KR20210086840A - Method for alerting when surrounding situation of car is dangerous situation by driving guide, and device using the same - Google Patents

Abstract

In accordance with the present invention, a method for determining whether a surrounding situation of a vehicle is risky, and generating a driving guide to issue an alarm, comprises the steps of: (a) inputting, by a driving situation determining and guiding device, a surrounding image to an image analysis module to allow an image analysis module to analyze the surrounding image so as to output surrounding environmental information including information on at least one object existing around a vehicle when the surrounding image of the vehicle is obtained from at least one camera mounted on the vehicle; (b) inputting, by the driving situation determining and guiding device, the surrounding environmental information to a risk situation determining module to allow the risk situation determining module to perform deep-learning calculation on the surrounding environmental information so as to output predicted values as the probability to correspond to a plurality of preset risk situation categories, and determining whether a surrounding situation of the vehicle corresponds to a specific risk situation category with reference to the predicted values; and (c) allowing, by the driving situation determining and guiding device, a driving guide generating module to generate at least one of specific visual driving guide information and specific voice driving guide information corresponding to the specific risk situation category so as to enable a driver of the vehicle to recognize a specific risk situation corresponding to the specific risk situation category when it is determined that the surrounding situation of the vehicle corresponds to the specific risk situation category among the preset risk situation categories.

Description

Method for judging whether the surrounding situation of a car is dangerous and generating a driving guide to give an alert, and a device using the same

The present invention relates to a method for determining whether a surrounding situation of a vehicle is a dangerous situation, generating a driving guide and giving an alarm, and an apparatus using the same, and more particularly, to a surrounding video image obtained from at least one camera mounted on the vehicle. to generate information on the surrounding environment around the car by analyzing the data, and deep learning calculations on the generated surrounding environment information to output a predicted value as a probability corresponding to each of a plurality of preset risk situation categories, based on this, correspond to the surrounding environment of the car It relates to a method for judging a dangerous situation and an apparatus using the same.

With the recent development of automobile technology, driver assistance technologies that support drivers to drive safely have been developed and installed in automobiles. As a result, people with relatively poor perception and slow reaction speed, such as elderly drivers, can also drive. This became possible.

However, since these driver assistance technologies are technologies that assist driving and do not completely replace driving, there is still a need for drivers to accurately understand the driving situation, especially for older people who have relatively poor perception and reaction speed. Drivers, etc., are more likely to be unable to identify and respond to a dangerous situation when the vehicle is in danger, so there is still a possibility that it will eventually lead to an accident.

In the preceding document, Korean Patent Publication No. 10-0997412, when it is determined that the vehicle is being driven dangerously, the image captured by the camera mounted on the vehicle is analyzed to determine the separation distance from the vehicle in front and the adjacent vehicle in the left and right lanes. Thus, an invention that can determine the cause of dangerous driving is disclosed. However, in the prior literature, there is a method that can determine whether the surrounding situation of the vehicle corresponds to a specific dangerous situation from the surrounding image image obtained from the camera mounted on the vehicle, and a predetermined driving guide is generated for the driver based on the result. The contents of the method that can be provided are not disclosed.

Therefore, by using a predetermined algorithm to determine whether the surrounding environment of the car is a dangerous situation and what kind of dangerous situation it is, and based on the result, the driving guide information is provided to the driver to make the driver feel that the surrounding situation of the car is dangerous. There is still a need for the technology to accurately recognize the situation and to respond to it.

Accordingly, an object of the present invention is to solve all of the above problems.

In addition, the present invention analyzes the surrounding situation image obtained from the camera mounted on the vehicle in order to generate surrounding situation information for determining whether the surrounding situation of the vehicle is a dangerous situation to detect at least one object included therein. Another purpose is to provide a way to

Another object of the present invention is to provide a method capable of determining a dangerous situation corresponding to the surrounding environment of a vehicle by receiving surrounding situation information.

In addition, the present invention is to provide a method of generating at least one of voice driving guide information and visual driving guide information corresponding thereto when the surrounding environment of the vehicle is determined to be a dangerous situation, and using it to provide a method for alerting the driver for other purposes.

Another object of the present invention is to provide a convolutional neural network capable of more accurately and quickly performing real-time object detection or real-time instance segmentation on an input surrounding image.

In order to achieve the object of the present invention as described above and to realize the characteristic effects of the present invention to be described later, the characteristic configuration of the present invention is as follows.

According to one aspect of the present invention, there is provided a method for determining whether a surrounding situation of a vehicle is a dangerous situation, generating a driving guide and giving an alarm, (a) a video image of the surrounding area of the vehicle from at least one camera mounted on the vehicle When is obtained, the driving situation determination and guide device inputs the surrounding video image to the video analysis module to cause the video analysis module to analyze the surrounding video image, and information on at least one object existing in the vicinity of the vehicle outputting surrounding environment information including; (b) the driving situation determination and guide device inputs the surrounding environment information to the dangerous situation determination module, and allows the dangerous situation determination module to perform deep learning operations on the surrounding environment information to correspond to each of a plurality of preset dangerous situation categories outputting predicted values as a probability of becoming the vehicle, and determining whether the surrounding situation of the vehicle corresponds to a specific dangerous situation category with reference to the predicted values; and (c) when the driving situation determination and guide device determines that the surrounding situation of the vehicle corresponds to the specific dangerous situation category among the plurality of preset dangerous situation categories, causes the driving guide generation module to cause the specific dangerous situation generating at least one of specific visual driving guide information and specific voice driving guide information corresponding to the category so that the driver of the vehicle can recognize a specific dangerous situation corresponding to the specific dangerous situation category; A method comprising:

As an example, a plurality of audio guide templates in a predetermined database - each of the audio guide templates has at least one slot, which is a part whose contents can be changed and input according to circumstances, and in each of the slots, the Any one of location information, size information, movement status information, and type information corresponding to each of at least one object included in the surrounding environment information may be inputted - classified and stored according to a plurality of preset dangerous situation categories Each of the plurality of preset dangerous situation categories is characterized in that it further includes information on at least one dangerous condition that can be determined as each dangerous situation, wherein (c) In the step, the driving situation determination and guide device causes the driving guide generation module to (i) information on the specific dangerous situation category, (ii) information on a specific dangerous condition corresponding to the specific dangerous situation category, and (iii) with reference to information on a specific voice guide template corresponding to the specific dangerous situation category, each of the slots included in the specific voice guide template is input to generate the completed specific voice driving guide information; There is provided a method characterized in that the generated specific voice driving guide information is directly provided to the driver or transmitted to a predetermined driver's terminal and supported to be provided.

As an example, the driving guide generating module is characterized in that it is interlocked with a predetermined text-to-speech (TTS) engine, and in step (c), the driving situation determination and guide device includes the driving guide generating module a process of generating specific TTS data by applying the TTS engine to the generated specific voice driving guide information to directly provide specific voice information reproducing the specific TTS data to the driver; or A method is provided, characterized in that it is transmitted to a predetermined driver's terminal and supported to be provided.

As an example, each of the plurality of preset dangerous situation categories is characterized in that it further includes information on at least one dangerous condition that can be determined as each dangerous situation, and in step (b), the The driving situation determination and guide device causes the dangerous situation determination module to (i) the specific dangerous situation category information, (ii) information on a specific dangerous condition corresponding to the specific dangerous situation category, and (iii) the surrounding environment A process of specifying a risk object corresponding to the risk condition among at least one of the objects included in the surrounding video image with reference to information and generating risk factor information including information on the risk object is additionally performed A method is provided, characterized in that

As an example, in the step (c), the driving situation determination and guide device causes the driving guide module to receive the risk factor information, so that the surrounding image image corresponds to the coordinates corresponding to each of the dangerous objects To generate an image additionally displayed with a preset danger guide signal as the specific visual driving guide information, and to directly provide the generated specific visual driving guide information to the driver or transmit it to a predetermined driver terminal to support the provision A method is provided, characterized in that

As an example, before the step (a), a predetermined dangerous situation determination module learning device, (i) each of at least one first learning surrounding video image obtained from the camera mounted on the vehicle or prepared separately - the first 1 Each of the surrounding image images for learning includes information about a specific correct dangerous situation category corresponding to each of the first surrounding image images among the preset dangerous situation categories as the first GT (Ground Truth) - as learning data , cause the image analysis module to receive each of the first surrounding image images for learning and output first surrounding environment information for learning as a result of analysis, and use the outputted first surrounding environment information for learning to determine the dangerous situation module or (ii) at least one piece of separately prepared second learning surrounding environment information - each of the second learning surrounding environment information includes information on a specific correct dangerous situation category corresponding to the second GT ( Including as ground truth) as learning data, characterized in that learning for the dangerous situation determination module is performed, and the learning of the dangerous situation determination module is performed by the learning device of the dangerous situation determination module, (i ) Let the dangerous situation determination module receive the first learning surrounding environment information and perform the predetermined deep learning operation, and the first as a probability corresponding to each of the plurality of preset dangerous situation categories output as a result A process of optimizing a plurality of parameters included in the dangerous situation determination module is performed so that the difference is minimized by comparing the information on the learning prediction values with the first GT, or (ii) the dangerous situation determination module causes the first GT 2 Receive information on the surrounding environment for learning to perform the predetermined deep learning operation, and output as a result information on second prediction values for learning as a probability corresponding to each of the plurality of preset risk situation categories, the second Multiple included in the dangerous situation judgment module to minimize the difference compared to the GT There is provided a method characterized in that it is achieved by performing a process of optimizing the parameters.

As an example, in step (a), the driving condition determination and guide device causes the image analysis module to detect each of the at least one object included in the surrounding image image using a predetermined algorithm, and the detection There is provided a method for outputting information including at least some of position information, size information, movement status information, and type information of each object as the surrounding environment information.

As an example, before the step (a), the predetermined image analysis module learning apparatus may perform the predetermined real-time object detection algorithm or the predetermined real-time instance segmentation with respect to the image analysis module. time instance segmentation) algorithm, characterized in that the learning of the image analysis module is obtained by the image analysis module learning apparatus from the camera mounted on the vehicle, or at least one separately prepared A second surrounding image image for learning - Each of the second surrounding image images for learning includes correct object information for each of at least one learning object included in each as a third GT (Ground Truth) - as learning data In this way, the image analysis module receives the second learning peripheral image image and analyzes it using the predetermined real-time object detection algorithm or the predetermined real-time instance segmentation algorithm, and as a result, There is provided a method characterized by performing a process of optimizing a plurality of parameters included in the image analysis module so that the difference is minimized by comparing the information on the third GT.

As an example, the driving situation determination and guidance device causes the image analysis module to use a predetermined real-time object detection algorithm or a predetermined real-time instance segmentation algorithm to determine the surrounding Detecting each of the at least one object included in the video image or performing instance segmentation of the surrounding video image, wherein the step (a) includes: (a1) When the surrounding video image is obtained, determining the driving situation and the guide device causes the image analysis module to input the peripheral image image to the backbone block of ResNET, and causes the backbone block to sequentially perform a convolution operation on the peripheral image image to down-sample the first down-sampling feature map to the second m - wherein m is an integer greater than or equal to 2 - outputting a down-sampling feature map; (a2) the driving situation determination and guidance device causes the image analysis module to input a specific down-sampling feature map to the first (1*1) convolutional layer to the first (1*1) convolutional layer (1*1) convolution operation on the specific down-sampling feature map to generate a first feature map with an adjusted number of channels, and expand the first feature map to (1*r) - where r is an integer greater than or equal to 1 - (k*k) with a ratio - where k is an integer greater than or equal to 2 - a first (k*k) convolutional layer including a kernel to (n*r) - where n is an integer greater than or equal to 2 - (k) with an extension ratio *k) each input to an nth (k*k) convolutional layer including a kernel to cause each of the first (k*k) convolutional layer to the nth (k*k) convolutional layer to form the first The channels of the feature map are divided into at least two groups, and each of the first feature maps corresponding to the at least two groups is divided into a (k*k) convolution operation using an (1*r) expansion ratio to (n* r) a first process for generating a 2_1 th feature map to a 2_n th feature map by performing a (k*k) convolution operation using an extension ratio, and a 2_1 (1) *1) Convolutional layers to 2_n (1*1) convolutional layers are input to each of the 2_1 (1*1) convolutional layers to the 2_n (1*1) convolutional layers. A (1*1) convolution operation is performed on the 2_1 feature map to the 2_nth feature map to generate 3_1 to 3_n feature maps with the number of channels adjusted, and the 3_1 to 3_n feature maps are convolutional. Kate and input as a third (1*1) convolution layer to cause the third (1*1) convolution layer to convert the concatenated 3_1 to 3_n feature maps (1*1) Convolution is performed to generate a fourth feature map with the number of channels adjusted, and the specific down-sampling feature map and the fourth feature map are combined. Through a second process of concatenating to generate a transformed down-sampling feature map, the m-th down-sampling feature map to (mj) - wherein j is an integer greater than or equal to 1 and less than m - is added to each of the down-sampling feature maps generating an m-th transformed down-sampling feature map to an (mj)-th transformed down-sampling feature map by applying the first process and the second process; and (a3) the driving situation determination and guide device causes the image analysis module to input the m-th transformed down-sampling feature map to the (mj)-th transformed down-sampling feature map into a feature pyramid network to input the feature The pyramid network generates an m-th up-sampling feature map to (mj)-th up-sampling feature map through a deconvolution operation with reference to the m-th transformed down-sampling feature map to the (mj)-th transformed down-sampling feature map. input the (mj) up-sampling feature map to an object detection network or an instance segmentation network to cause the object detection network to detect an object on the surrounding video image, or to cause the instance segmentation network to detect the surrounding video image causing instance segmentation; A method comprising:

As an example, in step (a3), the image analysis module causes the feature pyramid network to down-sample the m-th up-sampling feature map to generate an (m+j)-th up-sampling feature map, (mj) an up-sampling feature map to the (m+j)-th up-sampling feature map are additionally input to the instance segmentation network, so that the instance segmentation network causes the (mj)-th up-sampling feature map to the (m+j)-th up-sampling feature map ) A method for instance segmenting the surrounding video image with further reference to an up-sampling feature map is provided.

As an example, in step (a3), the image analysis module causes the feature pyramid network to up-sample the feature map with i-th-i is an integer greater than or equal to (m-(j+1)) and less than or equal to m-up-sampling to generate a specific up-sampling feature map, and concatenate the specific up-sampling feature map with the (i-1)-th down-sampling feature map corresponding thereto to generate the (i-1)-th up-sampling feature map this is provided

In addition, according to another aspect of the present invention, there is provided an apparatus for judging whether a surrounding situation of a vehicle is a dangerous situation, and generating and alerting a driving guide, comprising: at least one memory for storing instructions; and at least one processor configured to execute the instructions. including, wherein the processor (I) when the surrounding video image of the vehicle is obtained from at least one camera mounted on the vehicle, inputs the surrounding video image to an image analysis module to cause the video analysis module to cause the surrounding image a process of analyzing an image to output surrounding environment information including information on at least one object existing in the vicinity of the vehicle; (II) input the surrounding environment information to the dangerous situation determination module so that the dangerous situation determination module performs a deep learning operation on the surrounding environment information to output predicted values as probabilities corresponding to each of a plurality of preset dangerous situation categories, a process of determining whether the surrounding situation of the vehicle corresponds to a specific dangerous situation category with reference to the predicted values; and (III) when it is determined that the surrounding situation of the vehicle corresponds to the specific dangerous situation category among the plurality of preset dangerous situation categories, causes the driving guide generation module to cause specific visual driving guide information corresponding to the specific dangerous situation category and generating at least one of specific voice driving guide information so that the driver of the vehicle can recognize a specific dangerous situation corresponding to the specific dangerous situation category. There is provided an apparatus for performing the

As an example, a plurality of audio guide templates in a predetermined database - each of the audio guide templates has at least one slot, which is a part whose contents can be changed and input according to circumstances, and in each of the slots, the Any one of location information, size information, movement status information, and type information corresponding to each of at least one object included in the surrounding environment information may be inputted - classified and stored according to a plurality of preset dangerous situation categories Each of the plurality of preset dangerous situation categories, characterized in that it further includes information on at least one dangerous condition that can be determined as each dangerous situation, wherein (III) In the process, the processor causes the driving guide generation module to (i) information on the specific dangerous situation category, (ii) information on a specific dangerous condition corresponding to the specific dangerous situation category, and (iii) the specific dangerous situation category. With reference to information on a specific voice guide template corresponding to a dangerous situation category, each of the slots included in the specific voice guide template is input to generate the completed specific voice driving guide information, and the generated specific voice driving guide information is generated. There is provided an apparatus characterized in that the guide information is directly provided to the driver or is transmitted to a predetermined driver's terminal and supported to be provided.

As an example, the driving guide generating module is characterized in that it is linked with a predetermined text-to-speech (TTS) engine, and in the process (III), the processor causes the driving guide generating module to generate the generated A process of generating specific TTS data by applying the TTS engine to specific voice driving guide information is additionally performed, so that specific voice information reproduced from the specific TTS data is directly provided to the driver or to a predetermined driver terminal An apparatus is provided, characterized in that it supports to be provided by transmitting.

As an example, each of the plurality of preset dangerous situation categories is characterized in that it further includes information on at least one dangerous condition that can be determined as each dangerous situation, and in the process (II), the The processor is configured to cause the dangerous situation determination module to refer to (i) the specific dangerous situation category information, (ii) information about a specific dangerous condition corresponding to the specific dangerous situation category, and (iii) the surrounding environment information. Further performing a process of specifying a risk object corresponding to the risk condition among at least one of the objects included in the surrounding image image and generating risk factor information including information on the risk object A device is provided.

As an example, in the process (III), the processor causes the driving guide module to receive the risk factor information, so as to correspond to the coordinates corresponding to each of the danger objects with respect to the surrounding image image. An image additionally displayed with a signal is generated as the specific visual driving guide information, and the generated specific visual driving guide information is directly provided to the driver or transmitted to a predetermined driver terminal and supported to be provided A device is provided.

As an example, before the (I) process, a predetermined dangerous situation determination module learning device is configured to (i) each of at least one first learning surrounding video image obtained from the camera mounted on the vehicle or prepared separately - the first 1 Each of the surrounding image images for learning includes information about a specific correct dangerous situation category corresponding to each of the first surrounding image images among the preset dangerous situation categories as the first GT (Ground Truth) - as learning data , cause the image analysis module to receive each of the first surrounding image images for learning and output first surrounding environment information for learning as a result of analysis, and use the outputted first surrounding environment information for learning to determine the dangerous situation module or (ii) at least one piece of separately prepared second learning surrounding environment information - each of the second learning surrounding environment information includes information on a specific correct dangerous situation category corresponding to the second GT ( Including as ground truth) as learning data, characterized in that learning for the dangerous situation determination module is performed, and the learning of the dangerous situation determination module is performed by the learning device of the dangerous situation determination module, (i ) Let the dangerous situation determination module receive the first learning surrounding environment information and perform the predetermined deep learning operation, and the first as a probability corresponding to each of the plurality of preset dangerous situation categories output as a result A process of optimizing a plurality of parameters included in the dangerous situation determination module is performed so that the difference is minimized by comparing the information on the learning prediction values with the first GT, or (ii) the dangerous situation determination module causes the first GT 2 Receive information on the surrounding environment for learning to perform the predetermined deep learning operation, and output as a result information on second prediction values for learning as a probability corresponding to each of the plurality of preset risk situation categories, the second In order to minimize the difference compared to the GT, the An apparatus is provided, characterized in that it is achieved by performing a process of optimizing a plurality of parameters.

As an example, in the process (I), the processor causes the image analysis module to detect each of the at least one object included in the surrounding image image using a predetermined algorithm, and a position of each detected object There is provided an apparatus for outputting information including at least some of information, size information, movement status information, and type information as the surrounding environment information.

As an example, before the (I) process, a predetermined image analysis module learning apparatus may perform the predetermined real-time object detection algorithm or the predetermined real-time instance segmentation with respect to the image analysis module. time instance segmentation) algorithm, characterized in that the learning of the image analysis module is obtained by the image analysis module learning apparatus from the camera mounted on the vehicle, or at least one separately prepared A second surrounding image image for learning - Each of the second surrounding image images for learning includes correct object information for each of at least one learning object included in each as a third GT (Ground Truth) - as learning data In this way, the image analysis module receives the second learning peripheral image image and analyzes it using the predetermined real-time object detection algorithm or the predetermined real-time instance segmentation algorithm, and as a result, There is provided an apparatus characterized in that by performing a process of optimizing a plurality of parameters included in the image analysis module so that the difference is minimized by comparing the information on the third GT.

As an example, the processor causes the image analysis module to use a predetermined real-time object detection algorithm or a predetermined real-time instance segmentation algorithm to be included in the surrounding video image. Detecting each of the at least one object or performing instance segmentation of the surrounding video image, wherein the (I) process includes: (I-1) When the surrounding video image is obtained, the processor performs the image analysis The first down-sampling feature map to mth down-sampled by a module inputting the surrounding video image to the backbone block of ResNET and causing the backbone block to sequentially perform a convolution operation on the surrounding video image - wherein m is an integer greater than or equal to 2 - a sub-process for outputting a down-sampling feature map; (I-2) the processor causes the image analysis module to input a specific down-sampling feature map as a first (1*1) convolutional layer to cause the first (1*1) convolutional layer to cause the specific downsampling feature map (1*1) convolution operation on the down-sampling feature map to generate a first feature map with the number of channels adjusted, and (1*r) - where r is an integer greater than or equal to 1 - has an expansion ratio (k*k) - wherein k is an integer greater than or equal to 2 - a first (k*k) convolutional layer including a kernel to (n*r) - where n is an integer greater than or equal to 2 - (k*k) with an extension ratio Each of the first (k*k) convolutional layer to the nth (k*k) convolutional layer is inputted to the nth (k*k) convolutional layer including the kernel, so that each of the first feature map The channels are divided into at least two groups, and each of the first feature maps corresponding to the at least two groups is expanded by a (k*k) convolution operation or (n*r) by a (1*r) extension ratio. A first process for generating a 2_1 th feature map to a 2_n th feature map by performing a (k*k) convolution operation by a ratio, and a 2_1 (1*1) The 2_1 (1*1) convolution layer to the 2_n (1*1) convolution layer are inputted as the 2_n (1*1) convolutional layer to each of the 2_1 feature maps. to (1*1) convolution operation on the 2_n-th feature map to generate 3_1 to 3_n feature maps with the number of channels adjusted, and concatenate the 3_1 to 3_n feature maps to By inputting the third (1*1) convolutional layer, the third (1*1) convolution layer performs a (1*1) convolution operation on the concatenated 3_1 to 3_n feature maps. to generate a fourth feature map with an adjusted number of channels, and concatenate the specific down-sampling feature map and the fourth feature map through a second process of generating a transformed down-sampling feature map by performing a second process, the m-th down-sampling feature map to (mj) - the j is an integer greater than or equal to 1 and less than m - the first process in each of the down-sampling feature maps and a sub-process for generating an m-th transformed down-sampling feature map to an (mj)-th transformed down-sampling feature map by applying the second process; and (I-3) the processor, causing the image analysis module to input the m-th transformed down-sampling feature map to the (mj)-th transformed down-sampling feature map into the feature pyramid network to the feature pyramid network. to generate an m-th up-sampling feature map to (mj)-th up-sampling feature map through a deconvolution operation with reference to the m-th transformed down-sampling feature map to the (mj)-th transformed down-sampling feature map, input the (mj)th up-sampling feature map to an object detection network or an instance segmentation network to cause the object detection network to detect an object on the surrounding video image or to cause the instance segmentation network to instance segment the surrounding video image subprocess; There is provided an apparatus for performing the

As an example, in the process (I-3), the image analysis module causes the feature pyramid network to down-sample the m-th up-sampling feature map to generate a (m+j)-th up-sampling feature map, The (mj)th up-sampling feature map to the (m+j)th up-sampling feature map are additionally input to the instance segmentation network to cause the instance segmentation network to make the (mj)th up-sampling feature map to the (m)th up-sampling feature map. +j) An apparatus is provided, characterized in that the surrounding video image is instance-segmented by further referring to an up-sampling feature map.

As an example, in step (I-3), the image analysis module causes the feature pyramid network to generate an up-sampling feature map for i-th i is an integer greater than or equal to (m-(j+1)) and less than or equal to m. up-sampling to generate a specific up-sampling feature map, and concatenating the specific up-sampling feature map and a corresponding (i-1)-th down-sampling feature map to generate an (i-1)-th up-sampling feature map There is provided an apparatus, characterized in that

In addition to this, a computer-readable recording medium for recording a computer program for executing the method of the present invention is further provided.

According to the present invention, the following effects are obtained.

The present invention is capable of detecting at least one object included in the surrounding situation image obtained from a camera mounted on the vehicle to generate surrounding situation information for determining whether the surrounding situation of the vehicle is a dangerous situation. There is an effect that can provide a method.

In addition, the present invention has an effect of providing a method for determining a dangerous situation corresponding to the surrounding environment of the vehicle by receiving the surrounding situation information.

In addition, the present invention can provide a method of generating at least one of voice driving guide information and visual driving guide information corresponding to the surrounding environment of the vehicle as a dangerous situation, and alerting the driver using the generated information. there is an effect

In addition, the present invention is effective in providing a convolutional neural network capable of more accurately and quickly performing real-time object detection or real-time instance segmentation with respect to an input surrounding image.

1 is a diagram schematically illustrating a driving situation determination and guide device for determining whether a surrounding situation of a vehicle is a dangerous situation, generating a driving guide, and giving an alarm, according to an embodiment of the present invention.
FIG. 2 is a diagram schematically illustrating an entire system for determining whether a surrounding situation of a vehicle is a dangerous situation, generating a driving guide, and providing an alarm according to an embodiment of the present invention.
3 is a method of performing real-time object detection or real-time instance segmentation on a surrounding video image obtained from a camera mounted on a vehicle according to an embodiment of the present invention; is a diagram schematically showing
4 is a diagram for performing real-time object detection or real-time instance segmentation on a surrounding video image obtained from a camera mounted on a vehicle according to an embodiment of the present invention; It is a diagram schematically illustrating a method of generating a feature map of a convolutional neural network.
5 is a diagram for performing real-time object detection or real-time instance segmentation on a surrounding video image obtained from a camera mounted on a vehicle according to an embodiment of the present invention; It is a diagram schematically showing a state in which a convolution operation is performed by different expansion ratios in a method of generating a feature map of a convolutional neural network.
6A is a flowchart schematically illustrating a process in which learning of a dangerous situation determination module is performed using an image of a surrounding for learning as learning data, according to an embodiment of the present invention.
6B is a flowchart schematically illustrating a process in which learning of a dangerous situation determination module is performed using, as learning data, surrounding environment information for learning separately prepared according to an embodiment of the present invention.
7 is a flowchart schematically illustrating a process in which learning of an image analysis module is performed by using an image of a surrounding image for learning as learning data, according to an embodiment of the present invention.
8A is a view illustrating an image of a surrounding area in a case in which a bus stop and a stopped bus exist in front of a right front of a driving lane according to an embodiment of the present invention.
FIG. 8B is a view exemplarily showing a surrounding image image when a large truck is driving in a right lane of the driving lane according to an embodiment of the present invention.
FIG. 8C is a view illustrating an image of a surrounding area when a taxi is driving in front of a driving lane according to an embodiment of the present invention.
FIG. 8D is a view exemplarily illustrating a surrounding image image in a case in which bicycles and motorcycles exist on the right side of the driving lane according to an embodiment of the present invention.
FIG. 8E is a view exemplarily illustrating a surrounding image image when a plurality of cars are parked on the right side of the driving lane according to an embodiment of the present invention.
8F is an exemplary view of a surrounding video image corresponding to a situation in which many vehicles are driving in the left lane in a situation where it is impossible to change lanes due to the presence of other vehicles in the front and right lanes of the driving lane according to an embodiment of the present invention; It is a drawing showing
FIG. 8G is a view exemplarily showing a surrounding video image corresponding to a situation in which a four-way intersection and a crosswalk exist in front, according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS [0010] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS [0010] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS [0023] Reference is made to the accompanying drawings, which show by way of illustration specific embodiments in which the invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the present invention. It should be understood that the various embodiments of the present invention are different but need not be mutually exclusive. For example, certain shapes, structures, and characteristics described herein with respect to one embodiment may be embodied in other embodiments without departing from the spirit and scope of the invention.

In addition, it should be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the present invention. Accordingly, the detailed description set forth below is not intended to be taken in a limiting sense, and the scope of the invention, if properly described, is limited only by the appended claims, along with all scope equivalents to those claimed. Like reference numerals in the drawings refer to the same or similar functions throughout the various aspects.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings in order to enable those of ordinary skill in the art to easily practice the present invention.

1 is a diagram schematically illustrating a driving situation determination and guide device for determining whether a surrounding situation of a vehicle is a dangerous situation, generating a driving guide, and giving an alarm, according to an embodiment of the present invention.

Referring to FIG. 1 , the driving situation determination and guide device 10 for determining whether the surrounding situation of the vehicle is a dangerous situation, generating a driving guide, and giving an alert may include a memory 11 and a processor 12 . . At this time, the memory 11 may store the instructions of the processor 12. Specifically, the instructions are generated for the purpose of causing the driving situation determination and guide device 10 to function in a specific manner. , stored in a computer usable or computer readable memory that may be directed to a computer or other programmable data processing equipment. The instructions may perform processes for performing the functions described herein.

In addition, the processor 12 may include a hardware configuration such as a micro processing unit (MPU) or a central processing unit (CPU), a cache memory, and a data bus. It may also include an operating system, a software configuration of an application that performs a specific purpose.

Next, the driving situation determination and guide device 10 may be linked with a database (not shown) including information used to determine whether the surrounding situation of the vehicle is a dangerous situation, and generate a driving guide to alert. At this time, the database is a flash memory type (flash memory type), hard disk type (hard disk type), multimedia card micro type (multimedia card micro type), card type memory (eg SD or XD memory), RAM ( At least one of Random Access Memory (RAM), Static Random Access Memory (SRAM), ReadOnly Memory (ROM), Electrically Erasable Programmable ReadOnly Memory (EEPROM), Programmable ReadOnly Memory (PROM), magnetic memory, magnetic disk, and optical disk It may include any type of storage medium, but is not limited thereto, and may include any medium capable of storing data. In addition, the database may be installed inside the driving situation judgment and guide device 10 to transmit data or record data received, which may vary depending on the implementation conditions of the invention.

Refer to a separate drawing (FIG. 2) for the entire system as an example for determining whether the surrounding situation of the vehicle 1 is a dangerous situation using such a driving situation determination and guide device 10, and generating a driving guide to alert. It is explained as follows.

FIG. 2 is a diagram schematically illustrating an entire system for determining whether a surrounding situation of a vehicle is a dangerous situation, generating a driving guide, and providing an alarm according to an embodiment of the present invention.

Referring to FIG. 2 , the process of determining whether the driving situation determination and guidance device 10 is a dangerous situation around the vehicle 1 and generating and alerting the driving guide is performed by first using at least one camera mounted on the vehicle ( 20), it starts with acquiring a surrounding video image of the corresponding vehicle 1 . At this time, the driving situation determination and guide device 10 may include an image analysis module 30 , a dangerous situation determination module 40 , and a driving guide generation module 50 , respectively, and allows each module to obtain A predetermined process may be performed by receiving a video image or data output from another module. In addition, although it is shown that the driving situation determination and guide device 10 includes each module in FIG. 2 , each module is a separate device and is interlocked with the driving situation determination and guide device 10 . Also, this may be modified according to the operating conditions of the invention. And, when the voice driving guide that can be generated in the practice of the present invention is converted to voice in a text-to-speech (TTS) method and provided to the driver, a TTS engine 51 for this purpose is additionally included to guide the driving. It may be linked with the generation module 50 .

Next, the driving situation determination and guide device 10 inputs the acquired surrounding image image to the image analysis module 30, and causes the corresponding image analysis module 30 to analyze the input surrounding image image of the vehicle 1 It is possible to output surrounding environment information including information on at least one object existing around. At this time, the object may include other vehicles, moving objects, pedestrians, and obstacles that may affect the vehicle while driving or parked around the vehicle. A curb as a boundary between a marked lane, a roadway and a sidewalk may also be included, and various existences around the vehicle may be the subject according to the implementation conditions of the invention, and the surrounding environment information is provided by the driving situation determination and guide device 10 The image analysis module 30 detects each of the at least one object included in the surrounding video image by using a predetermined algorithm, and at least one of location information, size information, movement status information, and type information of each detected object. Information including a part may be output as the surrounding environment information.

At this time, the image analysis module 30 may use a convolutional neural network (CNN)-based real-time object detection algorithm or real-time instance segmentation algorithm to analyze the surrounding image image obtained from the camera 20, , which will be described in more detail as follows.

Referring to FIG. 3 , when the surrounding video image is acquired, the video analysis module 30 inputs the surrounding video image to the backbone block 150 of ResNET, and causes the backbone block 150 to sequentially perform a convolution operation to download the image. The sampled first down-sampling feature map to the m-th down-sampling feature map are output. In this case, m may be an integer of 2 or more. Also, FIG. 2 shows only the fifth down-sampling feature map for convenience of explanation.

Next, the image analysis module 30 inputs each of the m-th down-sampling feature map to the (mj)-th down-sampling feature map to each bottleneck block 200 so that each bottleneck block 200 causes the m-th down-sampling feature map. The transformed down-sampling feature map to the (mj)-th transformed down-sampling feature map are generated.

That is, the operation of the bottleneck block 200 will be described with reference to FIG. 4 as follows.

The image analysis module 30 inputs a specific down-sampling feature map corresponding to each of the m-th down-sampling feature map to the (m-j)-th down-sampling feature map as the first (1*1) convolutional layer 210 .

Then, the first (1*1) convolutional layer 210 may perform a (1*1) convolution operation on a specific down-sampling feature map to generate a first feature map with an adjusted number of channels.

In this case, in order to reduce the amount of computation in the subsequent process, the number of channels of the first feature map may be smaller than that of a specific downsampling feature map. For example, if the number of channels of a specific down-sampling feature map is C, the number of channels of the first feature map may be C/S. Here, C may be a constant of 1 or more, S may be an integer of 2 or more, and C may be a multiple of S.

Next, the image analysis module 30 converts the first feature map to the first (k*k) convolutional layers 220-1 to (220-1) including the (k*k) kernel having the (1*r) extension ratio. n*r) is input to the n-th (k*k) convolutional layer 220-n including a (k*k) kernel having an extension ratio, respectively. Here, r may be an integer of 1 or more, k may be an integer of 1 or more, and n may be an integer of 2 or more. For reference, in FIG. 2, r is 1 and k is 3

Then, each of the first (k*k) convolutional layer 220-1 to the nth (k*k) convolutional layer 220-n divides the channels of the first feature map into at least two groups, , (k*k) convolution operation by (1*r) extension ratio to (k*k) convolution operation by (n*r) extension ratio for each of the first feature maps corresponding to the at least two divided groups A second_1st feature map to a 2nd_nth feature map may be generated by performing a root solution operation. In this case, the first (k*k) convolutional layer 220-1 to the nth (k*k) convolutional layer 220-n may share a convolution parameter.

That is, each of the first (k*k) convolutional layer 220-1 to the nth (k*k) convolutional layer 220-n does not perform a convolution operation on the first feature map at once, In order to minimize the amount of calculation, the channels of the first feature map may be divided into at least two groups, and a convolution operation may be performed for each of the divided at least two groups.

At this time, the number of channels in each of the 2_1 feature maps to the 2_n feature maps is maintained to be the same as the number of channels in the first feature map, and through this, the number of channels is divided for parallel processing and a small number of channels is convolved. By doing so, it is possible to prevent some characteristic information from being lost. For example, if the number of channels of a specific down-sampling feature map is C and the number of channels of the first feature map is C/S, the number of channels in each of the 2_1 feature maps to 2_n feature maps may be C/S. have.

Meanwhile, referring to FIG. 5 , the (k*k) convolution operation by the (1*r) expansion ratio to the (k*k) convolution operation by the (n*r) expansion ratio are performed according to the expansion ratio ( The convolution operation may be performed by selecting pixels on a feature map to which the convolution operation is applied using a kernel of size k*k). For reference, FIG. 5 schematically illustrates a state in which a convolution operation is performed in a state where k is 3 and r is 1.

As can be seen from FIG. 5 , the expansion ratio may be the distance between neighboring pixels of the feature map for convolution operation, and when the expansion ratio is 1 as in (a), the distance is 1, that is, After selecting neighboring pixels according to the kernel size, a convolution operation is performed. As in (b), when the expansion ratio is 2, pixels with a distance of 2 are selected according to the kernel size and then the convolution operation is performed.

Referring again to FIG. 4 , the image analysis module 30 converts each of the 2_1 th feature map to the 2_n th feature map into a 2_1 (1*1) convolution layer 230-1 to 2_n (1*1) convolution. Input to the solution layer 230-n.

Then, each of the 2_1 (1*1) convolutional layer 230-1 to the 2_n (1*1) convolutional layer 230-n is a (1*) 1) The 3_1 to 3_n feature maps in which the number of channels are adjusted may be generated by performing a convolution operation. In this case, the 2_1 (1*1) convolution layer 230-1 to the 2_n (1*1) convolution layer 230 - n may share a convolution parameter.

In addition, the number of channels in the 3_1 feature map to the 3_n feature map may be adjusted to be equal to the number of channels in the specific down-sampling feature map.

Next, the image analysis module 30 concatenates the 3_1 to 3_n feature maps and inputs them to the third (1*1) convolutional layer 240 .

In this case, the image analysis module 30 concatenates the 3_1 th feature map to the 3_n feature map, then rearranges the concatenated 3_1 th feature map to the 3_n th feature map, and the rearranged 3_1 feature map to 3_n feature maps may be input as the third (1*1) convolutional layer 240 .

For example, if each of the 3_1 feature maps to 3_n feature maps is a tensor of a CHW volume having a channel C, a height H, and a width W, the image analysis module 30 may Concatenate to create a tensor of 3CHW volume. Then, after rearranging the 3CHW tensor in the C3HW order, the (3C)HW volume tensor may be input to the third (1*1) convolutional layer 240 .

Then, the third (1*1) convolutional layer 240 performs a (1*1) convolution operation on the concatenated 3_1 to 3_n feature maps to generate a fourth feature map with an adjusted number of channels. can

In this case, the number of channels of the fourth feature map may be adjusted to be equal to the number of channels of the specific down-sampling feature map. That is, when the 3_1 to 3_n feature maps having the same number of channels as the specific down-sampling feature map are concatenated, the number of channels may be 3C, and the third (1*1) convolutional layer 240 . The number of channels can be adjusted to be equal to a specific down-sampling feature map by a (1*1) convolution operation in .

Next, the image analysis module 30 may generate a converted down-sampling feature map by concatenating the specific down-sampling feature map and the fourth feature map.

Through this, the m-th down-sampling feature map to the (mj)-th transformed down-sampling feature map that is robust to scale variations corresponding to each of the m-th down-sampling feature map to the (mj)-th down-sampling feature map by the bottleneck block 200 . You can create a sampling feature map.

Referring back to FIG. 3 , the image analysis module 30 inputs the m-th transformed down-sampling feature map to the (mj)-th transformed down-sampling feature map into the feature pyramid network 300 to the feature pyramid network 300 . to generate an m-th up-sampling feature map to (mj)-th up-sampling feature map through a deconvolution operation with reference to the m-th transformed down-sampling feature map to the (mj)-th transformed down-sampling feature map, mj) Input the up-sampling feature map to the object detection network or the instance segmentation network 400 to cause the object detection network to detect an object on the image or to cause the instance segmentation network 400 to instance segment the image.

At this time, the image analysis module 30 causes the feature pyramid network 300 to down-sample the m-th up-sampling feature map to generate the (m+j)-th up-sampling feature map, and the (mj)-th up-sampling feature map. to (m+j) th up-sampling feature map is additionally input to the instance segmentation network 400 to cause the instance segmentation network 400 to generate (mj) th up-sampling feature map to (m+j) th up-sampling feature map For further reference, you can have the image instance segmented.

In addition, the image analysis module 30 causes the feature pyramid network 300 to up-sample the i-th up-sampling feature map to generate a specific up-sampling feature map, and the specific up-sampling feature map and the (i-1)th corresponding up-sampling feature map. ) by concatenating the down-sampling feature map to generate the (i-1)-th up-sampling feature map. In this case, i may be an integer greater than or equal to (m-(j+1)) and less than or equal to m.

At this time, the instance segmentation network 400 generates a prototype mask by referring to the (mj)th up-sampling feature map, and at the same time, referring to the (mj)th up-sampling feature map to the (m+j)th up-sampling feature map. to predict the coefficient to reflect each prototype for each region of interest. In addition, the instance segmentation network 400 may generate a final mask by linearly combining the prototype masks with reference to the predicted coefficients, and may instant-segment objects on the surrounding video image with reference to the generated final mask.

Through this, by generating a feature map that minimizes the processing speed and is robust to scale fluctuations through the bottleneck block and inputs it to the instance segmentation network 400, the instance segmentation network 400 can accurately recognize the object. This is improved and real-time instance segmentation becomes possible.

Referring back to FIG. 2 , the driving situation determination and guide device 10 inputs the surrounding environment information output from the image analysis module 30 to the dangerous situation determination module 40, and the corresponding dangerous situation determination module 40 to output the predicted values as the probability corresponding to each of a plurality of preset dangerous situation categories by deep learning operation on the input surrounding environment information, and whether the surrounding situation of the car 1 is a dangerous situation with reference to the output predicted values can be judged. Specifically, the driving situation determination guide device 10 may cause the dangerous situation determination module 40 to perform a predetermined deep learning operation on the input surrounding environment information, in which case the deep learning operation is used. It may be determined according to an algorithm. As an example, when a deep learning algorithm based on a CNN (Convolutional Neural Network) is used, the corresponding deep learning operation may be a convolutional operation. As a result, a predicted value as a probability corresponding to each of a plurality of preset dangerous situation categories may be output from the dangerous situation determination module 40, wherein a plurality of preset dangerous situation categories may be defined and set in advance, , as an example, 'a bus/taxi stop exists (C1)', 'a large vehicle exists in the front/right lane/left lane of the driving lane (C2)', 'a taxi exists in front of the driving lane (C3)' When the dangerous situation category of is preset, if the dangerous situation determination module 40 outputs predicted values of 0.9 for the C1 category, 0.2 for the C2 category, and 0.1 for the C3 category, C1 corresponding to the highest predicted value among them The category may be determined as a dangerous situation category corresponding to the surrounding environment information. In addition, when the predicted values for each of a plurality of preset dangerous situation categories are all below or below a predetermined reference value, it is determined that the surrounding situation of the vehicle 1 is not a dangerous situation, or it is determined that it is a new situation that is not a preset dangerous situation You may. As such, the output predicted value and a specific method for determining a specific dangerous situation category with reference thereto may be made differently by using a separate formula or the like according to the implementation conditions of the invention.

[Table 1] below is an example in which the plurality of preset dangerous situation categories can be determined.

Example of preset dangerous situation category 'There is a bus/taxi stop' 'A large vehicle exists in the front/right/left lane of the driving lane' 'There is a taxi ahead of the driving lane' 'Bicycles/motorcycles exist in front/rear on the right/left side of the driving lane' 'There are cars parked on the right/left side of the driving lane' 'There are traffic jams in the right/left lane of the driving lane' 'There is a four-way intersection in front of the driving lane' ...

As shown in [Table 1] above, by determining in advance the dangerous situations that the vehicle may face while driving, and determining a specific dangerous situation category corresponding to the surrounding situation image obtained from the camera 20 mounted on the vehicle 1, It may be possible to determine whether the surrounding situation of the vehicle is a dangerous situation and what kind of dangerous situation it corresponds to. At this time, a method as an example of determining a specific dangerous situation category from the surrounding situation image corresponding to each situation in [Table 1] will be described with reference to separate drawings ( FIGS. 8A to 8G ) as follows.

8A to 8G are diagrams exemplarily illustrating a case in which an image of a surrounding situation obtained from a camera 20 mounted on a vehicle 1 corresponds to a predetermined specific dangerous situation category, according to an embodiment of the present invention. .

In FIG. 8A, there is shown a surrounding image image corresponding to a situation in which a bus stop 812 and a stopped bus 811 exist in front of the right front of the driving lane on which the vehicle 1 is traveling. is input, the image analysis module 30 detects the object corresponding to the left lane 814 and the right curb 813 to determine the driving lane, and corresponds to the bus stop 812 and the bus 811 object can be detected. At this time, as shown in FIG. 8A , a bounding box is generated for each object according to an algorithm used for detecting each object, and coordinates and sizes of the respective objects are located in the surrounding video image. Information on the object may be acquired, and the type (class) corresponding to each object may be analyzed together, so that position information, size information, and type information of the corresponding object may be obtained respectively. In addition, with respect to the object, the object is tracked by analyzing the driving state information (speed, acceleration, etc.) of the vehicle 1 and information on the object in each of the corresponding time-by-time surrounding video images. It is also possible to obtain movement status information. As described above, when a plurality of objects included in the surrounding video image are detected and surrounding environment information including the corresponding information is output from the image analysis module 30, the driving situation determination and guide device 10 determines the dangerous situation. It is input to the module 40 so that the dangerous situation determination module can determine a specific dangerous situation category corresponding thereto, and the surrounding image image shown in FIG. 8a is a risk corresponding to 'there is a bus/taxi stop' in the end. It may be judged by the situation category.

Hereinafter, since the method of performing object detection for each of the peripheral video images shown in FIGS. 8B to 8G is similar to the case of FIG. 8A , a detailed description thereof will be omitted.

FIG. 8B shows an image of a surrounding situation corresponding to a situation in which the large vehicle 821 loaded with cargo in front of the right lane of the driving lane in which the vehicle 1 is traveling is driving. At this time, the image analysis module 30 may detect the left lane 822 object and the right lane 823 object to obtain information on the driving lane based on this, and detect the object corresponding to the large vehicle 821 . Thus, it can be determined that the surrounding situation of the vehicle 1 falls under the dangerous situation category of 'large vehicle is present in the front/right lane/left lane of the driving lane', and furthermore, 'large vehicle in front of the right lane of the driving lane It can also be judged more specifically, such as 'existence'.

FIG. 8C shows an image of a surrounding situation corresponding to a situation in which a taxi 831 exists in front of a driving lane in which the vehicle 1 is traveling. At this time, the image analysis module 30 detects each of the objects corresponding to the lanes 832 and 833 and the taxi 831 from the image of the surrounding situation, and generates and outputs information about the surrounding situation based on this. Finally, it can be determined that the surrounding situation of the vehicle 1 corresponds to the dangerous situation category of 'a taxi exists in front of the driving lane'.

FIG. 8D shows an image of a surrounding situation corresponding to a situation in which bicycles and motorcycles 841 and 842 exist on the right side of the driving lane in which the vehicle 1 is traveling. At this time, the image analysis module 30 detects each object corresponding to the lane 844 , the curb 843 , and the bicycles and motorcycles 841 and 842 from the image of the surrounding situation, and generates information about the surrounding situation based on this. Finally, by the dangerous situation determination module 40, it is determined that the surrounding situation of the vehicle 1 corresponds to the dangerous situation category of 'there is a bicycle/motorcycle in the front/rear side of the right/left side of the driving lane. In addition, it is possible to make a more specific judgment such as 'there are bicycles and motorcycles in front of the roadside on the right side of the driving lane'.

FIG. 8E shows an image of a surrounding situation corresponding to a situation in which a plurality of parked cars 851 and 852 exist on the right side of the driving lane on which the vehicle 1 is traveling. At this time, the image analysis module 30 detects each object corresponding to the lane 855 , the curb 854 , and the parked cars 851 , 852 , 853 from the surrounding situation image, and based on this, the surrounding situation By generating and outputting information, it is finally determined by the dangerous situation determination module 40 that the surrounding situation of the vehicle 1 corresponds to the dangerous situation category of 'there are cars parked on the right/left side of the driving lane'. In addition, it is also possible to make a more specific judgment such as 'there are cars parked on the right side of the driving lane'.

8f shows an image of a surrounding situation corresponding to a situation in which a plurality of cars are congested in the left lane in a state in which acceleration or right overtaking is impossible due to the presence of a vehicle in the right lane and in front of the driving lane in which the vehicle 1 is traveling. have. At this time, the image analysis module 30 from the image of the surrounding situation to the lanes 866 and 867, the front car 864, the car 865 in the right lane, and the cars 861, 862, 863 in the left lane. By detecting each of the corresponding objects, generating and outputting information about the surrounding situation based on this, the dangerous situation determination module 40 finally determines the surrounding situation of the vehicle 1 as 'the vehicle is congested in the right/left lane of the driving lane. It can be judged that it falls under the dangerous situation category of 'there are cars', and furthermore, more specific, such as 'there are cars that are congested in the left lane of the driving lane' and 'there are cars in the front/right lane of the driving lane'. may be judged as

In FIG. 8G , a stop line 872 and a traffic light 871 corresponding to the driving lane exist in front of the driving lane in which the vehicle 1 is traveling, and crosswalks 873, 874, 875, 876 crossing the 4 intersections are shown. An image of a surrounding situation corresponding to an existing situation is shown. At this time, the image analysis module 30 detects each object corresponding to the stop line 872 , the traffic light 871 , and the crosswalk 873 , 874 , 875 , 876 , and generates and outputs surrounding situation information based on this. By doing so, it can be finally determined by the dangerous situation determination module 40 that the surrounding situation of the vehicle 1 corresponds to the dangerous situation category of 'there is a four-way intersection in front of the driving lane'.

Next, when it is determined that the surrounding situation of the vehicle 1 corresponds to a specific dangerous situation category among the plurality of preset dangerous situation categories, the driving situation determination guide device 10 causes the driving guide generation module 50 to By generating at least one of specific visual driving guide information and specific voice driving guide information corresponding to a specific dangerous situation category, the driver of the vehicle 1 may recognize a specific dangerous situation corresponding to a specific dangerous situation category. At this time, the driving situation determination guide device 10 causes the driving guide generation module 50 to transmit at least one of the generated specific visual driving guide information and specific voice driving guide information to a predetermined driver terminal 60 , It may be provided to the driver, and the driver terminal 60 may be a device such as a navigation device or a black box that is installed in a vehicle and capable of reproducing images and sounds, and is not installed in the vehicle, but is not installed in the vehicle, but a driving situation determination guide device (10). ) and a device capable of transmitting and receiving data via wire/wireless, a device such as a smartphone that can obtain at least one of the generated specific visual driving guide information and specific voice driving guide information and provide it to the driver is also adopted as the driver terminal 60 and can be used

In addition, the voice driving guide information that can be generated from the driving guide generation module 50 through the process as described above is generated using a voice guide template classified and stored in a plurality of dangerous situation categories preset in a predetermined database. At this time, the voice guide template has at least one slot, which is a part in which the content can be changed and input according to the situation, and each of the slots includes at least one slot included in the surrounding environment information. Any one of location information, size information, movement status information, and type information corresponding to each object may be input. In addition, each of the plurality of preset dangerous situation categories may additionally include information on at least one dangerous condition that can be determined as a dangerous situation, for example, if the dangerous situation category is 'bus/taxi' In the case of 'there is a stop', the corresponding dangerous conditions may be 'there is a bus stop on the right front of the driving lane' and 'there is a bus stopped on the right front of the driving lane'. At this time, the driving situation determination and guide device 10 causes the driving guide generation module 50 to respond to (i) information on the specific dangerous situation category, (ii) a specific dangerous condition corresponding to the specific dangerous situation category. Specific voice driving completed by inputting each slot included in the specific voice guide template corresponding to the specific dangerous situation category with reference to information on information about the specific voice guide template corresponding to the specific dangerous situation category, and (iii) information on the specific voice guide template corresponding to the specific dangerous situation category The guide information may be generated, and the generated specific voice driving guide information may be directly provided to the driver, or transmitted to a predetermined driver terminal and supported to be provided. For example, if the hazardous situation category is 'bus/taxi stop exists', the corresponding specific audio guide template is "[slot-1] to [slot-2] may come out suddenly. Reduce the speed and [slot] -2] carefully", and at this time, the driving situation judgment and guide device 10 causes the driving guide generation module 50 to have 'right front' for the [slot-1] slot. input, and 'bus' is input for the [slot-2] slot, so that specific voice driving guide information that has been completed, such as "The bus may come out of the front right side suddenly. Reduce your speed and watch the bus carefully" can be made to create

[Table 2] below is an example of voice driving guide information that can be generated in response to the plurality of preset dangerous situation categories.

Example of preset dangerous situation category Example of voice driving guide information 'There is a bus/taxi stop' "[Bus] may suddenly come out of [Right front]. Slow down and watch [Bus]",
"[People] may pass in front of [bus]. Slow down and keep an eye on [Right front]" 'A large vehicle exists in the front/right/left lane of the driving lane' "It is not recommended to drive side by side with [large car]. Reduce your speed and move [in the left lane]" 'There is a taxi ahead of the driving lane' "The [Taxi] in [forward] may make a sudden stop, so please keep a safe distance." 'Bicycles/motorcycles exist in front/rear on the right/left side of the driving lane' "Please note that there is a [bicycle/motorcycle] in [right front]." 'There are cars parked on the right/left side of the driving lane' “People may suddenly come out of [parked cars] in [right front], so please slow down.” 'There are traffic jams in the right/left lane of the driving lane' "Drive with caution as [driving cars] in the [left lane] may cut you in." 'There is a four-way intersection in front of the driving lane' "There is a [4-way intersection] in [ahead]. Please be careful." ... ...

At this time, in each example of the voice driving guide information in [Table 2], the part marked with a separate symbol ([]) may correspond to a slot part in the voice guide template and may be an input part. The guide template and voice driving guide information are not limited as shown in [Table 2], and may be determined differently depending on the implementation conditions of the invention. In addition, as an example of the invention, the driving guide generating module 50 may provide a predetermined TTS (Text-To-Speech) may be interlocked with the engine 51, and the driving situation determination and guide device 10 causes the driving guide generating module 50 to generate the TTS engine 51 for specific voice driving guide information. ) to generate specific TTS data by additionally performing a process to generate specific TTS data to directly provide specific voice information that reproduces specific TTS data to the driver or to transmit to a predetermined driver terminal 60 to support the provision have.

Next, the driving situation determination and guide device 10 may cause the driving guide generating module 50 to generate visual driving guide information in which a predetermined danger guide signal is additionally displayed on the surrounding image image, a plurality of preset Each of the dangerous situation categories further includes information on at least one dangerous condition that can be determined as each dangerous situation, and the driving situation determination and guide device 10 causes the dangerous situation determination module to ( i) the specific dangerous situation category information, (ii) information on a specific dangerous condition corresponding to the specific dangerous situation category, and (iii) the surrounding environment information, among at least one object included in the surrounding video image A process of specifying a risk object corresponding to a specific risk condition and generating risk factor information including information about the risk object may be additionally performed, and the driving situation judgment and guide device 10 is By allowing the driving guide module 50 to receive the risk factor information, an image in which a predetermined danger guide signal is additionally displayed to correspond to the coordinates corresponding to each danger object with respect to the surrounding image image is generated as specific visual driving guide information. In this case, the generated specific visual driving guide information may be directly provided to the driver or transmitted to a predetermined driver terminal and supported to be provided. For example, when the specific dangerous situation category is 'there is a bus/taxi stop', the surrounding video image may include a bus and a bus stop as an object on the right front side of the driving lane, and the corresponding bus on the surrounding video image and an image displaying a danger guide signal such as a predetermined geometric structure to correspond to the coordinates corresponding to the bus stop is generated as specific visual driving guide information, provided directly to the driver, or transmitted to a predetermined driver terminal and displayed. can be provided as

Next, the risk situation determination module may output information about the surrounding environment using a predetermined deep learning algorithm. For this, learning of the risk situation determination module based on the deep learning algorithm may be performed in advance. This will be described in detail with reference to separate drawings ( FIGS. 6A and 6B ) as follows.

6A is a flowchart schematically illustrating a process in which learning for a dangerous situation determination module is performed using surrounding image images for learning as learning data according to an embodiment of the present invention, and FIG. 6B is an embodiment of the present invention , is a flowchart schematically showing a process in which learning for the dangerous situation determination module is performed using the separately prepared surrounding environment information for learning as learning data.

Referring to FIG. 6A , as an embodiment of the present invention, a predetermined dangerous situation determination module learning apparatus uses, as learning data, each of at least one first learning surrounding video image obtained from the camera mounted on a vehicle or prepared separately , the image analysis module receives each of the first surrounding image images for learning (S601) and outputs the first surrounding environment information for learning as an analysis result (S602), and using the outputted first surrounding environment information for learning, the risk Learning for the situation determination module can be performed, in which case each of the first learning surrounding image images includes information on a specific correct dangerous situation category corresponding to each of the first surrounding video images among the preset dangerous situation categories. It may be included as GT (Ground Truth). In this state, the learning of the dangerous situation determination module causes the dangerous situation determination module learning device to receive the first learning surrounding environment information (S603) and perform a predetermined deep learning operation (S604), and , as a result of comparing the information (S605) on the first prediction values for learning as the probability corresponding to each of the plurality of preset dangerous situation categories output as a result with the first GT, so that the difference is minimized. This may be accomplished by performing a process of optimizing a plurality of parameters. At this time, the difference between the first learning prediction values and the first GT may be calculated as a loss value using a predetermined formula (S606), and backpropagation is iteratively performed in order to minimize the loss value. A plurality of parameters included in the dangerous situation determination module 40 may be optimized ( S407 ).

In addition, as another embodiment of the present invention, referring to FIG. 6B , a predetermined dangerous situation determination module learning apparatus uses at least one separately prepared second learning surrounding environment information as learning data, In this case, each of the second learning surrounding environment information may include information on a specific correct answer risk situation category corresponding to each as a second ground truth (GT). In this state, the learning of the dangerous situation determination module causes the dangerous situation determination module learning device to receive the second learning surrounding environment information (S611) and perform a predetermined deep learning operation (S612), and , as a result of comparing the information (S613) on the second prediction values for learning as the probability corresponding to each of the plurality of preset dangerous situation categories output as a result with the second GT, the difference is minimized. It may be achieved by performing a process of optimizing a plurality of parameters. In addition, the difference between the second prediction values for learning and the second GT may be calculated as a loss value using a predetermined formula (S614), and backpropagation is repeatedly performed to minimize the loss value. Thus, a plurality of parameters included in the dangerous situation determination module 40 may be optimized (S615).

And, the learning of the dangerous situation determination module 40, as described above, may be performed by the dangerous situation determination module learning device, but unlike this, the dangerous situation determination module learning device learns the separate dangerous situation determination module for learning to apply the learning results, such as optimized parameters, to the dangerous situation determination module 40, and the driving situation determination and guide device 10 functions as a dangerous situation determination module learning device and the dangerous situation determination module 40 It is also possible to perform learning on the , which may vary depending on the implementation conditions of the invention.

Next, as another embodiment of the invention, the predetermined algorithm used by the image analysis module 30 to analyze the surrounding situation image is a predetermined real-time object detection algorithm or the predetermined real-time object detection algorithm. When a deep learning algorithm is used as a real-time instance segmentation algorithm, learning based on the algorithm may be performed on the image analysis module 30 . This will be described in detail with reference to a separate drawing (FIG. 7) as follows.

7 is a flowchart schematically illustrating a process in which learning of an image analysis module is performed by using an image of a surrounding image for learning as learning data, according to an embodiment of the present invention.

Referring to FIG. 7 , the predetermined image analysis module learning apparatus uses at least one second learning surrounding image image obtained from the camera 20 mounted on the vehicle 1 or separately prepared as learning data, the image analysis module ( 30), each of the second learning peripheral image images may include correct answer object information for each of the at least one learning object included in each as the third GT (Ground Truth). have. In this state, in the learning of the image analysis module 30, the image analysis module learning device causes the image analysis module 30 to receive the second surrounding image image for learning (S701), and the predetermined real-time object detection algorithm or the Analyze (S702) using a predetermined real-time instance segmentation algorithm, and compare the information (S703) for each learning object output as a result with the third GT so that the difference is minimized. Included in the image analysis module This may be accomplished by performing a process of optimizing a plurality of parameters. At this time, the difference between the information on each learning object and the third GT may be calculated as a loss value using a predetermined formula (S704), and backpropagation is iteratively performed in order to minimize the loss value. , so that a plurality of parameters included in the image analysis module 30 can be optimized (S705).

And, as for the learning of the image analysis module 30, as in the learning of the dangerous situation determination module 40, the image analysis module learning device performs learning on a separate image analysis module for learning, and the learning result is transferred to the image analysis module 40 may be applied, and the driving situation determination and guide device 10 may function as an image analysis module learning device to perform learning on the image analysis module 30, which may vary depending on the implementation conditions of the invention. can

The embodiments according to the present invention described above may be implemented in the form of program instructions that can be executed through various computer components and recorded in a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the computer-readable recording medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include hard disks, magnetic media such as floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and magneto-optical media such as floppy disks. , and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware device may be configured to operate as one or more software modules to perform processing according to the present invention, and vice versa.

In the above, the present invention has been described with specific matters such as specific components and limited embodiments and drawings, but these are provided to help a more general understanding of the present invention, and the present invention is not limited to the above embodiments. , those of ordinary skill in the art to which the present invention pertains can make various modifications and variations from these descriptions.

Therefore, the spirit of the present invention should not be limited to the above-described embodiments, and not only the claims described below, but also all modifications equivalently or equivalently to the claims described below belong to the scope of the spirit of the present invention. will do it

1: Car 10: Driving situation judgment and guide device
20: camera 30: image analysis module
40: dangerous situation determination module 50: driving guide generation module
60: driver terminal
150: backbone block 200: bottleneck block
300: feature pyramid network 400: instance segmentation network

Claims (22)

In the method of determining whether the surrounding situation of the vehicle is a dangerous situation, and generating a driving guide to give an alarm,
(a) when the surrounding video image of the vehicle is obtained from at least one camera mounted on the vehicle, the driving condition determination and guide device inputs the surrounding video image to the video analysis module to cause the video analysis module to cause the surrounding image analyzing the image to output surrounding environment information including information on at least one object existing in the vicinity of the vehicle;
(b) the driving situation determination and guide device inputs the surrounding environment information to the dangerous situation determination module, and allows the dangerous situation determination module to perform deep learning operations on the surrounding environment information to correspond to each of a plurality of preset dangerous situation categories outputting predicted values as a probability of becoming the vehicle, and determining whether the surrounding situation of the vehicle corresponds to a specific dangerous situation category with reference to the predicted values; and
(c) when the driving situation determination and guide device determines that the surrounding situation of the vehicle corresponds to the specific dangerous situation category among the plurality of preset dangerous situation categories, causes the driving guide generation module to cause the specific dangerous situation category generating at least one of specific visual driving guide information and specific voice driving guide information corresponding to , so that the driver of the vehicle can recognize a specific dangerous situation corresponding to the specific dangerous situation category;
How to include. According to claim 1,
A plurality of voice guide templates in a predetermined database - Each of the voice guide templates has at least one slot, which is a part whose contents can be changed and input according to circumstances, and each of the slots includes the information about the surrounding environment. Any one of location information, size information, movement status information, and type information corresponding to each of the at least one object included in the . characterized,
Each of the plurality of preset dangerous situation categories is characterized in that it further includes information on at least one dangerous condition that can be determined as each dangerous situation,
In step (c),
The driving situation determination and guide device causes the driving guide generation module to (i) information on the specific dangerous situation category, (ii) information on specific dangerous conditions corresponding to the specific dangerous situation category, and (iii) With reference to information on a specific voice guide template corresponding to the specific dangerous situation category, each of the slots included in the specific voice guide template is input to generate the specific voice driving guide information completed, and the generated specific voice driving guide information is generated. The method of claim 1, wherein the voice driving guide information is directly provided to the driver or transmitted to a predetermined driver's terminal and supported to be provided. 3. The method of claim 2,
The driving guide generation module is characterized in that it is linked with a predetermined TTS (Text-To-Speech) engine,
In step (c),
The driving situation determination and guide device further performs a process of causing the driving guide generation module to generate specific TTS data by applying the TTS engine to the generated specific voice driving guide information, and the specific TTS data A method characterized in that the specific voice information reproduced by the . is directly provided to the driver or transmitted to a predetermined driver's terminal and supported to be provided. According to claim 1,
Each of the plurality of preset dangerous situation categories is characterized in that it further includes information on at least one dangerous condition that can be determined as the respective dangerous situation,
In step (b),
The driving situation determination and guide device causes the dangerous situation determination module to (i) the specific dangerous situation category information, (ii) information on a specific dangerous condition corresponding to the specific dangerous situation category, and (iii) the surroundings A process of specifying a risk object corresponding to the risk condition among at least one of the objects included in the surrounding image image with reference to environmental information, and generating risk factor information including information on the risk object A method characterized in that it is performed. 5. The method of claim 4,
In step (c),
The driving situation determination and guide device causes the driving guide module to receive the risk factor information, so that a preset danger guide signal is additionally displayed to correspond to the coordinates corresponding to each of the danger objects with respect to the surrounding image image. An image is generated as the specific visual driving guide information, and the generated specific visual driving guide information is directly provided to the driver or transmitted to a predetermined driver terminal and supported to be provided. According to claim 1,
Before step (a),
A predetermined danger situation determination module learning device, (i) each of the at least one first surrounding video image for learning obtained from the camera mounted on the vehicle or separately prepared - Each of the first surrounding video image for learning is a preset risk In the context category, the information on the specific correct answer risk situation category corresponding to each of the first surrounding image images for learning is included as the first GT (Ground Truth) - as the learning data, the image analysis module causes the first learning Each of the surrounding video images is received and analyzed to output the first surrounding environment information for learning, and learning about the dangerous situation determination module is performed using the outputted first surrounding environment information for learning, or (ii) separately Prepared at least one second surrounding environment information for learning - Each of the second learning surrounding environment information includes information on a specific correct answer risk situation category corresponding to each as a second GT (Ground Truth) - as learning data It is characterized in that learning about the dangerous situation determination module is performed using
In the learning of the dangerous situation determination module, the dangerous situation determination module learning device (i) causes the dangerous situation determination module to receive the first learning surrounding environment information and perform the predetermined deep learning operation, and the A plurality of parameters included in the risk situation determination module so that the difference is minimized by comparing the information on the first learning prediction values as probabilities corresponding to each of the plurality of preset dangerous situation categories output as a result with the first GT performing a process of optimizing, or (ii) causing the risk situation determination module to receive the second learning surrounding environment information and perform the predetermined deep learning operation, and the plurality of preset risks output as a result It is characterized by performing a process of optimizing a plurality of parameters included in the dangerous situation determination module so that the difference is minimized by comparing the information on the second learning prediction values as the probability corresponding to each situation category with the second GT. how to do it According to claim 1,
In step (a),
The driving situation determination and guide device causes the image analysis module to detect each of the at least one object included in the surrounding image image by using a predetermined algorithm, position information of each detected object, size information, The method of claim 1, wherein information including at least a part of movement status information and type information is output as the surrounding environment information. 8. The method of claim 7,
Before step (a),
The predetermined image analysis module learning apparatus performs learning based on the predetermined real-time object detection algorithm or the predetermined real-time instance segmentation algorithm for the image analysis module. It is characterized by performing
In the learning of the image analysis module, the image analysis module learning apparatus obtains from the camera mounted on the vehicle, or at least one separately prepared second surrounding image image for learning - each of the second learning surrounding image image is, Including correct object information for each of the at least one learning object included in each as the third GT (Ground Truth) - by using as learning data, the image analysis module receives the surrounding image image for the second learning Analysis is performed using the predetermined real-time object detection algorithm or the predetermined real-time instance segmentation algorithm, and the resultant information on each of the learning objects is compared with the third GT to minimize the difference. A method characterized in that by performing a process of optimizing a plurality of parameters included in the module. 8. The method of claim 7,
The driving situation determination and guidance device causes the image analysis module to apply a predetermined real-time object detection algorithm or a predetermined real-time instance segmentation algorithm to the surrounding video image. Detecting each of the at least one included object or performing instance segmentation of the surrounding video image,
The step (a) is,
(a1) When the surrounding video image is acquired, the driving situation determination and guide device causes the video analysis module to input the surrounding video image to the backbone block of ResNET and cause the backbone block to sequentially process the surrounding video image outputting a down-sampled first down-sampling feature map to m-th, where m is an integer greater than or equal to 2, by performing a convolution operation to output a down-sampling feature map;
(a2) the driving situation determination and guidance device causes the image analysis module to input a specific down-sampling feature map to the first (1*1) convolutional layer to the first (1*1) convolutional layer (1*1) convolution operation on the specific down-sampling feature map to generate a first feature map with an adjusted number of channels, and expand the first feature map to (1*r) - where r is an integer greater than or equal to 1 - (k*k) with a ratio - where k is an integer greater than or equal to 2 - a first (k*k) convolutional layer including a kernel to (n*r) - where n is an integer greater than or equal to 2 - (k) with an extension ratio *k) each input to an nth (k*k) convolutional layer including a kernel to cause each of the first (k*k) convolutional layer to the nth (k*k) convolutional layer to form the first The channels of the feature map are divided into at least two groups, and each of the first feature maps corresponding to the at least two groups is divided into a (k*k) convolution operation using an (1*r) expansion ratio to (n* r) a first process for generating a 2_1 th feature map to a 2_n th feature map by performing a (k*k) convolution operation using an extension ratio, and a 2_1 (1) *1) Convolutional layers to 2_n (1*1) convolutional layers are input to each of the 2_1 (1*1) convolutional layers to the 2_n (1*1) convolutional layers. A (1*1) convolution operation on the 2_1 feature map to the 2_nth feature map is performed to generate 3_1 to 3_n feature maps with the number of channels adjusted, and the 3_1 to 3_n feature maps are convolutional. Kate and input as a third (1*1) convolution layer to cause the third (1*1) convolution layer to convert the concatenated 3_1 to 3_n feature maps (1*1) Convolution is performed to generate a fourth feature map with the number of channels adjusted, and the specific down-sampling feature map and the fourth feature map are combined. Through a second process of concatenating to generate a transformed down-sampling feature map, the m-th down-sampling feature map to (mj) - wherein j is an integer greater than or equal to 1 and less than m - is added to each of the down-sampling feature maps generating an m-th transformed down-sampling feature map to an (mj)-th transformed down-sampling feature map by applying the first process and the second process; and
(a3) the driving situation determination and guide device causes the image analysis module to input the m-th transformed down-sampling feature map to the (mj)-th transformed down-sampling feature map into a feature pyramid network, and the feature pyramid To cause the network to generate an m-th up-sampling feature map to (mj)-th up-sampling feature map through a deconvolution operation referring to the m-th transformed down-sampling feature map to the (mj)-th transformed down-sampling feature map. and input the (mj)th up-sampling feature map to an object detection network or an instance segmentation network to cause the object detection network to detect an object on the surrounding video image, or cause the instance segmentation network to instantiate the surrounding video image segmentation;
How to include. 10. The method of claim 9,
In step (a3),
The image analysis module causes the feature pyramid network to down-sample the m-th up-sampling feature map to generate a (m+j)-th up-sampling feature map, the (mj)-th up-sampling feature map to the (mj)-th up-sampling feature map to the (m+j) additionally input an up-sampling feature map to the instance segmentation network to cause the instance segmentation network to further refer to the (mj)-th up-sampling feature map to the (m+j)-th up-sampling feature map. How to enable instance segmentation of the surrounding video image. 10. The method of claim 9,
In step (a3),
The image analysis module causes the feature pyramid network to generate a specific up-sampling feature map by up-sampling an up-sampling feature map, where i is an integer greater than or equal to (m-(j+1)) and less than or equal to m; A method of concatenating the specific up-sampling feature map and a corresponding (i-1)-th down-sampling feature map to generate an (i-1)-th up-sampling feature map. In the device for judging whether the surrounding situation of the vehicle is a dangerous situation, and generating a driving guide to warn it,
at least one memory storing instructions; and
at least one processor configured to execute the instructions; including,
the processor,
(I) When the surrounding video image of the vehicle is obtained from at least one camera mounted on the vehicle, the surrounding video image is input to the video analysis module, and the video analysis module analyzes the surrounding video image to analyze the surrounding video image of the vehicle. a process of outputting surrounding environment information including information on at least one object existing in the . (II) input the surrounding environment information to the dangerous situation determination module so that the dangerous situation determination module performs a deep learning operation on the surrounding environment information to output predicted values as probabilities corresponding to each of a plurality of preset dangerous situation categories, a process of determining whether the surrounding situation of the vehicle corresponds to a specific dangerous situation category with reference to the predicted values; and (III) when it is determined that the surrounding situation of the vehicle corresponds to the specific dangerous situation category among the plurality of preset dangerous situation categories, causes the driving guide generation module to cause specific visual driving guide information corresponding to the specific dangerous situation category and generating at least one of specific voice driving guide information so that the driver of the vehicle can recognize a specific dangerous situation corresponding to the specific dangerous situation category. device that does it. 13. The method of claim 12,
A plurality of voice guide templates in a predetermined database - Each of the voice guide templates has at least one slot, which is a part whose contents can be changed and input according to circumstances, and each of the slots includes the information about the surrounding environment. Any one of location information, size information, movement status information, and type information corresponding to each of the at least one object included in the . characterized,
Each of the plurality of preset dangerous situation categories is characterized in that it further includes information on at least one dangerous condition that can be determined as each dangerous situation,
In the process (III) above,
The processor is configured to cause the driving guide generation module to (i) information on the specific dangerous situation category, (ii) information on a specific dangerous situation corresponding to the specific dangerous situation category, and (iii) the specific dangerous situation category. With reference to information on a specific voice guide template corresponding to , each of the slots included in the specific voice guide template is input to generate the completed specific voice driving guide information, and the generated specific voice driving guide information is generated. Device characterized in that to directly provide to the driver or to transmit to a predetermined driver terminal to support the provision. 14. The method of claim 13,
The driving guide generation module is characterized in that it is linked with a predetermined TTS (Text-To-Speech) engine,
In the process (III) above,
The processor may further perform a process of causing the driving guide generation module to generate specific TTS data by applying the TTS engine to the generated specific voice driving guide information, thereby reproducing the specific TTS data. Device characterized in that the information is directly provided to the driver or transmitted to a predetermined driver's terminal to support the provision. 13. The method of claim 12,
Each of the plurality of preset dangerous situation categories is characterized in that it further includes information on at least one dangerous condition that can be determined as the respective dangerous situation,
In the process (II) above,
The processor causes the dangerous situation determination module to refer to (i) the specific dangerous situation category information, (ii) information about a specific dangerous condition corresponding to the specific dangerous situation category, and (iii) the surrounding environment information. Further performing a process of specifying a risk object corresponding to the risk condition among at least one of the objects included in the surrounding image image, and generating risk factor information including information on the risk object device to do. 16. The method of claim 15,
In the process (III) above,
The processor causes the driving guide module to receive the risk factor information, so that the predetermined danger guide signal is additionally displayed to correspond to the coordinates corresponding to each of the danger objects with respect to the surrounding image image at the specific time. An apparatus characterized in that it is generated as driving guide information, and the generated specific visual driving guide information is directly provided to the driver or transmitted to a predetermined driver terminal to support the provided. 13. The method of claim 12,
Prior to the (I) process,
A predetermined danger situation determination module learning device, (i) each of the at least one first surrounding video image for learning obtained from the camera mounted on the vehicle or separately prepared - Each of the first surrounding video image for learning is a preset risk In the context category, the information on the specific correct answer risk situation category corresponding to each of the first surrounding image images for learning is included as the first GT (Ground Truth) - as the learning data, the image analysis module causes the first learning Each of the surrounding video images is received and analyzed to output the first surrounding environment information for learning, and learning about the dangerous situation determination module is performed using the outputted first surrounding environment information for learning, or (ii) separately Prepared at least one second surrounding environment information for learning - Each of the second learning surrounding environment information includes information on a specific correct answer risk situation category corresponding to each as a second GT (Ground Truth) - as learning data It is characterized in that learning about the dangerous situation determination module is performed using
In the learning of the dangerous situation determination module, the dangerous situation determination module learning device (i) causes the dangerous situation determination module to receive the first learning surrounding environment information and perform the predetermined deep learning operation, and the A plurality of parameters included in the risk situation determination module so that the difference is minimized by comparing the information on the first learning prediction values as probabilities corresponding to each of the plurality of preset dangerous situation categories output as a result with the first GT performing a process of optimizing, or (ii) causing the risk situation determination module to receive the second learning surrounding environment information and perform the predetermined deep learning operation, and the plurality of preset risks output as a result It is characterized by performing a process of optimizing a plurality of parameters included in the dangerous situation determination module so that the difference is minimized by comparing the information on the second learning prediction values as the probability corresponding to each situation category with the second GT. device to do. 13. The method of claim 12,
In the process (I) above,
The processor causes the image analysis module to detect each of the at least one object included in the surrounding image image using a predetermined algorithm, and position information, size information, movement status information and type of each detected object. An apparatus, characterized in that the information including at least a part of the information is output as the surrounding environment information. 19. The method of claim 18,
Prior to the (I) process,
The predetermined image analysis module learning apparatus performs learning based on the predetermined real-time object detection algorithm or the predetermined real-time instance segmentation algorithm for the image analysis module. It is characterized by performing
In the learning of the image analysis module, the image analysis module learning apparatus obtains from the camera mounted on the vehicle, or at least one separately prepared second surrounding image image for learning - each of the second learning surrounding image image is, Including correct object information for each of the at least one learning object included in each as the third GT (Ground Truth) - by using as learning data, the image analysis module receives the surrounding image image for the second learning Analysis is performed using the predetermined real-time object detection algorithm or the predetermined real-time instance segmentation algorithm, and the resultant information on each of the learning objects is compared with the third GT to minimize the difference. An apparatus, characterized in that by performing a process of optimizing a plurality of parameters included in the module. 19. The method of claim 18,
the processor causes the image analysis module to use a predetermined real-time object detection algorithm or a predetermined real-time instance segmentation algorithm to perform at least one It is characterized in that each of the objects is detected or the surrounding video image is instance-segmented,
The (I) process is
(I-1) When the surrounding video image is obtained, the processor causes the image analysis module to input the surrounding video image to a backbone block of ResNET and cause the backbone block to sequentially perform a convolution operation on the surrounding video image a sub-process for outputting down-sampled first down-sampling feature maps to m-th - wherein m is an integer greater than or equal to 2 - down-sampling feature maps;
(I-2) the processor causes the image analysis module to input a specific down-sampling feature map as a first (1*1) convolutional layer to cause the first (1*1) convolutional layer to cause the specific downsampling feature map (1*1) convolution operation on the down-sampling feature map to generate a first feature map with the number of channels adjusted, and (1*r) - where r is an integer greater than or equal to 1 - has an expansion ratio (k*k) - wherein k is an integer greater than or equal to 2 - a first (k*k) convolutional layer including a kernel to (n*r) - where n is an integer greater than or equal to 2 - (k*k) with an extension ratio Each of the first (k*k) convolutional layer to the nth (k*k) convolutional layer is inputted to the nth (k*k) convolutional layer including the kernel, so that each of the first feature map The channels are divided into at least two groups, and each of the first feature maps corresponding to the at least two groups is expanded by a (k*k) convolution operation or (n*r) by a (1*r) extension ratio. A first process for generating a 2_1 th feature map to a 2_n th feature map by performing a (k*k) convolution operation by a ratio, and a 2_1 (1*1) The 2_1 (1*1) convolutional layer to the 2_n (1*1) convolutional layer are inputted to the 2_n (1*1) convolutional layer to cause each of the 2_1st feature maps. to (1*1) convolution operation on the 2_n second feature map to generate 3_1 to 3_n feature maps with the number of channels adjusted, and concatenate the 3_1 to 3_n feature maps to By inputting the third (1*1) convolutional layer, the third (1*1) convolution layer performs a (1*1) convolution operation on the concatenated 3_1 to 3_n feature maps. to generate a fourth feature map with an adjusted number of channels, and concatenate the specific down-sampling feature map and the fourth feature map through a second process of generating a transformed down-sampling feature map by performing a second process, the m-th down-sampling feature map to (mj) - the j is an integer greater than or equal to 1 and less than m - the first process in each of the down-sampling feature maps and a sub-process for generating an m-th transformed down-sampling feature map to an (mj)-th transformed down-sampling feature map by applying the second process; and
(I-3) the processor causes the image analysis module to input the m-th transformed down-sampling feature map to the (mj)-th transformed down-sampling feature map to a feature pyramid network to cause the feature pyramid network to generate an m-th up-sampling feature map to (mj)-th up-sampling feature map through a deconvolution operation referring to the m-th transformed down-sampling feature map to the (mj)-th transformed down-sampling feature map, and (mj) input the up-sampling feature map to an object detection network or an instance segmentation network to cause the object detection network to detect an object on the surrounding video image or to cause the instance segmentation network to instance segment the surrounding video image subprocess; device that does it. 21. The method of claim 20,
In the process (I-3) above,
The image analysis module causes the feature pyramid network to down-sample the m-th up-sampling feature map to generate a (m+j)-th up-sampling feature map, the (mj)-th up-sampling feature map to the (mj)-th up-sampling feature map to the (m+j) additionally input an up-sampling feature map to the instance segmentation network to cause the instance segmentation network to further refer to the (mj)-th up-sampling feature map to the (m+j)-th up-sampling feature map. Apparatus characterized in that it enables the instance segmentation of the surrounding video image. 21. The method of claim 20,
In step (I-3),
The image analysis module causes the feature pyramid network to generate a specific up-sampling feature map by up-sampling an up-sampling feature map, where i is an integer greater than or equal to (m-(j+1)) and less than or equal to m; and concatenating the specific up-sampling feature map and a corresponding (i-1)-th down-sampling feature map to generate an (i-1)-th up-sampling feature map.

Images