KR20180047149A - Apparatus and method for risk alarming of collision - Google Patents

Abstract

The present invention comprises: a memory in which a collision risk predicting program is stored; and a processor performing the program stored in the memory. At this moment, the processor extracts at least one object from image data based on an object recognizing model in accordance with the performance of the program, and reconstructs three-dimensional space information from the image data. Moreover, the processor predicts a movement passage of a camera photographing the movement passage of each one of the objects and the image data based on the three-dimensional space information and a movement passage predicting model. In addition, the processor determines whether or not an object of which a movement passage intersects with the movement passage of the camera among the objects.

Description

[0001] APPARATUS AND METHOD FOR RISK ALARMING OF COLLISION [0002]

The present invention relates to a crash risk warning apparatus and method.

The ADAS includes a variety of technologies that can assist drivers such as lane departure and collision detection while driving. Among the technologies of the driver assistance system, the collision prevention technique mainly uses the sensor. At this time, the detection sensors include microwave, laser, radar, and lidar. This sensor-based technology makes it easy to detect obstacles. However, sensor-based technology can easily be affected by ambient conditions or weather conditions. In addition, the sensor-based technology has a problem that the sensor installation cost is high.

In recent years, driver assistance systems have been attracting attention as they apply computer vision technology. This method can detect or track nearby pedestrians through image data to determine the risk of collision between a driving vehicle and a pedestrian. This method is suitable for real-time applications. However, there are disadvantages of frequent misunderstandings in complex urban areas or in a changing driving environment.

In the driver assistance system using computer vision technology, image data taken from the camera is used. In addition, the driver assistance system using the computer vision technology can estimate and use image information such as depth information, relative distance information, and speed information from the image data. At this time, the camera may be a stereo camera or a monocular camera. However, there are various problems to be solved in order to estimate the image information from the image data generated by using the monocular camera.

In this regard, Korean Patent Registration No. 10-1489836 (entitled "Pedestrian Detection and Collision Avoiding Device and Method") combines information related to at least one or more pedestrians and information related to a vehicle, And a pedestrian detection / collision avoiding apparatus and method for estimating situations in which collision avoidance is possible when a collision risk is determined, and outputting a control signal for collision avoidance for each situation.

Disclosure of Invention Technical Problem [8] Accordingly, the present invention has been made in view of the above problems, and it is an object of the present invention to provide a collision risk warning apparatus and method capable of predicting and warning a collision risk between a vehicle and a pedestrian from image data.

It should be understood, however, that the technical scope of the present invention is not limited to the above-described technical problems, and other technical problems may exist.

As a technical means for achieving the above technical object, a crash risk warning device according to the first aspect of the present invention includes a memory for storing a crash risk prediction program, and a processor for executing a program stored in the memory. At this time, according to the execution of the program, the processor extracts one or more objects from the image data based on the object recognition model, reconstructs the three-dimensional spatial information from the image data, Predicts a movement path of the camera that captured each movement path and image data, and determines whether there is an object that intersects the camera movement path and the movement path among the at least one object.

According to a second aspect of the present invention, there is provided a method for detecting a collision risk of a collision risk warning device, comprising: extracting at least one object from image data; Reconstructing three-dimensional spatial information from the image data; Estimating a movement path of each of the objects based on the three-dimensional space information and the generated motion path prediction model; Estimating a movement path of a camera that captures image data based on the three-dimensional spatial information and the generated motion path prediction model; And determining whether there is an object that intersects the movement path of the camera among the at least one object.

A third aspect of the present invention is a method for generating a model of a collision risk warning device, comprising: extracting learning image data including an object from a collected image data set; And generating an object recognition model based on the training image data and the machine learning algorithm. At this time, the object recognition model extracts a plurality of objects from the image data. In addition, the extracted plurality of objects are compared with the movement path for the camera that captures the image data after the movement path is predicted based on the reconstructed three-dimensional spatial information corresponding to the image data, Is extracted as a collision risk object, and the machine learning algorithm is either a neural network algorithm, a support vector machine algorithm, or a clustering algorithm.

The present invention can restore three-dimensional spatial information from two-dimensional image data and detect a collision risk between a pedestrian and a vehicle. In addition, the present invention can remove noise that may be caused by movement of the vehicle. Therefore, the present invention can reduce the mistrust of dangerous pedestrians, reduce the risk of accidents, and can contribute to safety of the driver.

FIG. 1 is an exemplary view of a dangerous pedestrian detection in a conventional driver assistance system. FIG.
2 is a block diagram of a collision risk warning device according to an embodiment of the present invention.
3 is a block diagram of a collision risk prediction program according to an embodiment of the present invention.
4 is an exemplary view of a pedestrian object extracted from image data according to an embodiment of the present invention.
FIG. 5 is an exemplary diagram illustrating feature matching for image data according to an exemplary embodiment of the present invention. Referring to FIG.
FIG. 6 is an exemplary view illustrating positions of a camera module and a pedestrian object according to an exemplary embodiment of the present invention.
FIG. 7 is an exemplary view of image data in which an identified pedestrian object is displayed according to an embodiment of the present invention. FIG.
FIG. 8 is an exemplary view of image data in which an identified pedestrian object is displayed according to an embodiment of the present invention.
9 is a flowchart illustrating a method of detecting a collision risk of a collision risk warning device according to an embodiment of the present invention.
10 is a flowchart illustrating a method of generating an object recognition model of a collision risk warning device according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings, which will be readily apparent to those skilled in the art. The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. In order to clearly illustrate the present invention, parts not related to the description are omitted, and similar parts are denoted by like reference characters throughout the specification.

Throughout the specification, when a part is referred to as being "connected" to another part, it includes not only "directly connected" but also "electrically connected" with another part in between . Also, when a part is referred to as "including " an element, it does not exclude other elements unless specifically stated otherwise.

Conventional driver assistance systems can predict a behavior pattern of a pedestrian based on a Kalman filter based trajectory model. In addition, the conventional driving assistance system can detect a dangerous pedestrian according to a behavior pattern of a pedestrian.

FIG. 1 is an exemplary view of a dangerous pedestrian detection in a conventional driver assistance system. FIG.

1 (a), the line 102 of the pedestrian 100 represents the locus information of the pedestrian 100, and the arrow 101 represents the locus information of the pedestrian 100 that is predicted based on the locus information of the pedestrian 100, The future direction of the future. The conventional driving assistance system extracts the pedestrian shown in Fig. 1 (a) as a dangerous pedestrian. However, the actual pedestrian is misunderstood as a driver getting off the other vehicle, or moving the standing situation to the direction of the vehicle corresponding to the driving assistance system. 1 (b) is a view similar to FIG. 1 (a), in which the driver 110 departing from another vehicle standing at one position is mistaken as a dangerous pedestrian.

Referring to FIG. 1 (c), in the conventional driving assist system, when a pedestrian is waiting for a signal at an intersection or a noise occurs due to movement of another vehicle, the moving direction of the pedestrian 122 is indicated by an arrow 121 Predicted, and detected as a dangerous pedestrian. As described above, the conventional driving assist system may cause noise due to ego-motion of another vehicle.

Such noise can be easily corrected in image data collected using a binocular camera using two lenses. However, correction using a monocular camera such as a black box device installed in a vehicle may be difficult.

Next, a crash risk warning system 200 according to an embodiment of the present invention will be described with reference to FIGS.

2 is a block diagram of a crash risk warning system 200 in accordance with an embodiment of the present invention.

The collision risk warning system 200 detects an object at risk of collision based on the image data collected through the camera module 240. At this time, the collision risk warning system 200 may include a camera module 240 and a collision risk warning device 210.

The collision risk warning device 210 extracts an object from the image data and predicts a movement path of the object. The collision risk warning device 210 detects the collision risk by comparing the movement path of the moving subject with the movement path of the object extracted from the image data.

At this time, the image data may be photographed through the camera module 240 attached to the object according to the movement path of the object. For example, the moving entity may be a transportation means such as a vehicle, a bicycle, and a train. Alternatively, the moving subject may be a person, but is not limited thereto.

In addition, the object extracted from the image data may be a vehicle such as a vehicle, a bicycle, a train, or a person, an animal, etc., but is not limited thereto.

Hereinafter, real-time image data collected through the camera module 240 installed in the vehicle, and pedestrians around the road on which the vehicle travels are described as an example.

The collision risk warning device 210 includes a memory 230 and a processor 220.

At this time, the collision risk prediction program 300 is stored in the memory 230. The memory 230 is collectively referred to as a non-volatile storage device that keeps stored information even when power is not supplied, and a volatile storage device that requires power to maintain stored information.

The processor 220 may extract a plurality of pedestrians from the image data according to the execution of the collision risk prediction program 300. [ The processor 220 can detect the possibility of collision by comparing the moving direction of the camera module 240 that captured the image data with the moving direction of each of the plurality of extracted pedestrians. At this time, the collision risk prediction program 300 will be described in detail with reference to FIG.

3 is a block diagram of a collision risk prediction program 300 according to an embodiment of the present invention.

The collision risk prediction program 300 includes a camera motion estimation module 320, a three-dimensional space construction module 321, a vehicle motion estimation module 322, a vehicle movement path prediction module 323, a pedestrian detection module 330, A motion estimation module 331, a pedestrian movement path prediction module 332, and a conflict determination module 340.

In addition, the collision risk prediction program 300 can detect the possibility of a collision of the pedestrian from the image data 310. The image data 310 may be acquired from the camera module 240 connected to the collision risk warning device 210 or may be captured through a separate camera or camera module and then transmitted to the collision risk warning device 210 . In addition, the image data 310 may be collected from a separate device including the camera module 240 and then transmitted to the collision risk warning device 210. For example, the image data 310 may be collected in real time from the camera module 240 included in the black box device attached to the running vehicle and transmitted to the collision risk warning device 210.

First, the processor 220 can extract a plurality of pedestrian objects from the image data 310 through the pedestrian detection module 330.

Specifically, the processor 220 can divide a plurality of frames included in the image data 310 into a plurality of windows through a sliding window technique. At this time, the sliding window technique can consider multi-scale. Therefore, the size of the window may be a predetermined multiple of a predetermined value from a predetermined value. In addition, the size of each window may be different, and may be extracted overlapping with the previously extracted window in time by a predetermined frame.

For example, the size of the window may be a value between 5 frames, which is a predetermined threshold value, and 6 frames, which is a multiple of 1.2. Also, the size of the overlap may be less than the size of one or more windows.

In addition, the processor 220 may extract features from the frames contained in each window. In this case, the feature may be a histogram of gradient (HoG), edge information, and a color histogram. Thus, the processor 220 may extract one or more features of a histogram of gradient (HoG), edge information, and color histogram, and generate a feature vector corresponding to the window based on the extracted feature.

The processor 220 can compare the extracted object recognition model with the extracted feature vector corresponding to each window. At this time, the pre-learned object recognition model may be generated based on a machine learning algorithm for classification or clustering. For example, the pre-learned object recognition model may be generated based on a support vector machine and a neural network.

The processor 220 may calculate the degree of similarity between the object recognition model previously learned and the feature vectors corresponding to the respective windows. If the calculated degree of similarity is equal to or greater than a predetermined value, the processor 220 can determine that a pedestrian object exists in the corresponding window. The processor 220 can extract the window including the pedestrian into the candidate area.

4 is an exemplary view of a pedestrian object extracted from the image data 310 according to an embodiment of the present invention.

For example, the processor 220 may extract a pedestrian object for each frame included in the window through the object recognition model. The processor 220 may extract a frame including the extracted pedestrian object as a candidate area. 4A and 4B, the processor 220 may display the area of the pedestrian extracted from the candidate area as a rectangle.

At this time, one pedestrian can be continuously extracted in a plurality of candidate regions. In addition, since the image data 310 is taken by a moving vehicle, the size and position of the same pedestrian included in each candidate region may be different from each other.

Therefore, when a plurality of candidate regions are extracted from the image data 310, the processor 220 can calculate the similarity between adjacent candidate regions out of the extracted plurality of candidate regions. The processor 220 may then determine whether the pedestrian extracted in the adjacent candidate region is the same pedestrian.

Alternatively, the processor 220 may determine that the positions of the pedestrians extracted from the continuous candidate regions among the plurality of candidate regions are similar or coincident, or the moving directions corresponding to the feature vectors are matched, as the same pedestrian.

Through this process, the processor 220 can extract a plurality of candidate pedestrian objects from the image data 310. [

Then, the processor 220 can extract the center of gravity of the pedestrian zone in which the candidate pedestrian object is extracted within the specific frame to the position of the candidate pedestrian. The processor 220 can extract a plurality of pedestrian objects through the pedestrian motion estimation module 331. [

Specifically, the processor 220 uses the pedestrian tracking model included in the pedestrian detection module 330 to recognize the area of the candidate pedestrian on the continuous frame included in the image data 310 as a pedestrian object in time . At this time, the pedestrian tracking model can be generated using a dynamic model such as a nonlinear dynamic model or a linear dynamic model. For example, it may be based on either a dynamic model or a mean-shift algorithm, a Kalman filter, and a particle filter.

Hereinafter, a case where a Kalman filter is used as a dynamic model will be described as an example.

The processor 220 may extract candidate pedestrian information corresponding to the candidate pedestrian object when the candidate pedestrian object detects the specific frame. At this time, the candidate pedestrian information includes the identifier of the candidate pedestrian, the position information, and the position and tracking information of the corresponding pedestrian in the previous frame.

In addition, the candidate pedestrian information may include the life information of the pedestrian object. At this time, the life information is set to have a high value when a specific pedestrian newly appears in the corresponding image data 310, and may be set to have a low value when the pedestrian disappears. Therefore, a pedestrian whose life information is a specific value or more may be a pedestrian newly appearing in the photographing area of the camera module 240. In addition, the pedestrian whose life information is less than or equal to a specific value can be a pedestrian out of the shooting area of the camera module 240.

The processor 220 can track the pedestrian object through the information of the candidate pedestrian and the pedestrian tracking model. At this time, the processor 220 can estimate the position change in the previous frame and the current frame, the previous window, and the current window through the pedestrian tracking model, assuming that the pedestrian maintains a constant velocity to track the temporal pedestrian object.

For example, if processor 220 extracts n pedestrians in a first window and m pedestrians in a second window, processor 220 may generate a cost matrix of size n X m.

In this case, the cost is calculated by comparing how the position predicted in the previous window corresponds to the position detected in the actual window corresponding to the specific pedestrian. The cost can be calculated using the distance measurement scale or the similarity measurement scale. For example, the distance measure may be, but is not limited to, Euclidian distance and Manhattan distance. The similarity measure may be, but not limited to, a Pearson's correlation coefficient, a Spearman correlation coefficient, a cosine similarity, and a Jacquard coefficient.

The processor 220 may then extract the pedestrian object based on the cost matrix. For example, processor 220 may extract a pedestrian object based on a least cost problem using a greedy algorithm.

At this time, if m extracted from the second window is larger than n extracted from the first window, the processor 220 can detect that a new pedestrian object that is not present in the previous frame or the previous window is added to the image data. If m extracted from the second window is smaller than n extracted from the first window, the processor 220 may determine that the pedestrian object detected in the previous frame has moved outside the area of the camera module 240. [

Thus, the processor 220 may reduce the life cycle of the pedestrian object tracked as moving out of the area of the camera module 240, or may set it to a predetermined value. The processor 220 may then terminate tracing for a pedestrian object whose life cycle is less than or equal to a predetermined value.

At this time, if two specific pedestrian objects are overlapped on the image data 310, or the two pedestrian objects are staggered, the recognized pedestrian object may be confused. Therefore, the processor 220 may compare the feature vectors in some current windows of the feature vectors in the previous window, and apply a Kalman filter to correct them so that they can be tracked even if the pedestrian object is out of the predicted position. The processor 220 can prevent confusion of the pedestrian object through this process.

The processor 220 extracts a pedestrian object through the pedestrian detection module 330 and traces the motion through the pedestrian motion estimation module 331. The pedestrian motion estimation module 332 then detects a plurality of pedestrian objects Can be predicted. In this case, the route prediction process for the pedestrian object will be described later.

Meanwhile, the processor 220 can estimate the motion of the camera module 240 through the camera motion estimation module 320. For this purpose, the processor 220 may perform matching of features extracted from the image data 310. [

5 is an exemplary view of feature matching for image data 310 according to an embodiment of the present invention.

Specifically, the processor 220 can set a reference point to calculate a transformation matrix between two specific windows or two frames included in the image data 310 based on a feature descriptor. For example, the feature descriptor may use a descriptor having characteristics such as a Harris corner, a Scale-invariant feature transformation (SIFT), and a Speeded-up robust feature (SURF).

At this time, if only a single feature descriptor is considered, many abnormal values may be included. Therefore, if only a single feature descriptor is used, the transformation matrix may become inaccurate. Accordingly, the processor 220 according to an embodiment of the present invention computes a transformation matrix using one or more feature descriptors between windows or frames. And the processor 220 may match one or more feature descriptors based on the Delaunay triangulation. For example, processor 220 may compute a transformation matrix between two windows or two frames based on ten or more feature descriptors and Trine triangulation.

In addition, the processor 220 may estimate the motion of the camera module 240 based on the information of the camera and a random sample consensus (RANSAC) algorithm. At this time, the information of the camera may include a focal distance of the camera for collecting the image data 310, a main point, installation information, and the like. The installation information of the camera may be the height and angle of the camera module 240 and the like.

At this time, the camera information may be input by the user of the collision risk warning device 210. Alternatively, the information of the camera may be collected through meta information of the image data 310, or analogized through the information of the vehicle in which the camera module 240 is installed.

The processor 220 randomly selects one of the pair of feature descriptors selected as the feature matching based on the RANSAC algorithm to calculate a three-dimensional transformation matrix between the two frames. The processor 220 may repeat the three-dimensional transformation matrix calculation process for the two windows or frames included in the image data 310. The processor 220 can select a transformation matrix including the most pairs of feature descriptors having a minimum error in the repeated calculation of the three-dimensional transformation matrix. The processor 220 may then extract the selected transformation matrix as a camera motion matrix between two windows or frames.

In this manner, the processor 220 can repeatedly estimate the camera motion for a plurality of windows or a plurality of frames included in the image data 310. [

Then, the processor 220 can execute the motion of the vehicle and the motion of the pedestrian object in the three-dimensional space based on the calibration formula. For example, the calibration formula is as shown in Equation (1) or Equation (2).

'(X, Y, Z)' is an actual three-dimensional coordinate and '(u, v)' is a two-dimensional coordinate on the camera module 240 in Equations (1) and (2). Also, '[R | t]' is the 3D rotation transformation matrix, 'A' is the camera information matrix, '(cx, cy)' is the camera principal point and '(fx, fy)' is the camera focal length.

The processor 220 may generate a three-dimensional graph based on the calculated three-dimensional information.

FIG. 6 is an exemplary view of a camera module 240 and a position of a pedestrian object according to an embodiment of the present invention. 6B is a graph 600 illustrating the three-dimensional information of the camera module 240 and the pedestrian object extracted from the image data 310 of FIG. 6A.

In FIG. 6 (b), the center line 620 in the graph 600 represents the movement of the camera module 240. At this time, since the camera module 240 is attached to the vehicle, the center line 620 may indicate the movement of the vehicle. In addition, the short solid lines 640 and 650 near the center line 620 indicate the movement of the pedestrian object. 6B, the position of the first pedestrian 640 and the position of the second pedestrian 630 can be confirmed on the side of the vehicle 620 by confirming the enlarged graph 610 obtained by enlarging a part of the graph 600 have.

The processor 220 can predict the movement path of the vehicle and the pedestrian object corresponding to the camera module 240 through the vehicle movement path prediction module 323 and the pedestrian movement path prediction module 332. [ At this time, the processor 220 can use the dynamic model as described above. For example, the dynamic model may be a Kalman filter.

Specifically, the processor 220 may use the continuous three-dimensional position information corresponding to each of the plurality of objects as an input of the dynamic model to learn the movement path. The processor 220 may then use this to predict the location of the next frame in the current frame. The processor 220 can again learn the dynamic model by using the position of the predicted frame again as the input of the dynamic model. The processor 220 may then predict the location of the object in the next frame.

That is, the processor 220 may repeat the t times to predict the movement path of the object. If the t value is too small, the predicted position may be too short. If the t value is too large, the prediction error can be accumulated. Therefore, a value between 15 and 30 can be used for t times.

The processor 220 may predict the movement path of the camera module 240 and the pedestrian object through a method based on the dynamic model described above. The processor 220 can detect a collision risk of the pedestrian object when the movement path of the camera module 240 and the movement path of the specific pedestrian object intersect with each other through the collision determination module 340. [

Specifically, the processor 220 may calculate parallelogram coordinates so that an area corresponding to a specific pedestrian object and an area including a predicted path of the pedestrian object are included. The processor 220 may then calculate the trapezoidal coordinates including the predicted path corresponding to the vehicle. At this time, the trapezoidal area can be generated considering the current vehicle speed. For example, the trapezoidal area may have a shape that becomes wider as it gets farther away from the vehicle.

The processor 220 may calculate the degree of overlap based on the parallelogram and trapezoid coordinates. The processor 220 may then calculate the risk of the pedestrian object based on the degree of overlap. The processor 220 can classify the state of the pedestrian object into a dangerous state when the risk of the pedestrian object is greater than or equal to a predetermined value. For example, the processor 220 may calculate a risk based on the degree of overlap, and may determine a potential risk if the risk is greater than 30%.

In another embodiment of the present invention, the processor 220 may determine the risk of collision of the pedestrian object by calculating the combination probability of the movement path and the vehicle movement path corresponding to the specific pedestrian object.

For example, the processor 220 may express the moving path of the pedestrian object and the moving path of the vehicle in a normal distribution form based on the change amount of the current speed. Then, the processor 220 can calculate the coupling probability between the two movement paths. The processor 220 may determine that the pedestrian object is a potentially dangerous state if the joint probability is greater than or equal to a predetermined value.

The processor 220 may then warn of the risk of a collision if the pedestrian object is determined to be a potentially dangerous or dangerous state.

For example, the processor 220 may warn of a crash risk through a voice module (not shown) included in the crash risk warning device 210. Alternatively, the processor 220 may display a crash risk warning and pedestrian information in a display module (not shown) included in the crash risk warning device 210.

In addition, when the vehicle is connected to the vehicle through a communication module (not shown), the processor 220 may transmit a warning of collision risk and pedestrian information to the vehicle so that the vehicle warns the driver of the risk of collision. Alternatively, the processor 220 may transmit collision risk warning and pedestrian information to a black box device for collecting the image data 310 or a navigation device connected to the collision risk warning device 210.

Meanwhile, the processor 220 may display the extracted pedestrian on the image data through a display module (not shown) or a display device connected to the collision risk warning device 210. At this time, the display device may be a separate device including a display module such as a black box device and a navigation device, and may be a head-up display device capable of displaying an augmented reality, but is not limited thereto.

FIGS. 7 and 8 are illustrations of image data in which an identified pedestrian object is displayed according to an embodiment of the present invention.

The processor 220 may generate a layer based on the extracted positional information and identification information of the pedestrian object. For example, referring to FIG. 7, the processor 220 may generate a layer so that the position of a pedestrian object may appear as a rectangle in the image data.

In addition, the processor 220 may display the movement path of the pedestrian object at the position of the pedestrian object. For example, the processor 220 may position the points of the "o" shape continuously to indicate the direction of movement of the pedestrian object.

Referring to FIG. 7A, the processor 220 can display the position and the movement path of the pedestrian 700.

7 (b), (c), and (d), if the image data includes a plurality of pedestrians, the processor 220 can generate the layers so that the colors of the rectangles of the respective pedestrians are different . 7 (b) and 7 (c), the processor 220 can display only the moving path of the moving pedestrians 710 and 720 among the plurality of pedestrians. Alternatively, the processor 220 may display the movement path to all of the plurality of pedestrians 730 and 740, as shown in FIG. 7B.

The processor 220 may create a layer to separately display the high-risk pedestrian of the pedestrian based on the risk of each pedestrian.

The processor 220 can express the movement path and position of the pedestrian by applying transparency and gradation. For example, the processor 220 may be represented by a dark color near the position of the pedestrian, and may be represented by a light color at a position away from the position of the pedestrian. In addition, the processor 220 can display a triangle indicating the position of the pedestrian on the upper part of the pedestrian.

For example, referring to FIG. 8, the processor 220 may cause the center of gravity of the pedestrian 800 to be displayed in red with transparency. In addition, the processor 220 may apply a gradient so that the red color becomes lighter and the yellow color becomes darker as the pedestrian 800 moves further away.

The processor 220 may merge the previously described layers into the image data. In addition, the processor 220 may display the layered image data through a display module (not shown) or may transmit the layered image data to a separate device.

Meanwhile, the processor 220 can generate an object recognition model using the collected learning image data. At this time, the collected learning image data may be the image data collected to generate the object recognition model or the traveling image data collected in the past.

Specifically, the processor 220 can extract a frame including a pedestrian object from the collected learning image data. The processor 220 may then generate an object recognition model based on a frame containing the extracted pedestrian object and a machine learning algorithm for classification or clustering.

The processor 220 may store the object recognition model in the memory so that the pedestrian detection module 330 can use the generated object recognition model.

Next, a collision risk detection method of the collision risk warning device 210 according to an embodiment of the present invention will be described with reference to FIG.

9 is a flowchart of a collision risk detection method of the collision risk warning device 210 according to an embodiment of the present invention.

The collision risk warning device 210 extracts one or more objects from the image data (S900).

Specifically, the collision risk warning device 210 may extract a candidate object from one or more windows included in the image data based on the sliding window technique. The collision risk warning device 210 may extract a plurality of objects by combining a plurality of windows from which candidate objects are extracted. The collision risk warning device 210 can track the extracted objects, respectively. At this time, the sizes of the one or more windows may be different from each other or may be the same.

After one or more objects have been extracted, the collision risk warning device 210 reconstructs three-dimensional spatial information from the image data (S910). The collision risk warning device 210 may calculate a transformation matrix including a predetermined number of feature descriptors for each frame included in the image data and the previous frame of each frame based on Delaunay triangulation .

The collision risk warning device 210 predicts the movement path of each object based on the three-dimensional spatial information and the created movement path prediction model (S920).

In addition, the collision risk warning device 210 predicts the movement path of the camera module 240 that has captured the image data based on the three-dimensional spatial information and the generated motion path prediction model (S930). At this time, the collision risk warning device 210 repeatedly inputs the reconstructed three-dimensional spatial information of a predetermined number of frames included in the image data to the dynamic model so that the movement path for each object or the movement of the camera module 240 The path can be predicted.

In step S940, the collision risk warning device 210 determines whether there is an object that intersects the movement path and the movement path of the camera module 240 among at least one object.

Next, a method of generating a movement path prediction model of the collision risk warning device 210 will be described with reference to FIG.

10 is a flowchart illustrating a method of generating an object recognition model of the collision risk warning device 210 according to an embodiment of the present invention.

The collision risk warning device 210 extracts learning image data including an object from the collected image data sets (S1000).

The collision risk warning device 210 generates an object recognition model based on the learning image data and the machine learning algorithm (S1010). At this time, the object recognition model extracts a plurality of objects from the image data. The machine learning algorithm is either a neural network algorithm, a support vector machine algorithm, or a clustering algorithm.

That is, after the movement route is predicted based on the reconstructed three-dimensional spatial information corresponding to the image data, the collision risk warning device 210 compares the movement route to the camera module 240, The movement path and object to the module 240 may be extracted as a collision risk object.

The collision risk warning device 210 and method according to an embodiment of the present invention can restore 3D space information from 2D image data and detect a collision risk between a pedestrian and a vehicle. In addition, the collision risk warning device 210 and the method may eliminate noise that may be caused by movement of the vehicle. Therefore, the collision risk warning device 210 and method can reduce the mistrust of a dangerous pedestrian, reduce the risk of accident, and can help the driver's safety.

One embodiment of the present invention may also be embodied in the form of a recording medium including instructions executable by a computer, such as program modules, being executed by a computer. Computer readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. The computer-readable medium may also include computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.

While the methods and systems of the present invention have been described in connection with specific embodiments, some or all of those elements or operations may be implemented using a computer system having a general purpose hardware architecture.

It will be understood by those skilled in the art that the foregoing description of the present invention is for illustrative purposes only and that those of ordinary skill in the art can readily understand that various changes and modifications may be made without departing from the spirit or essential characteristics of the present invention. will be. It is therefore to be understood that the above-described embodiments are illustrative in all aspects and not restrictive. For example, each component described as a single entity may be distributed and implemented, and components described as being distributed may also be implemented in a combined form.

The scope of the present invention is defined by the appended claims rather than the detailed description and all changes or modifications derived from the meaning and scope of the claims and their equivalents are to be construed as being included within the scope of the present invention do.

200: crash hazard warning system
210: crash hazard warning device
220: Processor
230: Memory
240: Camera module

Claims (15)

In a crash hazard warning device,
The crash risk prediction program stores the memory, and
A processor for executing a program stored in the memory,
The processor extracts one or more objects from the image data based on the object recognition model, reconstructs the three-dimensional spatial information from the image data, and based on the three-dimensional spatial information and the motion path prediction model, And estimating a movement path of each of the at least one object and a moving path of a camera that has captured the image data, Device. The method according to claim 1,
Wherein the processor is configured to extract one or more objects from a plurality of windows included in the image data based on a sliding window technique and to track each object in time by matching the plurality of windows,
Wherein the image data comprises one or more frames,
Wherein the window comprises the same or a different number of frames, respectively. 3. The method of claim 2,
The processor generating a cost matrix based on the one or more objects, performing tracking on each of the objects based on the cost matrix and the greedy algorithm,
Wherein the cost matrix computes and generates a similarity between the predicted position and the actual position corresponding to each of the objects. 3. The method of claim 2,
The processor integrates the plurality of windows through a dynamic model to extract the one or more objects,
Wherein the dynamic model is based on any one of a Mean-shift algorithm, a Kalman filter, and a particle filter. 3. The method of claim 2,
The processor repeatedly inputs reconstructed three-dimensional spatial information of a predetermined number of frames included in the image data to the dynamic model to predict a movement path for each of the objects or a movement path of the camera,
Wherein the dynamic model is based on any one of a Mean-shift algorithm, a Kalman filter, and a particle filter. The method according to claim 1,
The processor calculates a transform matrix including at least 10 feature descriptors for each frame included in the image data and a previous frame of each frame based on Delaunay triangulation, And predicts a moving path of each of the objects and a moving path of the camera based on the moving path of the object. The method according to claim 1,
Wherein the processor extracts learning image data including an object from the collected image data set and generates the object recognition model based on the learning image data and the machine learning algorithm,
Wherein the machine learning algorithm is one of a neural network algorithm, a support vector machine algorithm, and a clustering algorithm. The method according to claim 1,
Wherein the processor generates a layer corresponding to each of the objects, merges the layer into the image data,
Wherein the layer includes identification information of the object and a predicted travel path corresponding to the object. The method according to claim 1,
Wherein the camera is a monocular camera including one lens,
Wherein the object extracted from the image data is a pedestrian on the road,
Wherein the image data is two-dimensional image data photographed in a vehicle using the monocular camera. A method for detecting a collision risk of a collision hazard warning device,
Extracting one or more objects from the image data;
Reconstructing three-dimensional spatial information from the image data;
Estimating a movement path of each of the objects based on the three-dimensional spatial information and the created motion path prediction model;
Estimating a moving path of a camera that has photographed the image data based on the three-dimensional spatial information and the pre-generated moving path prediction model; And
Determining whether there is an object that intersects a movement path and a movement path of the camera among the one or more objects. 11. The method of claim 10,
Wherein the extracting of the one or more objects comprises:
Extracting one or more objects from a plurality of windows included in the image data based on a sliding window technique; And
And tracking each of the objects by temporally matching the plurality of windows,
Wherein the image data comprises one or more frames,
Wherein the windows each comprise the same or a different number of frames. 11. The method of claim 10,
Wherein reconstructing the three-dimensional spatial information comprises:
A transformation matrix including at least a predetermined number of feature descriptors for each frame included in the image data and a previous frame of each frame based on Delaunay triangulation,
Wherein the step of predicting the movement path of each of the at least one object or the step of predicting the movement path of the camera comprises:
And predicts the movement path of each of the objects and the movement path of the camera based on the calculated transformation matrix. 11. The method of claim 10,
Wherein the step of predicting the moving path of each of the objects or the step of predicting the moving path of the camera comprises:
Dimensional space information of a predetermined number of frames included in the image data is repeatedly input to the dynamic model to predict a moving path for each of the objects or a moving path of the camera,
Wherein the dynamic model is based on a Mean-shift algorithm, a Kalman filter, and a particle filter. A method for generating a model of a crash risk warning device,
Extracting learning image data including an object from the collected image data sets; And
And generating an object recognition model based on the learning image data and the machine learning algorithm,
The object recognition model extracts a plurality of objects from the image data,
Wherein the extracted plurality of objects are compared with a movement path for a camera that captures the image data after the movement path is predicted based on the reconstructed three-dimensional space information corresponding to the image data, And objects are extracted as collision dangerous objects,
Wherein the machine learning algorithm is one of a neural network algorithm, a support vector machine algorithm, and a clustering algorithm. A computer-readable recording medium recording a program for performing the method according to any one of claims 10 to 14 on a computer.

Images