KR102467446B1 - Ai based collision recognition method and device - Google Patents

Abstract

It relates to an artificial intelligence-based object collision recognition method, which includes the steps of detecting a 3D object for a surrounding object by applying an input image to an artificial intelligence-based 3D object detection model, the detected surrounding object Predicting the movement of the surrounding object by applying an artificial intelligence-based motion prediction model, calculating the movement speed of the surrounding object using the detected 3D object, and based on the calculated movement speed, determining the movement speed of the reference object and the surrounding object. Calculating a collision time between objects, determining whether a collision threshold corresponding to the characteristics of the surrounding objects has been reached in consideration of at least one information of the characteristics of the surrounding objects, the predicted movement, and the moving speed and re-predicting the motion of the surrounding object based on the key-point used in the motion prediction model when it is determined that the collision threshold has been reached.

Description

Artificial intelligence based collision recognition method and device {AI BASED COLLISION RECOGNITION METHOD AND DEVICE}

The present application relates to an artificial intelligence-based collision recognition apparatus and method.

Recently, 2D object recognition technology has been sufficiently developed and commercialized, but 3D object recognition technology is being developed to expand to a little higher accuracy and various application fields.

3D object recognition technology currently used in vehicles uses radar and lidar, but this requires a relatively high price compared to processing with cameras. In addition, there is a problem that the accuracy may be reduced due to the influence of the weather.

In addition, the number one place where traffic accidents occur the most is the intersection. This shows a high accident rate because it is impossible to predict when an object will appear in a corner and an unexpected situation will occur. Currently, a sensor detection intersection accident prevention system is being installed on a pilot basis, but since it uses a simple sensor, its accuracy is low and the probability of malfunction is high.

The background technology of the present application is disclosed in Korean Registered Patent Publication No. 10-2101986.

The present invention is intended to solve the problems of the prior art described above, and artificial intelligence that can recognize and prevent collisions of objects by developing from 2D object recognition and recognizing objects in 3D based on artificial intelligence to extract the accuracy range of the object An object of the present invention is to provide an apparatus and method for recognizing object collision based on the object.

The present invention is to solve the above-mentioned problems of the prior art, by applying an artificial intelligence-based 3D object detection method to detect a collision situation in real time, and when a collision situation is detected, the situation is immediately transmitted to user terminals such as drivers and pedestrians. An object of the present invention is to provide an artificial intelligence-based object collision recognition device and method that can be informed by

However, the technical problem to be achieved by the embodiments of the present application is not limited to the technical problems described above, and other technical problems may exist.

As a technical means for achieving the above technical problem, an artificial intelligence-based object collision recognition method according to an embodiment of the present invention applies an input image to an artificial intelligence-based 3D object detection model to detect a 3D object for a surrounding object. Step, predicting the motion of the surrounding object by applying the detected surrounding object to an artificial intelligence-based motion prediction model, calculating the moving speed of the surrounding object using the detected 3D object, and the calculated moving speed Calculating a collision time between a reference object and the surrounding object based on, a collision threshold corresponding to the characteristic of the surrounding object in consideration of at least one information of the characteristic of the surrounding object, the predicted movement, and the moving speed. ), and if it is determined that the collision threshold has been reached, re-predicting the motion of the surrounding object based on the key-point used in the motion prediction model. have.

The object collision recognition method based on artificial intelligence may further include providing notification information when it is determined that a collision between the reference object and the surrounding object will occur based on the re-predicted motion of the surrounding object. have.

In the re-predicting the motion of the surrounding object, feature points of the surrounding object may be extracted using the key-point, and the motion of the surrounding object may be predicted based on the movement of the feature point.

In addition, the detecting of the 3D object may include applying the input image to the 3D object detection model, which is a deep neural network training model based on 3D object detection, to obtain a 3D image corresponding to at least one of the surrounding objects included in the input image. object can be detected.

In the step of predicting the motion of the surrounding object, the next motion of the surrounding object included in the input image may be estimated from the motion prediction model, which is a pose-estimation-based deep neural network training model.

In addition, the calculating of the collision time may include determining the location of the 3D object based on the location information of the 3D object included in the 3D object detection result of the previous frame and the location information of the 3D object included in the 3D object detection result of the current frame. Movement speed can be calculated.

According to an embodiment of the present invention, an artificial intelligence-based object collision recognition apparatus includes a 3D object detection unit that detects a 3D object of a surrounding object by applying an input image to an artificial intelligence-based 3D object detection model, and detects the detected surrounding object. A prediction unit for predicting the motion of the surrounding object by applying an artificial intelligence-based motion prediction model, calculating the movement speed of the surrounding object using the detected 3D object, and based on the calculated movement speed, the reference object and the Whether a collision threshold corresponding to the characteristics of the surrounding objects has been reached based on at least one information of a calculation unit for calculating the collision time between the surrounding objects and the characteristics of the surrounding objects, the predicted movement, and the moving speed. A determination unit for determining, wherein the prediction unit may re-predict the motion of the surrounding object based on a key-point used in the motion prediction model when the determination unit determines that a collision threshold has been reached. .

In addition, the artificial intelligence-based object collision recognition apparatus may further include a notification providing unit providing notification information when it is determined that a collision between the reference object and the surrounding object will occur based on the re-predicted motion of the surrounding object. can

According to an embodiment of the present invention, an artificial intelligence-based collision recognition system generates an input image by photographing a predetermined surveillance target space, and applies the input image to an artificial intelligence-based 3D object detection model to obtain information about surrounding objects. Detecting a 3D object, predicting the motion of a surrounding object by applying the detected surrounding object to an artificial intelligence-based motion prediction model, calculating a collision time between a reference object and the surrounding object, and corresponding to the characteristics of the surrounding object A collision recognition device for re-predicting the motion of the surrounding object based on a result of determining whether a collision threshold has been reached, wherein the collision recognition device includes: an image receiving unit for receiving an input image captured by the camera module; A 3D object detection unit for detecting a 3D object of a surrounding object by applying an intelligence-based 3D object detection model, a prediction unit for predicting the motion of the surrounding object by applying the detected surrounding object to an artificial intelligence-based motion prediction model, detection A calculation unit for calculating a moving speed of the surrounding object using the 3D object and calculating a collision time between a reference object and the surrounding object based on the calculated moving speed, and characteristics of the surrounding object, predicted motion, and and a determination unit for determining whether a collision threshold corresponding to a characteristic of the surrounding object has been reached based on at least one piece of information among movement speeds, wherein the estimation unit determines whether the collision threshold is reached in the determination unit. When it is determined that the motion has been reached, the motion of the surrounding object may be re-predicted based on the key-point used in the motion prediction model.

The above-described problem solving means are merely exemplary and should not be construed as intended to limit the present disclosure. In addition to the exemplary embodiments described above, additional embodiments may exist in the drawings and detailed description of the invention.

According to the above-described problem solving means of the present application, when a driver is driving in the field of autonomous driving, surrounding objects are recognized as 3D based on the position and direction (X, Y, Z), so that the object appears or the distance is close A collision can be prevented by giving a warning faster and more accurately than in the case of a loss.

According to the above-mentioned problem solving means of the present application, it can be applied to a wide range of fields such as the field of using robots, the field of image processing, the field of transportation, etc., and can perform work with higher accuracy by applying artificial intelligence-based 3D object detection technique .

According to the above-described problem solving means of the present application, it is used not only in general traffic situations but also in areas such as situations in which illegally parked vehicles suddenly appear on living roads, unexpected situations that may occur in narrow parking lots, and driving mechanical vehicles in factories to reduce accident rates. can be lowered

However, the effects obtainable herein are not limited to the effects described above, and other effects may exist.

1 is a schematic configuration diagram of an artificial intelligence-based object collision recognition system according to an embodiment of the present invention.
2 is a schematic configuration diagram of an object collision recognition system based on artificial intelligence according to another embodiment of the present invention.
3 is a diagram schematically showing an application of 3D object recognition of an artificial intelligence-based object collision recognition apparatus according to an embodiment of the present disclosure.
4 is a diagram for explaining the prediction of a motion of a person among surrounding objects by applying an artificial intelligence-based motion prediction model of an apparatus for recognizing object collision based on artificial intelligence according to an embodiment of the present disclosure.
5 is a schematic block diagram of an object collision recognition apparatus based on artificial intelligence according to an embodiment of the present disclosure.
6 is an operation flowchart of an object collision recognition method based on artificial intelligence according to an embodiment of the present invention.

Hereinafter, embodiments of the present application will be described in detail so that those skilled in the art can easily practice with reference to the accompanying drawings. However, the present disclosure may be implemented in many different forms and is not limited to the embodiments described herein. And in order to clearly describe the present application in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the present specification, when a part is said to be “connected” to another part, it is not only “directly connected”, but also “electrically connected” or “indirectly connected” with another element in between. "Including cases where

Throughout the present specification, when a member is referred to as being “on,” “above,” “on top of,” “below,” “below,” or “below” another member, this means that a member is located in relation to another member. This includes not only the case of contact but also the case of another member between the two members.

Throughout the present specification, when a certain component is said to "include", it means that it may further include other components without excluding other components unless otherwise stated.

1 is a schematic configuration diagram of an object collision recognition system based on artificial intelligence according to an embodiment of the present invention, and FIG. 2 is a schematic configuration diagram of an object collision recognition system based on artificial intelligence according to another embodiment of the present disclosure.

Referring to FIGS. 1 and 2 , the AI-based object collision recognition system 1 may include a collision recognition device 10 , a camera module 20 and a user terminal 30 . The collision perception device 10, the camera module 20, and the user terminal 30 may interwork through a network.

According to an embodiment of the present application, the artificial intelligence-based object collision recognition system 1 may detect collision occurrence and recognize the possibility of occurrence using an AI-based 3D object detection method. In addition, the artificial intelligence-based object collision recognition system 1 can extract an accurate range of an object in 3D object recognition compared to 2D object recognition. The artificial intelligence-based object collision recognition system (1) uses the characteristics of 3D object recognition technology to recognize collisions through accurate distance and acceleration between objects, and also to detect changes in the relative position of each key-point once through key-point extraction technology. Further analysis can predict collision avoidance between objects more accurately.

According to an embodiment of the present disclosure, the collision perception apparatus 10 may detect a 3D object of a surrounding object by applying an input image provided from the camera module 20 to an artificial intelligence-based 3D object detection model. In other words, the collision perception apparatus 10 may detect a 3D object of the surrounding object through an artificial intelligence-based 3D object detection model in order to detect the surrounding object included in the input image. The input image may include a video sequence input in real time. The collision perception apparatus 10 may detect (extract) 3D object information using a bounding box or an object area displayed in an arbitrary shape (contour). For example, the surrounding object may be an object that is located in a region away from a predetermined reference object and may collide with the reference object.

The camera module 20 shown in FIG. 1 may be a vehicle black box provided in the vehicle 2 . Also, the camera module 20 may be a photographing device provided on a roof of the vehicle 2 . The collision perception device 10 may receive an input image captured from a vehicle black box or a photographing device and apply it to an artificial intelligence-based 3D object detection model to detect a 3D object for a surrounding object. For example, the input image may be an image of the front, rear, left, and right regions of the vehicle 2 currently driving. In addition, the collision perception device 10 may receive an input image centered on the currently running vehicle 2 from a CCTV recording device installed on a road or sidewalk. As an example, the input image may be an image obtained by capturing a predetermined space to be monitored centered on the currently running vehicle 2 . Meanwhile, the surrounding object may include a vehicle, a bicycle, a motorcycle, a pedestrian, an animal, and the like located in front, rear, left, or right of the currently driving vehicle.

The camera module 20 shown in FIG. 2 may be a CCTV recording device provided in a space (eg, a factory) where work is performed using a robot. The collision perception device 10 may detect a 3D object of a surrounding object by receiving an input image captured from a CCTV camera and applying it to an artificial intelligence-based 3D object detection model. As an example, the input image may be an image obtained by capturing a predetermined surveillance target space centered on a reference object (eg, a robot). In addition, the surrounding objects may include workers, unmanned robots, mobile robots, and the like located in front, rear, left, and right sides of the robot.

Since the process of recognizing a collision situation from the input image provided from the camera module 20 in the artificial intelligence-based collision perception system shown in FIGS. 1 and 2 is performed substantially similarly, hereinafter, for convenience of description, FIG. A process of recognizing a collision situation based on an input image collected from a vehicle black box provided in the car 2, which is the camera module 20 shown in, or a photographing device installed on the roof of the car 2, will be described. .

3 is a diagram schematically showing an application of 3D object recognition of an artificial intelligence-based object collision recognition apparatus according to an embodiment of the present disclosure.

According to an embodiment of the present disclosure, the collision perception apparatus 10 may detect a 3D object of a surrounding object using a 3D object detection model built through deep neural network training to which a 3D object detection technique is applied. The 3D object detection technique is an algorithm that detects a specific object such as a person or a vehicle in an image. Illustratively, the collision perception apparatus 10 may extract object information by detecting an object for each of a plurality of frames included in the input image. Also, the collision perception apparatus 10 may extract object information by detecting one or more objects (objects) included in the input image. For example, the collision perception apparatus 10 may extract object (object) information using an object (object) area displayed in a bounding box or an arbitrary shape (contour). Also, the collision perception apparatus 10 may detect a 3D object with respect to a surrounding object by inputting a plurality of input images to a 3D object detection model. The collision perception apparatus 10 may recognize an object and display an object area in a bounding box or an arbitrary shape on the object.

Referring to FIG. 3 , the collision detection apparatus 10 may extract a surrounding object in the form of a bounding box with respect to the detected surrounding object. The bounding box may be a 3D definition of the boundary of an object in the shape of a rectangular parallelepiped or regular hexahedron. Meanwhile, an arbitrary shape (contour) may be set as a shape corresponding to an object by various techniques that are already known or will be developed in the future.

The collision perception apparatus 10 may display 3D objects for surrounding objects (eg, cars, pedestrians, bicycles, motorcycles, etc.) in the form of a bounding box. For example, as shown in FIG. 3 , the collision perception apparatus 10 detects a first surrounding object (a), a second surrounding object (b), a third surrounding object (c), and a second surrounding object (c) in one image (video) frame. For each of the fourth surrounding objects d, the object area may be divided and displayed in the form of a bounding box. Also, the collision perception apparatus 10 may change the display color of the bounding box for the pre-classified surrounding objects. For example, if the surrounding object is a vehicle such as the first surrounding object (a), the second surrounding object (b), and the third surrounding object (c), a bounding box of a first color (eg, yellow) is used. to display the object. On the other hand, if the surrounding object is a person (pedestrian) like the fourth surrounding object d, the object may be displayed using a bounding box of a second color (eg, green).

Also, the collision perception apparatus 10 may extract 3D object information by detecting an object for each frame included in the input image. Here, the object corresponds to a surrounding object and may include a person, a car, a motorcycle, a bicycle, and the like. The collision perception apparatus 10 may calculate the positions and sizes of a plurality of objects included in one video frame. In addition, the collision perception apparatus 10 may extract attributes (eg, object type, position, direction, speed, etc.) of each image (video) frame from the input image as 3D object information (3D object detection result). .

According to an embodiment of the present disclosure, the collision perception apparatus 10 may predict the movement of the surrounding object by applying the detected surrounding object to an artificial intelligence-based motion prediction model. The collision perception apparatus 10 may estimate the next motion of a nearby object by applying an input image to a motion prediction model, which is a pose-estimation-based deep neural network training model. In other words, the collision perception apparatus 10 may predict the next motion by performing deep neural network training by applying the pose-estimation technique to the detected object. For example, the collision perception apparatus 10 may estimate the next motion by applying the detected fourth surrounding object d (pedestrian) to an artificial intelligence-based motion prediction model. The estimated next motion may include moving direction, speed, and the like.

Posture estimation (pose-estimation) is a method of detecting the pose by detecting key-points of the main connection parts (joints) of the body and connecting them. In the present application, a top-down method of detecting an object in an image and estimating it inside a bounding box may be applied to predict the motion of a surrounding object.

According to another embodiment of the present invention, the collision recognition apparatus 10 determines the corresponding 3D object based on the characteristic information of the 3D object included in the 3D object detection result of the previous frame and the 3D object characteristic information included in the 3D object detection result of the current frame. It is possible to determine whether the 3D objects are the same object, and to predict the movement of the corresponding object (object). In other words, the collision perception apparatus 10 determines whether the first surrounding object a included in the previous frame and the first surrounding object a included in the current frame are the same first surrounding object a, and 1 It is possible to predict the movement of the surrounding object (a) related to whether it is currently moving in the same direction as before or traveling at the same speed.

According to an embodiment of the present disclosure, the collision perception apparatus 10 may calculate a movement speed of a surrounding object using the detected 3D object. In addition, the collision recognition apparatus 10 calculates the moving speed of the 3D object based on the location information of the 3D object included in the 3D object detection result of the previous frame and the location information of the 3D object included in the 3D object detection result of the current frame can do. The collision perception apparatus 10 may calculate the moving speed of the 3D object based on the change in the location information of the 3D object in the previous frame and the location information of the 3D object in the current frame.

For example, based on the detected 3D object information, the collision perception apparatus 10 may extract one or more pieces of object track information including an appearance time interval for each object and a frame-by-frame attribute within the appearance time interval. In this case, the appearance time section may be a section obtained by adding the appearance time and the stay time. The object track information may include a start frame number, an end frame number, and object region display information for each frame within a track. A criterion for object area display information may include a top left vertex or a center of gravity. For example, the collision perception apparatus 10 may track the detected object and store object track information in association with the tracked object. In other words, the collision perception device 10 converts the appearance time of the detected object, the dwell time within the video frame, and the attributes (eg, position, movement speed, color, texture, etc.) of each video frame within the stay time into object track information. can be stored (recorded). In addition, the collision perception apparatus 10 may calculate the ID, movement speed, and movement direction of the tracked object using the position, size, and pixel data of each detected object, but is not limited thereto.

Also, the collision detection apparatus 10 may calculate a collision time between the reference object and the surrounding object based on the calculated movement speed. The collision perception apparatus 10 may calculate a collision time between the reference object and the surrounding object based on the calculated current moving speed of the surrounding object. Also, the collision perception apparatus 10 may acquire the current moving speed of the reference object. The collision detection apparatus 10 may calculate a time when a collision occurs between the reference object and the surrounding objects in consideration of the moving speed of the reference object and the moving speed of the surrounding objects.

According to an embodiment of the present invention, the collision perception apparatus 10 determines whether a collision threshold corresponding to the characteristics of the surrounding objects has been reached based on at least one information of the characteristics of the surrounding objects, the predicted movement, and the moving speed. can determine whether When a plurality of surrounding objects are detected, the collision perception apparatus 10 may determine whether a collision threshold corresponding to a characteristic of each surrounding object has been reached. For example, the collision perception apparatus 10 determines whether a collision threshold has been reached for each of the first surrounding object (a), the second surrounding object (b), the third surrounding object (c), and the fourth surrounding object (d). can judge

The characteristics of the surrounding object may include a type of object, a direction of movement, a distance from a reference object, and the like. The predicted motion may be estimated by applying an input image to a motion prediction model, which is a deep neural network training model based on pose-estimation. Also, the movement speed may be information calculated based on a change in location information of the 3D object in the previous frame and location information of the 3D object in the current frame. Collision thresholds may be set differently in response to the characteristics of surrounding objects. For example, when a nearby object is a car, it may be set as the first collision threshold. In addition, in the case of a pedestrian, it may be set as the second collision threshold. Also, in the case of a bicycle (bicyclist), it may be set as a third collision threshold.

For example, the collision perception apparatus 10 may calculate a collision time between a reference object and a plurality of surrounding objects as 5 minutes. In this case, the first collision threshold for the first surrounding object (car) may be, for example, 4 minutes and 30 seconds. Also, the second collision threshold for the second surrounding object (pedestrian) may be, for example, 2 minutes and 30 seconds. Also, the third collision threshold for the third surrounding object (bicycle) may be, for example, 3 minutes and 30 seconds.

As another example, the collision perception apparatus 10 sets a collision prediction point when the current moving speed is maintained in consideration of the current speed and direction of the reference object and the current speed and direction of the surrounding objects, and sets the collision prediction point based on the collision prediction point. It is possible to set a collision threshold suitable for each object's characteristics. For example, when the first surrounding object is a car, it may be set as the first collision threshold. Also, when the second surrounding object is a pedestrian, it may be set as the second collision threshold. In addition, when the third surrounding object is a bicycle (cyclist), it may be set as a third collision threshold.

According to an embodiment of the present disclosure, when it is determined that the collision threshold has been reached, the collision perception apparatus 10 may re-predict the motion of the surrounding object based on the key-point used in the motion prediction model. The collision perception apparatus 10 may re-predict the motions of a plurality of surrounding objects based on key-points used in the motion prediction model, based on whether or not the collision threshold determined in correspondence with each of the plurality of surrounding objects has reached. In other words, the collision perception device 10 may re-predict the motion of the surrounding object based on the key-point used in the motion prediction model in order to determine whether the surrounding object continues to move despite reaching the collision threshold. have.

For example, the collision perception apparatus 10 determines that the first surrounding object (a) has reached the collision threshold among the results of determining whether the collision threshold has been reached for the first to fourth surrounding objects (a) to (b). In this case, the motion of the first surrounding object a may be re-predicted based on the key-point used in the motion prediction model.

4 is a diagram for explaining the prediction of motion of a person (pedestrian) among surrounding objects by applying an artificial intelligence-based motion prediction model of an apparatus for recognizing object collision based on artificial intelligence according to an embodiment of the present disclosure.

Referring to FIG. 4 , a plurality of key-points may be detected for one surrounding object (eg, a person), and the collision perception device 10 determines the key of the body to be identified (eg, a person). A preset number of key-points corresponding to joint positions may be detected. According to one embodiment of the present application, the collision perception device 10 Coordinate values of each detected key-point may be calculated. For example, 2D coordinate values based on the horizontal axis and the vertical axis of each frame of the input image may be calculated for each detected key-point. For example, the main key-points may include Head, Neck, Shoulder, Elbow, Wrist, Hip, Knee, Ankle, and the like.

According to an embodiment of the present invention, as the number of key-points detected for each of the surrounding objects (eg, people) increases, the action or motion of the surrounding objects can be accurately measured, but immediate action is required when a collision situation occurs. Considering that it must be done, the calculation for recognizing the collision situation must be performed at a high speed. Therefore, instead of considering all positions of the body for each of the surrounding objects (eg, people), the number of key-points to be detected falls within the range of 24 or less in consideration of the calculation speed for collision detection in the present application. can However, it is not limited thereto, and if the body of a surrounding object (eg, person) is partially covered by a predetermined object or feature in the CCTV image, considering this, the surrounding object (eg, person) For , only a smaller number of key-points can be detected.

In particular, according to an embodiment of the present application, the collision perception device 10 A first feature obtained by extracting the position of a predetermined joint from the learning image and a second feature indicating which peripheral object (eg, a person) the extracted joint corresponds to are obtained, and the first feature and the second feature are previously secured. It can be implemented to detect a key-point based on a deep learning-based key-point extraction algorithm generated based on a feed-forward network that optimizes the first feature and the second feature by comparing the obtained human posture data. . According to an embodiment of the present application, the first feature may be Confidence Maps, and the second feature may be Affinity Field.

As an example, a deep learning-based key-point extraction algorithm obtains 2-dimensional confidence maps (first features) for each body part of a surrounding object (eg, a person) in an input color image (eg, a training image). Predictable. Although the first feature may be extracted insignificantly at the beginning of learning for generating a key-point extraction algorithm based on deep learning, comparison with human posture data previously obtained based on a feed-forward network When optimized through , key-points close to the joint positions of surrounding objects (eg, people) are gradually extracted. The second feature, Affinity field, reflects the characteristics (vector field set) indicating whose joint position the extracted joint position belongs to. A human skeleton including a plurality of key-points can be extracted by extracting 2 features and combining them. In particular, when a Human Skeleton is created by combining each part of the body of a surrounding object (eg, a person), it is possible to determine which surrounding object (eg, a person) each part belongs to through Greedy Relaxation. have.

According to an embodiment of the present disclosure, the collision perception apparatus 10 may provide notification information when it is determined that a collision between a reference object and a nearby object will occur based on the re-predicted movement of the surrounding object. For example, when it is determined that a collision between a reference object and a surrounding object will occur, the collision recognition device 10 detects a navigation device provided in the vehicle 2, a head-up display (HUD), or a terminal such as a driver's smartphone and the like. Notification information may be provided through a screen and/or a sound device. In addition, the collision detection device 10 may provide notification information to a portable terminal possessed by a pedestrian or a portable terminal possessed by a cyclist or motorcyclist.

According to one embodiment of the present application, the camera module 20 may generate an input image by capturing a predetermined space to be monitored. For example, referring to FIG. 1 , the camera module 20 may be provided in the vehicle 2 . The camera module 20 provided in the vehicle 2 may be a vehicle black box. A vehicle black box (vehicle video accident recorder, Event Data Recorder, EDR) can record accidents before and after a collision to provide information necessary to understand the circumstances of the accident. The black box for a vehicle may generate input images obtained by photographing the front, rear, left, and right sides of the currently driving vehicle 2 .

As another example, referring to FIG. 2 , the camera module 20 may be a CCTV recording device. The camera module 20 may be provided in a space where work is performed using a robot. The robot is an industrial robot and may be an automated guided vehicle (AGV). A situation in which the robot is performing a task is photographed using the camera module 20, and a collision between the robot (reference object) and a surrounding object (eg, a worker, an unmanned transport vehicle, etc.) through the collision recognition device 10 The accident rate can be reduced by determining whether it has occurred and providing notification information.

1 and 2 show that the collision perception device 10 and the camera module 20 are provided separately, but in the collision perception system 1, the collision perception device 10 itself has a camera module to capture images. A collision situation may be recognized from and when a collision situation is detected, a notification signal may be generated and transmitted to the user terminal 30 .

According to another embodiment of the present application, the collision perception device 10 may receive input images from a plurality of camera modules 20 . The collision detection device 10 may recognize a collision between the reference object and a neighboring object by identifying one reference object in input images received from a plurality of CCTV photographing devices. In addition, the collision detection apparatus 10 may recognize a collision between the reference object and the surrounding objects by using an input image received from a vehicle black box and a CCTV photographing device that captures the surrounding area of the reference object.

According to an embodiment of the present disclosure, the user terminal 30 may receive notification information from the collision perception device 10 . The user terminal 30 may be a navigation device 32 mounted in a vehicle or a portable terminal 31 possessed by a pedestrian, cyclist, or motorcycle driver. For example, the notification information is information provided when a collision is predicted to occur, and sounds such as a warning sound, a warning sound, or a siren may be emitted to induce an object to stop driving (moving) since a collision may occur. In addition, the user terminal 30 shown in FIG. 2 may be a terminal of a manager who manages a space where work is performed using a robot.

According to an embodiment of the present disclosure, the collision perception apparatus 10 may provide a collision notification providing menu to the user terminal 30 . For example, the user terminal 30 downloads and installs an application program provided by the collision perception device 10, and a collision notification providing menu may be provided through the installed application.

The collision recognition apparatus 10 may include all kinds of servers, terminals, or devices that transmit and receive data, content, and various communication signals with the user terminal 30 through a network, and have functions of storing and processing data.

The user terminal 30 is a device that interworks with the collision recognition device 10 through a network, and is, for example, a personal communication system (PCS) with a smartphone, a smart pad, a tablet PC, a wearable device, and the like. , GSM (Global System for Mobile communication), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, It may include all types of wired/wireless communication devices such as W-Code Division Multiple Access (W-CDMA) and Wireless Broadband Internet (Wibro) terminals. The user terminal 30 in the present application may include a portable terminal 31 possessed by a car driver, a pedestrian, a cyclist, and a motorcycle driver, and a navigation device 32 mounted in the vehicle 2 .

The collision perception device 10, the camera module 20, and the user terminal 30 may be connected to each other through the network 40, and the network 40 allows information exchange between terminals and nodes such as servers. It means a possible connection structure, and examples of such networks include a 3rd Generation Partnership Project (3GPP) network, a Long Term Evolution (LTE) network, a 5G network, a World Interoperability for Microwave Access (WIMAX) network, the Internet, LAN (Local Area Network), Wireless LAN (Wireless Local Area Network), WAN (Wide Area Network), PAN (Personal Area Network), wifi network, Bluetooth network, satellite broadcasting network, analog broadcasting network, DMB (Digital Multimedia Broadcasting) networks, etc. are included, but are not limited thereto.

5 is a schematic block diagram of an object collision recognition apparatus based on artificial intelligence according to an embodiment of the present disclosure.

Referring to FIG. 5 , the collision recognition device 10 includes an image receiving unit 11, a 3D object detection unit 12, a prediction unit 13, a calculation unit 14, a determination unit 15, and a notification providing unit 16. can include However, the configuration of the collision perception device 10 is not limited thereto. For example, the collision perception apparatus 10 may further include a storage unit (not shown) for storing the input image received by the image receiving unit 11 .

According to an embodiment of the present application, the image receiving unit 11 may receive an input image captured by the camera module 20. In other words, the image receiving unit 11 captures a predetermined surveillance target space and generates an input image. may be received from the camera module 20. In addition, the image receiver 11 may receive an input image obtained by photographing the surrounding environment of the reference object. The image receiver 11 receives the input image input in real time. can do.

According to an embodiment of the present application, the 3D object detection unit 12 may detect a 3D object of a surrounding object by applying the input image to an artificial intelligence-based 3D object detection model. In addition, the 3D object detection unit 12 can detect a 3D object corresponding to at least one surrounding object included in the input image by applying the input image to a 3D object detection model, which is a deep neural network training model based on 3D object detection. have. The 3D object detector 12 may detect a 3D object for one or more surrounding objects for each of a plurality of frames included in the input image. Also, the 3D object detection unit 12 may detect a plurality of 3D objects for a plurality of surrounding objects included in one frame. The 3D object detection unit 12 may detect (extract) a 3D object of a surrounding object using a bounding box or an object area displayed in an arbitrary shape (contour). The bounding box may be a 3D definition of the boundary of an object in the shape of a rectangular parallelepiped or regular hexahedron. Meanwhile, an arbitrary shape (contour) may be set as a shape corresponding to an object by various techniques that are already known or will be developed in the future. In this case, the object may include a person (pedestrian), a car, a motorcycle, a bicycle, and the like.

According to an embodiment of the present disclosure, the prediction unit 13 may predict the motion of the surrounding object by applying the detected surrounding object to an artificial intelligence-based motion prediction model. In addition, the prediction unit 13 may estimate (predict) the next motion of a nearby object included in an input image from a motion prediction model, which is a pose-estimation-based deep neural network training model.

According to an embodiment of the present disclosure, the calculation unit 14 may calculate the moving speed of the surrounding object using the detected 3D object, and calculate the collision time between the reference object and the surrounding object based on the calculated moving speed. . In addition, the calculation unit 14 calculates the moving speed of the 3D object based on the location information of the 3D object included in the 3D object detection result of the previous frame and the location information of the 3D object included in the 3D object detection result of the current frame. can

According to an embodiment of the present invention, the determination unit 15 determines whether a collision threshold corresponding to the characteristics of the surrounding objects has been reached based on at least one information of the characteristics of the surrounding objects, the predicted movement, and the moving speed. can judge Exemplarily, the collision threshold may include at least one of time, distance, and possibility of collision. A collision threshold may be differently applied according to the characteristics of surrounding objects.

Meanwhile, when the determination unit 15 determines that the collision threshold has been reached, the prediction unit 13 may re-predict the motion of the surrounding object based on the key-point used in the motion prediction model.

According to an embodiment of the present disclosure, the notification providing unit 16 may provide notification information when it is determined that a collision between the reference object and the surrounding object will occur based on the re-predicted movement of the surrounding object. The notification providing unit 16 may provide notification information to the user terminal 30 associated with the reference object and surrounding objects. The notification providing unit 16 may provide notification information through a navigation device provided in the vehicle 2 or a terminal such as a driver's smart phone, a screen, and/or a sound device. In addition, the notification providing unit 16 may provide notification information through a peripheral object, for example, a smartphone of a pedestrian, a cyclist, a motorcycle driver, or a car driver, and a screen and/or a sound device.

Hereinafter, based on the details described above, the operation flow of the present application will be briefly reviewed.

6 is an operation flowchart of an object collision recognition method based on artificial intelligence according to an embodiment of the present invention.

The artificial intelligence-based object collision recognition method shown in FIG. 6 may be performed by the collision recognition apparatus 10 described above. Therefore, even if omitted below, the description of the collision recognition apparatus 10 can be equally applied to the description of the artificial intelligence-based object collision recognition method.

In step S601, the collision perception apparatus 10 may detect a 3D object of a surrounding object by applying the input image to an artificial intelligence-based 3D object detection model.

In step S602, the collision perception apparatus 10 may predict the motion of the surrounding object by applying the detected surrounding object to an artificial intelligence-based motion prediction model.

In step S603, the collision perception apparatus 10 may calculate the moving speed of the surrounding object using the detected 3D object, and calculate the collision time between the reference object and the surrounding object based on the calculated moving speed.

In step S604, the collision perception apparatus 10 may determine whether a collision threshold corresponding to the characteristics of the surrounding objects has been reached in consideration of at least one information among the characteristics of the surrounding objects, the predicted movement, and the moving speed. have.

In step S605, when it is determined that the collision threshold has been reached, the collision perception apparatus 10 may re-predict the motion of the surrounding object based on the key-point used in the motion prediction model.

In the foregoing description, steps S601 to S605 may be further divided into additional steps or combined into fewer steps, depending on an embodiment of the present invention. Also, some steps may be omitted if necessary, and the order of steps may be changed.

The object collision recognition method based on artificial intelligence according to an embodiment of the present disclosure may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. Program instructions recorded on the medium may be those specially designed and configured for the present invention or those known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. - includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, as well as machine language codes such as those produced by a compiler. The hardware devices described above may be configured to act as one or more software modules to perform the operations of the present invention, and vice versa.

In addition, the artificial intelligence-based object collision recognition method described above may be implemented in the form of a computer program or application stored in a recording medium and executed by a computer.

The above description of the present application is for illustrative purposes, and those skilled in the art will understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present application. Therefore, the embodiments described above should be understood as illustrative in all respects and not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined form.

The scope of the present application is indicated by the following claims rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and equivalent concepts thereof should be construed as being included in the scope of the present application.

1: Collision Recognition System
10: Collision Recognition Device
11: image receiving unit 12: 3D object detection unit
13: prediction unit 14: calculation unit
15: determination unit 16: notification providing unit
20: camera module
30: user terminal
40: network

Claims (10)

In the object collision recognition method based on artificial intelligence,
Detecting a 3D object for a surrounding object by applying the input image to an artificial intelligence-based 3D object detection model;
predicting motion of the surrounding object by applying the detected surrounding object to an artificial intelligence-based motion prediction model;
calculating a moving speed of the surrounding object using the detected 3D object, and calculating a collision time between a reference object and the surrounding object based on the calculated moving speed;
determining whether a collision threshold has been reached in consideration of at least one information of the characteristics of the surrounding object, the predicted motion, and the moving speed; and
If it is determined that the collision threshold has been reached, re-predicting the motion of the surrounding object based on a key-point used in the motion prediction model;
Including,
The determining step is
In consideration of the current speed and direction of the reference object and the current speed and direction of the surrounding objects, a collision prediction point is set when the reference object and the surrounding objects maintain their current speeds, and the collision prediction point is used as a reference to the surrounding objects. Setting the collision threshold corresponding to the characteristics of each object,
The step of re-predicting the movement of the surrounding object,
The feature points of the surrounding objects are extracted using the key-points, and the movement of the surrounding objects is re-predicted based on the movement of the feature points. A method for recognizing a collision that is detected within a preset number according to surrounding objects. According to claim 1,
Providing notification information when it is determined that a collision between the reference object and the surrounding object will occur based on the re-predicted motion of the surrounding object;
Further comprising a collision recognition method. delete According to claim 1,
The step of detecting the 3D object,
Detecting a 3D object corresponding to at least one of the surrounding objects included in the input image by applying the input image to the 3D object detection model, which is a deep neural network training model based on 3D object detection. According to claim 1,
Predicting the movement of the surrounding object,
and estimating the next motion of the surrounding object included in the input image from the motion prediction model, which is a pose-estimation-based deep neural network training model. According to claim 1,
Calculating the collision time,
Calculating a movement speed of the 3D object based on the location information of the 3D object included in the 3D object detection result of the previous frame and the location information of the 3D object included in the 3D object detection result of the current frame. . In the artificial intelligence-based object collision recognition device,
a 3D object detection unit for detecting a 3D object of a surrounding object by applying the input image to an artificial intelligence-based 3D object detection model;
a predictor for predicting motion of the surrounding object by applying the detected surrounding object to an artificial intelligence-based motion prediction model;
a calculation unit that calculates a movement speed of the surrounding object using the detected 3D object, and calculates a collision time between a reference object and the surrounding object based on the calculated movement speed; and
a determination unit for determining whether a collision threshold corresponding to a characteristic of the surrounding object has been reached based on at least one information of a characteristic of the surrounding object, a predicted movement, and a moving speed;
Including,
The prediction unit,
When the determination unit determines that a collision threshold has been reached, the movement of the surrounding object is re-predicted based on the key-point used in the motion prediction model;
The judge,
In consideration of the current speed and direction of the reference object and the current speed and direction of the surrounding objects, a collision prediction point is set when the reference object and the surrounding objects maintain their current speeds, and the collision prediction point is used as a reference to the surrounding objects. Setting the collision threshold corresponding to the characteristics of each object,
The prediction unit,
The feature points of the surrounding objects are extracted using the key-points, and the movement of the surrounding objects is re-predicted based on the movement of the feature points. A collision-aware device that detects within a preset number according to surrounding objects. According to claim 7,
a notification providing unit providing notification information when it is determined that a collision between the reference object and the surrounding object will occur based on the re-predicted movement of the surrounding object;
Further comprising a collision recognition device. In the artificial intelligence-based collision recognition system,
a camera module generating an input image by photographing a predetermined space to be monitored; and
The input image is applied to an artificial intelligence-based 3D object detection model to detect a 3D object for a surrounding object, the detected surrounding object is applied to an artificial intelligence-based motion prediction model to predict the motion of the surrounding object, and the motion of the surrounding object is predicted. A collision perception device that calculates a collision time between the surrounding objects and re-predicts the motion of the surrounding objects based on a result of determining whether a collision threshold corresponding to the characteristics of the surrounding objects has been reached;
Including,
The collision recognition device,
an image receiving unit for receiving an input image captured by the camera module;
a 3D object detection unit for detecting a 3D object of a surrounding object by applying the input image to an artificial intelligence-based 3D object detection model;
a predictor for predicting motion of the surrounding object by applying the detected surrounding object to an artificial intelligence-based motion prediction model;
a calculation unit that calculates a movement speed of the surrounding object using the detected 3D object, and calculates a collision time between a reference object and the surrounding object based on the calculated movement speed; and
a determination unit for determining whether a collision threshold has been reached based on at least one information of the characteristics of the surrounding object, the predicted motion, and the moving speed;
including,
The prediction unit,
When the determination unit determines that a collision threshold has been reached, the movement of the surrounding object is re-predicted based on the key-point used in the motion prediction model;
The judge,
In consideration of the current speed and direction of the reference object and the current speed and direction of the surrounding objects, a collision prediction point is set when the reference object and the surrounding objects maintain current speeds, and the collision prediction point is used as a reference. Setting the collision threshold corresponding to the characteristics of each object,
The prediction unit,
The feature points of the surrounding objects are extracted using the key-points, and the movement of the surrounding objects is re-predicted based on the movement of the feature points. A collision recognition system that detects within a preset number according to surrounding objects. A computer-readable recording medium recording a program for executing the method of any one of claims 1, 2, and 4 to 6 in a computer.

Images