KR20220048959A - Moving body collision avoidance device, collision avoidance method and electronic device - Google Patents

Abstract

A collision avoidance method of a moving object collision avoidance device comprises the steps of: acquiring a driving image of a moving object; recognizing an object in the driving image acquired using a neural network model; calculating a relative distance between the moving object and the object based on the recognized object; calculating a required collision time between the object and the moving object based on the calculated relative distance; and controlling the operation of the moving object based on the calculated required collision time. An object of the present invention is to provide the collision avoidance method of a moving object collision avoidance device that can provide collision warning guidance or control the moving object to be braked when there is a possibility of a collision based on the driving image of the moving object.

Description

Moving body collision avoidance device, collision avoidance method and electronic device

The present invention relates to an apparatus for preventing collision with a moving object, a method for preventing collision with a moving object, and an electronic device for reducing collision accidents of a moving object.

Since moving objects such as automobiles and motorcycles may face various environments in the course of driving, measures to reduce collision accidents of moving objects have been actively discussed from the past to the present.

In addition, recently, personal mobility (hereinafter, PM) such as electric kickboards, electric bicycles, and ninebot-electric wheels, which are not only eco-friendly but also have strengths in parking and short-distance driving, are attracting attention as a means of transportation in the future.

As the usage rate of such personal mobility increases, the problem of personal mobility safety is emerging.

In particular, personal mobility is highly likely to be exposed to collisions with cars, motorcycles, and people because it is driven on a road or sidewalk.

As described above, for the safety of users of various moving objects, a method for reducing collision accidents of moving objects is required.

An object of the present invention is to provide an apparatus for preventing collision with a moving object, a method for preventing collision, and an electronic device capable of providing a collision warning guidance or controlling a moving object to be braked when there is a possibility of a collision based on a driving image of a moving object.

A collision avoidance method of an apparatus for preventing collision with a moving object according to an embodiment of the present invention for achieving the above object includes acquiring a driving image of the moving object, and recognizing an object in the acquired driving image using a neural network model , calculating the relative distance between the moving object and the object based on the recognized object, calculating the required collision time between the object and the moving object based on the calculated relative distance, and calculating the calculated required collision time and controlling the operation of the moving object based on the base.

In addition, the neural network model may recognize an object in the driving image, classify and output a bounding box indicating an object region in the driving image and an object in the bounding box.

The method may further include tracking the movement of the object by tracking a change in the position of the object recognized in the driving image.

The method may further include normalizing a driving image in which the object is recognized.

The method may further include determining a position state of an object recognized in the driving image and determining an object to be tracked among objects recognized in the driving image based on the position state according to the type of the object. there is.

In addition, the calculating of the relative distance may generate a virtual horizontal line in the driving image based on the bounding box.

In addition, the calculating of the relative distance may include calculating the expected relative distance based on the generated virtual horizontal line and a vertical coordinate of a bottom surface of the bounding box.

In addition, the calculating of the relative distance may include calculating the expected relative distance based on the vertical height of the bounding box and the actual height of the object.

In addition, the calculating of the relative distance may include calculating the relative distance through prediction and update by applying the calculated expected relative distance to a Kalman filter.

In the calculating of the required collision time, the required collision time may be calculated in consideration of the speed of the moving object.

In addition, when there is a possibility of colliding with the object, the controlling of the operation may provide a collision warning guidance or control the moving object to be braked.

On the other hand, the apparatus for preventing collision with a moving object according to an embodiment of the present invention for achieving the above object is an image acquisition unit that acquires a driving image of the moving object, and object recognition for recognizing an object in the acquired driving image using a neural network model a calculator for calculating a relative distance between the moving object and the object based on the recognized object, and calculating a required collision time between the object and the moving object based on the calculated relative distance; and the calculated required collision time and a control unit for controlling the operation of the moving object based on the .

In addition, the neural network model may recognize an object in the driving image, classify and output a bounding box indicating an object region in the driving image and an object in the bounding box.

The apparatus may further include an object tracking unit configured to track the movement of the object by tracking a change in the position of the object recognized in the driving image.

In addition, a position state determination unit for determining the position state of the object recognized in the driving image, and a tracking determining unit for excluding from the tracking target an object with a low probability of colliding with the moving object based on the position state according to the type of the object may include

Also, the calculator may generate a virtual horizontal line in the driving image based on the bounding box.

In addition, the calculator may calculate the expected relative distance based on the generated virtual horizontal line and the vertical coordinates of the bottom surface of the bounding box.

Also, the calculator may calculate an expected relative distance based on a vertical height of the bounding box and an actual height of the object.

In addition, the calculating of the relative distance may include calculating the relative distance through prediction and update by applying the calculated expected relative distance to a Kalman filter.

In addition, the calculator may calculate the required time required for the collision in consideration of the speed of the moving object.

On the other hand, the electronic device for a moving object according to an embodiment of the present invention for achieving the above object includes a photographing unit that captures a driving image of a moving object, an image acquisition unit that acquires a driving image captured by the photographing unit, and a neural network model. A recognition unit that recognizes an object in the obtained driving image using and an output unit for outputting information for guiding the traveling of the moving object based on a calculation unit for calculating a time and the calculated required collision time.

In addition, a program code for executing the above-described collision avoidance method may be recorded in a program according to an embodiment of the present invention for achieving the above object.

According to the present invention, it is possible to determine the possibility of a collision in the driving image, and when the probability of a collision is high, to provide a collision warning guidance or to control a moving object to be braked.

In addition, the present invention can identify the cause of the loss, damage, collision accident, etc. of the moving object through the event information.

1 is a block diagram illustrating an apparatus for preventing collision of a moving object according to an embodiment of the present invention.
2 is an exemplary diagram illustrating the structure of a neural network model according to an embodiment of the present invention.
3 to 6 are exemplary views illustrating driving images of a moving object according to an embodiment of the present invention.
7 is an exemplary diagram illustrating a photographing environment of personal mobility according to an embodiment of the present invention.
8 is an exemplary diagram illustrating a Kalman filter according to an embodiment of the present invention.
9 is an exemplary diagram illustrating a coordinate system based on a moving object according to an embodiment of the present invention.
10 is a flowchart illustrating a collision avoidance method according to an embodiment of the present invention.
11 is an exemplary diagram illustrating a moving object management system according to an embodiment of the present invention.
12 is a block diagram illustrating an electronic device according to an embodiment of the present invention.
13 is a timing diagram illustrating a method for monitoring a moving object according to an embodiment of the present invention.
14 is a block diagram illustrating an autonomous driving system according to an embodiment of the present invention.
15 is a block diagram illustrating the configuration of an autonomous driving vehicle according to an embodiment of the present invention.

The following is merely illustrative of the principles of the invention. Therefore, those skilled in the art can devise various devices that, although not explicitly described or shown herein, embody the principles of the invention and are included in the spirit and scope of the invention. In addition, it should be understood that all conditional terms and examples listed herein are, in principle, expressly intended only for the purpose of understanding the inventive concept and are not limited to the specifically enumerated embodiments and states as such. .

The above-described objects, features and advantages will become more apparent through the following detailed description in relation to the accompanying drawings, and accordingly, those of ordinary skill in the art to which the invention pertains will be able to easily practice the technical idea of the invention. .

In addition, in the description of the invention, if it is determined that a detailed description of a known technology related to the invention may unnecessarily obscure the gist of the invention, the detailed description thereof will be omitted. Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram illustrating an apparatus for preventing collision of a moving object according to an embodiment of the present invention.

Referring to FIG. 1 , the apparatus 10 for preventing collision of a moving object 200 includes an image acquisition unit 11 , an object recognition unit 12 , a position state determination unit 13 , a tracking determination unit 14 , and an object tracking unit (15), a normalization processing unit 16, a calculation unit 17, and a control unit 18 may be included. The moving object 10 may provide various guidance or control for preventing the moving object from collided based on a driving image captured by the photographing device of the moving object 200 or a separate photographing apparatus.

Here, the moving object is a movable object and requires guidance, and may be, for example, a person, a dog, a vehicle, a motorcycle, a bicycle, personal mobility, and the like. Hereinafter, for convenience of description, each component module constituting the collision avoidance apparatus 10 will be described in more detail by taking the case in which the moving object is implemented as personal mobility as an example.

The image acquisition unit 11 may acquire a driving image captured by a self-photographing apparatus of the moving object or a separate photographing apparatus. For example, the image acquisition unit 11 may acquire a driving image captured by a photographing device installed on the moving object in real time while the moving object is traveling. As another example, the image acquisition unit 11 may record a driving image captured by a photographing device (eg, a photographing device installed on the wearer's helmet, a photographing device held by the wearer's hand, etc.) can be obtained with

Here, the acquired driving image may include various objects such as pedestrians, motorcycles, vehicles, and obstacles in the sidewalk or road according to the driving environment of the moving object.

The object recognizer 12 may recognize objects in the driving image obtained based on the neural network model.

Specifically, the object recognition unit 12 inputs the acquired driving image to the neural network model in units of frames, and includes a bounding box indicating an object area in the driving image and type information of the object in the bounding box. A driving video can be acquired. Here, the neural network model may be, for example, a single shot multibox detector (SSD) algorithm as an object recognition model based on a CNN network. In this regard, it will be described with reference to FIGS. 2 and 3 .

2 is an exemplary diagram illustrating the structure of a neural network model according to an embodiment of the present invention. 3 is an exemplary view showing a driving image according to an embodiment of the present invention.

2, the neural network model passes through the input driving image to conv5_3 of the pre-trained VGG-16 model, extracts the base feature, and convolves the extracted feature map to the next layer ( layer) and convolution is possible up to a 1x1 size feature map. Then, by performing object detection in the feature map using the Detector & Classifier for the feature maps extracted from each layer, the bounding box coordinate information corresponding to the object in each feature map and the object are classified. Outputs a feature map including type information.

Thereafter, a driving image including one detection result may be finally output by applying non-maximum suppression (MNS) to the feature maps.

For example, referring to FIG. 3 , when one frame of the driving image as shown in (a) of FIG. 3 is input to the neural network, the neural network generates a bounding box corresponding to an object in the driving image as shown in FIG. 3 (b). (31a, 32a, 33a, 34a) and object classification type information (31b, 32b, 33b, 34b) including the driving image may be output. Here, the bounding box includes coordinate information of the bounding box.

That is, the neural network model may recognize an object in the driving image and classify and output a bounding box indicating an object region in the driving image and an object in the corresponding bounding box by type.

Meanwhile, the type information of the bounding box and the object in the bounding box of FIG. 3 is an example, and may be expressed differently as needed. In addition, the above-described neural network model is an example, and in addition to the SSD algorithm, various algorithms such as RefineDet and YOLO may be used.

Again, returning to FIG. 1 , the position state determination unit 13 may determine the position state of the object recognized in the driving image. In this case, the position state determination unit 13 may determine the position state of the object based on the bounding box positions of the objects recognized in the driving image. Here, the position state of the object indicates where the object is located, and for example, the position state includes a state in which the recognized object is located on a sidewalk, a state located on a road, a state located on an overpass, a state located on a building railing, etc. This may include

Specifically, the position state determination unit 13 may determine the position state of the object by comparing the positions of the bounding boxes of the recognized objects with respect to the virtual horizontal line in the driving image.

For example, the position state determination unit 13 determines to what extent the bounding box of the recognized object is located to the left or right with respect to the center of the virtual horizontal line in the driving image, and the bounding box of the recognized object is the virtual The positional state of the corresponding object may be determined based on whether it is located above or below the horizontal line or the ratio of being located above the virtual horizontal line in the driving image.

Also, the position state determination unit 13 may determine the position state of the corresponding object based on the relative position difference between the bounding boxes of the recognized objects and other adjacent objects among the recognized objects in the driving image.

The tracking determiner 14 may determine a tracking target based on a location state according to the type of object.

Specifically, the tracking determining unit 14 determines the possibility of colliding with the moving object based on the positional state of the corresponding object according to the type of the recognized object, and the moving object and the object with a low probability of colliding may be excluded from tracking. And, an object with a high probability of colliding can be determined as a tracking target. In this case, the tracking determining unit 14 may determine the possibility of a collision in consideration of the current driving position of the moving object.

For example, when the type of moving object is personal mobility, the tracking determiner 14 determines that the probability of colliding with a vehicle located on a road among the objects recognized in the driving image is low, and may exclude the object from tracking.

As another example, when the type of the moving object is a vehicle, the tracking determiner 14 determines that a person located on foot among the objects recognized in the driving image has a low probability of collide and excludes the object from tracking. When it is determined that the corresponding object is misrecognized based on the positional state of the corresponding object according to the type of the recognized object, the tracking determining unit 14 may exclude the corresponding object from the tracking target.

For example, when an object of a large electric signboard or poster is recognized, the tracking determining unit 14 may exclude the corresponding object from tracking.

That is, the tracking determiner 14 may determine a meaningful object among the objects recognized in the driving image, and determine only the corresponding object as a tracking target.

Through this, the collision avoidance apparatus 10 may use the limited resource resources more efficiently.

The object tracking unit 15 may track a movement path, movement speed, and movement direction of the object by tracking a change in the position of the bounding box of the object recognized in the driving image.

Specifically, the object tracking unit 15 tracks a change in the position of a bounding box of an object recognized by comparing the previous frame and the current frame of the driving image, and the movement path and movement direction of the object based on the change in the position of the object's bounding box. can be tracked. In this case, the object updater 15 may use various types of object tracking algorithms.

For example, the object tracking unit 15 may track the position of the bounding box of the object using a centroid tracking algorithm (Centroid Tracking Algorithm). In this regard, it will be described with reference to FIG.

Referring to FIG. 4 , the object tracking unit 15 includes the center coordinates (41a, 42a, 43a, 44a, 45a: x, y) of the object recognized in the previous frame (a) of the driving image and the current frame (b). ) calculates the center coordinates of the bounding box (41b, 43b, 44b, 45b) of the recognized object, respectively, and tracks the bounding box by calculating the Euclidean distance between the center coordinates of the bounding box in the previous frame (a) and the current frame (b) can do. At this time, the object tracking unit 15 tracks the bounding box of the object assuming that the recognized objects move to the shortest distance, and a tracking ID (or unique ID) for identifying the object is assigned to the bounding box of each object. can

In addition, the object tracking unit 15 assigns a tracking ID to a new object when an object that was not in the previous frame is recognized in the current frame, and when the object in the previous frame is not recognized in the current frame, the tracking ID of the object can be deleted.

And, after tracking the bounding box of the object, the object tracking unit 15 tracks the moving path and the moving direction of the object based on the amount of change in the position of the center coordinates of the previous frame (a) and the current frame (b) of the driving image. can be calculated. In this case, the moving speed and the moving direction of the movable body may be considered.

That is, the object tracking unit 15 may track the tracking target object to track the movement of the object in the driving image, and may determine the movement path, movement speed, and movement direction of the object.

The normalization processing unit 16 may normalize the driving image including the bound box of the object.

Specifically, the normalization processing unit 16 may normalize the driving image by using a low pass filter. For example, the normalization processing unit 16 may normalize the driving image as shown in FIG. 5( a ) into the driving image as shown in FIG. 5( b ). In this case, the bound box of the object in the driving image may more accurately display the position of the object through normalization.

The calculator 17 may calculate a relative distance between the moving object and the object based on the recognized object.

First, the calculator 17 may generate a virtual horizontal line in the driving image based on the bounding box of the object recognized in the driving image.

Specifically, the calculator 17 may calculate a virtual horizontal line in the driving image by using Equation 1 below.

는 가상 수평선의 픽셀 수직 좌표 ,

은 주행 영상 내 인식된 바운딩 박스 개수,

는 i 번째 인식된 바운딩 박스의 바닥면 픽셀 수직 좌표,

는 주행 영상을 촬영한 카메라의 실제 설치된 높이,

는 i 번째 인식된 바운딩 박스의 픽셀 너비,

는 오브젝트의 실제 평균 너비,

는 오브젝트의 실제 평균 너비 오프셋,

는 전체 바운딩박스 바닥면의 평균 픽셀 수직 좌표,

는 인식된 오브젝트의 평균 너비를 의미한다. 여기서, 인식된 오브젝트의 평균 너비는 오브젝트의 바운딩 박스 평균 너비를 의미한다.in Equation 1

is the pixel vertical coordinate of the imaginary horizontal line,

is the number of bounding boxes recognized in the driving video,

is the bottom pixel vertical coordinate of the i-th recognized bounding box,

is the actual installed height of the camera that recorded the driving video,

is the pixel width of the i-th recognized bounding box,

is the actual average width of the object,

is the actual average width offset of the object,

is the average pixel vertical coordinate of the bottom of the entire bounding box,

is the average width of the recognized object. Here, the average width of the recognized object means the average width of the bounding box of the object.

)을 생성할 수 있다. In this regard, referring to FIG. 6 , the calculator 17 calculates the average vertical coordinates of a plurality of bounding boxes in the driving image, the actual height of the camera photographing the driving image, the actual average width of the object, and the average width of the recognized object. A virtual horizontal line in the driving image (

) can be created.

은 주행 영상에서 인식된 오브젝트의 바운딩 박스 각각의 너비(

,

,

,

)의 평균을 의미하며,

는 인식된 오브젝트의 바운딩박스 각각의 바닥면 높이(

,

,

,

)의 평균을 의미한다. 이때, 좌표는 픽셀단위일 수 있다.In addition,

is the width of each bounding box of the object recognized in the driving image (

,

,

,

) means the average of

is the height of the bottom of each bounding box of the recognized object (

,

,

,

) means the average of In this case, the coordinates may be in units of pixels.

In addition, the calculator 17 may calculate an expected relative distance between the corresponding object and the moving body based on the generated virtual horizontal line and the vertical coordinates of the bottom surface of the bounding box.

Specifically, the calculator 17 may calculate the expected relative distance between the object and the moving body using Equation 2 below.

는 오브젝트와 이동체의 예상 상대 거리,

는 카메라의 초점 거리,

는 주행 영상을 촬영한 카메라의 실제 설치된 높이,

는 가상 수평선,

는 인식된 바운딩박스의 바닥면 픽셀 수직 좌표를 의미할 수 있다. 이와 관련하여 도 7의 (a)를 참조하여 하나의 예시를 설명한다.in Equation 2

is the expected relative distance between the object and the moving object,

is the focal length of the camera,

is the actual installed height of the camera that recorded the driving video,

is the virtual horizon,

may mean vertical coordinates of pixels of the bottom surface of the recognized bounding box. In this regard, one example will be described with reference to FIG. 7A.

7A is an example of a photographing environment of personal mobility, and reference numeral 71 is an example of a driving image captured in the corresponding photographing environment.

)와 초점거리(

), 주행 영상(71) 내 인식된 보행자의 바운딩 박스(72) 바닥면의 높이(

), 산출된 가상 수평선의 수직 좌표(

)를 이용하여 인식된 보행자와 퍼스널 모빌리티 사이의 예상 상대 거리(

)를 산출할 수 있다.Referring to (a) of FIG. 7 , the calculator 17 calculates the actual installed height (

) and focal length (

), the height of the bottom of the bounding box 72 of the pedestrian recognized in the driving image 71 (

), the vertical coordinate of the calculated virtual horizontal line (

), the estimated relative distance between the recognized pedestrian and personal mobility (

) can be calculated.

Meanwhile, when the virtual horizontal line cannot be calculated, the calculator 17 may calculate the expected relative distance based on the vertical coordinate of the bounding box of the recognized object and the actual height of the object. Here, when the virtual horizontal line cannot be calculated, the case where the actual installed height of the camera cannot be determined or the pixel vertical coordinates of the bottom surface of the recognized object bounding box cannot be determined may be included.

Specifically, the calculator 17 may calculate the expected relative distance between the object and the moving body using Equation 3 below.

는 오브젝트와 이동체의 예상 상대 거리,

는 카메라의 초점 거리,

는 오브젝트의 실제 평균 높이,

는 오브젝트의 바운딩박스 높이를 의미할 수 있다. 이와 관련하여 도 7의 (b)를 참조하여 하나의 예시를 설명한다.in Equation 3

is the expected relative distance between the object and the moving object,

is the focal length of the camera,

is the actual average height of the object,

may mean the height of the object's bounding box. In this regard, one example will be described with reference to FIG. 7B.

7 (b) is an example of a photographing environment of personal mobility, and (73) is an example of a driving image captured in the corresponding photographing environment.

), 주행 영상(73) 내 인식된 보행자의 바운딩박스(74) 높이(

) 및 인식된 보행자의 실제 평균 신장(

)을 이용하여 인식된 보행자와 퍼스널 모빌리티 사이의 예상 상대 거리(

)를 산출할 수 있다.Referring to FIG. 7A , the calculator 17 calculates the focal length (

), the height of the pedestrian's bounding box 74 recognized in the driving image 73 (

) and the actual average height of the perceived pedestrian (

), the estimated relative distance between the recognized pedestrian and personal mobility (

) can be calculated.

Meanwhile, although personal mobility has been described as an example in FIG. 7 , the calculation process may be equally applied to other moving objects such as a vehicle.

Also, the calculator 17 may calculate the relative distance by applying the Kalman filter to the calculated expected relative distance. In this regard, it will be described with reference to FIG. 8 .

)를 칼만필터에 적용하여 예측(Predict) 및 업데이트(Update)를 통해 상대 거리(

)를 산출할 수 있다. 여기서 상대 거리(

)는 예상 상대 거리(

)를 기초로 인식된 오브젝트와 이동체 사이의 거리를 보다 정확하게 산출한 값을 의미한다.Referring to FIG. 8 , the calculator 17 calculates the calculated expected relative distance (

) to the Kalman filter to predict and update the relative distance (

) can be calculated. where the relative distance (

) is the expected relative distance (

) means a value obtained by more accurately calculating the distance between the recognized object and the moving object.

)는 [

;

;

]로 정의되며,

는 또는 기초로 산출된 예상 상대 거리,

는 이동체의 속도,

는 이동체의 가속도를 의미한다.In Fig. 8, the state variable (

)Is [

;

;

] is defined as

is the estimated relative distance calculated on the basis of

is the speed of the moving object,

is the acceleration of the moving object.

In addition, the system model may be composed of a state transition matrix, a measurement value transition matrix, system noise, and measurement value noise, and the system model may be defined as in Equation 4 below.

is the state transition matrix, is the measured value transition matrix, is the system noise, and is the measured value noise.

)의 오차를 줄인 상대 거리(

)를 산출할 수 있다. That is, the calculator 17 uses the Kalman filter to calculate the expected relative distance (

) of the relative distance (

) can be calculated.

는 이동체의 진행 방향인 종방향의 거리(Longitudinal Distance)를 의미하며 이와 관련하여 도 9를 참조하여 설명한다.On the other hand, the above-mentioned relative distance (

denotes a longitudinal distance, which is the moving direction of the moving body, and will be described with reference to FIG. 9 in this regard.

9 is an exemplary diagram illustrating a coordinate system based on a moving object according to an embodiment of the present invention.

)이고 이동체의 진행 방향(종방향)이 X축되며, 이동체의 측면 방향이 Y축이 되는 좌표계이다.Fig. 9(a) shows the moving object-based coordinate system as a 3D view, and Fig. 9(b) shows the moving-object-based coordinate system as a top view. The position is the reference point (

), the moving direction (longitudinal direction) of the moving object is the X-axis, and the lateral direction of the moving object is the Y-axis.

)는 이동체 기준의 좌표계에서 오브젝트와 이동체 사이의 X축 방향의 거리를 의미할 수 있다.That is, the relative distance (

) may mean a distance in the X-axis direction between the object and the moving body in the moving body-based coordinate system.

)를 산출할 수 있다.In addition, the calculation unit 17 uses the following Equation 5 to calculate the lateral distance (

) can be calculated.

는 오브젝트와 이동체 사이의 Y축 방향의 거리, x는 오브젝트의 바운딩 박스 픽셀 수평 좌표,

는 이동체 기준의 좌표계에서 오브젝트와 이동체 사이의 X축 방향의 거리,

는 카메라의 초점 거리를 의미할 수 있다.in Equation 5

is the distance in the Y-axis direction between the object and the moving object, x is the horizontal coordinate of the object's bounding box pixel,

is the distance in the X-axis direction between the object and the moving body in the coordinate system of the moving body,

may mean a focal length of the camera.

)를 기초로 추돌 대상 여부를 판단할 수 있다. In addition, the calculation unit 17 calculates the side distance (

) can be used to determine whether a collision target exists.

)를 비교하여 추돌 대상 여부를 판단할 수 있다. 여기서, 안전 여유 거리는 퍼스널 모빌리 기준의 좌표계에서 Y축 방향의 소정 거리로써, 이동체가 진행 방향으로 이동할 때, 이동체와 오브젝트가 서로 추돌하지 않을 최소한의 Y축 방향의 거리를 의미할 수 있다. 이와 관련하여 도 9의 (b)를 참조하여 설명한다.Specifically, the calculation unit 17 calculates the side distance and the safety margin distance,

) can be compared to determine whether a collision target exists. Here, the safety clearance distance is a predetermined distance in the Y-axis direction in the coordinate system based on personal mobility, and may mean the minimum distance in the Y-axis direction at which the movable body and the object do not collide with each other when the movable object moves in the traveling direction. In this regard, it will be described with reference to FIG. 9(b).

,

)를 산출하고, 산출된 측면 거리(

,

)와 안전 여유 거리(

)를 비교하고, 산출된 측면 거리가 안전 거리보다 작은 오브젝트(91)는 이동체의 추돌 대상으로 판단하고, 측면 거리가 안전 거리보다 큰 오브젝트(93)은 추돌 비대상으로 판단할 수 있다. Referring to (b) of FIG. 9 , the calculator 17 calculates the respective lateral distances (

,

) is calculated, and the calculated lateral distance (

,

) and the safety clearance (

), an object 91 having a calculated side distance smaller than the safety distance may be determined as a collision target of the moving object, and an object 93 having a side distance greater than the safety distance may be determined as non-collision target.

In addition, the calculation unit 17 calculates the safety margin based on at least one of the type of the moving object, the traveling speed, the driving location, and the user's physical condition (height, weight, shoulder width, arm spacing, etc.) when the moving object is personal mobility. can be variably determined and used.

For example, when the traveling speed of personal mobility is high, or when the vehicle is traveling in a place where the movement change or movement speed of an object is high, the calculator 17 may determine a wider safety margin distance.

)를 기초로 인식된 오브젝트와 이동체 간의 추돌까지 소요되는 시간인 추돌소요시간을 산출할 수 있다. In addition, when the collision target is determined, the calculation unit 17 calculates the calculated relative distance (

), it is possible to calculate the required collision time, which is the time required to collide between the recognized object and the moving object.

Specifically, the calculator 17 may calculate the required collision time between the moving object 200 and the recognized object using Equation 6 below.

In Equation 6, T is the collision time required, D is the relative distance between the moving object 200 and the recognized object, and V is the speed of the moving object.

That is, the calculator 17 may calculate the required time required to collide with the recognized object in consideration of the speed of the moving object.

Meanwhile, the calculator 17 may determine a relative distance calculation method according to a calculation condition, and calculate the relative distance using the determined relative distance calculation method. Here, the relative distance calculation method is a method using the above-described virtual horizontal line (Equation 2) and a method using the bounding box size (Equation 3). At least one of the size of the box), the degree of inclination change while driving, and the number of recognized objects may be included.

For example, the calculator 17 calculates the relative distance by using the bounding box size when the change in inclination during driving exceeds a predetermined criterion and the calculation condition is that there is one recognized object. can

That is, the calculator 17 determines the relative distance calculation method according to the calculation conditions, calculates the relative distance between the moving object and the object based on the object recognized by the determined calculation method, and based on the calculated relative distance, the object and the moving object It is possible to calculate the time required for a collision between the two.

The controller 18 may determine the possibility of colliding with an object and provide various control commands for preventing the moving object from colliding.

Specifically, the control unit 18 may determine the possibility of collision with the recognized object based on the calculated required collision time. In this case, the control unit 18 may determine the possibility of collision in consideration of the collision factor. Here, the collision factors include the brake performance of the moving object, the driving environment of the moving object (paved road, sidewalk, unpaved road driving, etc.), the weight of the user using the moving object, and the surrounding environment (the area with many pedestrians, the area requiring deceleration, etc.) There may be several factors that influence the

Also, when it is determined that there is a possibility of a collision, the control unit 18 may provide a collision warning guidance or control the moving object to be braked.

For example, if it is determined that there is a possibility of a collision with the object based on the calculated required collision time, the controller 18 may generate a control command for controlling the operation of the personal mobility and provide it as the personal mobility.

Next, with reference to FIG. 10, the collision prevention method of the moving object of the collision avoidance apparatus 10 is demonstrated.

10 is a flowchart illustrating a collision avoidance method according to an embodiment of the present invention.

Referring to FIG. 10 , a driving image of the moving object 200 photographed from its own camera or an external camera device may be acquired ( S10 ).

Also, the collision avoidance apparatus 10 may recognize an object in the driving image obtained using the neural network model (S20). Here, the neural network model may recognize an object in the driving image and classify and output a bounding box indicating an object region in the driving image and an object in the bounding box.

Specifically, the step of recognizing the object (S20) is to input the acquired driving image to the neural network model in frame units, and to obtain a driving image including a bounding box indicating an object region in the driving image and type information of an object in the bounding box. Thus, an object in the driving image can be recognized.

Then, it is possible to determine the state of the position of the object recognized in the driving image (S30).

Specifically, in the step of determining the position state ( S30 ), the position state of the object may be determined based on the positions of the bounding boxes of the objects recognized in the driving image.

And, it is possible to determine the tracking target based on the location state according to the type of object (S40).

Specifically, the tracking target determination step (S40) may determine whether the recognized object is misrecognized based on the positional state of the corresponding object according to the type of the recognized object or by determining the possibility of colliding with a moving object to determine the tracking target.

For example, the tracking target determination step S40 determines the possibility of collision with the moving object based on the positional state of the corresponding object according to the recognized object type, and excludes the moving object and the low probability of collision with the tracking target (S40: N), the object recognition step (S20) may be re-performed.

In addition, in the tracking target determination step (S40), an object with a high probability of colliding with a moving object may be determined as a tracking target (S40: Y).

In addition, the movement of the object may be tracked by tracking a change in the position of the object in the driving image determined as the tracking target ( S50 ).

Specifically, the collision avoidance apparatus 10 tracks a change in the position of the recognized object's bounding box by comparing the previous frame and the current frame of the driving image, and the movement path and movement direction of the object based on the change in the position of the object's bounding box. can be tracked.

Then, the driving image including the recognized object may be normalized (S60).

Then, a relative distance between the moving object and the object may be calculated based on the recognized object (S70).

Specifically, the step of calculating the relative distance (S70) is to generate a virtual horizontal line in the driving image based on the bounding boxes of the recognized objects, and calculate the generated virtual horizontal line and the vertical coordinates of the bottom surface of the bounding box of the recognized object. Based on this, the expected relative distance can be calculated.

In addition, when the virtual horizontal line cannot be generated, the step of calculating the relative distance ( S70 ) may calculate the expected relative distance based on the vertical height of the recognized object's bounding box and the actual height of the object.

In the step of calculating the relative distance ( S70 ), the relative distance may be calculated through prediction and update by applying the calculated expected relative distance to the Kalman filter.

Then, calculate the side distance, determine whether a collision target is based on the calculated side distance, and if there is a collision target, calculate the required collision time between the object and the moving body based on the calculated relative distance of the collision target object It can be (S80). Specifically, in calculating the required collision time ( S80 ), the required collision time may be calculated in consideration of the speed of the moving object.

Then, it is possible to control the operation of the moving object based on the calculated required collision time (S90). Specifically, in the controlling step ( S90 ), when there is a possibility of colliding with an object, a collision warning guide may be provided or a moving object may be controlled to be braked.

Meanwhile, the above-described collision avoidance apparatus 10 may be implemented using software, hardware, or a combination thereof. For example, according to the hardware implementation, ASICs (application specific integrated circuits), DSPs (digital signal processors), DSPDs (digital signal processing devices), PLDs (programmable logic devices), FPGAs (field programmable gate arrays), processors (processors) ), controllers, micro-controllers, micro-processors, and other electrical units for performing other functions may be implemented using at least one.

11 is an exemplary diagram illustrating a moving object management system 1000 according to an embodiment of the present invention.

Referring to FIG. 11 , the moving object management system may include an electronic device 100 , a moving object 200 , a monitoring server 300 , and a related organization 400 .

The electronic device 100 includes the function of the above-described collision avoidance device 10 as one component, and is a smart phone, tablet computer, notebook computer, and personal digital assistant (PDA) capable of providing driving-related guidance to a user of a moving object. ), a portable multimedia player (PMP), smart glasses, project glasses, navigation, black box, dash cam or video recorder, etc., may be implemented in various devices, and may be provided in the mobile body 200 .

The mobile device 200 may include both a non-shared mobile device used by one user and a shared mobile device used by a plurality of users. Hereinafter, the shared mobile device will be described. Here, the shared mobile may be rented, returned, and paid for rental through a user application installed in the user terminal.

The monitoring server 300 may continuously monitor the moving object 200 .

Specifically, the monitoring server 300 may monitor the rental, return, and rental fee payment of the moving object 200 through the user application, and may transmit user information to the electronic device 100 . Here, the user information is information about the user who has rented the moving object 200, and may include whether or not the rental fee of the moving object 200 is paid and the user's face information.

In addition, the monitoring server 300 receives event information from the electronic device 100 , and the manager 310 sends the electronic device 100 or the mobile device 300 to the corresponding mobile device 200 or electronic device 300 based on the event information. 200) may transmit an administrator command to control the operation or contact the relevant authority.

That is, the monitoring server 300 may respond to an event situation occurring to the user based on the event information.

The related organization 400 is an organization related to the event situation, and may be a police station, a medical institution, a rescue organization, etc. depending on the type of event.

Hereinafter, the configuration of the electronic device 100 of the moving object management system 1000 will be described.

12 is a block diagram illustrating an electronic device according to an embodiment of the present invention.

12 , the electronic device 100 includes a photographing unit 110 , a collision prediction unit 120 , an output unit 130 , an input unit 140 , a sensing unit 150 , a communication unit 160 , and an event sensing unit. 170 , a storage unit 180 , and a control unit 190 may be included.

The photographing unit 110 may perform a function of photographing the periphery of the moving object.

Specifically, the photographing unit 110 may obtain a driving image or a parking image of the moving object by photographing the front. Also, the photographing unit 110 may obtain an image of the user by photographing the user of the moving object.

The collision prediction unit 120 may perform the functions of the above-described collision avoidance apparatus 10 .

Specifically, the collision prediction unit 120 includes the image acquisition unit 11, the object recognition unit 12, the position state determination unit 13, the tracking determination unit 14, and the object update unit of the above-described collision avoidance device 10. Each function of the unit 15 , the normalization processing unit 16 , and the calculation unit 17 may be performed.

That is, the collision prediction unit 120 acquires the driving image captured by the photographing unit 110 to recognize the object, calculates the relative distance to the recognized object, and calculates the required collision time based on the calculated relative distance can do.

The output unit 130 is a device for outputting various data of the electronic device 100 to the user as images and/or audio.

For example, the output unit 130 may output information for guiding the driving of the moving object 200 based on the required collision time. Here, the output unit 130 may include all or part of the display unit 131 and the audio output unit 132 .

The display unit 131 is a device that outputs data that can be visually recognized by a user.

Specifically, the display unit 131 may output driving guidance information, driving speed, driving image, boarding time, payment time, remaining rental time, and the like.

For example, when it is determined that there is a possibility of a collision, guide information warning of a collision may be output on the display unit 131 .

The audio output unit 132 is a device for outputting data that can be recognized aurally. Here, the audio output unit 132 may be implemented as a speaker expressing sound of data to be notified to the user of the electronic device 100 .

Specifically, the audio output unit 132 may output driving guidance information, driving speed, remaining rental time of driving image, and the like.

The input unit 140 functions to convert a physical input from the outside of the electronic device 100 into a specific electrical signal. Here, the input unit 140 may include all or part of the user input unit 141 and the microphone unit 142 .

The user input unit 141 may receive a user input such as a touch or a push operation. Here, the user input unit 141 may be implemented using at least one of various button shapes, a touch sensor for receiving a touch input, and a proximity sensor for receiving an approaching motion.

The microphone unit 142 may receive a user's voice and a sound generated inside or outside the vehicle.

The sensing unit 150 may sense and acquire various data necessary for the operation of the electronic device 100 .

Specifically, the sensing unit 150 may acquire impact data by sensing an impact of the electronic device 100 or a moving object. Also, the sensing unit 150 may acquire current location data of the electronic device 100 or a moving object.

The communication unit 150 may transmit/receive various data by communicating with the mobile body 200 , the monitoring server 300 , an external device, and a related organization 400 . Here, the external device may mean another electronic device or another movable body 200 .

Specifically, the communication unit 150 may transmit a control command for controlling the operation of the moving object to the moving object.

In addition, the communication unit 150 may transmit or receive user information or event information from the monitoring server 300 . Also, the communication unit 150 may receive a manager command from the monitoring server 300 .

The event detection unit 170 may detect whether various events of the moving object have occurred.

Specifically, the event detecting unit 170 may detect whether a boarding event has occurred based on the user image. Here, the boarding event is information indicating whether the user of the shared mobile device wears safety equipment when boarding the shared mobile device, whether or not to make a prepayment, and whether the user himself or herself.

For example, the event detection unit 170 image-processes the user image to recognize whether the user wears safety equipment (helmet, knee protector, elbow protector, etc.), and when the user does not wear safety equipment, boarding It can be detected that an event has occurred.

As another example, the event detection unit 170 image-processes the user image to recognize the face of the riding user, compares it with the face information of the registered user of the shared moving object, and says that the face information of the riding user is different from the face information of the user. If it is determined, it may be detected that a boarding event has occurred.

Also, the event detection unit 170 may detect whether an accident event occurs based on at least one of a driving image, a user input, and impact data. Here, the accident event relates to whether or not an emergency situation occurs to the user of the shared mobile object, and is event information indicating whether the user of the shared mobile object has a collision accident while driving, a slip accident, and the like. Here, the user input may include a user's emergency rescue request command.

For example, in the case of colliding with an object recognized in the driving image, the event detecting unit 170 may detect that an accident event has occurred in the user of the shared moving object.

Also, the event detecting unit 170 may detect whether an attention event occurs based on at least one of a driving image and current location data. Here, the caution event relates to whether or not a situation requiring attention to the user of the shared moving object occurs, and is event information indicating whether the user is driving at a speed or abnormal driving (illegal overtaking, illegal interruption, etc.).

For example, when it is determined that the vehicle travels adjacent to the object recognized in the driving image, the event detection unit 170 may detect that an attention event has occurred in the user of the shared moving object.

As another example, if it is determined based on the current location data that the speed of the current shared moving object is faster than the prescribed speed of the corresponding area, the event detecting unit 170 may detect that a caution event has occurred to the user of the shared mobile.

Also, the event detecting unit 170 may detect whether an alighting event occurs based on at least one of a user image and current location data. Here, the getting off event is event information indicating whether the user of the shared mobile device is dismounting and storing safety equipment when the user of the shared mobile device gets off the shared mobile device, and whether the shared mobile device is illegally parked.

For example, when the shared mobile is parked in an illegal parking area based on the current location data, the event detection unit 170 may determine that an alighting event has occurred.

That is, the event detection unit 170 may detect whether at least one of a boarding event, an accident event, a caution event, and an alighting event occurs to the user of the shared moving object based on the driving image, the user image, the impact data, and the current location data. there is.

The storage unit 160 may perform a function of storing various data necessary for the operation of the electronic device 100 .

Also, the storage 160 may store data generated by the operation of the electronic device 100 , for example, a driving image, a recognized object, event information, a driving route, and the like.

The storage unit 160 includes random access memory (RAM), flash memory, read only memory (ROM), erasable programmable ROM (EPROM), electrically erasable and programmable ROM (EEPROM), registers, hard disk, removable disk, and memory. It may be implemented as a storage device in a built-in type such as a card or a Universal Subscriber Identity Module (USIM), as well as a storage device in a removable type such as a USB memory.

The controller 170 may control the overall operation of the electronic device 100 .

Specifically, when the event detection unit 170 detects the occurrence of an event, the control unit 170 generates event information and transmits the generated event information to the monitoring server 300 or a related organization 400 related to the conversion event. It is possible to control the communication unit 150 as possible.

Here, the event information includes generated event type information (boarding event, accident event, attention event, alighting event, etc.), event video information (eg, driving video before and after the event, user video, etc.), event occurrence location and time Information, a movement path of an object in the driving image, a movement speed and a movement direction, etc. may be included.

In addition, the control unit 170 determines the urgency for each generated event information, and controls the communication unit 150 to selectively transmit the event information generated according to the urgency to the monitoring server 300 or a related organization 400 . can In this case, the controller 170 may determine the degree of urgency according to the degree of damage to the user according to the occurrence of the event, and may determine the degree of urgency based on at least one of event type information, impact data, and the speed of the moving object 200 . there is.

Also, the controller 170 may control the operation of the electronic device 100 or the moving object 200 based on a manager command. Here, the manager command may include a control command for controlling the emergency light of the moving object to operate, for controlling the rescue siren to operate, or for controlling the moving object to be braked.

Through this, it is possible to prevent an accident occurring in the moving object 200 from escalating and to minimize damage to the user of the moving object 200 .

Meanwhile, the control unit 170 may perform all functions of the control unit 18 of the above-described collision avoidance device 10 .

Hereinafter, a moving object monitoring method of the moving object management system 1000 will be described with reference to FIG. 13 .

13 is a timing diagram illustrating a method for monitoring a moving object of a moving object management system according to an embodiment of the present invention.

Referring to FIG. 13 , the monitoring server 300 may transmit user information to the electronic device 100 ( S100 ). Here, user information may be input through a user application.

Next, the electronic device 100 may acquire various data necessary to detect the occurrence of an event (S200). Specifically, the electronic device 100 may acquire a front image, a surrounding image, a user image, impact data, and current location data of the moving object 200 .

Next, the electronic device 100 may detect whether an event has occurred to the user of the moving object 200 based on the acquired data (S300).

Specifically, the electronic device 100 may detect whether a boarding event, an accident event, a caution event, or an alighting event occurs to the user of the shared moving object based on the driving image, the user image, the impact data, and the current location data.

And, when an event occurs, the electronic device 100 generates event information ( S400 ), and transmits the generated event information to the monitoring server 300 , or to a related organization 400 according to the urgency of the generated event. It can be transmitted (S500).

Next, the monitoring server 300 may monitor the corresponding moving object 200 or the user of the moving object in real time based on the received event information (S600).

In addition, the monitoring server 300 may transmit event information and a dispatch request command to the related organization 400 according to the determination of the manager 310 (S700). In this case, the related organization 400 may be dispatched to the place where the corresponding event occurred based on the event information.

Also, the monitoring server 300 may transmit a manager command corresponding to the generated event to the electronic device 100 (S800). For example, the monitoring server 300 may transmit a manager command for controlling the moving object to be braked to the electronic device 100 .

Then, the electronic device 100 may control the operation of the electronic device 100 or the moving object 200 based on the received manager command (S900).

Through the above-described moving object management system 1000 , the user of the moving object can be monitored in real time, and when an event occurs to the user of the moving object, it can be quickly dealt with.

In addition, the monitoring server 300 may identify the cause of the loss, damage, accident, etc. of the moving object through the event information.

Furthermore, through the mobile body management system 1000, the government policy can be revitalized by inducing insurance discount through consultation with the insurance company of the mobile body, and it is possible to lead the market through the greening of the moving object and mandatory collision prevention.

Meanwhile, at least one of the modules constituting the device for preventing collision of a moving object 10 and the electronic device 100 according to the present invention may be implemented as one module of a system for autonomous driving to perform a route guidance function. This will be described in more detail with reference to FIGS. 14 to 15 .

Referring to FIG. 14 , the autonomous driving vehicle 2000 according to the present embodiment may include a control device 2100 , sensing modules 2004a , 2004b , 2004c , 2004d , an engine 2006 , and a user interface 2008 . can

The autonomous driving vehicle 2000 may have an autonomous driving mode or a manual mode. For example, according to a user input received through the user interface 2008 , the manual mode may be switched to the autonomous driving mode, or the autonomous driving mode may be switched to the manual mode.

When the vehicle 2000 is operated in the autonomous driving mode, the autonomous driving vehicle 2000 may be operated under the control of the control device 2100 .

In this embodiment, the control device 2100 may include a controller 2120 including a memory 2122 and a processor 2124 , a sensor 2110 , a communication device 2130 , and an object detection device 2140 .

Here, the object detection apparatus 2140 may perform all or some functions of the above-described collision avoidance apparatus 10 .

That is, in the present embodiment, the object detection apparatus 2140 is a device for detecting an object located outside the vehicle 2000, and the object detection apparatus 2140 detects an object located outside the vehicle 2000, Object information may be generated according to the detection result.

The object information may include information on the existence of an object, location information of the object, distance information between the vehicle and the object, and relative speed information between the vehicle and the object.

The object may include various objects positioned outside the vehicle 2000 , such as a lane, another vehicle, pedestrian, traffic signal, light, road, structure, speed bump, terrain, and animal. Here, the traffic signal may be a concept including a traffic light, a traffic sign, a pattern or text drawn on a road surface. In addition, the light may be light generated from a lamp provided in another vehicle, light generated from a street lamp, or sunlight.

In addition, the structure may be an object located around the road and fixed to the ground. For example, the structure may include a street lamp, a street tree, a building, a power pole, a traffic light, and a bridge. Features may include mountains, hills, and the like.

The object detection apparatus 2140 may include a camera module. The controller 2120 may extract object information from an external image captured by the camera module and allow the controller 2120 to process the information.

Also, the object detection apparatus 2140 may further include imaging apparatuses for recognizing an external environment. In addition to LIDAR, RADAR, GPS, Odometry and other computer vision devices, ultrasonic sensors, and infrared sensors can be used, and these devices can be selected or operated simultaneously as needed to enable more precise detection .

On the other hand, the collision avoidance apparatus 10 according to an embodiment of the present invention may determine the position state of the recognized object, and determine a tracking target based on the position state according to the type of object. Then, the collision avoidance apparatus 10 tracks the movement of the object by tracking the change in the position of the object in the driving image determined as the tracking target, normalizes the driving image including the recognized object, and A relative distance between the moving object and the object may be calculated based on the object.

In addition, the collision avoidance apparatus 10 may control the operation of the moving object based on the calculated required collision time in connection with the control apparatus 2100 of the autonomous driving vehicle 2000 .

As an example, if there is a possibility of a collision between the autonomous vehicle 2000 and an object, the autonomous vehicle 2000 may control a brake to slow down or stop the vehicle. As another example, when the object is a moving object, the autonomous driving vehicle 2000 may control the driving speed of the autonomous driving vehicle 2000 to move at the same speed as the object.

The collision avoidance apparatus 10 according to an embodiment of the present invention may be configured as one module in the control apparatus 2100 of the autonomous vehicle 2000 . That is, the memory 2122 and the processor 2124 of the control device 2100 may implement the collision avoidance method according to the present invention in software.

Also, the sensor 2110 may be connected to the sensing modules 2004a, 2004b, 2004c, and 2004d for the internal/external environment of the vehicle to acquire various types of sensing information. Here, the sensor 2110 is an attitude sensor (eg, a yaw sensor, a roll sensor, a pitch sensor, a collision sensor, a wheel sensor, a speed sensor, and a tilt sensor). , weight sensor, heading sensor, gyro sensor, position module, vehicle forward/reverse sensor, battery sensor, fuel sensor, tire sensor, steering sensor by steering wheel rotation, vehicle interior It may include a temperature sensor, an in-vehicle humidity sensor, an ultrasonic sensor, an illuminance sensor, an accelerator pedal position sensor, a brake pedal position sensor, and the like.

Accordingly, the sensor 2110 includes vehicle posture information, vehicle collision information, vehicle direction information, vehicle location information (GPS information), vehicle angle information, vehicle speed information, vehicle acceleration information, vehicle inclination information, vehicle forward/reverse information, Sensing signals for battery information, fuel information, tire information, vehicle lamp information, vehicle interior temperature information, vehicle interior humidity information, steering wheel rotation angle, exterior illumination, accelerator pedal pressure, and brake pedal pressure can be obtained

In addition, the sensor 2110 includes, in addition, an accelerator pedal sensor, a pressure sensor, an engine speed sensor, an air flow sensor (AFS), an intake air temperature sensor (ATS), a water temperature sensor (WTS), and a throttle position. It may further include a sensor (TPS), a TDC sensor, a crank angle sensor (CAS), and the like.

In this way, the sensor 2110 may generate vehicle state information based on the sensed data.

The wireless communication device 2130 is configured to implement wireless communication between the autonomous vehicles 2000 . For example, it enables the autonomous vehicle 2000 to communicate with a user's mobile phone, another wireless communication device 2130 , another vehicle, a central device (traffic control device), a server, and the like. The wireless communication device 2130 may transmit/receive a wireless signal according to an access wireless protocol. The wireless communication protocol can be Wi-Fi, Bluetooth, Long-Term Evolution (LTE), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Global Systems for Mobile Communications (GSM), and the communication protocol is It is not limited thereto.

Also, in the present embodiment, the autonomous vehicle 2000 may implement inter-vehicle communication through the wireless communication device 2130 . That is, the wireless communication device 2130 may communicate with other vehicles and other vehicles on the road through vehicle-to-vehicle communication (V2V). The autonomous vehicle 2000 may transmit/receive information such as a driving warning and traffic information through vehicle-to-vehicle communication, and may request or receive a request for information from another vehicle. For example, the wireless communication device 2130 may perform V2V communication as a dedicated short-range communication (DSRC) device or a cellular-V2V (C-V2V) device. In addition to communication between vehicles, communication (V2X, Vehicle to Everything communication) between a vehicle and another object (eg, an electronic device carried by a pedestrian) may also be implemented through the wireless communication device 2130 .

In this embodiment, the controller 2120 is a unit that controls the overall operation of each unit in the vehicle 2000 , and may be configured by the manufacturer of the vehicle during manufacturing or may be additionally configured to perform the function of autonomous driving after manufacturing. Alternatively, a configuration for continuously performing additional functions through an upgrade of the controller 2120 configured during manufacturing may be included. Such a controller 2120 may be referred to as an Electronic Control Unit (ECU).

The controller 2120 collects various data from the connected sensor 2110, the object detection device 2140, the communication device 2130, etc., and sends a control signal based on the collected data to the sensor ( 2110 ), the engine 2006 , the user interface 2008 , the communication device 2130 , and the object detection device 2140 . Also, although not shown, the control signal may be transmitted to an acceleration device, a braking system, a steering device, or a navigation device related to driving of a vehicle.

In the present embodiment, the controller 2120 may control the engine 2006, for example, to detect the speed limit of the road on which the autonomous vehicle 2000 is traveling and to prevent the driving speed from exceeding the speed limit. ) or the engine 2006 to accelerate the driving speed of the autonomous vehicle 2000 within a range that does not exceed the speed limit.

In addition, if the autonomous driving vehicle 2000 approaches or departs from the lane while the autonomous driving vehicle 2000 is driving, the controller 2120 determines whether such lane proximity and departure is according to a normal driving situation or other driving conditions. The engine 2006 may be determined to determine whether or not the vehicle is driven according to the determination result, and the engine 2006 may be controlled to control the driving of the vehicle. Specifically, the autonomous vehicle 2000 may detect lanes formed on both sides of a lane in which the vehicle is traveling. In this case, the controller 2120 determines whether the autonomous driving vehicle 2000 approaches or departs from the lane, and if it is determined that the autonomous driving vehicle 2000 approaches or departs from the lane, such It may be determined whether the driving is according to an accurate driving situation or other driving situations. Here, as an example of a normal driving situation, it may be a situation in which a vehicle lane change is required. And, as an example of other driving conditions, it may be a situation in which a lane change of the vehicle is not required. If the controller 2120 determines that the autonomous driving vehicle 2000 is approaching or departing from the lane in a situation in which a vehicle lane change is not required, the autonomous driving vehicle 2000 does not deviate from the lane and the corresponding vehicle may control the driving of the autonomous driving vehicle 2000 to drive normally.

When there is another vehicle or an obstacle in front of the vehicle, the engine 2006 or the braking system may be controlled to decelerate the driving vehicle, and in addition to the speed, the trajectory, the driving path, and the steering angle may be controlled. Alternatively, the controller 2120 may control the driving of the vehicle by generating a necessary control signal according to recognition information of other external environments, such as a driving lane of the vehicle, a driving signal, and the like.

The controller 2120 may control the driving of the vehicle by performing communication with a surrounding vehicle or a central server in addition to generating its own control signal and transmitting a command for controlling the surrounding devices through the received information.

In addition, when the position of the camera module 2150 is changed or the angle of view is changed, the controller 2120 may have difficulty in accurate vehicle or lane recognition according to the present embodiment. In order to prevent this, calibration of the camera module 2150 ( Calibration) may be generated to control a control signal to be performed. Accordingly, in the present embodiment, the controller 2120 generates a calibration control signal to the camera module 2150 so that the mounting position of the camera module 2150 is determined by vibration or shock generated according to the movement of the autonomous vehicle 2000 . Even if it is changed, the normal mounting position, direction, angle of view, etc. of the camera module 2150 may be continuously maintained. In the controller 2120, the initial mounting position, direction, and angle of view information of the camera module 2120 stored in advance and the initial mounting position, direction, and angle of view information of the camera module 2120 measured while the autonomous vehicle 2000 is driving are critical. When the value is different than the value, a control signal may be generated to perform calibration of the camera module 2120 .

In this embodiment, the controller 2120 may include a memory 2122 and a processor 2124 . The processor 2124 may execute software stored in the memory 2122 according to a control signal of the controller 2120 . Specifically, the controller 2120 stores data and instructions for performing the lane detection method according to the present invention in the memory 2122, and the instructions may be executed by the processor 2124 to implement one or more methods disclosed herein. there is.

In this case, the memory 2122 may be stored in a recording medium executable by the non-volatile processor 2124 . Memory 2122 may store software and data via suitable internal and external devices. The memory 2122 may include random access memory (RAM), read only memory (ROM), a hard disk, and a memory 2122 device connected to a dongle.

The memory 2122 may store at least an operating system (OS), a user application, and executable commands. The memory 2122 may also store application data and array data structures.

Processor 2124 may be a microprocessor or suitable electronic processor, a controller, microcontroller, or state machine.

The processor 2124 may be implemented as a combination of computing devices, and the computing device may be a digital signal processor, a microprocessor, or a suitable combination thereof.

Meanwhile, the autonomous vehicle 2000 may further include a user interface 2008 for a user's input to the aforementioned control device 2100 . User interface 2008 may allow a user to enter information with appropriate interaction. For example, it may be implemented as a touch screen, a keypad, an operation button, and the like. The user interface 2008 may transmit an input or command to the controller 2120 , and the controller 2120 may perform a vehicle control operation in response to the input or command.

Also, the user interface 2008 may be an external device of the autonomous vehicle 2000 and may communicate with the autonomous vehicle 2000 through the wireless communication device 2130 . For example, the user interface 2008 may be capable of interworking with a mobile phone, tablet, or other computer device.

Furthermore, although it has been described that the autonomous vehicle 2000 includes the engine 2006 in this embodiment, it is also possible to include other types of propulsion systems. For example, the vehicle may be driven by electric energy, and may be driven by hydrogen energy or a hybrid system combining them. Accordingly, the controller 2120 may include a propulsion mechanism according to the propulsion system of the autonomous vehicle 2000 , and may provide a corresponding control signal to the components of each propulsion mechanism.

Hereinafter, a detailed configuration of the control device 2100 according to the present invention according to the present embodiment will be described in more detail with reference to FIG. 15 .

The control device 2100 includes a processor 2124 . The processor 2124 may be a general purpose single or multi-chip microprocessor, a dedicated microprocessor, microcontroller, programmable gate array, or the like. A processor may also be referred to as a central processing unit (CPU). Also, in this embodiment, the processor 2124 may be used as a combination of a plurality of processors.

The control device 2100 also includes a memory 2122 . Memory 2122 may be any electronic component capable of storing electronic information. Memory 2122 may also include a combination of memories 2122 in addition to a single memory.

Data and instructions 2122a for performing the lane detection method according to the present invention may be stored in the memory 2122 . When the processor 2124 executes the instructions 2122a , all or part of the instructions 2122a and data 2122b necessary for execution of the instructions may be loaded 2124a and 2124b onto the processor 2124 .

The control device 2100 may include a transmitter 2130a, a receiver 2130b, or a transceiver 2130c for allowing transmission and reception of signals. One or more antennas 2132a, 2132b may be electrically coupled to transmitter 2130a, receiver 2130b, or each transceiver 2130c, and may additionally include antennas.

The control device 2100 may include a digital signal processor (DSP) 2170 . The DSP 2170 may enable the vehicle to quickly process a digital signal.

The control device 2100 may include a communication interface 2180 . The communication interface 2180 may include one or more ports and/or communication modules for connecting other devices with the control device 2100 . The communication interface 2180 may enable a user and the control device 2100 to interact.

The various components of the control device 2100 may be coupled together by one or more buses 2190 , which may include a power bus, a control signal bus, a status signal bus, a data bus, and the like. According to the control of the processor 2124 , the components may transfer information to each other through the bus 2190 and perform a desired function.

Meanwhile, in the specification and claims, terms such as "first", "second", "third", and "fourth" are used to distinguish between similar elements, if any, and this is not necessarily the case. Used to describe a specific sequence or sequence of occurrences. It will be understood that the terms so used are interchangeable under appropriate circumstances to enable the embodiments of the invention described herein to operate, for example, in sequences other than those shown or described herein. Likewise, where methods are described herein as including a series of steps, the order of those steps presented herein is not necessarily the order in which those steps may be performed, and any described steps may be omitted and/or Any other steps not described may be added to the method. For example, without departing from the scope of the present invention, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component.

Also, terms such as "left", "right", "front", "behind", "top", "bottom", "above", "below" in the specification and claims are used for descriptive purposes, It is not necessarily intended to describe an invariant relative position. It will be understood that the terms so used are interchangeable under appropriate circumstances to enable the embodiments of the invention described herein to operate otherwise than, for example, as shown or described herein. As used herein, the term “connected” is defined as being directly or indirectly connected in an electrical or non-electrical manner. Objects described herein as being "adjacent" to one another may be in physical contact with one another, in proximity to one another, or in the same general scope or area as appropriate for the context in which the phrase is used. The presence of the phrase “in one embodiment” herein refers to the same, but not necessarily, embodiment.

In addition, in the specification and claims, references to 'connected', 'connecting', 'fastened', 'fastening', 'coupled', 'coupled', etc., and various variations of these expressions, refer to other elements directly It is used in the sense of being connected or indirectly connected through other components.

On the other hand, when it is mentioned that a certain element is "directly connected" or "directly connected" to another element, it should be understood that no other element is present in the middle.

In addition, the suffixes "module" and "part" for the components used in this specification are given or mixed in consideration of the ease of writing the specification, and do not have distinct meanings or roles by themselves.

In addition, the terms used herein are for the purpose of describing the embodiments and are not intended to limit the present invention. As used herein, the singular expression includes the plural expression unless the context clearly dictates otherwise. In the present application, terms such as "consisting of" or "comprising" should not be construed as necessarily including all of the various components or various steps described in the specification, some of which components or some steps are It should be construed that it may not include, or may further include additional components or steps.

So far, the present invention has been looked at with respect to preferred embodiments thereof. All embodiments and conditional examples disclosed through this specification have been described with the intention of helping those of ordinary skill in the art to understand the principles and concepts of the present invention, and those skilled in the art It will be understood that the present invention may be implemented in a modified form without departing from the essential characteristics of the present invention.

Therefore, the disclosed embodiments are to be considered in an illustrative rather than a restrictive sense. The scope of the present invention is indicated in the claims rather than the foregoing description, and all differences within the scope equivalent thereto should be construed as being included in the present invention.

Meanwhile, the method according to various embodiments of the present invention described above may be implemented as a program and provided to a server or devices. Accordingly, each device can download the program by accessing the server or device in which the program is stored.

In addition, the above-described method according to various embodiments of the present invention may be implemented as a program and stored in various non-transitory computer readable media to be provided. The non-transitory readable medium refers to a medium that stores data semi-permanently, rather than a medium that stores data for a short moment, such as a register, cache, memory, and the like, and can be read by a device. Specifically, the above-described various applications or programs may be provided by being stored in a non-transitory readable medium such as a CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM, and the like.

In addition, although preferred embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above, and the technical field to which the present invention pertains without departing from the gist of the present invention as claimed in the claims In addition, various modifications are possible by those of ordinary skill in the art, and these modifications should not be individually understood from the technical spirit or perspective of the present invention.

Claims (22)

In the collision prevention method of the moving object collision avoidance device,
acquiring a driving image of the moving object;
recognizing an object in the obtained driving image using a neural network model;
calculating a relative distance between the moving object and the object based on the recognized object;
calculating a required collision time between the object and the moving body based on the calculated relative distance; and
and controlling an operation of the moving object based on the calculated required collision time. The method of claim 1,
The neural network model recognizes an object in the driving image and classifies and outputs a bounding box indicating an object region in the driving image and an object in the bounding box. 3. The method of claim 2,
The method of claim 1, further comprising: tracking the movement of the object by tracking a change in the position of the object recognized in the driving image. 4. The method of claim 3,
Collision avoidance method, characterized in that it further comprises the step of normalizing the driving image in which the object is recognized. 3. The method of claim 2,
determining a position state of an object recognized in the driving image; and
Determining an object to be tracked from among the objects recognized in the driving image based on the positional state according to the type of the object; 3. The method of claim 2,
The calculating of the relative distance comprises generating a virtual horizontal line in the driving image based on the bounding box. 7. The method of claim 6,
The calculating of the relative distance comprises calculating the expected relative distance based on the generated virtual horizontal line and the vertical coordinates of the bottom surface of the bounding box. 8. The method of claim 7,
The calculating of the relative distance comprises calculating the expected relative distance based on the vertical height of the bounding box and the actual height of the object. 9. The method of claim 8,
The calculating of the relative distance comprises calculating the relative distance through prediction and update by applying the calculated expected relative distance to a Kalman filter. 3. The method of claim 2,
In the calculating of the required time for collision, the method for preventing collision, characterized in that calculating the required time for the collision in consideration of the speed of the moving object. The method of claim 1,
In the controlling of the operation, when there is a possibility of colliding with the object, the collision avoidance method according to claim 1 , wherein a collision warning is provided or the moving object is controlled to be braked. In the moving object collision avoidance device,
an image acquisition unit for acquiring a driving image of the moving object;
an object recognition unit for recognizing an object in the obtained driving image using a neural network model;
a calculation unit calculating a relative distance between the moving object and the object based on the recognized object, and calculating a required collision time between the object and the moving object based on the calculated relative distance; and
Collision avoidance device comprising a; a control unit for controlling the operation of the moving object based on the calculated required collision time. 13. The method of claim 12,
The neural network model recognizes an object in the driving image and classifies and outputs a bounding box indicating an object region in the driving image and an object in the bounding box. 14. The method of claim 13,
The collision avoidance device further comprising: an object tracking unit for tracking the movement of the object by tracking a change in the position of the object recognized in the driving image. 14. The method of claim 13,
a position state determination unit for determining a position state of an object recognized in the driving image; and
Collision avoidance device further comprising a; a tracking determining unit that excludes the moving object and the object having a low collision probability based on the positional state according to the type of the object from the tracking target. 14. The method of claim 13,
The calculator is configured to generate a virtual horizontal line in the driving image based on the bounding box. 17. The method of claim 16,
The calculator is configured to calculate the expected relative distance based on the generated virtual horizontal line and the vertical coordinates of the bottom surface of the bounding box. 18. The method of claim 17,
Wherein the calculator calculates the expected relative distance based on the vertical height of the bounding box and the actual height of the object Collision avoidance device, characterized in that. 19. The method of claim 18,
The calculating of the relative distance comprises calculating the relative distance through prediction and update by applying the calculated expected relative distance to a Kalman filter. 13. The method of claim 12,
The collision avoidance device, characterized in that the calculation unit calculates the required time required for the collision in consideration of the speed of the moving object. A mobile electronic device comprising:
a photographing unit for photographing a moving image of a moving object;
an image acquisition unit for acquiring the driving image photographed by the photographing unit;
a recognition unit for recognizing an object in the obtained driving image using a neural network model;
a calculation unit calculating a relative distance between the moving object and the object based on the recognized object, and calculating a required collision time between the object and the moving object based on the calculated relative distance; and
and an output unit for outputting information for guiding the driving of the moving object based on the calculated required collision time. A program stored in a computer-readable recording medium comprising a program code for executing the method for preventing a collision according to any one of claims 1 to 11.

Images