KR101947280B1 - System and method for Pedestrian Protection - Google Patents

Abstract

The present invention relates to a system and a method for pedestrian protection. According to embodiments of the present invention, the method for pedestrian protection comprises: a candidate generation step of extracting pedestrian candidate objects from an input image; a step of calculating a region of prediction path (ROPP) which is a region where a future path of a vehicle may exist by statistical analysis of driving data; a step of expanding an outer region of a left edge and a right edge of the ROPP in accordance with a preset standard to calculate a region of danger (ROD); a step of selecting and outputting only pedestrian candidates positioned in the ROD among the pedestrian candidate objects extracted by the candidate generation step; and a candidate verification step of performing candidate verification on the filtered pedestrian candidate objects to detect a pedestrian in a danger state. The step of calculating the ROPP uses the speed and angular speed of the vehicle to set a direction error range, and uses the direction error range and maximum acceleration to calculate the ROPP.

Description

{System and method for pedestrian protection}

The present invention relates to a pedestrian protection system and method, and more particularly, to a pedestrian protection system and method for minimizing a time required for detecting a pedestrian, and in particular, by setting a direction error range using a speed and an angular velocity, .

The pedestrian protection system predicts pedestrian pedestrian accidents and prevents them by emergency braking.

Conventional pedestrian protection system generally performs three stages of Hypothesis Generation (HG), Hypothesis Verification (HV) and Risk Assessment (RA) in sequence.

HG is implemented as a relatively lightweight algorithm, and candidates are selected for areas where pedestrians are expected to exist in the entire image. The HV is implemented with a relatively heavy but accurate algorithm and evaluates whether the generated candidate is a pedestrian. The RA assesses the risk of an accident by evaluating whether the detected pedestrian is on the route of the vehicle.

Such a pedestrian protection system uses a non-irreversible means such as emergency braking as compared with a system that provides only an alarm, so that the cost due to malfunction is very large. Malfunctions are caused by false detection of pedestrians as false pedestrians and path prediction errors that are considered to be dangerous to safe pedestrians by mistakenly predicting the route of the vehicle. Therefore, in order to secure practicality and consumer acceptability, it is necessary to minimize malfunction.

A typical example of a method for minimizing the detection of pedestrian misfire is a method of performing HV through CNN (Convolutional Neural Network). CNN has been reported to outperform existing systems in various recognition fields and has a very low false detection rate in pedestrian recognition. However, CNN is considerably heavier than existing systems and has a lot of difficulties in real-time implementation. Therefore, there have been proposed methods of using high-speed hardware such as GPU (Graphic Processing Unit) or FPGA and reducing the number of pedestrian candidates by using relatively light detector such as ACF (Aggregated Channel Feature) as HG, It has a disadvantage that it is difficult to secure.

Therefore, it is expected that the real - time improvement of HV implemented through CNN will be an important means of minimizing the detection of pedestrian error for the time being.

In addition, in the case of Korean Patent No. 10-1770924 of the applicant, a system and a method for improving the real-time property in order to solve the above problems have been proposed. However, the present invention is not limited to the ROPP (Region of Prediction Path) And the maximum value of the acceleration and direction error is analyzed only according to the current speed and there is a problem that the ROPP is set to be wide. Since the maximum value of the acceleration and the direction error is derived as a function, .

Korean Patent No. 10-1770924

David Gerbenimo, Antonio M. L. Pez, Angel D. Sappa, and Thorsten Graf, "Survey of Pedestrian Detection for Advanced Driver Assistance Systems," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 32, no. 7, Jul. 2010, pp. 1239-1258.

In order to solve the above problems, the present invention extracts only a dangerous pedestrian candidate through a risk assessment (RA) and performs a candidate verification (HV), thereby reducing the burden of relatively heavy HV processing So as to improve the real-time performance of the HV.

It is another object of the present invention to provide a pedestrian protection system and method for reducing the time required for the pedestrian verification by narrowing the area ROPP in which a future route can exist in consideration of the speed and the angular speed.

Another object of the present invention is to provide a system and method for protecting a pedestrian, which can easily learn and utilize a region ROPP in which a future route may exist, taking a speed and an angular speed into consideration by using a lookup table.

According to another aspect of the present invention, there is provided a pedestrian protection method comprising: a candidate generation step of extracting a pedestrian candidate object from an input image; Calculating a Region of Prediction Path (ROPP), which is an area where a future route of the vehicle may exist, through statistical analysis of the driving data; Expanding the left and right outer regions of the ROPP according to a preset reference to calculate the ROD; And selecting and outputting only the pedestrian candidates located in the ROD among the pedestrian candidate objects extracted through the candidate generating step; And a candidate verification step of performing a candidate verification on each of the filtered candidate pedestrian objects to detect a pedestrian in a dangerous state. The step of calculating the ROPP may include setting a direction error range using the speed and angular velocity of the vehicle And the ROPP is calculated using the direction error range and the maximum acceleration.

The step of calculating the ROPP may include calculating a ROPP using a preset lookup table using the maximum acceleration and the direction error range.

" 및 "

"를 근거로 예측시간 T 동안의 예상 경로 끝점(x(T), y(T))을 계산하는 단계; "Φ(T)=tan^-1(y(T)/x(T))"의 식에 따라 상기 예상 경로 끝점의 위상각(Φ(T))을 계산하는 단계; CTRV 모션 모델을 근거로 EKF(Extended Kalman Filtering)를 적용하는 CTRV_EKF에 근거해 보정된 차속(v') 및 각속도(ω')가 속하는 셀(cell)을 선택하는 단계; 상기 선택된 셀로부터 방향오류 평균(μ _φ )과 표준편차(σ _φ )를 추출하여 “Δφ _max = μ _φ + βσ _φ , Δφ _min = μ _φ - βσ _φ ”의 식에 따라 최대 방향오류(Δφ _max ) 및 최소 방향오류(Δφ _min )를 계산하는 단계; 상기 최대 방향오류(Δφ _max ) 및 최소 방향오류(Δφ _min )를 기반하여 “Φ(T) + Δφ _min ~ Φ(T) + Δφ _max ”를 ROPP의 위상각 범위로 정의하는 단계; 상기 선택된 셀로부터 가속도의 평균(μ _a )과 표준편차(σ _a )를 추출하여 “a _max = μ _a + βσ _a ”의 식에 따라 상기 최대가속도(a _max )를 계산하는 단계; "

"에 근거해 최대 도달거리(s_max(v))를 계산하는 단계; 및 반지름이 s_max(v)이고 위상각이 Φ(T) + Δφ _min ~ Φ(T) + Δφ _max 범위인 부채꼴로 정의되는 ROPP를 산출하는 단계를 포함하며, 상기 셀을 선택하는 단계에서, 해당되는 셀이 없을 경우 거리가 가장 가까운 활성영역의 셀을 선택하고, 상기 x 및 y는 차량 전방과 좌측을 x축과 y축으로 하는 좌표계 상의 좌표를, 상기 v은 속도를, 상기 ω는 각속도를, 상기 t는 시각을, 상기 T는 예측 시간을 나타내는 것을 특징으로 한다.When calculating the ROPP according to the lookup table, the step of calculating the ROPP includes: inputting the vehicle speed v and the angular speed omega; "

"And"

"Calculating a predicted path end point (x (T), y ( T)) for the prediction time T on the basis of;" Φ (T) = tan -1 (y (T) / x (T)) " of Calculating a phase angle (? (T)) of the estimated path end point in accordance with the equation (4), calculating a corrected vehicle speed (v ') and angular velocity (?) Based on CTRV_EKF applying Extended Kalman Filtering ω ') is extracted from the selected cell and a direction error average ( μ _φ ) and a standard deviation ( σ _φ ) are extracted from the selected cell to obtain "Δ φ _max = μ _φ + βσ _φ , Δ φ _min = μ _φ - calculating a maximum orientation errors (Δ φ _max) and the minimum direction errors (Δ φ _min) on the basis of a formula βσ _φ "; The maximum orientation errors (Δ φ _max) and the minimum direction errors (Δ φ _min) the base by "Φ (T) + Δ φ min ~ Φ (T) + Δ φ max" the step of defining a phase angle range of ROPP ; From the selected cell by the extracting mean (μ _a) and the standard deviation (σ _a) of the acceleration "a _max = according to the expression of μ _a + βσ _a "calculating the maximum acceleration (a _max); "

"By calculating a maximum reach (s _max (v)) based on; a and the radius of s _max (v) a phase angle _{Φ (T) + Δ φ min} ~ Φ (T) + Δ φ max range Wherein the step of selecting the cell selects a cell of the active region closest to the cell in the absence of the corresponding cell, wherein x and y are set to x Axis and y-axis, v is a velocity,? Is an angular velocity, t is a time, and T is a predicted time.

The step of calculating the ROD includes: determining a vehicle lateral width (W) stored in advance; And obtaining the ROD by expanding the left and right outer regions of the ROPP by half (W / 2) of the vehicle lateral width (W), respectively.

Selecting and outputting only the pedestrian candidates located in the ROD may include projecting the ROD onto the input image; And comparing the ROD position projected on the input image with the position of the candidate pedestrian object and selecting and outputting only the pedestrian candidate objects located in the ROD as a verification object.

"에 따라 ROD의 차량 중심 좌표계를 입력 영상 좌표계로 변환하며, 상기 x(u, v, 1)는 입력 영상 좌표계, 상기 X(X, Y, Z, 1)는 차량 중심 좌표계, 상기 P는 카메라의 투사 매트릭스인 것을 특징으로 한다.Wherein projecting the ROD to the input image comprises:

"Converts the vehicle center coordinate of ROD to the input image coordinate system, the x (u, v, 1) is the input image coordinate system, the X (X, Y, Z, 1) is a vehicle central coordinate system, wherein P is the camera according to the Is a projection matrix of < / RTI >

According to another aspect of the present invention, there is provided a pedestrian protection system including: a candidate generator for extracting a pedestrian candidate object from an input image; An ROPP calculation unit for calculating a region of prediction path (ROPP), which is an area where a future route of the vehicle can exist through statistical analysis of the traveling data; An ROD calculation unit for calculating the ROD by expanding the left and right outer regions of the ROPP according to predetermined criteria; And a pedestrian candidate selecting unit for selecting only the pedestrian candidates located in the ROD among the pedestrian candidate objects extracted through the candidate generating unit and transmitting the pedestrian candidates to the candidate verifying unit. And a candidate verifying unit for performing a candidate verification on each of the filtered candidate candidate objects to detect a pedestrian in a dangerous state. The ROPP calculating unit sets a direction error range using the speed and angular velocity of the vehicle, And the ROPP is calculated using the error range and the maximum acceleration.

And the ROPP calculator calculates ROPP using a preset lookup table using the maximum acceleration and the direction error range.

According to the pedestrian protection system and method of the present invention, when the pedestrian for active pedestrian protection is detected according to the three steps of candidate generation, candidate verification, and risk evaluation, considering the relatively large processing load of the candidate verification, Risk assessment should be performed prior to candidate verification. As a result, the processing load of the candidate verification is minimized and the processing time is reduced, thereby improving the real-time performance of the entire system.

Also, according to the present invention, the ROP can be set to be narrower by narrowly setting the area ROPP in which the future route may exist in consideration of the speed and the angular speed, thereby saving time required for the pedestrian verification.

Further, the present invention implements the maximum value of the acceleration and the direction error as a look-up table of the speed and the angular velocity space, so that the learning and use becomes easier.

1 is a view for explaining a pedestrian protection method according to an embodiment of the present invention.
2 is a diagram illustrating a problem of a physics-based path prediction using a CTRV motion model that can be generated at the time of vehicle rotation.
Fig. 3 is a view for explaining a range of longitudinal motion state which varies according to the relationship between the velocity-angular velocity and the acceleration. Fig.
4 is a diagram for explaining the range of the lateral motion state, which is changed according to the relationship between the velocity-angular velocity and the direction error.
5 is a view for explaining a range of a direction error according to the speed when the vehicle is running.
6 is a diagram for explaining the definition of a direction error.
FIG. 7 is a diagram for explaining cell division of a velocity-angular velocity plane for generating a look-up table according to an embodiment of the present invention.
8 is a diagram for explaining the RODPP calculation step according to an embodiment of the present invention in more detail.
9 and 10 are views for explaining the ROD calculation step according to an embodiment of the present invention in more detail.
11 is a diagram for explaining the step of selecting a candidate for a pedestrian according to an embodiment of the present invention in more detail.
12 is a diagram for explaining ROPP set in various situations when the vehicle is running.
13 is a view for explaining an ROD set in various situations when driving the vehicle.
Figs. 14 and 15 are diagrams for explaining the ROD set in various situations when driving the vehicle, in a forward-looking manner.
16 is a view for explaining a pedestrian protection system according to an embodiment of the present invention.

It is noted that the technical terms used in the present invention are used only to describe specific embodiments and are not intended to limit the present invention. In addition, the technical terms used in the present invention should be construed in a sense generally understood by a person having ordinary skill in the art to which the present invention belongs, unless otherwise defined in the present invention, Should not be construed to mean, or be interpreted in an excessively reduced sense. In addition, when a technical term used in the present invention is an erroneous technical term that does not accurately express the concept of the present invention, it should be understood that technical terms can be understood by those skilled in the art. In addition, the general terms used in the present invention should be interpreted according to a predefined or prior context, and should not be construed as being excessively reduced.

Furthermore, the singular expressions used in the present invention include plural expressions unless the context clearly dictates otherwise. In the present invention, terms such as " comprising " or " comprising " and the like should not be construed as encompassing various elements or various steps of the invention, Or may further include additional components or steps.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings, wherein like reference numerals refer to like or similar elements throughout the several views, and redundant description thereof will be omitted.

In the following description, well-known functions or constructions are not described in detail since they would obscure the invention in unnecessary detail. It is to be noted that the accompanying drawings are only for the purpose of facilitating understanding of the present invention, and should not be construed as limiting the scope of the present invention with reference to the accompanying drawings.

1 is a view for explaining a pedestrian protection method according to an embodiment of the present invention.

Referring to FIG. 1, the pedestrian protection method of the present invention includes a candidate generation (HG) step S10 for extracting a pedestrian candidate object from an input image, an ROD (Region of Danger) from the travel data, A risk assessment (RA) step S20 for selecting candidates of possible pedestrian candidates, and a candidate verification operation based on CNN (Convolutional Neural Network) are performed for each of the selected candidate pedestrian objects, And a candidate verification (HV) step S30 to notify.

That is, in the present invention, the RA step S20 is performed before the HV step S30, unlike the conventional method. If the operation sequence is changed, a larger number of inputs must be processed in the RA step S20 than in the conventional case, but a smaller number of inputs are processed in the HV step S30 than in the conventional case. However, since the calculation request amount of the HV step S30 is much larger than that of the RA step S20, the method of the present invention can have a shortened operation time as compared with the conventional method.

For example, it is assumed that each of the HV step (S30) and the RA step (S20) consumes 10 ms and 0.1 ms to process one pedestrian candidate (this means that the calculation requirement of the HV step (S30) Assuming that it is 100 times as large as it is, it generally has a larger value).

In this case, if 10 pedestrian candidates are output at the HG level, HV outputs 4 pedestrians, and 2 pedestrians are output at the RA stage, HV and RA are operated 10 times and 4 times, respectively, The time consumed is "10 ms × 10 + 0.1 ms × 4 = 100.4 ms".

On the other hand, if 10 pedestrian candidates are output at the HG level, the RA outputs 4 dangerous pedestrian candidates, and the HV outputs 2 dangerous pedestrians, the present invention operates RA 10 times and HV 4 times, Is " 0.1 ms x 10 + 10 ms x 4 = 41 ms ".

That is, since the execution time of the HV step is much larger than the execution time of the RA step, the system execution time is approximately proportional to the number of inputs processed by the HV step.

Therefore, the present invention reduces the information processing amount in the HV step, thereby shortening the operation time of the entire system, and consequently improving system real-time performance.

1, the RA step S20 of the present invention includes a step S21 of calculating a region of a prediction path (ROPP) which is an area where a future route of the vehicle can exist through statistical analysis of the traveling data of the vehicle (Step S22) of calculating a dangerous area ROD (Region of Danger) based on the ROPP, and a step of selecting only a pedestrian candidate located in the ROD among a plurality of pedestrian candidates as a candidate pedestrian candidate , And transmitting it to the HV step S30 (S23).

Hereinafter, the RA step S20 according to an exemplary embodiment of the present invention will be described in more detail with reference to FIG. 2 to FIG.

For reference, vehicle path prediction can be largely performed in three ways: physics-based, maneuver-based, and interaction-based. The physics-based prediction method is to predict the path by extending the current motion state such as speed, acceleration, and angular velocity. This is a relatively light algorithm that is easy to implement in real time. However, it has a limited predictable time and a large error Lt; / RTI > The reason for this uncertainty is that the vehicle movement is basically steered continuously according to the intention of the driver. The intention-based prediction method has the advantage of reducing the uncertainty by grasping the driver's intention. However, since it takes a burden to process various large-volume information such as map, forward image, and driver image in order to understand the intention of the driver, Is still limited. Interaction-based prediction method is a method to consider interaction between vehicle and other vehicle, and is still in the early stage of development.

Accordingly, the present invention predicts ROPP using a physics-based prediction method.

Physics-based path prediction is based on a predicted path endpoint (x (t), y (t)) based on the Constant Turn-Rate and Velocity (CTRV) motion model assuming that the vehicle velocity and angular velocity are constant over a short period of time. Can be calculated.

[Equation 1]

&Quot; (2) "

In this case, x and y are coordinates on the coordinate system having the x-axis and the y-axis as the front and the left sides of the vehicle, v is the vehicle speed, ω is the angular speed, and t is the time. t can be increased from the current time point 0s to the maximum predicted time T (for example, 3 seconds).

On the other hand, the CTRA motion model frequently used for vehicle pose estimation has a problem that it is not suitable for path prediction including low-speed travel. It is because the CTRA motion model has a problem of predicting the future route in the vehicle backward direction.

Also, the CTRV motion model has a problem of generating a large error during rotation. In contrast to the curved line which keeps the steering angle constant, the driver gradually increases the steering angle gradually and then adjusts it gradually smaller. At this time, since the turning radius becomes smaller as the steering angle becomes larger, Because a large error is generated when the future path is predicted.

2 is a diagram illustrating a problem of a physics-based path prediction using a CTRV motion model that can be generated at the time of vehicle rotation.

In FIG. 2, it is assumed that the blue dotted line expresses the actual path during the period of t = 0 to T, the red dotted line represents the predicted path, and the vehicle rotates along the green path from the left to the upper right.

2 (a) is a view showing the actual path and the predicted path of the vehicle immediately before the left turn. In the actual path, the vehicle speed is decreased and the direction change is started upward. However, in the physics-based path prediction method, The straight line path that maintains the current speed is predicted.

FIG. 2 (b) is a diagram showing an actual path and a predicted path of the vehicle in a situation where the left turn is almost completed. In the actual path, the vehicle speed increases and starts to go straight upward. However, Because it is in the path, it predicts the curved path that maintains the current turning radius.

2 (c) is a graph of estimated curvature (assumed to be an actual curvature, blue solid line) and estimated curvature (red solid line) calculated by fitting a real path and a predicted path to a circle during a left turn, It can be seen that it is difficult to compensate it by a simple method because the estimated curvature is smaller than the estimated curvature and the size of the error is not constant.

The limitation of such a physics-based path prediction occurs because the driver constantly adjusts the vehicle motion. In the case of rotating at a low speed, the above-mentioned error necessarily occurs because the magnitude and rate of change of the steering angle are both large.

Therefore, if it is not possible to determine the precise future path, the next best way is to predict the area where the future path can exist as small as possible. In the present invention, the probability of occurrence of the error is minimized by statistically analyzing the traveling data to predict the ROPP, which is a region where a future route can exist, in a sector shape having a minimum size.

The vehicle basically includes an electronic control chassis control system such as an electronic stability control (ESC), etc., and the ESC can provide driving data including a speed and an angular speed.

Therefore, the present invention predicts the vehicle path based on the speed and the angular speed included in the running data. In particular, the driver models the range of longitudinal / lateral motion according to the vehicle speed considering the considerably different driving behavior according to the vehicle speed, and then calculates the area ROPP in which the future route can exist based on the model.

At this time, the range of the longitudinal motion state is determined by the relationship between the velocity-angular velocity and the acceleration, and the lateral motion state range is characterized by the relationship between the velocity-angular velocity and the direction error.

First, referring to FIG. 3, a description will be made of a range of a longitudinal motion state varying according to a relationship between a velocity-angular velocity and an acceleration.

Fig. 3 (a) is a diagram showing the range of the acceleration according to the velocity, and Fig. 3 (b) is the diagram showing the range of the acceleration according to the angular velocity.

Referring to FIG. 3 (a), when the vehicle speed is close to zero, the range of the acceleration is relatively wide, and when the vehicle speed reaches a certain speed, the range of the acceleration greatly decreases. Thus, the maximum acceleration according to the speed can be approximated in an inverse proportion relation.

Referring to FIG. 3 (b), when the absolute value of the angular velocity is smaller than 0.1 (|? | <0.1), the vehicle is in the straight running state and when the absolute value is greater than or equal to 0.1 (|? |? 0.1) And the sign of the acceleration indicates the direction of rotation.

3 (b), the range of the acceleration is wide because of a high possibility of occurrence of a large change in the speed in the case of the straight running in which the absolute value of the angular velocity is close to zero. In the case of the rotational running in which the absolute value of the angular velocity is large, It is possible to confirm that the range of the acceleration is narrowed. On the other hand, it can be seen that the maximum acceleration is kept constant regardless of the angular velocity. Since the maximum acceleration is required when determining the ROPP, the analysis according to the angular velocity is not significant in the longitudinal direction.

Referring to FIG. 4, a description will be made of a range of a lateral motion state that changes according to a relationship between a velocity-angular velocity and a direction error.

FIG. 4A is a diagram showing a range of a direction error according to a speed, and FIG. 4B is a diagram showing a range of a direction error according to an angular velocity.

Referring to FIG. 4 (a), as in the case of the acceleration, the direction error can be approximated to an inverse relationship in which the range decreases as the velocity increases.

On the other hand, referring to (b) of FIG. 4, it can be seen that the range of the direction error is relatively constant according to the value of the angular velocity, but the position of the range is different.

When the angular velocity increases in the positive direction (left turn), the range of the direction error has a negative value (the actual path is on the right side with respect to the expected path), and when the angular velocity increases in the negative direction The range of directional errors has a positive value (the actual path is on the left with respect to the expected path). That is, when analyzing the lateral motion state, analyzing into one motion state without distinguishing between the straight travel and the rotational travel results in obtaining an unnecessarily wide ROPP in both the straight travel and the rotation travel.

Therefore, when analyzing the lateral motion state, it is necessary to analyze the distinction between the straight running and the running.

5 is a view for explaining a range of a direction error according to the speed when the vehicle is running.

Referring to FIG. 5, it can be confirmed that when the direction error range is set in consideration of the angular velocity, the ROPP is expected to be reduced in the lateral direction.

Among the methods of setting the direction error range, there is a method of setting using the maximum and minimum functions of the direction error absolute value according to the speed proposed in Korean Patent No. 10-1770924. The above method assumes that the same directional error range is obtained at the same speed regardless of the angular velocity.

However, when checking the running data, it can be confirmed that the direction error range is different depending on the angular velocity value.

5 shows the direction error distribution (green dots) during the straight running and the directional error distribution (red dots) during the left turning. The dark blue line and the bright blue line represent the set maximum and minimum error functions, respectively. The direction error range in the straight run is included between the maximum and minimum direction error functions, but the direction error range at the left turn not only goes out between the two functions but also narrows.

For example, when the speed is 5 m / s and the speed is 5 m / s, the method proposed in Korean Patent No. 10-1770924 sets the range of the direction error to A. However, considering the angular speed, When analyzing, the range of the direction error is set to B.

As a result, the method proposed in Korean Patent No. 10-1770924 is considered to reduce the efficiency because it generates unnecessary areas as much as AB. If the angular velocity of the direction error range can be set in consideration of the efficiency, the phase angle range of ROPP Can be reduced.

Therefore, as shown in FIG. 5, the definition of the direction error range set in consideration of the angular velocity will be described in order to reduce the phase angle range of the ROPP.

6 is a diagram for explaining the definition of a direction error.

The method proposed in Korean Patent No. 10-1770924 defines a direction error as a directional angle between the end point of the expected path obtained based on the CTRV and the end point of the actual path.

In the case of the above method, as shown in FIG. 6 (a), there is a problem that ROPP does not include all the actual paths when the actual path is bent.

The vehicle is a left turn along the green line, and the blue dotted line and the red dotted line represent the actual path and the expected path from the present to the T seconds, respectively.

6 (a), the direction error is only Δφ _min , which is the angle between the line segment a connecting the current position o and the end point of the predicted path and the line segment b connecting o and the end point of the actual path . In this case, the ROPP is set to a red sector having a range of ± Δ φ _min based on the segment a as shown in FIG. 6 (a), which does not include the actual path for the remaining time except one T- . On the other hand, if a direction error Δ φ _max obtained from a line segment c that has the largest angle with the line segment a among the points passing through the actual path is applied, ROPP including the real path as in the case of the green sector can be obtained.

Therefore, if all of the direction errors for all points on the actual path are used in the analysis, ROPP can be set to include all the actual paths.

The definition of the direction error is as shown in FIG. 6 (b), in which the line segment a connecting the end point of the predicted path at the current position o and the line segments b ₁ , b ₂ , b connecting the respective points on the actual path moved for T seconds _3, ..., b _n) angle between the direction of both error _{(Δ φ 1, Δ φ 2} , Δ φ 3, ..., can be defined as Δ φ _n).

If the maximum and minimum ranges are obtained by using the direction error data, the coverage ratio of the actual path in the ROPP can be further improved.

Hereinafter, an implementation method of using a look-up table to determine a range of motion state on a three-dimensional space based on a velocity-angular velocity space will be described in setting a direction error range.

As shown in FIGS. 3A and 3B and FIGS. 4A and 4B, when the motion state is analyzed in the two-dimensional space, the velocity and acceleration, angular velocity and acceleration, velocity and direction error, angular velocity It is possible to express the data distribution of the direction error by a simple relation.

However, as shown in FIGS. 3 (c) and 4 (c), when the acceleration and the range of the direction error are analyzed in the three-dimensional space according to the velocity-angular velocity, It is not easy.

According to the method of generating the lookup table in advance using the learning data for the acceleration and the direction error range in the velocity-angular velocity space and setting the ROPP by using the lookup table while driving, the acceleration due to the velocity- It is possible to analyze the range of directional errors.

FIG. 7 is a diagram for explaining cell division of a velocity-angular velocity plane for generating a look-up table according to an embodiment of the present invention.

First, the maximum value and the minimum value of the velocity, the maximum value and the minimum value of the angular velocity are obtained from the learning data as shown in FIG. 7A, and the region where the learning data exists in the velocity and angular velocity space is shown by the solid line .

The area of the solid red line is divided at constant intervals with respect to the velocity and the angular velocity axis, and a small area determined by the velocity and the angular velocity in a specific range is defined as a cell. The learning data is assigned to the cell corresponding to the velocity and the angular velocity of each moment.

The green region in (b) of FIG. 7 represents the active region in which the allocated data exists, and the region other than green represents the inactive region in which there is no allocated data. A lookup table for the longitudinal motion range and the lateral motion range can be generated by obtaining the acceleration and the direction error of the moments assigned to the cell for each active area cell and obtaining the average and variance thereof.

Each cell assumes that the data have a Gaussian distribution. During traveling, the maximum acceleration can be calculated by using the information of the longitudinal lookup table cell corresponding to the current vehicle speed and angular velocity. The maximum acceleration can be calculated using the information of the lateral lookup table cell and the maximum direction error (left direction based on the estimated route) Directional error (rightward with respect to the expected path) can be calculated.

When such longitudinal / lateral motion state range modeling is completed, the present invention can calculate a region ROPP in which a future path can exist based on the longitudinal / lateral motion state range model through the ROPP calculation step S21.

8 is a diagram for explaining the ROPP calculation step according to an embodiment of the present invention in more detail.

First, the current speed v and the angular speed omega of the vehicle included in the running data are extracted and acquired (S211).

And then calculating equation (1) and equation (2) based on a predicted path end point (x (T), y ( T)) to (S212), the phase of the predicted path of each end point ^{(Φ (T) = tan -1} (y ( T ) / x ( T )) (S213).

Then, it selects the cell to which the vehicle speed V 'and the angular velocity?', Corrected through the CTRV_EKF, belongs (S214). At this time, when there is no corresponding cell, the cell of the active region closest to the distance is selected. The CTRV_EKF means applying the EKF (Extended Kalman Filtering) based on the CTRV motion model, and performs the role of removing the noise of the vehicle speed and the angular velocity in consideration of the vehicle behavior characteristics.

Then, a maximum direction error (DELTA phi _max ) is calculated from the selected cell using a direction error average ( mu _phi ) and a standard deviation ( sigma _phi ) = μ _φ + βσ _φ ) and the minimum direction error (Δ φ _min = mu _] - [ beta] [ _phi] ) (S215).

And determining a range of phase angles of the ROPP based on the maximum direction error and the minimum direction error ( ? ( T ) + ?? _min - ? ( T ) + ?? _max ) is calculated (S216).

)를 계산하도록 한다(S218).Then, the average ( μ _a ) and standard deviation ( σ _a ) of the acceleration are extracted from the selected cells and the maximum acceleration ( a _max = mu _a + beta _a ) is calculated (S217), and the maximum reaching distance (

(S218).

Finally, ROPP defined as a sector whose radius is s _max (v) and whose phase angle is in the range of ? ( T ) + ?? _min to ? ( T ) + ?? _max is calculated (S219).

When the ROPP is calculated through the ROPP calculation step S21 of FIG. 8, the present invention calculates the ROD based on the ROPP as shown in FIG. 9 and projects it to the input image, Candidate objects are selected.

FIGS. 9 and 10 are views for explaining the ROD calculation step S22 according to an embodiment of the present invention in more detail.

Referring to FIG. 9, in the ROD calculation step of the present invention, the lateral width W of the vehicle stored in advance is determined (S221), and the outer side region and the right lateral side region of the ROPP are divided by half (W / To thereby obtain the ROD (S222).

The red dotted line in FIG. 10 (a) represents the estimated path calculated based on the speed and the acceleration, and the blue sector represents the ROPP calculated based on the maximum arrival distance and the maximum direction error. The blue region in FIG. 10 (b) represents the ROD set to extend horizontally by half of the vehicle lateral width on both sides of the set ROPP.

11 and 12 are views for explaining the pedestrian candidate object selection step S23 according to an embodiment of the present invention in more detail.

Referring to FIG. 11, the step of selecting a candidate for a pedestrian according to the present invention includes a step S231 of projecting an ROD to an input image by converting the center coordinate system of the ROD into an input image coordinate system (S231) Comparing the positions of the objects, selecting only the pedestrian candidate objects located in the ROD, and transmitting them to the HV step S30 (S232).

Since the ROD is calculated by statistical analysis of the running data, it has a coordinate system different from that of the input image, that is, a vehicle center coordinate system. Therefore, it is necessary to convert the vehicle center coordinate system of the ROD into the input image coordinate system using the projection matrix ( P ) of the camera.

If a camera projection matrix P is multiplied by a point X (X, Y, Z, 1) on the vehicle center coordinate system as in Equation 6, a point x (u, v, 1) on the image can be obtained, The ROD can be projected onto the input image.

&Quot; (6) &quot;

At this time, the size of P is 3 × 4, and X and x use a homogeneous coordinate system. Since the ROD on the vehicle center coordinate system exists on the ground surface, Z is set to a negative -H of the camera height. The ROD on the image can be calculated by sampling the boundary of the ROD with a predetermined number of points and calculating the points on the image corresponding to these points.

A rough fan-shaped figure drawn with a solid line in FIG. 12 (a) shows the ROD on the vehicle center coordinate system, and FIG. 12 (b) shows the ROD projected on the corresponding input image. However, although the ROD is roughly fan-shaped, the ROD projected on the input image may not appear as a sector shape due to perspective transformation.

Since the ROD means an area occupied by the vehicle within a predetermined time, the present invention can determine that the pedestrian is dangerous if any pedestrian exists in the ROD as shown in FIG. 12 (c). In addition, when using the vehicle center coordinate system ROD, it is possible to evaluate whether the pedestrian center coordinate (X, Y) exists in the ROD. In addition, when using the image coordinate system ROD, it is possible to evaluate whether the base center of the pedestrian zone rectangle exists in the ROD.

The red solid line in FIG. 12 (c) shows the ROD. Referring to this, it can be seen that when the vehicle speed is small, the maximum reachable distance is small and the ROD with the wide angle range is calculated. In the case of a pedestrian marked with a red square, it is classified as a dangerous pedestrian because the center of the bottom exists in the ROD, and a pedestrian marked with a blue square is classified as a non-dangerous pedestrian because it exists outside the ROD.

FIG. 13 is a view for explaining ROPP set in various situations when driving the vehicle.

The red sector represents ROPP of the look-up table (direction error average-direction error standard deviation) (LUT ( μ-σ )) scheme, and the black sector represents ROPP of the method proposed in Korean Patent No. 10-1770924. In addition, the red dotted line represents the predicted path based on the Constant Turn-Rate and Velocity (CTRV) motion model, and the blue dotted line represents the CTRV motion model based actual path.

13 (a) and 13 (b) are diagrams showing a low-speed forward state and a high-speed forward state. By analyzing the angular velocity and velocity together at low speed, the range of direction error and the maximum acceleration become small, so that the unnecessary range is reduced and the ROPP can be set smaller. Since there is only data in a straight line state at high speed with almost no rotation, the method proposed in Korean Patent No. 10-1770924 and the size of ROPP are almost similar.

13C, 13D and 13E show the ROPP at the rotation initial point, the rotation intermediate point, and the rotation end point, respectively. At the beginning of the turn, most drivers reduce speed. In this case, when the method proposed in Korean Patent No. 10-1770924 is applied, semicircular ROPP having an azimuth angle range of 180 degrees is drawn. On the other hand, when using the lookup table method of the present invention, The size of ROPP is smaller than that of ROPP. Likewise, a smaller size ROPP may appear, including both the actual path and the rotation midpoint and end point.

FIGS. 14 and 15 are diagrams for explaining an ROD set in various situations when driving the vehicle, on an input image.

The red solid line represents ROD by the method of the present invention, and the blue solid line represents ROD by the method proposed in Korean Patent No. 10-1770924. In addition, the red dotted line and the blue dotted line represent the predicted path and the actual path.

14 (a) and 14 (b) show the ROD in the low-speed straight-ahead state. FIG. 14 (a) shows the ROD when the vehicle starts in a stopped state, which is a very low speed state, and a short ROD is generated in the right and left directions and short in the longitudinal direction. However, according to the method of the present invention, it is possible to have a smaller ROD than both in the longitudinal / transverse direction.

FIG. 14 (b) shows the ROD in the low speed straight state, and according to the method of the present invention, the ROD is significantly reduced. FIG. 14C shows an ROD in a fast forward straight state, and a long ROD is generated in the right and left directions and in the longitudinal direction. In the high-speed straight ahead, the method according to the present invention and the method proposed in Korean Patent No. 10-1770924 are almost similar.

Figures 15 (a), (b) and (c) show ROD at the beginning, middle, and end of rotation. According to the method of the present invention, the ROD is considerably smaller than the method proposed in Korean Patent No. 10-1770924, but the actual path is included.

A pedestrian protection method by a method of setting a direction error range using a lookup table of the present invention and a pedestrian protection method by a method proposed by Korean Patent No. 10-1770924 were compared and evaluated.

As an evaluation method, raw data of KITTI Vision Benchmark Suite was used.

Data provides data and calibration information collected by a vehicle equipped with a front color stereo camera, a front monochrome stereo camera, a 3D laser scanning rider, and a high precision positioning system while traveling in various environments.

Among them, the positioning information was acquired using OXTS® RT 3003, and the distance and angle resolution were 0.02 m and 0.1 °, respectively. The data consist of 71 sequences, of which 67 sequences are used.

The excluded sequence includes cases where the positioning error is severe including the shadow area of the GPS and the case where the inertial measurement unit (IMU) error is severe in the stationary state and the length is too short. The 67 sequences consist of 35,637 frames acquired at 10 Hz and correspond to approximately 59 minutes of travel. Data are categorized into four categories (city, residential, road, campus) according to the area of acquisition.

When the data for learning and evaluation are used in the same manner as in Table 1 below, the actual path inclusion ratio and the ROD ratio with respect to the entire image are calculated using the method proposed in Korean Patent No. 10-1770924, the LUT (min-max), the LUT mu] - [sigma] ).

Method % Actual Path Containment ROD Ratio vs. Total Image (%) The method proposed in Korean Patent No. 10-1770924 99.28 28.96 LUT (min-max) 99.95 24.02 LUT (μ-σ) 99.24 18.38

The method proposed in Korean Patent No. 10-1770924 is a method in which the range of longitudinal / transverse motion states is expressed as a function, and LUT (min-max) is a method of determining the maximum value of data , And the minimum value is stored as it is. The LUT (μ-σ) is a method of determining the maximum and minimum ranges based on the average and standard deviation of the data included in the cell.

The number of cells belonging to the lookup table was determined to be 27 × 33, and β = 3 (maximum and minimum values in the range of three times the dispersion, μ ± 3 σ) were used in the LUT (μ-σ)

In addition, the maximum predicted time T is set to 3 seconds. The performance of ROPP is evaluated by how much the set ROPP includes the actual path (ratio of the actual path included), and the ROD performance is calculated by projecting the generated ROD on the input image, Respectively.

For reference, the higher the actual path inclusion ratio is, the better the better, and the lower the ROD ratio is, the better.

In Table 1, since the LUT (min-max) reflects the maximum and minimum values obtained from the data distribution, the actual path inclusion ratio is the highest at 99.95% and the LUT (μ-σ) is 99.24% 10-1770924. &Lt; / RTI &gt;

On the other hand, the ROD ratio of the whole image is 18.38% for the LUT (μ-σ), which is about 10% lower than that proposed in Korean Patent No. 10-1770924.

Accordingly, it can be seen that the LUT (μ-σ) method, which has the largest reduction in the ROD ratio compared with the entire image, is the best in comparison with the method proposed in Korean Patent No. 10-1770924 have.

Table 2 below is a table comparing the results of the actual path inclusion ratio and the ROD ratio with respect to the whole image when the learning and evaluation data are separated again for the LUT (min-max) and LUT (μ-σ) methods.

Method Direction error using endpoint Direction error using all points % Actual Path Containment ROD Ratio vs. Total Image (%) % Actual Path Containment ROD Ratio vs. Total Image (%) LUT (min-max) 94.38 21.87 98.14 22.23 LUT (μ-σ) 96.94 19.73 98.25 18.16

For learning and evaluation, 2/3 of the 67 sequences were used for learning and the remaining 1/3 were used for evaluation and 3-fold cross validation was performed. Also, when calculating the direction error, the result of the method proposed in Korean Patent No. 10-1770924 using only the angular error between the estimated path and the end points of the actual path, and the angle between the end point of the estimated path and all points on the actual path The methods using all the errors were also compared.

First, the results of direction errors using all the points in Table 2 show that ROPP and ROD performances are almost similar when learning and evaluation data are used in the same way as learning and evaluation data are used .

In addition, the LUT (μ-σ) method of the present invention shows a superior performance to the LUT (min-max) with an actual path inclusion ratio of 98.25% and an ROD ratio of 18.16% with respect to the whole image.

Table 2 shows that the actual path inclusion ratio is higher when the direction error data using all points are used than the direction error data using only the end point, and the ratio of the ROD to the total image is larger than that of the LUT (μ-σ) .

Taking all the above results into consideration, it is analyzed that the use of all the points on the actual path increases the learning data sufficiently, as compared with the case where only the end point is used, thereby helping to model the cell distribution by the Gaussian distribution.

16 is a view for explaining a pedestrian protection system according to an embodiment of the present invention.

16, the pedestrian protection system of the present invention includes an input image acquiring unit 10 for acquiring and providing an input image through various image acquisition devices such as a front camera, an electronic control chassis control system, A candidate generator 30 for extracting a pedestrian candidate object from the input image, and an ROD (Region of Danger) from the running data to obtain a running data including a vehicle collision A candidate evaluating unit 40 for selecting candidates for possible pedestrian candidates, and a candidate verifying unit 50 for selecting and notifying only pedestrians by verifying respective pedestrian candidate objects based on CNN (Convolutional Neural Network) And a vehicle control unit 60 for performing various vehicle control operations for preventing collision in consideration of the vehicle's running state and the like.

In addition, the risk evaluation unit 40 further includes an ROPP calculating unit 41 for calculating ROPP, which is an area where a future route of the vehicle can exist, through a statistical analysis of the running data of the vehicle, a ROPP calculating unit 41 for calculating ROPP, And a pedestrian candidate selecting unit 43 for selecting only the pedestrian candidate objects that are likely to be out of the vehicle based on the ROD and providing the pedestrian candidate objects to the candidate verifying unit 50. [

That is, considering that the processing load of the candidate verifying unit 50 is much larger than that of the risk evaluating unit 40, the pedestrian protection system of the present invention is configured such that the candidate verifying unit 40 50 to minimize the amount of information input. As a result, the operation time of the system of the present invention is greatly reduced compared with the conventional system, and the system real time can be obtained.

The features, structures, effects and the like described in the foregoing embodiments are included in at least one embodiment of the present invention and are not necessarily limited to one embodiment. Further, the features, structures, effects and the like illustrated in the embodiments can be combined and modified by other persons skilled in the art to which the embodiments belong.

Therefore, it should be understood that the present invention is not limited to these combinations and modifications. While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is clearly understood that the same is by way of illustration and example only and is not to be construed as limiting the scope of the present invention. It can be seen that various modifications and applications are possible. For example, each component specifically shown in the embodiments can be modified and implemented. It is to be understood that the present invention may be embodied in many other specific forms without departing from the spirit or essential characteristics thereof.

Claims (8)

A candidate generating step of extracting a pedestrian candidate object from an input image;
Calculating a Region of Prediction Path (ROPP), which is an area where a future route of the vehicle may exist, through statistical analysis of the driving data;
Calculating an ROD (Region of Danger) by extending the left and right outer regions of the ROPP according to a preset reference; And
Selecting and outputting only the pedestrian candidates located in the ROD among the pedestrian candidate objects extracted through the candidate generation step; And
And a candidate verification step of performing a candidate verification on each of the filtered candidate pedestrian candidates to detect a pedestrian in a dangerous state,
Wherein the step of calculating the ROPP comprises:
Obtaining a speed and an angular velocity of the vehicle included in the running data;
Calculating a predicted path endpoint for a predicted time using the velocity and acceleration and calculating a phase angle of the predicted path endpoint;
Selecting a cell to which said velocity and angular velocity belongs from a look-up table;
Reading average and standard deviation of direction errors from the selected cell and calculating a maximum direction error and a minimum direction error with respect to a phase angle of the estimated path end point using an average and standard deviation of the read direction error;
Calculating a range of the phase angle of the ROPP using the calculated maximum direction error and minimum direction error;
Reading the mean and standard deviation of the acceleration from the selected cell and calculating a maximum reach distance using the average and standard deviation of the read acceleration; And
And calculating the ROPP defined by the range of the phase angle and the maximum reach distance,
Wherein the lookup table includes a plurality of cells defined according to a speed and an angular rate of the vehicle and calculates an acceleration and a direction error using a speed and an angular velocity assigned to each cell for each cell, Wherein the mean and standard deviation of the standard deviation and the calculated direction error are calculated and recorded.
delete The method according to claim 1,
Wherein the step of calculating the ROPP comprises:
"

"And"

(X (T), y (T)) for the predicted time T based on the estimated path endpoint
Calculating a phase angle? (T) of the expected path end point according to an equation of? (T) = tan ^-1 (y (T) / x (T));
Selecting a cell to which the corrected velocity v 'and the angular velocity omega' belongs based on CTRV_EKF applying Extended Kalman Filtering (EKF) based on the CTRV motion model;
Extracting an average _(φ μ) and standard deviation (σ _φ) of the selected cell from the direction error "Δ φ _max = μ _φ + βσ _φ , Δ φ _min = μ _φ - calculating a maximum orientation errors (Δ φ _max) and the minimum direction errors (Δ φ _min) on the basis of a formula βσ _φ ";
The maximum orientation errors (Δ φ _max) and the minimum direction errors (Δ φ _min) the base by "Φ (T) + Δ φ min ~ Φ (T) + Δ φ max" the step of defining a phase angle range of ROPP ;
From the selected cell by the extracting mean (μ _a) and the standard deviation (σ _a) of the acceleration "a _max = calculating a maximum acceleration ( a _max ) according to an equation of? _a +? _a ?
"

(S _max (v)) on the basis of &quot;
Calculating an ROPP defined by a sector whose radius is s _max (v) and whose phase angle is in the range of ? ( T ) + ?? _Min to ? ( T ) + ?? _Max ,
In the step of selecting the cell, if there is no corresponding cell, a cell of the active region closest to the cell is selected,
Wherein x and y are coordinates on a coordinate system having the x-axis and the y-axis as the front and the left sides of the vehicle, v is the velocity,? Is the angular velocity, t is the time and T is the predicted time. Pedestrian protection methods.
The method of claim 3,
The step of calculating the ROD
Determining a vehicle lateral width (W) stored in advance in the ROPP; And
And obtaining the ROD by expanding the left and right side outer regions of the ROPP by half (W / 2) of the vehicle lateral width (W), respectively.
The method according to claim 1,
The step of selecting and outputting only the pedestrian candidates located in the ROD
Projecting the ROD onto the input image; And
Comparing the ROD position projected on the input image with the position of the candidate pedestrian object, and selecting and outputting only the pedestrian candidate object located in the ROD as the object to be verified.
6. The method of claim 5,
The step of projecting the ROD onto the input image
"

"Converts the vehicle center coordinate of ROD to the input image coordinate system, the x (u, v, 1) is the input image coordinate system, the P (X, Y, Z, 1) is a vehicle central coordinate system, wherein P is the camera according to the Wherein the projection matrix is a projection matrix of the pedestrian.
A candidate generator for extracting a pedestrian candidate object from an input image;
An ROPP calculation unit for calculating a region of prediction path (ROPP), which is an area where a future route of the vehicle can exist through statistical analysis of the traveling data;
An ROD calculation unit for calculating an ROD (Region of Danger) by extending the left and right outer regions of the ROPP according to predetermined criteria; And
A pedestrian candidate selecting unit for selecting only the pedestrian candidates located in the ROD among the pedestrian candidate objects extracted through the candidate generating unit and transmitting the pedestrian candidates to the candidate verifying unit; And
And a candidate verifying unit for performing a candidate verification for each of the filtered pedestrian candidate objects to detect a pedestrian in a dangerous state,
The ROPP calculation unit may calculate,
Obtaining a speed and an angular velocity of the vehicle included in the running data,
Calculating a predicted path endpoint for the predicted time using the velocity and acceleration, calculating a phase angle of the predicted path endpoint,
Selects a cell to which the velocity and the angular velocity belong from a look-up table,
Calculating a maximum direction error and a minimum direction error with respect to a phase angle of the estimated path end point by using an average and a standard deviation of the read direction error,
Calculating a range of the phase angle of the ROPP using the calculated maximum direction error and the minimum direction error,
Reads the mean and standard deviation of the acceleration from the selected cells, calculates a maximum reach distance using the average and standard deviation of the read acceleration,
Calculating the ROPP defined by the range of the phase angle and the maximum reach distance,
Wherein the lookup table includes a plurality of cells defined according to a speed and an angular rate of the vehicle and calculates an acceleration and a direction error using a speed and an angular velocity assigned to each cell for each cell, Wherein the mean and standard deviation of the standard deviation and the calculated direction error are calculated and recorded.
delete

Images