KR102283053B1 - Real-Time Multi-Class Multi-Object Tracking Method Using Image Based Object Detection Information - Google Patents

Abstract

The present invention relates to a real-time multi-class multi-object tracking method using image-based object detection information, a first step of reading detection data from image-based object detection information, and a first step of performing location-based filtering on the read detection data Step 2, a third step of performing matching and Kalman filter processing on the detected data filtered in the second step, a fourth step of performing an initial object filtering process after the third step, and the data after the fourth step a fifth step of performing processing and management, wherein the first to fifth steps are sequentially performed for each frame, and the third step proceeds up to 5 frames and matches the same object 4 frames within 5 frames The candidate data set is stored and updated, and the candidate data set is used in the matching process in the matching and Kalman filter processing steps.

Description

Real-Time Multi-Class Multi-Object Tracking Method Using Image Based Object Detection Information

The present invention relates to a real-time multi-class multi-object tracking method using image-based object detection information, and more particularly, by utilizing a general imaging device attached to a vehicle, detection data through an image stored through a camera while driving , and develop a tracker (hereinafter referred to as a tracking algorithm) to correct misrecognized or unrecognized parts through the obtained detection data in various steps in a step-by-step manner to detect major objects in real time. It relates to a real-time multi-class multi-object tracking method using a new type of image-based object detection information that can be detected and tracked.

Conventionally, since the object detection performance is not perfect, problems such as unrecognized, misrecognized, and incorrect location occur.

In addition, if only detection information is used, there is a difficulty in actual use in the application because there is no data or the use of wrongly recognized data.

In addition, there are limitations in algorithm development because limited computing power and resources must be used in an embedded environment.

In addition, in general, a detector (hereinafter referred to as a detection algorithm) requires more computing power than a tracker (tracking algorithm), and since it detects multiple classes or multiple objects in the entire image, computing There is a problem that consumes a lot of power.

[Prior art literature]

Republic of Korea Patent Publication No. 10-2016-0110773 (published on September 22, 2016) (Title of the invention: Method and apparatus for tracking a moving object using a Kalman filter model)

An object of the present invention is to develop a tracking algorithm (Tracker) that is proposed in view of the above problems, operates in real-time while using minimal resources in a conventional environment, and can cope with unrecognized and misrecognized It is to provide a real-time multi-class multi-object tracking method using image-based object detection information that can detect and track major objects in real time.

According to an embodiment of the present invention for achieving the above object, a real-time multi-class multi-object tracking method using image-based object detection information is a real-time multi-class multi-object tracking method using image-based object detection information,

A first step of reading detection data from image-based object detection information;

A second step of performing location-based filtering on the read detection data;

A third step of performing matching and Kalman filter processing on the detected data filtered in the second step;

a fourth step of performing initial object filtering processing after the third step;

Including a fifth step of performing data processing and management after the fourth step,

The first to fifth steps are sequentially performed for each frame,

In the third step, if the same object is matched with 4 frames within 5 frames, it is stored and updated as a candidate data set, and the candidate data set is used in the matching process in the matching and Kalman filter processing steps. but,

delete

The third step is

Initiating the Kalman Filter prediction process,

matching the filtering data and the tracking data set;

If matched, performing a Kalman filter correction process, and

Comprising the step of updating the tracking data when the correction is completed,

delete

The fourth step is

Matching the remaining filtered data with the candidate data set,

updating candidate data if matched;

After checking the candidate dataset, passing it to the tracking data,

It characterized in that it comprises the step of adding the remaining filtering data to the candidate dataset,
It reduces misrecognition through initial object filtering, and then goes through the Kalman filtering process to reduce unrecognized recognition to improve the recognition rate that can recognize multiple objects in real time.
An image-based object characterized in that it is possible to process detected data in real time by using only a class, score, and detection result box, which are information that can be obtained from detection data rather than image-based, using a very small amount of computation compared to image processing-based data processing. A real-time multi-class multi-object tracking method using detection information is provided.

Also, preferably,

The step of matching the filtering data and the tracking data set is

Nearest Neighbor (NN), heuristic similarity matching, and Multi Class Score Calculation methods are sequentially performed.

As described above, according to the real-time multi-class multi-object tracking method using image-based object detection information according to the present invention, misrecognition is reduced through initial object filtering, and then multiple objects are recognized in real time by reducing unrecognized through Kalman filtering process. It has the effect of improving the recognition rate that can be done.

According to the present invention, it is possible to process detected data in real time using a very small amount of computation compared to image processing-based data processing using only the class, score, and detection result box, which are information that can be obtained from detection data, not image-based. .

1 is a flowchart illustrating a real-time multi-class multi-object tracking method using image-based object detection information according to the present invention.
2 shows an example of a camera image screen for explaining the present invention.
3 shows an expression expressing the Kalman algorithm used in the present invention.
4 shows an example of a result screen of a real-time multi-class multi-object tracking method using image-based object detection information according to the present invention.
5 to 7 show examples of a result screen for changing a location-based ROI in the real-time multi-class multi-object tracking method using image-based object detection information according to the present invention.
8 shows an example of a result screen for changing a region of interest according to an object size in the real-time multi-class multi-object tracking method using image-based object detection information according to the present invention.
9 to 12 show examples of result screens in each frame of a detected image frame and a corrected image frame in the real-time multi-class multi-object tracking method using image-based object detection information according to the present invention.

Since the present invention can have various changes and can have various embodiments, specific embodiments are illustrated in the drawings and described in detail.

However, this is not intended to limit the present invention to specific embodiments, and it should be understood to include all modifications, equivalents and substitutes included in the spirit and scope of the present invention.

The terms used in the present application are only used to describe specific embodiments, and are not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly dictates otherwise. In the present application, terms such as “comprise” or “have” are intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, but one or more other features It should be understood that this does not preclude the existence or addition of numbers, steps, operations, components, parts, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical and scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present application. does not

Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the accompanying drawings. In describing the present invention, in order to facilitate the overall understanding, the same reference numerals are used for the same components in the drawings, and duplicate descriptions of the same components are omitted.

Hereinafter, a real-time multi-class multi-object tracking method using image-based object detection information according to the present invention will be described in detail with reference to the accompanying drawings.

A brief look at the features of the present invention.

First, the information received from the detection algorithm (Detector) is top class information, top score (confidence, 0~1 value), bounding box information (cx,cy,w,h), score information for other classes, and the above detection (Detection) Use only information (no image processing).

Table 1 is data obtained through detection data, which is not included in the scope of the present invention, but is intended to be used in the present invention. In the present invention, the process of acquiring detected data is not included in the scope of rights.

Referring to Table 1, object data that can be classified is used in the detection data.

In addition, the present invention is a non-image processing method, and the amount of computation is small because the calculation is performed only with the value of the bounding box of the object. (The image processing method is generally Therefore, there is a problem that the amount of computation is high). For this reason, in the present invention, it is possible to process related data using a real-time algorithm.

Features of the present invention, multi-class (Multi Class), multi-object tracking (Multi Object tracking), Kalman filter estimation (Kalman Filter Estimation) - filtering based tracking (Filtering based tracking), matching method (Matching method) is Nearest Neighbor and Heuristic method is used, camera calibration and 2D to 3D transformation are performed, and object information extraction and processing for application are performed.

Hereinafter, before explaining the initial filtering process and the Kalman filtering process in FIG. 1, assuming that one image processing screen frame is input every 0.3 seconds, it is performed in the process of FIG. 1 for each successive frame, from the beginning of the Kalman filtering process. Since the filtering process is not performed, the initial filtering is performed first after the location filtering process. If 4 out of 5 frames are matched in the initial filtering process, a candidate data set is created, and then the Kalman filter process is performed. After that, frame input is continuously performed, and thereafter, the processing of the detection data set and the Kalman filter result is continuously performed for each frame, and the corrected data as useful data is updated and managed.

Meanwhile, the Nearest Neighbor and Heuristic method and the Multi Class Score Calculation process described below represent the process of step S5, which is a Kalman filter matching process. In the Kalman filter matching process, the prediction process is performed using NN, heuristic method, and multi-class score calculation method step by step (step 3). In the present invention, when matching is completed in one frame set of the left and right screens in the tracking data set (see FIGS. 11 and 12), it can be displayed as correction, and when matching is not completed, it is not updated with the prediction process will remain

Meanwhile, since data cannot be stored indefinitely in the process of predicting and correcting each frame set, only a certain size is continuously stored and deleted repeatedly.

In addition, in the present invention, in the initial object filtering (steps S9 to S12) and Kalman filtering (S4 to S7) processes, the initial object filtering process is performed first, and when a candidate dataset is determined, the Kalman filtering process is performed. At this time, the candidate data Only index information is stored for the set and tracking data set, and in the case of initial object filtering and Kalman filtering, access to the storage device (not shown) of the central processing unit (not shown) through the corresponding index to read each data and perform the corresponding processing repeat to do

Hereinafter, NN and a heuristic method will be described.

Description of Nearest Neighbor and Heuristic method

[Nearest Neighbor] (hereinafter referred to as NN)

It is an algorithm that finds the other closest distance from a given object. Instead of finding the closest object, the searching space was variably adjusted according to the size and class of a given object. Filtering was performed based on the size and variable distance for each object (coordinate distance within the image).

It refers to “finding another object that has the closest distance to one object in a set”. It is a general method used in the image processing field.

When the time interval between image frames is short, the object is more likely to be nearby. Therefore, close objects are more likely to be the same object than distant objects. For this reason, the Euclidean distance root (square of the difference in x value + the square of the difference in y value) is used as a condition in the x,y two-dimensional coordinate system in the image.

Objects of the same class are generally closer as their size increases in the image. In the 3D real world, objects close to the camera appear larger in the image. And as the distance increases, even if the same amount is moved, the amount of movement in the image is generally proportional to the size. Therefore, the search area proportional to the size of the object is limited, and matching described below is used only within this area. This is a method to save the amount of computation that does not search the entire area.

Also, although it is an algorithm that finds the closest one, it is used to filter out objects that are more than a certain distance because matching only based on a close criterion results in erratic results.

[Heuristic Method]

The heuristic method is a method of finding a good solution empirically because it is a heuristic method. It is different from the method designed based on the theory, so it is a method to find a solution that is satisfactory to some extent rather than finding the optimal solution. An arbitrary method used based on human experience rather than a design and theoretical method is called a heuristic.

For example, the Kalman filter is not a heuristic method. It has been proven that the Kalman filter has the best optimal performance for a linear system and a model with Gaussian noise.

The matching through the heuristic method mentioned here refers to the class matching and the heuristic similarity method. Referring to FIG. 1, steps S5 and S9 in FIG. 1 are shown, and steps S5 and S9 include Class matching, Nearest Neighbor (NN), and heuristic similarity matching are all included.

In the present invention, the camera is attached to the front of the vehicle, and main multi-class object information after detecting the current frame time t from this image is stored in the memory.

Referring to FIG. 2 , objects detected through a detection process are displayed on a screen.

At this time, the recognition process for each object is varied and continues for each frame step by step through an algorithm. As the data used in the object recognition process, various types of data other than image data are used. For example,

Object 1's class, score, bounding box(cx, cy, w, h), and score values for other classes.

Object 2's class, score, bounding box(cx, cy, w, h), and score values for other classes.

Object 3's class, score, bounding box(cx, cy, w, h), and score values for other classes.

this can be used Although the class will be described later, the data obtained in the detection process (see Table 1) is used as it is classified according to the size and type of the object.

According to the present invention, on the assumption that the camera is fixed to the vehicle, the positions where objects can appear in the image are limited. Based on this, objects out of the area are removed and stored in a data set.

The variables necessary for Kalman filter calculation are obtained using the location history values of objects currently in tracking (the process of processing the detected data later is called tracking. Hereinafter also referred to as tracking data). R). Then, the object being tracked and the detection values entered into the data set are matched through the Nearest Neighbor and the heuristic method. To calculate the correction of the Kalman filter with data matched with the object being tracked, z (observed value) is obtained, and the correction of the Kalman filter is performed. At this time, prediction According to the ratio of covariance of and z, the value of the tracking set is updated with a correction value.

In matching with tracking, data sets that are not matched are used for candidate group matching (Initial Object Filter). In candidate group matching, like tracking set matching, it is determined whether the same object continuously appears within a few frames (3 to 5 frames) before transferring to tracking. When a few frames (3 to 5 frames) are filled, it is judged as a candidate check to determine whether to newly add or delete a tracking set. This serves to filter out intermittent misrecognition that does not result in continuous detection.

Finally, the data of the tracking set updated by correction is processed and extracted for use in the application. And if there is an application, you can use it in it. For example, it is possible to create a collision time (Time to Collision) using the position and speed so that the driver can be notified by calculating the probability and speed of a collision with the vehicle in front. Alternatively, meaningful information may be transmitted to the driver by using the distance from the surrounding object and information on the objects, thereby recognizing a route guidance or a dangerous caution situation.

Kalman Filter

When there is no matching object due to unrecognized detection in Detection, if only the detection value is used in the application, it may be difficult to use in the application because there is no data. So, using a Kalman filter, a prediction is made with a prediction, and it is continued through appropriate motion modeling and operation. And since this value can be used in the application, the problem caused by unrecognized can be solved.

And, in the case of object matching, the currently predicted data and variance can be used by updating the new estimate and variance using the observed (matching) value and variance. there is.

The main purpose is to obtain an optimal state estimate in the shaky detection data and to cope with the unknown.

The Kalman filter is a recursive filter that estimates a linear state based on measurements (observed values) that contain errors (noise).

The Kalman filter is displayed in the form of FIG. 3 . Hereinafter, FIG. 3 will be described.

In Fig. 3, the left side is called prediction and the right side process is called correction.

The current location is predicted using the previously accumulated past tracking data, and if there is matching observation data among the new detection data, the ratio of the variance between the predicted data and the observed data is used and the ratio of the variance between the two This will determine the final state.

<Prediction>

(1) Prediction of state at time k

는 상태변수를 말하며 시간 k일 때의 물체의 bounding box의 값을 상태로써 가진다.

is a state variable and has the value of the object's bounding box at time k as a state.

가 가지는 cx, cy, w, h는 각각이 독립이어서 A는 4x4 단위행렬이다.A is the state transition matrix. state variable

Each of cx, cy, w, and h of is independent, so that A is a 4x4 identity matrix.

,u_k가 같은 원소값을 가지므로 B는 4x4 단위행렬이다. u _k refers to user input, but there is no input value for where the objects appearing in the image will move. Therefore, by calculating the value u _k from the previous trace data it must put. It is the amount of movement of an object in the image in unit time t (time interval between frames). B is the state transition matrix for user input.

Since ,u _k have the same element value, B is a 4x4 identity matrix.

(2) Prediction of covariance at time k

는 시간 t가 k일 시점의 공분산 행렬이다. 4x4 단위행렬에 일정 수가 곱해져 초기값으로 정해진 후, 직접 접근하여 변경하는 일 없이 예측과 보정과정을 통해 자동으로 업데이트 된다. n은 100으로 주었다. Qk는 예측에 의한 공분산 오차 공분산 행렬이다. 이미지 내에서 물체의 검출데이터가 일정하지 않고 값이 흔들리는 상태로 매칭되었을 때에는 uk의 원소값들의 변화가 크다. 이러한 점을 들어 Q를 다음과 같은 형태로 정의하여 움직임이 많은 물체가 큰 공분산행렬을 갖도록 하였다. 그리고 상태변수

의 원소들에 해당하는 Q값은 물체의 크기가 클수록 이미지상에서의 움직임이 크게크게 나타나기 때문에 다음과 같이 물체의 크기에 비례하도록 하였다. 원소에 곱해진 0.1, 0.2과 같은 상수들은 임의의 값으로 상황에 따라 바뀔 수 있다. 그리고 Q식에서 루트 안의 값이 음수를 가진다면

으로 계산한다.

is the covariance matrix at time t k. After the 4x4 identity matrix is multiplied by a certain number and set as an initial value, it is automatically updated through prediction and correction without direct access and change. n was given as 100. Qk is the covariance error covariance matrix by prediction. When the detection data of an object in the image is not constant and the value is matched in a shaking state, the change in the element values of uk is large. For this point, Q was defined in the following form so that a moving object has a large covariance matrix. and state variables

The Q value corresponding to the elements of is proportional to the size of the object as follows, because the larger the object, the greater the movement on the image. Constants such as 0.1 and 0.2 multiplied by elements are arbitrary values and can be changed according to circumstances. And in the Q expression, if the value in the root is negative,

calculated as

,

는 다음과 수학식 6 및 수학식 7과 같이 정리된다.According to the above,

,

is arranged as in Equations (6) and (7) as follows.

<Correction>

Kk represents the Kalman gain. It is calculated as the ratio of variance to which of the state predicted value or the observed value to believe more.

가 가진 형태와 같다. H행렬은 상태

를 관측값 Zk 의 형태로 변환시켜준다. 즉 상태값에서 관측값을 도출한다. 하지만 상태와 관측값의 형태가 같기 때문에 H는 4x4 단위행렬을 가진다.Zk represents the observation at time k. The shape of the observation is the state

has the same form as H matrix is state

is converted into the form of the observation value Zk. That is, the observed value is derived from the state value. However, since the state and observation values are the same, H has a 4x4 identity matrix.

(1) Kalman Gain Calculation

The Kalman gain is calculated by the above equation, and due to the above, the equation is arranged as shown in Equation 10 below.

R is the observation measurement noise covariance matrix. In fact, this value depends on the performance of the detector. If the performance of the detector is good, the value of R is small, and if it is accurate, there is no need for R and there is no need for a Kalman filter. However, since there is measurement noise in the detection result with error, this value must be entered. Since the error increases as the size of the object increases, this point was used to define R.

(2) Update state estimate using observations

값의 차이를 칼만게인 Kk만큼 비중을 두어 상태

를 보정한다.State prediction at observation z _{k and time k}

State by weighting the difference in values as much as the Kalmangain Kk

to correct

식은 위의 내용으로 다음과 같이 정리된다.

The above expression is summarized as follows.

(3) Covariance update

After the state is updated, the state's covariance matrix is updated with the Kalman gain. I is a 4x4 identity matrix.

식은 위의 내용으로 다음과 같이 정리된다.

The above expression is summarized as follows.

The flowchart of FIG. 1 will be sequentially described.

만 하여 추가로 넣어준다. 매칭된 후 남은 데이터들은 새로운 후보 데이터의 후보군으로 추가한다(S12). S13에서는 전체적인 데이터 관리 및 추가 삭제(메모리 관리), 다중 물체 점수 계산, 어플리케이션에서 사용하기 위한 정보를 추출하는 부분을 나타낸다. 데이터 관리에는 추적 구간 데이터 중 오래된 것을 삭제하는 것과, 다중 물체 점수 계산과정에서 물체의 점수가 일정점수 이하로 떨어지는 경우(매칭이 안되는 것으로 인한)에 해당 데이터를 삭제하여 메모리 관리를 한다. 그리고 어플리케이션에서 사용하기 위한 데이터 가공부분이 해당한다.When the tracker is executed after the detector completes the detection, the tracker calls the detection data from the predetermined point. Based on the front camera mounted on the vehicle in motion, the positions of the objects appearing on the screen are determined. Based on this, data suspected of being misrecognized in the detected data is not used. In S3 of S4, the state (position and size) of the object is predicted through the Kalman filter from the previously accumulated tracking data. For matching between the predicted data and the detected data in S5, it is primarily filtered using class information. Then, it is filtered secondarily using the center point distance of the object on the image. In this case, the search area is limited to be proportional to the size of the object. Then, an evaluation function is created based on the size and aspect ratio of the object to find the matching partner with the highest score. The matched tracking data is updated to a more accurate position by updating the state x and state covariance P through the Kalman filter correction of S6. The tracking data that fails to match is updated with the predicted state x and the predicted state covariance P calculated, so that the location is more inaccurate than before. The initial object filtering of the S8 is a device to filter out misrecognition like location-based filtering, and it plays a role in filtering intermittent misrecognition so that it is not tracked. And, if n frames out of m frames are actually matched through the inspection of candidate data accumulated over a certain number, this data is transferred as new data so that it can be used in S3. After matching with the tracking data, the remaining detection data is matched with the candidate data accumulated in the initial object filtering of S8 (S9). In S10, the same method as the matching method used in S5 is used without using the Kalman filter. Candidate data matching the detection data is updated by adding additional detection data. Candidate data that does not match the detected data is state prediction like S4 prediction.

Just add more. The remaining data after matching is added as a candidate group of new candidate data (S12). In S13, the overall data management and addition and deletion (memory management), multi-object score calculation, and information extraction for use in applications are shown. In data management, the old data of the tracking section is deleted, and when the score of an object falls below a certain point in the multi-object score calculation process (due to inability to match), the data is deleted and memory is managed. And the data processing part for use in the application corresponds.

Description of the algorithm used

It shows location-based filtering of S2.

Objects that can appear while driving a car use the approximate location of the image and reduce misrecognition.

Modeling and using S4 and S6 Kalman filters in step S3

An optimal solution is estimated from the detection result including the error.

It copes with the unknown with the prediction of the Kalman filter.

Matching method for S5 and S10

The amount of computation is reduced by subtracting the search area considering the characteristics of the object appearing in the image.

Match using object class, location, size, and aspect ratio (Nearest Neighbor to location, class, size and aspect ratio heuristic method).

Initial object filtering in S8

A way to deal with misconceptions. A simple and efficient method.

Using the y-value coordinates among the coordinates of the center point of an object, if it is out of the range set for each class, it is filtered.

In S8, candidate data with a certain cumulative abnormality is added as new data in the tracking dataset through inspection.

Let's look at the features of the present invention again.

It is a general-purpose multi-object tracker that can be used in real time even in an embedded environment.

Most of the information detected by the image-based object detector uses only the class, score, and detection result box it has.

It has the versatility to work most of the time even if a tracker is attached behind many other detectors.

Using only class, score, and bounding box information, it has a very small amount of computation compared to image processing-based algorithms.

Location-based filtering of step S2 in consideration of the location characteristics of objects appearing in a general camera attached to a vehicle

A method of heuristic matching of objects between frames for real-time use in the tracker (see steps S5, S9).

The process performed in the present invention will be described once again.

The data obtained from Detection is the value of the bounding box. Let the position value be the state, and let u be the velocity change.

A and B are 4 by 4 identity matrices. On the image, cx, cy, w, and h are independent relationships that do not affect each other.

x =[ cx ; cy; w ; h ] (state)

z =[ x' ; y' ; w' ; h’ ] (variation)

P represents a Gaussian error generated during prediction as a covariance matrix.

H is a matrix that allows the state x value to be expressed in the form of z, but since x and z have the same form, it is a 4 by 4 unit matrix.

K is the Kalman gain, and it is determined as the ratio between the current covariance P due to prediction and the covariance R determined at the time of observation. It has a value between 0 and 1.

In correction, the update of the estimate of the current state = prediction + K (observed value - prediction).

Then, the P value is updated using the K value.

Repeat the above process.

The basic characteristics of an object that may appear while driving a vehicle will be described as follows.

Objects (crosswalks, road markers, etc.) in the form of floor markers change abruptly in y.

For objects attached to the ground, such as cars, people, and bicycles, the movement of the x-axis and y-axis increases as the distance increases, and decreases as the distance increases. And the movement to the y-axis is a 2D (two-dimensional) projection of an object in the real world onto an image, and the data on the y-axis is compressed, so the amount of movement in the y-axis as much as the actual movement. It doesn't appear that big. The x-axis is displayed as much as it moves.

As the distance increases, all objects appear less motion on the image.

When the size of an object decreases during prediction, the u-value also decreases (generally), and the larger the object, the greater the u-value. And the amount of increase in covariance is also proportional to this.

Based on the above, u, Q, and R values were modeled, and the characteristics are as follows.

For objects attached to the ground, such as cars, people, bicycles, and motorcycles, the x of u is maintained at the speed between frames, and the y value of u is smoothed using the average value because the amount of movement is small. . W and h are the same as y.

For objects such as floor markers and traffic lights appearing in the air, the amount of change in y is large, so x and y are maintained at the speed between frames, and smoothing is applied to w and h.

Q is the cumulative amount of error covariance due to prediction, which increases when moving over the image, so it was set as the size of the object/n + |u value/m|, and Q(1) by the square root of (w * h) ), Q(4), Q(11), Q(14).

R is set to be proportional to the size of the object.

Prediction limit

An aspect ratio that an object can have in an image plane is determined to some extent. (Traffic lights are long horizontally, pedestrians are tall. Cars can be both horizontal and vertical, but cannot be too thin. Road traffic signs have an aspect ratio close to a square, etc.).

In object prediction, since past information is used, in general, decreasing ones continue to decrease and increasing ones continuously increase. For example, when a person walks, the bounding box approaches a square when the legs are spread apart, but the proportions become thinner when standing sideways. So, in general, when prediction is made, it becomes thinner and the matching fails. To prevent this, the minimum and maximum aspect ratio is determined for each object class. This is to prevent the matching failure due to the aspect ratio being misaligned during prediction.

Matching Algorithm

Images are input in units of frames. Except in special circumstances, the amount of movement of an object within the image between frames is small. In other words, “An object in the previous frame will be located around it in the next frame”, “The size of the object will also be similar”, “Even if misrecognition occurs, it is highly likely to occur within the same broad category”. That is, objects with the closest object within a certain distance, similar size and shape, and classes belonging to the same class or large category are highly likely to be the same object.

Class Matching

The main classes are listed and grouped by characteristics as follows.

Group A: Car, Bus, Truck, Van (VAN)

Group B: Two-wheeled vehicles (bicycles, motorcycles), people

Group C: Types of signs

Group D: Types of floor markers, crosswalks

Other: the rest

Classification as above will be referred to as a major classification. Groups with similar position and dynamic characteristics within the image. When the detection stage recognizes the car or truck of group A, the score may be similar even in the same area. Therefore, the classes corresponding to the large classification were allowed to match. When matching, it was filtered based on whether it falls within the broad category.

Heuristic Similarity method

From Detection, we get class, score, bounding box points, and class's scores. To match based on this, we try to compare the size and aspect ratio of the object with the above large classification class, NN (Nearest Neighbor). called similarity. Define a cost function that calculates the similarity between area and aspect ratio as a ratio, and it matches the highest object.

In the heuristic similarity method,

A class is matched with the NN-filtered detection data, and the size and aspect ratio are compared.

First, size comparison will be described as follows.

Find the area of the bounding box (bbox) of the detection data and the area of the bounding box (bbox) of the tracking data set or candidate data set.

Assuming D_area (the area of one object in the detection data) and M_area (the area of one object in the data to be matched),

Sim_area = D_area/M_area (area similarity),

If sim_area is greater than 1, the reciprocal of sim_area is stored, if the area is the same, it is 1, and as the area differs, it is closer to 0.

In addition, when comparing the aspect ratio, the aspect ratio is the vertical/horizontal value, called D_ratio and M_ratio. Also, sim_ratio = D_ratio/M_ratio (aspect ratio similarity). If Sim_ratio is greater than 1, the reciprocal of sim_ratio is stored, and as above, it is close to 1, and the difference is closer to 0.

Also, if the process of bundling into one cost function is described, it is called gain_area and gain_ratio, both of which are positive numbers, and the sum of the two is 1.

By default, both have a value of 0.5. However, if the object aspect ratio is constant, increase the aspect ratio score ratio.

Therefore, result = gain_area x sim_area + gain_ratio x sim_ratio,

Here, result has a value between 0 and 1, sets a minimum threshold value, exceeds this value, and stores the one with the highest result as a matching result.

Multi Class Score Calculation

Describes the update of the score of an object in the tracking set matched by the matching method described above.

In the above formula, S _{t- 1} is the score value for all classes of the tracking set, and Ot represents the score value for all classes of the matched object. And the St value obtained by η(St) is in a very small state. Because the sum of all score values of one object is 1, the score value for each class has a value between 0 and 1. So, when real numbers less than 1 are multiplied by element, the number becomes very small. η is a representation of the normalize function that makes the sum of Sts equal to 1.

And the problem with the above method is that if tracking is continued, the top score value converges to 1, so that the top is maintained even if there is a match within the same large category. In order to solve this problem, opportunity points were given to class scores belonging to the same major category. Then, even if tracking continues, there is room for transition to another large classification class. And giving opportunity points to money care (“don't care”) (where the object does not correspond to all classes and is nothing, i.e. the background) is added as an element to the class score calculation above, Calculated. This is so that the score of the top class naturally decreases at a certain rate. (Later, there is a section to remove the tracking object that has fallen below a certain score.).

Object Filtering

In the problem of non-recognition and misrecognition in detection, the Kalman filter solves the unrecognized part, and from incomplete detection information (detection information including error) optimal If it is used for state estimation, the method to be described now is a part to deal with misrecognition.

Location Filtering

If the car is equipped with a camera and fixed, and if the camera has been calibrated, it is possible to filter the objects that appear in the driving situation by location. For example, a person cannot fly in the sky and a traffic light cannot appear on the ground. If you're using it on a handheld camera, this Location Filter won't work.

Initial Object Filtering

If it is said that the wrong data according to the location has been filtered out above, this time it is to remove the misrecognition from the limited and simple detection information. Initially, 3 to 5 frames are inspected, stored in a candidate set, and matched, and data matched with actual data during a predetermined reference frame are newly added to the tracking set.

This method is easy to filter out the false recognition that appears blinking, but it has a disadvantage that it cannot filter out the false recognition that continuously enters a similar location, similar size, similar ratio, and class belonging to a large category.

It is used in the same way as used in Tracking Set matching, but without Kalman filter applied.

Briefly explaining candidate filtering, if an object actually passed the above conditions 4 times during 5 frames and was matched, this data is delivered to the tracking set.

result

Referring to FIG. 4 , the colored box represents the result after final tracking, the green ellipse represents the prediction variance (prediction process for Kalman filter matching), and the white ellipse represents the observation variance (class data bounding box obtained from the detection data). The red ellipse indicates the correction variance (corrected state after matching with the Kalman filter applied).

Referring to FIG. 5 , as a screen displayed on the front of the vehicle, a region of interest (ROI) is displayed as a red box on the front screen image that can be viewed through the camera.

FIG. 6 is a view similar to FIG. 5 and is a screen in which the region of interest (ROI) is displayed upward, and FIG. 7 is a view similar to FIG. 5, in which the region of interest (ROI) is displayed downward.

5 to 7 show the S1 location-based filtering process in FIG. 1 .

8 is a screen showing the front of the vehicle, a vehicle is displayed at a position called a vanishing point, a bounding box (bbox) is displayed on the vehicle, and a region of interest is displayed as a rectangular area around it to perform NN processing. is a diagram showing

9 and 10 are views showing a detection data image and a Kalman filter matching image frame by frame on the left and right sides.

9 and 10 , objects classified into various classes are detected in a bounding box (bbox) in the left frame. 9 and 10 are initial object filtering for removing misrecognition. 5 frames are displayed from top to bottom in FIG. 9 . In each frame, the detection data is displayed on the left, and the Kalman filter prediction and correction screen is displayed on the right. Objects that appear continuously from frame 1 to frame 5 are set to be passed as tracking data if 4 frames are matched. It can be seen that, among the detected data in FIGS. 9 and 10, the crosswalk continuously appears, and the right central red signboard and the left central Homeplus billboard are intermittently misrecognized.

11 and 12 , the left side is the detection data, and the right side is the final tracking data result.

At the same time, the position is corrected through the Kalman filter, and if the detection algorithm does not recognize an object, the prediction part of the Kalman filter, a tracking algorithm, predicts the current expected position based on previous data and uses it in the application stage even if unrecognized occurs. Fill in the blanks so that there is no problem.

When it becomes difficult to detect an object that is too far away, as it travels, a distant traffic light is continuously detected, passes through the initial object filtering, and appears in the final tracking result through the final Kalman filter.

Referring to FIGS. 11 and 12 , a screen of a total of 10 frames is displayed, and in the right tracking result data, correction and prediction processes appear, and it can be seen that the red box is displayed.

As described above, according to the real-time multi-class multi-object tracking method using image-based object detection information according to the present invention, misrecognition is reduced through initial object filtering, and then multiple objects are recognized in real time by reducing unrecognized through Kalman filtering process. It will improve the recognition rate that can be done.

In addition, according to the present invention, it is possible to process detected data in real time using a very small amount of computation compared to image processing-based data processing using only the class, score, and detection result box, which are information that can be obtained from detection data, not image-based. .

As described above, the best embodiment has been disclosed in the drawings and the specification. Although specific terms are used herein, they are used only for the purpose of describing the present invention and are not used to limit the meaning or the scope of the present invention described in the claims. Therefore, it will be understood by those skilled in the art that various modifications and equivalent other embodiments are possible therefrom. Accordingly, the true technical protection scope of the present invention should be defined by the technical spirit of the appended claims.

Claims (4)

In the real-time multi-class multi-object tracking method using image-based object detection information,
A first step of reading detection data from image-based object detection information;
A second step of performing location-based filtering on the read detection data;
A third step of performing matching and Kalman filter processing on the detected data filtered in the second step;
a fourth step of performing initial object filtering processing after the third step;
Including a fifth step of performing data processing and management after the fourth step,
The first to fifth steps are sequentially performed for each frame,
In the third step, if 4 frames are matched with the same object within 5 frames, the third step is stored and updated as a candidate data set, and the candidate data set is used in the matching process in the matching and Kalman filter processing steps but,
The third step is
Initiating the Kalman Filter prediction process,
matching the filtering data and the tracking data set;
If matched, performing a Kalman filter correction process, and
Comprising the step of updating the tracking data when the correction is completed,
The fourth step is
Matching the remaining filtered data with the candidate data set,
updating candidate data if matched;
After checking the candidate dataset, the step of passing it to the tracking data;
It characterized in that it comprises the step of adding the remaining filtering data to the candidate dataset,
It reduces misrecognition through initial object filtering, and then goes through the Kalman filtering process to reduce unrecognized recognition to improve the recognition rate that can recognize multiple objects in real time.
An image-based object characterized in that it is possible to process detected data in real time by using only a class, score, and detection result box, which are information that can be obtained from detection data rather than image-based, using a very small amount of computation compared to image processing-based data processing. Real-time multi-class multi-object tracking method using detection information. delete delete The method of claim 1,
The step of matching the filtering data and the tracking data set is
Nearest Neighbor (NN), heuristic similarity matching, and Multi Class Score Calculation are sequentially performed in real-time using image-based object detection information. Class multi-object tracking method.

Images