KR20160132731A - Device and method for tracking pedestrian in thermal image using an online random fern learning - Google Patents

Abstract

According to an apparatus and a method suggested in the present invention for tracking a pedestrian in a thermal image using an online random ferns learning, a pedestrian is detected by using a random forest based on a Haar-like feature and a random forest based on an OCS-LBP feature, and the detected pedestrians can be tracked in real time by using an online random ferns learning and a particle filter based on the OCS-LBP feature and an LID feature, thereby tracking multiple pedestrians in real time.

Description

TECHNICAL FIELD [0001] The present invention relates to a device and a method for tracking a pedestrian using an online random fun learning in a thermal image,

The present invention relates to an apparatus and method for tracking a pedestrian by using online random fun learning in a thermal image, and more particularly, to an apparatus and a method for tracking a pedestrian using a random- And more particularly, to an apparatus and method for tracking a pedestrian using online random fun learning in a thermal image capable of tracking a pedestrian separately from a background and other pedestrians.

Detecting and tracking pedestrians is widely applied in many fields such as image synthesis, motion capture, security surveillance, and Human Computer Interaction (HCI). Particularly, as the demands for security and surveillance in real life have increased in recent years, pedestrian detection and tracking technology in video surveillance system has become increasingly important.

Until recently, many algorithms have been developed for pedestrian detection and tracking. Algorithms for detecting and tracking pedestrians include adaptive background generation, background subtraction, area - based pedestrian tracking, and shape - based pedestrian extraction. Also, if the pedestrian detection and tracking method is divided according to the number of pedestrians to be tracked, category-free tracking (CFT) is a method that excludes the background manually or automatically from the first frame, ≪ / RTI > Next, association-based tracking (ABT) is a fully automatic process associated with detection that responds to a specific trace. ABT-based detection approach based on ABT is becoming more and more popular in real life because ABT is more practical than CFT, but the problem of ABT-based tracking approach is that the lack of detection results and false detection results Can be seen.

On the other hand, pedestrian tracking using a general CCD camera (using visible light) is not effective in dark environments such as night. To overcome these limitations, methods have been studied for tracking pedestrians with infrared, near infrared and thermal infrared cameras. Cameras that use infrared or near infrared rays are cheaper than infrared infrared cameras, but people who are under or behind the light when tracking a pedestrian in infrared or near infrared rays have difficulties in tracking because they are not distinguished from the background.

On the other hand, in the case of a thermal image using a thermal infrared camera, robust detection can be performed irrespective of the lighting condition or the posture of the pedestrian, and the pedestrian can be traced not only indoors but also outdoors. However, it is costly and requires a lot of data for performance evaluation than a study on pedestrian tracking in a general CCD camera image. Also, as in the case of a conventional CCD camera image, tracking is performed in the case of occlusion, large shape change, This was difficult.

That is, the existing technology has a problem that it is difficult to trace an illumination condition, an obstruction, a large shape change, a non-linear motion, and a similar pedestrian when there is a pedestrian, and a database is required because of a lot of data for performance evaluation.

Korean Patent Laid-Open Publication No. 10-1350922 (Publication Date: Jan. 14, 2009) discloses a pedestrian tracking method and apparatus using a thermal imaging camera. The pedestrian tracking method includes: setting a temperature range of a pedestrian to be traced and a temperature range of a background; acquiring a first thermal image of a pedestrian to be traced based on a thermal imaging camera; The method comprising the steps of generating a second thermal image by separating the background from the first thermal image and tracking the tracked object based on the second thermal image, In the case of a pedestrian having a similar brightness to the pedestrian, or a pedestrian having a large shape change, as pointed out in the preceding problem, the problem of deteriorating the tracking performance is still not solved.

The present invention has been proposed in order to solve the above-mentioned problems of the previously proposed methods. In the image input from the thermal imaging camera, a pedestrian using Haar-like feature based random forest and OCS-LBP feature based random forest And detects the detected pedestrians using online random fun learning and particle filter based on OCS-LBP feature and LID feature and can track many pedestrians in real time. It is an object of the present invention to provide an apparatus and method for tracking a pedestrian.

In addition, the present invention can easily track a pedestrian by repeatedly estimating the posterior position distribution of the pedestrian through the particle filter, and also measure the observation likelihood for determining the particle weight, And an object of the present invention is to provide an apparatus and method for tracking a pedestrian using online random fun learning in a thermal image, which can increase the success rate of tracking using online random fun learning.

In addition, the present invention performs on-line learning on an initial fun using a booster random funn, and then determines whether the random fun for the tracer model is based on the positional distribution where the pedestrian of the fun selected in the subsequent frame is observed The present invention aims to provide an apparatus and a method for tracking a pedestrian by using online random fun learning in a thermal image, which can solve the point that a lot of time is required for learning by reducing the amount of necessary sample.

In addition, the present invention maintains the identity of the pedestrian by maintaining the identity of the pedestrian through the association test based on the calculated value obtained by combining the detected distance between the pedestrian and the tracer, the detected pedestrian's probability using the tracer's model, It is an object of the present invention to provide an apparatus and method for tracking a pedestrian by using online random fun learning in a thermal image which can increase the success rate.

Finally, the present invention applies a correlation checking algorithm and an online random fun learning based on pedestrian detection, so that it is possible to search for a new random pedestrian having not only overlapping or moving camera environments with pedestrians, And an object of the present invention is to provide an apparatus and a method for tracking a pedestrian using a fun learning.

According to an aspect of the present invention, there is provided a method for tracking a pedestrian using an online random fun learning method,

Detecting a pedestrian using a random forest based on Haar-like features and a random forest based on an OCS-LBP (oriented center-symmetric local binary pattern) feature in an input thermal image; And

And tracking the detected pedestrian in real time using online random ferns learning and particle filter based on OCS-LBP feature and LID (local intensity distribution) feature And the like.

Preferably, the pedestrian detection may be performed in the first frame of the column image, and the pedestrian detection and the pedestrian tracking may be simultaneously performed in the second frame.

More preferably, in the first frame,

In addition to detecting pedestrians, it is also possible to initialize the initial pedestrian state, extract the particle samples, learn the random fun using the detected pedestrian information, and assign the learned random fun to each tracker assigned to the detected pedestrian have.

Even more preferably,

Through the boosting algorithm, which evaluates the binary test and selects the optimal ferns from the candidate ferns of the window that has slid in the first frame before learning the random fun, And can learn the selected booster random fun.

Preferably, the pedestrian detecting step includes:

Sliding a window in a specific frame of an input thermal image;

Extracting a Haar-like feature from the sliding window;

Applying a random forest classifier based on the extracted Haar-like feature;

Extracting an oriented center-symmetric local binary pattern (OCS-LBP) feature in the sliding window;

Applying a random forest classifier based on the extracted OCS-LBP characteristics;

Applying a random forest classifier based on the Haar-like feature to set a candidate region, applying a random forest classifier based on the OCS-LBP feature to determine whether to detect a pedestrian for the sliding window; And

And detecting a pedestrian position of the specific frame using a pedestrian detection result of the sliding window.

Advantageously, the step of tracking the pedestrian comprises:

Filtering the particles of each frame including the area where the pedestrian is found, from the second frame of the input thermal image, the particle filter;

Estimating an observation likelihood of the filtered particles and classifying the filtered particles into particles having an upper weight; And

And estimating a state of the pedestrian using the classified particles.

More preferably, the step of estimating the observed likelihood of the particles and classifying the particles into particles having an upper weight value,

Applying an OCS-LBP feature and a LID feature extracted from the filtered particle to a boiled random-function classifier using each feature model, and estimating an observation likelihood;

Combining the estimated observation likelihoods to set weights of the particles; And

And updating the particles based on the weights of the set particles and classifying the particles having the higher weights.

Preferably, in the second frame,

Estimating the state of the final pedestrian by combining the detected information of the pedestrian and the tracker according to the result of the association test; And

And determining whether to update the random fun and re-extract the particles by applying an online random fun learning algorithm.

More preferably, the step of estimating the state of the final pedestrian by combining the detected information of the pedestrian and the tracker according to the result of the association test,

Calculating an estimated value by combining the distance between the pedestrian detected in the pedestrian detecting step and the tracer assigned to the detected pedestrian, the probability of the pedestrian detected using the tracer's model, and the overlapping rate;

Checking associations using a Hungarian algorithm based on the calculated estimates; And

And a step of estimating a final pedestrian state by combining the information of the pedestrian and the tracker detected according to the result of the association test.

More preferably, the step of determining whether to update the random fun and re-extract the particles by applying the online random fun learning algorithm comprises:

After estimating the observation likelihood of the final pedestrian using the above-mentioned booster random fun classifier, an online random fun learning algorithm using two maximum and minimum thresholds (U-max, U-min) is applied,

If the observed likelihood is higher than the maximum threshold value (U-max), determining a general situation without overlapping, updating the random fun and re-extracting the particles;

If the observed likelihood is lower than the minimum threshold value (U-min), determining that the image is completely overlapped and using the previous particle information again without updating the random funn; And

If the observed likelihood is between the maximum threshold value (U-max) and the minimum threshold value (U-min), judging the partial overlap or small state change and re-extracting the particles without updating the random fun can do.

According to an aspect of the present invention, there is provided an apparatus for tracking a pedestrian using an on-line random fun learning in a thermal image, the apparatus including a processor,

Detecting a pedestrian using a random forest based on a Haar-like feature and a random forest based on an OCS-LBP (oriented center-symmetric local binary pattern) feature in an input thermal image; And

And tracking the detected pedestrian in real time using online random ferns learning and particle filter based on OCS-LBP feature and LID (local intensity distribution) feature And the like.

Advantageously,

The pedestrian detection may be performed in the first frame of the column image, and the pedestrian detection and the pedestrian tracking may be performed simultaneously from the second frame.

More preferably, in the first frame of the column image,

In addition to detecting pedestrians, it is also possible to initialize the initial pedestrian state, extract the particle samples, learn the random fun using the detected pedestrian information, and assign the learned random fun to each tracker assigned to the detected pedestrian have.

Even more preferably,

Before the random fun is learned, a boosted random funn is selected through a boosting algorithm for evaluating a binary test and selecting optimal ferns among candidate ferns of a window sliding in a first frame And the selected booster random fun can be learned.

Preferably, the pedestrian detecting step includes:

Sliding a window in a specific frame of an input thermal image;

Extracting a Haar-like feature from the sliding window;

Applying a random forest classifier based on the extracted Haar-like feature;

Extracting an oriented center-symmetric local binary pattern (OCS-LBP) feature in the sliding window;

Applying a random forest classifier based on the extracted OCS-LBP characteristics;

Applying a random forest classifier based on the Haar-like feature to set a candidate region, applying a random forest classifier based on the OCS-LBP feature to determine whether to detect a pedestrian for the sliding window; And

And detecting a pedestrian position of the specific frame using a pedestrian detection result of the sliding window.

Advantageously, the step of tracking the pedestrian comprises:

(2-1) from the second frame of the inputted thermal image, the particle filter filters particles of each frame including an area where a pedestrian is found;

(2-2) estimating an observation likelihood of the filtered particles and classifying the particles into particles having an upper weight value; And

(2-3) Estimating the state of the pedestrian using the classified particles.

More preferably, the step of estimating the observed likelihood of the particles and classifying the particles into particles having an upper weight value,

Applying an OCS-LBP feature and a LID feature extracted from the filtered particle to a boiled random punster classifier using each feature model, estimating an observation likelihood, combining the estimated observation likelihoods, And classifying particles having higher weights after updating the particles based on the weights of the set particles.

Preferably, in the second frame,

Estimating the state of the final pedestrian by combining the detected information of the pedestrian and the tracker according to the result of the association test; And

And determining whether to update the random fun and re-extract the particles by applying an online random fun learning algorithm.

More preferably, the step of estimating the state of the final pedestrian by combining the detected information of the pedestrian and the tracker according to the result of the association test,

Calculating an estimated value by combining a distance between the detected pedestrian and a tracer assigned to the detected pedestrian, a probability of a detected pedestrian using a tracer's model, and an overlap ratio;

Checking associations using a Hungarian algorithm based on the calculated estimates; And

And a step of estimating a final pedestrian state by combining the information of the pedestrian and the tracker detected according to the result of the association test.

More preferably, the step of determining whether to update the random fun and re-extract the particles by applying the online random fun learning algorithm comprises:

After estimating the observation likelihood of the final pedestrian using the above-mentioned booster random fun classifier, an online random fun learning algorithm using two maximum and minimum thresholds (U-max, U-min) is applied,

If the observed likelihood is higher than the maximum threshold value (U-max), determining a general situation without overlapping, updating the random fun and re-extracting the particles;

Determining if the observed likelihood is less than a minimum threshold value (U-min), re-using the previous particle information without updating the random fun; And

If the observation likelihood is between a maximum threshold value (U-max) and a minimum threshold value (U-min), judging the partial overlap or small state change and re-extracting the particle without updating the random fun .

According to the apparatus and method for tracking a pedestrian using online random fun learning in a thermal image proposed in the present invention, a Haar-like feature-based random forest and an OCS-LBP feature-based random forest And the detected pedestrians can track many pedestrians in real time using online random fun learning and particle filter based on OCS-LBP feature and LID feature.

In addition, according to the present invention, it is possible to easily estimate the posture of a pedestrian by repeatedly estimating the posture distribution of the posture of the pedestrian through the particle filter, and to measure the observation likelihood for determining the particle weight. Instead, you can increase your success rate by using online random fun learning.

In addition, according to the present invention, an on-line learning is performed on an initial fun using a booster random funn, and then, a random fun for a tracer model calculates a position distribution on which a pedestrian of a fun selected in a subsequent frame is observed It is possible to solve the problem that the amount of sample required is reduced and the learning takes a long time.

Further, according to the present invention, by maintaining the identity of the pedestrian through association test based on a value calculated by combining the detected distance between the pedestrian and the tracer, the probability of the detected pedestrian using the tracer's model and the overlap ratio, And the tracking success rate can be increased.

Finally, according to the present invention, the relevance checking algorithm and the online random fun learning are applied based on the detection of the pedestrian, so that the tracking performance is excellent not only for the overlapping of the pedestrians or the moving camera environment but also for the newly appearing pedestrians.

1 is a functional block diagram of an apparatus and method for tracking a pedestrian using on-line random fun learning in a thermal image according to an embodiment of the present invention.
2 is a flowchart of a method of tracking a pedestrian using on-line random fun learning in a thermal image according to an embodiment of the present invention.
3 is a flowchart illustrating a method of tracking a pedestrian using online random fun learning in a thermal image according to an exemplary embodiment of the present invention.
FIG. 4 is a conceptual diagram of a step S100 in a method of tracking a pedestrian using on-line random fun learning in a thermal image according to an embodiment of the present invention.
FIG. 5 is a flowchart of a step S200 of a method for tracking a pedestrian using on-line random fun learning in a thermal image according to an embodiment of the present invention.
FIG. 6 is a conceptual diagram of a step S200 of a method for tracking a pedestrian using on-line random fun learning in a thermal image according to an embodiment of the present invention.
FIG. 7 is a flowchart illustrating a method of tracking a pedestrian using on-line random fun learning in a thermal image according to an embodiment of the present invention in step S220.
FIG. 8 is a conceptual diagram of a method for selecting a booster random funn in an apparatus and method for tracking a pedestrian using online random fun learning in a thermal image according to an embodiment of the present invention.
FIG. 9 is a flowchart of a step S300 of a method for tracking a pedestrian using on-line random fun learning in a thermal image according to an exemplary embodiment of the present invention.
FIG. 10 is a conceptual diagram of a step S300 of a method for tracking a pedestrian using online random fun learning in a thermal image according to an embodiment of the present invention.
11 is a flowchart of a step S400 of a method for tracking a pedestrian using on-line random fun learning in a thermal image according to an embodiment of the present invention.
FIG. 12 is a conceptual diagram of a step S400 of a method for tracking a pedestrian using on-line random fun learning in a thermal image according to an embodiment of the present invention.
FIG. 13 is an experimental result table for multiple-object tracking precision (MOTP) of an apparatus and method for tracking a pedestrian using on-line random fun learning in a thermal image according to an embodiment of the present invention.
FIG. 14 is an experimental result table for multiple objects tracking accuracy (MOTA) of an apparatus and method for tracking a pedestrian using on-line random fun learning in a thermal image according to an exemplary embodiment of the present invention.
15 is a diagram illustrating a result of tracking a pedestrian using an overlap ratio in an apparatus and method for tracking a pedestrian using on-line random fun learning in a thermal image according to an exemplary embodiment of the present invention.
16 is a diagram illustrating an image showing a result of tracking an apparatus and method for tracking a pedestrian using on-line random fun learning in a thermal image according to an embodiment of the present invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings, in order that those skilled in the art can easily carry out the present invention. In the following detailed description of the preferred embodiments of the present invention, a detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present invention rather unclear. In the drawings, like reference numerals are used throughout the drawings.

In addition, in the entire specification, when a part is referred to as being 'connected' to another part, it may be referred to as 'indirectly connected' not only with 'directly connected' . Also, to "include" an element means that it may include other elements, rather than excluding other elements, unless specifically stated otherwise.

1 is a functional block diagram of an apparatus and method for tracking a pedestrian using on-line random fun learning in a thermal image according to an embodiment of the present invention. As shown in FIG. 1, an apparatus and method for tracking a pedestrian using on-line random fun learning in a thermal image according to an exemplary embodiment of the present invention includes detecting a pedestrian in a first frame, And can be configured to simultaneously perform detection and pedestrian tracking.

2 is a flowchart of a method of tracking a pedestrian using on-line random fun learning in a thermal image according to an embodiment of the present invention. As shown in FIG. 2, a method for tracking a pedestrian using an online random fun learning method in a thermal image according to an exemplary embodiment of the present invention includes a random forest based on a Haar-like feature, (Step S100) of detecting a pedestrian using a random forest based on an OCS-LBP (oriented center-symmetric local binary pattern) feature, and detecting the detected pedestrian based on OCS-LBP feature and LID (S200) of real-time tracking using an online random fern learning and a particle filter. In addition, the step S300 of estimating the state of the final pedestrian by combining the detected information of the pedestrian and the tracker according to the result of the relevance test, and applying an online random fun learning algorithm to determine whether the random fun is updated and re- (S400) whether or not to determine whether or not to perform the operation.

Step S100 is a step of detecting a pedestrian in each frame of the input thermal image. FIG. 3 is a flowchart illustrating a method of tracking a pedestrian using an on-line random fun learning method in a thermal image according to an exemplary embodiment of the present invention. FIG. And a method of tracking a pedestrian using the learning is a conceptual diagram of step S100. 3 and 4, in the method of tracking a pedestrian using online random fun learning in a thermal image according to an exemplary embodiment of the present invention, a step S100 includes sliding a window in a specific frame of the input thermal image (S120) of extracting a Haar-like feature from a sliding window (S120), setting a candidate region by applying a random forest classifier based on the extracted Haar-like feature (S130) A step of extracting an OCS-oriented center-symmetric local binary pattern (OCS-LBP) feature (S140), a step of determining whether to detect a pedestrian for a sliding window by applying a random forest classifier based on the extracted OCS- S150), and detecting a pedestrian position for a specific frame using the pedestrian detection result for the sliding window (S160) have.

The purpose of pedestrian detection is to initialize the area where the pedestrian is detected in each frame, and to update the position and size of the pedestrian in the subsequent frame. In the preprocessing process, the Haar-like feature is used as a filter to remove tracking obstacles and shortens the time to classify the pedestrian to be tracked. The OCS-LBP feature is extracted from the candidate window and applied to a random forest classifier that separates candidate windows from pedestrian classes and background classes.

Step S200 is to track the detected pedestrian in real time using online random ferns learning and particle filter based on OCS-LBP feature and LID (local intensity distribution) feature to be. FIG. 5 is a flowchart of a step S200 of a method for tracking a pedestrian using on-line random fun learning in a thermal image according to an exemplary embodiment of the present invention. FIG. And a method of tracking a pedestrian using the learning is a conceptual diagram of step S200. 5 and 6, in an apparatus and method for tracking a pedestrian using on-line random fun learning in a thermal image according to an exemplary embodiment of the present invention, a step S200 includes the steps of repeatedly estimating a position distribution of a posterior of a pedestrian (S210) filtering the particles of each frame of the area where the pedestrian is found from the second frame of the inputted thermal image by using the particle filter capable of estimating the observation likelihood of the filtered particle, (S220) classification of the particles into particles having an upper weight, and estimating a state of the pedestrian using the classified particles (S230). Step S220 will be described in more detail below with reference to FIGS. 7 and 8. FIG.

Step S220 is a step of estimating an observation likelihood of the filtered particles and classifying them into particles having an upper weight value. FIG. 7 is a flowchart illustrating a method of tracking a pedestrian using on-line random fun learning in a thermal image according to an embodiment of the present invention in step S220. As shown in FIG. 7, in an apparatus and method for tracking a pedestrian using online random fun learning in a thermal image according to an exemplary embodiment of the present invention, step S220 includes an OCS (Step S222); estimating an observation likelihood by applying the LBP feature and the LID feature to a boiled random fun classifier using each feature model; setting a weight of each particle by combining the estimated observation likelihoods; And sorting the particles having the upper weights after updating the particles based on the weights of the set particles (S226).

In step S222, the observation likelihood is estimated for weight setting of the particles, and online random fun learning can be used instead of the general distance measurement. In this case, before the random fun is learned, a boosting algorithm is performed in which optimal ferns are selected by evaluating a binary test on candidate ferns of a window sliding in a first frame including a detected area of a pedestrian , A booster random fun is selected through the selection, and the selected booster random fun is learned. The reason for using the Boosted random fun is that random fern requires a large number of learning samples and not all fins are suitable for classification. The method of selecting the boosted random funn will be described in detail below with reference to FIG.

FIG. 8 is a conceptual diagram of a method for selecting a booster random funn in an apparatus and method for tracking a pedestrian using online random fun learning in a thermal image according to an embodiment of the present invention. As shown in FIG. 8, in the apparatus and method for tracking a pedestrian using online random fun learning in a thermal image according to an exemplary embodiment of the present invention, a boosting random funnel selection method uses a boosting algorithm. The boosting algorithm is described in detail in Algorithm 1 below.

The reason for choosing the Boosted random fun is that in step S222, an online random fun learning is used instead of the distance measurement to measure the observation likelihood for particle weight estimation, as alluded to above, This is because a large number of samples are required and it takes a long time to learn random fun. That is, it is aimed at shortening the time by performing online random fun learning using the optimal Funn Boosted random fun. Also, unlike the random forest and the adaboost algorithm, the random fun does not evaluate the binary test that makes up the fun. Therefore, classification performance of random fun depends on the size and quality of learning data. That is, in order to improve the classification performance of the random funn, it is necessary to increase the quality as well as the size, so it is necessary to select the optimum T fun in M candidate ferns. The process of selecting these booster random funns can be explained in Algorithm 1. The algorithm 1 will be described below.

[Algorithm 1]

1: T: Given weak classifiers.

S: (X _i , Y _i ) ... A data set consisting of N samples with labels of (X _n , Y _n ). Where Y _i ∈ {+1, -1} is the label for the pedestrian and background class, respectively.

M: A pool of random ferns calculated over the entire sample.

2: initialize positive sample weights D ₁ (x _i ) = 1 / N _pos and negative sample weights D ₁ (x _i ) = - 1 / N _neg .

3: for t = 1 to T do

for m = 1 to M do

Under the current distribution D _t , the hypothesis h _m (x) using the following equation 1 is calculated. The hypothesis is defined by the co-occurrence of the random-fun observation (output) z. The observation variable z of the fun is captured by the variable k.

? is a smoothing coefficient, and the probability P is estimated by Equations (6) and (7) to be described later.

Next, the Bhattacharyya distance Q _m of the hypothesis h _m (x) is estimated by the following equation (2).

end.

We choose h _t that minimizes Q _m and update the sample weights by the following equation (3).

The strong classifier is updated by the following equation (4).

end.

4: Finally, the strong classifier is obtained by the following equation (5).

Here,? _E is a classifier threshold value.

At each boosting iterations, the probability _{P (z t (x) =} k | c target) and _{P (z t (x) =} k | c background) is

Is calculated in the weight distribution D of the learning samples as shown in the following equations (6) and (7).

와 같이 자동으로 설정된다. 여기서, w와 h는 검색의 바운딩 박스(bounding box)의 폭과 높이이고, (cx,cy)는 보행자 위치의 중심이다.
In the initial detection and tracer setting, we combine a particle filter for predicting the position of a pedestrian with a motion model and a random - funn - based on - line tracer to localize the pedestrian. This combination is affected by time critical and online applications. When the pedestrian is detected, in the first step, the pedestrian is assigned to the detection set D, and the jth sense initial vector is

As shown in FIG. Where w and h are the width and height of the bounding box of the search, and (cx, cy) is the center of the position of the pedestrian.

와 같이 자동으로 설정된다. 여기서, RF₁ ⁱ는 t=1에서 온라인 부스티드 랜덤 펀 학습에 의해 결정되는 i번째 보행자에 대한 랜덤 펀 분류기이다.
The initial tracker set T and the i-th tracker state vector

As shown in FIG. Here, RF ₁ ⁱ is a random-funciner for i-th pedestrian determined by on-line booster random fun learning at t = 1.

In the second frame, the position of the tracer tr ⁱ is updated using the particle and its likelihood. The random fern and its distribution are learned for each tracker from the second frame in the random fun online learning method after being determined in the first frame.

Reference features The histogram, ie LID and particle OCS-LBP, is applied to the corresponding random fun for the tracer. Therefore, the overall likelihood of the particle j is estimated by the likelihood of the random fun as shown in the following equation (8).

This process is repeated until the likelihood for all particles is calculated. If the final likelihood P ^j for the jth particle is estimated, then the weight w _t ^j of the ^jth particle at time t is replaced using the likelihood P ^j obtained from the random fun, and each weight is normalized.

The state of the current pedestrian is updated with the state vector for the top j particle with a large weight, except for the random funn.

Step S300 is a step of estimating the pedestrian state through the correlation test. FIG. 9 is a flowchart of a step S300 in a method of tracking a pedestrian using an online random fun learning in a thermal image according to an exemplary embodiment of the present invention. FIG. 10 is a flowchart illustrating an online random fern And a method of tracking the pedestrian using the learning is a conceptual diagram of the step S300. As shown in FIGS. 9 and 10, in an apparatus and method for tracking a pedestrian using online random fun learning in a thermal image according to an exemplary embodiment of the present invention, step S300 includes steps of detecting (Step S310) of calculating the estimated value by combining the distance between the pedestrian and the tracer assigned to the detected pedestrian, the probability of the pedestrian detected using the tracer's model, and the overlap ratio, and using the Hungarian algorithm (S320), and estimating the final pedestrian state by combining the information of the pedestrian and the tracer detected according to the relevance test result (S330).

According to an embodiment, it is possible to allocate a pedestrian detected in the current frame from the second frame to the existing tracker by maximizing the estimated value. First, an estimation value for assigning each detection to each track is calculated using the matching function sim (tr ⁱ , d ^j ) as shown in the following equation (9). At this time, the higher the estimated value between the detection target and the tracking target, the better the match.

Where P (tr ⁱ , d ^j ) is the likelihood that detection is applied to the tracer tr ⁱ , and dist (d ^j -tr ⁱ ) is the Euclidean distance between the tracer and the detected d ^j . Size (d ^j -tr ⁱ ) is an indicator function that causes the overlap rate between the detection d ^j and the tracer tr ⁱ to return to 1 when the threshold is below. The tracing function uses the Hungarian algorithm to calculate the maximum assignment of the total score.

Step S400 is a step of determining whether to update by applying an online random fun learning algorithm. FIG. 11 is a flowchart of a step S400 of a method for tracking a pedestrian using on-line random fun learning in a thermal image according to an exemplary embodiment of the present invention. FIG. And a method of tracking a pedestrian using learning is a conceptual diagram of step S400. 11 and 12, in an apparatus and method for tracking a pedestrian using on-line random fun learning in a thermal image according to an exemplary embodiment of the present invention, a step S400 includes the steps of using a booster random- If the observed likelihood is higher than the maximum threshold value (U-max), then an online random-fun learning algorithm using two maximum and minimum thresholds (U-max, U-min) (Step S410). If the observation likelihood is lower than the minimum threshold value (U-min), it is determined to be completely overlapped and the random fun is updated (S420), and if the observed likelihood is between the maximum threshold value (U-max) and the minimum threshold value (U-min), it is determined that the partial overlapping or small state change is present, Update Fun If not, it can be configured to include a step (S430) to re-extract the particles.

Depending on the embodiment, the tracer tr ⁱ state after the association check can be updated by combining the state of the current tracker tr ⁱ and the detection d ^j using the following equation:

Where a is an adjustable parameter for updating the state of the tracer set to 0.5. Because Boosted random fun is not suitable for real-time learning, it learns random fun instead of boosted random fun in the second frame. For learning of the random funn, a positive sample is sampled in the bounding box of the association detection. A negative learning set is randomly sampled at least a distance l from a nearby object. If the search does not overlap with another search, or if the matching score exceeds the threshold, the random-functors classifier is updated. Given the learning input for the tracer tr ^{i *} , simply apply the same binary test for each fun, accumulate the initial distribution of the busted random fun, and update the distribution of the random fun.

13 is an experimental result table of multiple-object tracking precision (MOTP) of an apparatus and method for tracking a pedestrian using on-line random fun learning in a thermal image according to an embodiment of the present invention, Is an experimental result table for multiple objects tracking accuracy (MOTA) of an apparatus and method for tracking a pedestrian using on-line random fun learning in a thermal image according to an exemplary embodiment of the present invention. 13 and 14, the multiple-object tracking precision (MOTP) of an apparatus and method for tracking a pedestrian using on-line random fun learning in a thermal image according to an exemplary embodiment of the present invention Experimental results on multiple objects tracking accuracy (MOTA) and experimental results on multi-object tracking accuracy (MOTA) indicate that a partial or complete occlusion at night, a complex background, a sudden shape change, an unexpected behavior, , And 10 kinds of thermal infrared video and 320 × 240 pixel size video were tested for performance. Seven videos recorded in winter and three recorded in summer. First, Multilateral Tracking Accuracy (MOTP) was used. MOTP is the total error of estimated positions of pedestrian and tracker pairs matched in all frames. This represents the tracer's ability to estimate accurate pedestrian position. The experiment used the overlap ratio instead of the distance.

As a second criterion, MOTA was used for CLEAR MOT recognition. MOTA represents the setting errors of all pedestrians created by the tracer over all frames.

As shown in FIG. 13, the average MOTP performance of the apparatus and method for tracking a pedestrian using online random fun learning in a thermal image according to an embodiment of the present invention is 89%, which is higher than the performance of the comparison target method . Since video 10 was recorded by a mobile camera in hot summer, method 1 in FIG. 13 showed the best MOTP performance. However, in FIG. 14, it can be seen that the performance of the MOTA performance is lower than that of the apparatus and method for tracking the pedestrian using the online random fun learning in the thermal image according to the embodiment of the present invention. That is, the online random fun based on Boosted Random Fun has strong and excellent tracking results compared to simple random Firm (Method 1) or Hot Spot based Tracking (Method 2).

As shown in FIG. 14, the overall average MOTA of an apparatus and method for tracking a pedestrian using on-line random fun learning in a thermal image according to an exemplary embodiment of the present invention is 79.5%. Video 2 and Video 4 showed slightly lower results than Method 1 due to the complete closure of some pedestrians. In Method 2, videos 2 and 10 have negative values because some erroneous response has occurred. Because video 10 was recorded by a mobile camera, an inaccurate value due to weak vibrations was produced. The apparatus and method for tracking a pedestrian using on-line random fun learning in a thermal image according to an exemplary embodiment of the present invention can detect a stronger result and a smaller It can be seen that it has a ratio error.

FIG. 15 is a view illustrating a result of tracking a pedestrian using an overlap ratio of an apparatus and method for tracking a pedestrian using an online random fun learning method in a thermal image according to an exemplary embodiment of the present invention. As shown in FIG. 15, the result of tracking the pedestrian using the overlap ratio of the apparatus and method for tracking the pedestrian using the online random fun learning in the thermal image according to the embodiment of the present invention, Provides a detailed description of relative tracking performance. In other words, three videos show a plurality of pedestrian tracking methods for the overlapping rate between the ground-truth (GT) and the pedestrian (ST) detected by the system.

16 is a diagram illustrating an image showing a result of tracking an apparatus and method for tracking a pedestrian using on-line random fun learning in a thermal image according to an embodiment of the present invention. As shown in FIG. 16, a plurality of pedestrian tracking results obtained from ten videos are shown using an apparatus and method for tracking a pedestrian using on-line random fun learning in a thermal image according to an embodiment of the present invention . Video 10 is recorded by a mobile camera.

The present invention may be embodied in many other specific forms without departing from the spirit or essential characteristics of the invention.

S100: Detecting the pedestrian using the Haar-like feature-based random forest and the OCS-LBP feature-based random forest in the input thermal image
S110: Sliding the window in a specific frame of the inputted thermal image
S120: Extract Haar-like feature from the sliding window
S130: setting a candidate region by applying a random forest classifier based on the extracted Haar-like feature
S140: Extracting the OCS-LBP feature from the sliding window
S150: determining whether to detect a pedestrian for the sliding window by applying a random forest classifier based on the extracted OCS-LBP feature
S160: Detecting the position of the pedestrian with respect to the specific frame using the result of the detection of whether or not the pedestrian is detected
S200: Tracking detected pedestrians in real time using online random fun learning and particle filter based on OCS-LBP feature and LID feature
S210: From the second frame of the inputted thermal image, the particle filter filters particles of each frame including an area where a pedestrian is found
S220: Estimating the observation likelihood of the filtered particles and classifying them into particles having an upper weight value
S222: Estimating the observation likelihood by applying the OCS-LBP feature and the LID feature extracted from the filtered particle to a boiled random-function classifier using each feature model
S224: Combining the estimated observation likelihoods to set the weight of each particle
S226: Classifying the particles with the highest weight after updating the particle based on the weight of each set particle
S230: Estimating the state of the pedestrian using the classified particles
S300: estimating the state of the final pedestrian by combining the information of the pedestrian and the tracer detected according to the result of the relevance test
S310: calculating the estimated value by combining the distance between the detected pedestrian and the tracker given to the detected pedestrian, the probability of the detected pedestrian using the tracer's model and the overlapping rate
S320: checking the association using a Hungarian algorithm based on the calculated estimation value
S330: estimating the final pedestrian state by combining the information of the pedestrian and the tracker detected according to the result of the association test
S400: Determining whether to update the random fun and re-extract the particle by applying the online random fun learning algorithm
S410: If the observed likelihood is higher than the maximum threshold value (U-max), it is determined that there is no overlapping condition and the random fun is updated and the particle is re-extracted
S420: when the observed likelihood is lower than the minimum threshold value (U-min), it is judged as a complete overlap and the previous particle information is used again without updating the random fun
S430: a step of re-extracting the particles without updating the random fun, judging the partial overlap or small state change when the observed likelihood is between the maximum threshold value (U-max) and the minimum threshold value (U-min)

Claims (20)

A method for tracking a pedestrian using an online random fun learning in a thermal image,
(1) detecting a pedestrian using a random forest based on a Haar-like feature and a random forest based on an OCS-LBP (oriented center-symmetric local binary pattern) feature in an input thermal image; And
(2) tracking the detected pedestrian in real time using online random ferns learning and particle filter based on OCS-LBP feature and LID (local intensity distribution) feature Wherein the step of tracking the pedestrian using the online random fun learning in the thermal image is performed.
The method according to claim 1,
Wherein the step of detecting the pedestrian is performed at the first frame of the column image and the step of detecting the pedestrian and the step of tracking the pedestrian are simultaneously performed from the second frame.
3. The method of claim 2, wherein in the first frame of the column image,
In addition to the detection of the pedestrian, the initial pedestrian state is initialized, the particle sample is extracted, the random fun is learned using the detected information of the pedestrian, and the learned random fun is assigned to each tracker assigned to the detected pedestrian A method of tracking a pedestrian using an online random fun learning in a thermal image.
The method of claim 3,
Before the random fun is learned, a boosted random funn is selected through a boosting algorithm for evaluating a binary test and selecting optimal ferns among candidate ferns of a window sliding in a first frame And the selected booster random fun is learned. A method for tracking a pedestrian using on-line random fun learning in a thermal image.
2. The method of claim 1, wherein the step (1)
(1-1) sliding a window in a specific frame of an inputted column image;
(1-2) extracting Haar-like features from the sliding window;
(1-3) setting a candidate region by applying a random forest classifier based on the Haar-like feature extracted in the step (1-2);
(1-4) extracting an OCS-LBP feature in the sliding window;
(1-5) determining whether to detect a pedestrian for the sliding window by applying a random forest classifier based on the OCS-LBP feature extracted in the step (1-4); And
(1-6) detecting a pedestrian position of the specific frame by using a pedestrian detection result of the sliding window, and detecting a pedestrian position of the specific frame using the result of the pedestrian detection of the sliding window. How to track.
2. The method of claim 1, wherein step (2)
(2-1) from the second frame of the inputted thermal image, the particle filter filters particles of each frame including an area where a pedestrian is found;
(2-2) estimating an observation likelihood of the filtered particles and classifying the particles into particles having an upper weight value; And
(2-3) A method for tracing a pedestrian using online random fun learning in a thermal image, comprising the step of estimating a state of a pedestrian using the classified particles.
7. The method according to claim 6, wherein the step (2-2)
Applying an OCS-LBP feature and a LID feature extracted from the filtered particle to a boiled random punster classifier using each feature model, estimating an observation likelihood, combining the estimated observation likelihoods, And classifying particles having higher weights after updating the particles based on the weights of the set particles. The method of claim 1, How to track.
3. The method of claim 2, wherein in the second frame,
(3) estimating the state of the final pedestrian by combining the detected information of the pedestrian and the tracker according to the result of the association test; And
(4) applying an online random fun learning algorithm to determine whether to update the random fun and whether to re-extract the particles. The method according to claim 1, further comprising the step of tracking the pedestrian using the online random fun learning How to.
9. The method of claim 8, wherein step (3)
(3-1) calculating an estimated value by combining the distance between the pedestrian detected in the step (1) and the tracker given to the detected pedestrian, the probability of the detected pedestrian using the tracker's model, and the overlapping rate;
(3-2) checking association using a Hungarian algorithm based on the calculated estimated value; And
(3-3) combining the detected information of the pedestrian and the tracker according to the result of the association test to estimate the final pedestrian state. How to.
9. The method according to claim 8, wherein in the step (4)
After estimating the observation likelihood of the final pedestrian using the above-mentioned booster random fun classifier, an online random fun learning algorithm using two maximum and minimum thresholds (U-max, U-min) is applied,
(4-1) if the observed likelihood is higher than the maximum threshold value (U-max), determining a general situation without overlapping, updating the random fun and re-extracting the particles;
(4-2) if the observed likelihood is lower than the minimum threshold value (U-min), judging the completely overlapping and using the previous particle information again without updating the random funn; And
(4-3) When the observation likelihood is between the maximum threshold value (U-max) and the minimum threshold value (U-min), it is determined that the partial overlap or the small state change, Wherein the step of tracking the pedestrian using the on-line random fun learning in the thermal image is performed.
An apparatus for tracking a pedestrian using an online random fun learning in a thermal image,
The apparatus includes a processor,
(1) detecting a pedestrian using a random forest based on a Haar-like feature and a random forest based on an OCS-LBP (oriented center-symmetric local binary pattern) feature in an input thermal image; And
(2) tracking the detected pedestrian in real time using online random ferns learning and particle filter based on OCS-LBP feature and LID (local intensity distribution) feature Wherein the step of tracking the pedestrian using the online random fun learning in the thermal image.
12. The apparatus of claim 11,
Wherein the step of detecting the pedestrian is performed at the first frame of the thermal image and the detection of the pedestrian and the tracking of the pedestrian are simultaneously performed at the second frame.
13. The method of claim 12, wherein in a first frame of the column image,
In addition to the detection of the pedestrian, the initial pedestrian state is initialized, the particle sample is extracted, the random fun is learned using the detected information of the pedestrian, and the learned random fun is assigned to each tracker assigned to the detected pedestrian To track pedestrians using on-line random fun learning in thermal imaging.
14. The apparatus of claim 13,
Before the random fun is learned, a boosted random funn is selected through a boosting algorithm for evaluating a binary test and selecting optimal ferns among candidate ferns of a window sliding in a first frame And the selected booster random fun is learned. The apparatus for tracking a pedestrian by using online random fun learning in a thermal image.
12. The method of claim 11, wherein step (1)
(1-1) sliding a window in a specific frame of an inputted column image;
(1-2) extracting Haar-like features from the sliding window;
(1-3) setting a candidate region by applying a random forest classifier based on the Haar-like feature extracted in the step (1-2);
(1-4) extracting an oriented center-symmetric local binary pattern (OCS-LBP) feature in the sliding window;
(1-5) determining whether to detect a pedestrian for the sliding window by applying a random forest classifier based on the OCS-LBP feature extracted in the step (1-4); And
(1-6) detecting a pedestrian position of the specific frame by using a pedestrian detection result of the sliding window, and detecting a pedestrian position of the specific frame using the result of the pedestrian detection of the sliding window. Tracking device.
12. The method of claim 11, wherein step (2)
(2-1) from the second frame of the inputted thermal image, the particle filter filters particles of each frame including an area where a pedestrian is found;
(2-2) estimating an observation likelihood of the filtered particles and classifying the particles into particles having an upper weight value; And
(2-3) estimating the state of the pedestrian using the classified particles, and tracking the pedestrian using the online random fun learning in the thermal image.
The method as claimed in claim 16, wherein the step (2-2)
Applying an OCS-LBP feature and a LID feature extracted from the filtered particle to a boiled random punster classifier using each feature model, estimating an observation likelihood, combining the estimated observation likelihoods, And classifying particles having higher weights after updating the particles based on the weights of the set particles. The method of claim 1, Lt; / RTI >
13. The method of claim 12, wherein in the second frame,
(3) estimating the state of the final pedestrian by combining the detected information of the pedestrian and the tracker according to the result of the association test; And
(4) applying an online random fun learning algorithm to determine whether to update the random fun and whether to re-extract the particles. The method according to claim 1, further comprising the step of tracking the pedestrian using the online random fun learning .
19. The method of claim 18, wherein step (3)
(3-1) calculating an estimated value by combining the distance between the pedestrian detected in the step (1) and the tracker assigned to the detected pedestrian, the probability of the detected pedestrian using the tracker's model, and the overlapping rate;
(3-2) checking association using a Hungarian algorithm based on the calculated estimated value; And
(3-3) combining the detected information of the pedestrian and the tracker according to the result of the association test to estimate the final pedestrian state. .
19. The method of claim 18, wherein in step (4)
After estimating the observing likelihood of the final pedestrian using the booster random fun classifier, an online random fun learning algorithm using two maximum and minimum thresholds (U-max, U-min) is applied,
(4-1) if the observed likelihood is higher than the maximum threshold value (U-max), determining a general situation without overlapping, updating the random fun and re-extracting the particles;
(4-2) if the observed likelihood is lower than the minimum threshold value (U-min), judging the completely overlapping and using the previous particle information again without updating the random funn; And
(4-3) When the observation likelihood is between the maximum threshold value (U-max) and the minimum threshold value (U-min), it is determined that the partial overlap or the small state change, Wherein the step of tracing the pedestrian using the online random fun learning in the thermal image is performed.

Images