KR102318397B1 - Object tracking method and device that is robust against distance and environment change - Google Patents

Abstract

The size of an image patch for tracking an object is changed using the distance from the distance meter to the object obtained, and preliminary tracking points are obtained using feature patches extracted from the image patch based on different features, and preliminary tracking points are obtained. By fusing the tracking points, a tracking point for tracking the object is determined.

Description

Object tracking method and device that is robust against distance and environment change

A method and apparatus for tracking an object whose distance from the device changes.

As a method of tracking an object, there is a tracking method using a Kernelized Correlation Filter. In this method, the object is tracked using the size of the image patch specified in the first image frame. Accordingly, this method has a disadvantage in that a tracking error increases when the size of the object in the image frame is changed due to the distance between the tracking device and the object being changed due to the movement of the tracking device or the object.

Also, when a method of tracking an object using only a single image sensor is applied to a device operating in a changing environment, such as a weapon, military equipment, or unmanned aerial vehicle, a tracking error may occur.

[1] J. F. Henriques et. al., “High-Speed Tracking with Kernelized Correlation Filters”, in IEEE Trans. on PAMI, 2015. [2] D. S. Bolme et. al., “Visual Object Tracking Using Adaptive Correlation Filters” in CVPR, 2010, pp. 2544-2550.

An apparatus and method for robustly tracking an object in response to a change in a distance from the object and a change in a surrounding environment are proposed.

The problems of the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the following description.

As a means for solving the above technical problem, a method for tracking an object according to an embodiment includes: acquiring a current image frame from an image sensor; obtaining a current distance by measuring a distance to an object using a range finder; determining, based on the current distance, a current image patch that is a subset of the current image frame; generating a plurality of current feature patches by extracting different features from the current image patch; generating a plurality of current frequency patches by transforming the plurality of current feature patches into a frequency domain; generating a plurality of correlation coefficient patches by obtaining correlation coefficients for each of the plurality of current frequency patches with a corresponding model frequency patch from among a plurality of model frequency patches generated in advance; obtaining a plurality of maximum correlation coefficients by determining a correlation coefficient having a maximum value for each of the plurality of correlation coefficient patches; obtaining a plurality of preliminary tracking points from positions of pixels corresponding to the plurality of maximum correlation coefficients; obtaining a plurality of PSRs by calculating a peak to sidelobe ratio (PSR) for each of the plurality of correlation coefficient patches; selecting at least one maximum correlation coefficient from among the plurality of maximum correlation coefficients and selecting at least one preliminary tracking point from among the plurality of preliminary tracking points based on the plurality of PSRs; and determining a current tracking point by calculating a weighted average of the selected at least one preliminary tracking point using the selected at least one maximum correlation coefficient.

According to another embodiment, a method for tracking an object includes: acquiring a plurality of current image frames by acquiring a current image frame from each of a plurality of image sensors; obtaining a current distance by measuring a distance to an object using a range finder; obtaining a plurality of current image patches by determining a subset current image patch for each of the plurality of current image frames based on the current distance; generating a plurality of current frequency patches by converting features extracted for each of the plurality of current image patches into a frequency domain; generating a plurality of correlation coefficient patches by obtaining correlation coefficients for each of the plurality of current frequency patches with a corresponding model frequency patch from among a plurality of model frequency patches generated in advance; obtaining a plurality of maximum correlation coefficients by determining a correlation coefficient having a maximum value for each of the plurality of correlation coefficient patches; obtaining a plurality of preliminary tracking points from positions of pixels corresponding to the plurality of maximum correlation coefficients; obtaining a plurality of PSRs by calculating a peak to sidelobe ratio (PSR) for each of the plurality of correlation coefficient patches; selecting at least one maximum correlation coefficient from among the plurality of maximum correlation coefficients and selecting at least one preliminary tracking point from among the plurality of preliminary tracking points based on the plurality of PSRs; and determining a current tracking point by calculating a weighted average of the selected at least one preliminary tracking point using the selected at least one maximum correlation coefficient.

According to another embodiment, an apparatus for tracking an object includes: an image sensor; rangefinder; and a processor, wherein the processor obtains a current image frame from the image sensor, obtains a current distance by measuring a distance to an object using the distance finder, and based on the current distance, By determining a current image patch that is a subset of an image frame, and extracting different features from the current image patch, to generate a plurality of current feature patches, and transforming the plurality of current feature patches into a frequency domain, a plurality of current frequencies generating a plurality of correlation coefficient patches by generating patches, and obtaining, for each of the plurality of current frequency patches, correlation coefficients with a corresponding model frequency patch from among a plurality of model frequency patches generated in advance, a plurality of correlation coefficient patches; obtaining a plurality of maximum correlation coefficients by determining a correlation coefficient having a maximum value for each of the correlation coefficient patches of , obtaining a plurality of preliminary tracking points from positions of pixels corresponding to the plurality of maximum correlation coefficients; , obtains a plurality of PSRs by calculating a peak to sidelobe ratio (PSR) for each of the plurality of correlation coefficient patches, and based on the plurality of PSRs, the maximum correlation of at least one of the plurality of maximum correlation coefficients By selecting a coefficient, selecting at least one preliminary tracking point from among the plurality of preliminary tracking points, and calculating a weighted average for the selected at least one preliminary tracking point by using the selected at least one maximum correlation coefficient, Determine the current tracking point.

According to another embodiment, an apparatus for tracking an object includes: a plurality of image sensors; rangefinder; and a processor, wherein the processor obtains a plurality of current image frames by obtaining a current image frame from each of a plurality of image sensors, and obtains a current distance by measuring a distance to an object using a range finder. and, based on the current distance, determining a subset current image patch for each of the plurality of current image frames, thereby obtaining a plurality of current image patches, and a feature extracted for each of the plurality of current image patches by transforming into the frequency domain to generate a plurality of current frequency patches, and for each of the plurality of current frequency patches, by obtaining correlation coefficients with a corresponding model frequency patch among a plurality of pre-generated model frequency patches, By generating a plurality of correlation coefficient patches and determining a correlation coefficient having a maximum value for each of the plurality of correlation coefficient patches, a plurality of maximum correlation coefficients is obtained, and pixels corresponding to the plurality of maximum correlation coefficients are obtained. Obtaining a plurality of preliminary tracking points from locations, calculating a peak to sidelobe ratio (PSR) for each of the plurality of correlation coefficient patches, to obtain a plurality of PSRs, and based on the plurality of PSRs, the plurality of selecting at least one maximum correlation coefficient among the maximum correlation coefficients of By calculating a weighted average of the preliminary tracking points, the current tracking point is determined.

When the small UAV approaches or retreats toward the device, the size of the object captured in the image may change significantly. Since the object to which the size change is reflected can be tracked using the distance measured from the range finder, tracking error can be reduced.

It is possible to obtain the distance to the target by using the rangefinder mounted on the military surveillance and reconnaissance weapon.

By acquiring a plurality of preliminary tracking points from a plurality of image frames acquired through a plurality of image sensors and fusing the plurality of preliminary tracking points to determine the tracking point, the object may be robustly tracked against changes in the surrounding environment. In addition, by acquiring a plurality of preliminary tracking points based on different features from one image frame and fusion of the plurality of preliminary tracking points to determine the tracking point, the object may be robustly tracked against changes in the surrounding environment.

The tracking method according to the present embodiment can be applied to a tracker based on a correlation filter, and is not limited by the type of image sensor or the type of feature.

The effect according to the present invention is not limited by the contents exemplified above, and more various effects are included in the present specification.

1 illustrates an example of a device for tracking an object.
2 is a diagram for explaining an example of a method for tracking an object.
3 is a diagram for explaining an example of a method of determining a current image patch.
4 is a diagram for explaining an example of a method of obtaining a maximum correlation coefficient, a preliminary tracking point, and a PSR.
5 is a diagram for explaining an example of a method for tracking an object.
6 is a flowchart illustrating an example of a method for tracking an object.
7 is a flowchart illustrating an example of a method for tracking an object.
8 is a graph illustrating a tracking error of an object.

Hereinafter, exemplary embodiments will be described in detail with reference to the accompanying drawings. It goes without saying that the following description is only for specifying the embodiments and does not limit or limit the scope of the invention. What can be easily inferred by an expert in the art from the detailed description and examples is construed as belonging to the scope of the right.

As used herein, terms such as 'consisting' or 'comprising' should not be construed as necessarily including all of the various components or various steps described in the specification, and some components or some steps thereof. It should be construed that they may not be included, or may further include additional components or steps.

The terms used in this specification have been selected as currently widely used general terms as possible while considering the functions in the present invention, which may vary depending on the intention or precedent of a person skilled in the art, the emergence of new technology, and the like. In addition, in a specific case, there is a term arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the corresponding invention. Therefore, the term used in the present invention should be defined based on the meaning of the term and the overall content of the present invention, rather than the name of a simple term.

1 illustrates an example of a device for tracking an object.

The device 100 may perform a function of tracking an object. Optionally, the device 100 may be mounted on equipment operated outdoors, such as military equipment. The device 100 may be used to track a moving object, such as an unmanned aerial vehicle, missile, vehicle, or the like.

The device 100 may include at least one image sensor 110 , a range finder 120 , and a processor 130 . However, only the components related to the present embodiments are illustrated in the apparatus 100 illustrated in FIG. 1 . Accordingly, it is apparent to those skilled in the art that the device 100 may further include other general-purpose components in addition to the components illustrated in FIG. 1 . For example, the device 100 may further include a memory, a power source (connecting means with an internal power source or an external power source), a communication means, an input means, an output means, and an electrical or physical connection means with an external device. .

The apparatus 100 may photograph an object using at least one image sensor 110 . At least one image sensor 110 may include a photoelectric conversion element. For example, the at least one image sensor 110 may include at least one of a Charge Coupled Device (CCD) image sensor, a Complementary Metal-Oxide Semiconductor (CMOS) image sensor, a gray image sensor generating a gray image, and an infrared image sensor. may include, but is not limited thereto. The infrared image sensor may be a Medium Wave Infrared (MWIR) image sensor for mid-infrared rays, a Short Wave Infrared (SWIR) image sensor for near-infrared rays, and the like, but is not limited thereto.

When the apparatus 100 includes a plurality of image sensors 110 , the plurality of image sensors 110 may be of the same type or different from each other. For example, apparatus 100 may include a CCD image sensor, a gray scale image sensor, a MWIR image sensor, and a SWIR image sensor. Since the apparatus 100 includes different types of image sensors, it is possible to acquire an image of an object robustly against changes in the surrounding environment, noise, and sensor failure.

In other embodiments, the device 100 may not include at least one image sensor. In another embodiment, the device 100 may be mounted on a device equipped with at least one image sensor, and the device 100 may acquire an image frame from at least one image sensor of the device.

The device 100 may measure a distance to the object by using the range finder 120 . The distance measurer 120 may measure a distance from the device 100 to the object. The range finder 120 may be a laser range finder, a light wave range finder, or an electromagnetic wave range finder (Radio Range Finder), but is not limited thereto.

In other embodiments, device 100 may not include a range finder. In another embodiment, the device 100 may be mounted on a device equipped with a range finder, and the device 100 may obtain a distance from the range finder of the device to the object.

The processor 130 serves to perform overall functions for controlling the device 100 . The processor 130 may be implemented as an array of a plurality of logic gates, or may be implemented as a combination of a general-purpose microprocessor and a memory in which a program executable in the microprocessor is stored.

The processor 130 may obtain an image frame from the at least one image sensor 110 , and obtain a distance from the distance measurer 120 to the object. The processor 130 may track the object based on the image frame obtained from the at least one image sensor 110 and the distance obtained from the distance measurer 120 .

2 is a diagram for explaining an example of a method for tracking an object.

In FIG. 2 , processes performed by the processor 130 to track an object are represented by blocks. In the embodiment with reference to FIG. 2 , the processor 130 acquires an image frame from one image sensor, but in another embodiment, the processor 130 may acquire a plurality of image frames from a plurality of image sensors.

The processor 130 may obtain the current image frame IF_crt from the image sensor 110 . In this case, the current image frame IF_crt may be an arbitrary nth image frame obtained from the image sensor 110 .

The processor 130 may obtain a current distance d_crt that is a distance from the distance measurer 120 to the object. In this case, the current distance d_crt is a distance corresponding to the current image frame IF_crt, and may be a distance obtained at the same/similar time as when the current image frame IF_crt is obtained. That is, the current distance d_crt may be a distance to the object captured in the current image frame IF_crt.

The processor 130 may obtain a distance change factor d_fct indicating a change in the distance between the device 100 and the object based on the current distance d_crt.

The processor 130 may obtain the distance change factor d_fct from the current distance d_crt and the reference distance d_ref. The processor 130 may obtain the distance change factor d_fct from Equation (1).

The reference distance d_ref is a distance corresponding to the predetermined reference image frame IF_ref, and may be a distance obtained at the same time point at which the reference image frame IF_ref is obtained. That is, the reference distance d_ref may be a distance to the object captured in the reference image frame IF_ref. The reference image frame IF_ref may be an image frame acquired at a time different from the current image frame IF_crt. For example, the reference image frame IF_ref may be the first image frame. As another example, the reference image frame IF_ref may be a previous image frame (eg, an n−1 th image frame).

The processor 130 may determine the current image patch IP_crt, which is a subset of the current image frame IF_crt, based on the distance change factor d_fct.

3 is a diagram for explaining an example of a method of determining a current image patch.

The processor 130 may determine a previous tracking point P_prev with respect to the previous image frame IF_prev. The previous image frame IF_prev may be an n-1 th image frame. In this case, the tracking point is one pixel.

The processor 130 may set the previous tracking point P_prev in the current image frame IF_crt as the center point of the current image patch IP_crt. For example, when the previous tracking point P_prev is one pixel, the previous tracking point P_prev may be set as the center point of the current image patch IP_crt.

The processor 130 multiplies the size (lx_ref, ly_ref) of the reference image patch (IP_ref) of the reference image frame (IF_ref) by the distance change factor (d_fct) to determine the size (lx_crt, ly_crt) of the current image patch (IP_crt) can decide For example, when lx_ref=120, ly_ref=100, and d_fct=0.8, it may be lx_crt=96, ly_crt=80. In this case, the unit of size may be the number of pixels.

That is, the processor 130 sets the center point of the current image patch IP_crt from the previous tracking point P_prev, and the size (lx_ref, ly_ref) of the reference image patch IP_ref and the current image patch from the distance change factor d_fct. By determining the sizes (lx_crt, ly_crt) of (IP_crt), the current image patch IP_crt may be determined from the current image frame IF_crt.

Since the current image patch can be determined according to the size of the object captured in the current image frame by using the distance change factor, the object can be robustly tracked against a change in the distance between the device 100 and the object or a change in the size of the object in the image frame. can

Referring back to FIG. 2 , the processor 130 may extract a plurality of current feature patches FtP_crt based on different features from the current image patch IP_crt.

The feature may be an edge of an image, color, gray level, luminous intensity, chroma, hue, RGB, etc., but is not limited thereto. For example, the current image frame IF_crt may be a gray image frame obtained from a gray image sensor, the features may be a gray level, a gray lateral edge, and a gray longitudinal edge, and the plurality of current feature patches may be gray It may be a level image, a gray lateral edge image, and a gray longitudinal edge image. For another example, the current image frame IF_crt may be a color image frame obtained from a CCD image sensor, and the features may be R (red), G (green), B (blue), and a plurality of current feature patches. These may be an R-channel image, a G-channel image, or a B-channel image. The size of the current feature patch FtP_crt may be the same as the size of the current image patch IP_crt.

The processor 130 may generate a plurality of current frequency patches FP_crt by converting the plurality of current feature patches FtP_crt into a frequency domain. For example, the processor 130 may generate a plurality of current frequency patches FP_crt based on a Fourier transform, a discrete Fourier transform, or a fast Fourier transform. The size of the current frequency patch FP_crt may be the same as the size of the current image patch IP_crt.

For example, the processor 130 generates the first current frequency patch FP_crt_1 by transforming the first current feature patch FtP_crt_1 into the frequency domain, and converts the second current feature patch FtP_crt_2 into the frequency domain to generate the second current feature patch FtP_crt_2 The mth current frequency patch FP_crt_m may be generated by generating the 2nd current frequency patch FP_crt_2 and converting the mth current feature patch FtP_crt_m into the frequency domain.

The processor 130 may generate a plurality of correlation coefficient patches CP by calculating correlation coefficients of the plurality of current frequency patches FP_crt and the plurality of model frequency patches FP_mdl. The processor 130 calculates an element-wise product between elements with a corresponding model frequency patch FP_mdl for each of the plurality of current frequency patches FP_crt, and performs Inverse Fourier Transform to , a plurality of correlation coefficient patches CP may be generated.

Before calculating the product between the elements, the processor 130 may normally calculate the product between the elements by changing the size of the plurality of model frequency patches FP_mdl to be the same as the size of the current frequency patch FP_crt. The size of the correlation coefficient patch CP may be the same as the size of the current frequency patch FP_crt.

For example, the product of the elements of the first model frequency patch FP_mld_1 and the first current frequency patch FP_crt_1 is transformed into a spatial domain, whereby the first correlation coefficient patch CP_1 may be generated, and the second model frequency By transforming the product between the elements of the patch FP_mld_2 and the second current frequency patch FP_crt_2 into the spatial domain, the second correlation coefficient patch CP_2 may be generated, and the mth model frequency patch FP_mld_m and the mth current frequency patch FP_mld_m A product of elements of the frequency patch FP_crt_m is transformed into a spatial domain, so that an m-th correlation coefficient patch CP_m may be generated.

The model frequency patch FP_mdl may include frequency components including information on an object to be tracked. The model frequency patch FP_mdl may be updated for each image frame to reflect a change in the object captured by the image sensor 110 .

Each of the plurality of model frequency patches FP_mld may be based on the same characteristic as the corresponding current frequency patch FP_crt. For example, the first model frequency patch FP_mld_1 and the first current frequency patch FP_crt_1 are based on a gray level image, and the second model frequency patch FP_mld_2 and the second current frequency patch FP_crt_2 have a gray transverse direction. Based on the edge image, the mth model frequency patch FP_mld_3 and the mth current frequency patch FP_crt_m may be based on the gray longitudinal edge image.

The processor 130 may obtain a plurality of maximum correlation coefficients Cmax by determining a correlation coefficient having a maximum value for each of the plurality of correlation coefficient patches CP.

The processor 130 may obtain a plurality of preliminary tracking points P_tmp from positions of pixels corresponding to each of the plurality of maximum correlation coefficients Cmax.

The processor 130 may obtain a plurality of PSRs by calculating a peak to sidelobe ratio (PSR) for each of the plurality of correlation coefficient patches CP.

4 is a diagram for explaining an example of a method of obtaining a maximum correlation coefficient, a preliminary tracking point, and a PSR.

4 shows a part of the correlation coefficient patch CP.

The processor 130 may determine the correlation coefficient having the maximum value among the correlation coefficient patches CP as the maximum correlation coefficient Cmax. For example, when pixels of the correlation coefficient patch CP each have a value within a range of 0 to 1, a maximum value of 1 may be determined as the maximum correlation coefficient Cmax.

The processor 130 may determine the pixel having the maximum correlation coefficient Cmax in the correlation coefficient patch CP as the preliminary tracking point P_tmp.

The processor 130 determines the values of pixels included in the area (hatched area) except for the kxk size area centered on the preliminary tracking point P_tmp among the qxq size area centered on the preliminary tracking point P_tmp ( That is, the PSR can be obtained from the standard deviation (σ) and the mean (μ) of the correlation coefficients). For example, k may be set to 5.

Specifically, the processor 130 may obtain a PSR based on Equation (2).

Referring back to FIG. 2 , the processor 130 selects at least one maximum correlation coefficient (Cmax_i) from among the plurality of maximum correlation coefficients (Cmax) based on the plurality of PSRs and selects at least one of the plurality of preliminary tracking points (P_tmp). At least one preliminary tracking point P_tmp_i may be selected.

If there is a PSR having a value greater than the reference value α among the plurality of PSRs, the processor 130 may select a maximum correlation coefficient and a preliminary tracking point corresponding to the PSR. The reference value α is a preset value. For example, the reference value α may be 0.8.

For example, when the first PSR(PSR_1)=0.9, the second PSR(PSR_2)=0.5, the mth PSR(PSR_m)=0.85, and α=0.8, the processor 130 performs the first maximum correlation coefficient Cmax_1 ), the first preliminary tracking point (P_tmp_1), the mth maximum correlation coefficient (Cmax_m), and the mth preliminary tracking point (P_tmp_m) may be selected.

For another example, when the first PSR (PSR_1) = 0.9, the second PSR (PSR_2) = 0.5, the m th PSR (PSR_m) = 0.85, and α = 0.9, the processor 130 is the first maximum correlation coefficient ( Cmax_1) and the first preliminary tracking point P_tmp_1 may be selected.

If there is no PSR having a value greater than the reference value α among the plurality of PSRs, the processor 130 selects a maximum correlation coefficient and a preliminary tracking point corresponding to the PSR having the largest value, or selects a plurality of maximum correlation coefficients. Both (Cmax) and the plurality of preliminary tracking points (P_tmp) may be selected.

For example, when the first PSR(PSR_1)=0.9, the second PSR(PSR_2)=0.5, the mth PSR(PSR_m)=0.85, and α=0.95, the processor 130 determines the first value having the largest value. The first maximum correlation coefficient Cmax_1 and the first preliminary tracking point P_tmp_1 corresponding to the PSR (PSR_1) may be selected.

As another example, when the first PSR(PSR_1)=0.9, the second PSR(PSR_2)=0.5, the mth PSR(PSR_m)=0.85, and α=0.95, the processor 130 determines the two largest PSRs. The first maximum correlation coefficient (Cmax_1) corresponding to the first PSR (PSR_1) and the mth PSR (PSR_m), the first preliminary tracking point (P_tmp_1), the mth maximum correlation coefficient (Cmax_m), and the mth preliminary tracking point ( P_tmp_m) can be selected.

As another example, when the first PSR (PSR_1) = 0.9, the second PSR (PSR_2) = 0.5, the m th PSR (PSR_m) = 0.85, and α = 0.95, the processor 130 may include a plurality of maximum correlation coefficients. All of Cmax and the plurality of preliminary tracking points P_tmp may be selected.

As another example, when the first PSR(PSR_1)=0.9, the second PSR(PSR_2)=0.5, the mth PSR(PSR_m)=0.85, and α=0.95, the processor 130 determines the second value having the lowest value. A plurality of maximum correlation coefficients and a plurality of preliminary tracking points corresponding to the remaining PSRs except for the PSR (PSR_2) may be selected.

A small PSR means that the reliability of the obtained preliminary tracking point is low, for example, because the object is obscured or it is difficult to distinguish the object from neighboring clusters. The processor 130 determines the current tracking point P_crt from the maximum correlation coefficient corresponding to the relatively large PSR and the preliminary tracking point, thereby robustly tracking the object against changes in the surrounding environment and noise.

The processor 130 may determine the current tracking point P_crt by fusing the selected at least one preliminary tracking point using the at least one selected maximum correlation coefficient. The processor 130 may determine the current tracking point P_crt by calculating a weighted average for the selected at least one preliminary tracking point (P_tmp_i=(x_tmp_i, y_tmp_i)) using the selected at least one maximum correlation coefficient. . Specifically, the processor 130 may determine the current tracking point P_crt based on Equation (3).

When the current tracking point P_crt is configured as a single pixel, the tracking points x_crt and y_crt obtained by fusing at least one selected preliminary tracking point may be determined as the current tracking point P_crt.

The processor 130 may update the plurality of model frequency patches FP_mdl to determine the next tracking point for the next image frame. The processor 130 may update the plurality of model frequency patches FP_mdl based on the plurality of current frequency patches FP_crt, the current tracking point P_crt, and the like.

The processor may update the model frequency patch corresponding to the PSR having a value greater than the reference value α among the plurality of PSRs, and may not update the model frequency patch corresponding to the remaining PSRs.

For example, when the first PSR(PSR_1)=0.9, the second PSR(PSR_2)=0.5, the mth PSR(PSR_m)=0.85, and α=0.8, the processor 130 generates the first model frequency patch FP_mdl_1 ) and the mth model frequency patch FP_mdl_m may be updated, but the second model frequency patch FP_mdl_2 may not be updated.

5 is a diagram for explaining an example of a method for tracking an object.

In FIG. 5 , processes performed by the processor 130 to track an object are represented by blocks. In the embodiment with reference to FIG. 5 , the processor 130 acquires a plurality of image frames from a plurality of image sensors. The same configuration as in the embodiment with reference to FIG. 2 will be omitted.

The processor 130 may obtain a plurality of current image frames IF_crt from the plurality of image sensors 110 . The processor 130 may obtain a plurality of current image frames IF_crt by obtaining each current image frame from the plurality of image sensors 110 .

The processor 130 may obtain a plurality of current image patches IP_crt by determining a current image patch that is a subset of the plurality of current image frames IF_crt based on the distance change factor d_fct.

Since the current image patch can be determined according to the size of the object captured in the current image frame by using the distance change factor, the object can be robustly tracked against a change in the distance between the device 100 and the object or a change in the size of the object in the image frame. can

The field of view (FOV) of the plurality of image sensors 110 may be the same, and the plurality of current image patches IP_crt may have the same location and size. For example, the position of the first current image patch (IP_crt_1) in the first current image frame (IF_crt_1) and the position of the second current image patch (IP_crt_2) in the second current image frame (IF_crt_2) and the mth current image frame ( In IF_crt_m), the position of the mth current image patch IP_crt_m may be the same.

The processor 130 may extract a plurality of current feature patches FtP_crt based on at least one feature from the plurality of current image patches IP_crt. For example, the first current image frame (IF_crt_1) is a gray image frame obtained from a gray image sensor, the second current image frame (IF_crt_2) is a color image frame obtained from a CCD image sensor, and an mth current image frame ( IF_crt_m) may be a near-infrared image frame obtained from a near-infrared image sensor. For example, the first current feature patch FtP_crt_1 may be a gray lateral edge image, the second current feature patch FtP_crt_2 may be an R channel image, and the mth current feature patch FtP_crt_m may be a near-infrared level image.

The processor 130 may generate a plurality of current frequency patches FP_crt by converting the plurality of current feature patches FtP_crt into a frequency domain.

The processor 130 may generate a plurality of correlation coefficient patches CP by calculating correlation coefficients of the plurality of current frequency patches FP_crt and the plurality of model frequency patches FP_mdl. The processor 130 calculates, for each of the plurality of current frequency patches FP_crt, a product (element-wise product) between elements with a corresponding model frequency patch FP_mdl, and performs an inverse Fourier transform to perform a plurality of correlation coefficients Patches CP may be generated. In this case, before calculating the product between the elements, the processor 130 normally calculates the product between the elements by changing the size of the plurality of model frequency patches FP_mdl to be the same as the size of the plurality of current frequency patches FP_crt. can do.

Each of the plurality of model frequency patches FP_mld may be based on the same characteristic as the corresponding current frequency patch FP_crt. For example, the first model frequency patch FP_mld_1 and the first current frequency patch FP_crt_1 are based on the gray lateral edge image, and the second model frequency patch FP_mld_2 and the second current frequency patch FP_crt_2 are R Based on the channel image, the mth model frequency patch FP_mld_3 and the mth current frequency patch FP_crt_m may be based on a near-infrared level image.

The processor 130 may obtain a plurality of maximum correlation coefficients Cmax by determining a correlation coefficient having a maximum value for each of the plurality of correlation coefficient patches CP.

The processor 130 may obtain a plurality of preliminary tracking points P_tmp from positions of pixels corresponding to each of the plurality of maximum correlation coefficients Cmax.

The processor 130 may obtain a plurality of PSRs by calculating a peak to sidelobe ratio (PSR) for each of the plurality of correlation coefficient patches CP.

The processor 130 selects at least one maximum correlation coefficient Cmax_i from among the plurality of maximum correlation coefficients Cmax based on the plurality of PSRs, and selects at least one preliminary tracking point from among the plurality of preliminary tracking points P_tmp. (P_tmp_i) can be selected.

If there is a PSR having a value greater than the reference value α among the plurality of PSRs, the processor 130 may select a maximum correlation coefficient and a preliminary tracking point corresponding to the PSR.

If there is no PSR having a value greater than the reference value α among the plurality of PSRs, the processor 130 selects a maximum correlation coefficient and a preliminary tracking point corresponding to the PSR having the largest value, or selects a plurality of maximum correlation coefficients. Both (Cmax) and the plurality of preliminary tracking points (P_tmp) may be selected.

A small PSR means that the reliability of the obtained preliminary tracking point is low, for example, because the object is obscured or it is difficult to distinguish the object from neighboring clusters. The processor 130 determines the current tracking point P_crt from the maximum correlation coefficient corresponding to the relatively large PSR and the preliminary tracking point, thereby robustly tracking the object against changes in the surrounding environment and noise.

The processor 130 may determine the current tracking point P_crt by fusing the selected at least one preliminary tracking point using the at least one selected maximum correlation coefficient. Specifically, the processor 130 may determine the current tracking point P_crt by calculating a weighted average of the selected at least one preliminary tracking point using the at least one selected maximum correlation coefficient.

The processor 130 may update the plurality of model frequency patches FP_mdl to determine the next tracking point for the next image frame. The processor 130 may update the plurality of model frequency patches FP_mdl based on the plurality of current frequency patches FP_crt, the current tracking point P_crt, and the like.

The processor may update the model frequency patch corresponding to the PSR having a value greater than the reference value α among the plurality of PSRs, but may not update the model frequency patch corresponding to the remaining PSRs.

6 is a flowchart illustrating an example of a method for tracking an object.

In operation 601 , the processor 130 may obtain a current image frame from the image sensor 110 .

In step 602 , the processor 130 may obtain the current distance by measuring the distance to the object using the range finder 120 .

In operation 603 , the processor 130 may determine a current image patch that is a subset of the current image frame based on the current distance.

For example, when the current image frame is the first image frame, the current image patch may be determined by a user's setting or a method in which the processor 130 identifies an object in the image.

As another example, when the current image frame is the nth image frame, the current image patch may be determined based on a previous tracking point determined in the previous image frame (n−1th image frame).

The size of the current image patch may be determined based on the current distance, and may be determined to be the same as the size of the previous image patch or may be changed differently.

In operation 604 , the processor 130 may generate a plurality of current feature patches by extracting different features from the current image patch.

For example, the features may be a gray level, a gray lateral edge, and a gray longitudinal edge, and the plurality of current feature patches may be a gray level image, a gray lateral edge image, and a gray longitudinal edge image.

In operation 605 , the processor 130 may generate a plurality of current frequency patches by transforming the plurality of current feature patches into a frequency domain.

In step 606, for each of the plurality of current frequency patches, the processor 130 may generate a plurality of correlation coefficient patches by obtaining correlation coefficients with a corresponding model frequency patch from among a plurality of model frequency patches generated in advance. can

For each of the plurality of current frequency patches, the processor 130 may generate a plurality of correlation coefficient patches by transforming an element-wise product between elements with a corresponding model frequency patch into a spatial domain.

In operation 607, the processor 130 may obtain a plurality of maximum correlation coefficients by determining a correlation coefficient having a maximum value for each of the plurality of correlation coefficient patches.

In operation 608 , the processor 130 may obtain a plurality of preliminary tracking points from positions of pixels corresponding to a plurality of maximum correlation coefficients.

In step 609 , the processor 130 may obtain a plurality of PSRs by calculating a peak to sidelobe ratio (PSR) for each of the plurality of correlation coefficient patches.

The processor 130 calculates the standard deviation and average of values (ie, correlation coefficients) of pixels included in an area excluding the kxk size area centered on the preliminary tracking point among the qxq size area centered on the preliminary tracking point. You can get PSR from

In operation 610, the processor 130 may select at least one maximum correlation coefficient from among the plurality of maximum correlation coefficients based on the plurality of PSRs and may select at least one preliminary tracking point from among the plurality of preliminary tracking points.

If there is a PSR having a value greater than the reference value among the plurality of PSRs, the processor 130 may select a maximum correlation coefficient and a preliminary tracking point corresponding to the corresponding PSR.

In step 611 , the processor 130 may determine the current tracking point by calculating a weighted average of the selected at least one preliminary tracking point using the at least one selected maximum correlation coefficient.

7 is a flowchart illustrating an example of a method for tracking an object.

In operation 701 , the processor 130 may acquire a plurality of current image frames by acquiring a current image frame from each of the plurality of image sensors 110 .

In operation 702 , the processor 130 may obtain the current distance by measuring the distance to the object using the range finder 120 .

In operation 703 , the processor 130 may obtain a plurality of current image patches by determining a current image patch that is a subset for each of the plurality of current image frames based on the current distance.

In operation 704 , the processor 130 may generate a plurality of current frequency patches by transforming the extracted features for each of the plurality of current image patches into a frequency domain.

For example, the plurality of current image patches may be a gray image patch, a color image patch, a near-infrared image patch, the features may be a longitudinal edge, R(Red), and a near-infrared level, and the plurality of current feature patches may be a gray species. It may be a directional edge image, an R-channel image, or a near-infrared level image.

In operation 705 , the processor 130 may generate a plurality of current frequency patches by transforming the plurality of current feature patches into a frequency domain.

In step 706, the processor 130 generates a plurality of correlation coefficient patches by obtaining correlation coefficients with a corresponding model frequency patch among a plurality of model frequency patches generated in advance for each of the plurality of current frequency patches. can

For each of the plurality of current frequency patches, the processor 130 may generate a plurality of correlation coefficient patches by transforming an element-wise product between elements with a corresponding model frequency patch into a spatial domain.

In operation 707 , the processor 130 may obtain a plurality of maximum correlation coefficients by determining a correlation coefficient having a maximum value for each of the plurality of correlation coefficient patches.

In operation 708 , the processor 130 may obtain a plurality of preliminary tracking points from positions of pixels corresponding to the plurality of maximum correlation coefficients.

In operation 709 , the processor 130 may obtain a plurality of PSRs by calculating a peak to sidelobe ratio (PSR) for each of the plurality of correlation coefficient patches.

The processor 130 calculates the standard deviation and average of values (ie, correlation coefficients) of pixels included in an area excluding the kxk size area centered on the preliminary tracking point among the qxq size area centered on the preliminary tracking point. You can get PSR from

In operation 710 , the processor 130 may select at least one maximum correlation coefficient from among the plurality of maximum correlation coefficients based on the plurality of PSRs and select at least one preliminary tracking point from among the plurality of preliminary tracking points.

If there is a PSR having a value greater than the reference value among the plurality of PSRs, the processor 130 may select a maximum correlation coefficient and a preliminary tracking point corresponding to the corresponding PSR.

In operation 711 , the processor 130 may determine the current tracking point by calculating a weighted average of the selected at least one preliminary tracking point using the at least one selected maximum correlation coefficient.

8 is a graph illustrating a tracking error of an object.

8 is a graph for comparing a tracking error (SKCF) of tracking an object using the method according to the present embodiment with a tracking error (KCF) of tracking an object using the conventional method. The tracking error is a root mean square (RMS) error between the object (eg, the center of the object) and the current tracking point (eg, the center of the current tracking point).

From the graph of FIG. 8 , it can be seen that the tracking error (SKCF) of the method according to the present embodiment is lower than the tracking error (KCF) of the conventional method, and from this, tracking performance of an object using the method according to the present embodiment can be seen to be improved.

Table 1 is a table for comparing the tracking error of tracking the object using the method according to the present embodiment with the tracking error of tracking the object using the conventional method.

Method 1 (KCF) Method 2 (SKCF1) 3rd method (SKCF2) RMS error 13.729 [Pixels] 12.2841 [Pixels] 10.9909 [Pixels]

In the experiment, an image frame obtained from a gray image sensor was used. In the first method, the subject was tracked using a conventional method. That is, in the first method, the size of the image patch or the frequency patch is not changed, and the fusion of preliminary tracking points based on different characteristics is not performed. In the second method, the size of the image patch or frequency patch is changed according to the present embodiment, but the tracking point is determined based on only the gray level, and fusion of preliminary tracking points based on different characteristics is not performed. In the third method, the size of the image patch or frequency patch was changed according to the method according to the present embodiment, and fusion of preliminary tracking points based on the gray level image, the gray longitudinal edge image, and the gray lateral edge image was performed. From Table 1, it can be seen that the RMS error of the second method is lower than that of the first method. This means that according to the present embodiment, by changing the size of the image patch or the frequency patch based on the distance conversion factor, the object can be tracked by reflecting the change in the distance between the apparatus 100 and the object.

In addition, from Table 1, it can be seen that the RMS error of the third method is lower than that of the first method-2 method. This is achieved by changing the size of an image patch or a frequency patch based on a distance conversion factor according to the present embodiment, and fusing preliminary tracking points based on different characteristics, thereby changing the distance between the device 100 and the object and the surroundings. This means that the object can be tracked by reflecting changes in the environment.

Although the embodiments have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements by those skilled in the art using the basic concept of the present invention as defined in the following claims are also included in the scope of the present invention. belongs to

100: device
110: image sensor
120: range finder
130: processor

Claims (12)

delete delete delete delete delete A method for tracking an object, comprising:
obtaining a plurality of current image frames by obtaining a current image frame from each of a plurality of image sensors of different types;
obtaining a current distance by measuring a distance to an object using a range finder;
obtaining a plurality of current image patches by determining a subset current image patch for each of the plurality of current image frames based on the current distance;
generating a plurality of current frequency patches by converting features extracted for each of the plurality of current image patches into a frequency domain;
generating a plurality of correlation coefficient patches by obtaining correlation coefficients for each of the plurality of current frequency patches with a corresponding model frequency patch from among a plurality of previously generated model frequency patches;
obtaining a plurality of maximum correlation coefficients by determining a correlation coefficient having a maximum value for each of the plurality of correlation coefficient patches;
obtaining a plurality of preliminary tracking points from positions of pixels corresponding to the plurality of maximum correlation coefficients;
obtaining a plurality of PSRs by calculating a peak to sidelobe ratio (PSR) for each of the plurality of correlation coefficient patches;
selecting at least one maximum correlation coefficient from among the plurality of maximum correlation coefficients and selecting at least one preliminary tracking point from among the plurality of preliminary tracking points based on the plurality of PSRs; and
determining a current tracking point by calculating a weighted average for the selected at least one preliminary tracking point using the selected at least one maximum correlation coefficient;
Obtaining the plurality of current image frames comprises:
obtaining the plurality of current image frames from the plurality of different types of image sensors comprising at least two of a CCD image sensor, a gray scale image sensor, a MWIR image sensor, and a SWIR image sensor;
The step of selecting the at least one preliminary tracking point from among the plurality of preliminary tracking points includes:
If any one of the plurality of PSRs is smaller than a predetermined reference value, the at least one maximum correlation coefficient is selected, excluding the maximum correlation coefficient corresponding to the one of the plurality of maximum correlation coefficients, and selecting the at least one preliminary tracking point excluding the preliminary tracking point corresponding to the one PSR from among the plurality of preliminary tracking points. 7. The method of claim 6,
Determining the plurality of current image patches comprises:
obtaining a distance change factor by calculating a ratio of the current distance to a reference distance obtained by measuring a distance to the object corresponding to a predetermined reference image frame; and
determining a current size, which is the size of the plurality of current image patches, by multiplying the distance varying factor by a reference size that is a size of a reference image patch that is a subset of the reference image frame. 7. The method of claim 6,
The step of obtaining the plurality of correlation coefficient patches,
changing the size of the plurality of model frequency patches to be the same as the size of the plurality of current frequency patches. 7. The method of claim 6,
The step of obtaining the plurality of correlation coefficient patches,
obtaining, for each of the plurality of current frequency patches, the plurality of correlation coefficient patches by transforming an element-wise product between elements with a corresponding model frequency patch into a spatial domain; . 7. The method of claim 6,
The step of selecting the at least one preliminary tracking point comprises:
and selecting a maximum correlation coefficient and a preliminary tracking point corresponding to a PSR exceeding a reference value among the plurality of PSRs. delete A device for tracking an object, comprising:
a plurality of different types of image sensors;
rangefinder; and
including a processor;
The processor is
obtaining a plurality of current image frames by obtaining a current image frame from each of the plurality of image sensors of different types;
Obtain the current distance by measuring the distance to the object using a range finder,
obtaining a plurality of current image patches by determining a subset current image patch for each of the plurality of current image frames based on the current distance;
generating a plurality of current frequency patches by transforming the extracted features for each of the plurality of current image patches into a frequency domain;
For each of the plurality of current frequency patches, by obtaining correlation coefficients with a corresponding model frequency patch from among a plurality of model frequency patches generated in advance, a plurality of correlation coefficient patches are generated,
By determining a correlation coefficient having a maximum value for each of the plurality of correlation coefficient patches, a plurality of maximum correlation coefficients is obtained,
obtaining a plurality of preliminary tracking points from positions of pixels corresponding to the plurality of maximum correlation coefficients;
By calculating a peak to sidelobe ratio (PSR) for each of the plurality of correlation coefficient patches, a plurality of PSRs is obtained,
selecting at least one maximum correlation coefficient from among the plurality of maximum correlation coefficients and selecting at least one preliminary tracking point from among the plurality of preliminary tracking points based on the plurality of PSRs;
determine a current tracking point by calculating a weighted average for the selected at least one preliminary tracking point using the selected at least one maximum correlation coefficient;
The different types of the plurality of image sensors include at least two of a CCD image sensor, a gray scale image sensor, a MWIR image sensor, and a SWIR image sensor,
The processor is
If any one of the plurality of PSRs is smaller than a predetermined reference value, the at least one maximum correlation coefficient is selected, excluding the maximum correlation coefficient corresponding to the one of the plurality of maximum correlation coefficients, and select the at least one preliminary tracking point except for the preliminary tracking point corresponding to the one of the plurality of preliminary tracking points.

Images