KR102313795B1 - Video stabilization method based on SURF for enhancing accuracy and reliability - Google Patents

Abstract

The present invention relates to a video stabilization method based on speed-up robust feature (SURF) tracking, capable of maximizing accuracy and reliability of feature point tracking. The method is configured to: generate restoration information including the position (x,y) of a corresponding feature point and a position (x',y') of a second frame by using a transformation matrix (H) estimated by a transformation matrix estimator when a first error occurs in a first mismatch determination step (S70) and a first mismatch restoration step (S80); and generate a new tracker corresponding to the feature point when a second error occurs in a second mismatch determination step (S90) and a second mismatch restoration step (S100) at the same time. Since the generated restoration information and the new tracker are utilized in a Kalman filter update step (S110) and an image mapping step (S120), whether the feature points are mismatched is accurately determined to restore mismatching, thereby increasing accuracy of image stabilization. In addition, the Kalman filter update step (S110) is configured to update the weight of the Kalman filter to reflect the feature point matching by the transform matrix (H) estimated by the transform matrix estimation step (S60), the restoration information generated by the first mismatch restoration step (S80), and the new tracker generated by the second mismatch restoration step (S100), thereby maximizing the accuracy and reliability of feature point tracking. ​

Description

SURF feature point tracking-based image stabilization method with increased accuracy and reliability {Video stabilization method based on SURF for enhancing accuracy and reliability}

The present invention relates to image stabilization based on SURF feature point tracking with improved accuracy and reliability. Specifically, when mapping an image, the weight of a Kalman filter is updated to reflect a transformation matrix used for mapping, but first, 2 It relates to a SURF keypoint tracking-based image stabilization method that can improve the accuracy and reliability of image stabilization and at the same time improve the processing speed of image stabilization by compensating for keypoint tracking even when an error occurs.

In general, the real-time feature point matching technique for finding the same feature points between different images based on feature points extracted from input images is to extract feature points from each image to generate descriptive markers, and then compare and Through collation, the same object can be identified by finding feature points with the same descriptive mark.

In this case, the feature point is defined as representative pixels capable of expressing objects in the image, and the descriptive mark expresses various information about the feature point.

Meanwhile, with the recent spread of compact cameras that are easy to operate and miniaturized with mobile cameras, digital cameras, camcorders, drones, and wearable cameras, the demand for personal cameras is increasing, and at the same time, due to the development of cloud services, As sharing becomes more popular, the demand for high-quality image acquisition increases, and accordingly, various studies on image stabilization methods for image stabilization are being conducted.

1 is a flowchart illustrating a video stabilization method using particle-based feature points disclosed in Korean Patent Registration No. 10-1851896 (Title of the Invention: Method and Apparatus for Video Stabilization Using Particle-Based Feature Points).

The video stabilization method using particle-based feature points of FIG. 1 (hereinafter referred to as the first prior art) (S900) includes the steps of estimating motion between a plurality of adjacent frames and dividing the input image into regions (S910), and a plurality of frames generating a flat area map using (S915), extracting a plurality of feature points using the flat area map (S920), and generating particle keypoints in a flat background area using the flat area map (S925), redefining a feature point using the generated particle keypoint and a plurality of feature points, estimating a camera path using the redefined feature point (S930), and smoothing the estimated camera path (S935); The step (S940) is to generate a stabilized image by warping the image by applying the smoothed camera path.

Also, in step S930, descriptors for matching are generated using the distances between the generated particle keypoints and the plurality of feature points, and at the same time, the keypoints are matched using the generated descriptors to redefine the feature points.

The first prior art S900 configured in this way can effectively remove video shake through adaptive camera path estimation according to the divided area by redefining the characteristics of an image using particle keypoints, and also feature points in a flat area. It has the advantage of solving the problem of the existing stabilization technique in which homography estimation performance is deteriorated due to erroneous detection.

However, in the first prior art (S900), when an error in detecting a feature point detected in the previous frame but not detected in the current frame occurs or an error in detecting a feature point detected in the next frame but not detected in the previous frame occurs, Since a technique and a method for resolving the error of the metric are not described at all, the accuracy and reliability of the feature point tracking are low.

2 is a block diagram illustrating an image stabilization apparatus disclosed in Korean Patent Registration No. 10-0985805 (Title of the Invention: Image stabilization apparatus and method using an adaptive Kalman filter).

The image stabilization apparatus (hereinafter referred to as the second prior art) 100 of FIG. 2 includes a feature point tracking unit 110 , a unit motion measurement unit 122 , a global motion measurement unit 124 , a gain adjustment unit 126 , and Kalman. It consists of a filter unit 128 and a motion compensation unit 130 .

The feature point tracking unit 110 extracts a feature point from a reference image frame among a plurality of image frames constituting the input original image, and then searches for a corresponding point corresponding to the feature point on a current image frame temporally continuous to the reference image frame.

The unit motion measurement unit 122 measures a unit motion that is a motion between temporally adjacent image frames with respect to an image frame between the reference image frame and the current image frame.

The global motion measurement unit 124 measures the global motion with respect to the current image frame by accumulating unit motions.

The gain adjusting unit 126 determines the gain of the Kalman filter according to the frequency with which the sign of each measured unit motion is changed for a plurality of image frames input during the measurement time.

The Kalman filter unit 128 estimates the intentional motion from which the noise component is removed from the global motion.

The motion compensator 130 calculates an unintentional motion representing a noise component based on the difference between the global motion and the intentional motion, and generates a resultant image frame by compensating for the unintentional motion with respect to the current image frame.

The second prior art 100 configured in this way estimates the motion of the image using the feature points extracted from the reference image frame and the corresponding points, thereby compensating for the rotational component as well as the translational component in the parallel direction of the motion shown in the image. This has the advantage of accurately estimating intentional motion by using a Kalman filter corresponding to the frequency of sign change of a unit motion with respect to an image frame when estimating intentional motion.

However, in the second prior art 100, the feature point tracking unit 110 extracts feature points of temporally successive frames and estimates motion through comparison of these feature points, but after extracting the feature points of the reference image frame, reference Since it is configured to estimate motion through searching for corresponding points corresponding to feature points on frames temporally successive to an image frame, it has a disadvantage in that computational throughput and computational processing time increase.

In addition, in the second prior art 100, when the scale and rotation of the feature points change due to a change in scale or rotation between frames, their technicians do not consider the scale and rotation characteristics, so the feature points cannot be accurately matched. This happens.

Also, in the second prior art 100 , like the first prior art S900 , an error occurs in that a feature point detected in the previous frame but not detected in the current frame is detected, or is detected in the re-frame but not detected in the previous frame. When an error in which a keypoint is detected occurs, there is a disadvantage in that accuracy and reliability of tracking the keypoint are lowered because a technique and method for solving the error are not described at all.

The present invention is to solve this problem, and the present invention solves two frames (F1) and (F2) input using a Speed-Up Robust Features (SURF) algorithm and an Approximate Nearest Neighbor (ANN) search algorithm. An object of the present invention is to provide a SURF feature point tracking-based image stabilization method capable of compensating for both rotation and translation components as well as increasing the accuracy and reliability of feature point matching by extracting and matching feature points at the same time.

In addition, another problem to be solved in the present invention is a feature point that was detected in the previous frame F1 through the first mismatch determination step S70 and the first mismatch restoration step S80, but is not detected in the current frame F2. When the first error occurs, the position (x', y') on the second frame F2 corresponding to the position (x, y) of the corresponding feature point using the transformation matrix H estimated by the transformation matrix estimator ), the restoration information including the position (x, y) of the corresponding feature point and the position (x', y') of the second frame is generated, and the generated restoration information is updated in the Kalman filter update step (S110) and the image It is intended to provide an image stabilization method based on SURF feature point tracking, which is configured to be utilized in the mapping step (S120), thereby accurately determining whether the feature points are mismatched and restoring the image stabilization accuracy.

In addition, another problem of the present invention is a feature point that is detected in the current frame F2 through the second mismatch determination step S90 and the second mismatch restoration step S100, but is not detected in the previous frame F1. When an existing second error occurs, a new tracker corresponding to the corresponding feature is created, and the created new tracker is configured to be utilized in the Kalman filter update step (S110) and the image mapping step (S120), thereby further improving the accuracy of image stabilization. An object of the present invention is to provide an image stabilization method based on SURF feature point tracking that can be improved.

In addition, another solution to the present invention is that the Kalman filter update step (S110) is performed by matching the feature points by the transform matrix (H) estimated by the transform matrix estimation step (S60) and the first mismatch restoration step (S80). SURF key point tracking base that can maximize the accuracy and reliability of key point tracking by being configured to update the weight of the Kalman filter so that the generated restoration information and the new tracker generated by the second mismatch restoration step (S100) are reflected It is intended to provide an image stabilization method.

In an image stabilization method (S1) based on SURF feature point tracking for stabilizing an image by tracing feature points of an image input from a camera, the solution of the present invention for solving the above problem is: the SURF feature point tracking-based image stabilization method (S1) ) is a step 10 (S10) of receiving a frame from the camera; Step 20 (S20) of extracting feature points by analyzing the frame input by the frame input step (S10) using a preset SURF (Speed-Up Robust Features) algorithm; a step 30 (S30) of removing noise by applying a Kalman filter to the feature points extracted by the step 20 (S20); Using a preset Approximate Nearest Neighbor (ANN) search algorithm, the feature points of the second frame F2 that is the current frame that has passed the Kalman filter by step 30 (S30) and the feature points of the first frame F1 that is the previous frame Step 40 (S40) of detecting a motion vector by comparing the positional coordinates of the same feature points after matching the feature points by removing the feature points that are not identical while leaving the same feature points; Step 60 (S60) of estimating a transformation matrix ((Homography, H) that is a corresponding relation of feature point matching between the first and second frames (F1) and (F2) by using the motion vector detected in step 40 (S40); Step 120 (S120) of generating a mapping image F′ in which the feature points of the first frame F1 are mapped to the second frame F2 by using the transformation matrix H estimated in step 60 (S60) , wherein the SURF feature point tracking-based image stabilization method (S1) proceeds after step 60 (S60), and compares the feature points detected in the first frame F1 and the feature points detected in the second frame F2. In comparison, a step 70 (S70) of determining whether a first error occurs in which a first mismatch point, which is a feature point detected in the first frame F1 but not detected in the second frame F2, occurs (S70); ), the transformation matrix (H) estimated by the step 60 (S60) at the position (x, y) of the first mismatch point on the first frame F1. to extract the position (x', y') of the first mismatch point on the second frame (F), and (x, y), (x', y') coordinates of the first mismatch point Step 80 (S80) of generating the restoration information including

delete

In the present invention, the SURF feature point tracking-based image stabilization method (S1) proceeds after the step 70 (S70) or the step 80 (S80), and includes the feature points detected in the first frame F1 and the second frame ( Comparing the feature points detected in F2), it is determined whether a second error occurs in which a second mismatch point, which is a feature point detected in the second frame F2 but not detected in the first frame F1, occurs (S90). ); It proceeds when it is determined that a second error has occurred in the step 90 (S90), and further includes a step 100 (S100) of creating a new tracker corresponding to the second mismatch point, and the step 120 (S120) is the mapping When generating an image, it is preferable to utilize the new tracker created by the step 100 (S100).

In addition, in the present invention, the SURF feature point tracking-based image stabilization method (S1) further includes a step 110 (S110) performed after the step 90 (S90) or the step 100 (S100), and the step 110 (S110) is Kalman filter so that the feature point matching by the transformation matrix (H) by the step 60 (S60), the restoration information generated by the step 80 (S80), and the new tracker generated by the step 100 (S100) are reflected It is desirable to update the weights.

In addition, in the present invention, the SURF feature point tracking-based image stabilization method (S1) further includes a step 50 (S50) performed after the step 40 (S40), and the step 50 (S50) is performed in the step 40 (S40). After calculating the error value of each of the motion vectors detected by the motion vector, if the calculated error value is greater than or equal to a set value (TH, Threshold), it is preferable to determine the corresponding value as an outlier and remove it.

According to the present invention having the above problems and solutions, feature points of two frames (F1) and (F2) inputted using a Speed-Up Robust Features (SURF) algorithm and an Approximate Nearest Neighbor (ANN) search algorithm are extracted and at the same time By being configured to match, both rotation and translation components can be compensated, and the accuracy and reliability of feature point matching can be improved.

In addition, according to the present invention, through the first mismatch determination step (S70) and the first mismatch restoration step (S80), a first feature point detected in the previous frame F1 but not detected in the current frame F2 exists. When an error occurs, the position (x', y') on the second frame F2 corresponding to the position (x, y) of the corresponding feature point is extracted using the transformation matrix (H) estimated by the transformation matrix estimator. After that, restoration information including the position (x, y) of the corresponding feature point and the position (x', y') of the second frame is generated, and the generated restoration information is updated in the Kalman filter update step (S110) and the image mapping step ( By being configured to be utilized in S120), it is possible to accurately determine whether the feature points are mismatched and restore them to increase the accuracy of image stabilization.

In addition, according to the present invention, through the second mismatch determination step ( S90 ) and the second mismatch restoration step ( S100 ), a feature point detected in the current frame F2 but not detected in the previous frame F1 is present. When an error occurs, a new tracker corresponding to the corresponding feature is created, and the created new tracker is configured to be utilized in the Kalman filter update step (S110) and the image mapping step (S120), so that the accuracy of image stabilization can be further improved. do.

In addition, according to the present invention, in the Kalman filter update step (S110), the feature point matching by the transform matrix (H) estimated by the transform matrix estimation step (S60) and the restoration information generated by the first mismatch restoration step (S80) , it is possible to maximize the accuracy and reliability of feature point tracking by updating the weight of the Kalman filter to reflect the new tracker generated by the second mismatch restoration step (S100).

1 is a flowchart illustrating a video stabilization method using particle-based feature points disclosed in Korean Patent Registration No. 10-1851896 (Title of the Invention: Method and Apparatus for Video Stabilization Using Particle-Based Feature Points).
2 is a block diagram illustrating an image stabilization apparatus disclosed in Korean Patent Registration No. 10-0985805 (Title of the Invention: Image stabilization apparatus and method using an adaptive Kalman filter).
3 is a flowchart illustrating an image stabilization method based on SURF feature point tracking according to an embodiment of the present invention.
FIG. 4 is a block diagram conceptually illustrating the operation process of FIG. 3 .
5 (a) and (b) are exemplary diagrams showing the feature points of each frame extracted by the SURF feature point extraction step of FIG. 3 .
6 is a block diagram illustrating the step of extracting the SURF feature point of FIG. 3 .
7 is an exemplary diagram for explaining the ANN search algorithm applied to the motion vector detection step of FIG. 3 .
FIG. 8 is an exemplary diagram for explaining a motion vector detected by the motion vector detection step of FIG. 3 .
9 is an exemplary diagram illustrating a mapping image generated by the image mapping step of FIG. 3 .

Hereinafter, the present invention will be described with reference to the accompanying drawings.

3 is a flowchart illustrating an image stabilization method based on SURF feature point tracking according to an embodiment of the present invention, and FIG. 4 is a block diagram conceptually illustrating the operation process of FIG. 3 .

The SURF feature point tracking-based image stabilization method (S1), which is an embodiment of the present invention, extracts feature points of each of the two input frames (F1) and (F2), and then uses the extracted feature point information for the two frames (F1), ( This is a method for estimating a transformation matrix (H) that is a matching correspondence between F2) and matching two frames (F1) and (F2) according to the estimated transformation matrix.

In addition, the SURF feature point tracking-based image stabilization method (S1) of the present invention, as shown in FIGS. 3 and 4, includes a frame input step (S10), a SURF feature point extraction step (S20), a Kalman filter prediction step (S30), and a motion Vector detection step (S40), outlier filtering step (S50), transformation matrix estimation step (S60), first mismatch determination step (S70), first mismatch restoration step (S80), second mismatch determination step ( S90), a second mismatch restoration step (S100), a Kalman filter update step (S110), and an image mapping step (S120).

The platform providing the service of the SURF feature point tracking-based image stabilization method (S1) of the present invention may be implemented as hardware such as a controller and a terminal, or may be implemented as software such as an application program or an application.

The frame input step S10 is a step of receiving a frame of an image photographed from an external camera.

In the SURF feature point extraction step (S20), the size and rotation of each of the previous frame (F1) and the current frame (F2) input through the frame input step (S10) using a preset SURF (Speed-Up Robust Features) algorithm This is a step of extracting invariant feature point candidates.

Typically, a Speed-Up Robust Features (SURF) algorithm generates an integral image with respect to an input image. In this case, if the integral image is used, it is possible to obtain the sum of all pixels in the selected area through four operations, no matter what size of the image area is selected.

In addition, the SURF algorithm uses a hessian matrix defined by Equation 1 below to extract feature points of an image. In this case, when a Hessian matrix is used, a determinant is obtained by using a box filter with similar results but a high operation speed without using a Gaussian filter unlike SIFT. That is, Gaussian has a real value, but the box filter can perform integer-type operation, so the operation processing speed can be increased.

In this case, Lxx denotes a convolution value between the black-and-white image luminance value at the x position and the x-direction second-order differential value of a Gaussian having a variance of σ, and the remaining Lyy and Lxy are the y-direction second-order derivatives and xy-directions. It means a convolution value with a Gaussian filter differentiated by .

Also, the SURF algorithm detects a candidate group of feature points defined by the following Equation (2) using the determinant of the Hessian matrix (Happrox) approximated by Equation (1).

In this case, Dxx, Dyy, and Dxy represent the approximated second derivatives along the x-axis and y-axis, respectively.

That is, SURF can improve the speed by simplifying the operation by using the approximated filter and the integral image.

In addition, the SURF algorithm expresses the feature of the feature point candidate group detected by Equation 2 as a feature vector that is a descriptor, and in detail, for the direction component of the feature point detected using a Haar wavelet response It can be described as a 64-dimensional vector.

In other words, in the SURF algorithm, the descriptor is expressed as 64-dimensional immutable information calculated through the result of a Har wavelet with a filter size of 2.

5 (a) and (b) are exemplary diagrams showing the feature points of each frame extracted by the SURF feature point extraction step of FIG. 3 .

In Fig. 5 (a), the feature point candidates of the first frame (F1), which is the previous frame detected by the SURF feature point extraction step (S20), are displayed, and in Fig. 4 (b), the SURF feature point extraction step (S20) is shown. Feature point candidates of the second frame F2, which is the current frame detected by , are displayed.

That is, the SURF feature point extraction step S20 extracts feature point candidate groups of each of the two input frames F1 and F2.

This SURF feature point extraction step (S20) will be described in detail with reference to FIG. 6 to be described later.

6 is a block diagram illustrating the step of extracting the SURF feature point of FIG. 3 .

As shown in FIG. 6 , the SURF feature point extraction step (S20) includes an integral image detection step (S21) of acquiring integral images for the input frames (F1) and (F2), and an integral image detection step (S21). The scale space generation step (S22) of generating a scale space while changing the size of the box filter using the integral image by x-direction Haar wavelet response and y-direction Haar wavelet response within a 6σ radius for the keypoint candidate groups selected by the keypoint candidate group selection step (S23) and the keypoint candidate group selection step (S23) The main direction detection step (S24) of adding a wavelet response and summing the vectors to detect the main direction, and the main direction detected by the main direction detecting step (S24) divided into 4X4 sub-regions based on the main direction, and then 5X5 for each area An arithmetic processing step (S25) of calculating dx and dy by applying the magnitude x- and y-direction Har wavelets, and sum(dx), sum( dy), sum(ldxl), and sum(ldyl) are calculated to obtain four features for each sub-region, and a descriptor vector of a total of 64 dimensions of 16X4 is obtained (S26).

The Kalman filter prediction step (S30) is performed by applying the Kalman filter updated by the Kalman filter update step (S120) to the feature points extracted by the SURF feature point extraction step (S20), such as shaking for each feature point. This is a step of obtaining a feature vector from which the noise component has been removed.

FIG. 7 is an exemplary diagram for explaining an ANN search algorithm applied to the motion vector detection step of FIG. 3 , and FIG. 8 is an exemplary diagram for explaining a motion vector detected by the motion vector detection step of FIG. 3 .

The motion vector detection step (S40) is the direction and magnitude of the 64-dimensional descriptor vector that is the feature point of the frames F1 and F2 detected by the SURF feature point detection step S20 and the Kalman filter prediction step S30. After comparing information, leaving identical feature points and removing non-identical feature points, a motion vector is detected by comparing the position coordinates of the same feature points.

In other words, the motion vector detection step (S40) is a SURF feature point detection step (S20) and a Kalman filter prediction step (S30) in order to obtain the parameters of movement and rotation size change between the two frames F1 and F2. Matching of feature points is performed.

In this case, the matching of each feature point between the two images is performed using a preset Approximate Nearest Neighbor (ANN) search algorithm, where ANN means a feature point having the minimum Euclidean distance between invariant descriptor vectors.

)와 쿼리(q)와 동일한 차원의 특징점 후보군인 후보셋(p1, p2, ..., pn)들이 주어졌을 때, 쿼리(q)로부터 가장 가까운 거리를 갖는 특징점 후보셋(pj)을 가장 가까운 이웃으로 결정함과 동시에 쿼리에서 가장 가까운 지점까지의 거리(r)를 c배로 곱한 값을 반환한다.That is, the motion vector detection step (S40) utilizes a preset ANN (Approximate Nearest Neighbor) search algorithm to detect the SURF feature point (S20) and the Kalman filter prediction step (S30), the detected frames (F1), (F2) Matches the feature points between, in detail, as shown in FIG. 7, query q(

) and the candidate sets (p1, p2, ..., pn) that are the feature point candidates of the same dimension as the query (q), the feature point candidate set (pj) having the closest distance from the query (q) is selected as the nearest At the same time as determining a neighbor, it returns a value multiplied by c times the distance (r) to the nearest point in the query.

Also, in the motion vector detection step S40, as shown in FIG. 8, the motion vector MV of each matched feature point is detected through the ANN search algorithm.

In the outlier filtering step (S50), after calculating an error value of each of the motion vectors (MV) detected by the motion vector detecting step (S40), if the calculated error value is greater than or equal to a set value (TH, Threshold), the corresponding value It is a step of determining that is an outlier and removing it.

The transformation matrix estimation step (S60) utilizes the motion vectors from which the outlier has been removed by the outlier filtering step (S50), and the transformation matrix (Homography, H), which is a corresponding relationship of feature point matching of the two frames (F1) and (F2) is the step of estimating

At this time, the transformation matrix H, which is the correspondence between the two frames F1 and F2 estimated by the transformation matrix estimation step S60, is defined by Equation 3 below.

’는 제1 프레임(F1)의 특징점의 위치좌표이고, ‘

’는 제2 프레임(F2)의 특징점의 위치좌표이고, 변환행렬’H’의 ‘

’는 어떤 회전각으로 회전할지를 나타내는 회전(rotation) 정보와, x,y,z 방향으로 얼마만큼 이동할지를 나타내는 평행이동(translation) 정보, x,y,z 방향으로 얼마만큼 크기를 변화시킬지를 나타내는 크기 변환(scaling) 정보를 포함한다.At this time, '

' is the position coordinate of the feature point of the first frame F1, and '

' is the position coordinate of the feature point of the second frame F2, and ' of the transformation matrix 'H'

' denotes rotation information indicating at which rotation angle to rotate, translation information indicating how much to move in the x, y, and z directions, and indicating how much to change the size in the x, y, z directions. Includes scaling information.

The first mismatch determination step S70 compares the feature points detected in the first frame F1 with the feature points detected in the second frame F2, which were detected in the previous frame F1, but were detected in the current frame F2. ), it is determined whether a first error in which a first mismatch point, which is a feature point that is not detected, occurs.

In addition, in the first mismatch determination step S70 , if it is determined that a first error having a first mismatch point has occurred, the first mismatch restoration step S80 is performed as a next step.

Also, in the first mismatch determination step S70 , if it is determined that the first error does not occur because the first mismatch point does not exist, the second mismatch determination step S90 proceeds to the next step.

The first mismatch restoration step S80 is performed when it is determined that the first error has occurred in the first mismatch determination step S70, and the transformation matrix H estimated by the transformation matrix estimation step S60 is utilized. After extracting the position (x', y') on the second frame F2 corresponding to the position (x, y) of the corresponding feature point, the position (x, y) of the first frame F1 of the first mismatching point ) and generating restoration information indicating the positions (x', y') of the second frame extracted by the transformation matrix.

At this time, the restoration information generated by the first mismatch restoration step (S80) is used in the Kalman filter update step (S110) and the image mapping step (S120).

In the second mismatch restoration step S90 , it is determined whether a second error has occurred in which a second mismatch point, which is a feature point detected in the current frame F2 , is not detected in the previous frame F1 .

Also, in the second mismatch determination step S90 , if it is determined that a second error with a second mismatch point exists, a second mismatch restoration step S100 is performed as a next step.

Also, in the second mismatch determination step S90, if it is determined that the second error does not occur because the second mismatch point does not exist, the Kalman filter update step S110 is performed as a second mismatch determination step S90. proceed with

The second mismatch restoration step S100 is performed when it is determined that the second error has occurred in the second mismatch determination step S90. This is the step to create a new Tracker.

At this time, the new tracker generated by the second mismatch restoration step (S100) is utilized in the Kalman filter update step (S110) and the image mapping step (S120).

The Kalman filter update step (S110) includes matching feature points by the transform matrix (H) estimated by the transform matrix estimation step (S60), the restoration information generated by the first mismatch restoration step (S80), and the second mismatch The weight of the Kalman filter is updated so that the new tracker generated by the restoration step S100 is reflected.

In addition, the updated Kalman filter is stored by the Kalman filter update step S110 and is utilized in the aforementioned Kalman filter prediction step S30.

9 is an exemplary diagram illustrating a mapping image generated by the image mapping step of FIG. 3 .

The image mapping step (S120) includes the transformation matrix (H) estimated by the transformation matrix estimation step (S60), the restoration information generated by the first mismatch restoration operation (S80), and the second mismatch restoration operation (S100). As shown in FIG. 9 by utilizing the new tracker generated by

10 is a block diagram illustrating an image stabilization system based on SURF feature point tracking to which the present invention is applied.

The SURF feature point tracking-based image stabilization system 1 of FIG. 10 is a system to which the above-described SURF feature point tracking-based image stabilization method S1 of FIGS. 2 to 9 is applied.

In addition, as shown in FIG. 10 , the SURF key point tracking-based image stabilization system 1 includes a control unit 30, a memory 31, a data input/output unit 32, a SURF key point extraction unit 33, and a Kalman filter unit 934. ), a motion vector detection unit 35 , an outlier filtering unit 36 , a transformation matrix estimation unit 37 , a mismatching check and restoration unit 38 , a Kalman filter update unit 39 , and an image mapping unit 40 . is done

The control unit 30 is an OS (Operating System) of the SURF feature point tracking-based image stabilization system 1, and the control target 31, (32), (33), (34), (35), (36), ( 37), (38), (39), and (40) manage and control the operation.

Also, the control unit 30 inputs adjacent frames F1 and F2 input through the data input/output unit 32 to the SURF feature point extraction unit 33 and temporarily stores them in the memory 31 .

In addition, the control unit 30 stores the Kalman filter updated by the Kalman filter update unit 39 in the memory 31 .

The memory 31 stores the mapping image generated by the image mapping unit 40 .

In addition, the Kalman filter updated by the Kalman filter update unit 39 is stored in the memory 31 .

In addition, a preset SURF algorithm is applied to the memory 31 .

Also, a preset Approximate Nearest Neighbor (ANN) search algorithm is stored in the memory 31 .

The data input/output unit 32 inputs and outputs data to and from an external device.

The SURF feature point extraction unit 33 is a processor that performs the arithmetic processing of the SURF feature point extraction step S20 of FIGS. 3 and 6 described above, and in detail, through the data input/output unit 32 using a preset SURF algorithm. The size and rotation invariant feature point candidates for each of the two input frames F1 and F2 are extracted.

The Kalman filter unit 34 is a processor that performs the arithmetic processing of the Kalman filter prediction step S30 of FIG. 3 described above, and in detail, a Kalman filter preset to the key points extracted by the SURF key point extraction unit 33 By applying , we obtain a feature vector from which noise components such as vibration for each feature point are removed.

The motion vector detection unit 35 is a processor that performs the arithmetic processing of the motion vector detection step S40 of FIG. 3 described above, and in detail, detected by the SURF feature point detection module 33 and the Kalman filter unit 34 . After comparing the direction and size information of the 64-dimensional descriptor vector that is the feature point of the two frames (F1) and (F2), leaving the same feature point and removing the feature point that is not the same, the motion through the comparison of the position coordinates of the same feature points A motion vector is detected.

The outlier filtering unit 36 is a processor that performs the arithmetic processing of the outlier filtering step S50 of FIG. 3 described above, and in detail, each of the motion vectors MV detected by the motion vector detection unit 35 . After calculating the error value of , if the calculated error value is greater than or equal to the set value (TH, Threshold), the value is determined as an outlier and is removed.

The transformation matrix estimating unit 37 is a processor that performs the arithmetic processing of the transformation matrix estimation step S60 of FIG. 3 described above, and in detail, the motion vectors from which the outliers are removed by the outlier filtering unit 36 are calculated. Using this method, a transformation matrix (Homography, H), which is a corresponding relationship of feature point matching of the two frames (F1) and (F2), is estimated.

The mismatch check and restore unit 38 is the first mismatch determination step (S70), the first mismatch recovery step (S80), the second mismatch determination step (S90) and the second mismatch recovery of FIG. It is a processor that performs the arithmetic processing of step S100, and specifically, when the above-described first mismatching point is detected, while generating restoration information for the first mismatching point and at the same time a second mismatching point is detected , create a new tracker corresponding to the second mismatch point.

The Kalman filter update unit 39 is a processor that performs the arithmetic processing of the Kalman filter update step S110 of FIG. 3 described above, and in detail, the Kalman filter update unit 39 is applied to the transformation matrix H estimated by the transformation matrix estimator 37. The weight of the Kalman filter is updated so that the matching of the key points by , and the restoration information or the new tracker generated by the mismatch check and restoration unit 38 are reflected.

The image mapping unit 40 is a processor that performs the arithmetic processing of the image mapping step S120 of FIG. 3 described above, and the transformation matrix H estimated by the transformation matrix estimator 37 and the mismatching check and restoration The mapping image F' is generated by mapping the feature points of the first frame F1 to the second frame F2 using the restored information detected by the unit 38 or a new tracker.

As described above, the SURF feature point tracking-based image stabilization method (S1), which is an embodiment of the present invention, uses two frames (F1) and (F2) input using a Speed-Up Robust Features (SURF) algorithm and an Approximate Nearest Neighbor (ANN) search algorithm. ) by extracting and matching the feature points at the same time, it is possible to compensate for both rotation and translation components, as well as to increase the accuracy and reliability of feature point matching.

In addition, the SURF feature point tracking-based image stabilization method (S1) of the present invention was detected in the previous frame (F1) through the first mismatch determination step (S70) and the first mismatch restoration step (S80), but the current frame (F2) When a first error in which a feature point that is not detected occurs in , the transformation matrix H estimated by the transformation matrix estimator is used in the second frame F2 corresponding to the position (x, y) of the corresponding feature point. After extracting the position (x', y'), restoration information including the position (x, y) of the corresponding feature point and the position (x', y') of the second frame is generated, and the generated restoration information is applied to the Kalman filter By being configured to be utilized in the update step (S110) and the image mapping step (S120), it is possible to accurately determine whether the feature points are mismatched and restore it, thereby increasing the accuracy of image stabilization.

In addition, the SURF feature point tracking-based image stabilization method (S1) of the present invention is detected in the current frame (F2) through the second mismatch determination step (S90) and the second mismatch restoration step (S100), but the previous frame (F1) When a second error occurs in which a feature point that is not detected in , a new tracker corresponding to the feature point is generated, and the created new tracker is configured to be utilized in the Kalman filter update step (S110) and the image mapping step (S120). It is possible to further improve the accuracy of image stabilization.

In addition, in the SURF feature point tracking-based image stabilization method (S1) of the present invention, the Kalman filter update step (S110) is the feature point matching by the transform matrix (H) estimated by the transform matrix estimation step (S60), and the first mismatch restoration It is configured to update the weight of the Kalman filter so that the restoration information generated by step S80 and the new tracker generated by the second mismatch restoration step S100 are reflected, thereby maximizing the accuracy and reliability of feature point tracking. be able to

S1:SURF feature point tracking-based image stabilization method
S10: Frame input step S20: SURF feature point extraction step
S21: Integral image detection step S22: Scale space generation step
S23: feature point candidate group selection step S24: main direction detection step
S25: operation processing step S26: descriptor acquisition step
S30: Kalman filter prediction step S40: Motion vector detection step
S50: outlier filtering step S60: transformation matrix estimation step
S70: first mismatch determination step S80: first mismatch restoration step
S90: second mismatch determination step S100: second mismatch restoration step
S110: Kalman filter update step S120: Image mapping step

Claims (5)

In the image stabilization method (S1) based on SURF feature point tracking for stabilizing an image by tracking feature points of an image input from a camera:
The SURF feature point tracking-based image stabilization method (S1) is
Step 10 (S10) of receiving a frame from the camera;
Step 20 (S20) of extracting feature points by analyzing the frame input by the frame input step (S10) using a preset SURF (Speed-Up Robust Features) algorithm;
a step 30 (S30) of removing noise by applying a Kalman filter to the feature points extracted by the step 20 (S20);
Feature points of the second frame F2 that is the current frame that has passed the Kalman filter by the step 30 (S30) using a preset Approximate Nearest Neighbor (ANN) search algorithm and the feature points of the first frame F1 that is the previous frame Step 40 (S40) of detecting a motion vector by comparing the positional coordinates of the same feature points after matching the feature points by removing the feature points that are not identical while leaving the same feature points;
Step 60 (S60) of estimating a transformation matrix ((Homography, H) that is a corresponding relation of feature point matching between the first and second frames (F1) and (F2) by using the motion vector detected in step 40 (S40);
Step 120 (S120) of generating a mapping image F′ in which the feature points of the first frame F1 are mapped to the second frame F2 by using the transformation matrix H estimated in step 60 (S60) including,
The SURF feature point tracking-based image stabilization method (S1) is
It proceeds after step 60 (S60), and is detected in the first frame F1 by comparing the feature points detected in the first frame F1 with the feature points detected in the second frame F2, but is detected in the second frame ( In F2), a step 70 (S70) of determining whether a first error occurs in which a first mismatch point, which is a feature point that is not detected, occurs;
It proceeds when it is determined that the first error has occurred in step 70 (S70), and the position (x, y) of the first mismatch point on the first frame F1 is estimated by step 60 (S60). By applying the transformation matrix (H), the position (x', y') of the first mismatch point on the second frame (F) is extracted, and (x, y), (x', y') further comprising a step 80 (S80) of generating restoration information including coordinates,
The step 120 (S120) is a SURF feature point tracking-based image stabilization method (S1), characterized in that the restoration information generated by the step 80 (S80) is utilized when the mapping image is generated. delete According to claim 1, wherein the SURF feature point tracking-based image stabilization method (S1)
It proceeds after the step 70 (S70) or the step 80 (S80), and compares the feature points detected in the first frame F1 with the feature points detected in the second frame F2 in the second frame F2. Step 90 (S90) of determining whether a second error occurs in which a second mismatch point, which is a feature point that is detected but not detected in the first frame F1, occurs;
It proceeds when it is determined that a second error has occurred in the step 90 (S90), further comprising a step 100 (S100) of generating a new tracker corresponding to the second mismatch point,
The step 120 (S120) is a SURF feature point tracking-based image stabilization method (S1), characterized in that the new tracker generated by the step 100 (S100) is used when the mapping image is generated. The method of claim 3, wherein the SURF feature point tracking-based image stabilization method (S1) further comprises a step 110 (S110) performed after the step 90 (S90) or the step 100 (S100),
The step 110 (S110) is
Kalman filter so that the feature point matching by the transformation matrix (H) by the step 60 (S60), the restoration information generated by the step 80 (S80), and the new tracker generated by the step 100 (S100) are reflected SURF feature point tracking-based image stabilization method (S1), characterized in that the weight (weight) is updated. 5. The method of claim 4, wherein the SURF feature point tracking-based image stabilization method (S1) further comprises a step 50 (S50) performed after the step 40 (S40),
The step 50 (S50) is
After calculating the error value of each of the motion vectors detected in step 40 (S40), if the calculated error value is greater than or equal to the set value (TH, Threshold), the value is determined as an outlier and removed SURF feature point tracking-based image stabilization method (S1) characterized.

Images