KR101212316B1 - Movement path extracting method of center points of frame using spatial transformation and motion estimation - Google Patents

Abstract

PURPOSE: A movement path extracting method for center points of a frame using spatial transformation and motion estimation is provided to specify the outline of the surroundings of an object instead of a segment by using a specified route, to search for some information for object and to show a curved text line. CONSTITUTION: A current frame and a previous frame are extracted by sampling an input frame(S10). A SURF(Speeded-Up Robust Feature) characteristic point and a descriptor for the current frame are extracted(S20). A space transformation parameter is estimated(S30). A new position of the center point of the current frame is calculated(S40). By using estimation of block motion of the new location surroundings of the center point, the new position of the center point is redefined(S50). [Reference numerals] (AA) Input image sequence; (BB) Result path of a final frame; (S10) Sampling of an input frame; (S20) Extracting a SURF(Speeded-Up Robust Feature) characteristic point and a descriptor; (S30) Estimating a space transformation parameter; (S40) Calculating the new location of the center point; (S50) Redefining the new position of the center point

Description

Movement Path Extracting Method of Center Points of Frame Using Spatial Transformation and Motion Estimation

The present invention relates to a method for extracting a moving path of a frame center point using spatial transform and motion estimation. More particularly, the present invention relates to a method of extracting a path connecting a center point of an image sequence captured by a camera embedded in a mobile phone. This paper relates to a method of extracting a moving path of a frame center point using spatial transform and motion estimation that can obtain accurate and stable results by using the speeded-up robust feature to estimate the spatial transform parameter between the previous frame and the current frame. .

Recently, cell phones are widely used in everyday life. For example, in addition to telephony, the benefits of mobile phones include the ability to use a variety of applications such as games, web browsing, image and video processing, location services and information management. Because mobile phones are small in size, the traditional interactive method using a keyboard and mouse has the inconvenience of being limited in use.

Thus, new interactive methods for mobile phones are being developed, some using physical manipulations such as contact, pressure, tilt and implicit biometric information. In the above, the tilt input can be used for navigation menus, maps, and scrolling in 3-D scenes or documents and lists. Approaches to other interactive methods are based on mobile phone movement. One well-known method for estimating device movement, such as cell phones, is based on special motion sensors such as acceleration sensors. However, the method using the acceleration sensor has a problem that an external sensor must be installed.

Powerful desktop computers, on the other hand, often use computer vision as an interactive method. Computer vision includes multimodal interaction, gesture recognition, face tracking, and body tracking.

Recently, for mobile phones, TinyMotion system has been developed and known that can detect the movement of the hand by analyzing the image sequence captured by the built-in camera. In TinyMotion systems, it is possible for a user to use a camera to control the functionality of a mobile phone defined by the uniqueness of a series of movements, the execution of a series of movements.

The present invention has been made in view of the above, and the moving path of the frame center point using spatial transform and motion estimation that can provide a new interactive technology for portable devices by analyzing image sequences captured from the built-in camera. To provide an extraction method, the purpose is.

According to an embodiment of the present invention, a method for extracting a moving path of a frame center point using spatial transform and motion estimation includes sampling a frame to reduce complexity among input frames, and speed-up robust feature (SURF) characteristics of a current frame. Extracting points and descriptors, estimating spatial transformation parameters using feature points of previous and current frames, calculating new positions of center points of the current frame, and using motion estimation Redefining the location.

Sampling the frames above makes it possible to reduce the number of frames needed to be processed, which reduces complexity.

The SURF feature point (eg, center point) is extracted for the current frame, and the SURF feature point for the previous frame is reused.

The spatial transformation parameters are estimated using matching characteristic points to map the current frame and the previous frame.

The calculation of the new position of the center point is made using the estimated spatial transformation parameters of the positions of all the center points of the previous frame.

The step of redefining the position is made so that the motion estimation process in a small search range redefines the position of the center point in the current frame since the spatial transformation parameters are not perfect and the object shape may be distorted.

In the final frame obtained through the above steps, the path is defined as a curve connecting all the center points of the frame image together.

According to the moving path extraction method of the frame center point using spatial transformation and motion estimation according to an embodiment of the present invention, it is possible to specify a path including a series of center points in the scene caused by the camera movement.

Thus, the path can be used to specify a contour around an object instead of segments, find some information about the object, display a curved text line to extract from an image, and so on. Can be used for real applications.

1 is a flowchart schematically illustrating a method for extracting a moving path of a frame center point using spatial transform and motion estimation according to an embodiment of the present invention.
FIG. 2 is a block diagram illustrating an interval for sampling a frame in a method for extracting a moving path of a frame center point using spatial transform and motion estimation according to an embodiment of the present invention.
FIG. 3 is an image illustrating SURF characteristic points of a previous frame in the method for extracting a moving path of a frame center point using spatial transform and motion estimation according to an embodiment of the present invention.
FIG. 4 is an image corresponding to FIG. 3 illustrating SURF characteristic points of a current frame in the method for extracting a moving path of a frame center point using spatial transform and motion estimation according to an embodiment of the present invention.
FIG. 5 is an image illustrating a calculation of a SURF descriptor by aligning a direction of a square grid at characteristic points in a method of extracting a moving path of a frame center point using spatial transform and motion estimation according to an embodiment of the present invention.
FIG. 6 is an image illustrating a matching pair of characteristic points between a current frame and a previous frame in the method for extracting a moving path of a frame center point using spatial transform and motion estimation according to an embodiment of the present invention.
FIG. 7 is an image illustrating a previous frame when mapping a center point from a previous frame in a current frame in the method for extracting a moving center point of a frame center using spatial transform and motion estimation according to an embodiment of the present invention.
8 is a diagram illustrating a method for extracting a moving path of a frame center point using spatial transform and motion estimation according to an embodiment of the present invention. The image to represent.
9 is a flowchart of an Android application using the SURF library as a shared library in the method for extracting a moving path of a frame center point using spatial transform and motion estimation according to an embodiment of the present invention.

Next, a preferred embodiment of a moving path extraction method of a frame center point using spatial transform and motion estimation according to the present invention will be described in detail with reference to the accompanying drawings.

As those skilled in the art would realize, the described embodiments may be modified in various different ways, all without departing from the spirit or scope of the present invention. In the drawings, parts irrelevant to the description are omitted in order to clearly describe the present invention, and the same reference numerals are used for the same parts throughout the specification.

Throughout the specification, when an element is referred to as "including " an element, it means that it can include other elements, not excluding other elements, unless specifically stated otherwise.

First, as shown in FIG. 1, a method for extracting a moving path of a frame center point using spatial transform and motion estimation according to an embodiment of the present invention includes a frame sampling step S10 and a speed-up robust feature (SURF) characteristic point. (feature points) and descriptors extraction step S20, a spatial transform estimation step S30, a new position calculation step S40, and a position redefinition step S50.

The frame sampling step S10 samples some frames in order to reduce the number of frames required for processing in the input image sequences.

In the above, sampling of the frame can be configured to be performed at a constant interval (D).

For example, it is also possible to set D to 5, as shown in FIG. 2, for the interval D between two sampled frames.

The sampling process helps to reduce the complexity incurred in processing all image sequences.

Even when using the sampled frame as described above, it is possible to approximate a path connecting all frame center points in the image sequence.

When it is necessary to generate a path connecting the center point as a flexible path, it may be represented by a Bezier curve connecting the center point of the sampled frame obtained in the final frame.

 In addition, the spatial transform parameter in the spatial transform estimation step S30 can be effectively estimated when the interval between two successfully sampled frames is sufficiently large.

 It is also possible to reduce the resolution of the sampled frame to 320 * 240, and to convert the sampled frame to a gray scale image.

The scale invariant feature transform (SIFT) and the speeded-up robust feature (SURF) used in the SURF feature point and descriptor extraction step (S20) are powerful image detectors and descriptors.

The SIFT and SURF are stabilized with variations in scale, rotation, changes in illumination, and partial invariance. Thus, SIFT and SURF are suitable for the detection of feature points that can be used to estimate the spatial transformation between the previous frame and the current frame.

In the SIFT and SURF, the same characteristic point can be detected in both the previous frame and the current frame, and it is possible to determine the size of the region of interest of this characteristic point. The size of the region of interest is determined in relation to the size at which the detected characteristic point is located.

 The SIFT detector consists of three main steps. 1) In the first step, characteristic points are detected. This image is superimposed with a Gaussian filter at different sizes. The maximum / minimum of the Difference of Gaussian (DoG) generated at many sizes is obtained with the characteristic point. 2) The extreme detection of the size-space creates too many characteristic points and is mostly unstable, thus detailing the proximity data for the exact position, scale, and ratio of major curvature. In this process, points with low contrast or poorly located edges are removed. 3) Finally, each feature point is assigned one or more orientations based on the local image tilt direction. Since the characteristic point descriptor can be expressed in relation to this orientation, rotational invariance is achieved.

Based on such SIFT, SURF detector is proposed with many approximations to increase efficiency. The SURF detector is based on a Hessian matrix because of its low complexity and high accuracy.

The Hessian matrix is used instead of the Gaussian difference (DoG) scale.

The SURF detector then approximates the second derivative of the Gaussian for the box filter to reduce the computational complexity. Here, a box filter of any size is calculated for a period of time by utilizing the average of the complete image. In a complete image, each position represents the sum of the origin and all pixels of the input image within the rectangular region formed by this position. The sum of the intensity values of any vertical rectangular region can be calculated with three additions. Hessian pyramids are created by applying different sized box filters onto the original image. This characteristic point is located by finding neighborhoods beyond the maximum 3 * 3 * 3 area. The maximum decision of the Hessian matrix is interpolated in this scale and image space.

The SURF can produce good results compared to SIFT with lower computational complexity.

In the image sequence sampled in the frame sampling step S10, the current frame is represented by F _n , and the previous (reference) frame is represented by F _{n −1} .

The SURF characteristic point is located and the SURF descriptor calculates for the current frame F _n .

In the above, the characteristic point of the previous frame F _{n -1} is reused since it has already been extracted.

3 shows a state in which the SURF characteristic point is detected for the previous frame F _{n -1} , and FIG. 4 shows a state in which the SURF characteristic point is detected for the current frame F _n . 3 and 4 illustrate a state in which the SURF characteristic point (or center point) of the image sequence input through the camera embedded in the mobile phone is detected, and is obtained by moving the mobile phone along the outline of an object (object). The state which detected SURF characteristic point (or center point) in a frame is shown.

Here, two sets of the SURF characteristic point Φ _n of the current frame F _n and the SURF characteristic point Φ _n-1 of the previous frame F _{n -1} are represented by Equations 1 and 2 below. Is defined as:

In Equations 1 and 2, m1 and m2 represent the number of characteristic points detected in the current frame and the previous frame, and [(x _i , y _i ), s _i , α _i ] and [(x ' _j). , y ' _j ), s' _j , α' _j ] are the positions (x,) of the characteristic points (∀i∈ (1, ..., m1) and ∀j∈ (1, ..., m2)), respectively. y), scale (s) and direction (alpha) are shown.

Two sets (Ω) of SURF descriptors for the characteristic points are defined as in Equations 3 and 4 below.

In Equations 3 and 4, ω _i and ω _j (i∈ (1, ..., m1) and j∈ (1, ..., m2)) are characteristic points ([(x _i , y _i ), s _i , α _i ] and [(x ' _j , y' _j ), s ' _j , α' _j ]).

The SURF descriptor is calculated using the perimeter of the characteristic point.

First, the dominant direction of each characteristic point is estimated by building a circular region around the detected characteristic point.

As shown in Figure 5, the rectangular area is aligned and centered along the dominant direction.

In this case, the size of the region is 20s, and s represents a space size.

The area is divided into 4 * 4 subareas. Haar wavelets are calculated in the horizontal and vertical directions. The wavelet response is summarized for each subregion from the first set of feature vectors. The sum of the absolute values of this response is also added to this vector.

Thus, four elements are displayed for the subregions. Thus, descriptors for each characteristic point are obtained as 4 * 4 * 4 * = 64 dimensional vectors in SURF compared to 128 dimensional vectors in SIFT.

Since the SURF descriptor is calculated with respect to the dominant direction, rotational invariance is achieved. The SURF descriptors are standardized to improve invariability in illumination changes.

In the spatial transform estimation step (S30), after extracting SURF feature points, pairs ({ω _i ↔ω _j }, i∈ (1, ..., m1) and j∈ corresponding to feature points in the current frame and the previous frame are extracted. Proceed to the matching process to find (1, ..., m2)).

The similarity between the two characteristic points is examined by calculating the cosine similarity between the SIFT and SURF descriptors.

And matching in order to speed up the process, the set (Ω _n) for each descriptor (ω _i) 2 of the most likely descriptor (ω _'j and ω' in the previous set (Ω _n-1) for _k, in here ω _i uses a 2-proximity neighbor (2-NN) search strategy to find ω ' _j more similar to ω' _j than ω ' _k .

The two characteristic points ω _i and ω ' _j are pairs of corresponding points that satisfy the condition of Equation 5 below.

In Equation 5, DR is set to 0.6.

In the above, a match of characteristic points is accepted only if the ratio of the vector angles from the closest second nearest neighbor is less than DR.

6 shows a matching pair of characteristic points between the current frame and the previous frame.

In addition, the global motion parameter can be estimated using a spatial transformation matrix that can be calculated using a matching pair, and a non-reflective similarity model including deformation, rotation, and scaling. ).

The non-reflective pseudo-transformation model may be represented by a matrix capable of mapping all characteristic points of the previous frame in the current frame. Compared to the affine transformation model, the shape and angle are preserved in the non-reflective pseudo transformation model.

Since only two parameters are needed to represent the non-reflective pseudo-transformation model, at least two matching pairs are required, which can be easily estimated using Equation 6 below.

In Equation 6, sc and ss are defined as sc = scale * cos (angle) and ss = scale * sin (angle), respectively.

Four unknowns (sc, ss, tx, ty) can be solved using the matching pair. Equation 6 may be rewritten as in Equation 7 and Equation 8 below.

And with at least two matches, we can combine the equations of x and y (Equations 7 and 8) for one linear system to solve for four unknowns as

The matrix equation of Equation 9 can be rewritten as X = X '* r. Here, r is defined as r = [sc ss tx ty]. Therefore, r = X / X '.

In the new position calculation step S40, a new position is recalculated from the transformation matrix capable of mapping pixels of the previous (reference) frame in the current frame from the previous stage to the new position in the current frame.

For example, the positions of all center points of the previous sampled frame are recalculated in the current frame by using the transformation matrix of Equation 9 above.

7 shows an example of calculating the center point of the previous frame, and FIG. 8 shows an example of recalculating the center point of the current frame.

The center point of the previous frame is repositioned due to the movement of the camera at the new position of the current frame corresponding to the transformation matrix.

After recalculating the positions for all previous center points (indicated by red dots in FIG. 8), the center point of the current frame (indicated by green dots in FIG. 8) is added into the path.

In the position redefining step (S50), the position redefinition such as a path connection indicating the path of the camera movement is redefined.

Since the conversion parameter is not perfect and the shape may be distorted, the new position of the center point in the current frame may not be the same as the corresponding position of the previous frame.

Thus, a redefinition of the position is needed, and for each center point (excluding the new center point in the current frame), the block-based motion estimation process is done in a very small search area (less than 3).

Then, the surrounding block located in the current frame which is almost similar to the block found from the position of the previous frame can be found, and the path connecting all the center points in the final frame is a necessary path to find.

Next, the process of implementing SURF in an Android phone is explained.

In the moving path extraction method of the frame center point using spatial transform and motion estimation according to an embodiment of the present invention, most calculation time is consumed to locate the SURF characteristic point and calculate the descriptor.

The main programming language for Android phones using the Android SDK is Java (JAVA). Java applications are compiled into Java bytecode by the Java compiler. Java bytecode is compiled into a .dex file that can be run by the Dalvik Virtual Machine. Since the execution time of the Dalvik bytecode, which is the .dex file, is slow, it is necessary to reduce the execution time of the Dalvik virtual machine.

The SURF is implemented to reduce execution time and is built with the original C / C ++ library using the Android NDK. The Android NDK was developed to improve processing speed, reuse available C source code, and directly access hardware.

The SURF library is dynamically loaded and communicated with Java using the JAVA Native Interface (JNI). The Android NDK can be used with other libraries such as OpenCV. And, SURF can be implemented in OpenCV. 9 shows a system flowchart of application operation with Android's shared library.

Using this method, the computation time for SURF is much smaller than in Java.

Next, a result of evaluating the performance of the moving path extraction method of the frame center point using spatial transform and motion estimation according to an embodiment of the present invention will be described.

For evaluation, several test image sequences captured by the mobile phone built-in camera are used. This image sequence is captured at 15 frames per second with a resolution of 640 * 480. This resolution is downscaled to 320 * 240. One test image sequence with 149 frames (relative to 15s length) is shown. The algorithm according to an embodiment of the present invention is implemented and tested in an Android mobile phone with a 1GHZ processor and 512MB RAM. In the motion estimation step, the search range is set to 2 and the block size is set to 32.

Table 1 below shows the path results for several selected frames. In Table 1, the results of the second column and the third column are obtained according to whether motion estimation based on location redefinition is performed. In Table 1, the first column represents a frame index (ID) of a frame selected from an original image sequence. The interval between two successful sampled frames is set to 10 (ie 15 sampling frames). In the path result, the red point added to the path represents the center point of the previous frame and the green point represents the center point of the current frame.

The total processing time is 3.34 and 3.47 seconds, respectively, to find the path with motion estimation. Motion estimation based on the location redefinition requires less computation time compared to the total processing time, but significantly improves the results.

ID No location override Has location override


11

31

41

71

111

131

141
final

The above experimental results indicate that the method of extracting the moving path of the frame center point using the spatial transform and the motion estimation according to an embodiment of the present invention connects the frame center points accurately by combining the SURF based on the spatial transform and the motion estimation. Show that the path can be estimated. The corresponding position of the center point remains the same along the frame in the image sequence.

Table 2 below shows the path results obtained along with the different values of the spacing between the two sampled frames. The result shows a stable and accurate path result that can be obtained with a high value of D. Therefore, we can reduce the processing time by increasing the value of D. Instead of using a fixed value of D, it is possible to select an adaptive value that can be used, such as after completing the frame processing of the current frame in the captured real-time image sequence. And this system is always stable in real time. In the case where one wants to generate a flexible path, the resulting path can be represented by a Bezier curve containing the obtained points of the result path.

D Result path Processing time (s)


10

3.47
(2.94 for SURF)


15

2.29
(1.86 for SURF)


20

1.91
(1.47 for SURF)

In the above, a preferred embodiment of a method for extracting a moving path of a frame center point using spatial transformation and motion estimation according to the present invention has been described. However, the present invention is not limited thereto, and the scope of the claims, the specification, and the accompanying drawings. Various modifications can be made therein and this also falls within the scope of the present invention.

S10-frame sampling step, S20-SURF characteristic point and descriptor extraction step
S30-space transform estimation step, S40-new position calculation step
S50-Position Override Step

Claims (11)

Sampling the input frame to extract the current frame and the previous frame,
Extracting the SURF feature point and descriptor for the current frame,
Estimating a spatial transform parameter using the characteristic points of the previous frame and the current frame,
Calculating a new position of the center point in the current frame by converting the position of the center point of the previous frame to the current frame based on the transformation matrix related to the spatial transform parameter;
Estimating a block motion around a new position of the center point and redefining the new position of the center point. The method according to claim 1,
And extracting the current frame and the previous frame. The method of extracting a moving center point of a frame center point using spatial transformation and motion estimation configured to perform sampling of an input frame at a predetermined interval (D). The method according to claim 1,
And extracting the current frame and the previous frame. The method of extracting a center point of a frame using spatial transformation and motion estimation for reducing the resolution of the sampled frame or converting the sample to a gray scale image. The method according to claim 1,
And extracting the SURF characteristic point and the descriptor from the frame center point using spatial transform and motion estimation using a SURF detector based on a Hessian matrix. The method of claim 4,
Extracting the SURF feature point and the descriptor, and reusing the feature point of the previous frame, and extracting the center point of the frame using spatial transform and motion estimation calculated by the SURF descriptor only for the feature point of the current frame. The method according to claim 5,
The SURF descriptor is calculated using the periphery of the characteristic point, and the moving path of the frame center point using the spatial transformation and the motion estimation that estimates the dominant direction of each characteristic point by constructing the circular region around the detected characteristic point. Extraction method. The method according to claim 5,
In the estimating of the spatial transform parameter, the SURF characteristic point extracted in the extracting of the SURF characteristic point and the descriptor is performed in a matching process of finding a corresponding pair in the current frame and the previous frame.
In order to speed up the matching process, the frame center point is moved by using spatial transform and motion estimation to find a pair corresponding to the descriptor related to the characteristic point of the current frame based on the descriptor most similar to the characteristic point of the previous frame. Path extraction method. delete The method of claim 7,
In the estimating of the spatial transform parameter, a global motion parameter that can be estimated using a spatial transform matrix that can be calculated using a matching pair,
Moving path extraction method of frame center point using spatial transformation and motion estimation using non-reflective similarity model including deformation, rotation, and scaling. The method according to claim 9,
In calculating the new position of the center point, the spatial transformation and motion estimation are recalculated from the transformation matrix capable of mapping the pixels of the previous frame in the current frame to the new position in the current frame. Moving path extraction method of frame center point using delete

Images