KR101313644B1 - reuse methode for descriptor of image - Google Patents

Abstract

The present invention compares a current frame with a next frame in a video formed of a plurality of consecutive frames, and reuses a descriptor generated in the current frame when the similarity is high. Disclosed is a method for recycling a movie diskette that can reduce the amount of processing performed in a transform.

Description

Recycle methode for descriptor of image

The present invention relates to a video clipper recycling method for significantly reducing processing throughput by using the current frame's splitter as the next clipper in a video having a series of similarities.

As computing technology advances, the quantity and quality of image processing continues to improve, and what has been used only in special fields such as military and space exploration is spreading to real life applications such as image compression and face recognition.

Among the detailed fields of image processing, image recognition has not been applied to real life relatively much compared to other detailed fields, but many studies have shown that applications and implementations that are gradually applied to real life, such as face recognition, have emerged one after another. However, the problem of image recognition is difficult and complicated, and since image recognition algorithms require very complicated computational processes and require a large amount of database or memory, it is difficult to implement them.

Object recognition is one of the most important applications of image recognition. It receives an image from a camera and finds out whether there is an object to be found in the image and where the object is located. Since the quality of the input image is also very diverse, numerous object recognition algorithms for each characteristic have been studied.

The process of object recognition varies greatly depending on the application and algorithm. However, in general, image recognition, feature extraction, and detection + localization are performed as shown in FIG. 1. Is done.

In image enhancement, various filtering is performed to reduce noise of an input image and to highlight a feature to be used in a later step. For example, histogram equalization for equalizing the brightness of the input image. In the feature extraction step, noteworthy features are extracted from the enhanced image into a vector form. The operation after this step is performed with the feature vector, not the input image. In this step, a large amount of computation is required because it is necessary to perform operations on numerous input pixels to extract only necessary information. In addition, this step has a great influence on the object recognition result because only necessary information is left and the rest is discarded. Therefore, this step is very important in the object recognition process, and a lot of research has been done. The final stage represents all recognition procedures after the feature extraction phase, which depends on the application.

Feature vectors used in object recognition should be discriminant from features extracted from other objects, and invariant if the image is deformed or if the viewpoint changes if it comes from the same object.

In the early years, features that express the global characteristics of objects, such as those that represent the geometric shape of the object outline, or the features such as the length of the outline, the degree of rounding, etc., were mainly studied. However, it is often not appropriate to use global features for object recognition in real life images. A cluttered background may not be easy to extract the outline of an object, and a portion of the object is often occluded by another object.

In order to remedy this problem, research has been actively carried out over the last few years using features that represent local patches of an image. By creating local features that represent objects, we first look for prominent points in the image, which are called keypoints or interest points. After that, we create a feature vector representing the local patch around each keypoint, which is called a region descriptor or descriptor. As described above, object recognition is performed by matching a plurality of descriptors made from an input image with descriptors of a model image. With these local patches, it is possible to recognize objects by the remaining features, even if some of the objects are hidden or due to complex background.

Among the local features created according to this strategy, SIFT (Scale-Invariant Feature Transform) is widely used because of its robustness in various image transformations.

SIFT, D. Lowe, "Distinctive image features from scale-invariant keypoints," International Journal of Computer Vision , vol. 2, no. 60, pp. 91-110, 2004. This is a kind of local feature proposed in (Lowe), and it is widely used for object recognition because it shows better performance than other local features under various conditions. As the name suggests, features do not change well when the object's scale changes or rotates.

2 illustrates a SIFT feature calculation procedure. The computation procedure is largely divided into keypoint detection and descriptor generation. The first part, keypoint detection, finds points with noteworthy features in the input image and uses them as keypoints. The second part, descriptor generation, creates a descriptor for each local image patch that it finds for each keypoint it finds. This descriptor consists of a gradient histogram for the local patch of each keypoint.

Keypoint detection consists of three small operations, scale-space generation, local extrema detection, and keypoint detection. Descriptor generation consists of two small operations, orientation assignment and descriptor generation.

Since the SIFT performed in this way is based on pixel-by-pixel operation, the amount of computation increases as the size of the image increases. In addition, the amount of computation for each pixel is also very large. Because of these problems, research has been conducted to optimize the SIFT algorithm to speed up the computation. See also V. Bonato, E. Marques, and G. Constantinides, "A Parallel Hardware Architecture for Scale and Rotation Invariant Feature Detection," Circuits and Systems for Video Technology , IEEE Transactions on, vol. 18, no. 12, pp. As described in 1703-1712, Dec 01, 2008 (hereinafter referred to as 'Bonato'), studies have been made to speed up the computation by implementing some of the SIFT operations on hardware accelerators such as FPGAs. However, hardware and memory usage is still a problem in Bonato.

In the present invention, since the similarity between the frame and the frame is very large, the descriptors generated in the current frame are very similar to the descriptors generated in the previous frame. By using the almost identical to the generated descriptors, by reusing the descriptors generated in the previous frame in the video photographed from a fixed position, it is possible to significantly reduce the amount of SIFT descriptor generation operation and reduce the memory.

The solution for solving the above problems is to reuse the descriptor generated in the current frame in the video formed of a plurality of consecutive frames to reduce the amount of computation processed in the Scale-Invariant Feature Transform (SIFT) A method for recycling a video clipper, the method comprising: detecting keypoints by the SIFT in the current frame and producing a splitter for each keypoint; Generating keypoints by the SIFT in a next frame; Comparing keypoints generated in the next frame with keypoints in the current frame, if the keypoints generated in the next frame satisfy a preset reuse condition, the splitter generated in the current frame is assigned to the next frame. And reusing, wherein the preset reuse condition includes: 1) a keypoint at the same position as each keypoint position generated in the next frame must exist in the current frame; and 2) the next frame and the current frame. The keypoints must be the same scale.

In addition, in the present invention, the predetermined reuse condition is that 1) a keypoint at the same position as each keypoint position generated in the next frame must exist in the current frame, and 2) the scale of the next frame and the keypoint of the current frame. It is desirable that the scales be the same.

In addition, in the present invention, the predetermined reuse condition may further include 3) a condition of not reusing the current diskette in the case of a keypoint near the moving object.

In addition, in the present invention, each of the keypoints of the current frame is assigned an 'age', the position and the clipper of the keypoints are stored, and subsequent frames until the end of a lifetime determined by the number of consecutive frames. If keypoints are present at the same position as the current frame at, it is preferable to reuse a discarder of keypoints at the same position as the current frame in the subsequent frames and to initialize the reused keypoint if the keypoint is reused.

In addition, in the present invention, the number of burned frames of the lifetime is preferably three.

According to the present invention having the above-mentioned problems and solutions, it has the effect of significantly reducing the amount of computation of the STIF algorithm by reusing the descriptor under a specific condition for the part including the background in the frames of the video, giving age The improved effect can be obtained by operating in a certain life.

1 is a flowchart illustrating a processing procedure of recognizing a general object.
2 is a flowchart illustrating a calculation process of a general SIFT.
3 is a view for explaining the keypoint positional relationship between frames in the present invention.
4 is a flowchart of a Descriptor reuse operation procedure in the present invention.
5 is an image showing the location of a keypoint having a large Descriptor error in the present invention.
Figure 6 illustrates the overlapping local patch in the present invention: (a) when the background patch is changed by a moving object; (b) shows a case where the background patch is not changed by a moving object.
Figure 7 shows the Descriptor reuse results in the present invention.
8a, b, c, d is a graph showing the Descriptor distance according to the distance to the moving keypoint in the present invention.
9A is a graph in which an error accumulation rate and a reuse rate according to a distance from a moving keypoint are applied to a weather image.
9B is a graph in which the error accumulation rate and the reuse rate according to the distance from the moving keypoint in the present invention are applied to the [Paris] image.
Figure 10a is a graph illustrating the reuse rate according to the Descriptor life in the present invention. 10B is a graph illustrating a descriptor error according to the reuse rate of the descriptor in the present invention.
FIG. 10 is a graph illustrating descriptor mismatch according to a reuse rate of a descriptor in the present invention. FIG.
11A is an 'akiyo' image used as an example in the present invention.
11B is a 'container' image used by way of example in the present invention.
11C is a 'deadline' image used by way of example in the present invention.
11D is a 'paris' image used as an example in the present invention.
11 is a 'wash dc' image used by way of example in the present invention.

Since video has a lot of similarities between frames (high temporal correlation), the descriptors generated in the current frame are very similar to the descriptors generated in the previous frame. In particular, the descriptor created in the part that does not change like the background is almost identical to the descriptor generated in the previous frame. According to the present invention, the amount of SIFT descriptor generation can be reduced by reusing a descriptor generated in a previous frame in a video photographed at a fixed position.

1.1 Interframe keypoint  Positional relationship

In a movie shot at a fixed position, the background is almost the same, and the moving object and its surroundings change. 3 shows an example of the whether video. FIG. 3 (a) shows the first frame, and FIG. 3 (b) shows the second frame. The circled portion of each frame represents the detected keypoint. In FIG. 3 (c), for the second frame, keypoints detected at the same position as the previous frame are indicated by blue circles, and keypoints detected at other positions are indicated by red circles. 3 (d) shows only the keypoints detected at different positions by red circles. In this video, the background does not change, only the person moves, and the points shown in FIG. 3 (d) show the silhouette of the person, which is a moving object.

The present invention proposes a method of reusing a descriptor using such a property.

1.2 Descriptor  Reuse algorithm

4 illustrates a descriptor reuse algorithm proposed in the present invention. In the first frame, keypoints are detected using the original SIFT algorithm and a descriptor is created for each keypoint. In the following frame, keypoints are generated according to the original SIFT algorithm. For each keypoint created, check the reuse conditions, such as whether keypoints of the same scale, same scale exist in the previous frame (reuse conditions are described in the following subsections). If the reuse condition is satisfied, the descriptor is reused. Otherwise, the descriptor is created according to the original SIFT algorithm. The final keypoint and descriptor are stored for use in the next frame.

1.3 SIFT descriptor  Reuse condition

In order to reduce the amount of SIFT computation, the proposed method decides whether to create a new descriptor or reuse a descriptor created in a previous frame for each keypoint.

To reuse the Descriptor, for the keypoint of the current frame created in the same location as the previous frame, its descriptor and the descriptor created in the previous frame must match or be very similar. In the present invention, descriptor reuse conditions are presented below.

Condition 1: A keypoint must exist in the previous frame at the same position as each Keypoint position created in the current frame. If present, the descriptor of the keypoint in the same location is reused. Otherwise, the descriptor is created without reuse.

Condition 2: If the keypoint of the previous frame is different at the same position as the current frame, if the scale of the keypoint of the previous frame and the keypoint of the current frame are different, a new descriptor is created without reuse.

Condition 3: For keypoints in the vicinity of moving objects, the descriptor should not be reused even if it is determined to be reusable under conditions 1 and 2.

In the case of the background part, the difference between the following frames is very small. Therefore, the difference between the keypoint position and its descriptor between subsequent frames is very small, and there is very little error by reusing the descriptor. On the other hand, for keypoints created from moving objects, their position changes as the frame changes. Therefore, the descriptor is not reused and newly created, so there is no error at all.

In FIG. 5, the red circle indicates a keypoint detected at a different position from the previous frame, and the blue circle indicates a keypoint detected at the same position as the previous frame. The white circle is a keypoint with a large difference between the reused descriptor and the descriptor created by the original SIFT algorithm. According to FIG. 5, an error due to descriptor reuse mainly occurs around a moving object. This is true even if the keypoint is in the background and there is a keypoint at the same position in the previous frame. This is because in the case of a keypoint near a moving object, its local patch is likely changed by the moving object. Therefore, even if it is determined to be reusable according to the conditions 1 and 2, reusing the descriptor in such a case may cause considerable error.

In order to grasp the moving object without a great increase in the amount of calculation, the present invention assumes that the keypoint whose position is changed represents the moving object. Once you have identified the moving object, you should test whether it has changed the local patch of the surrounding keypoint. For this test, we need to know the boundaries of the moving object. G. Bradski, and A. Kaehler, Learning As described in OpenCV : O'Reilly Media, 2008. Object segmentation is an algorithm that divides the boundary of an object. Assuming object boundaries are identified through object segmentation, we can verify that the local patch of the keypoint overlaps with the surrounding moving objects. If the local patch of the keypoint is not overlapped by a moving object, it is possible to reuse the descriptor created in the previous frame.

6 illustrates the descriptor reuse condition described above. The keypoint in the background is marked with an 'x' and the keypoint on the moving object is marked with a 'y'. It is assumed that 'x' is detected at the same position as the previous frame, and 'y' is not assumed to be at the same position as the previous frame. Figure 4.4 (b) also assumes that the object segmentation results are available. The local patch of 'x' is indicated by the rectangle around the keypoint. In FIG. 6 (a), since the local patch of 'x' overlaps with the moving object, the descriptor of the previous frame cannot be reused, and the descriptor must be newly generated by the original SIFT algorithm. On the other hand, in FIG. 6B, since the local patch of 'x' does not overlap with the moving object, it is possible to reuse the descriptor generated in the previous frame.

7 shows the actual descriptor reuse result using object segmentation. FIG. 7 (a) shows keypoints extracted from the first frame of the weather ◎ image as blue circles. 7 (b) shows a second frame in L. Vincent, and P. Soille,? Atersheds in digital spaces: an efficient algorithm based on immersion simulations,? Pattern Analysis and Machine Intelligence, IEEE Transactions on, vol. 13, no. 6, pp. 583-598, 1991. and J.-W. Jang, and H.-J. Lee, ◎ lock-Based Predictive Watershed Transform for Parallel Video Segmentation, Journal of Semiconductor Technology and Science (Submitted) , 2011. Each segment is overlaid on the original image with a random color. 7 (c) shows a part of the descriptor overlap test. Shown by the green rectangle is the local patch of the keypoint extracted on the moving object. The patch marked with a red box at the top is a patch extracted from the background and you can see that much of it overlaps the moving object. Therefore, this background patch is not reused. On the other hand, the patch marked with a blue box is a patch extracted from the background and overlaps the green box shown below but does not overlap with the moving object. Therefore, this background patch is reused. 7 (d) shows a result of reusing a descriptor of a second frame. Blue circles indicate keypoints reused by descriptors and red circles indicate keypoints not reused.

Object segmentation is one of the important operations in the field of image recognition, and is implemented in high-end image processing processors. In this case, using object segmentation to determine reuse condition 3 does not significantly increase hardware cost. On the other hand, in an environment where object segmentation is not implemented, a large amount of computation is required to determine reuse condition 3. Therefore, the present invention proposes a simplified test method for the determination of reuse condition 3. In the present invention, the reuse condition 3 is determined without using object segmentation. A detailed method will be described later in detail.

4.Simplified overlap  Test method

In an environment where object segmentation is not available, the boundary of a moving object cannot be determined, so the boundary cannot determine whether the local patch of the surrounding keypoint has changed. In this section, we propose a simplified overlap test method that can be used instead of object segmentation.

8 shows a descriptor error when a descriptor is reused using only reuse conditions 1 and 2 for the second frame of the weather image. The x-axis of FIG. 8a represents the distance from the nearest moving keypoint (the keypoint detected at a different position from the previous frame), and the y-axis represents the Euclidean distance between the reused descriptor and the descriptor generated by the original SIFT algorithm. Since the descriptor is normalized, Euclidean distance between descriptors can have a maximum. FIG. 8B is an enlargement by narrowing the range of FIG. 8A. As shown in FIG. 8, it can be seen that keypoints having a large number of descriptor errors are gathered near the moving keypoint.

8C shows descriptor errors on the y-axis along the x-axis, arranged sequentially from the closest to the farthest distance from the moving keypoint. In FIG. 8D, the accumulated value of the descriptor error according to the distance from the moving keypoint is indicated by a solid line. In this case, the y-axis represents the ratio of the overall error. In addition, the dotted line indicates the ratio of the number of descriptors whose distance from the moving keypoint is larger than the value indicated on the x-axis.

8C and 8D show more intuitively that most of the overall descriptor errors are concentrated close to the moving keypoint.

FIG. 9 shows accumulated values of descriptor errors according to distances to moving keypoints for four moving pictures (weather in FIG. 3, paris in FIG. 11A, deadline in FIG. 11B, and container in FIG. 11). In this case, the y-axis represents the ratio of the overall error. In addition, the ratio of the number of descriptors whose distance from the moving keypoint is larger than the value indicated on the x-axis is indicated by a dotted line. This is the ratio of the number of descriptors to be reused among the total reusable keypoints when keypoints with a distance from the keypoint moving smaller than the value indicated on the x-axis are not reused. 9A and 9B are results for the weather and paris videos, respectively.

According to this result, while the ratio of the descriptor number has its own range for each video, the accumulated value of the descriptor error has a similar value as 0.6-0.8 when the distance from the moving keypoint is 10-20. have. Therefore, if the value of this range is set as threshold, a nearly constant descriptor error rate can be obtained.

Table 1 shows the distance of moving keypoint and descriptor reuse rate by video for a certain descriptor error rate. In the present invention, according to the result of this table, the threshold for the distance from the moving keypoint is set to 12. Descriptors for keypoints located less than this threshold are not reused.

 Distance from moving keypoint and descriptor reuse rate according to Descriptor error accumulation rate Error acc.
Seq. 70% 80% Distance Reuse [%] Distance Reuse [%] weather 11.82 85.63 12.75 83.98 paris 9.81 53.04 11.72 43.48 deadline 14.01 38.58 17.53 24.93 container 12.17 25.27 14.03 16.13

5. Keypoint  Extended location comparison

Depending on the characteristics of the video, even if the background looks almost unchanged, the keypoint may not be continuously detected at the same position. This is a case where the keypoint cannot be stably detected due to noise or the like present in the video. In this case, even though the background does not change visually, the descriptor reuse rate is low, and the amount of computation reduction is not sufficiently seen. However, in some cases, keypoints at the same location do not appear in successive frames, but at the same location across one or two frames.

In order to prepare for such a case, the present invention proposes a method for extending descriptor reuse. Age is assigned to each keypoint, and its location and descriptor are remembered even if the keypoint is not reused until the end of its lifespan as the number of frames elapses. At the end of its lifetime, keypoints are deleted from memory. After a keypoint is obtained for the current frame, the position is compared against all stored keypoints, and if there is a keypoint at the same position, the descriptor is reused. When the descriptor of a stored keypoint is reused, its age is initialized.

FIG. 10A shows a descriptor reuse rate according to the lifetime, and FIG. 10B shows a descriptor error according to the reuse rate. At this time, the descriptor error means Euclidean distance from the descriptor created by the original SIFT algorithm.

Descriptor mismatch is the ratio of matching the same keypoint but not the same keypoint when matching the descriptor set generated by the original SIFT algorithm and the descriptor set created by reuse with the nearest neighbor search described in Lowe. do. 10c shows a descriptor mismatch according to the reuse rate.

According to FIG. 10A, the reuse rate increases as the life is longer, but when the life is larger than 3 frames, the increase in reuse rate is slowed down. In addition, as the lifetime increases, the memory storing the location of the keypoint and its descriptor increases, and as shown in FIGS. 10B and 10C, the descriptor error also increases. Therefore, it can be seen that the gain when the lifetime is larger than 3 frames is not large. Therefore, in the present invention, for the extended descriptor reuse method, the lifetime is limited to three.

Claims (5)

In the video clipper recycling method to reduce the amount of computation processed in the Scale-Invariant Feature Transform (SIFT) by reusing the descriptor generated in the current frame in a video formed of a plurality of consecutive frames :
Detecting keypoints by the SIFT in the current frame and producing a discarder for each keypoint;
Generating keypoints by the SIFT in a next frame;
Comparing keypoints generated in the next frame with keypoints in the current frame, if the keypoints generated in the next frame satisfy a preset reuse condition, the splitter generated in the current frame is assigned to the next frame. Reusing,
The preset reuse condition is that 1) a keypoint at the same position as each keypoint position generated in the next frame must exist in the current frame, and 2) the scale of the keyframe of the next frame and the current frame is The method of recycling a movie diskette, characterized in that it must be the same. delete The method of claim 1, wherein the preset reuse condition further comprises: 3) a condition of not reusing the current decoder when the keypoint is located near the moving object. The method according to claim 1, wherein each of the keypoints of the current frame is assigned an age, the position and the clipper of the keypoints are stored, and subsequent frames are reached until the end of a lifetime determined by the number of consecutive frames. A keypoint at the same position as the current frame in the second frame, reuses a diskette of keypoints at the same position as the current frame in the subsequent frames, and initializes the reused keypoint if the keypoint is reused. How to recycle the diskette. 5. The method of claim 4, wherein the frame life of the frame is three.

Images