KR20130056171A - Real-time object recognition and tracking method using representative feature, and apparatus thereof - Google Patents

Abstract

PURPOSE: A method for recognizing and tracking an object in real-time using representative features and a device thereof are provided to recognize and track the object by performing simple calculations and the generations of simple histograms using features of a corresponding part in an image. CONSTITUTION: A representative feature extracting unit(11) extracts a representative feature of an object. A coefficient determining unit(12) determines the coefficient according to the feature. If there is not an interest boundary, an object recognizing unit(14) recognizes a basic object using a motion vector. If there is an interest boundary, an object tracking unit(15) tracks the object in real-time using the interest boundary. [Reference numerals] (11) Representative feature extracting unit; (12) Coefficient determining unit; (13) Verifying unit; (14) Object recognizing unit; (15) Object tracking unit

Description

Real-time Object Recognition and Tracking Method Using Representative Feature, and Apparatus Thereof}

The present invention relates to a real-time object recognition and tracking technology for effectively recognizing and tracking an object of interest in an image or video, and more particularly, to real-time object recognition based on a representative feature of an object (eg, a local feature presenter). And a method and apparatus for real time object recognition and tracking for tracking and tracking.

Recently, the development of ubiquitous technology has been very active in the field of computer vision and pattern analysis in the method of automatically recognizing and tracking objects from images or images.

In researching a reliable object recognition method, the biggest problem is that even the same kind of objects exist in various images in an image or an image. That is, although the same kind of objects may exist according to the color, observation angle, lighting, etc. of the object, they may exist in completely different forms. On the contrary, it is often observed that other kinds of objects are classified into the same kind by looking at some criteria. Therefore, the successful object recognition method must have the characteristics to distinguish the differences between different kinds of objects while ensuring diversity among the same objects.

In addition, in the object tracking method, a successful object tracking method is required only if the object has strong characteristics such as collision or change of exposure.

On the other hand, most of the object recognition and tracking methods that have been studied so far aim to improve the recognition rate, or most of the object recognition and tracking methods can recognize only a certain kind of objects or the recognition rate is reduced due to the benefit.

Therefore, it is necessary to study a new object recognition and tracking method that can satisfy both the recognition rate improvement and the fast processing speed.

As described above, the conventional object recognition and tracking methods that have been studied so far aim to improve the recognition rate, or even when considering the processing speed, the recognition rate may be lowered due to the opposite benefit or only some specific types of objects may be recognized. Most of the existing object recognition and tracking methods do not reflect the characteristics of each partial behavior in the process of expressing human behavior, and research for real-time analysis using these characteristics has not been well conducted. Therefore, there is a need for a new object recognition and tracking method and apparatus capable of satisfying both a recognition rate improvement and a high processing speed. Accordingly, it is an object of the present invention to solve this problem and meet the above requirements.

Accordingly, an object of the present invention is to provide a real-time object recognition and tracking method and apparatus for real time object recognition and tracking based on the representative characteristics of the object.

In other words, the present invention uses a learning scheme for object recognition to ensure a basic recognition rate, and uses a simple local feature to speed up processing, that is, not using the features of the entire image for object recognition, but within the image. To provide a real-time object recognition and tracking method and apparatus for enabling the tracking of a recognized object through simple histogram generation and a few simple operations, using the characteristics (representative features) of the parts considered to be features of the object Its purpose is to.

The objects of the present invention are not limited to the above-mentioned objects, and other objects and advantages of the present invention which are not mentioned can be understood by the following description, and will be more clearly understood by the embodiments of the present invention. It will also be readily apparent that the objects and advantages of the invention may be realized and attained by means of the instrumentalities and combinations particularly pointed out in the appended claims.

An apparatus of the present invention for achieving the above object, In the object recognition and tracking device, Representative feature extraction unit for extracting a representative feature of the object; A coefficient determination unit for determining coefficients according to characteristics of the object; A determination unit for determining whether a boundary of interest exists; An object recognition unit for recognizing a basic object using a motion vector when there is no boundary of interest as a result of the determination of the determination unit; And an object tracking unit for tracking an object in real time using the boundary of interest if the boundary of interest exists as a result of the determination of the determination unit.

On the other hand, the method of the present invention for achieving the above object, in the object recognition and tracking method, (a) extracting a representative feature of the object; (b) determining coefficients according to features of the object; (c) determining if a boundary of interest exists; (d) recognizing a basic object using a motion vector if there is no boundary of interest as a result of the determination of step (c); And (e) tracking the object in real time using the boundary of interest if the boundary of interest exists as a result of the determination of step (c).

As described above, the present invention has an effect of recognizing and tracking an object in real time based on a representative feature (eg, local feature presenter) of the object.

In other words, the present invention uses a learning scheme for object recognition to ensure a basic recognition rate, and uses a simple local feature to speed up processing, that is, not using the features of the entire image for object recognition, but within the image. By using the feature (representative feature) of the part considered to be the feature of the object, it is effective to enable the tracking of the recognized object through simple histogram generation and a few simple operations.

In addition, the present invention can be applied to various applications requiring object recognition and tracking by automatically detecting a representative feature of an object through training images and suggesting a technique for real time object recognition and tracking using the training image.

For example, real-time object recognition and tracking technology using the representative features of the present invention may be applied to a security system or the like, and may be utilized as a base technology for automatically analyzing image data of a predetermined period and analyzing occurrence of a specific event. That is, the present invention can be used for real-time object recognition and tracking services for security purposes, and real-time object recognition based services that can be used in devices such as smartphones.

1 is a configuration diagram of an embodiment of a real-time object recognition and tracking device using a representative feature according to the present invention,
2 is a flowchart illustrating a method for real time object recognition and tracking using a representative feature according to the present invention;
3 is a detailed flowchart of an embodiment of a pretreatment process;
4 is a detailed flowchart illustrating an embodiment of a real-time object recognition and tracking process according to the present invention;
5 is a view for explaining a process of generating a motion vector and detecting a moving object;
6 is a view for explaining a process of determining a threshold value of a representative point;
7 is a diagram illustrating an object recognition rate according to an average matching ratio;
8 is a diagram illustrating a result of a multi-object recognition and tracking experiment.

BRIEF DESCRIPTION OF THE DRAWINGS The above and other objects, features and advantages of the present invention will become more apparent from the following detailed description of the present invention when taken in conjunction with the accompanying drawings, It can be easily carried out. In the following description, well-known functions or constructions are not described in detail since they would obscure the invention in unnecessary detail. Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

And throughout the specification, when a part is referred to as being "connected" to another part, it includes not only "directly connected" but also "electrically connected" with another part in between. Also, when a component is referred to as " comprising "or" comprising ", it does not exclude other components unless specifically stated to the contrary .

In the present invention, in order to satisfy both the recognition rate improvement and the high-speed processing speed, a method of automatically extracting a representative feature (representative feature) of an object, an automatic coefficient determination method for each object, a basic object recognition method using a motion vector, and an interest It includes a real-time object tracking technique using boundary.

1 is a block diagram of an embodiment of a real-time object recognition and tracking device using a representative feature according to the present invention.

As shown in FIG. 1, the apparatus for real-time object recognition and tracking using the representative feature according to the present invention automatically extracts a representative feature (representative feature) of the object, that is, automatically extracts the representative feature of the object through probabilistic analysis. The representative feature extractor 11 for determining the coefficient according to the feature of the object, that is, the coefficient determiner 12 for automatically determining the coefficient for object recognition in consideration of the feature of the object, and determining whether there is a boundary of interest. Determination unit 13 for determining, the determination unit 13, the object recognition unit 14 for recognizing the basic object using a motion vector, if the boundary of interest does not exist, the determination of the determination unit 13 As a result, if there is a boundary of interest, it includes an object tracking unit 15 for tracking the object in real time using the boundary of interest.

The operation of the object recognition and tracking apparatus according to the present invention will be described in detail with reference to FIG. 2 as follows.

2 is a flowchart illustrating a method for real time object recognition and tracking using a representative feature according to the present invention.

First, the representative feature (representative feature) of the object is automatically extracted. That is, the representative feature of the object is automatically extracted through stochastic analysis (21).

In order to recognize a desired object in real time, a part which is considered to be a feature of the object is automatically extracted from the training images of the object. This is a process of determining which point of interest is a representative part of the object among various interest points. That is, all points of interest are compared with each other, and if the similarity exceeds a predetermined value, the part is selected as the representative feature.

Then, the coefficient is automatically determined according to the characteristics of the object. That is, the coefficient necessary for object recognition is automatically determined in consideration of the characteristics of the object (22).

When the task of detecting the representative points for each object is completed, the training set is performed to automatically select coefficients necessary for recognition through additional learning. Various thresholds are used throughout the preprocessing (21 and 22 are preprocessing) and object recognition processes. That is, all objects have their own characteristics, and the characteristics of these objects cannot be distinguished by a single threshold. Therefore, by identifying the inherent characteristics of the object, the threshold value that can maximize the recognition rate of each object is automatically determined.

Then, it is determined whether the boundary of interest exists (23).

As a result of the determination 23, if there is no boundary of interest, the base object is recognized using the motion vector (24).

The basic object recognition process 24 using the motion vector is repeatedly performed when an image is first input or when an object to be tracked is not recognized. In addition, even if there is an object being tracked, the basic object recognition process is periodically performed for the emergence of an unknown object. In the basic object recognition process, a motion vector is generated by applying the matching of SURF feature descriptors (a type of local feature presenter), and the motion vectors are detected by detecting the motion vectors. Detect and recognize base object.

As a result of the determination 23, if the boundary of interest exists, the object is tracked in real time using the boundary of interest (25).

This process is an object tracking process effectively using the boundary of interest generated in the object recognition process described above. If there is an area set as the boundary of interest, the object tracking process is relatively simple and has a small amount of computation, without performing the above-described complex object recognition process. That is, the histogram of the motion vector is generated by analyzing the correlation between the current frame and the immediately preceding frame, and the moving object is detected (tracked) by analyzing the histogram.

Next, a detailed embodiment of a real-time object recognition and tracking method using the representative feature and its apparatus according to the present invention will be described in detail with reference to FIGS. 3 to 8.

3 is a detailed flowchart of an embodiment of a preprocessing process.

In order to recognize and track various kinds of objects quickly and accurately, a preprocessing process is performed to calculate representative points of objects based on various images of objects of interest and extract and learn their feature information.

The preprocessing process is a process of pre-calculating and recording various pieces of information necessary for the actual query image processing, as shown in FIG. 3. In the present invention, for effective real-time object recognition, a feature of extracting features of each type of image is performed by using a matching point matching algorithm in a preprocessing process. In general, a point of interest refers to a part of an image that contains a lot of information, that is, a point where the surrounding change is severe. In the present invention, a feature descriptor is generated for each image of the training data using a SURF technique, and a representative feature is detected. In the next process, a representative point that can best represent the object among the detected points of interest is selected. If there is a sufficient number of points of interest similar to each other in the SURF descriptor similarity, the point of interest is considered a representative point.

When the detection of the representative point is completed for each object, coefficients necessary for recognition are automatically calculated through additional learning. The matching ratio is calculated for a plurality of training images belonging to each object type, and then their average is calculated. Similarly, for training images of other objects, the matching ratio for each image is measured and their average value is calculated. For each object representative point, we compare each image of the training set to calculate the average match ratio and determine the optimal threshold for recognizing each object.

4 is a detailed flowchart of an embodiment of a real-time object recognition and tracking process according to the present invention.

In the object recognition process, which object is included in the query image is determined using the representative features acquired in the preprocessing process. The basic object recognition process is similar to the preprocessing process. First, the interest point and the range detection are performed on the current frame of the query image. Comparing the detected points of interest with the representative features constructed in the preprocessing process, calculate how much agreement is made. With respect to the points of interest of the current frame, if it is determined that there are more than a certain ratio of representative features of an object, it is finally considered that the current frame includes the object. Compared with the conventional method, there is an advantage in that multiple objects can be detected simultaneously in one query image.

The entire process of actually recognizing and tracking an object based on a preprocessing process and an object recognition technique is illustrated in FIG. 4. In FIG. 4, the left side (tracking motion from two consecutive frames, motion vector processing, object search and boundary box generation) represents the object recognition process, while the right side (masking the current frame from the boundary box, determining whether the object is recognized, boundary box refresh) Represents an object tracking process. These two processes work together to recognize the objects of interest included in the image and track them.

Object recognition is repeatedly performed when an image is first input or when an object to be tracked is not recognized. Also, even if there is an object being tracked, this process is periodically performed in consideration of the appearance of another object that may be present. In this process, we apply the matching of SURF feature descriptors to generate motion vectors, and detect which objects are moving by detecting vectors showing motions different from the entire motion vectors. By creating an area containing the detected objects, the performance is improved by considering only the inside of the area, not the entire frame area of object tracking.

FIG. 5 is a diagram for describing a process of generating a motion vector and detecting a moving object. FIG. 5 illustrates a result of detecting a moving object using two motion vectors for two consecutive frames in an image in which the entire viewpoint moves to the upper left. .

First, SURF feature descriptors are extracted and matched for the current frame (top left) and the previous frame (top right), respectively. The white arrows represent the path traveled between the matched descriptors afterwards (lower left). In this case, it can be seen that the arrows mostly point to the lower left corner. Generating histograms based on direction and size shows that the variance is very large compared to others. This is because most of the feature points move in the same direction and size because the viewpoint of the image moves to one place as described above.

Therefore, in the present invention, a vector having such a large variable is defined as a representative motion vector. Subtracting the representative motion vector obtained from the white arrow vectors of the current frame (mid) and displaying the remaining portion is the same as the next frame (bottom). These remaining vectors represent SURF descriptors that show different motion than the full motion vector. That is, it can be seen that the object of interest is likely to have a different movement from the background. Therefore, the existence of the object is determined by comparing these descriptors with the representative feature descriptors generated during the preprocessing. If the object of interest is recognized through this process, the region is set as the region of interest and processing of the frame is completed.

Meanwhile, as shown in FIG. 4, in the object tracking process, object tracking is performed based on the ROI generated in the object recognition process. By masking the current frame of the input image using the ROI, an area excluding the ROI is excluded. The object is recognized in the corresponding area to determine whether the object actually exists. If it is determined that the object exists, the current region of interest must be updated to track the object in the next frame. Here, the region of interest is generated based on the region where the object is actually recognized in the previous frame and the current frame. In the present invention, the size of the ROI is set to 1.5 times the size of the area where the real object is recognized in the current frame, and the position is moved as the object moves from the previous frame to the current frame. This method is similar to the Kalman filter in that it uses only the information up to the previous frame, but has an advantage that the Kalman filter can have a similar effect with a very small amount of calculation.

Next, the performance of the object recognition and tracking method described above through experiments is as follows. The experiment was performed in "Intel Core 2 Duo 2.67Ghz, Windows 7 environment with 4GB memory specification." All processes including preprocessing and query processing were implemented based on "MATLAB", and the SURF algorithm module implemented in "OpenCV" was compiled and used as mex. "Caltech101" was used as a set of training images for the experiment. In this experiment, 20 images were selected and trained for the three categories of “stop sign”, “bikes” and “face”. The actual query was conducted by receiving a 30fps image with a resolution of 640x80 from the camera for about 4 minutes.

6 is a diagram for describing a process of determining a threshold value of a representative point.

First, we find the optimal representative point threshold by experiment. In FIG. 6, the broken line graphs show recognition rates for each classification of the frame, and the bar graph shows the sum of all frame recognition rates. As can be seen in Figure 6, when the threshold value is 0.3 showed the best recognition rate. If the threshold is lowered, the recognition rate drops because too many points are selected as representative points. On the contrary, if the threshold value is too high, the number of points selected as the representative point is too small, which adversely affects the recognition rate. However, even in this case, as shown in the graph, when no object is included in the frame, it shows a good recognition rate, but it is correct to interpret that it finds nothing more than the recognition rate is improved. Therefore, the representative point threshold was 0.3 in all subsequent experiments.

7 is a diagram illustrating an object recognition rate according to an average matching ratio.

Next, the experiment for the average matching ratio. As described above, two kinds of average matching ratios can be used, one of the minimum matching ratio values detected in corresponding object images, and the other of the highest matching ratio values measured in matching with other object images. . The experiment using two thresholds is as follows.

The graph labeled 'Minimum Average Match Rate' is the result of the minimum match rate value detected in the object images, and the graph labeled 'Maximum Average Match Rate' is detected in the match with other object images. This is the result of the highest match ratio value. In addition, 1 to 3 means each object used in the experiment. The two graphs are the results of experiments on accuracy and false alarms, respectively. This allows you to see the characteristics of object recognition when each threshold is used.

If the threshold value is used as the 'minimum average matching ratio', the overall recognition rate increases. If it is recognized correctly or if it is incorrectly recognized, it will be high. On the other hand, when the threshold value is used as the 'maximum average matching ratio', the overall recognition rate decreases, and the false recognition rate is almost less than 5%. This also shows the possibility that the object recognition algorithm can be used in various environments. Systems with a variety of purposes, such as using a 'minimum average match rate' threshold if not allowed, even if there is a slight chance of being misrecognized, and vice versa The algorithm can be used effectively in.

Table 1 below shows the results of the object recognition time and shows how many frames per second can be processed when real object recognition is performed using the method proposed by the present invention. From the experimental images in which various objects appear in various forms, the average number of processed frames was measured for the case of using only the basic object recognition process and using both the basic object recognition process and the object tracking process in parallel. As a result, the latter recognizes about 1.5 times faster object recognition speed than the former. This is the result of an object being recognized because it considers only a certain area, rather than the whole area, for a few frames. Experimental results show that only 3.21 frames per second can be processed using the basic object recognition method, but up to 4.43 frames per second can be processed by further object tracking.

Basic object recognition Basic Object Recognition + Object Tracking Average Processing Frames Per Second (fps) 3.21 4.43

8 is a diagram illustrating a result of a multi-object recognition and tracking experiment.

The three objects recognized in FIG. 8 are indicated by red, green, and blue boxes, respectively. In particular, the figure at the top shows that the "stop sign" object recognizes and tracks the object well in the moving image to the right. The figure below shows that all objects are recognized correctly when there are 0 to 3 different objects of interest in the image at the same time. In addition, since the SURF descriptor is used, it can be seen that it has robust characteristics in various sizes, rotations, lighting changes, and the like.

On the other hand, the real-time object recognition and tracking method according to the present invention as described above may be implemented in the form of program instructions that can be executed by various computer means may be recorded on a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be those specially designed and constructed for the present invention or may be available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tape, optical media such as CD-ROMs, DVDs, and magnetic disks, such as floppy disks. Magneto-optical media, and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. The medium may be a transmission medium such as an optical or metal line, a wave guide, or the like, including a carrier wave for transmitting a signal designating a program command, a data structure, or the like. Examples of program instructions include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. The hardware device may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, but, on the contrary, Various permutations, modifications and variations are possible without departing from the spirit of the invention.

Therefore, the scope of the present invention should not be construed as being limited to the embodiments described, but should be determined by the scope of the appended claims, as well as the appended claims.

11: representative feature extraction unit 12: coefficient determination unit
13 determination unit 14 object recognition unit
15: object tracking unit

Claims (6)

In the object recognition and tracking device,
A representative feature extracting unit for extracting a representative feature of the object;
A coefficient determination unit for determining coefficients according to characteristics of the object;
A determination unit for determining whether a boundary of interest exists;
An object recognition unit for recognizing a basic object using a motion vector when there is no boundary of interest as a result of the determination of the determination unit; And
As a result of the determination of the determination unit, if there is a boundary of interest, an object tracking unit for tracking an object in real time using the boundary of interest
Object recognition and tracking device comprising a.
The method of claim 1,
The object recognition unit,
Apparatus for recognizing a basic object by detecting a moving object by generating a motion vector by applying the matching of SURF feature descriptors, detecting vectors showing a motion different from the entire motion vector.
3. The method according to claim 1 or 2,
The object tracking unit,
And analyzing the correlation between the current frame and the immediately previous frame to generate a histogram of the motion vector and tracking the moving object by analyzing the generated histogram.
In the object recognition and tracking method,
(a) extracting representative features of the object;
(b) determining coefficients according to features of the object;
(c) determining if a boundary of interest exists;
(d) recognizing a basic object using a motion vector if there is no boundary of interest as a result of the determination of step (c); And
(e) tracking the object in real time using the boundary of interest if the boundary of interest exists as a result of the determination of step (c).
Object recognition and tracking method comprising a.
The method of claim 4, wherein
The step (d)
A method of recognizing and tracking an object by generating a motion vector by applying a match of a SURF feature descriptor, detecting vectors showing motions different from the entire motion vector, and detecting a moving object.
The method according to claim 4 or 5,
The step (e)
And analyzing the correlation between the current frame and the immediately preceding frame to generate a histogram of the motion vector and tracking the moving object by analyzing the generated histogram.

Images