KR20220115422A - ORB SLAM-based object tracking device and method thereof - Google Patents

Abstract

The present invention proposes an ORB SLAM-based object tracking device for improving an object (image) tracking function by reducing failure frequency in the image tracking of an autonomous vehicle, and a method thereof. The ORB SLAM-based object tracking device of the present invention includes: a configuration of increasing the number of feature points by pre-processing an input image so that common feature points between a current frame and a key frame are sufficient; a configuration of adding a new keyframe to maintain continuity between the current keyframe and the previous keyframe. The possibility of the tracking failure is minimized.

Description

ORB SLAM-based object tracking device and method thereof

The present invention relates to ORB SLAM technology, and more particularly, to an ORB SLAM-based object tracking device and method for improving the tracking function of an object (image) by reducing the frequency of failures during image tracking in an autonomous vehicle. it's about

Recent research on autonomous driving is advancing autonomous driving technology by using sensors that can detect various objects while driving based on automatic control and vehicle dynamics.

Autonomous vehicles are equipped with various technologies for safety. ADAS (AEB, Lane Assisting System), C-V2X, Deep Learning, LIDAR, SLAM (Simultaneous Localization And Mapping), etc. Among them, SLAM technology creates a map of the surrounding environment without external help, and can play a role in resolving errors caused by GPS through mapping of the user's surrounding environment. These SLAM technologies can be divided into EKF SLAM, Fast SLAM, Graph SLAM, BA (Bundle Adjustment) SLAM, ORB SLAM, etc. depending on the algorithm. Among them, ORB SLAM (Oriented FAST and Rotated BRIEF Simultaneous Localization And Mapping) technology is a - Map generation - It supports a parallel system consisting of three threads of loop combining (tracking, local mapping, and loop closing), so it is used a lot because it is fast and can calculate 3D information more accurately. For reference, the tracking is to extract a feature from an image to generate a key frame and to estimate the position of the camera, the local mapping is to optimize the overlapping key frame and the map point generated after tracking, and the loop closing is the map It optimizes all keyframes and map points by correcting the errors accumulated during creation. In the ORB SLAM technique, a feature point is extracted from an input image by using the FAST detector of the ORB.

However, in the above-described method for extracting key points, in the case of an image with severe shaking or a large difference in environmental change, the number of sufficient key points shared between the current frame and the last key frame is small, so the problem of track loss is reduced. Occurs. Since the tracking failure occurs when the number of keyframes matched between the last keyframe and the current frame is typically less than 30, if the number of keyframes is less than this, it is difficult to track an object.

In addition, in the conventional ORB SLAM technology, when the number of feature points extracted from consecutive frames refers to less than 90% of the feature points of the previous keyframe, there is a possibility of tracking failure. In this case, a new keyframe must be created and provided, but the condition is not clear. As mentioned above, if the reference is less than 90%, a keyframe cannot be added, and continuity between the current keyframe and the previous keyframe could not be maintained.

Accordingly, it is an object of the present invention to provide an ORB SLAM-based object tracking apparatus and method for minimizing the frequency of tracking failure occurring due to a small number of sufficient feature points shared between a current frame and a last frame.

Another object of the present invention is to generate and provide a new keyframe according to the relaxed keyframe insertion condition when the number of feature points extracted from consecutive frames is less than 90% of the previous keyframe feature points to perform object tracking. To provide an ORB SLAM-based object tracking device and method for minimizing the frequency of tracking failure.

In order to achieve the above object, the present invention provides an object tracking apparatus based on ORB SLAM (Oriented FAST and Rotated BRIEF Simultaneous Localization And Mapping) technology, comprising: a feature point extraction unit for extracting feature points from input image information; a feature point number determination unit for determining the number of feature points shared between the current frame and the previous key frame; a frame preprocessor for preprocessing a current frame when the number of shared feature points is less than a predetermined number; a matching unit that performs a matching operation between the current frame and the previous key frame if the number is greater than or equal to a predetermined number after the frame preprocessing unit; and a keyframe generator that generates a new keyframe according to a new keyframe generation condition and provides the generated keyframe as the input image information.

The number of the feature points may be a number in which a tracking failure probability is minimized or a number in which a tracking failure does not occur.

The preprocessor further extracts the number of feature points by making the image of the current frame clearer than the current one.

The keyframe generation condition is that, when the number of feature points in the current frame is less than 90% of the previous keyframe feature points, one new keyframe is generated per 20 consecutive frames, but a new keyframe is generated in every frame. Change it so that it is provided.

According to another feature of the present invention, an object tracking apparatus based on ORB SLAM (Oriented FAST and Rotated BRIEF Simultaneous Localization And Mapping) includes extracting feature points from an input image; determining the number of keyframes shared between the current frame and the previous keyframe by using the extracted keyframes; pre-processing a current frame when the number of feature points is less than a predetermined number according to the determination; further extracting the number of feature points according to the preprocessing, and matching the current frame with the previous keyframe when the number is greater than or equal to a predetermined number; generating a new keyframe according to a keyframe generation condition when the number of feature points of the current frame is less than 90% of the feature points of a previous keyframe to track an object; and an ORB SLAM-based object tracking method that provides the generated keyframe to the input image, and performs object recognition and tracking while repeating the above steps.

In the step of extracting the key points, the key points are extracted using Oriented FAST and Rotated BRIEF (ORB).

The keyframe generation condition is provided by changing from generating one new keyframe per 20 consecutive frames to generating a new keyframe in every frame.

The predetermined number is a threshold number in which tracking failure occurs due to the number of keyframes matched between the current keyframe and the previous keyframe, and the threshold number is 30.

According to the ORB SLAM-based object tracking apparatus and method of the present invention as described above, the number of feature points is increased by preprocessing the input image so that the common feature points between the current frame and the key frame are sufficient, and the number of feature points is increased between the current key frame and the previous key frame. In order to maintain continuity between keyframes, new keyframes are added to maximize the input image information.

Accordingly, the present invention can minimize the possibility of tracking failure caused by insufficient number of keyframes matched between the current frame and the previous keyframe or failure to add a new keyframe. As a result, the improvement of object recognition or object tracking functions in autonomous vehicles can be expected.

1 is a block diagram of an ORB SLAM-based object tracking device according to a preferred embodiment of the present invention;
2 is a flowchart illustrating an ORB SLAM-based object tracking method according to a preferred embodiment of the present invention.

Objects and effects of the present invention, and technical configurations for achieving them will become clear with reference to the embodiments described below in detail in conjunction with the accompanying drawings. In describing the present invention, if it is determined that a detailed description of a well-known function or configuration may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted.

In addition, the terms described below are terms defined in consideration of functions in the present invention, which may vary according to intentions or customs of users and operators.

However, the present invention is not limited to the embodiments disclosed below and may be implemented in various different forms. Only the present embodiments are provided so that the disclosure of the present invention is complete, and to completely inform those of ordinary skill in the art to which the present invention belongs, the scope of the invention, the present invention is defined by the scope of the claims it will only be Therefore, the definition should be made based on the content throughout this specification. Therefore, the spirit of the present invention should not be limited to the described embodiments, and not only the claims to be described later, but also all those with equivalent or equivalent modifications to the claims will be said to belong to the scope of the spirit of the present invention. .

Hereinafter, the present invention will be described in more detail based on the embodiments shown in the drawings.

1 is a block diagram of an ORB SLAM-based object tracking apparatus according to a preferred embodiment of the present invention. As shown in FIG. 1 , the apparatus 100 of the present invention includes a feature point extracting unit 110 , a feature point number determining unit 120 , a frame preprocessing unit 130 , a matching unit 140 , and a keyframe generating unit 150 . ) is included.

The key point extraction unit 110 is a unit for extracting key points from image information input through an input device (not shown). For image tracking, since the number of keyframes matched between the current frame and the last keyframe must be a predetermined number or more, it is necessary to extract keypoints for each frame. The reason for extracting the keyframe in this way is that a process of selecting a keyframe must be performed based on the number of keyframes shared between the current frame and the previously generated frame.

The feature point number determining unit 120 is a unit that determines whether the number of feature points shared between the current frame and the previous key frame is less than or equal to a predetermined number (that is, 30). Depending on the determination result, the next process may be performed differently.

The frame preprocessor 130 is a unit that pre-processes the current frame so that the number of key points can be further extracted when the number of key points is less than the above-described number (ie, 30). The pre-processing may be, for example, an operation to sharpen the frame. In other words, there are many common features between the current frame and the previous keyframe.

The matching unit 140 is a unit that performs a matching operation between the current frame and a previously generated key frame when the number of feature points is 30 or more. That is, the matching unit 140 is performed when the number of matched feature points between the current frame and the previous keyframe is 30 or more to minimize the image tracking failure probability. Here, the number of feature points is mentioned as 30, but it is not necessarily limited thereto. This is only one embodiment, and even when the number of shared feature points is less than that, if the probability of image tracking failure is low, the number of feature points that are determined may be changed.

The keyframe generating unit 150 is a unit that generates a new keyframe in every frame from which one new frame is generated per 20 consecutive frames in order to prevent the possibility of tracking failure. At this time, the keyframe generation unit 150 reflects the new keyframe generation condition. This condition is that a new keyframe is generated in every frame instead of one new frame per 20 consecutive frames. In other words, if the number of keyframes refers to only 90% of the keyframe keyframes, there is a possibility of image tracking failure.

The present invention having such a configuration is to further facilitate image tracking by reducing the keyframe matching failure rate by modifying image sharpening and keyframe generation conditions, which will be described in detail with reference to FIG. 2 .

2 is a flowchart illustrating an ORB SLAM-based object tracking method according to a preferred embodiment of the present invention. As shown, peripheral image information is input from an input device provided in an autonomous vehicle or the like (s100).

Then, the key point extraction unit 110 extracts the key point from the image information input through the input device using the ORB (s100). In order to minimize the image tracking failure, the number of feature points shared between the current frame and the last keyframe should be sufficient. The key point extraction unit 110 extracts a key point for each frame of the input image information.

When the keypoints are extracted from the current frame, the number of keypoints determining unit 120 determines whether the number of keypoints shared between the current frame and the previous keyframe is less than a predetermined condition, for example, 30 (s120). The reason for determining the number of keyframes is that if the number of keyframes between the current frame and the previously generated keyframe is less than 30, image tracking fails.

Therefore, as a result of the determination, if the number of matched feature points between the current frame and the key frame is 30 or more, the matching operation between the current frame and the key frame will be immediately performed (s140). On the other hand, as a result of the determination of the number of keyframes, the number of keyframes in the frame is less than 30, and even if the current frame and the previous keyframe are matched, there is a possibility that the image cannot be tracked. Therefore, in this case, a process of increasing the number of feature points of the current frame is required. In the present invention, when the number of feature points is less than 30, the frame preprocessor 130 performs a preprocessing process to improve the image tracking function (s130). That is, the pre-processing process enables the extraction of the number of feature points from the current frame further, and for this purpose, the image of the current frame is sharpened to extract the feature points. A sharpening mask may be used as an example of a method of image sharpening. In addition, the preprocessing process may be repeatedly performed until the number of feature points shared between the current frame and the previous keyframe reaches 30 or more.

According to this process, when the number of feature points is 30 or more, the aforementioned matching unit 140 performs a matching operation (s140) and selects a keyframe.

After matching, it is necessary to maintain continuity between the current keyframe and the previous keyframe, and in some cases a new keyframe is required. This is also a method to reduce the possibility of image tracking failure.

As shown in Figure 2, if the number of feature points of the current key frame refers to 90% or more of the feature points of the previous key frame ('No' in s150), the probability of image tracking failure is low, so return to the image input step and repeat the above process. will be performed repeatedly. However, when the number of feature points of the current keyframe refers to a number less than 90% of the feature points of the previous keyframe, the possibility of image tracking failure still exists. Therefore, in this case, it is necessary to create a new keyframe to maintain continuity between the current keyframe and the previous keyframe.

In the present invention, a new keyframe is added by relaxing the keyframe insertion condition. According to the relaxed condition, one per 20 consecutive frames, which is a condition for generating a new keyframe, may be changed to all frames, and the keyframe generator 150 may generate a new keyframe (s160). Here, the new keyframe may be referred to as a frame for maintaining continuity between the current keyframe and the previous keyframe as described above. Then, when a new key frame is generated, after adding the key frame, the image input step proceeds (s170).

As such, the present invention increases the number of feature points through a pre-processing process through image sharpening when the number of feature points between the current frame and the keyframe is small, and furthermore, to maintain continuity between the current keyframe and the previous keyframe, the By adding new keyframes according to conditions, it can be seen that the image tracking performance is improved by lowering the matching failure rate of keyframes.

Although described with reference to the illustrated embodiments of the present invention as described above, these are merely exemplary, and those of ordinary skill in the art to which the present invention pertains can use various methods without departing from the spirit and scope of the present invention. It will be apparent that modifications, variations, and other equivalent embodiments are possible. Therefore, the true technical protection scope of the present invention should be determined by the technical spirit of the appended claims.

110: feature point extraction unit
120: feature point number determination unit
130: frame preprocessor
140: matching unit
150: keyframe generator

Claims (8)

In an object tracking device based on ORB SLAM (Oriented FAST and Rotated BRIEF Simultaneous Localization And Mapping) technology,
a feature point extraction unit for extracting a feature point from the input image information;
a feature point number determination unit for determining the number of feature points shared between the current frame and the previous key frame;
a frame preprocessor for preprocessing a current frame when the number of shared feature points is less than a predetermined number;
a matching unit for performing a matching operation between a current frame and a previous key frame if the number is greater than or equal to a predetermined number after the frame preprocessing unit; and
ORB SLAM-based object tracking apparatus, comprising: a key frame generator that generates a new key frame according to a new key frame generation condition and provides the generated key frame as the input image information. The method of claim 1,
The number of feature points is
ORB SLAM-based object tracking device, which is the number that minimizes the probability of tracking failure or the number that does not cause tracking failure. The method of claim 1,
The preprocessor is
ORB SLAM-based object tracking apparatus for further extracting the number of feature points by making the image of the current frame clearer than the current one. The method of claim 1,
The keyframe generation condition is:
ORB SLAM-based object tracking device that provides a change from one frame to all frames per 20 consecutive frames to generate a new keyframe when the number of keyframe points in the current frame is less than 90% of the previous keyframe keyframe . ORB SLAM (Oriented FAST and Rotated BRIEF Simultaneous Localization And Mapping) based object tracking device,
extracting feature points from the input image;
determining the number of keyframes shared between the current frame and the previous keyframe by using the extracted keyframes;
pre-processing a current frame when the number of feature points is less than a predetermined number according to the determination;
further extracting the number of feature points according to the pre-processing, and matching the current frame with the previous key frame when the number is greater than or equal to a predetermined number;
generating a new keyframe according to a keyframe generation condition when the number of feature points of the current frame is less than 90% of the feature points of a previous keyframe to track an object; and
ORB SLAM-based object tracking method, characterized in that the generated keyframe is provided to the input image, and object recognition and tracking are performed while repeating the above steps. 6. The method of claim 5,
In the step of extracting the feature point,
The feature point is an ORB SLAM-based object tracking method that is extracted using Oriented FAST and Rotated BRIEF (ORB). 6. The method of claim 5,
The keyframe generation condition is:
ORB SLAM-based object tracking method that generates new keyframes by changing from 1 frame to every frame per 20 consecutive frames. 6. The method of claim 5,
The predetermined number is
ORB SLAM-based object tracking method, which is the threshold number of which tracking failure occurs due to the number of keyframes matched between the current keyframe and the previous keyframe.

Images