KR102664901B1 - Apparatus for estimating visual odometry and method thereof - Google Patents

Abstract

The present invention discloses a visual mileage estimation device and method. The visual mileage estimation device of the present invention includes a camera unit that acquires images by photographing the surrounding environment while the vehicle is running; An image processing unit that receives an image from a camera unit, divides the ROI, extracts feature points, and removes outliers; and a driving distance estimation unit that estimates the driving distance based on the feature points extracted from the image processing unit.

Description

Visual mileage estimation device and method {APPARATUS FOR ESTIMATING VISUAL ODOMETRY AND METHOD THEREOF}

The present invention relates to a visual driving distance estimation device and method. More specifically, when extracting feature points from an image to estimate the visual driving distance, ROI (Region Of Interest) is divided and extracted uniformly for each ROI. This relates to a visual mileage estimation device and method for dividing ROI by dynamically changing the ROI factor according to the number of points (outliers) and adjusting the degree of ROI division.

In general, when measuring the driving distance for autonomous driving in a mobile robot, etc., the driving distance is measured using encoder data attached to the mobile robot. When using encoder data attached to a mobile robot, it is limited to only mobile robots, so it is difficult to measure the driving distance by applying it to various types of robots.

Recently, visual driving distance has been measured using temporally continuous video images.

Here, visual driving distance means estimating the change in pose of the robot between two consecutive images through image processing. At this time, visual odometry is based on measurements in a static environment, that is, an environment in which nothing moves.

In this way, visual inertial odometry, or visual odometer, which measures visual driving distance, is a technology that analyzes camera images without using GPS to determine location and direction based on terrain features, and is used in the robot and AR industries. It is widely applied

The background technology of the present invention is disclosed in Republic of Korea Patent Publication No. 10-1866660 (announced on June 12, 2018, RGB-D visual odometry measurement method and device based on a background model in a dynamic environment).

In this way, the visual odometer based on visual odometry has a significant impact on the positioning performance depending on the outlier removal performance.

Here, if the feature points detected through the image before removing the outliers are concentrated in a moving object, not only will the number of feature points remaining after removing the outliers decrease, but the probability that the outliers will not be properly removed increases, reducing the driving distance. There was a problem in that the positioning performance deteriorated by increasing the contamination level of the Kalman filter for estimation.

Additionally, when driving in a section where there are many dynamic objects, there is a problem that outlier removal performance deteriorates because the probability that feature points are concentrated in dynamic objects increases.

The present invention was created to improve the problems described above. The purpose of the present invention according to one aspect is to divide the ROI and extract the feature points uniformly for each ROI when extracting feature points from an image to estimate the visual driving distance. It provides a visual mileage estimation device and method that divides the ROI by dynamically changing the ROI factor according to the number of points (outliers) and adjusting the degree of ROI division.

A visual mileage estimation device according to one aspect of the present invention includes a camera unit that acquires images by photographing the surrounding environment while the vehicle is running; An image processing unit that receives an image from a camera unit, divides the ROI (Region of Interest), extracts feature points, and removes outliers; and a driving distance estimation unit that estimates the driving distance based on the feature points extracted from the image processing unit.

In the present invention, the image processing unit includes an image dividing unit that receives an image and divides the image according to a set ROI factor; a feature point detection unit that detects feature points for each ROI divided in the image segmentation section and selects and extracts feature points with strong metrics; and an outlier point removal unit that determines a dynamic object from the feature points extracted from the feature point detection unit and removes the outliers.

In the present invention, the image processing unit further includes an ROI factor setting unit that adjusts and sets the ROI factor according to the number of outliers.

In the present invention, the ROI factor setting unit adjusts the ROI factor to a greater extent as the number of outliers increases.

In the present invention, the feature point detection unit detects feature points using a corner-based algorithm.

In the present invention, the feature point detection unit generates metric information for each feature point using the difference in brightness of the feature points, sets priority using the metric information, sets the minimum distance between each feature point, and generates metric information for each feature point based on the metric information and minimum distance. It is characterized by equally selecting and extracting feature points for each ROI as many as are used in the filter, and then merging the extracted feature points.

In the present invention, the outlier removal unit determines dynamic objects using the RANSAC algorithm.

A visual mileage estimation method according to another aspect of the present invention includes the steps of an image processing unit receiving an image from a camera unit; A step where the image processing unit divides the image based on the set ROI factor; A step where the image processing unit detects and extracts feature points for each ROI of the divided image; A step where the image processing unit removes outliers from the extracted feature points; and a step of the driving distance estimating unit estimating the driving distance based on feature points from image measurements from which outliers have been removed.

The present invention is characterized in that the image processing unit further includes a step of adjusting and setting the ROI factor according to the number of outliers.

In the present invention, when adjusting the ROI factor, the image processing unit adjusts the ROI factor to be larger as the number of outliers increases.

In the present invention, the step of detecting and extracting feature points includes: an image processing unit detecting feature points and then generating metric information for each feature point using the brightness difference; Setting priorities by an image processing unit using metric information; Setting the minimum distance between each feature point by the image processing unit; A step where the image processing unit equally selects and extracts feature points for each ROI as many as are used in the filter based on priority and minimum distance; and a step of the image processing unit merging the extracted feature points for each ROI.

In the present invention, when detecting a feature point, the image processing unit detects the feature point using a corner-based algorithm.

In the present invention, the step of removing outliers is characterized in that the image processing unit determines the dynamic object from the extracted feature points and removes the outliers.

In the present invention, the step of removing outliers is characterized in that the image processing unit determines the dynamic object using the RANSAC algorithm.

When extracting feature points from an image to estimate the visual driving distance, the visual driving distance estimation device and method according to an aspect of the present invention divide ROI (Region Of Interest) and extract it uniformly for each ROI, and outlier points (outliers) are extracted uniformly for each ROI. ) By dynamically changing the ROI factor according to the number and dividing the ROI by adjusting the degree of ROI division, the problem of feature points being detected crowded in a specific section can be solved, and outlier removal performance and filter stability can be improved as well. By dynamically adjusting the degree of ROI division, resource efficiency can be improved by reducing the resources of the system used to divide the ROI and merge the extracted feature points.

1 is a block diagram showing a visual mileage estimation device according to an embodiment of the present invention.
Figure 2 is a flowchart illustrating a visual mileage estimation method according to an embodiment of the present invention.
Figure 3 is a flowchart illustrating the feature point extraction process in the visual mileage estimation method according to an embodiment of the present invention.

Hereinafter, a visual mileage estimation device and method according to the present invention will be described with reference to the attached drawings. In this process, the thickness of lines or sizes of components shown in the drawings may be exaggerated for clarity and convenience of explanation. In addition, the terms described below are terms defined in consideration of functions in the present invention, and may vary depending on the intention or custom of the user or operator. Therefore, definitions of these terms should be made based on the content throughout this specification.

1 is a block diagram showing a visual mileage estimation device according to an embodiment of the present invention.

As shown in FIG. 1, the visual driving distance estimation device according to an embodiment of the present invention may include a camera unit 10, an image processing unit 20, and a driving distance estimating unit 30.

The camera unit 10 may provide images captured of the surrounding environment while the vehicle is driving to the image processing unit 20.

The image processing unit 20 may receive an image from the camera unit 10, divide the ROI, extract feature points, and remove outliers.

Here, the image processing unit 20 may include an image segmentation unit 210, a feature point detection unit 220, an outlier point removal unit 230, and an ROI factor setting unit 240.

The image segmentation unit 210 may receive an image and divide the image according to a set ROI factor.

For example, if the ROI factor is set to 1, it is not divided, if the ROI factor is set to 2, the image can be divided into 4 parts (2x2), and if the ROI factor is set to 3, the image can be divided into 9 parts (3x3). At this time, the more congested the road is with many dynamic objects, the more segmentation can be done by increasing the ROI factor.

The feature point detection unit 220 may detect feature points for each ROI divided by the image segmentation unit 210 and select and extract feature points with a strong metric.

Here, the feature point detection unit 220 may detect feature points using corner-based algorithms such as FAST (Feature form Accelerated Segment Test), Shi-Tomasi, and Harris-corner.

At this time, the feature point detection unit 220 can adjust the number of detected feature points by adjusting the threshold factor value of the applied corner-based algorithm. Here, if the threshold is lowered, a large number of feature points are detected and the amount of computation increases when selecting feature points. Therefore, tuning can be performed to find the optimal value of the threshold.

Additionally, the feature point detection unit 220 generates metric information for each feature point using the difference in brightness of the feature points, and sets the priority in descending order using the metric information. Additionally, the feature point detection unit 220 may set a minimum distance between each feature point in order to equally extract feature points as many as the number used in the filter.

Based on the priority-set metric information and minimum distance, the feature point detection unit 220 equally selects and extracts feature points for each ROI as many as are used in the filter, and then merges the feature points extracted for each ROI.

The outlier point removal unit 230 may determine a dynamic object from the feature points extracted by the feature point detection unit 220 and remove the outliers.

Here, the outlier removal unit 230 uses the RANSAC (RANdom SAmple Consensus) algorithm to select the model with the maximum consensus, that is, supported by the largest number of data, to determine the dynamic object and remove the outlier. You can.

The ROI factor setting unit 240 adjusts and sets the ROI factor according to the number of outliers so that the image can be divided according to the ROI factor set in the image segmentation unit 210.

Here, the ROI factor setting unit 240 can adjust the ROI factor to be larger as the number of outliers increases.

For example, the ROI factor setting unit 240 may determine whether the number of outliers is greater or less than the threshold, and if it is greater than the threshold, set the ROI factor to be greater than 1. In addition, when the threshold is 50 and the number of outliers is 50 to 75, it is judged to be a normal driving road and the ROI factor is set to 2, allowing the image to be divided into 4 (2x2). If the number of outliers is 75 to 100, the vehicle is It is judged to be a congested road and the ROI factor is set to 3 so that the image can be divided into 9 segments (3x3). If the number of outliers is less than the threshold, the ROI factor can be initially set to 1 by determining that it is a highway or general road with few vehicles.

The driving distance estimation unit 30 can estimate the visual driving distance by determining the position and direction between images by applying a Kalman filter based on feature points from image measurements from which outliers have been removed in the image processing unit 20.

As described above, according to the visual mileage estimation device according to an embodiment of the present invention, when extracting feature points from an image to estimate the visual mileage, ROI (Region Of Interest) is divided and extracted uniformly for each ROI, By dividing the ROI by dynamically changing the ROI factor according to the number of outliers and adjusting the degree of ROI division, the problem of feature points being detected concentrated in a specific section can be solved and the outlier removal performance and filter stability can be improved. In addition, by dynamically adjusting the degree of ROI division, resource efficiency can be improved by reducing the resources of the system used to divide the ROI and merge the extracted feature points.

Figure 2 is a flowchart for explaining a visual mileage estimation method according to an embodiment of the present invention, and Figure 3 is a flowchart for explaining a feature point extraction process in the visual mileage estimation method according to an embodiment of the present invention.

As shown in FIG. 2, in the visual mileage estimation method according to an embodiment of the present invention, first, the image processing unit 20 receives an image from the camera unit 10 (S10).

After receiving the image in step S10, the image processing unit 20 divides the image based on the set ROI factor (S20).

For example, if the ROI factor is set to 1, it is not divided, if the ROI factor is set to 2, the image can be divided into 4 parts (2x2), and if the ROI factor is set to 3, the image can be divided into 9 parts (3x3). At this time, the more congested the road is with many dynamic objects, the more segmentation can be done by increasing the ROI factor.

After dividing the image in step S20, the image processing unit 20 detects and extracts feature points for each ROI of the divided image (S30).

Here, the image processing unit 20 may detect feature points using corner-based algorithms such as FAST (Feature form Accelerated Segment Test), Shi-Tomasi, and Harris-corner. At this time, the number of detected feature points can be adjusted by adjusting the threshold factor value of the corner-based algorithm applied. Here, if the threshold is lowered, a large number of feature points are detected and the amount of computation increases when selecting feature points. Therefore, tuning can be performed to find the optimal value of the threshold.

To explain this in more detail, as shown in FIG. 3, the image processing unit 20 detects a feature point and then uses the brightness difference to generate metric information for each feature point (S300).

After generating metric information for each feature point in step S300, the image processing unit 20 sets priorities in descending order using the metric information (S310).

Additionally, the image processing unit 20 sets the minimum distance between each feature point to equally extract the number of feature points used in the filter (S320).

After setting priorities for feature points and setting the minimum distance between each feature point in steps S310 and S320, the image processing unit 20 equalizes the number of ROIs used in the filter based on the priority-set metric information and the minimum distance. Feature points are selected and extracted (S330).

After extracting the feature points for each ROI in step S330, the image processing unit 20 merges the feature points extracted for each ROI (S340).

After extracting the feature points in this way, the image processing unit 20 uses the RANSAC (RANdom SAmple Consensus) algorithm from the extracted feature points to select the model with the maximum consensus, that is, the model supported by the largest number of data to determine the dynamic object. and remove outliers (S40).

Meanwhile, after removing the outliers in step S40, the image processing unit 20 adjusts and sets the ROI factor according to the number of outliers (S50).

Here, the image processing unit 20 can adjust the ROI factor to be larger as the number of outliers increases.

For example, the ROI factor setting unit 240 may determine whether the number of outliers is greater or less than the threshold, and if it is greater than the threshold, set the ROI factor to be greater than 1. In addition, when the threshold is 50 and the number of outliers is 50 to 75, it is judged to be a normal driving road and the ROI factor is set to 2, allowing the image to be divided into 4 (2x2). If the number of outliers is 75 to 100, the vehicle is It is judged to be a congested road and the ROI factor is set to 3 so that the image can be divided into 9 segments (3x3). If the number of outliers is less than the threshold, the ROI factor can be initially set to 1 by determining that it is a highway or general road with few vehicles.

From the image measurements from which outliers have been removed in step S40, the driving distance estimation unit 30 applies a Kalman filter based on the feature points to determine the position and direction between images to estimate the visual driving distance (S60).

As described above, according to the visual driving distance estimation method according to an embodiment of the present invention, when extracting feature points from an image to estimate the visual driving distance, ROI (Region Of Interest) is divided and extracted uniformly for each ROI, By dividing the ROI by dynamically changing the ROI factor according to the number of outliers and adjusting the degree of ROI division, the problem of feature points being detected concentrated in a specific section can be solved and the outlier removal performance and filter stability can be improved. In addition, by dynamically adjusting the degree of ROI division, resource efficiency can be improved by reducing the resources of the system used to divide the ROI and merge the extracted feature points.

Implementations described herein may be implemented, for example, as a method or process, device, software program, data stream, or signal. Although discussed only in the context of a single form of implementation (eg, only as a method), implementations of the features discussed may also be implemented in other forms (eg, devices or programs). The device may be implemented with appropriate hardware, software, firmware, etc. The method may be implemented in a device such as a processor, which generally refers to a processing device that includes a computer, microprocessor, integrated circuit, or programmable logic device. Processors also include communication devices such as computers, cell phones, portable/personal digital assistants (“PDAs”) and other devices that facilitate communication of information between end-users.

The present invention has been described with reference to the embodiments shown in the drawings, but these are merely exemplary, and those skilled in the art will recognize that various modifications and other equivalent embodiments are possible therefrom. You will understand.

Therefore, the true technical protection scope of the present invention should be determined by the claims below.

10: camera unit 20: image processing unit
30: mileage estimation unit 210: image division unit
220: Feature point detection unit 230: Outlier point removal unit
240: ROI factor setting unit

Claims (14)

A camera unit that acquires images by photographing the surrounding environment while the vehicle is driving;
An image processing unit that receives the image from the camera unit, divides the ROI (Region of Interest), extracts feature points, and removes outliers; and
Includes a driving distance estimation unit that estimates the driving distance based on the feature points extracted from the image processing unit,
The image processing unit divides the ROI by adjusting the ROI factor, detects feature points for each ROI, and then equally selects and extracts feature points for each ROI using metric information of the feature points.
The method of claim 1, wherein the image processing unit,
an image segmentation unit that receives the image and divides the image according to a set ROI factor;
a feature point detection unit that detects the feature points for each ROI divided by the image division unit and selects and extracts feature points with a strong metric; and
A visual mileage estimation device comprising: an outlier point removal unit that determines a dynamic object from the feature points extracted from the feature point detection unit and removes the outlier points.
The visual mileage estimation device according to claim 2, wherein the image processing unit further includes an ROI factor setting unit that adjusts and sets the ROI factor according to the number of outliers.
The visual mileage estimation device according to claim 3, wherein the ROI factor setting unit adjusts the ROI factor to be larger as the number of outliers increases.
The visual mileage estimation device according to claim 2, wherein the feature point detection unit detects the feature point using a corner-based algorithm.
The method of claim 2, wherein the feature point detector generates metric information for each feature point using a brightness difference between the feature points, sets a priority using the metric information, and sets a minimum distance between each feature point to determine the metric. A visual mileage estimation device characterized in that, based on information and the minimum distance, the feature points are equally selected and extracted for each ROI as many as used in the filter, and then the extracted feature points are merged.
The visual mileage estimation device according to claim 2, wherein the outlier removal unit determines the dynamic object using the RANSAC algorithm.
An image processing unit receiving an image from a camera unit;
The image processing unit dividing the image based on a set ROI (Region Of Interest) factor;
The image processing unit detecting and extracting feature points for each ROI of the divided image;
removing outliers from the extracted feature points by the image processing unit; and
Including a step where the driving distance estimator estimates the driving distance based on the feature points from the image measurements from which the outliers have been removed,
The step of detecting and extracting the feature point is,
A visual mileage estimation method wherein the image processing unit adjusts the ROI factor to detect feature points for each divided ROI, and then equally selects and extracts feature points for each ROI using metric information of the feature points.
The method of claim 8, further comprising the step of the image processing unit adjusting and setting the ROI factor according to the number of outliers.
The method of claim 9, wherein when adjusting the ROI factor, the image processing unit adjusts the ROI factor to be larger as the number of outliers increases.
The method of claim 8, wherein the step of detecting and extracting the feature point comprises:
After the image processing unit detects the feature point, generating metric information for each feature point using the brightness difference;
setting a priority by the image processing unit using the metric information;
The image processing unit setting a minimum distance between each feature point;
the image processing unit selecting and extracting the feature points equally for each ROI as many as the number used in the filter based on the priority and the minimum distance; and
A visual mileage estimation method comprising: the image processing unit merging the feature points extracted for each ROI.
The method of claim 11, wherein when detecting the feature point, the image processing unit detects the feature point using a corner-based algorithm.
The method of claim 8, wherein in the step of removing the outliers, the image processing unit determines a dynamic object from the extracted feature points and removes the outliers.
The method of claim 8, wherein in the step of removing the outlier, the image processing unit determines a dynamic object using a RANSAC algorithm.

Images