WO2020010841A1

WO2020010841A1 - Autonomous vacuum cleaner positioning method and device employing gyroscope calibration based on visual loop closure detection

Info

Publication number: WO2020010841A1
Application number: PCT/CN2019/073757
Authority: WO
Inventors: 周毕兴; 张立新
Original assignee: 深圳市沃特沃德股份有限公司
Priority date: 2018-07-10
Filing date: 2019-01-29
Publication date: 2020-01-16
Also published as: CN109141395B; CN109141395A

Abstract

Provided are an autonomous vacuum cleaner positioning method and device using gyroscope calibration based on visual loop closure detection. The method comprises: acquiring, according to gyroscope information and odometer information correspondingly obtained between two key image frames of adjacent time points, actual location information with respect to a current image frame and a target historical image frame, and completing positioning. The actual location information of the current image frame can be quickly calculated on the basis of visual loop closure detection, thereby reducing computational complexity, and completing updating of the gyroscope and odometer information while achieving accurate positioning.

Description

Positioning method and device of sweeper based on visual loop calibration gyroscope

Technical field

The present application relates to the technical field of sweepers, and in particular to a sweeper positioning method and device based on a visual loop calibration gyroscope.

Background technique

As a kind of intelligent household appliances, the sweeper can automatically complete the floor cleaning work in the room by virtue of certain artificial intelligence, which is loved by consumers. The existing sweeper senses the environment and its own state through sensors, and then realizes autonomous movement in an environment with obstacles. Due to the existence of errors between the sweeper's own gyroscope and the odometer sensor, the sweeper will inevitably experience inaccurate positioning. Therefore, how to quickly calibrate the sweeper gyroscope and achieve accurate positioning is of great significance to the intelligence of the sweeper.

technical problem

The purpose of this application is to provide a positioning method and device for a sweeper based on a visual loop calibration gyroscope, to correct a positioning error generated during the work of the sweeper, and to reduce calculation complexity.

Technical solutions

To achieve the above objective, the present application provides a positioning method for a sweeper based on a visual loop calibration gyroscope, including:

Collect a picture of the current environment and choose to save the key frame picture in the picture of the current environment;

Acquiring and saving the corresponding gyroscope information and odometer information between two key frame pictures at adjacent times;

Find historical frame pictures similar to the current frame picture to form a visual loop;

Obtain the real position information between the current frame picture and the target historical frame picture through the gyroscope information and the odometer information, and the target historical frame picture is formed from the current frame picture in the historical frame picture Visual loopback pictures;

Update the gyroscope data and odometer data corresponding to the current frame picture according to the real position information to complete positioning.

The present application also provides a positioning device for a sweeper based on a visual loop calibration gyroscope, including:

An acquisition module, configured to acquire a picture of the current environment according to a preset frequency through a visual sensor, and select to save a key frame picture in the picture of the current environment;

An acquisition module for collecting a picture of the current environment and selecting a key frame picture in the picture of the current environment;

A detection module, configured to acquire and save gyroscope information and odometer information corresponding to two key frame pictures at adjacent times;

A calculation module, configured to obtain real position information between the current frame picture and the target historical frame picture through the gyroscope information and the odometer information;

An update module is configured to complete positioning according to the gyroscope data and odometer data corresponding to the current frame picture and the real position information.

The present application also provides a computer device including a processor, a memory, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, any one of the foregoing is implemented. The positioning method of the sweeper based on the visual loop calibration gyroscope.

Beneficial effect

The present application provides a positioning method and device for a sweeper based on a visual loop calibration gyroscope, which can quickly calculate the real position information of the current frame based on the visual loop. Under the premise of ensuring the positioning accuracy of the sweeper, a simpler method is used. Calibrate the gyro of the sweeper, reduce the computational complexity, correct the errors generated by the sweeper during its movement, and achieve accurate positioning.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic flowchart of a positioning method of a sweeper based on a visual loop calibration gyroscope according to an embodiment of the present application; FIG.

2 is a structural block diagram of a positioning device for a sweeper based on a visual loop calibration gyroscope according to an embodiment of the present application;

3 is a structural block diagram of a calculation module according to an embodiment of the present application;

4 is a structural block diagram of an update module according to an embodiment of the present application;

5 is a structural block diagram of an acquisition module according to an embodiment of the present application;

6 is a structural block diagram of a detection module according to an embodiment of the present application;

FIG. 7 is a schematic block diagram of a computer device according to an embodiment of the present application.

Best Mode of the Invention

It should be understood that the specific embodiments described herein are only used to explain the application, and are not used to limit the application.

Those skilled in the art will understand that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art to which this application belongs. It should also be understood that terms such as those defined in the general dictionary should be understood to have meanings consistent with the meanings in the context of the prior art, and unless specifically defined like this, they would not be idealized or overly Formal meaning to explain.

Referring to FIG. 1, a method for positioning a sweeper based on a visual loop calibration gyroscope according to an embodiment of the present application includes:

S1: Collect a picture of the current environment and choose to save the key frame picture in the picture of the current environment.

In this embodiment, the vision sensor of the sweeper starts to collect pictures of the surrounding environment when the sweeper starts to move, and selects a picture with rich reference objects as features to save as a key frame picture. Because the sweeper did not move at the beginning, there was no error in the gyroscope information and odometer information of the key frame picture at this time.

S2: Obtain and save the corresponding gyroscope information and odometer information between two key frame pictures at adjacent times.

In this embodiment, the sweeper will simultaneously record the gyroscope information and odometer information between two key frame pictures at the same time while collecting key frame pictures, so as to ensure that The interval is too long and the error is too large.

S3: Find historical frame pictures similar to the current frame picture.

In this embodiment, a bag-of-words model is established to accelerate feature matching, and each feature in a key frame picture is trained one-to-one into a visual dictionary, and the DBOW mapping of the feature points of the current frame picture in the visual dictionary is calculated. The key frame picture with the most similar frame picture is judged to form a visual loop.

S4: Obtain the real position information between the current frame picture and the target historical frame picture through the gyroscope information and odometer information. The target historical frame picture is the historical frame picture and the current frame picture. A picture that forms a visual loop.

In this embodiment, since there is a specific distortion coefficient between picture position information and odometer information between adjacent historical frame pictures, the coefficient S can be obtained through calculation. Based on the picture position information between the current frame picture and the historical frame picture that forms a visual loop with the current frame picture, the real position information between the two can be quickly calculated.

S5: Update gyroscope data and odometer data corresponding to the current frame picture according to the real position information to complete positioning.

In this embodiment, the sweeper automatically completes calibration when it detects that the current frame picture and the historical frame picture form a visual loop. For example, when the sweeper detects the first visual loop, the historical frame picture that forms a visual loop with the current frame picture is actually the initial position information of the sweeper without error. The odometry data and gyroscope data of the current frame picture are updated according to the calculated real position information and the gyroscope information of the historical frame. The sweeper can correct errors and achieve accurate positioning.

The step of obtaining the real position information between the current frame picture and the target historical frame picture through the gyroscope information and the odometer information includes:

S401: Acquire picture position information and odometer information between a specified number of the historical frame pictures adjacent to the current frame picture;

S402: Acquire picture position information between the current frame picture and the target historical frame picture;

S403: The true position information between the current frame picture and the target historical frame picture is calculated by the following formula:

Real position relationship = picture position information between the current frame picture and the target historical frame picture × S;

Wherein, S = the odometer information between the specified number of the historical frame pictures / picture position information between the specified number of the historical frame pictures.

In this embodiment, the calculation module performs feature matching on the current frame picture and the historical frame picture of the sweeper. When two frames of pictures are detected to form a loop, the current number of adjacent frames of the current frame picture captured by the monocular camera is called. The picture position information between the frame pictures and the odometer information of the position where the sweeper passes these pictures. Because the data calculated by the monocular camera directly from the picture is distorted, the odometer information records the actual motion path of the sweeper, which represents the true distance information between the pictures. There is a specific ratio between the calculated picture position information and the real distance information of the pictures, and the picture position information between these frames of pictures and the odometer information can be compared to obtain the proportionality coefficient between them. Then according to the picture position relationship between the current frame picture and the historical frame picture that forms a visual loop, the real position information of the sweeper can be obtained.

Updating the gyroscope data and odometer data corresponding to the current frame picture according to the real position information, and completing the positioning step includes:

S501: Obtain gyroscope information of the target historical frame to update the gyroscope data of the current frame picture;

S502: Replace the odometer data corresponding to the current frame picture with the real position;

S503: Integrate gyroscope data and odometer data corresponding to the current frame picture, and complete positioning again.

In this embodiment, since the information of the gyroscope and the odometer will gradually appear during the movement of the sweeper, the gyroscope information recorded in the historical frame picture that first forms a visual loop with the current frame picture is essentially the beginning of the sweeper The pose information does not cause errors, and can be directly used to update the gyroscope data of the current frame picture. The calculated actual position information error between the current frame and the historical frame picture that forms a visual loop with the current frame picture has the smallest error, and the odometer data of the current frame picture can be directly replaced to complete the update. The combination of gyroscope data and odometer data of the current frame picture can accurately complete the positioning.

The steps of collecting a picture of the current environment through a visual sensor according to a preset frequency and selecting a key frame picture in the picture of the current environment include:

S101: Determine whether the first picture in the picture of the current environment collected by the visual sensor has a feature that reaches a preset number;

S102: if yes, determine that the first picture is the key frame picture;

S103: Save the key frame picture.

In this embodiment, because the image occupies a relatively large space, in order to reduce data redundancy, the sweeping machine automatically selects and collects images with rich features as key frames, such as pictures with reference objects and differences reaching a set number.

Steps to find a historical frame picture similar to the current frame picture, including:

S301: One-to-one correspondence of each feature in the collected picture into words, and the words are summarized into a word tree-type visual dictionary.

In this embodiment, the key to finding similar frames is how to determine the similarity between the two frames of pictures. The most intuitive way is to use feature matching, and compare whether the number of feature point matches is sufficient. Because feature matching is very time-consuming, loop detection needs to match all past key frames. This amount of calculation is absolutely unbearable for the calculation module configured by the sweeper. Therefore, a bag-of-words model is needed to accelerate the speed of feature matching. Dictionary training is actually a clustering process. Assuming a total of 10,000,000 features are extracted from all the pictures, they can be aggregated into 100,000 words using the K-means method. However, it is still too inefficient to match only these 100,000 words, because each feature needs to be compared 100,000 times to find its own corresponding word. In order to improve the efficiency, the dictionary constructs a tree with k branches and a depth of d during the training process. Intuitively, the upper nodes provide coarse classification and the lower nodes provide fine classification up to the leaf nodes. Using this tree, the time complexity can be reduced to the logarithmic level, which greatly speeds up feature matching.

S302: Calculate a DBOW mapping of the features of the current frame picture in the visual dictionary;

S303: Find the historical frame picture most similar to the current frame picture by comparing the DBOW mapping to form a visual loop.

In this embodiment, DBOW provides two methods for calculating similarity. The first is to directly compare two pictures; the second is to construct a picture set into a database and then compare it with another picture. The more similar the pictures are, the closer the score is to 1. We can use this score to determine whether the two pictures are the same scene and form a loop.

Referring to FIG. 2, the present application further provides a positioning device for a sweeper based on a visual loop calibration gyroscope, including:

An acquisition module 1 is configured to collect a picture of the current environment and select a key frame picture in the picture of the current environment.

Further, the device further includes:

A judging module, which is used for judging whether the current movement is started;

A generating module for generating an instruction for collecting a picture of the current environment.

In this embodiment, the vision sensor starts to collect pictures of the surrounding environment when the sweeper starts to move, and selects pictures with rich reference objects as features to save as key frame pictures. Because the sweeper did not move at the beginning, there was no error in the gyroscope information and odometer information of the key frame picture at this time.

The obtaining module 2 is configured to obtain and save gyroscope information and odometer information corresponding to two key frame pictures at adjacent times.

The detection module 3 is configured to find a historical frame picture similar to the current frame picture.

In this embodiment, a bag-of-words model is established to accelerate feature matching, and each feature in a key frame picture is trained one-to-one into a visual dictionary, and the DBOW mapping of feature points in the current frame picture in the visual dictionary is calculated. The key frame picture with the most similar frame picture is judged to form a visual loop.

A calculation module 4 is configured to obtain real position information between the current frame picture and the target historical frame picture through the gyroscope information and the odometer information.

An update module 5 is configured to complete positioning according to the gyroscope data and odometer data corresponding to the current frame picture and the real position information.

Referring to FIG. 3, the calculation module 4 includes:

A first obtaining unit, configured to obtain picture position information and odometer information between a specified number of the historical frame pictures adjacent to the current frame picture;

A second obtaining unit, configured to obtain picture position information between the current frame picture and the target historical frame picture;

A calculation unit is configured to calculate and obtain real position information between the current frame picture and the target historical frame picture.

In this embodiment, the calculation module performs feature matching on the current frame picture and historical frame picture of the sweeper. When detecting that two frames form a loop, the first obtaining unit 401 calls the current frame picture captured by the monocular camera to be adjacent. A specified number of historical frame pictures between the picture position information and the odometer information of the position where the sweeper passed these pictures. Because the data calculated by the monocular camera directly from the picture is distorted, the odometer information records the actual motion path of the sweeper, which represents the true distance information between the pictures. There is a specific ratio between the calculated picture position information and the real distance information of the pictures, and the picture position information between these frames of pictures and the odometer information can be compared to obtain the proportionality coefficient between them. The second obtaining unit 402 can obtain the real position information of the sweeper according to the picture position relationship between the current frame picture and the historical frame picture that forms a visual loop.

Referring to FIG. 4, the update module 5 includes:

A calling unit 501, configured to obtain gyroscope information of the target historical frame, and thereby update the gyroscope data of the current frame picture;

A replacement unit, configured to replace the odometer data corresponding to the current frame picture with the real position information;

The positioning unit 502 is configured to integrate gyroscope data and odometer data corresponding to the current frame picture, and complete positioning again.

Referring to FIG. 5, the acquisition module 1 includes:

A judging unit 101, configured to judge whether a first picture in a picture of a current environment collected by a visual sensor has a feature that reaches a preset number;

A determining unit 102, configured to determine that the first picture in the picture of the current environment has a preset number of features, and determine that the first picture is the key frame picture;

The saving unit 103 is configured to save the key frame picture.

Further, the acquisition module 1 further includes:

A deleting unit is configured to determine that the first picture is not the key frame picture and delete the first picture to reduce data redundancy.

Referring to FIG. 6, the detection module 3 includes:

A summary unit 301 is configured to map each feature in the collected picture into a word one by one, and to summarize each word into a word tree-type visual dictionary.

A mapping unit 302, configured to calculate a DBOW mapping of the features of the current frame picture in the visual dictionary;

A comparison unit 303 is configured to find a historical frame picture most similar to the current frame picture by comparing the DBOW mapping to form a visual loop.

During the cleaning process, the sweeper collects information about the external environment through a monocular camera. The information collection includes taking pictures with rich feature points as key frame pictures and storing them in memory, and then using a three-axis gyroscope to collect the sweeper ’s movement angle and position, Record the odometer information of the odometer sensor to locate the sweeper, so the sweeper can move autonomously in an environment with obstacles. During the cleaning process of the sweeping machine, the calculation module will call the historical frame pictures in the memory and the current frame pictures taken by the monocular camera, and quickly perform special diagnosis matching through the bag of words model. When a loop is detected between the current frame picture and the historical frame picture, the calculation module calculates the pose relationship between the two frames based on the algorithm, that is, the real position relationship of the current frame is calculated, and the sweeper is automatically updated according to the real position relationship. Gyro and odometer data to complete positioning. During the operation of the sweeper, the customer can also actively send an update instruction to the sweeper through the remote control. After the sweeper receives the command information through the signal receiver, it will update the gyroscope data and odometer data first to realize repositioning.

The present application provides a method and system for positioning a sweeper based on a visual loop calibration gyroscope, which can quickly calculate the real position information of the current frame based on the visual loop. Under the premise of ensuring the positioning accuracy of the sweeper, a simpler method is used. Calibrate the gyro of the sweeper, reduce the computational complexity, correct the errors generated by the sweeper during its movement, and achieve accurate positioning.

A computer device is also provided in the embodiment of the present application, and its internal structure may be as shown in FIG. 7. The step counting device includes a memory, a processor, and at least one application program stored in the memory and configured to be executed by the processor, the application program being configured to execute the sweeper in any one of the above embodiments. Positioning method.

Those skilled in the art can understand that the computer device according to the present invention and the above-mentioned device for performing one or more of the methods described in the present application. These devices may be specially designed and manufactured for the required purpose, or they may include known devices in general-purpose computers. These devices have computer programs or applications stored therein that are selectively activated or reconstructed. Such a computer program may be stored in a device (eg, a computer) readable medium or in any type of medium suitable for storing electronic instructions and coupled to a bus, respectively, including, but not limited to, any Types of disks (including floppy disks, hard disks, CD-ROMs, CD-ROMs, and magneto-optical disks), ROM (Read-Only Memory), RAM (Random Access Memory (random memory), EPROM (Erasable Programmable Read-Only Memory (EEPROM), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory, magnetic card or optical card. That is, a readable medium includes any medium that stores or transfers information in a readable form by a device (eg, a computer).

Although the embodiments of the present application have been shown and described, it will be understood by those skilled in the art that various changes, modifications, and replacements of these embodiments can be made without departing from the principle and spirit of the present application. And variations, the scope of the application is defined by the appended claims and their equivalents.

Claims

A positioning method of a sweeper based on a visual loop calibration gyroscope, characterized in that it includes:

Collect a picture of the current environment and choose to save the key frame picture in the picture of the current environment;

Acquiring and saving the corresponding gyroscope information and odometer information between two key frame pictures at adjacent times;

Find historical frame pictures similar to the current frame picture to form a visual loop;

Obtain the real position information between the current frame picture and the target historical frame picture through the gyroscope information and the odometer information, and the target historical frame picture is formed from the current frame picture in the historical frame picture Visual loopback pictures;

Update the gyroscope data and the odometer data corresponding to the current frame picture according to the real position information to complete positioning.
The positioning method of a sweeper based on a visual loop calibration gyroscope according to claim 1, wherein the obtaining between the current frame picture and the target historical frame picture is performed by using the gyroscope information and the odometer information. Steps for real location information, including:

Acquiring picture position information and odometer information between a specified number of the historical frame pictures adjacent to the current frame picture;

Acquiring picture position information between the current frame picture and the target historical frame picture;

The true position information between the current frame picture and the target historical frame picture is calculated by the following formula:

Real position relationship = picture position information between the current frame picture and the target historical frame picture × S;

Wherein, S = the odometer information between the specified number of the historical frame pictures / picture position information between the specified number of the historical frame pictures.
The positioning method for a sweeper based on a visual loop calibration gyroscope according to claim 1, wherein the updating of the gyroscope data and odometer data corresponding to the current frame picture according to the real position information completes the positioning Steps, including:

Acquiring gyroscope information of the target historical frame, thereby updating the gyroscope data of the current frame picture;

Replacing the odometer data corresponding to the current frame picture with the real position information;

Integrate gyroscope data and odometer data corresponding to the current frame picture, and complete positioning again.
The method for positioning a sweeper based on a visual loop calibration gyroscope according to claim 1, wherein the step of collecting a picture of the current environment and selecting a key frame picture in the picture of the current environment comprises:

Judging whether the first picture in the picture of the current environment collected by the visual sensor has a feature that reaches a preset number;

If so, determine that the first picture is the key frame picture;

Save the key frame picture.
The positioning method of a sweeper based on a visual loop calibration gyroscope according to claim 4, wherein the judgment is performed on whether the first picture among the pictures of the current environment collected by the vision sensor has a preset number of features. After the steps, include:

If there are no features that reach a preset number, it is determined that the first picture is not the key frame picture, and the first picture is deleted.
The positioning method of a sweeper based on a visual loop calibration gyroscope according to claim 1, wherein the step of finding a historical frame picture similar to the current frame picture comprises:

Map each feature in the collected picture into words one by one, and summarize each word into a word tree visual dictionary;

Calculating a DBOW mapping of the features of the current frame picture in the visual dictionary;

By comparing the DBOW mapping, a historical frame picture most similar to the current frame picture is found to form a visual loop.
The method for positioning a sweeper based on a visual loop calibration gyroscope according to claim 6, wherein the step of one-to-one mapping each feature in the collected picture into each word comprises:

A K-means algorithm is used to correspond each feature in the picture to each of the words, and the words are summarized into a word tree-type visual dictionary.
The method for positioning a sweeper based on a visual loop calibration gyroscope according to claim 1, wherein before the step of collecting a picture of the current environment and selecting to save a key frame picture in the picture of the current environment, comprising:

Determine whether the sweeper is currently moving;

If motion is started, an instruction to collect a picture of the current environment is generated.
A positioning device for a sweeper based on a visual loop calibration gyroscope, which comprises:

An acquisition module, configured to acquire a picture of the current environment according to a preset frequency through a visual sensor, and select to save a key frame picture in the picture of the current environment;

An acquisition module for collecting a picture of the current environment and selecting a key frame picture in the picture of the current environment;

A detection module, configured to acquire and save gyroscope information and odometer information corresponding to two key frame pictures at adjacent times;

A calculation module, configured to obtain real position information between the current frame picture and the target historical frame picture through the gyroscope information and the odometer information;

An update module is configured to complete positioning according to the gyroscope data and odometer data corresponding to the current frame picture and the real position information.
The positioning device for a sweeper based on a visual loop calibration gyroscope according to claim 9, wherein the calculation module comprises:

A first obtaining unit, configured to obtain picture position information and odometer information between a specified number of the historical frame pictures adjacent to the current frame picture;

A second obtaining unit, configured to obtain picture position information between the current frame picture and the target historical frame picture;

A calculation unit is configured to calculate and obtain real position information between the current frame picture and the target historical frame picture.
The positioning device for a sweeper based on a visual loop calibration gyroscope according to claim 9, wherein the update module comprises:

A calling unit, configured to obtain the gyroscope information of the target historical frame, thereby updating the gyroscope data of the current frame picture;

A replacement unit, configured to replace the odometer data corresponding to the current frame picture with the real position information;

A positioning unit is configured to synthesize gyroscope data and odometer data corresponding to the current frame picture, and complete positioning again.
The positioning device for a sweeper based on a visual loop calibration gyroscope according to claim 9, wherein the acquisition module comprises:

A judging unit, configured to judge whether the first picture among the pictures of the current environment collected by the visual sensor has a feature that reaches a preset number;

A determining unit, configured to determine that the first picture in the current environment has a preset number of features, and determine that the first picture is the key frame picture;

A saving unit, configured to save the key frame picture.
The positioning device for a sweeper based on a visual loop calibration gyroscope according to claim 12, wherein the acquisition module further comprises:

A deleting unit is configured to determine that the first picture is not the key frame picture and delete the first picture to reduce data redundancy.
The positioning device for a sweeper based on a visual loop calibration gyroscope according to claim 9, wherein the detection module comprises:

A summary unit, configured to map each feature in the collected picture into words one by one, and summarize each word into a word tree visual dictionary;

A mapping unit, configured to calculate a DBOW mapping of features of the current frame picture in the visual dictionary;

A comparison unit is configured to find a historical frame picture most similar to the current frame picture by comparing the DBOW mapping to form a visual loop.
The positioning device for a sweeper based on a visual loop calibration gyroscope according to claim 14, wherein the summary unit is specifically configured to use a K-means algorithm to map each feature in the picture to each of the one-to-one The words are summarized into a word tree visual dictionary.
A computer device, characterized in that it comprises a processor, a memory, and a computer program stored on the memory and executable on the processor, and the processor implements the computer program as claimed in claim 1 when executing the computer program. The positioning method of a sweeper based on a visual loop calibration gyroscope according to any one of -8.