CN108544494B - Positioning device, method and robot based on inertia and visual characteristics - Google Patents
Positioning device, method and robot based on inertia and visual characteristics Download PDFInfo
- Publication number
- CN108544494B CN108544494B CN201810543857.6A CN201810543857A CN108544494B CN 108544494 B CN108544494 B CN 108544494B CN 201810543857 A CN201810543857 A CN 201810543857A CN 108544494 B CN108544494 B CN 108544494B
- Authority
- CN
- China
- Prior art keywords
- image
- landmark
- camera
- positioning device
- feature
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 33
- 230000000007 visual effect Effects 0.000 title claims abstract description 33
- 238000012545 processing Methods 0.000 claims abstract description 40
- 230000004927 fusion Effects 0.000 claims abstract description 20
- 238000007781 pre-processing Methods 0.000 claims abstract description 14
- 230000008569 process Effects 0.000 claims abstract description 14
- 238000013519 translation Methods 0.000 claims abstract description 7
- 230000001133 acceleration Effects 0.000 claims abstract description 6
- 238000003384 imaging method Methods 0.000 claims description 19
- 238000001914 filtration Methods 0.000 claims description 6
- 230000003287 optical effect Effects 0.000 claims description 6
- 238000000605 extraction Methods 0.000 claims description 4
- 238000005259 measurement Methods 0.000 claims description 4
- 230000008859 change Effects 0.000 claims description 3
- 238000013507 mapping Methods 0.000 claims description 2
- 230000036544 posture Effects 0.000 claims description 2
- 230000007613 environmental effect Effects 0.000 abstract 1
- 238000010586 diagram Methods 0.000 description 7
- 238000005516 engineering process Methods 0.000 description 7
- 238000010408 sweeping Methods 0.000 description 5
- 230000009471 action Effects 0.000 description 4
- 230000009466 transformation Effects 0.000 description 4
- 238000001514 detection method Methods 0.000 description 3
- 238000006243 chemical reaction Methods 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 206010063385 Intellectualisation Diseases 0.000 description 1
- 241001417527 Pempheridae Species 0.000 description 1
- 238000009825 accumulation Methods 0.000 description 1
- 238000004140 cleaning Methods 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000008030 elimination Effects 0.000 description 1
- 238000003379 elimination reaction Methods 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 230000000873 masking effect Effects 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J9/00—Programme-controlled manipulators
- B25J9/16—Programme controls
- B25J9/1694—Programme controls characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion
- B25J9/1697—Vision controlled systems
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J19/00—Accessories fitted to manipulators, e.g. for monitoring, for viewing; Safety devices combined with or specially adapted for use in connection with manipulators
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J19/00—Accessories fitted to manipulators, e.g. for monitoring, for viewing; Safety devices combined with or specially adapted for use in connection with manipulators
- B25J19/02—Sensing devices
- B25J19/04—Viewing devices
Abstract
The application discloses a positioning device, a positioning method and a robot based on inertial data and visual characteristics, wherein the positioning device is a movable visual positioning device and comprises: the camera of the image acquisition module is kept backwards and placed in a preset position of the positioning device, and is used for avoiding collision by an obstacle in the advancing process; the image processing module comprises an image preprocessing sub-module and a characteristic matching sub-module and is used for processing the acquired image data; the inertial data acquisition processing module is used for sensing the rotation angle information, the acceleration information and the translation speed information of the inertial sensor in real time; and the fusion positioning module is used for fusing the environmental information acquired by each sensor module to realize positioning. Compared with the prior art, the method has the advantages that the camera is effectively prevented from being blocked by the obstacle when the machine advances, the inertial data are fused by using the image data matched by the features, the landmark is updated by combining the relative position relationship, the feature matching result is more accurate, and the robustness of the positioning algorithm is enhanced.
Description
Technical Field
The application relates to a positioning method and a positioning device, in particular to a positioning device, a positioning method and a robot based on inertial data and visual characteristics.
Background
The robot realizes intellectualization, one basic technology is that the robot can position and walk by itself, and an indoor navigation technology is a key technology. The current indoor navigation technology comprises inertial sensor navigation, laser navigation, visual navigation, radio navigation and the like, and each technology has own advantages and disadvantages. The inertial sensor navigation is to use a gyroscope, an odometer and the like for navigation and positioning, so that the cost is low, but the problem of long-time drift exists; the laser navigation precision is high, but the price is higher, and the service life is also a problem; visual navigation in the traditional sense is complex in calculation, and has higher requirements on the performance of a processor, and higher power consumption and price; the radio requires a plurality of fixed radio emission sources, is inconvenient to apply and is relatively expensive. The integration of multiple technologies, low cost and high precision, is one development direction of robotic navigation technology.
In the existing visual sweeper products, the camera is arranged in front of the machine, and generally needs to be slightly protruded to obtain a good visual angle, however, the camera lens is easily touched by some objects which are difficult to detect, and the lens is easily scratched; and the front of the machine is generally provided with more sensors, for example, many machines are provided with collision bars and cylindrical 360-degree infrared receiving devices, which are easy to block the camera, so that the angle of the camera needs to be increased.
Disclosure of Invention
The positioning device based on the inertial data and the visual characteristics is a movable visual positioning device and comprises an image acquisition module, an image processing module, an inertial data acquisition module and a fusion positioning module;
the camera of the image acquisition module is kept backwards and placed in a preset position of the positioning device and used for avoiding collision by obstacles in the advancing process of the positioning device, wherein the number of the cameras is at least one;
the image processing module comprises an image preprocessing sub-module and a characteristic matching sub-module and is used for processing the image data acquired by the image acquisition module; the image preprocessing sub-module is used for converting the data acquired by the camera into a gray image, and the feature matching sub-module is used for extracting feature data from the image preprocessed by the image preprocessing sub-module and matching the feature data with associated features of landmark images in a landmark database; the landmark database is a landmark database built in the image processing module, and comprises image feature points of a region associated with a given landmark;
the inertial data acquisition processing module consists of a series of inertial data measurement units and senses the rotation angle information, the acceleration information and the translation speed information of the inertial sensor in real time;
and the fusion positioning module is used for carrying out data fusion on the inertial data acquired by the inertial data acquisition processing module according to the feature matching result in the image processing module, and correcting the current position information through the data fusion result.
Further, the preset position is a concave and/or convex structure with a backward opening at the middle of the top surface of the positioning device.
Further, the preset position is a concave and/or convex structure with a backward tail opening on the top surface of the positioning device.
Further, the angle formed by the optical axis of the camera and the top surface of the positioning device spans 0-80 degrees.
Further, in the fusion positioning module, when the feature matching in the image processing module is successful, the current position coordinate of the positioning device is calculated according to the geometrical relationship between the gray image feature and the imaging feature of the landmark in the actual scene, and the inertial data is used for updating and correcting;
when feature matching in the image processing module fails, a rigid connection relation between the inertial sensor and the camera is obtained according to the accumulated value of the inertial data, and meanwhile, a new landmark is calculated by combining the gray image feature and the relative gesture of the landmark image associated feature in the landmark database, and is stored and recorded in the landmark database to complete creation of the new landmark;
the image sensor comprises an inertial sensor, a camera, a gray level image sensor, a landmark image sensor, a gray level image sensor and a landmark image sensor, wherein the inertial sensor is connected with the camera, mapping association exists between the camera and the gray level image feature or landmark image associated feature, and the feature can be obtained through extraction of the gray level image; the rigid connection relation is a position relation established based on pose changes corresponding to the inertial data between two adjacent frames of images acquired by the camera.
A positioning method based on inertial data and visual features, the positioning method being applied to the positioning device, comprising the steps of:
preprocessing an image collected by a camera aiming at a landmark in an actual scene to obtain a gray image, extracting characteristic points from the gray image, generating gray image characteristics, and determining descriptors of the characteristic points;
performing feature matching on the generated descriptors of the gray image features and descriptors of associated features of landmark images stored in a landmark database;
judging whether the gray image features are matched with the associated features of the landmark images in the landmark database, if so, fusing inertial data according to the geometric relationship between the gray image features and the imaging features of the landmarks in the actual scene to obtain the current position coordinates; otherwise, according to the rigid connection relation between the inertial sensor and the camera, combining the relative postures of the gray image features and the landmark image associated features in a landmark database, fusing the inertial data, calculating a new landmark, storing and recording the new landmark in the landmark database, and completing the creation of a new landmark;
the inertial data is subjected to calibration filtering processing, and the rigid connection relation is a position relation established based on pose changes corresponding to the inertial data between two adjacent frames of images acquired by the camera.
Further, the feature matching process includes: under the current frame image, calculating the Hamming distance between the descriptors of the gray image features and the corresponding descriptors in the associated features of the landmark images of the landmark database;
if the Hamming distance is smaller than a preset threshold value, the similarity of the collected image and the associated feature of the landmark image corresponding to the landmark database is high, and the matching is considered to be successful;
wherein the preset threshold corresponds to a determined numerical relationship of the relative pose.
Further, the fused inertial data includes: when the feature matching fails, according to the pose change corresponding to the inertial data between two adjacent frames of images acquired by the camera, the rigid connection relation between the inertial sensor and the camera is obtained, under the condition that the internal parameters of the camera are known, the feature point coordinates of the current frame of image predicted by the inertial sensor are obtained through calculation according to the rigid connection relation between the inertial sensor and the camera, and then the feature point coordinates of the current frame of image predicted by the inertial sensor are compared with the feature point coordinates of the current frame of image acquired by the camera by combining the relative pose of the gray image features and the landmark images associated features in a landmark database.
Further, the imaging feature geometric relationship is a similar triangle relationship established based on the position relationship between feature points in the current frame image acquired on the camera at the preset position and road signs in the acquired actual scene.
A robot, which is a mobile robot provided with the positioning device.
Compared with the prior art, the one or more cameras are arranged at the backward protruding structure of the surface of the positioning device, so that the machine is prevented from being blocked by undetected obstacles when advancing, and the mounting position of the machine is more selected and more flexible. According to the positioning of the camera on the positioning device, the application uses the image features of the landmark database to match the shot landmark images, fuses inertial data, creates new landmarks for the landmark database while positioning, improves the positioning precision, shortens the feature searching range of the landmark database in the positioning and matching process, shortens the feature searching time, and improves the navigation efficiency.
Drawings
FIG. 1 is a block diagram of a positioning device based on inertial data and visual features according to an embodiment of the present application;
FIG. 2 is a flow chart of a positioning method based on inertial data and visual features provided by the embodiment of the application;
FIG. 3 is a diagram of a robotic system based on inertial data and visual features (a camera is positioned at a protruding structure on the surface of the positioning device) according to an embodiment of the present application;
fig. 4 is a diagram of another robot system structure based on inertial data and visual features (a camera is positioned at a concave structure of the surface of the positioning device) according to the embodiment of the present application.
Detailed Description
The following is a further description of embodiments of the application, taken in conjunction with the accompanying drawings:
the positioning device based on the inertial data and the visual characteristics is implemented in a robot mode and comprises a sweeping robot, an AGV and the like. The following assumes that the obstacle avoidance device is mounted on a robot for sweeping floor. However, it will be appreciated by those skilled in the art that the configuration according to the embodiment of the present application can be extended to be applied to a mobile terminal, except for being particularly used for a mobile robot.
The application provides a positioning device based on inertial data and visual characteristics, which is a movable visual positioning device, as shown in figure 1, and comprises an image acquisition module, an image processing module, an inertial data acquisition module and a fusion positioning module; the camera of the image acquisition module is placed in a preset position of the positioning device and used for avoiding collision by obstacles in the advancing process of the positioning device. In the advancing process of the positioning device, if the camera is placed in front of the positioning device, a slight protrusion is generally required, and a better visual angle can be obtained by keeping a preset angle, however, the camera lens is easily touched by some objects which are difficult to detect, and the lens is easily scratched; in the embodiment of the application, the collision strip and the cylindrical 360-degree infrared receiving device are arranged, and the camera is easy to block, so that the camera can keep a preset angle to obtain a better visual angle.
The number of the cameras can be multiple, and the cameras are arranged at different positions of the middle part and/or the tail part of the top surface of the positioning device; during forward driving of the positioning device, the obstacle reduces the quality of the image captured by the camera for use in navigation positioning and thereby reduces the accuracy of navigation. In order to maintain a sufficient level of navigation performance, the positioning device is provided with a plurality of cameras to acquire effective image information, the sensing field range of the positioning device is expanded from a plurality of view angles, and the system performance of the image acquisition module is improved.
Preferably, the preset position is at a protruding structure which is at the tail of the visual positioning device and is opened backwards. As shown in fig. 3, in the implementation of the present application, the orientation of the camera 106 keeps backward relative to the positioning device, and is placed at the backward protruding structure of the tail of the surface cover of the positioning device, so that the camera is prevented from being collided by the rear of the camera, and the lens of the camera has more accurate imaging characteristics. In particular, the angle formed by the optical axis of the camera 106 and the top surface of the positioning device is preferably 45 degrees in this embodiment, increasing the effective field of view of the positioning device, preventing unwanted imaging problems, such as light reflection and/or refraction that may prevent effective imaging features of the camera.
Preferably, the preset position is a concave structure with a backward opening at the middle part of the visual positioning device. As shown in fig. 4, in the implementation of the present application, the orientation of the camera 108 is kept backward relative to the positioning device, and is placed in a concave structure with a backward opening near the geometric center of the surface of the cover of the positioning device, so that the camera is prevented from being collided by the rear position of the camera, and the lens of the camera has more accurate imaging characteristics. In particular, the range of angles formed by the optical axis of the camera 108 and the top surface of the positioning device is preferably 45 degrees in this embodiment, increasing the effective field of view of the positioning device, preventing unwanted imaging problems such as light reflection and/or refraction that may prevent the camera from effectively imaging features.
It should be noted that, in the embodiment where the preset position is the tail of the visual positioning device and the opening is at the concave structure of the backward direction, reference may be made to the structure of fig. 4, and the concave structure of fig. 4 may be moved to the tail of the positioning device; the embodiment in which the preset position is the middle part of the visual positioning device and the protruding structure with the backward opening is referred to the structure in fig. 3, and the protruding structure in fig. 3 is moved to the middle part of the positioning device. The camera is arranged at the preset position, and the landmark with higher precision is obtained than the landmark obtained under the condition that a 360-degree omnibearing camera is used on the surface of the positioning device.
As shown in fig. 1, the image processing module includes an image preprocessing sub-module and a feature matching sub-module, and is configured to process image data acquired in the image acquisition module. The image preprocessing sub-module receives the image data acquired by the image acquisition module to establish a unique repeated identifiable landmark in the surrounding environment, binarizes the color image data acquired by the camera, converts the color image data into a gray image and completes the preprocessing process of the image; and then the feature matching sub-module extracts feature data from the images preprocessed by the image preprocessing sub-module and matches the associated features of the landmark images in the landmark database.
The landmark database is a landmark database built in the image processing module, and comprises image feature points of a given landmark associated area. The landmark database contains information about a number of previously observed landmarks with which the positioning device can perform navigational positioning actions. Landmarks may be considered as a collection of features having a specific two-dimensional structure. Any of a variety of features may be used to identify a landmark, which may be, but is not limited to, a set of features identified based on a two-dimensional structure of corners of a photo frame when the positioning device is configured as a house cleaning robot. Such features are based on static geometry within the room and, although features have some illumination and dimensional variation, they are generally more easily discernable and identifiable as landmarks relative to objects in lower regions of the environment that are frequently displaced (e.g., chairs, garbage cans, pets, etc.).
As shown in fig. 1, the inertial data acquisition and processing module consists of a series of inertial data measurement units, and senses the rotation angle information, acceleration information and translation speed information of the inertial sensor in real time; the module is used for acquiring inertial data through an inertial sensor, and then performing calibration filtering processing and transmitting the inertial data to the fusion positioning module. The original data processing of the inertial data comprises a masking of a maximum value and a minimum value; static drift elimination; kalman filtering of data. Wherein the inertial sensor comprises an odometer, a gyroscope, an accelerometer and the like for inertial navigation.
As shown in fig. 1, the fusion positioning module is configured to perform data fusion on the inertial data acquired by the inertial data acquisition processing module according to the feature matching result in the image processing module, and correct the current position information according to the data fusion result. The module is used for matching the new image with the corresponding landmark image stored in the landmark database based on the image acquired by the camera and the travelling distance acquired by the inertial sensor, and then carrying out data fusion to realize positioning.
As shown in fig. 1, in the fused positioning module, when feature matching in the image processing module is successful, according to the geometrical relationship (similar triangle relationship) between the gray image feature and the imaging feature of the landmark in the actual scene, the current position coordinate of the positioning device is calculated, and the inertial data is used for updating and correcting; when feature matching in the image processing module fails, a rigid connection relation between the inertial sensor and the camera is obtained according to the accumulated value of the inertial data, and meanwhile, a new landmark is calculated by combining the gray image feature and the relative gesture of the landmark image associated feature in the landmark database, and is stored and recorded in the landmark database to complete creation of the new landmark; the rigid connection relation is a position relation established based on pose changes corresponding to the inertial data between two adjacent frames of images acquired by the camera; the inertial sensor is mapped to the camera, the camera is mapped to the gray level image features or the landmark image associated features, the features can be obtained through extraction of the gray level images, and iterative operation is carried out by using the inertial sensor data between two continuous frames of images according to a rigid connection relationship to obtain a prediction of the current position of the positioning device, so that the search area is smaller when the features are matched, and the matching speed is higher.
Further, in the implementation of the application, the camera can be placed at a backward protruding or recessed structure of the tail part of the surface of the positioning device face cover, and the problem that the camera is collided is avoided by the rear placement of the camera. In a specific embodiment, an infrared collision sensor in the inertial data measurement unit is installed in front of the positioning device, and senses an obstacle to obtain a first position coordinate, but the camera cannot shoot, the obtained shooting object is located behind the positioning device, and a second position coordinate obtained by matching and calculating image features is different from the first position coordinate, so that a rigid connection relationship based on the camera and the infrared collision sensor exists; and when the camera shoots the obstacle, the image features are matched and calculated to obtain the first position coordinates, the infrared collision sensor senses other objects, the obtained position information is not the first position coordinates, and the image processing module performs position relation conversion on the image data of the image acquisition module to realize positioning according to the description of translation and/or rotation of the positioning device relative to the landmark gesture associated with the landmark database.
Further, the positioning device may also include different types of sensor systems, enabling reliable and robust multi-sensor data fusion to enhance robustness of autonomous movement. Additional sensor systems may be used in conjunction with each other to create a perception of the environment of the positioning device that is sufficient to allow the positioning device to make intelligent decisions regarding actions to be taken in the environment. ) A contact sensor in the buffer sensor system 550, a laser scanner, and/or imaging sonar. Wherein the sensors are typically placed such that they have a maximum achieved coverage of the sensing area around the positioning device. Second, the sensor is typically placed in such a way that the positioning device itself causes an absolute minimum obstruction to the sensor. Third, the placement and installation of the sensors should not encroach upon the rest of the industrial design of the platform. The sensor is more aesthetically pleasing to be mounted to a location within the positioning device that is not apparent. In terms of practicality, the sensors should be mounted in a way so as not to interfere with normal robot operation.
Based on the same inventive concept, the embodiment of the application also provides a positioning method based on the inertial data and the visual characteristics, and because the hardware device for solving the positioning problem of the positioning method is based on the positioning device based on the inertial data and the visual characteristics, the implementation of the positioning method can refer to the application implementation of the positioning device based on the inertial data and the visual characteristics, and the repetition is omitted.
In specific implementation, the positioning method based on inertial data and visual characteristics provided by the embodiment of the application, as shown in fig. 2, specifically includes:
preprocessing an image collected by a camera aiming at a landmark in an actual scene to obtain a gray image, extracting characteristic points from the gray image, generating gray image characteristics, and determining descriptors of the characteristic points;
specifically, gaussian filtering processing is performed on an environment two-dimensional color image shot by a camera, noise is removed, and graying is performed; extracting feature points from the gray level image to generate image features, comparing the extracted feature points with the pixel gray levels of 256 positions in the field of the feature points, recording a result in binary mode, wherein 0 represents that the pixel gray level of the feature points is smaller than one of 256 pixel gray levels in the field of the feature points, 1 represents that the pixel gray level of the feature points is larger than one of 256 pixel gray levels in the field of the feature points, and storing the result in a 256-dimensional vector as a descriptor of the feature points. The field of the feature point is a circular plane with the feature point as a center and r as a radius, and the value of r is determined according to the gray level of the actually acquired image.
And step two, matching the descriptors of the gray image features generated in the step one, wherein the matching object is the associated features of the landmark images stored in the landmark database, and the descriptors of the corresponding feature points are obtained by processing the landmark images, so that the generated descriptors of the gray image features are matched with the descriptors of the landmark image features stored in the landmark database.
Judging whether the gray image features are matched with landmark image associated features in a landmark database, if so, calculating the current position coordinate of the positioning device under a geodetic coordinate system according to the geometric relationship between the gray image features and imaging features of landmarks in an actual scene under the condition of known camera internal parameters, and then obtaining a predicted position coordinate of the positioning device through the inertial sensor by combining the rigid connection relationship between the camera and the inertial sensor, and correcting and updating the current position coordinate to obtain an accurate current position coordinate; otherwise, according to the rigid connection relation between the inertial sensor and the camera, combining the relative gesture of the gray image feature and the landmark image associated feature in the landmark database, merging the landmark image coordinates corresponding to the feature points of the gray image feature, merging the image feature point coordinates predicted by the inertial data in the landmark image of the current frame, calculating a new landmark, storing and recording in the landmark database, and completing the creation of the new landmark. Wherein the inertial data has been subjected to a calibration filtering process; the inertial data includes angular velocity, acceleration and distance information; the rigid connection relation is a position relation established based on pose changes corresponding to the inertial data between two adjacent frames of images acquired by the camera.
As one mode of implementation of the present application, in the second step, the feature matching process includes: under the current frame image, calculating the Hamming distance between the descriptors of the gray image features and the corresponding descriptors in the associated features of the landmark images of the landmark database; if the Hamming distance is smaller than a preset threshold value, the method indicates that the similarity of the collected image and the associated feature of the landmark image corresponding to the landmark database is high, and the matching is considered to be successful. Specifically, in the implementation of the application, feature matching is performed by calculating the hamming distance of the descriptors of the feature points, and a large number of experiments show that the hamming distance of the descriptors of the feature points which are failed to match is about 128, and the hamming distance of the descriptors of the feature points which are successful to match is far less than 128; that is, the number of the same elements on the bit corresponding to the feature codes of the descriptors in the image templates of the database is less than 128, and the descriptors are not paired; the feature points on one graph are paired with the feature points with the largest number of identical elements on the bit corresponding to the feature codes on the other graph. Wherein the preset threshold corresponds to a determined numerical relationship of the relative gesture, which is set to 128 in the embodiment of the present application. The relative pose is dependent on one or more of the two-dimensional spatial features associated with the identifiable landmarks within the image captured by the camera, and the estimation of the relative pose is also varied by virtue of various sensor configuration variations of the positioning device. In the navigation process, the positioning device can add the newly determined relative gesture information into landmark image characteristic information of the landmark database, enriches newly created landmark images and improves characteristic recognition efficiency.
As one mode of implementation of the present application, in the third step, the fusing inertial data includes: when the feature matching fails, according to the pose change corresponding to the inertial data between two adjacent frames of images acquired by the camera, obtaining the rigid connection relation between the inertial sensor and the camera, under the condition that the internal parameters of the camera are known, calculating the feature point coordinates of the current frame of image predicted by the inertial sensor according to the rigid connection relation between the inertial sensor and the camera, and then combining the relative pose of the gray level image features and the landmark image associated features in a landmark database, comparing the feature point coordinates of the current frame of image predicted by the inertial sensor with the feature point coordinates of the current frame of image acquired by the camera, and updating and correcting the feature point coordinates of the current image acquired by the camera; when the feature matching is successful, calculating the current position coordinate of the positioning device according to the imaging feature geometric relation of the feature points of the gray image feature, converting according to the rigid connection relation between the camera and the inertial sensor, comparing with the current position coordinate obtained by the inertial sensor, and correcting the current position coordinate obtained by the feature points to obtain an accurate coordinate value of the current position.
Specifically, the inertial sensor is used for making a predictive motion model, the camera is used for making an observation model, and rigid connection between the inertial sensor and the camera is used as a parameter value to be estimated; and calculating accumulated values of the inertial data between two continuous frames of images, wherein the accumulated values comprise translation caused by speed and acceleration and rotation caused by angular speed, and because one inertial sensor is connected to the camera, the camera is connected to the image and the image is connected to the feature, meanwhile, the feature point can be obtained through image extraction, an optimization equation is constructed according to the imaging uniqueness principle of the same feature point in the image, and the pose provided by the inertial sensor is taken as an initial value to carry out iterative solution. And then, information fusion is carried out by using the predicted and observed covariance information to obtain an optimal estimation in the least square sense, and the accurate coordinate value of the positioning device in the current position is obtained by updating and correcting.
Recording the inertial data between two continuous frames of images shot by the camera and performing accumulation operation to obtain pose transformation recorded by the inertial sensor between two continuous frames, converting the pose transformation into pose transformation of the camera by utilizing a fixed rotation and translation transformation relation between the inertial sensor and the camera, and obtaining the coordinates of the characteristic coordinates of the previous frame in the current frame according to the camera internal reference matrix; and when the feature matching fails, predicting the current image coordinates through the current position coordinates acquired by the inertial sensor by using the conversion method, comparing the current image coordinates with the feature point coordinates in the current image features, updating and correcting the feature point coordinates in the current image features, and storing the feature point coordinates back to the landmark database as a new landmark created in the current position. When the feature matching is successful, the image processing module preprocesses the current position coordinates obtained by the imaging feature geometric relation operation of the coordinates of the gray image features, and compares the current position coordinates obtained by the inertial sensor with the current position coordinates obtained by the inertial sensor, namely, the current position coordinates obtained by the feature points are corrected by the observation model, so that correction and update of the current position coordinates obtained by the feature points are realized. The creation of a new landmark is optionally attempted when the locating device does not match the presence of a known landmark within the input image.
As an implementation manner of the present application, in the third step, the imaging feature geometric relationship is established based on a position relationship corresponding to an image acquired by the lens orientation angle of the camera at the preset position and inertial data acquired by the inertial sensor sensing road sign. The camera model adopts a traditional pinhole model, the internal parameters of the camera are known, and the geometric relationship of a similar triangle is constructed by combining the triangulation of the distance and the position of the feature on the road sign shot in the advancing process of the positioning device, so that the two-dimensional coordinates of the corresponding feature angular points on the road sign in the camera coordinate system can be calculated.
As an embodiment of a robot in the implementation of the present application, fig. 3 provides a structural diagram of a sweeping robot, which may be used as a specific application product structural diagram of a positioning device based on inertial data and visual characteristics provided in the implementation of the present application, and for convenience of explanation, only the parts related to the embodiment of the present application are shown. The image processing module and the fusion positioning module in the positioning device are arranged in the signal processing board 102; the image acquisition module comprises a camera 106, wherein the camera 106 is arranged at a backward protruding structure of the tail part of the machine body 101, and an optical axis of the camera 106 forms a certain inclination angle with the top surface of the positioning device, so that the camera 106 is far away from the collision detection sensor 105 and is prevented from being touched by some objects which are difficult to detect; the inertial data acquisition module comprises a collision detection sensor 105, the inertial data acquisition module senses under the action of the engine body 101 driven by the motion wheel 104 and the universal wheel 107, the inertial data acquisition module and the data acquired by the camera 106 are fused and corrected to position coordinates by applying the relative pose and the rigid connection relation, so that positioning navigation actions are executed, and the landmark database can be updated to be used as the basis for constructing a navigation map. And finally, the human-computer interface 103 outputs the accurate coordinate value of the current position of the sweeping robot calculated by the signal processing board.
As another embodiment of the robot according to the present application, fig. 4 provides a block diagram of a sweeping robot, which may be used as a block diagram of a specific application product of a positioning device based on inertial data and visual characteristics according to the embodiment of the present application, and for convenience of explanation, only a portion related to the embodiment of the present application is shown. In fig. 4, the camera 108 is mounted at a concave structure with a backward tail opening of the body 101, and an optical axis of the camera 108 forms a certain inclination angle with the top surface of the positioning device, so that the camera 108 is far away from the collision detection sensor 105, and the camera 108 has a good viewing angle range.
The above embodiments are merely for fully disclosing the present application, but not limiting the present application, and should be considered as the scope of the present disclosure based on the substitution of equivalent technical features of the inventive subject matter without creative work.
Claims (10)
1. The positioning device is a movable visual positioning device and is characterized by comprising an image acquisition module, an image processing module, an inertial data acquisition module and a fusion positioning module;
the camera of the image acquisition module is kept backwards and placed in a preset position of the positioning device and used for avoiding collision by obstacles in the advancing process of the positioning device, wherein the number of the cameras is at least one;
the image processing module comprises an image preprocessing sub-module and a characteristic matching sub-module and is used for processing the image data acquired by the image acquisition module; the image preprocessing sub-module is used for converting the data acquired by the camera into a gray image, and the feature matching sub-module is used for extracting feature data from the image preprocessed by the image preprocessing sub-module and matching the feature data with associated features of landmark images in a landmark database; the landmark database is a landmark database built in the image processing module, and comprises image feature points of a region associated with a given landmark;
the inertial data acquisition processing module consists of a series of inertial data measurement units and senses the rotation angle information, the acceleration information and the translation speed information of the inertial sensor in real time;
and the fusion positioning module is used for carrying out data fusion on the inertial data acquired by the inertial data acquisition processing module according to the feature matching result in the image processing module, and correcting the current position information through the data fusion result.
2. The positioning device of claim 1, wherein the predetermined position is a concave and/or convex configuration with a central opening rearward of a top surface of the positioning device.
3. The positioning device of claim 1, wherein the predetermined position is a concave and/or convex configuration of a rear opening of a top surface of the positioning device.
4. The positioning device of claim 1 wherein an angle formed by an optical axis of the camera head and a top surface of the positioning device spans between 0 and 80 degrees.
5. The positioning device according to claim 1, wherein in the fusion positioning module, when feature matching in the image processing module is successful, current position coordinates of the positioning device are calculated according to the geometrical relationship between the gray image features and imaging features of landmarks in an actual scene, and updating and correcting are performed by using the inertial data;
when feature matching in the image processing module fails, a rigid connection relation between the inertial sensor and the camera is obtained according to the accumulated value of the inertial data, and meanwhile, a new landmark is calculated by combining the gray image feature and the relative gesture of the landmark image associated feature in the landmark database, and is stored and recorded in the landmark database to complete creation of the new landmark;
the image sensor comprises an inertial sensor, a camera, a gray level image sensor, a landmark image sensor, a gray level image sensor and a landmark image sensor, wherein the inertial sensor is connected with the camera, mapping association exists between the camera and the gray level image feature or landmark image associated feature, and the feature can be obtained through extraction of the gray level image; the rigid connection relation is a position relation established based on pose changes corresponding to the inertial data between two adjacent frames of images acquired by the camera.
6. Positioning method based on inertial data and visual features, characterized in that it is applied to a positioning device according to any one of claims 1 to 5, comprising the steps of:
preprocessing an image collected by a camera aiming at a landmark in an actual scene to obtain a gray image, extracting characteristic points from the gray image, generating gray image characteristics, and determining descriptors of the characteristic points;
performing feature matching on the generated descriptors of the gray image features and descriptors of associated features of landmark images stored in a landmark database;
judging whether the gray image features are matched with the associated features of the landmark images in the landmark database, if so, fusing inertial data according to the geometric relationship between the gray image features and the imaging features of the landmarks in the actual scene to obtain the current position coordinates; otherwise, according to the rigid connection relation between the inertial sensor and the camera, combining the relative postures of the gray image features and the landmark image associated features in a landmark database, fusing the inertial data, calculating a new landmark, storing and recording the new landmark in the landmark database, and completing the creation of a new landmark;
the inertial data is subjected to calibration filtering processing, and the rigid connection relation is a position relation established based on pose changes corresponding to the inertial data between two adjacent frames of images acquired by the camera.
7. The positioning method of claim 6, wherein the feature matching process comprises: under the current frame image, calculating the Hamming distance between the descriptors of the gray image features and the corresponding descriptors in the associated features of the landmark images of the landmark database;
if the Hamming distance is smaller than a preset threshold value, the similarity of the collected image and the associated feature of the landmark image corresponding to the landmark database is high, and the matching is considered to be successful;
wherein the preset threshold corresponds to a determined numerical relationship of the relative pose.
8. The positioning method of claim 6 wherein the fusing inertial data comprises: when the feature matching fails, according to the pose change corresponding to the inertial data between two adjacent frames of images acquired by the camera, the rigid connection relation between the inertial sensor and the camera is obtained, under the condition that the internal parameters of the camera are known, the feature point coordinates of the current frame of image predicted by the inertial sensor are obtained through calculation according to the rigid connection relation between the inertial sensor and the camera, and then the feature point coordinates of the current frame of image predicted by the inertial sensor are compared with the feature point coordinates of the current frame of image acquired by the camera by combining the relative pose of the gray image features and the landmark images associated features in a landmark database.
9. The positioning method according to claim 6, wherein the imaging feature geometry is a similar triangle relationship established based on a positional relationship between feature points in a current frame image acquired on the camera at the preset position and landmarks in an acquired actual scene.
10. A robot, characterized in that the robot is a mobile robot provided with a positioning device according to any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810543857.6A CN108544494B (en) | 2018-05-31 | 2018-05-31 | Positioning device, method and robot based on inertia and visual characteristics |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810543857.6A CN108544494B (en) | 2018-05-31 | 2018-05-31 | Positioning device, method and robot based on inertia and visual characteristics |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108544494A CN108544494A (en) | 2018-09-18 |
| CN108544494B true CN108544494B (en) | 2023-10-24 |
Family
ID=63511456
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810543857.6A Active CN108544494B (en) | 2018-05-31 | 2018-05-31 | Positioning device, method and robot based on inertia and visual characteristics |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108544494B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109431381B (en) * | 2018-10-29 | 2022-06-07 | 北京石头创新科技有限公司 | Robot positioning method and device, electronic device and storage medium |
| CN109341724B (en) * | 2018-12-04 | 2023-05-05 | 中国航空工业集团公司西安航空计算技术研究所 | On-line calibration method for relative pose of airborne camera-inertial measurement unit |
| CN110039536A (en) * | 2019-03-12 | 2019-07-23 | 广东工业大学 | The auto-navigation robot system and image matching method of indoor map construction and positioning |
| CN110405774B (en) * | 2019-08-29 | 2021-03-30 | 广东博智林机器人有限公司 | Control system of robot and robot |
| CN111612765B (en) * | 2020-05-21 | 2023-06-16 | 华南师范大学 | Method for identifying and positioning round transparent lens |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103954283A (en) * | 2014-04-01 | 2014-07-30 | 西北工业大学 | Scene matching/visual odometry-based inertial integrated navigation method |
| CN104236548A (en) * | 2014-09-12 | 2014-12-24 | 清华大学 | Indoor autonomous navigation method for micro unmanned aerial vehicle |
| CN104574372A (en) * | 2014-12-21 | 2015-04-29 | 天津大学 | Image registration method based on similar feature triangles |
| CN106537186A (en) * | 2014-11-26 | 2017-03-22 | 艾罗伯特公司 | Systems and methods for performing simultaneous localization and mapping using machine vision systems |
| CN106662876A (en) * | 2014-08-18 | 2017-05-10 | 东芝生活电器株式会社 | Autonomous moving body |
| CN107255476A (en) * | 2017-07-06 | 2017-10-17 | 青岛海通胜行智能科技有限公司 | A kind of indoor orientation method and device based on inertial data and visual signature |
| CN212044739U (en) * | 2018-05-31 | 2020-12-01 | 珠海市一微半导体有限公司 | Positioning device and robot based on inertial data and visual characteristics |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8174568B2 (en) * | 2006-12-01 | 2012-05-08 | Sri International | Unified framework for precise vision-aided navigation |
| US8180146B2 (en) * | 2009-12-22 | 2012-05-15 | The Chinese University Of Hong Kong | Method and apparatus for recognizing and localizing landmarks from an image onto a map |
-
2018
- 2018-05-31 CN CN201810543857.6A patent/CN108544494B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103954283A (en) * | 2014-04-01 | 2014-07-30 | 西北工业大学 | Scene matching/visual odometry-based inertial integrated navigation method |
| CN106662876A (en) * | 2014-08-18 | 2017-05-10 | 东芝生活电器株式会社 | Autonomous moving body |
| CN104236548A (en) * | 2014-09-12 | 2014-12-24 | 清华大学 | Indoor autonomous navigation method for micro unmanned aerial vehicle |
| CN106537186A (en) * | 2014-11-26 | 2017-03-22 | 艾罗伯特公司 | Systems and methods for performing simultaneous localization and mapping using machine vision systems |
| CN104574372A (en) * | 2014-12-21 | 2015-04-29 | 天津大学 | Image registration method based on similar feature triangles |
| CN107255476A (en) * | 2017-07-06 | 2017-10-17 | 青岛海通胜行智能科技有限公司 | A kind of indoor orientation method and device based on inertial data and visual signature |
| CN212044739U (en) * | 2018-05-31 | 2020-12-01 | 珠海市一微半导体有限公司 | Positioning device and robot based on inertial data and visual characteristics |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108544494A (en) | 2018-09-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108406731B (en) | Positioning device, method and robot based on depth vision | |
| CN108544494B (en) | Positioning device, method and robot based on inertia and visual characteristics | |
| JP7150773B2 (en) | Mobile robot and its control method | |
| CN108481327B (en) | Positioning device, positioning method and robot for enhancing vision | |
| CN108647646B (en) | Low-beam radar-based short obstacle optimized detection method and device | |
| US10192113B1 (en) | Quadocular sensor design in autonomous platforms | |
| CN108290294B (en) | Mobile robot and control method thereof | |
| EP3283843B1 (en) | Generating 3-dimensional maps of a scene using passive and active measurements | |
| WO2019144541A1 (en) | Cleaning robot | |
| KR101784183B1 (en) | APPARATUS FOR RECOGNIZING LOCATION MOBILE ROBOT USING KEY POINT BASED ON ADoG AND METHOD THEREOF | |
| US11187790B2 (en) | Laser scanning system, laser scanning method, movable laser scanning system, and program | |
| WO2019138678A1 (en) | Information processing device, control method for same, program, and vehicle driving assistance system | |
| US11703334B2 (en) | Mobile robots to generate reference maps for localization | |
| KR20150144731A (en) | Apparatus for recognizing location mobile robot using edge based refinement and method thereof | |
| KR102056147B1 (en) | Registration method of distance data and 3D scan data for autonomous vehicle and method thereof | |
| CN112740274A (en) | System and method for VSLAM scale estimation on robotic devices using optical flow sensors | |
| CN108332752B (en) | Indoor robot positioning method and device | |
| Khan et al. | Stereovision-based real-time obstacle detection scheme for unmanned ground vehicle with steering wheel drive mechanism | |
| CN212044739U (en) | Positioning device and robot based on inertial data and visual characteristics | |
| KR101965739B1 (en) | Mobile robot and method for controlling the same | |
| Birk et al. | Simultaneous localization and mapping (SLAM) | |
| US11709499B2 (en) | Controlling method for artificial intelligence moving robot | |
| Lu | Vision-enhanced lidar odometry and mapping | |
| Onoguchi et al. | Planar projection stereopsis method for road extraction | |
| JP7354528B2 (en) | Autonomous mobile device, method and program for detecting dirt on lenses of autonomous mobile device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: 519000 2706, No. 3000, Huandao East Road, Hengqin new area, Zhuhai, Guangdong Applicant after: Zhuhai Yiwei Semiconductor Co.,Ltd. Address before: Room 105-514, No.6 Baohua Road, Hengqin New District, Zhuhai City, Guangdong Province Applicant before: AMICRO SEMICONDUCTOR Co.,Ltd. |
|
| CB02 | Change of applicant information | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |