WO2022027611A1

WO2022027611A1 - Positioning method and map construction method for mobile robot, and mobile robot

Info

Publication number: WO2022027611A1
Application number: PCT/CN2020/107863
Authority: WO
Inventors: 崔彧玮
Original assignee: 苏州珊口智能科技有限公司
Priority date: 2020-08-07
Filing date: 2020-08-07
Publication date: 2022-02-10
Also published as: CN112041634A

Abstract

A map construction method and a positioning method for mobile robot, a serving end, a mobile robot, and a computer-readable storage medium. The map construction method comprises: controlling an image capturing device and a ranging and sensing device to respectively acquire image data and measurement data synchronously at at least two positions (S110); analyzing the movement state of a mobile robot in a physical space by using at least the measurement data measured at different positions, so as to obtain at least first position data of a landmark feature in each piece of image data mapped to a physical coordinate system, and/or obtain second position data corresponding to the at least two positions in the physical coordinate system (S120); and recording the obtained first position data and second position data in map data constructed on the basis of the physical coordinate system (S130). The method takes the advantages and disadvantages of various sensors into consideration, and improves the accuracy and reliability of map construction by using a multi-sensor fusion manner.

Description

Positioning method of mobile robot, method of constructing map and mobile robot

technical field

The present application relates to the field of positioning technology, and in particular to a method for positioning a mobile robot, a method for constructing a map, and a mobile robot.

Background technique

A mobile robot is a machine device that automatically performs specific tasks. It can be commanded by people, run pre-programmed programs, or act according to principles and programs formulated with artificial intelligence technology. Mobile robots are widely used in large indoor occasions such as airports, railway stations, warehouses, hotels, etc. due to their autonomous mobility. For example, commercial cleaning robots, handling/distribution robots, welcome robots, etc., use the autonomous movement function to realize functions such as cleaning, handling, and guiding.

Affected by external factors such as the complexity of indoor venues and internal factors such as the hardware accuracy configured by mobile robots, mobile robots sometimes rely on the environmental information provided by a single detection device, which is not conducive to accurate positioning. For example, in an open area where obstacles are far away, it is not conducive for a mobile robot to measure the surrounding environment only by relying on laser measurement, and thus is not conducive to determining the exact positional relationship between it and the obstacle. For another example, in an area with a high similarity of the roof environment and a large range, it is not conducive for the mobile robot to obtain the landmark data only by relying on the camera device to determine the exact position in the area where it is located.

SUMMARY OF THE INVENTION

In view of the above-mentioned shortcomings of the related art, the purpose of the present application is to provide a method for overcoming the positioning and composition accuracy problems existing in the above-mentioned related art.

In order to achieve the above purpose and other related purposes, a first aspect disclosed in the present application provides a method for a mobile robot to construct a map, including: controlling the image capturing device and the ranging sensing device to obtain image data and measurement data synchronously at different positions, respectively; Use the measurement data obtained at different positions to analyze the moving state of the mobile robot in the physical space, so as to obtain the landmark feature in each image data mapped to the landmark position data in the physical coordinate system, and/or to obtain the different positions in the physical space. Corresponding positioning position data in the physical coordinate system; record the obtained location data and/or positioning position data in the map data constructed based on the physical coordinate system.

A second aspect disclosed in the present application provides a method for positioning a mobile robot, including: controlling an image capturing device and a ranging sensing device to acquire image data synchronously at a first position of the mobile robot and a second position different from the first position, respectively. and measurement data; wherein, the first position is mapped to the first positioning position data in the map data; the measurement data measured at the first position and the second position are used to analyze the moving state of the mobile robot in the physical space to When the mobile robot is in a second position, second positioning position data of the second position in the map data is determined.

A third aspect disclosed in the present application provides a server, including: an interface device for performing data communication with a mobile robot; a storage device for storing at least one program; a processing device, connected to the storage device and the interface device, for Execute the at least one program to coordinate the storage device and the interface device to execute and implement the method for constructing a map by a mobile robot as described in any one of the first aspects disclosed in the present application, or as in the second aspect disclosed in the present application Any of the described positioning methods for a mobile robot.

A fourth aspect disclosed in the present application provides a mobile robot, including: an image capturing device for acquiring image data; a ranging sensing device for acquiring measurement data; a moving device for performing a moving operation; and a storage device for Store at least one program; a processing device, connected to the mobile device, the image capturing device, the ranging sensing device, and the storage device, for executing the at least one program, so as to execute any one of the first aspects disclosed in this application. The method for constructing a map for a mobile robot described above, or the positioning method for a mobile robot according to any one of the second aspects disclosed in this application.

A fifth aspect disclosed in the present application provides a computer-readable storage medium, which stores at least one computer program, and when the computer program is run by a processor, the computer program controls the device where the storage medium is located to execute the first aspect disclosed in the present application. Any of the methods for constructing a map for a mobile robot, or a method for positioning a mobile robot according to any one of the second aspects disclosed in the present application.

To sum up, the positioning method of the mobile robot, the method of constructing the map, the server, the mobile robot and the storage medium in this application combine the advantages that multiple sensors can obtain more information in the physical space, and utilize the advantages of the image capturing device. The positioning capability and the measurement capability of the ranging sensing device, and the optimization of errors in multiple sensors, provides a new method and structure to realize the process of building maps and positioning, improving the accuracy and reliability of composition and positioning sex.

Other aspects and advantages of the present application can be readily appreciated by those skilled in the art from the following detailed description. Only exemplary embodiments of the present application are shown and described in the following detailed description. As those skilled in the art will recognize, the content of this application enables those skilled in the art to make changes to the specific embodiments disclosed without departing from the spirit and scope of the invention to which this application relates. Accordingly, the drawings and descriptions in the specification of the present application are only exemplary and not restrictive.

Description of drawings

The invention to which this application relates is set forth with particularity characteristic of the appended claims. The features and advantages of the inventions involved in this application can be better understood by reference to the exemplary embodiments described in detail hereinafter and the accompanying drawings. A brief description of the drawings is as follows:

FIG. 1 shows a schematic diagram of a method of constructing a map in the present application in one embodiment.

Figures 2a and 2b show a brief schematic diagram of a map constructed in this application in one embodiment.

Fig. 3 shows a schematic diagram of measurement data obtained by the mobile robot in the present application at two locations, respectively, in one embodiment.

FIG. 4 is a schematic diagram of another embodiment of the measurement data obtained by the mobile robot in the present application at two positions respectively.

FIG. 5 shows a schematic diagram of the positioning method of the mobile robot in the present application in one embodiment.

FIG. 6 is a schematic diagram of a server in an embodiment of the present application.

FIG. 7 is a schematic diagram of a module structure of a mobile robot in an embodiment.

detailed description

The embodiments of the present application are described below by specific specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification.

In the following description, reference is made to the accompanying drawings, which describe several embodiments of the present application. It is to be understood that other embodiments may be utilized and modular or unit compositional, electrical, as well as operational changes may be made without departing from the spirit and scope of the present disclosure. The following detailed description should not be considered limiting, and the scope of embodiments of the present application is limited only by the claims of the issued patent. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application.

Although in some instances the terms first, second, etc. are used herein to describe various positioning position data, information or parameters, these positioning position data or parameters should not be limited by these terms. These terms are only used to distinguish one location data or parameter from another. For example, first positioning position data may be referred to as second positioning position data, and similarly, second positioning position data may be referred to as first positioning position data, without departing from the scope of the various described embodiments. The first positioning position data and the second positioning position data are both describing one positioning position data, but unless the context clearly indicates otherwise, they are not the same positioning position data. Depending on the context, for example, the word "if" as used herein can be interpreted as "at the time of" or "when".

Also, as used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context dictates otherwise. It should be further understood that the terms "comprising", "comprising" indicate the presence of the stated features, steps, operations, location data, components, items, categories, and/or groups, but do not exclude one or more other features, steps, The existence, appearance, or addition of operations, location data, components, items, categories, and/or groups. The terms "or" and "and/or" as used herein are to be construed to be inclusive or to mean any one or any combination. Thus, "A, B or C" or "A, B and/or C" means "any of the following: A; B; C; A and B; A and C; B and C; A, B and C" . Exceptions to this definition arise only when combinations of location data, functions, steps or operations are inherently mutually exclusive in some way.

At present, the positioning technologies of mobile robots mainly include the following: positioning based on dead reckoning (DR), positioning based on map matching, and positioning based on laser SLAM (Simultaneous Localization and Mapping, SLAM) or visual SLAM, etc.

Among them, an example of the positioning method based on dead reckoning includes using an inertial measurement unit (Inertial Measurement Unit, IMU) and an odometer installed in the wheel set of the robot to measure the acceleration and angular velocity of the mobile robot during the movement process, and use The increments of these data are accumulated to derive the relative position of the mobile robot at a certain moment relative to the starting moment, so as to realize the positioning of the mobile robot. However, this method has the problem of accumulation of errors. For example, when the mobile robot is moved during a large-scale operation, the ground material (carpet, tile, wooden board, etc.) that the wheel group passes through is different, and the actual distance traveled is different from the sensor sensed. There are errors in the data, and the accumulated errors will gradually increase over time. If there is no additional positioning data to help correct it, the positioning of the mobile robot will eventually fail.

Secondly, an example of a localization method based on map matching includes using sensors on the mobile robot to detect the surrounding environment, constructing a local map, and matching it with the pre-stored complete map, so as to obtain the current position of the mobile robot in the entire environment. However, this method is limited by the layout of the environment and can only be applied to environments with relatively simple sites and obvious changes.

Furthermore, examples of positioning methods based on laser SLAM or visual SLAM include the use of laser detectors or cameras on mobile robots to detect the surrounding environment, combined with laser SLAM or visual SLAM technology, by using lasers to measure the distance and orientation data of landmark features. , and build an environment map corresponding to the measured landmark features to determine the movement trajectory and pose of the mobile robot. The reason why the above-mentioned positioning technology is not easy to be directly transferred to the workplace with the characteristics of large moving range, complex surrounding environment, or high similarity of top-view decoration environment, usually consider the following factors:

On the one hand, the indoor space of the family is small, and the environmental similarity of each location is low. The landmark features of the space area close to the ceiling, such as ceilings, walls, and wardrobes, are easily converted into data through images and ranging to achieve positioning. The above-mentioned workplace has a wider spatial range, which makes it possible that if the above-mentioned positioning technology is used for positioning, the object image space within the viewing angle of the same image capturing device is far away from the mobile robot, and the distance measuring sensor cannot provide accurate distance data. , which makes the map constructed by the mobile robot inconsistent with the real environment.

On the other hand, due to the high complexity of the environment such as the fluidity of objects around the mobile robot or the occlusion of landmark features, the mobile robot cannot rely on the configured image capture device to provide stable image data sources of landmark features for robot positioning. As a result, due to the failure to continuously detect the matchable landmark features, the actual path of the mobile robot may not match the planned path when it moves autonomously.

In view of this, an embodiment of the first aspect of the present application provides a method for constructing a map, so as to jointly optimize the error through data provided by the image capturing device and the ranging sensing device, respectively, to construct a map of the physical space where the mobile robot is located.

The method for constructing a map may be executed by a processor in the mobile robot, or may be executed by a server communicating with the mobile robot.

Among them, the mobile robot refers to an autonomous mobile device with the ability to build a map, including but not limited to: drones, industrial robots, home companion mobile devices, medical mobile devices, household cleaning robots, commercial cleaning robots, intelligent One or more of vehicles, and patrol mobile equipment.

The physical space refers to the actual three-dimensional space in which the mobile robot is located, and can be described by abstract data constructed in the space coordinate system. For example, the physical space includes, but is not limited to, family residences, public places (eg, offices, shopping malls, hospitals, underground parking lots, and banks), and the like. For a mobile robot, the physical space usually refers to an indoor space, that is, the space has boundaries in the directions of length, width, and height. In particular, it includes physical spaces such as shopping malls, waiting halls and other physical spaces with large spatial scope and high scene repetition.

The mobile robot is provided with an image capturing device, wherein the image capturing device is a device for providing a two-dimensional image according to a preset pixel resolution. In some embodiments, the image capturing device includes, but is not limited to, a camera, a video camera, a camera module integrated with an optical system or a CCD chip, a camera module integrated with an optical system and a CMOS chip, and the like. According to actual imaging requirements, lenses that can be used by the camera or video camera include, but are not limited to, standard lenses, telephoto lenses, fisheye lenses, and wide-angle lenses.

In some embodiments, the image capturing device may be disposed on the mobile robot and the main optical axis is between the horizontal plane and the vertical direction to the ceiling, such as the main optical axis is located at 0° in the vertical direction to the ceiling and within the range of 0±90°. Taking the mobile robot as a commercial cleaning robot as an example, the image capturing device is assembled on the upper half of the commercial cleaning robot, and its main optical axis is a preset angle obliquely upward to obtain image data within a corresponding viewing angle range.

The images captured by the image capturing device may be one or more of a single image, a continuous image sequence, a non-sequential image sequence, or a video. If the image capturing device captures an image sequence or video, one or more image frames can be extracted from the sequence or video as image data for subsequent processing.

Wherein, the image data reflects the perception of the physical space where the mobile robot is located by the image capturing device. In some cases, the image acquired by the image capturing device may be directly used as image data; in other cases, the image acquired by the image capturing device may also be processed as data. For example, when the image acquired by the image capturing device is a single image, the single image captured at the first position and the second position can be directly used as the image data. For another example, when the image acquired by the image capturing device is an image sequence or video, an image frame may be extracted from the image sequence or video as image data.

In some embodiments, the mobile device stores the captured images in a local storage medium. Wherein, the storage medium may include read-only memory, random access memory, EEPROM, CD-ROM or other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices, flash memory, U disk, removable hard disk, or can be used for storage Any other medium that has the desired program code in the form of instructions or data structures and can be accessed.

In some embodiments, the mobile robot transmits the captured image to an external device connected in communication for storage, and the communication connection includes a wired or wireless communication connection. The external device may be a server located in the network, and the server includes but is not limited to one or more of a single server, a server cluster, a distributed server group, and a cloud server. In a specific implementation, the cloud server may be a cloud computing platform provided by a cloud computing provider. Based on the architecture of the cloud server, the types of the cloud server include but are not limited to: Software-as-a-Service (Software as a Service, SaaS), Platform-as-a-Service (Platform as a Service, PaaS) , and Infrastructure-as-a-Service (Infrastructure as a Service, IaaS). Based on the nature of the cloud server, the types of the cloud server include but are not limited to: public cloud (Public Cloud) server, private cloud (Private Cloud) server, and hybrid cloud (Hybrid Cloud) server and the like.

In some embodiments, the public cloud server is, for example, Amazon's elastic computing cloud (Amazon EC2), IBM's Blue Cloud, Google's AppEngine, and Windows' Azure service platform, etc.; the private cloud server For example, Alibaba Cloud Computing Service Platform, Amazon Cloud Computing Service Platform, Baidu Cloud Computing Platform, and Tencent Cloud Computing Platform.

The mobile robot is further provided with a ranging sensing device, and the ranging sensing device can measure the distance of the landmark feature in the physical space where the mobile robot is located relative to the mobile robot.

The landmark feature refers to a feature that is easy to distinguish from other objects in the physical space where the mobile robot is located. For example, the landmark feature can be a table corner, a contour feature of a ceiling light on a ceiling, a space between a wall and the ground. connection etc.

Examples of the ranging sensing device include ranging sensing devices that provide one-dimensional measurement data such as laser sensors and ultrasonic sensors, or include two-dimensional or binocular camera devices such as ToF sensors, multi-line lidars, millimeter-wave radars, or binocular cameras. The ranging sensing device for three-dimensional measurement data may also include the above two ranging sensing devices. For example, a laser sensor can determine its distance from a landmark feature based on the time difference between the time it emits a laser beam and the time it receives the laser beam; for another example, an ultrasonic sensor can determine movement based on the vibration signal that the sound wave it emits is bounced back by the landmark feature. The distance of the robot relative to the landmark feature; another example, the binocular camera device can use the triangulation principle to determine the distance of the mobile robot relative to the landmark feature according to the images captured by its two cameras; another example, ToF (Time of Flight, The infrared light projector of the time-of-flight) sensor projects infrared light outward, and the infrared light is reflected after encountering the measured object and received by the receiving module. The depth of the illuminated object is calculated by recording the time from infrared light emission to being received. information.

The range sensing device generates corresponding measurement data by detecting entities in the surrounding environment of the mobile robot. In order to prevent the mobile robot from abnormal movements such as collision and entanglement with entities in the surrounding environment, or to prevent abnormal movements such as the mobile robot falling, the ranging sensing device is installed on the body side of the mobile robot. The ranging sensing device is used to detect the edge of the corresponding image data in the physical space or the environmental data in the area that cannot be covered by the image data. Taking the mobile robot being a commercial cleaning robot as an example, the ranging sensing device may include, but is not limited to, be disposed at a position between 10-80 cm from the ground on the side of the commercial cleaning robot.

In some embodiments, the ranging sensing device is mounted on one side of the mobile robot in the direction of travel, so that the mobile robot can learn about landmark features in the direction of travel so as to avoid or take other behavioral controls. Of course, in some cases, the ranging sensing device can also be installed at other positions of the mobile robot, as long as the distances relative to each landmark feature in the surrounding physical space can be obtained.

In some embodiments, there may be one or more types of ranging sensing devices installed on the mobile robot. For example, only a laser sensor may be installed on the mobile robot, a laser sensor and a binocular camera device may be installed at the same time, or a laser sensor, a binocular camera device and a ToF sensor may be installed at the same time. In addition, the installed quantity of the same sensor can also be configured according to requirements to obtain measurement data in different directions.

The measurement data reflects the perception of the physical space where the mobile robot is located by the ranging sensing device. In other words, the measurement data reflects the physical quantities with physical meanings in the physical space that can be detected by the ranging sensing device. For example, relative positions of entities in physical space relative to mobile robots, etc. Wherein, the physical meaning means that the measurement data can provide physical data in basic units such as time unit/length unit/angle unit.

The measurement data at least reflects the positional relationship between the mobile robot and the entities in the surrounding environment, wherein the measurement data includes the distances of each landmark feature that can be observed in the physical space where the mobile robot is located relative to the mobile robot and the relative orientation. In some cases, the data acquired by the ranging sensing device can be directly used as the measurement data; in other cases, the data acquired by the ranging sensing device can also be processed to obtain the measurement data. For example, when the ranging sensing device is a laser sensor, the distance detected by the laser sensor relative to each landmark feature can be directly used as the measurement data; for another example, when the ranging sensing device is a binocular camera device , then the image obtained by the binocular camera device is processed to obtain the distance relative to each landmark feature and use it as the measurement data.

Generally, since there are multiple landmark features in the physical space, there are distances and relative orientations relative to the multiple landmark features in the measurement data, thereby presenting the depth point cloud data.

In some examples, the measurement data also reflects the mobile robot's perception data of entities in its surroundings. For example, the measurement sensing device is a laser sensor, which emits a light beam to a solid surface in the surrounding environment to detect a set of measurement points that can reflect the light beam, and acquires corresponding measurement data. The relative position is obtained by 3D reconstruction of the acquired measurement data through predetermined physical parameters.

It should be noted that although an image capturing device and a ranging sensing device are installed in the mobile robot, the viewing angle range of the image capturing device and the measuring range of the ranging sensing device may not overlap, or the image There is at most partial overlap between the viewing angle range of the capturing device and the measurement range of the ranging sensing device, that is, the execution of each step in this application does not necessarily depend on the landmark features detected by the image capturing device and the ranging sensing device, even if There is no common landmark feature in the image capturing device and the ranging sensing device, nor does it affect the process of constructing the map in this application.

In an exemplary embodiment, please refer to FIG. 1 , which shows a schematic diagram of a method of constructing a map in the present application in one embodiment. As shown in the figure, in step S110, the image capturing device and the ranging sensing device are controlled to acquire image data and measurement data simultaneously at at least two positions, respectively.

Here, the mobile robot travels through a number of locations during its movement in the current physical space. In order to correlate the image data and the measurement data accurately, the image capturing device and the ranging sensing device of the mobile robot respectively acquire the image data and the measurement data corresponding to each position at each position.

Take mobile robots working in airports, supermarkets and other working environments as an example, because the physical space where mobile robots are located includes moving objects such as walking passengers and their suitcases, or walking customers and their shopping carts, as well as fences, shelves, etc. , standing people and other fixed objects, therefore, the measurement data measured at at least two locations includes both fixed objects and possibly moving objects. The image data acquired at the same two positions at least include image data of entities in the physical space whose vertical direction is higher than the detection range of the ranging sensing device, and may include image data of corresponding part of the measurement data.

In order to improve the data accuracy when the data from two independent devices is used in fusion, the image data and measurement data belonging to the same location are acquired synchronously. Here, the synchronous acquisition method includes that the mobile robot stops at multiple locations along the way and acquires image data and measurement data, and then continues to move to achieve synchronous acquisition of image data and measurement data; Synchronized acquisition of image data and measurement data.

In a possible implementation manner, the synchronization acquisition method may be based on an external synchronization signal, and the image capturing device and the ranging sensing device may acquire the image data and measurement data corresponding to the position synchronously; it may also be based on the The synchronization signal in the image capturing device or the ranging sensing device is used to obtain the image data and measurement data corresponding to the position synchronously by the image capturing device and the ranging sensing device. For example, during the working process of the mobile robot in the current physical space, the control device in the mobile robot sends a synchronization signal to the image capturing device and the ranging sensing device at certain time intervals, so as to make the image capturing device and the ranging device The sensing device acquires image data and measurement data, respectively. For another example, the image capturing device and the ranging sensing device are provided with a clock module, and the same clock signal mechanism is preset to send out signals synchronously. When the image capturing device and the ranging sensing device receive their respective synchronization signals, Perform the steps to acquire image data and measurement data.

In some cases, the external synchronization signal may also be generated based on the signal in the inertial navigation sensor of the mobile robot. The inertial navigation sensor (IMU) is used to acquire inertial navigation data of the mobile robot. The inertial navigation sensor includes, but is not limited to, one or more of a gyroscope, an odometer, an optical flow meter, and an accelerometer. The inertial navigation data acquired by the inertial navigation sensor includes, but is not limited to, one or more of the speed data, acceleration data, moving distance, rolling circles of the roller, and deflection angle of the roller, etc. of the mobile robot.

For example, an IMU is configured in the driving component of the mobile robot (that is, the component such as the wheel set for advancing the mobile robot), the IMU is connected with the lower computer of the mobile robot, and the lower computer is connected with the image capturing device; and, in An IMU is also arranged in the ranging sensing device, and the ranging sensing device and the image capturing device are connected with the upper computer of the mobile robot. Here, the time stamps of the IMU in the ranging sensing device and the IMU in the mobile robot driving assembly are kept synchronized, so that the IMU in the mobile robot driving assembly generates a synchronization signal to the image capturing device so that the image capturing device acquires image data , the IMU in the ranging sensing device also generates a synchronization signal at the same time, so that the ranging sensing device acquires measurement data.

For another example, an IMU is configured in the driving component of the mobile robot (ie, components such as a wheel set for moving the mobile robot forward), and the IMU is connected with the lower computer of the mobile robot, and the lower computer is respectively connected with the image capturing device and the distance measuring device. Induction device connection. The IMU sends a synchronization signal to the image capturing device and the ranging sensing device when it detects that the wheel set rotates a preset number of turns, so that the image capturing device can obtain image data and the ranging sensing device can obtain measurement data.

For ease of description, in the following embodiments, how the mobile robot constructs the map at two positions (ie, the first position and the second position) will be taken as an example.

Please continue to refer to FIG. 1, in step S120, the mobile robot at least analyzes the movement state of the mobile robot in the physical space by using the measurement data obtained at different positions, so as to obtain at least the mapping of landmark features in each image data to physical coordinates location data of the landmark in the system, and/or to obtain positioning location data corresponding to the at least two locations in the physical coordinate system.

Here, both the measurement data and the image data obtained in step S110 include data helpful for positioning/building a map within different spatial height ranges in the physical space where the mobile robot is located, such as landmark data, distance/ angle etc. In addition, since the ranging sensing device and the image capturing device move as a whole with the mobile robot, at least the movement state determined by the measurement data obtained at different positions can solve the problem that the landmark features reflected in the image data have no physical scale. The location of landmarks in the coordinate system (such as scale), and only using measurement data is not conducive to obtaining high-precision positioning. An example of the set coordinate system includes a camera coordinate system, or a virtual coordinate system that lacks a mapping relationship determined based on a physical scale with the physical coordinate system. Thus, the moving state of the mobile robot described by the measurement data and the landmark data corresponding to the landmark features in the image data are used to calculate the landmark position data mapped from the landmark features captured by the image data to the physical coordinate system.

Specifically, if there is no reference to reflect the actual movement of the mobile robot in the physical space, the mobile robot cannot determine the physical distance of its own movement, the physical distance between it and an entity, etc. based on the image data to reflect the actual physical Data about positional relationships in space. To this end, the mobile robot obtains the moving state of the mobile robot in the physical space by analyzing the measurement data with physical meaning; the mobile robot is assembled on the mobile robot by using the moving state and the preset image capturing device and distance measuring device. As well as the internal and external parameters during image acquisition, the mobile robot constructs the correspondence between the landmark data in the image data and the landmark position data in the physical coordinate system, thereby determining the landmark position data of the corresponding landmark feature in the physical coordinate system.

Wherein, the moving state includes at least one of the following: data related to the positioning of the mobile robot itself, data used to help the mobile robot determine positioning, or used to reflect the change of the moving position of the mobile robot in the physical space and the landmark feature reflected in the image data Data such as the physical scale of the mapping relationship between image pixel position changes. Wherein, the data related to the positioning of the mobile robot itself includes, but is not limited to: changes in the posture of the mobile robot relative to the entity, changes in the posture and posture between the front and rear positions of the mobile robot, or information on each position the mobile robot passes through, etc. ; The data for helping the mobile robot to determine positioning includes, but is not limited to: the relative positional relationship (such as pose change data, etc.) between the landmark data corresponding to the same landmark feature in the measurement data and the mobile robot, the Landmark locations, etc.

It should be noted that the landmark feature described in the measurement data and the landmark feature described in the image data are not necessarily the same landmark feature. In the following, for convenience of description, the landmark data reflecting the landmark features in the measurement data is referred to as landmark measurement data, and the landmark data reflecting the landmark features in the image data is referred to as landmark image data.

The manner in which the mobile robot analyzes the movement state of the mobile robot in the physical space by at least using the measurement data obtained at different positions includes the following examples:

In some examples, the mobile robot obtains the corresponding movement state by analyzing the measurement data measured at different positions to reflect the change of its physical position. Wherein, the analysis process reflecting the change of its physical position includes a process of analyzing the movement behavior of the mobile robot by using measurement data with physical meaning. The mobile robot analyzes the data values of the measurement data relative to the same entity at at least two positions, and the positional deviation between the measurement data relative to the same entity, etc., to obtain the mobile robot's position and posture change caused by the movement. mobile state.

In order to improve the measurement accuracy of the moving state, in some specific examples, the mobile robot calculates the moving state by using the landmark measurement data corresponding to the common landmark feature among the identified measurement data.

Taking the measurement data obtained by the mobile robot at the first position and the second position as an example, the measurement data obtained by the mobile robot at the first position and the second position respectively have at least one common landmark feature. The moving state of the mobile robot can be calculated relative to the distances of the common landmark features.

In the process of constructing the map from the mobile robot's preset initial moving position as the starting point, the mobile robot, when the positioning position information of the previous position in the physical coordinate system is known, uses the above-mentioned example to obtain the difference between the different positions before and after. In the moving state, positioning position data corresponding to the at least two positions in the physical coordinate system, and/or landmark position data corresponding to the landmark feature in the measurement data in the physical coordinate system, etc. are obtained. It should be understood that, since the measurement data acquired by the distance-measuring sensing device is data with actual physical units, the calculated movement state also has actual physical units.

In some other specific examples, the mobile robot obtains the moving state by calculating the coincidence degree of each measurement data obtained at different positions. Among them, the single measurement data provides the positional relationship between the entity part in the corresponding measurement plane and the mobile robot in the physical space. When the mobile robot measures the same entity at different positions, it reflects the changes caused by the pose change of the mobile robot. A change in the measurement position of the same entity in the measurement data. For example, a mobile robot can rotate, translate, and zoom the measurement data obtained at one position to obtain measurement data that coincides with some measurement points in the measurement data at another position. The processing process reflects the process of simulating a mobile robot to perform a moving operation in order to make the corresponding measurement points coincide. To this end, by performing the coincidence optimization operation, the mobile robot can simulate the data and physical scale of the mobile robot's pose change between different positions and other moving states, and use the pose change data and other moving states, and the coincident The physical quantity provided by the measurement data determines other data in the moving state, such as the positioning position data corresponding to the at least two positions in the physical coordinate system, and/or the landmark measurement data in the measurement data corresponding to the physical coordinate system Landmark location data, etc.

Since the measurement data includes the physical quantity of the fixed object and the physical quantity of the corresponding moving object measured by the ranging sensing device at the corresponding position, for this reason, in an exemplary embodiment, the mobile robot uses the measurement data obtained at different positions The degree of coincidence between them is optimized to obtain at least the coincident measurement data in the measurement data that meet the coincidence conditions and map to the landmark position data under the physical coordinate system, and/or the mobile robot obtains at least the data when the data is obtained. Positioning position data corresponding to two positions in the physical coordinate system. Wherein, the mobile robot may regard the overlapping measurement data part as the landmark data corresponding to the landmark feature. An example of the coincidence condition includes that the number of iterative optimizations of the coincidence degree reaches a preset threshold of the number of iterations, and/or the gradient value of the coincidence degree is smaller than a preset gradient threshold value, and the like. This method is beneficial to calculate the movement state by reflecting the physical quantity of the fixed object in each measurement data as much as possible.

Taking the measurement data as one-dimensional data as an example, please refer to FIG. 3 , which is a schematic diagram of an embodiment of the measurement data obtained by the mobile robot in the present application at two positions respectively. As shown in the figure, the mobile robot obtains measurement data MData_1 and MData_2 at the first position and the second position, respectively, wherein if there is an entity with a fixed position in the physical space, the measurement data MData_1 and MData_2 have corresponding and available entities in the physical space. Measurement points that are coincident after data processing. From the measurement points of entity 311, entity 312 and entity 313 in the measurement data MData_1 and MData_2 in the figure, it can be seen that both entity 311 and entity 313 in the two measurement data can be overlapped after data processing, while entity 312 cannot Coincidence with entity 311 and entity 313 at the same time, so entity 311 and entity 313 can be regarded as entities with fixed positions. The data processing of the two measurement data MData_1 and MData_2 by the mobile robot including rotation, translation, and zooming reflects the relative positional relationship between the first position and the second position of the mobile robot. To this end, the mobile robot uses an optimization function constructed according to the two measurement data MData_1 and MData_2 to perform an optimal calculation of the coincidence degree until the coincidence degree of the two measurement data MData_1 and MData_2 meets the preset coincidence condition. Thereby, the pose change data reflecting the relative position relationship is obtained, and under the preset coincidence condition, the coincident measurement data points in the two measurement data are used as the landmark measurement data, and based on the obtained pose change data, we obtain In the measurement data, at least the landmark measurement data is mapped to the landmark position data in the physical coordinate system, and/or the positioning position data corresponding to the at least two positions in the physical coordinate system. It should be noted that the above example is also applicable to the calculation processing in which the measurement data is two-dimensional data.

In order to improve computational efficiency, in another exemplary embodiment, the mobile robot optimizes the degree of coincidence between the measurement data of landmarks in the measurement data obtained at different positions, so as to obtain the movement of the mobile robot under the coincidence condition. state.

Wherein, the optimization process is similar to the above example of one-dimensional data. The difference is that, in an exemplary embodiment, the method for constructing a map further includes the step of extracting landmark features in each measurement data, so as to use the landmark measurement data corresponding to the landmark features in each measurement data to pair the movement. status is analyzed. The landmark measurement data includes landmark measurement data corresponding to moving objects and landmark measurement data corresponding to stationary objects. For example, the measurement data obtained by the ranging sensing device includes its distance relative to each feature in the physical space. It includes some static features (such as flower pots, coffee tables, shelves, etc.) and some moving features (such as people, pets, shopping carts in progress, etc.).

It should be understood that in some scenarios, some features in the physical space are not suitable to be used as landmark features, for example, movable objects such as people and pets in the physical space. If these inappropriate features are marked in the map data, it will affect the subsequent positioning of the mobile robot, and easily lead to large errors. Therefore, it is necessary to filter each data in the measurement data to determine which ones can be used as landmark features, so as to use these extracted landmark features to analyze the moving state and improve the calculation accuracy.

In one embodiment, the features that meet the requirements are selected as landmark features by means of a threshold. For example, the measurement data obtained at the first position and the measurement data obtained at the second position have five common features a, b, c, d, and e, wherein the calculated values based on a, b, c, and d are respectively The numerical difference between the moving states is within 1cm, and the numerical difference between the moving state calculated based on e and the moving state calculated by other features is in the range of 74-75cm. If the screening threshold is set to 5cm, then a, b , c, and d can be extracted as landmark features, while e may be an active feature and cannot be used as landmark features.

In another embodiment, landmark features may also be extracted directly based on numerical changes in measurement data acquired at each location. Specifically, in the measurement data obtained from different positions, the changes of static features in each position data are relatively regular, while the changes of active features are relatively different from other static features. Therefore, a predetermined change threshold can be used to find a feature whose numerical value changes relatively regularly in the measurement data relative to the distance of each feature, and use it as a landmark feature. For example, the measurement data obtained at the first position and the measurement data obtained at the second position have five common features a, b, c, d, and e. At the first position, the distance of feature a relative to the mobile robot is 2m, the distance of feature b relative to the mobile robot is 3.4m, the distance of feature c relative to the mobile robot is 4.2m, and the distance of feature d relative to the mobile robot is 2.8m, the distance of the e feature relative to the mobile robot is 5m; at the second position, the distance of the a feature relative to the mobile robot is 2.5m, the distance of the b feature relative to the mobile robot is 3.8m, and the distance of the c feature relative to the mobile robot is 3.8m. The distance of the mobile robot is 4.9m, the distance of the d feature relative to the mobile robot is 3.6m, and the distance of the e feature relative to the mobile robot is 2m. It can be seen that the variation of a feature at two positions is 0.5m, the variation of b feature at two positions is 0.4m, the variation of c feature at two positions is 0.7m, and the variation of d feature at two positions is 0.7m. The variation at each position is 0.8m, and the variation of the e-feature at two positions is 3m. Assuming that the preset change threshold is 0.5m, a, b, c, and d can be extracted as landmark features, while e may be an active feature and cannot be used as landmark features.

When determining the static landmark feature determined in any one of the above two embodiments, the coincidence degree is calculated by using the landmark measurement data corresponding to the static landmark feature in the measurement data to determine the corresponding moving state. For example, the coincidence degree is calculated according to the positional deviation between the landmark measurement data corresponding to the same static landmark feature at different locations, so that the positional deviation between the landmark measurement data is minimized, and the corresponding coincidence degree is determined when the coincidence degree condition is met. mobile state.

In still other embodiments, the coincidence degree optimization is performed on each landmark measurement data including the moving object and the fixed object, and when the coincidence degree satisfies the coincidence degree condition, it is determined that the coincident landmark measurement data corresponds to the landmark feature of the fixed object.

The process of optimizing the coincidence degree mentioned in any of the above embodiments can be exemplified as follows: please refer to FIG. 4 , which shows the measurement data obtained by the mobile robot in the present application at two positions respectively in another embodiment. Schematic. As shown in the figure, the measurement data MData_3 and MData_4 obtained by the mobile robot at the first position and the second position, wherein the circle represents the matching landmark measurement data in the two measurement data, and the star represents the two measurement data in one measurement data. Coincidence calculates the coincident landmark measurement data. The mobile robot iteratively optimizes the coincidental landmark measurement data so that the number of landmark measurement data represented by the star in previous coincidence calculations reaches the coincidence condition, and determines the movement state obtained in the corresponding times of coincidence calculation; Using the obtained movement state and the coincident landmark measurement data, at least the landmark feature corresponding to the landmark measurement data in the measurement data is mapped to the landmark position data in the physical coordinate system, and/or the mobile robot acquires data Positioning position data corresponding to at least two positions in the physical coordinate system.

Using the movement state obtained in any of the above examples, the mobile robot also determines the landmark position information in the physical coordinate system of the landmark feature corresponding to the landmark image data in the synchronously obtained image data.

During the movement of the mobile robot, the image capturing device also acquires image data synchronously at each position. By comparing the landmark image data corresponding to the common landmark features in the image data captured at each position, the movement amount and rotation amount of the mobile robot in the set coordinate system are calculated. The relative position relationship of the mobile robot, etc. Assuming that in the set coordinate system, the initial position of the mobile robot is taken as the coordinate origin, the coordinates of each positioning position of the mobile robot in the set coordinate system and the landmark features reflected by the image data can be obtained in the set coordinate system. The coordinates of the landmark location below.

Although the mobile robot can calculate the coordinates of each position the mobile robot passes through in its set coordinate system and the coordinates of each landmark image feature, these coordinates lack the actual physical unit, that is, the mobile robot cannot know the coordinates in its image data. The mapping relationship between the position coordinates of a pixel point and its corresponding physical measurement point in the physical space under the set coordinate system and the position in the actual physical coordinate system.

The mobile robot converts the coordinates of each position marked in the set coordinate system to the physical coordinate system by using the movement state with physical units calculated by the measurement data, so as to obtain the landmark features in each image data and map them to the physical coordinate system. and/or to obtain positioning position data corresponding to the at least two positions in the physical coordinate system, etc. Wherein, since the ranging sensing device and the image capturing device move with the mobile robot as a whole, the moving state obtained by the mobile robot using the measurement data reflects the movement of the mobile robot relative to the landmark feature corresponding to the landmark image data in the image data. Here, based on the mapping relationship between the set coordinate system and the physical coordinate system determined by the moving state, the mobile robot maps the landmark image features in the set coordinate system to the physical coordinate system, so as to obtain The corresponding landmark features are mapped to the landmark location data in the physical coordinate system.

Using a mobile robot to optimize the degree of coincidence between the measurement data acquired at different positions, the corresponding landmark features in the image data that meet the coincidence conditions are mapped to the landmark position data in the physical coordinate system, and/ Or the positioning position data corresponding to the at least two positions in the physical coordinate system are described as an example to describe the execution process of obtaining each position data in the physical coordinate system:

The mobile robot converts the measurement data obtained at the first position and the second position into the coordinate system of the same measurement data, and performs iterative calculation of the coincidence degree to obtain the coincidence degree of the two measurement data in the corresponding coordinate system. The coincidence condition is met, thereby obtaining the moving state of the mobile robot under the coincidence condition. Wherein, the movement state includes the physical scale between the relative positional relationship between the first position and the second position of the mobile robot and the measured positional relationship between the coincident landmark measurement data, the relative position of the mobile robot in the second position relative to the first position. Position and pose change data, etc. Wherein, under the action of the overall movement of the mobile robot, the physical scale also corresponds to the conversion relationship between the set coordinate system and the physical coordinate system (ie, the map scale). For example, the conversion relationship between the set coordinate system and the physical coordinate system is determined based on the physical scale and internal and external parameters of the image pickup device, wherein the internal and external parameters include the internal optical parameters and assembly parameters of the image pickup device, and the image Assembly parameters between the capture device and the ranging sensing device, etc.

After obtaining the conversion relationship between the set coordinate system and the physical coordinate system, the mobile robot calculates the positioning position data corresponding to each position of the mobile robot in the set coordinate system in the physical coordinate system, and the corresponding positioning position data in the set coordinate system The landmark features under the system are mapped to the landmark location data under the physical coordinate system. Of course, in practical applications, the required data can be obtained according to the requirements. For example, in some cases, only the landmark location data of the landmark feature mapped to the physical coordinate system needs to be obtained; Positioning position data corresponding to each position in the physical space in the physical coordinate system; in some cases, the landmark position data and the positioning position data need to be obtained at the same time.

It should be added that the coincidence condition may be a preset coincidence degree condition, or select a local optimal coincidence degree, which includes but is not limited to: the coincidence degree of common landmark features in the two measurement data obtained based on multiple coincidence iterations The average value is the highest, or the standard deviation of the coincidence degree of each common landmark feature in the two measurement data is the smallest, and the gradient of the coincidence degree is locally minimal.

It should be noted that the positioning position data may also be determined according to the movement state obtained when the measurement data meets the coincidence condition, or the positioning position data may be determined after error correction according to the above two calculation methods.

The following will use a specific example to illustrate how to optimize the degree of coincidence between the measurement data obtained at different locations, so as to obtain the landmark location data and the landmark location data mapped to the physical coordinate system from the landmark feature that meets the coincidence condition. Positioning position data corresponding to at least two positions in the physical coordinate system. It should be noted that this example is only used to explain the present application and not to limit the present application.

Assuming that the positions of the image capturing device itself in the current physical space at two moments (time 1 and time 2) before and after the set coordinate system are _Twc1 and _Twc2 , and the position of the ranging sensing device in the set coordinate system is T _wb1 and T _wb2 . Using the assembly parameters between the image capturing device and the ranging sensing device, that is, the position T _bc of the ranging sensing device relative to the image capturing device, it can be obtained: T _wc1 =T _wb1 ×T _bc , T _wc2 =T _wb2 ×T _bc .

Assuming that the scale factor between the distance unit of the set coordinate system and the actual physical coordinate system is s, the coordinates of T _wc1 in the set coordinate system (in matrix form) are:

Then the coordinates in the real physical coordinate system are:

The coordinates of T _wc2 in the set coordinate system are:

Then the coordinates in the real physical coordinate system are:

Among them, R _wc1 and R _wc2 both represent the rotation amount, and t _wc1 and t _wc2 both represent the translation amount.

Similarly, according to the distance of each landmark feature relative to the mobile robot in the set coordinate system, the coordinates of each landmark feature in the set coordinate system and the coordinates corresponding to the physical coordinate system can also be obtained in a similar way. This will not be repeated.

Using the T′ _wc1 matrix, the relative positions of the image pickup device at time 1 and time 2 can be solved T _c1c2 =T _c1w T _wc2 . in:

_Twc2 is the position of the image pickup device in the physical coordinate system at time 2 .

Using T _c1c2 =T _c1w T _wc2 , the measurement data at time 2 is mapped to the coordinate system where the measurement data at time 1 is located to form simulated measurement data SData (or the measurement data at time 1 can also be mapped to time 2, the principle The same, and will not be repeated), and the measurement data acquired by the ranging sensing device at time 1 is taken as the actual measurement data RData. It should be understood that by mapping the measurement data at time 2 to time 1, the positions and angles at time 2 relative to the positions and angles of various landmark features at time 1 can be simulated theoretically.

Here, one of the simulated measurement data SData and the actual measurement data RData is adjusted by a scaling factor (ie, physical scale), so that the coincidence degree of the simulated measurement data SData and the actual measurement data RData reaches the coincidence degree condition, thereby determining the scaling The value of the coefficient, the scale scaling factor is

and

The value of s in . Therefore, by substituting s, the coordinates of the mobile robot in the physical coordinate system (that is, the positioning position data) and the coordinates of each landmark feature in the physical coordinate system (that is, the landmark position data) at time 1 and time 2 can be determined. .

In other examples, in order to prevent the mobile robot from interfering with the acquisition of corresponding data by the ranging sensing device and the image capturing device due to external factors such as bumping and being picked up during the movement, the mobile robot measures the measurement data and The image data is analyzed to reflect the change of its physical position, and the corresponding movement state is obtained.

In an exemplary embodiment, the present application considers the error in the image capturing device to jointly optimize the error of the image capturing device and the ranging sensing device, thereby improving the accuracy of constructing the map.

Here, the degree of coincidence between the measurement data acquired at different positions and the degree of coincidence between the acquired image data are jointly optimized to obtain the corresponding landmarks in the image data under the coincidence conditions that meet the joint optimization. The feature is mapped to landmark location data in the physical coordinate system, and/or positioning location data corresponding to the at least two locations in the physical coordinate system.

In one embodiment, each measurement data and each image data are adjusted respectively to optimize the degree of coincidence of each measurement data and the degree of coincidence of each image data; when the errors between the moving states obtained based on the respective degrees of coincidence conform to the error When conditions are met, the corresponding movement states are used to determine the landmark position data in the image data mapped to the corresponding landmark feature in the physical coordinate system, and/or the positioning position data corresponding to the at least two positions in the physical coordinate system. An example of the error condition includes a preset error threshold, or a minimum error value selected based on a preset number of adjustments.

Taking the image data and measurement data obtained by the mobile robot at the first position and the second position synchronously as an example, the measurement data obtained when the mobile robot is at the first position is mapped to the second position, and the simulation is obtained by calculating the coincidence degree. The measured data corresponding to the second position, that is, the mapped measured data determined by using the current coincidence degree corresponds to the simulated moving state of the mobile robot, which represents the measured data at the first position according to the corresponding The measurement data at the second location inferred from the movement state of .

At the same time, the image capturing device also calculates the moving state of the mobile robot in the set coordinate system (that is, in the set coordinate system) by comparing the landmark image data corresponding to the common landmark feature in the image data captured at the two positions. amount of movement and rotation below). Wherein, the movement state obtained by using the image data and the movement state obtained by the measurement data correspond to the assembly parameters between the image capturing device and the ranging sensing device.

Using this correspondence, the error between the two moving states is adjusted iteratively so that the adjusted error meets an error condition, and the two moving states are used to determine the landmark position data of the landmark feature in the physical coordinate system, and/or the first Positioning position data of a position and a second position in a physical coordinate system. When the coincident measurement data is used as the landmark measurement data corresponding to the landmark feature, the landmark position data of each landmark measurement data in the physical coordinate system can also be determined.

In another embodiment, the degree of coincidence between the measurement data of the two positions is adjusted, and the movement state obtained under the corresponding degree of coincidence determines the degree of coincidence of the landmark image data in the image data; When the degree of coincidence between the image data and the adjusted image data meets the coincidence condition, use the corresponding movement state to determine that the corresponding landmark feature in the image data is mapped to the landmark position data in the physical coordinate system, and/or Positioning position data corresponding to the at least two positions in the physical coordinate system.

Still taking the image data and measurement data obtained by the mobile robot synchronously at the first position and the second position as an example, the measurement data obtained when the mobile robot is at the first position is mapped to the second position, and obtained by calculating the coincidence degree. The simulated measurement data corresponding to the second position, that is, the mapped measurement data determined by using the current coincidence degree corresponds to the simulated movement state of the mobile robot, which represents the measurement data at the first position according to The measurement data at the second location inferred from the corresponding movement state.

The moving state obtained by using the image data and the moving state obtained by the measurement data are corresponding to the assembly parameters between the image capturing device and the ranging sensing device. The degree of coincidence of the matched landmark image data in the image data.

Evaluate the coincidence degree obtained by using the measurement data and the coincidence degree obtained by using the image data to determine whether the evaluation structure meets the coincidence conditions. , if not, continue to adjust the coincidence of the measured data until the coincidence conditions are met.

Through the optimization method in this embodiment, the errors of the image capturing device and the ranging sensing device can be jointly optimized, so as to utilize the advantages of each sensor while avoiding the accumulation of errors and improving the accuracy of composition.

In an exemplary embodiment, the range-measuring sensing devices include multiple types, and each range-measuring sensing device synchronously acquires its own measurement data according to its own measurement method, so as to analyze the movement state. For example, the ranging sensing device includes any two or more of the following: a laser ranging sensing device, a binocular camera device, a ToF ranging sensing device, a structured light ranging sensing device, and the like.

Specifically, when the mobile robot has multiple ranging sensing devices, the respective measurement data can be obtained synchronously based on the respective measurement methods of the ranging sensing devices, and the measurement data of the ranging sensing devices can be integrated to determine the movement of the mobile robot. state.

In one embodiment, the measurement data from each ranging sensing device belonging to the same position can be fused, and then the fused measurement data corresponding to each position can be optimized to obtain the measurement data that is consistent with each measurement data. The landmark feature under the coincidence condition between them is mapped to the position data in the physical coordinate system, and/or the position data corresponding to the at least two positions in the physical coordinate system.

Here, each ranging sensing device synchronously acquires measurement data at each position according to their respective measurement methods, and after obtaining the measurement data from each ranging sensing device, the measurement data from each ranging sensing device belonging to the same position The measurement data is fused to perform optimization processing based on the fused measurement data. For example, when the ranging sensing device includes a binocular camera device and a laser sensor, at the first position, the binocular camera device uses the triangulation principle to determine the characteristics of the mobile robot relative to each target according to the images captured by its two cameras. At the same time, the laser sensor also determines the distance relative to the landmark feature at the first position according to the time difference between the time it emits the laser beam and the time it receives the laser beam to obtain the laser beam. The first measurement data acquired by the sensor. At the second position, the binocular camera device uses the triangulation principle to determine the distance of the mobile robot relative to each landmark feature according to the images captured by its two cameras to obtain the second measurement data obtained by the binocular camera device; at the same time, The laser sensor also determines its distance from the landmark feature at the second location based on the time difference between the time it emits the laser beam and the time it receives the laser beam to obtain second measurement data acquired by the laser sensor. After the fusion processing of each first measurement data and the fusion processing of each second measurement data, the coincidence degree optimization process is performed based on each first measurement data after fusion and the second measurement data after fusion; similar to that mentioned in the foregoing example various ways to obtain each position data in the physical coordinate system, which will not be described in detail here.

The fusion processing includes: performing landmark analysis on measurement data provided by different ranging sensing devices to obtain landmark measurement data corresponding to common landmark features. For example, the measurement data obtained by each ranging sensing device has a distance relative to the same landmark feature, and the distances detected by each ranging sensing device relative to the same landmark feature are processed by means of averaging or median. The fusion processing further includes: performing interpolation, averaging and other processing on the measurement data provided by different ranging sensing devices according to the respective measured direction angles. For example, the measurement value of a certain voxel position measured by the binocular camera device and the measurement data of the corresponding voxel position measured by the single-point laser ranging sensing device are averaged. For another example, the ranging value D1 at a certain direction angle measured by the single-point laser ranging sensing device corresponds to the direction between the measured values D21 and D22 of two voxel positions measured by the binocular camera device. angle, the distance measurement value D1 and the measurement values D21 and D22 are respectively averaged to obtain new measurement values D21' and D22' corresponding to the two voxel positions, and replace the measurement values D21 and D22. For another example, the ranging value D1 at a certain direction angle measured by the single-point laser ranging sensing device corresponds to the direction between the measured values D21 and D22 of two voxel positions measured by the binocular camera device. angle, then add a column C21 between the columns C1 and C2 where the measured values D21 and D22 corresponding to the two voxel positions in the two-dimensional distance lattice data measured by the binocular camera device are located, and the other columns on the added column C21 The value of the voxel position can be interpolated according to the mean of the measured values of the rows where columns C1 and C2 are located.

In another embodiment, the coincidence degree optimization process may be performed on the measurement data obtained from different ranging sensing devices obtained at different positions, and the obtained results are obtained under the condition that the coincidence degrees between the measurement data meet the coincidence conditions. The landmark feature is mapped to the position data in the physical coordinate system, and/or the position data corresponding to the at least two positions in the physical coordinate system.

Here, each ranging sensing device synchronously acquires measurement data at each position according to their respective measurement methods, and after obtaining the measurement data from each ranging sensing device, analyzes the moving state of the mobile robot according to the respective measurement data, The coincidence degree of the measurement data obtained by each ranging sensing device at different positions is optimized respectively to obtain each moving state under the coincidence condition. Perform data processing such as average value and median value on each moving state to obtain a moving state that can be used to map the landmark feature data in the image data to the physical coordinate system. The location data under the system will not be described in detail here.

Please continue to refer to FIG. 1 , in step S130 , the obtained location data of each landmark and positioning location data are recorded in the map data constructed based on the physical coordinate system.

Here, after performing steps S110 and S120, it is determined that the landmark features in each image data are mapped to the landmark position data in the physical coordinate system and/or the positioning position data corresponding to at least two positions in the physical coordinate system, and the obtained The landmark location data and/or positioning location data are recorded in the map data constructed based on the physical coordinate system, thereby obtaining a map of the current physical space, and the map includes each location the mobile robot passes through, as well as each landmark feature s position. Here, the map data is data stored in a storage medium through a database, which can be displayed to the user through grids/unit vectors or the like.

In some cases, each image data captured by the image pickup device at each location is also stored in the map for relocation. The relocation method includes: after the map data of the current physical space is constructed for the first time, the mobile robot can pass the image data captured by the image capturing device at a certain position with the map during the working process of the current physical space. Each image data stored in the data is compared, so as to determine its own position in the physical space through the comparison result.

In other cases, the map also stores historical measurement data (eg, point cloud data) acquired by the ranging sensing device at each location for relocation. The relocation method includes: after the map data of the current physical space is constructed for the first time, the mobile robot can pass the image data captured by the image capturing device at a certain position with the map during the working process of the current physical space. Compare the image data stored in the data, and compare the measurement data obtained by the ranging sensing device at a certain position with the measurement data stored in the map data, so as to determine the physical presence of the image data through the comparison results. position in space.

Among them, the comparison method of measurement data includes but is not limited to iterative Closest Point (Iterative Closest Point, ICP) algorithm. The way of spatial transformation corresponds each point in the two sets of point sets one by one, so as to judge the similarity of the two sets of point sets.

In an exemplary embodiment, when the ranging sensing device includes a binocular camera device, it also includes data corresponding to each positioning position of the mobile robot in the physical coordinate system, where the binocular camera device is located. The procedure for storing each captured image in the map data.

Here, each image captured by the binocular camera device at each position is also stored in the map for relocation.

Wherein, the relocation method includes: after the map data of the current physical space is constructed for the first time, the mobile robot can pass the image captured by the binocular camera device at a certain position during the working process of the current physical space. Compare with each image stored in the map data, so as to determine its own position in the physical space through the comparison result.

In some cases, each image data captured by the image pickup device at each location and each image captured by the binocular camera device are simultaneously stored in the map. After the map data of the current physical space is constructed for the first time, during the working process of the current physical space, the mobile robot can compare the image data captured by the image capturing device at a certain position with each image data stored in the map data. Comparing the image captured by the binocular camera device with each image stored in the map data, so as to determine its own position in the physical space through the comparison result.

In an exemplary embodiment, the method for constructing a map further includes the step of determining an environment boundary in the map data according to the measurement data measured by the range-finding sensing device. Wherein, examples of the environmental boundary include, but are not limited to, space dividers such as doors, walls, and screens.

Please refer to FIG. 2a and FIG. 2b, which show a brief schematic diagram of the map constructed in the present application in one embodiment. As shown in the figure, during the movement of the mobile robot in the current physical space, the position and distance obtained by the ranging sensing device relative to each landmark feature are marked in the map data. Since the space divider is a continuous and uninterrupted object in space, in the process of sensing it by the ranging sensing device, the measurement data also presents continuous or dense lines based on the type of the ranging sensing device, thus forming a pattern as shown in Figure 2a and the environmental boundary 11 shown in Figure 2b. Among them, in Figure 2a and Figure 2b, the horizontal axis and the vertical axis represent the X axis and the Y axis in the physical coordinate system, respectively, the discrete point (0,0) in Figure 2a represents the initial position of the mobile robot, and the Other discrete points are the locations the mobile robot passes through during the movement and other landmark features in the physical environment.

It should be noted that, although the above embodiment takes the mobile robot to acquire image data and measurement data synchronously at two positions as an example, in practical applications, it can also be three positions, four positions, etc., I won't go into details here.

Aiming at the problem that the mobile robot is not easy to locate in the physical space with the characteristics of a large area and a certain similarity of the decoration environment, the present application obtains the measurement data reflecting the high complexity of the mobile robot's body-side environment and other environments by synchronously acquiring and including the mobile robot. Image data with lower environmental complexity, such as the above, realizes a wider environmental perception in physical space; and the method of constructing a map using measurement data and image data solves the problem of low accuracy in the map constructed by using any of these data. , Landmark density low difference and other issues that help to accurately locate.

An embodiment of the second aspect of the present application provides a positioning method to jointly optimize the error through data provided by the image capturing device and the ranging sensing device, respectively, to determine the position of the mobile robot in the physical space.

The positioning method may be executed by a processor in the mobile robot, or may be executed by a server communicating with the mobile robot.

Wherein, map data of the current physical space may be pre-stored in the mobile robot or the server, and the current position of the mobile robot in the physical space may be determined based on the map data. Alternatively, when there is no pre-stored map data, the current position of the mobile robot in the physical space can also be determined based on the data in the physical space obtained by the mobile robot at the current position and the data in the physical space obtained at the previous position, and Build map data of the current physical space while moving.

Please refer to FIG. 5 , which is a schematic diagram of an embodiment of the positioning method of the mobile robot in the present application. As shown in the figure, in step S210, the image capturing device and the ranging sensing device are controlled to obtain image data and measurement data simultaneously at least at the first position and the second position of the mobile robot, respectively; wherein, the first position is mapped on the map The first positioning position data in the data.

The first location map corresponds to the first positioning location data in the map data. Wherein, the first position may be a position passed by the mobile robot in the current moving operation. Alternatively, when the map data of the current physical space is pre-stored, the first position may also be a certain position stored in the map data by the mobile robot during historical movement operations.

In some cases, the external synchronization signal may also be generated based on signals in the inertial navigation sensor of the mobile robot. The inertial navigation sensor (IMU) is used to acquire inertial navigation data of the mobile robot. The inertial navigation sensor includes, but is not limited to, one or more of a gyroscope, an odometer, an optical flow meter, and an accelerometer. The inertial navigation data acquired by the inertial navigation sensor includes, but is not limited to, one or more of the speed data, acceleration data, moving distance, rolling circles of the roller, and deflection angle of the roller, etc. of the mobile robot.

For ease of description, in the following embodiments, how the mobile robot is positioned at two positions (ie, the first position and the second position) will be taken as an example.

Please continue to refer to FIG. 5, in step S220, at least use the measurement data respectively measured at the first position and the second position to analyze the moving state of the mobile robot in the physical space to determine when the mobile robot is in a position relative to the first position. When the second location is the second location, the second location is the second positioning location data in the map data.

Here, both the measurement data and the image data acquired in step S210 include data helpful for positioning within different spatial height ranges in the physical space where the mobile robot is located, such as landmark data, distances/angles relative to landmark features, and the like. In addition, since the ranging sensing device and the image capturing device move as a whole with the mobile robot, at least the movement state determined by the measurement data obtained at different positions can solve the problem that the landmark features reflected in the image data have no physical scale. The location of landmarks in the coordinate system (such as scale), and only using measurement data is not conducive to obtaining high-precision positioning. An example of the set coordinate system includes a camera coordinate system, or a virtual coordinate system that lacks a mapping relationship determined based on a physical scale with the physical coordinate system. Thus, the moving state of the mobile robot described by the measurement data and the landmark data corresponding to the landmark features in the image data are used to calculate the landmark position data mapped from the landmark features captured by the image data to the physical coordinate system.

In the process that the mobile robot presets the initial movement position as the starting point for positioning, when the mobile robot knows the positioning position information of the previous position in the physical coordinate system, the movement between different positions before and after the above example is used. state, to obtain positioning position data corresponding to the at least two positions in the physical coordinate system. It should be understood that, since the measurement data acquired by the distance-measuring sensing device is data with actual physical units, the calculated movement state also has actual physical units.

In some other specific examples, the mobile robot obtains the moving state by calculating the coincidence degree of each measurement data obtained at different positions. Among them, the single measurement data provides the positional relationship between the entity part in the corresponding measurement plane and the mobile robot in the physical space. When the mobile robot measures the same entity at different positions, it reflects the changes caused by the pose change of the mobile robot. A change in the measurement position of the same entity in the measurement data. For example, a mobile robot can rotate, translate, and zoom the measurement data obtained at one position to obtain measurement data that coincides with some measurement points in the measurement data at another position. The processing process reflects the process of simulating a mobile robot to perform a moving operation in order to make the corresponding measurement points coincide. To this end, by performing the coincidence optimization operation, the mobile robot can simulate the data and physical scale of the mobile robot's pose change between different positions and other moving states, and use the pose change data and other moving states, and the coincident The physical quantity provided by the measurement data is used to determine other data in the moving state, such as the second positioning position data in which the second position is mapped in the map data.

Since the measurement data includes the physical quantity of the fixed object and the physical quantity of the corresponding moving object measured by the ranging sensing device at the corresponding position, for this reason, in an exemplary embodiment, the mobile robot uses the measurement data obtained at different positions The degree of coincidence between them is optimized, so as to obtain the second positioning position data corresponding to the second position in the map data under the coincidence condition between the measurement data. Wherein, the mobile robot may regard the overlapping measurement data part as the landmark data corresponding to the landmark feature. An example of the coincidence condition includes that the number of iterative optimizations of the coincidence degree reaches a preset threshold of the number of iterations, and/or the gradient value of the coincidence degree is smaller than a preset gradient threshold value, and the like. This method is beneficial to calculate the movement state by reflecting the physical quantity of the fixed object in each measurement data as much as possible.

Taking the measurement data as one-dimensional data as an example, please refer to FIG. 3 , which is a schematic diagram of an embodiment of the measurement data obtained by the mobile robot in the present application at two positions respectively. As shown in the figure, the mobile robot obtains measurement data MData_1 and MData_2 at the first position and the second position, respectively, wherein if there is an entity with a fixed position in the physical space, the measurement data MData_1 and MData_2 have corresponding and available entities in the physical space. Measurement points that are coincident after data processing. From the measurement points of entity 311, entity 312 and entity 313 in the measurement data MData_1 and MData_2 in the figure, it can be seen that both entity 311 and entity 313 in the two measurement data can be overlapped after data processing, while entity 312 cannot Coincidence with entity 311 and entity 313 at the same time, so entity 311 and entity 313 can be regarded as entities with fixed positions. The data processing of the two measurement data MData_1 and MData_2 by the mobile robot, including rotation, translation, zoom, etc., reflects the relative positional relationship between the first position and the second position of the mobile robot. To this end, the mobile robot uses an optimization function constructed according to the two measurement data MData_1 and MData_2 to perform an optimal calculation of the coincidence degree until the coincidence degree of the two measurement data MData_1 and MData_2 meets the preset coincidence condition. Thereby, the pose change data reflecting the relative position relationship is obtained, and under the preset coincidence condition, the coincident measurement data points in the two measurement data are used as the landmark measurement data, and based on the obtained pose change data, we obtain The second positioning position data corresponding to the second position in the map data under the condition of coincidence between the measurement data. It should be noted that the above example is also applicable to the calculation processing in which the measurement data is two-dimensional data.

In another embodiment, when the map data of the current physical space is pre-stored, and the first position corresponds to the first positioning position data in the map data, the movement state of the second position relative to the first position can be used to determine the first position based on the first position. A positioning position data is calculated, and the second positioning position data in which the second position is mapped in the map data is calculated. For example, in the historical working process of the mobile robot, after the map data of the current physical space is obtained through the method for constructing a map described in the embodiment of the first aspect of the present application (ie, the embodiment parts corresponding to FIG. 1 to FIG. 5 ), In the current moving operation, the mobile robot is located at the second position, and the second position has a common landmark feature with the first position, so the movement state of the second position relative to the first position can be determined based on the common landmark feature, And based on the first positioning position data corresponding to the first position and the moving state, the second positioning position data corresponding to the second position is determined.

Of course, in some cases, since the mobile robot has acquired and saved image data and measurement data corresponding to each location at multiple locations in the physical space when constructing the map data. Therefore, the second position where the mobile robot is currently located may be the same position or very close to the corresponding first position in the map data. At this time, by comparing the image data and measurement data obtained at the second position with the image data and measurement data corresponding to the first position saved in the map data, the first positioning position data corresponding to the first position can be determined as The second positioning position data corresponding to the second position.

In order to improve the calculation efficiency, in another exemplary embodiment, the mobile robot optimizes the coincidence degree between the landmark measurement data in the measurement data obtained at different positions, so as to obtain a coincidence condition that meets the coincidence conditions between the measurement data. Move the robot down to move the state.

Wherein, the optimization process is similar to the above example of one-dimensional data. The difference is that, in an exemplary embodiment, the positioning method further includes the step of extracting landmark features in each measurement data, so as to use the landmark measurement data corresponding to the landmark features in each measurement data to determine the movement state. analysis. Wherein, the landmark measurement data includes landmark measurement data corresponding to moving objects and landmark measurement data corresponding to fixed objects. For example, the measurement data obtained by the ranging sensing device includes its distance relative to each feature in the physical space. It includes some static features (such as flower pots, coffee tables, shelves, etc.) and some moving features (such as people, pets, shopping carts in progress, etc.).

It should be understood that in some scenarios, some features in the physical space are not suitable to be used as landmark features, for example, movable objects such as people and pets in the physical space. If these inappropriate features are used as landmark features, it will lead to large errors. Therefore, it is necessary to filter each data in the measurement data to determine which ones can be used as landmark features, so as to use these extracted landmark features to analyze the moving state and improve the calculation accuracy.

In another embodiment, landmark features may also be extracted directly based on numerical changes in measurement data acquired at each location. Specifically, in the measurement data obtained at different positions, the changes of static features in each position data are relatively regular, while the changes of moving features are relatively different from those of other static features. Therefore, a predetermined change threshold can be used to find a feature whose numerical value changes relatively regularly in the measurement data relative to the distance of each feature, and use it as a landmark feature. For example, the measurement data obtained at the first position and the measurement data obtained at the second position have five common features a, b, c, d, and e. At the first position, the distance of feature a relative to the mobile robot is 2m, the distance of feature b relative to the mobile robot is 3.4m, the distance of feature c relative to the mobile robot is 4.2m, and the distance of feature d relative to the mobile robot is 2.8m, the distance of the e feature relative to the mobile robot is 5m; at the second position, the distance of the a feature relative to the mobile robot is 2.5m, the distance of the b feature relative to the mobile robot is 3.8m, and the distance of the c feature relative to the mobile robot is 3.8m. The distance of the mobile robot is 4.9m, the distance of the d feature relative to the mobile robot is 3.6m, and the distance of the e feature relative to the mobile robot is 2m. It can be seen that the variation of a feature at two positions is 0.5m, the variation of b feature at two positions is 0.4m, the variation of c feature at two positions is 0.7m, and the variation of d feature at two positions is 0.7m. The variation at each position is 0.8m, and the variation of the e-feature at two positions is 3m. Assuming that the preset change threshold is 0.5m, a, b, c, and d can be extracted as landmark features, while e may be an active feature and cannot be used as landmark features.

The process of optimizing the coincidence degree mentioned in any of the above embodiments can be exemplified as follows: please refer to FIG. 4 , which shows the measurement data obtained by the mobile robot in the present application at two positions respectively in another embodiment. Schematic. As shown in the figure, the measurement data MData_3 and MData_4 obtained by the mobile robot at the first position and the second position, wherein the circle represents the matching landmark measurement data in the two measurement data, and the star represents the two measurement data in one measurement data. Coincidence calculates the coincident landmark measurement data. The mobile robot iteratively optimizes the coincidental landmark measurement data so that the number of landmark measurement data represented by the star in previous coincidence calculations reaches the coincidence condition, and determines the movement state obtained in the corresponding times of coincidence calculation; Using the obtained movement state and the coincident landmark measurement data, the second positioning position data corresponding to the second position in the map data is obtained under the coincidence condition between the measurement data.

Using the movement state obtained in any of the above examples, the mobile robot also determines the landmark position information in the physical coordinate system of the ground surface feature corresponding to the landmark image data in the synchronously obtained image data.

The mobile robot converts the coordinates of each position marked in the set coordinate system to the physical coordinate system by using the movement state with physical units calculated by relying on the measurement data, so as to obtain the position that meets the coincidence conditions between the measurement data. second positioning position data corresponding to the second position in the map data, etc. Wherein, since the ranging sensing device and the image capturing device move with the mobile robot as a whole, the moving state obtained by the mobile robot using the measurement data reflects the movement of the mobile robot relative to the landmark feature corresponding to the landmark image data in the image data. Here, based on the mapping relationship between the set coordinate system and the physical coordinate system determined by the moving state, the mobile robot maps the landmark image features in the set coordinate system to the physical coordinate system, so as to obtain The corresponding landmark features are mapped to the landmark location data in the physical coordinate system.

The mobile robot performs optimization processing on the coincidence degree between the measurement data obtained at different positions, and obtains the second positioning position corresponding to the second position in the map data under the coincidence conditions between the measurement data. The data is used as an example to describe the execution process of obtaining the second positioning position data in the physical coordinate system:

The measurement data obtained by the mobile robot at the first position and the second position respectively are converted into the coordinate system of the same measurement data, and the iterative calculation of the coincidence degree is performed to obtain the coincidence of the two measurement data in the corresponding coordinate system. meet the coincidence conditions. Thereby, the moving state of the mobile robot under the coincidence condition is obtained. Wherein, the movement state includes the physical scale between the relative positional relationship between the first position and the second position of the mobile robot and the measured positional relationship between the coincident landmark measurement data, the relative position of the mobile robot in the second position relative to the first position. Position and pose change data, etc. Wherein, under the action of the overall movement of the mobile robot, the physical scale also corresponds to the conversion relationship between the set coordinate system and the physical coordinate system (ie, the map scale). For example, the conversion relationship between the set coordinate system and the physical coordinate system is determined based on the physical scale and internal and external parameters of the image pickup device, wherein the internal and external parameters include the internal optical parameters and assembly parameters of the image pickup device, and the image Assembly parameters between the capture device and the ranging sensing device, etc.

After obtaining the conversion relationship between the set coordinate system and the physical coordinate system, the mobile robot calculates the positioning position data corresponding to each position of the mobile robot in the set coordinate system in the physical coordinate system.

It should be added that the coincidence condition may be a preset coincidence degree condition, or select a local optimal coincidence degree, which includes but is not limited to: the average coincidence degree of the common landmark feature in the two measurement data is the highest, or the two measurement data The standard deviation of the coincidence degree of each common landmark feature is the smallest, and the gradient of the coincidence degree is locally minimal, etc.

The following will use a specific example to describe how to optimize the degree of coincidence between the measurement data obtained at different locations, so as to obtain the second positioning location data of the second location in the map data under the coincidence condition. It should be noted that this example is only used to explain the present application and not to limit the present application.

Assuming that the positions of the image capturing device itself in the current physical space at two moments (time 1 and time 2) before and after the set coordinate system are _Twc1 and _Twc2 , and the position of the ranging sensing device in the set coordinate system is T _wb1 and T _wb2 . Using the installation parameters between the image capturing device and the ranging sensing device, that is, the position T _bc of the ranging sensing device relative to the image capturing device, it can be obtained: T _wc1 =T _wb1 T _bc , T _wc2 =T _wb2 T _bc .

Assuming that the scaling factor of the conversion relationship between the distance unit of the set coordinate system and the actual physical coordinate system is s, the coordinates of T _wc1 in the set coordinate system (in the form of a matrix) are:

Then the coordinates in the real physical coordinate system are:

The coordinates of T _wc2 in the set coordinate system are:

Then the coordinates in the real physical coordinate system are:

and

The value of s in . Therefore, by substituting s, the second positioning position data of the second position in the map data can be determined.

In an exemplary embodiment, the present application considers the error in the image pickup device to jointly optimize the error of the image pickup device and the ranging sensing device, thereby improving the accuracy of positioning.

Here, a joint optimization process is performed on the degree of coincidence between the measurement data acquired at different positions and the degree of coincidence between the acquired image data, so as to obtain that the second position is in the The second positioning position data in the map data.

In one embodiment, each measurement data and each image data are adjusted respectively to optimize the degree of coincidence of each measurement data and the degree of coincidence of each image data; when the errors between the moving states obtained based on the respective degrees of coincidence conform to the error When conditions are met, the corresponding movement state is used to determine the second positioning position data corresponding to the second position in the map data. An example of the error condition includes a preset error threshold, or a minimum error value selected based on a preset number of adjustments.

The degree of coincidence obtained by using the measurement data and the degree of coincidence obtained by using the image data are evaluated and processed to determine whether the evaluation structure meets the coincidence conditions. If so, map each position to the physical coordinate system with the corresponding moving state. , continue to adjust the coincidence of the measured data until the coincidence conditions are met.

Through the optimization method in this embodiment, the errors of the image capturing device and the ranging sensing device can be jointly optimized, so that the advantages of each sensor are used while the accumulation of errors is avoided, and the positioning accuracy is improved.

In one embodiment, the measurement data from each ranging sensing device belonging to the same position can be fused, and then the fused measurement data corresponding to each position can be optimized to obtain the measurement data that is consistent with each measurement data. The second positioning position data in the map data under the coincidence condition between the second position data.

Here, each ranging sensing device synchronously acquires measurement data at each position according to their respective measurement methods, and after obtaining the measurement data from each ranging sensing device, the measurement data from each ranging sensing device belonging to the same position The measurement data is fused to perform optimization processing based on the fused measurement data. For example, when the ranging sensing device includes a binocular camera device and a laser sensor, at the first position, the binocular camera device uses the triangulation principle to determine the characteristics of the mobile robot relative to each target according to the images captured by its two cameras. At the same time, the laser sensor also determines the distance relative to the landmark feature at the first position according to the time difference between the time it emits the laser beam and the time it receives the laser beam to obtain the laser beam. The first measurement data acquired by the sensor. At the second position, the binocular camera device uses the triangulation principle to determine the distance of the mobile robot relative to each landmark feature according to the images captured by its two cameras to obtain the second measurement data obtained by the binocular camera device; at the same time, The laser sensor also determines its distance from the landmark feature at the second location based on the time difference between the time it emits the laser beam and the time it receives the laser beam to obtain second measurement data acquired by the laser sensor. After the fusion processing of each first measurement data and the fusion processing of each second measurement data, the coincidence degree optimization processing is performed based on each first measurement data after fusion and the second measurement data after fusion; and various ways to obtain each position data in the physical coordinate system, which will not be described in detail here.

The fusion processing includes: performing landmark analysis on measurement data provided by different ranging sensing devices to obtain landmark measurement data corresponding to common landmark features. For example, the measurement data obtained by each ranging sensing device has a distance relative to the same landmark feature, and the distances detected by each ranging sensing device relative to the same landmark feature are processed by means of averaging or median. The fusion processing further includes: performing interpolation, averaging and other processing on the measurement data provided by different ranging sensing devices according to the respective measured direction angles. For example, the measurement value of a certain voxel position measured by the binocular camera device and the measurement data of the corresponding voxel position measured by the single-point laser ranging sensing device are averaged. For another example, the ranging value D1 at a certain direction angle measured by the single-point laser ranging sensing device corresponds to the direction between the measured values D21 and D22 of two voxel positions measured by the binocular camera device. angle, the distance measurement value D1 and the measurement values D21 and D22 are respectively averaged to obtain new measurement values D21' and D22' corresponding to the two voxel positions, and replace the measurement values D21 and D22. For another example, the ranging value D1 at a certain direction angle measured by the single-point laser ranging sensing device corresponds to the direction between the measured values D21 and D22 of two voxel positions measured by the binocular camera device. angle, then add a column C21 between the columns C1 and C2 where the measured values D21 and D22 corresponding to the two voxel positions in the two-dimensional distance lattice data measured by the binocular camera device are located, and the other columns on the added column C21 The value of the voxel position can be interpolated according to the mean value of the measured values of the rows where columns C1 and C2 are located.

In another embodiment, the coincidence degree optimization process may be performed on the measurement data obtained from different ranging sensing devices obtained at different positions, and the obtained results are obtained under the condition that the coincidence degrees between the measurement data meet the coincidence conditions. second positioning position data of the second position in the map data.

Here, each ranging sensing device synchronously acquires measurement data at each position according to their respective measurement methods, and after obtaining the measurement data from each ranging sensing device, analyzes the moving state of the mobile robot according to the respective measurement data, The coincidence degree of the measurement data obtained by each ranging sensing device at different positions is optimized respectively to obtain each moving state under the coincidence condition. Perform data processing such as average value and median value on each moving state to obtain a moving state that can be used to map the landmark feature data in the image data to the physical coordinate system. The location data under the system will not be repeated here.

Aiming at the problem that the mobile robot is not easy to locate in the physical space with the characteristics of a large area and a certain similarity of the decoration environment, the present application obtains the measurement data reflecting the high complexity of the mobile robot's body-side environment and other environments by synchronously acquiring and including the mobile robot. Image data with lower environmental complexity, such as above, realizes wider environmental perception in physical space; and the method of positioning using measurement data and image data solves the problem of using either of these data positioning accuracy is low, which is helpful for accurate positioning. problems such as low density of landmarks.

The present application also provides a server.

The server includes, but is not limited to, a single server, a server cluster, a distributed server cluster, a cloud server, and the like. Here, according to the actual design, the server is provided by a cloud server provided by a cloud provider. Wherein, the cloud server includes a public cloud (Public Cloud) server and a private cloud (Private Cloud) server, wherein the public or private cloud server includes Software-as-a-Service (Software as a Service, SaaS ), Platform-as-a-Service (Platform as a Service, PaaS) and Infrastructure-as-a-Service (Infrastructure as a Service, IaaS), etc. The private cloud service end is, for example, Alibaba cloud computing service platform, Amazon (Amazon) cloud computing service platform, Baidu cloud computing platform, Tencent cloud computing platform and so on.

The server can accept image data and measurement data from the mobile robot, so as to construct a map of the current physical space or obtain the position of the mobile robot in the physical coordinate system based on the acquired image data and measurement data.

The server is connected in communication with a mobile robot located in a physical space. Wherein, the physical space refers to the physical space provided for the mobile robot to navigate and move, and the physical space includes but is not limited to any of the following: indoor/outdoor space, road space, flight space, etc. For example, in some embodiments, the mobile robot is a drone, and the physical space corresponds to a flight space; in other embodiments, the mobile robot is a vehicle with an automatic driving function, then the physical space Corresponds to a tunnel road where positioning cannot be obtained or a road space where the network signal is weak but needs to be navigated; in still other embodiments, the mobile robot is a sweeping robot, and the physical space corresponds to an indoor or outdoor space. The mobile robot is equipped with sensing devices such as a camera device, a movement sensing device, and the like, which provide navigation data for autonomous movement.

Please refer to FIG. 6 , which is a schematic diagram of the server in an embodiment of the present application. The server 40 includes an interface device 41 , a storage device 42 , and a processing device 43 . The storage device 42 includes a non-volatile memory, a storage server, and the like. Wherein, the non-volatile memory is, for example, a solid state disk or a U disk. The storage server is used for storing various obtained power consumption related information and power supply related information. The interface device 41 includes a network interface, a data line interface, and the like. The network interfaces include but are not limited to: Ethernet network interface devices, network interface devices based on mobile networks (3G, 4G, 5G, etc.), network interface devices based on short-range communication (WiFi, Bluetooth, etc.), and the like. The data line interface includes but is not limited to: USB interface, RS232, RS485, etc. The interface device is data-connected with the control system, a third-party system, the Internet, and the like. The processing device 43 is connected to the interface device 41 and the storage device 42, and includes at least one of a CPU or a chip integrated with the CPU, a programmable logic device (FPGA) and a multi-core processor. The processing device 43 also includes memory, registers, etc., for temporarily storing data.

The interface device 41 is used for data communication with a mobile robot located in a physical space.

The storage device 42 is used to store at least one program. Here, the storage device 42 includes, for example, a hard disk disposed on the server and stores the at least one program.

The processing device 43 is configured to call the at least one program to coordinate the interface device and the storage device to execute the method for constructing a map mentioned in the example of the first aspect, or to execute the method mentioned in the example of the second aspect. positioning method. The method for constructing a map is shown in FIG. 1 and the corresponding description, and the positioning method is shown in FIG. 5 and the corresponding description, which will not be repeated here.

In addition, the map data obtained based on the method for constructing a map can be sent to the mobile robot through the interface device, and can also be stored on the server side, so that the map data can be used during positioning, or the server side can provide other devices with the information. Describe the map data. When the server needs to use the map data for positioning, the storage device can provide the map data to the processing device; when other devices need to call the map data, the storage device can provide the map data to the interface device, so that the interface device can map the map data to the interface device. data to other devices.

Here, a mobile robot is provided, please refer to FIG. 7 , which is a schematic diagram of a module structure of a mobile robot in an embodiment. As shown in the figure, the mobile robot 50 includes a storage device 54 , a moving device 53 , an image capturing device 51 , a distance sensing device 55 and a processing device 52 .

The image pickup device is used to collect image data. Wherein, the image capturing device is a device for providing a two-dimensional image according to a preset pixel resolution. In some embodiments, the image capturing device includes, but is not limited to, a camera, a video camera, a camera module integrated with an optical system or a CCD chip, a camera module integrated with an optical system and a CMOS chip, and the like. According to actual imaging requirements, lenses that can be used by the camera or video camera include, but are not limited to, standard lenses, telephoto lenses, fisheye lenses, and wide-angle lenses.

In some embodiments, the image capturing device may be disposed on a mobile robot with a main optical axis between a horizontal plane and a vertical ceiling direction, such as the main optical axis being located at 0° in a vertical ceiling direction and within the range of 0±90°. Taking the mobile robot as a commercial cleaning robot as an example, the image capturing device is assembled on the upper half of the commercial cleaning robot, and its main optical axis is a preset angle obliquely upward to obtain image data within a corresponding viewing angle range.

The ranging sensing device is used for acquiring measurement data, and the ranging sensing device can measure the distance of the landmark feature in the physical space where the mobile robot is located relative to the mobile robot.

The landmark feature refers to a feature that is easy to distinguish from other objects in the physical space where the mobile robot is located. For example, the landmark feature can be a table corner, a contour feature of a ceiling light on a ceiling, a space between a wall and the ground. connection etc. Examples of the ranging sensing device include ranging sensing devices that provide one-dimensional measurement data such as laser sensors and ultrasonic sensors, or include two-dimensional or binocular camera devices such as ToF sensors, multi-line lidars, millimeter-wave radars, or binocular cameras. The ranging sensing device for three-dimensional measurement data may also include the above two ranging sensing devices. For example, a laser sensor can determine its distance from a landmark feature based on the time difference between the time it emits a laser beam and the time it receives the laser beam; for another example, an ultrasonic sensor can determine movement based on the vibration signal that the sound wave it emits is bounced back by the landmark feature. The distance of the robot relative to the landmark feature; another example, the binocular camera device can use the triangulation principle to determine the distance of the mobile robot relative to the landmark feature according to the images captured by its two cameras; another example, ToF (Time of Flight, The infrared light projector of the time-of-flight) sensor projects infrared light outward, and the infrared light is reflected after encountering the measured object and received by the receiving module. The depth of the illuminated object is calculated by recording the time from infrared light emission to being received. information.

The range sensing device generates corresponding measurement data by detecting entities in the surrounding environment of the mobile robot. In order to prevent the mobile robot from abnormal movements such as collision and entanglement with the entities in the surrounding environment, or to prevent abnormal movements such as the mobile robot from falling, the ranging sensing device is assembled on the body side of the mobile robot. The ranging sensing device is used to detect the edge of the corresponding image data in the physical space or the environmental data in the area that cannot be covered by the image data. Taking the mobile robot as a commercial cleaning robot as an example, the ranging sensing device is arranged at a position between 10-80 cm from the ground on the side of the commercial cleaning robot.

The mobile device is configured to perform a movement operation based on a navigation route. The moving device includes components such as wheels arranged at the bottom of the mobile robot to drive the mobile robot to move.

The storage device is used to store at least one program. The storage device includes a nonvolatile memory, a storage server, and the like. Wherein, the non-volatile memory is, for example, a solid state disk or a U disk.

The processing device is connected with the storage device, the image capturing device, the distance measuring sensing device and the mobile device, and is used for calling and executing the at least one program to coordinate the storage device, the image capturing device, the distance measuring sensing device and the mobile device. The mobile device executes the method for building a map mentioned in the example of the aforementioned first aspect, or performs the positioning method mentioned in the example for the aforementioned second aspect. The method for constructing a map is shown in FIG. 1 and the corresponding description, and the positioning method is shown in FIG. 5 and the corresponding description, which will not be repeated here.

Furthermore, map data obtained based on the method of constructing a map may be stored in a storage device so that the map data can be used in positioning. The storage device may provide the map data to the processing device when the map data needs to be used for positioning.

In addition, the mobile robot may further include an interface device, so that map data can be provided to other devices or the acquired image data and measurement data can be sent to other devices through the interface device.

Wherein, the interface device includes a network interface, a data line interface, and the like. The network interfaces include but are not limited to: Ethernet network interface devices, network interface devices based on mobile networks (3G, 4G, 5G, etc.), network interface devices based on short-range communication (WiFi, Bluetooth, etc.), and the like. The data line interface includes but is not limited to: USB interface, RS232, RS485, etc. The interface device is data-connected with the control system, a third-party system, the Internet, and the like.

The mobile robot in the present application is installed with an image capturing device and a ranging sensing device, so that the advantages and disadvantages of each sensor are combined, and the accuracy and reliability of composition or positioning are improved by means of multi-sensor fusion.

The present application also provides a computer readable and writable storage medium, which stores a computer program. When the computer program is executed, the device where the storage medium is located implements at least one embodiment described above for the method for constructing a map, such as FIG. 1 . The corresponding embodiment described in any one of the embodiments; or when the computer program is executed, the device where the storage medium is located implements at least one of the embodiments described above for the positioning method for a mobile robot, such as the one corresponding to FIG. 5 . Any of the described embodiments in various implementations.

The functions, if implemented in the form of software functional units and sold or used as independent products, may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be embodied in the form of a software product in essence, or the part that contributes to the prior art or the part of the technical solution. The computer software product is stored in a storage medium, including Several instructions are used to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application.

In the embodiments provided in this application, the computer readable and writable storage medium may include read-only memory, random access memory, EEPROM, CD-ROM or other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices, flash memory, A USB stick, a removable hard disk, or any other medium that can be used to store the desired program code in the form of instructions or data structures and that can be accessed by a computer. Also, any connection is properly termed a computer-readable medium. For example, if the instructions are sent from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave Coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of the medium. It should be understood, however, that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transitory media, but are instead intended to be non-transitory, tangible storage media. Disk and disc, as used in the application, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and blu-ray disc, where disks typically reproduce data magnetically, while discs use lasers to optically reproduce data replicate the data.

In one or more exemplary aspects, the functions described by the computer programs of the methods described herein may be implemented in hardware, software, firmware, or any combination thereof. When implemented in software, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium. The steps of the methods or algorithms disclosed herein may be embodied in processor-executable software modules, where the processor-executable software modules may reside on a tangible, non-transitory computer readable and writable storage medium. Tangible, non-transitory computer-readable storage media can be any available media that can be accessed by a computer.

The flowcharts and block diagrams in the above-described figures of the present application illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. Based on this, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which contains one or more possible functions for implementing the specified logical function(s) Execute the instruction. It should also be noted that, in some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It is also noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented by dedicated hardware-based systems that perform the specified functions or operations , or can be implemented by a combination of dedicated hardware and computer instructions.

The above-mentioned embodiments merely illustrate the principles and effects of the present application, but are not intended to limit the present application. Anyone skilled in the art can make modifications or changes to the above embodiments without departing from the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by those with ordinary knowledge in the technical field without departing from the spirit and technical idea disclosed in this application should still be covered by the claims of this application.

Claims

A method for building a map for a mobile robot, comprising:

Control the image capturing device and the ranging sensing device to obtain image data and measurement data synchronously at different positions;

Use the measurement data obtained at different positions to analyze the moving state of the mobile robot in the physical space, so as to obtain the landmark feature in each image data mapped to the landmark position data in the physical coordinate system, and/or to obtain the different positions in the physical space. The corresponding positioning position data in the physical coordinate system;

The obtained landmark position data and/or positioning position data are recorded in map data constructed based on the physical coordinate system.
The method for constructing a map for a mobile robot according to claim 1, further comprising the step of determining an environment boundary in the map data according to the measurement data measured by the range-finding sensing device.
The method for building a map for a mobile robot according to claim 1, wherein when the ranging sensing device includes a binocular camera device, it further includes data corresponding to each positioning position of the mobile robot in the physical coordinate system , the step of saving each image captured by the binocular camera device in the map data.
The method for constructing a map by a mobile robot according to claim 1, wherein the measurement data obtained at different positions is used to analyze the moving state of the mobile robot in the physical space, so as to obtain the mapping of landmark features in each image data to the physical space. Landmark location data in the coordinate system, and/or positioning location data corresponding to the different locations in the physical coordinate system, including:

The degree of coincidence between the measurement data obtained at different positions is optimized, so that the landmark features in each image data that meet the coincidence conditions are mapped to the landmark position data under the physical coordinate system, and/or the landmark position data that meets the coincidence conditions are obtained. Positioning position data corresponding to the different positions under the conditions in the physical coordinate system.
The method for constructing a map by a mobile robot according to claim 1, wherein the measurement data obtained at different positions is used to analyze the moving state of the mobile robot in the physical space, so as to obtain the mapping of landmark features in each image data to the physical space. Landmark location data in the coordinate system, and/or positioning location data corresponding to the different locations in the physical coordinate system, including:

The degree of coincidence between the measurement data acquired at different locations and the degree of coincidence between the collected image data is jointly optimized, and the landmark features in each image data are mapped to physical objects under the conditions of coincidence that meet the joint optimization. Landmark location data in the coordinate system, and/or positioning location data corresponding to the different locations in the physical coordinate system.
The method for building a map for a mobile robot according to claim 1, 4 or 5, wherein the controlling the image capturing device and the ranging sensing device to obtain image data and measurement data synchronously at different positions, respectively, includes: controlling the image The capturing device, as well as a variety of ranging sensing devices, acquire image data and measurement data synchronously at different positions.
The method for constructing a map for a mobile robot according to claim 1, wherein the analyzing the movement state of the mobile robot in the physical space by using the measurement data obtained at different positions comprises: extracting the measurements obtained at different positions Landmark features in the data, and use the extracted landmark features to analyze the moving state of the mobile robot in the physical space.
The method for constructing a map by a mobile robot according to claim 1, wherein the controlling the image capturing device and the ranging sensing device to obtain image data and measurement data synchronously at different positions, respectively, comprises: controlling the image capturing device and the measuring device. The distance sensing device obtains image data and measurement data synchronously at different positions, wherein the synchronization signal is obtained from the outside or inside of the image capturing device and the distance measuring sensing device in the mobile robot.
The method for constructing a map for a mobile robot according to claim 8, wherein the synchronization signal is obtained from the outside of the image capturing device and the ranging sensing device in the mobile robot, comprising: the synchronization signal is obtained from the image capturing device in the mobile robot and the external inertial navigation sensor of the ranging sensing device.
The method for constructing a map for a mobile robot according to claim 1, wherein the viewing angle range of the image capturing device and the measurement range of the ranging sensing device partially overlap or do not overlap.
A method for positioning a mobile robot, comprising:

Control the image capturing device and the ranging sensing device to obtain image data and measurement data synchronously at a first position of the mobile robot and a second position different from the first position, respectively; wherein, the first position is mapped to the first position in the map data. a positioning location data;

The moving state of the mobile robot in the physical space is analyzed using the measurement data measured at the first position and the second position, respectively, to determine that when the mobile robot is at the second position, the second position is in the map data of the second positioning position data.
The method for positioning a mobile robot according to claim 11, wherein the measurement data measured at the first position and the second position is used to analyze the moving state of the mobile robot in the physical space to determine when the When the mobile robot is in the second position, the second positioning position data of the second position in the map data includes:

The degree of coincidence between the measurement data acquired at different positions is optimized to obtain second positioning position data corresponding to the second position in the map data under the condition of coincidence between the measurement data.
The method for positioning a mobile robot according to claim 11, wherein the measurement data measured at the first position and the second position is used to analyze the moving state of the mobile robot in the physical space to determine when the When the mobile robot is in the second position, the second positioning position data of the second position in the map data includes:

Perform joint optimization processing on the degree of coincidence between the measurement data acquired at different locations and the degree of coincidence between the collected image data, so as to obtain that the second position is in the map data under the condition of the degree of coincidence of the joint optimization. of the second positioning position data.
The method for positioning a mobile robot according to any one of claims 11 to 13, wherein the image capturing device and the distance measuring and sensing device are controlled at a first position of the mobile robot and a second position different from the first position Respectively acquire image data and measurement data synchronously, including: controlling the image capture device and various ranging sensing devices to acquire image data and measurement data synchronously at a first position of the mobile robot and a second position different from the first position, respectively .
The method for positioning a mobile robot according to claim 11, wherein the analyzing the movement state of the mobile robot in the physical space by using the measurement data measured at the first position and the second position respectively comprises: extracting the movement state of the mobile robot in the first position and the second position The landmark features in the measurement data obtained at the first position and the second position are used, and the movement state of the mobile robot in the physical space is analyzed by using the extracted landmark features.
The method for positioning a mobile robot according to claim 11, wherein the image capturing device and the distance measuring sensing device are controlled to acquire images synchronously at a first position of the mobile robot and a second position different from the first position, respectively. data and measurement data, including: controlling the image capturing device and the distance measuring sensing device to obtain image data and measurement data synchronously at a first position of the mobile robot and a second position different from the first position, respectively, wherein the synchronization signal is obtained from the mobile robot Outside or inside of image capture devices and ranging sensing devices in robots.
The method for positioning a mobile robot according to claim 16, wherein the synchronization signal is obtained from the outside of the image capturing device and the distance measuring sensing device in the mobile robot, comprising: obtaining the synchronization signal from the image capturing device in the mobile robot and Inertial navigation sensor outside the ranging sensing device.
The positioning method of a mobile robot according to claim 11, wherein the viewing angle range of the image capturing device and the measurement range of the distance measuring sensing device partially overlap or do not overlap.
A server, characterized in that it includes:

an interface device for data communication with the mobile robot;

a storage device that stores at least one program;

A processing device, connected with the storage device and the interface device, for executing the at least one program, so as to coordinate the storage device and the interface device to execute and implement the mobile robot construction map according to any one of claims 1 to 10 method, or the positioning method of a mobile robot according to any one of claims 11 to 18.
A mobile robot, comprising:

an image capturing device for collecting image data;

A ranging sensing device for acquiring measurement data;

a mobile device for performing mobile operations;

a storage device for storing at least one program;

A processing device, connected with the mobile device, the image capturing device, the distance measuring and sensing device, and the storage device, and used for executing the at least one program, so as to execute the map building process of the mobile robot according to any one of claims 1 to 10. method, or the positioning method of a mobile robot according to any one of claims 11-18.
A computer-readable storage medium, characterized in that it stores at least one computer program, and when the computer program is run by a processor, the computer program controls a device where the storage medium is located to perform the movement according to any one of claims 1 to 10. The method for constructing a map by a robot or the positioning method for a mobile robot according to any one of claims 11 to 18.