WO2019245320A1

WO2019245320A1 - Mobile robot device for correcting position by fusing image sensor and plurality of geomagnetic sensors, and control method

Info

Publication number: WO2019245320A1
Application number: PCT/KR2019/007499
Authority: WO
Inventors: 홍순혁; 명현; 김형진; 송승원; 현지음
Original assignee: 삼성전자주식회사; 한국과학기술원
Priority date: 2018-06-22
Filing date: 2019-06-21
Publication date: 2019-12-26

Abstract

Provided are a mobile robot device and a control method thereof. The mobile robot device comprises: a driving unit; an image sensor; a plurality of geomagnetic sensors; a memory for storing at least one instruction; and a processor for executing at least one instruction, wherein the processor may acquire a plurality of image data through the image sensor while the mobile robot device moves by means of the driving unit, acquire sensing data through the plurality of geomagnetic sensors, extract a feature point from the plurality of image data and obtain key nodes on the basis of the feature point, obtain a node sequence on the basis of the sensing data, generate a graph structure that estimates a position of the mobile robot device on the basis of the key nodes and the node sequence, and correct the graph structure when position recognition of the mobile robot device fails.

Description

Mobile robot device and control method by fusing an image sensor and a plurality of geomagnetic sensors

The present disclosure relates to a mobile robot device and a control method of fusion of an image sensor and a plurality of geomagnetic sensors to correct a position and to produce a map. More particularly, the present invention analyzes and simultaneously matches data acquired through each sensor and moves a mobile robot. An apparatus and a control method for estimating and correcting a current position of a.

Robots have improved work efficiency by performing tasks that are difficult to reach in various industries. In the industry related to the manufacturing industry, a robot arm type that has been fixed at a specific position and performing repetitive tasks has been used, but research and demand for a mobile robot device that is relatively free to move and can perform various tasks is gradually increasing. It is increasing.

However, in order to move, such a mobile robot device must determine where it is currently located, and at the same time, recognize a surrounding environment and create a path, that is, a map. The technology that accomplishes both processes is called SLAM (Simultaneous Localization And Mapping) technology.

The SLAM technology has recently been developed in various fields, and has been spotlighted as an essential technology for efficiently driving a cleaning robot or an autonomous vehicle. In addition, a graph-based SLAM that expresses the position and odometry of the robot as nodes and edges (constraint in other expressions) is widely used as one of the above SLAMs.

Conventionally, 2D or 3D LiDAR was commonly used to implement graph SLAM, but until now, it is inefficient to use for general robot due to its low price competitiveness.

In addition, in the case of the SLAM using a camera, there are disadvantages in that it is difficult to efficiently recognize a location in an environment such as an empty corridor having few features and is not robust due to the amount of light. .

In addition, the existing method of utilizing Beacon or Wi-Fi indoors had a limitation that a separate Beacon should be installed or communication should be smoothly performed in all indoor spaces.

In addition, the technology of recognizing the position through the existing geomagnetic sensor has a disadvantage in that the matching is not accurate in an environment in which the magnetic field distortion is not large enough, and even if it is used, only the similarity between the positions is required, and thus, another auxiliary means is required.

Therefore, in order to apply the SLAM technology to a mobile robot device, a technology for efficiently configuring sensors and processing data is required, and a search for a method of complementing each other's functions using the characteristics of each sensor is required. do.

SUMMARY The present disclosure has been made to solve the above-described problems, and an object of the present disclosure is to fuse a plurality of geomagnetic sensors with an image sensor having different characteristics, thereby enabling a mobile robot device to more efficiently use graph slam (SLAM) technology. The present invention provides a mobile robot device and a control method thereof.

Mobile robot device according to an embodiment of the present disclosure for achieving the above object is a traveling unit; An image sensor; A plurality of geomagnetic sensors; A memory for storing at least one instruction; And a processor configured to execute the at least one instruction, wherein the processor acquires a plurality of image data through the image sensor while the mobile robot apparatus moves through the driving unit, and through the plurality of geomagnetic sensors. Obtain sensing data, extract feature points from the plurality of image data, obtain key nodes based on the feature points, obtain a node sequence based on the sensing data, and based on the key nodes and the node sequence A graph structure for estimating the position of the mobile robot device may be generated, and if the position recognition of the mobile robot device fails, the graph structure may be corrected.

In addition, the processor may control to extract the feature points from the plurality of image data using an ORF (Oriented FAST and Rotated BRIEF) algorithm.

The processor may be configured to accumulate the feature points to form a submap, perform a random sample consensus (RANSAC) algorithm, and match the submap to the nearest node to obtain the key node.

The processor acquires a sensing data group by grouping the obtained sensing data, compares and matches a magnetic field value of the sensing data group with magnetic fields of a previously stored sensing data group, and matches the matched sensing data group. Based on this, it is possible to control to obtain a node bundle having a pair of graph structures.

The processor may determine whether the newly obtained node bundle exists between the previously stored node bundles, and if it is determined that the newly obtained node bundle exists between the previously stored node bundles, Only the newly acquired node bundle existing between the existing acquired node bundles may be extracted, and only the extracted newly obtained node bundle may be selected as the node sequence.

The processor updates an existing acquired node bundle having a smaller distance difference from the newly acquired node bundle, and the distance difference between the extracted newly acquired node bundle and the updated existing acquired node bundle is most significant. The small node bundle may be searched and controlled to find the position of the node sequence using the node bundle having the smallest distance difference.

In addition, the processor may control to update to an existing obtained node bundle having a smaller distance difference using a Gaussian process.

The processor detects location information different from the location information represented by the graph structure generated by the mobile robot device through a wheel sensor while the mobile robot device moves, and fails to recognize the location. The device may be rotated and controlled to acquire new image data and sensing data through the image sensor and the plurality of magnetic sensor.

In addition, the mobile robot device obtains a new keynode and node sequence based on the newly acquired image data and the sensing data and compares it with the graph structure previously generated based on the obtained new keynode and node sequence. Control to determine whether or not a match.

The processor may control to correct the graph structure based on the new keynode and node sequences matching the previously generated graph structure.

On the other hand, the control method of the mobile robot device according to an embodiment of the present disclosure for achieving the above object obtains a plurality of image data through the image sensor while the mobile robot device is moving, sensing through a plurality of geomagnetic sensors Obtaining data; extracting feature points from the plurality of image data and acquiring key nodes based on the feature points; acquiring a node sequence based on the sensing data; obtaining the key nodes and the node sequence Generating a graph structure estimating a position of the mobile robot device based on the result; And correcting the graph structure when the mobile robot device fails to recognize the position.

In the obtaining of the key nodes, the feature points may be extracted from the plurality of image data by using an ORF (Oriented FAST and Rotated BRIEF) algorithm.

The acquiring of the key nodes may include accumulating the feature points to construct a submap; And performing the submap random sample consensus (RANSAC) algorithm and matching the nearest node to obtain the key node.

The acquiring of the node sequence may include acquiring a sensing data group by grouping the acquired sensing data; Comparing and matching magnetic field values of the sensing data group with magnetic field values of the previously stored sensing data group; And acquiring a node bundle having a pair of graph structures based on the matched sensing data group.

The acquiring of the node sequence may include determining whether the newly obtained node bundle exists between the previously stored node bundles; If it is determined that the newly acquired node bundle exists between the previously stored node bundles, extracting only the newly acquired node bundle existing between the existing acquired node bundles; And selecting only the extracted newly obtained node bundle as the node sequence.

The locating of the node sequence may include updating an existing acquired node bundle having a smaller distance difference from the newly acquired node bundle; the newly acquired node bundle and the updated existing node bundle; Searching for a node bundle having a smallest distance difference among the obtained node bundles; And locating the node sequence using the node bundle having the smallest difference in distance.

In addition, the step of locating the node sequence may be updated with an existing acquired node bundle having a smaller distance difference using a Gaussian process.

In the correcting of the graph structure, while the mobile robot apparatus moves, the position sensor fails to detect position information different from the position information represented by the graph structure generated by the mobile robot apparatus through a wheel sensor. In this case, the mobile robot device rotates and acquires new image data and sensing data through the image sensor and the plurality of magnetic sensor.

The correcting of the graph structure may include: acquiring a new key node and a node sequence based on the image data and the sensing data newly acquired by the mobile robot device; And comparing the new key node and the node sequence based on the acquired new key node and the node sequence to determine whether they match.

The correcting of the graph structure may include correcting the graph structure based on the new keynode and node sequences matching the previously generated graph structure.

According to various embodiments of the present disclosure as described above, the mobile robot device can generate a graph structure based on the location information by matching and fusing data obtained from the image sensor and the plurality of geomagnetic sensors, respectively, and the mobile robot. If the device fails to recognize the position, the graph can be corrected.

1 is a block diagram briefly illustrating a configuration of a mobile robot apparatus according to an embodiment of the present disclosure;

2 is a block diagram illustrating in detail the configuration of a mobile robot apparatus according to an embodiment of the present disclosure;

3 is a block diagram illustrating a configuration of generating a graph by acquiring a sequence of key nodes and nodes according to an embodiment of the present disclosure;

4 is a block diagram illustrating a graph slam (SLAM) algorithm using an image sensor, according to an embodiment of the present disclosure;

5 is a flowchart illustrating a method of obtaining a key node from an image sensor according to an embodiment of the present disclosure;

6A is a diagram for describing a method of extracting feature points from image data, according to an exemplary embodiment.

6B is a diagram for describing a submap in which feature points extracted from image data are accumulated, according to an embodiment of the present disclosure;

7 is a flowchart illustrating a method of obtaining a node bundle from sensing data according to an embodiment of the present disclosure;

8 is a diagram for describing a method of obtaining a node sequence by matching a node bundle according to an embodiment of the present disclosure;

FIG. 9A is a diagram for describing a distance value between an existing node bundle and a newly obtained node bundle, according to an embodiment of the present disclosure; FIG.

FIG. 9B is a view for explaining a table for determining whether a distance value between a previously acquired node bundle and a newly acquired node bundle can be measured according to an embodiment of the present disclosure; FIG.

FIG. 10 is a diagram for describing a method of determining a location of a newly obtained node bundle using an existing node bundle, according to an embodiment of the present disclosure; FIG.

FIG. 11 is a diagram for describing a method of generating a graph structure using a key node and a node sequence, according to an embodiment of the present disclosure; FIG.

12A is a diagram for describing a method of matching a submap and a target node when the mobile robot apparatus fails to recognize a location according to an embodiment of the present disclosure;

12B is a diagram for describing a method of matching a feature point and a target node of a submap when the mobile robot apparatus fails to recognize a position according to an embodiment of the present disclosure;

FIG. 13A is a diagram for describing a method of matching a target node when a mobile robot apparatus fails to recognize a position, according to an exemplary embodiment.

FIG. 13B is a diagram for describing a method of correcting a graph by matching a target node when the mobile robot apparatus fails to recognize a position according to an embodiment of the present disclosure; FIG.

14 is a flowchart illustrating a method of matching with a target node when the mobile robot apparatus fails to recognize a position according to an embodiment of the present disclosure;

FIG. 15A is a diagram for describing a method of performing an experiment when a mobile robot apparatus fails to recognize a position, according to an exemplary embodiment.

FIG. 15B is a view for explaining a result of an experiment when a mobile robot apparatus fails to recognize a position according to one embodiment of the present disclosure; FIG.

16 is a flowchart illustrating a method of controlling a mobile robot device according to an embodiment of the present disclosure.

Embodiments described below are shown by way of example in order to help understanding of the present disclosure, it should be understood that the present disclosure may be implemented in various modifications different from the embodiments described herein. However, in the following description of the present disclosure, when it is determined that a detailed description of a related known function or component may unnecessarily obscure the subject matter of the present disclosure, the detailed description and specific illustration thereof will be omitted. In addition, the accompanying drawings may be exaggerated in dimensions of some of the components, rather than drawn to scale to facilitate understanding of the disclosure.

In the description of the present disclosure, the order of each step should be understood to be non-limiting unless the preceding step is to be performed logically and temporally prior to the later step. That is, except for the exceptional cases described above, even if the process described as the following step is performed before the process described as the preceding step, the nature of the disclosure is not affected and the scope of rights should be defined regardless of the order of the steps.

And as used herein, "A or B" is defined to mean not only selectively indicating any one of A and B, but also including both A and B. In addition, the term "comprising" in this specification has the meaning encompassing further including other components in addition to the listed elements.

In this specification, since the components required for the description of each embodiment of the present disclosure have been described, the present disclosure is not necessarily limited thereto. Thus, some components may be changed or omitted, and other components may be added. It may also be arranged distributed in different independent devices.

Hereinafter, with reference to the drawings will be described in more detail with respect to the present disclosure. 1 is a block diagram schematically illustrating a configuration of a mobile robot apparatus 10 according to an embodiment of the present disclosure.

As shown in FIG. 1, the mobile robot device 10 includes a processor 150, an image sensor 110, a plurality of geomagnetic sensors 120, a memory 130, and a driving unit 140.

The 3D image data including the color image and the depth image may be obtained using the image sensor 110, and a distance value corresponding to the pixel of the obtained 3D image data may be measured. The image sensor 110 may include a stereo camera, a time of flight camera, an infrared camera, and the like, and may obtain depth data values of each pixel. In addition, the image sensor 110 may operate in various situations such as the moving robot apparatus 10 moves, rotates, and stops.

The plurality of geomagnetic sensors 120 may be an IMU (Inertial Measurement Unit) sensor incorporating a three-axis geomagnetic sensor. Magnetic field sensing data may be obtained through the plurality of geomagnetic sensors 120. Sensing data obtained through the plurality of geomagnetic sensors 120 may be stored in the memory 130.

The memory 130 may store instructions or data related to at least one other component of the mobile robot device 10. In particular, the memory 130 may be implemented as a nonvolatile memory, a volatile memory, a flash-memory, a hard disk drive (HDD), or a solid state drive (SSD). The memory 130 may be accessed by the processor 150 and read / write / modify / delete / update of data by the processor 150 may be performed. In the present disclosure, the term memory refers to a memory card (not shown) mounted on the memory 130, a ROM (not shown), a RAM (not shown), or the mobile robot device 10 (eg, micro). SD card, memory stick). In addition, the memory 130 may store programs and data for configuring various screens to be displayed in the display area of the display.

In particular, the memory 130 may store the acquired 3D image data or the sensing data, and may store at least one instruction. Various program instructions may be included, but various program instructions may be partially omitted, modified, or added depending on the type and characteristics of the mobile robot apparatus 10.

The driving unit 140 is a device that helps the mobile robot device 10 to move, and may be configured as a wheel, or may be configured as a device that can move in a non-standard moving form such as N-group walking. In addition, the driving unit 140 may rotate left and right as well as forward and backward and may also be configured to rotate. Therefore, the driving unit 140 may be configured in various ways according to the type and characteristics of the mobile robot device 10.

The processor 150 may control the overall operation of the mobile robot apparatus 10 using various instructions and modules stored in the memory 130.

In detail, while the mobile robot apparatus 10 moves through the driving unit 140, the processor 150 acquires a plurality of image data through the image sensor 110 and senses through the plurality of geomagnetic sensors 120. Data can be obtained.

The processor 150 may extract feature points from the plurality of image data and obtain key nodes based on the feature points. In detail, the processor 150 may extract the feature points from the plurality of image data using an ORF (Oriented FAST and Rotated BRIEF) algorithm. The processor 150 may accumulate the extracted feature points to form a submap, perform a random sample consensus (RANSAC) algorithm on the submap, and match the nearest node to obtain a key node.

In addition, the processor 150 may obtain a node sequence based on the sensing data. Specifically, a graph structure for estimating the position of the mobile robot apparatus 10 based on the key nodes and the node sequence is generated, and when the position recognition of the mobile robot apparatus 10 fails, the graph structure is generated. You can correct it.

The processor 150 acquires a sensing data group by grouping the obtained sensing data, compares and matches the magnetic field values of the obtained sensing data group with the magnetic field values of a previously stored sensing data group, and matches Based on the sensing data group, a node bundle having a pair of graph structures may be obtained.

In addition, the processor 150 determines whether the newly obtained node bundle exists between the previously stored node bundles, and if it is determined that the newly obtained node bundle exists between the previously stored node bundles. Only the newly acquired node bundle existing between the existing obtained node bundles may be extracted, and only the newly acquired node bundle may be selected as the node sequence.

The processor 150 updates an existing acquired node bundle having a smaller distance difference from the newly acquired node bundle and extracts a distance difference between the newly acquired node bundle and the updated existing node bundle. The smallest node bundle may be searched and the node sequence having the smallest distance difference may be used to locate the node sequence.

In addition, the processor 150 may update a previously obtained node bundle having a smaller distance difference using a Gaussian process.

In addition, the processor 150 senses location information different from the location information represented by the graph structure generated by the mobile robot device 10 through the wheel sensor 160 while the mobile robot device 10 moves. If the position recognition fails, the mobile robot apparatus 10 may rotate and acquire new image data and sensing data through the image sensor 110 and the plurality of magnetic sensors.

In addition, the processor 150 acquires a new key node and node sequence based on the image data and the sensing data newly acquired by the mobile robot device 10, and based on the obtained new key node and node sequence. It may be determined whether or not a match is made by comparing the graph structure generated previously.

In addition, the processor 150 may correct the graph structure based on the new key node and node sequence matching the previously generated graph structure.

2 is a block diagram showing in detail the configuration of the mobile robot device 10 according to an embodiment of the present disclosure. As shown in FIG. 2, the mobile robot apparatus 10 includes a processor 150, an image sensor 110, a plurality of geomagnetic sensors 120, a wheel sensor 160, an input unit 170, a memory 130, The driving unit 140 may include a communication unit 180, a display unit 190, and a function unit 195.

Meanwhile, since the image sensor 110, the plurality of geomagnetic sensors 120, the memory 130, the driving unit 140, and the processor 150 illustrated in FIG. 2 have been described with reference to FIG. 1, redundant descriptions thereof will be omitted. do.

On the other hand, Figure 2 is a case where the mobile robot device 10 is a device having a variety of functions, such as content sharing function, communication function, display function, for example, shows a comprehensive view of the various components. Therefore, according to the exemplary embodiment, some of the components shown in FIG. 2 may be omitted or changed, and other components may be further added.

The number of the image sensors 110 may be plural and may be disposed in the central portion or the outer portion of the mobile robot apparatus 10 to photograph the surrounding environment. However, this is only an embodiment, and the number of image sensors 110 may be added, modified, or changed depending on the type and characteristics of the mobile robot apparatus 10.

The plurality of geomagnetic sensors 120 may measure magnetic field distortion of an indoor environment, and extract magnetic field characteristics and the like using a sequence. The plurality of geomagnetic sensors 120 may also operate regardless of the operating situation of the mobile robot device 10.

The plurality of geomagnetic sensors 120 may be symmetrically disposed to the left and right of the driving unit 140 of the mobile robot apparatus 10, but this is merely an example and may be disposed at various positions. According to the type and characteristics of the mobile robot device 10, the number of the plurality of geomagnetic sensors 120 may be added, modified or changed.

In an embodiment, the wheel sensor 160 may detect a situation in which the wheel and the floor of the mobile robot apparatus 10 are lifted without contact, that is, a kidnapping situation. In addition to the kidnapping situation, the wheel sensor 160 may detect a location recognition failure situation that detects location information different from a graph structure indicating the location information of the mobile robot device 10 acquired by the mobile robot device 10. . The wheel sensor 160 may be disposed on the driving unit 140 abutting the floor, but this is only an example and may be disposed anywhere touching the floor.

The input unit 170 may receive a user command for controlling the overall operation of the mobile robot device 10. In this case, the input unit 170 may be implemented as a touch screen. However, the input unit 170 may be implemented as another input device (for example, a mouse, a keyboard, or a microphone) according to the type and characteristics of the mobile robot apparatus 10. Can be.

The driving unit 140 may allow the mobile robot apparatus 10 to freely move, including forward and backward rotation. As described above, the wheel sensor 160 may be disposed on the driving unit 140. According to the type and characteristics of the mobile robot device 10, the driving unit 140 may be variously modified, such as quadruped walking.

The communication unit 180 is a component capable of communicating with various types of external devices according to various types of communication methods.

The communication unit 180 may include various communication chips such as a Wi-Fi chip, a Bluetooth chip, a wireless communication chip, and the like. In this case, the Wi-Fi chip and the Bluetooth chip may communicate with each other by the Wi-Fi method and the Bluetooth method. The wireless communication chip refers to a chip that performs communication according to various communication standards such as IEEE, Zigbee, 3rd Generatoin (3G), 3rd Generation Partnership Project (3GPP), Long term Evolution (LTE), and the like.

The display 190 may output image data under the control of the processor 150. In particular, the content received through the communication unit 180 may be displayed, and when generating a graph structure based on the data acquired by the mobile robot device 10, the corresponding graph is displayed to display the mobile robot device 10. It is also possible to estimate the movement path. This is only an example and may display various functions according to the type and characteristics of the mobile robot apparatus 10.

The function unit 195 may perform various functions according to the type and characteristics of the mobile robot device 10. For example, if the mobile robot apparatus 10 is a robot cleaner, it may adsorb dust in the surroundings through the function unit 195 while moving. If the mobile robot device 10 is a robot that moves and plays various multimedia contents, the function unit 195 may be hardware that plays multimedia contents.

The processor 150 may process a digital signal, a central processing unit (CPU), a micro controller unit (MCU), a micro processing unit (MPU), a controller, and an application processor (AP). Or a communication processor (CP), one or more of ARM processors, or may be defined in the corresponding terms. In addition, the processor 150 may be implemented in a System on Chip (SoC), a large scale integration (LSI) in which a processing algorithm is embedded, or may be implemented in the form of a field programmable gate array (FPGA). The processor 150 may perform various functions by executing computer executable instructions stored in the memory 130. In addition, the processor 150 may include at least one of a separate AI dedicated processor, a graphics-processing unit (GPU), a neural processing unit (NPU), and a visual processing unit (VPU) to perform an artificial intelligence function. have.

3 is a block diagram illustrating a configuration of generating a graph by acquiring a sequence of key nodes and nodes according to an embodiment of the present disclosure.

The processor 150 may include a key node acquirer 310, a node sequence acquirer 320, and a graph generator 330 illustrated in FIG. 3.

The key node acquisition unit 310 may obtain a key node by extracting and matching feature points by using image data acquired by the image sensor 110 stored in the memory 130. The image data is three-dimensional data including color data and depth data, and feature points may be extracted by using an ORB algorithm. ORB (Oriented FAST and Rotated BRIEF) algorithms are just one embodiment, and three-dimensional image data can also be used using other algorithms (e.g., algorithms such as The speeded up robust features (SURF) and the Scale Invariant Feature Transform (SIFT)). Feature points can be extracted from.

In addition, the key node acquisition unit 310 may overcome an environment in which the feature points are insufficient by accumulating the extracted feature points to form a submap. The submap is formed by accumulating three-dimensional feature points in the key node acquisition unit 310, and may be based on reliability of an odometry of the robot.

In an embodiment, when the wheel sensor 160 detects a location recognition failure situation such as kidnapping of the mobile robot device 10, the key node acquisition unit 310 may acquire the image from the image sensor 110 in a new environment. Key nodes can be obtained by using three-dimensional image data.

The node sequence acquirer 320 may acquire a node sequence based on sensing data acquired through the plurality of geomagnetic sensors 120 stored in the memory 130. The sensing data includes data in which magnetic field distortion of an indoor environment is reflected. The data measured by the magnetic field distortion, such as the sensing data, may derive the magnetic field value using a sequence of data instead of the value itself at a specific instant.

Accordingly, the node sequence obtainer 320 may group sensing data, match magnetic field values of the grouped sensing data, and generate a node bundle with the matched sensing data group. The node bundle may be matched again to obtain a node sequence in the form of a loop closing. The loop closing form refers to the closed loop form and means that the node is connected in the form of a sequence.

The node sequence obtainer 320 may apply a Gaussian process in the process of obtaining a node sequence by performing node bundle matching. The Gaussian process is a type of supervised learning that is a set of random variables. In this case, the set variables follow a joint Gaussian distribution. Specifically, this will be described with reference to FIG. 10.

In addition, the node sequence acquisition unit 320 compares a newly obtained node bundle query value with a previously obtained node bundle to extract a query having the smallest error score. Can be. The node bundle having the smallest distance difference can be selected as a node sequence in a loop closing form.

The graph generator 330 may generate a graph structure indicating the position information of the mobile robot apparatus 10 based on the keynode and node sequences acquired by the keynode acquirer 310 and the node sequence acquirer 320. have.

The graph structure generated by the graph generator 330 may be in the form of a loop closing, and may be composed of a key node and a node sequence. Such a graph structure may mean that 3D image data matching and sensing data matching may be simultaneously performed.

In addition, the graph generator 330 may not only be a kidnapping situation in which the wheel sensor 160 is currently held by the mobile robot device 10, but also an image sensor 110 and a plurality of geomagnetic sensors 120. The graph can be corrected based on the data obtained from

In addition, as an example, the graph structure generated by the graph generator 330 may be displayed on the display 190, and is preset in the input unit 170 (eg, a touch screen, a microphone, a keyboard, etc.). When performing the event, the display 190 can see the simulation process.

4 is a block diagram illustrating a graph slam (SLAM) algorithm using an image sensor 110 according to an embodiment of the present disclosure.

That is, in FIG. 4, a key node is obtained based on the 3D image data acquired by the image sensor 110, a graph structure is generated based on the obtained key node, and the position information of the mobile robot apparatus 10 is estimated. The process of making a map 400 is shown.

The 3D image data including the depth data is acquired through the image sensor 110 that is a component of the mobile robot apparatus 10 (410). According to an embodiment, the continuous or discontinuous three-dimensional image data is acquired through the image sensor 110, and the obtained three-dimensional image data is stored in the memory 130.

Using the 3D image data acquired through the image sensor 110, the mobile robot device 10 may perform a process of visual odometry 420. The visual odometry 420 process refers to a process of determining the position and direction of the mobile robot device 10 by analyzing the image data acquired by the mobile robot device 10. The mobile robot device 10 of the present disclosure uses a visual odometry 420 that can be effectively used for the mobile robot device 10 having a non-standard moving method such as a group N walking robot, but this is only one embodiment. Do.

In one embodiment, the visual odometry 420 process performed by the mobile robot apparatus 10 is to perform image correction on the first three-dimensional image data obtained. In detail, the mobile robot apparatus 10 may correct such as lens distortion that may occur while the image sensor 110 acquires 3D image data. The mobile robot apparatus 10 may extract a feature point of the three-dimensional image data after the image correction is completed, and build an optical flow field with the extracted feature point. The mobile robotic device 10 may determine the position and direction of the mobile robotic device 10 through the optical flow vector of the optical flow field.

In an embodiment, the mobile robot apparatus 10 may extract feature points from an ORB algorithm using three-dimensional image data. ORB (Oriented FAST and Rotated BRIEF) algorithm determines 16 pixel values on a circle with a radius of 3 around a specific point on a three-dimensional image frame, and brighter or darker pixels that are lighter or darker than a certain point. If more than nine are consecutive, the feature points can be extracted using the FAST-9 algorithm which selects the feature points as corners. However, this is only an example, and the number of consecutive bright or dark pixels may be changed according to the characteristics of the mobile robot device 10 while using an ORB algorithm. to be.

The mobile robot device 10 accumulates and matches feature points extracted from the obtained 3D image data. This process is referred to as node management 430. In detail, the mobile robot apparatus 10 may match a submap accumulated by collecting the feature points of the nodes, that is, the nodes of the mobile robot apparatus 10.

(One)

Equation (1) above is an equation for explaining that the processor 150 may configure a submap by accumulating the feature points extracted from the 3D image data. M on the left side means the j-th submap accumulated feature points. Right

Is a constant,

Denotes an odd geometry of the i-th mobile robot apparatus 10,

Denotes a coordinate transformation from the image sensor 110 to the mobile robot device 10,

Is the pose of the i-th robot, that is, the feature point of the i-th node. K and l are index numbers of the first and last nodes included in the j th submap.

The mobile robot apparatus 10 may match the submap formed by accumulating the feature points in two stages. In the first step, the mobile robot apparatus 10 may search for a submap closest to the target node by matching several submaps with the target node. Since the mobile robot device 10 is difficult to estimate the position of the mobile robot device 10 by performing matching on a single feature point in an environment in which the feature points are insufficient, the closest node, i. Rough matching may be performed with the target node.

In the present disclosure, the mobile robot device 10 may utilize a RANSAC algorithm based on rigid transformation when performing rough matching. Rigid body transformation refers to a transformation that changes only the position and direction while maintaining the shape and size of the object to be converted.

In one embodiment, the RANSAC algorithm is an algorithm that enables regression analysis of values outside the normal distribution called outliers to data belonging to the normal distribution. Specifically, the mobile robot device 10 selects any number of accumulated feature points, assumes the selected number of feature points as data (Inlier) belonging to a normal distribution, and then selects a regression model. You can get it. In addition, when comparing the remaining feature points with the regression model, the mobile robot apparatus 10 may determine whether the remaining feature points are within a preset tolerance, and if the tolerances are within the tolerance, include the data (Inlier) in a normal distribution and regression. Models can be constructed. In addition, the mobile robot apparatus 10 may measure the reconfigured regression model, the data (Inlier) and the error belonging to the normal distribution, and determine whether to repeat the above-described process according to whether the error exceeds a preset tolerance. .

In the second step, the mobile robot apparatus 10 may acquire a key node closest to the target node in the submap by matching the target node with the node including the feature points accumulated in the submap searched in the first step.

That is, in the first step, the mobile robot device 10 may look at one submap to which the RANSAC algorithm is applied and match the target node to search for the nearest submap, and the second step may be performed by the mobile robot device 10. A step of acquiring a key node which is a node closest to the target node by matching the node including the feature point of the submap with the target node.

After going through the node management 430, the mobile robot device 10 may determine depth based constraints 450 and feature based constraints of each of the keynodes. 460 may be generated.

The mobile robot device 10 may generate a constraint by connecting a region where the depth data of the key node matches the depth data of the target node, and the constraint may be a depth based constraint. (450)

The mobile robot device 10 may generate a constraint by connecting a keynode's feature to an area matching the feature of the target node, and the constraint may be a feature based constraints ( 460).

In addition, the mobile robot apparatus 10 generates depth based constraints 450 and feature based constraints 460 based on the key node to form a loop closing shape. In this case, optimizing the graph structure means that the mobile robot device 10 generates a graph based on the key node in order to best represent the position information of the mobile robot device 10. This means that it can be generated accurately and the error can be minimized.

Meanwhile, Bow (Bag of words) Scene Matching 440 is a matching method that the mobile robot apparatus 10 may use in a situation such as kidnapping. In an embodiment, when the wheel sensor 160 detects a kidnapping situation, the mobile robot apparatus 10 extracts a representative feature point from feature points extracted from previously acquired image data, and comprises a codebook consisting of representative feature points. Can be generated. Then, when the mobile robot device 10 is placed on the floor after the kidnapping situation, the mobile robot device 10 can check whether the current position matches the existing position using a codebook.

5 is a flowchart illustrating a method of obtaining a key node from the image sensor 110 according to an embodiment of the present disclosure.

First, as an example, the image sensor 110 such as an RGB-D sensor may acquire 3D image data. (S510) The mobile robot apparatus 10 may store the obtained 3D image data in the memory 130. Can be stored.

In addition, the mobile robot apparatus 10 may extract a feature point by applying an ORB algorithm, in one embodiment, from the plurality of image data stored in the memory 130 (S520).

The mobile robot apparatus 10 may accumulate the extracted feature points to form a submap (S530), apply a RANSAC algorithm, and perform matching. (S540) The matching method is divided into two steps as described above.

In the first step, the mobile robot device 10 may view the submap that has undergone the RANSAC algorithm as one node and perform rough matching with the target node.

In the second step, the mobile robot apparatus 10 may search for the nearest node by matching the nodes of the matched submap and the target node (S550). Thereafter, the mobile robot apparatus 10 may acquire a node that is closest to the target node, that is, a key node (S560).

6A and 6B are diagrams for describing a method of extracting feature points from image data, according to an exemplary embodiment.

6A illustrates the mobile robot device 10 extracting feature points from three-dimensional image data. The mobile robot apparatus 10 may extract and view the vicinity of the leg 610 of the whiteboard as a feature point, which is continuously brighter than the pixels near the floor of the corridor. At the same time, the mobile robot apparatus 10 may view and extract a portion 620 attached to a white tape which is brighter than pixels Pixel near the bottom of the hall as a feature point.

In FIG. 6B, the mobile robot apparatus 10 shows the accumulation of sub-maps of feature points extracted from the 3D image data. As described above, the mobile robot apparatus 10 may apply the RANSAC algorithm to the submap formed by accumulating the feature points extracted from the image data and match the target node. In addition, the mobile robot apparatus 10 may obtain a key node by matching the feature points constituting the matched submap and the target node.

7 is a flowchart illustrating a method of obtaining a node bundle from sensing data according to an embodiment of the present disclosure.

First, the sensing data may be acquired by the plurality of geomagnetic sensors 120. (S700) The sensing data is data obtained by measuring magnetic field distortion of an indoor environment, and should be formed in a sequence form. Can be grouped in a sequence form (S720).

The mobile robot apparatus 10 may compare and match the previously stored sensing data group with the newly acquired sensing data group and the magnetic field values. (S740) The mobile robot apparatus 10 compares and matches the magnetic field values of the sensing data group. By doing so, it is possible to construct a loop closing with the node sequence as a component.

(2)

Equation (2) above calculates a matching cost when the sensing data group is expressed as a vector. The mobile robot apparatus 10 may obtain a matching cost by calculating a Euclidean distance between the previously stored sensing data group and the newly acquired sensing data.

The mobile robot device 10 may generate a node bundle with the matched sensing data group (S760). That is, the mobile robot device 10 may generate a node bundle having a pair of graph structures as matched sensing data groups.

8 to 10 are diagrams for describing a method of obtaining a node sequence by matching node bundles according to an embodiment of the present disclosure.

FIG. 8 compares the distance difference between the node bundle 800 (or target) previously acquired by the mobile robot device 10 and the node bundle 810 (or query) newly acquired. The process of finding a matching point is shown.

The mobile robot device 10 may calculate an error score by pushing the query 810 with the target 800. The mobile robot device 10 may repeat the process 820 of obtaining a distance difference from the target 800 while continuously raising the position of the query 810. Therefore, it can be seen from the table 890 that the difference between the target 800 and the query 810 is the smallest in the case of (c) than in the case of (b) 840 or (d) 880. .

In FIG. 9A, a node bundle consisting of a pair of graph structures of a target_L 910 and a target_R 930 and a node bundle consisting of a pair of graph structures of a query_L 920 and a query_R 940 are illustrated. It is shown.

In FIG. 9B, a table shows whether the target 800 is between queries 810 so that the mobile robot device 10 is in a range capable of measuring physical values between each node bundle.

As shown in FIG. 9A, the mobile robot apparatus 10 may calculate a distance difference between the target_L 910 and the target_R 930 based on the query_L 920 and the query_R 940. have. For example, the mobile robot apparatus 10 calculates the distance between the target_L 920 and LL based on the query_L 920 and the distance between the query_L 920 and the target_R 930 is LR. Can be calculated as

As shown in FIG. 9B, the mobile robot device 10 may measure the distance difference between the query bundle and the graph structure constituting the target bundle only when the query bundle is on the left side or on the right side.

In FIG. 10, when the mobile robot device 10 determines that the query 1010 exists between the targets 1020, the target 1020 extracts only the query and the Gaussian process to find the location of the extracted query 1010. The process of updating the target 1030 having a smaller error score (Error Score) is shown.

The mobile robot device 10 may extract only a query existing between the targets 1020 and select the extracted query as a node sequence. In addition, the mobile robot apparatus 10 may generate a target 1030 having a smaller query and distance difference than the target 1040 by using a Gaussian process to determine the position of the node sequence.

The mobile robot apparatus 10 performs a Gaussian process repeatedly until the distance between the curry selected as the node sequence is almost disappeared, that is, until the distance difference is smallest, and finally finds the position of the node sequence. Can be.

FIG. 11 is a diagram for describing a method of generating a graph structure using a key node and a node sequence, according to an exemplary embodiment.

The mobile robot device 10 may simultaneously configure a graph structure 1110 having a key node 1100 obtained based on 3D image data and a node sequence obtained based on sensing data, as components.

In detail, the mobile robot apparatus 10 may create the constraint 1120 using the key node 1100 and the node sequence, and perform a graph slam by generating a graph structure in the form of a loop closing. have.

12 to 13 are diagrams for describing a method of calibrating a graph when the mobile robot apparatus 10 fails to recognize a position, according to an embodiment of the present disclosure, and FIG. 14 is a view according to an embodiment of the present disclosure. 2 is a flowchart illustrating a method of correcting a graph when the mobile robot apparatus 10 fails to recognize a position.

The wheel sensor 160 may detect a case where the mobile robot device 10 fails to recognize the position, such as a kid napping situation in which the mobile robot device 10 is heard, and the mobile robot device 10 may again detect an image sensor ( The key node and the node sequence may be obtained using the 110 and the plurality of geomagnetic sensors 120.

The mobile robot apparatus 10 may rough match the relocation node 1220 based on a submap in which feature points are equally accumulated in a situation such as kidnapping.

As shown in FIG. 12A, the mobile robot apparatus 10 may match a submap 1200 including a node relatively closer to the relocation node 1220 among various submaps. In addition, the mobile robot apparatus 10 may identify that the submap 1210 including the relatively distant node does not match the relocation node 1220.

As shown in FIG. 12B, the mobile robot apparatus 10 may match the submap 1200 including the node closer to the relocation node 1220 again to search for the best match.

FIG. 13 illustrates in detail a loop closed graph structure in which the mobile robot device 10 finds the best matching pair, that is, a node, in the kidnapping 1300 situation.

When the mobile robot device 10 finds the best-fit node 1310, the mobile robot device 10 places a constraint 1320 on the best-fit node 1310 and performs an existing graph slam. do.

14 is a flowchart for describing a method of correcting a graph when the mobile robot apparatus 10 fails to recognize a position, according to an exemplary embodiment.

First, the wheel sensor 160 may detect a kidnapping situation such as the mobile robot device 10 being lifted (S1410). At this time, the mobile robot device 10 may generate generation of an odometry constraint. Therefore, the mobile robot apparatus 10 may rotate and simultaneously acquire three-dimensional image data and sensing data using the image sensor 110 and the plurality of geomagnetic sensors 120. (S1430)

In addition, the mobile robot apparatus 10 may newly acquire a key node and a node sequence based on the acquired new 3D image data and sensing data, and attempt to match the relocation node (S1440).

The mobile robot device 10 may determine the next step according to whether the matching is successful (S1450). If the matching does not succeed, the mobile robot device 10 attempts matching again (S1440). In this case, the graph may be corrected based on the matched keynode and node sequences (S1460).

FIG. 15 is a diagram for describing a method and a result of assuming an experiment when the mobile robot apparatus 10 fails to recognize a position, according to an exemplary embodiment.

FIG. 15A illustrates an experimental method for understanding how the mobile robot apparatus 10 operates when an experimenter randomly generates a kidnapping situation 1510 in the mobile robot apparatus 10 in a rectangular experiment site 13 m long and 8.5 m long. It is shown.

In FIG. 15B, a result value when the experiment planned in FIG. 15A is actually performed is illustrated. The line 1520 indicated by the dotted line indicates the trajectory of the graph slam generated by the mobile robot device 10 in the kidnapping situation. In addition, the line indicated by the solid line 1530 is an odometry trajectory of the mobile robot device 10, and even though the kidnapping 1510 occurs, it can be seen that the result value is derived to match the trajectory 1520 of the graph slam. .

FIG. 16 is a flowchart illustrating a control method of the mobile robot apparatus 10 according to an exemplary embodiment.

First, the mobile robot apparatus 10 may acquire a plurality of image data through the image sensor 110 and acquire sensing data through the plurality of geomagnetic sensors 120 (S1610).

In addition, the mobile robot apparatus 10 may extract a feature point from a plurality of image data, obtain a key node, and obtain a node sequence based on the sensing data (S1620).

In addition, the mobile robot apparatus 10 may generate a graph structure indicating the position information of the mobile robot apparatus 10 using the obtained key node and node sequence (S1630).

When the mobile robot device 10 fails to recognize the location (for example, in a situation such as kidnapping) (S1640), the mobile robot device 10 acquires a new node and node sequence again to obtain the location information of the mobile robot device 10. The graph structure shown can be corrected. (S1650)

On the other hand, the term "part" or "module" as used in the present disclosure includes a unit composed of hardware, software, or firmware, and for example, may be used interchangeably with terms such as logic, logic block, component, or circuit. Can be. The "unit" or "module" may be an integrally formed part or a minimum unit or part of performing one or more functions. For example, the module may be configured as an application-specific integrated circuit (ASIC).

Various embodiments of the present disclosure may be implemented in software that includes instructions stored in a machine-readable storage media. The device may be a device capable of calling a stored command from a storage medium and operating according to the called command, and may include an electronic device according to the disclosed embodiments. When an instruction is executed by a processor, the processor may perform a function corresponding to the instruction directly or by using other components under the control of the processor. The instructions can include code generated or executed by a compiler or interpreter. The device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' means that the storage medium does not include a signal and is tangible, but does not distinguish that the data is stored semi-permanently or temporarily on the storage medium.

According to one embodiment, a method according to various embodiments disclosed herein may be provided included in a computer program product. The computer program product may be traded between the seller and the buyer as a product. The computer program product may be distributed online in the form of a device-readable storage medium (eg compact disc read only memory (CD-ROM)) or through an application store (eg Play StoreTM). In the case of an online distribution, at least a portion of the computer program product may be stored at least temporarily on a storage medium such as a server of a manufacturer, a server of an application store, or a relay server, or may be temporarily created.

Each component (eg, a module or a program) according to various embodiments may be composed of a singular or plural number of objects, and some of the above-described subcomponents may be omitted, or other subcomponents may be omitted. It may be further included in various embodiments. Alternatively or additionally, some components (eg, modules or programs) may be integrated into one entity to perform the same or similar functions performed by each corresponding component prior to integration. According to various embodiments, operations performed by a module, program, or other component may be executed sequentially, in parallel, repeatedly, or heuristically, or at least some of the operations may be executed in a different order, omitted, or another operation may be added. Can be.

Claims

In a mobile robotic device,

Driving unit;

An image sensor;

A plurality of geomagnetic sensors;

A memory for storing at least one instruction; And

A processor for executing the at least one instruction;

The processor,

While the mobile robot device is moving through the driving unit, a plurality of image data is obtained through the image sensor, and sensing data is obtained through the plurality of geomagnetic sensors.

Extracting feature points from the plurality of image data and obtaining key nodes based on the feature points;

Obtaining a node sequence based on the sensing data,

Generate a graph structure estimating the position of the mobile robot device based on the key nodes and the node sequence,

The mobile robot device for correcting the graph structure when the position recognition of the mobile robot device fails.
The method of claim 1,

The processor,

And extracting the feature points from the plurality of image data by using an oriented fast and rotated BRIEF (ORB) algorithm.
The method of claim 1,

The processor,

Accumulating the feature points to form a submap,

And performing the random sample consensus (RANSAC) algorithm on the submap and matching the closest node to obtain the key node.
The method of claim 1,

The processor,

Grouping the acquired sensing data to obtain a sensing data group,

Compare and match the magnetic field values of the sensing data group with the magnetic field values of the previously stored sensing data group,

And a node bundle having a pair of graph structures based on the matched sensing data group.
The method of claim 4, wherein

The processor,

It is determined whether the newly obtained node bundle exists between the previously stored node bundles.

If it is determined that the newly acquired node bundle exists between the previously stored node bundles, only the newly acquired node bundle existing between the existing acquired node bundles is extracted,

And only the extracted newly acquired node bundle as the node sequence.
The method of claim 5,

The processor,

Update the existing obtained node bundle with a smaller distance difference from the extracted newly acquired node bundle,

Search for the node bundle having the smallest distance difference between the extracted newly acquired node bundle and the updated existing acquired node bundle,

And locate the node sequence using the node bundle having the smallest difference in distance.
The method of claim 6,

The processor,

A mobile robotic device, characterized in that for updating a previously obtained node bundle having a smaller distance difference using a Gaussian process.
The method of claim 1,

The processor,

While the mobile robot device moves, if the mobile robot device detects location information different from the location information indicated by the graph structure generated by the mobile robot device and fails to recognize the location, the mobile robot device rotates. A mobile robot apparatus, wherein new image data and sensing data are acquired through the image sensor and the plurality of magnetic sensor sensors.
The method of claim 8,

The processor,

The mobile robot device obtains a new key node and node sequence based on the newly acquired image data and the sensing data.

And comparing the graph structure with the previously generated graph structure based on the acquired new key node and node sequence to determine whether or not a match is made.
The method of claim 9,

The processor,

The mobile robot device, characterized in that for correcting the graph structure on the basis of the new key node and node sequence matching the previously created graph structure
In the control method of a mobile robot device,

Acquiring a plurality of image data through an image sensor and sensing data through a plurality of geomagnetic sensors while the mobile robot apparatus moves;

Extracting feature points from the plurality of image data and obtaining key nodes based on the feature points;

Obtaining a node sequence based on the sensing data;

Generating a graph structure for estimating the position of the mobile robot device based on the key nodes and the node sequence; And

Correcting the graph structure when the mobile robot device fails to recognize a position.
The method of claim 11,

Obtaining the key nodes,

And extracting the feature points from the plurality of image data using an ORB algorithm.
The method of claim 11,

Obtaining the key nodes,

Accumulating the feature points to form a submap;

And performing the random sample consensus (RANSAC) algorithm on the submap and matching the closest node to obtain the key node.
The method of claim 11,

Acquiring the node sequence,

Grouping the obtained sensing data to obtain a sensing data group;

Comparing and matching magnetic field values of the sensing data group with magnetic field values of the previously stored sensing data group; And

Obtaining a node bundle having a pair of graph structures based on the matched sensing data group.
The method of claim 14,

Acquiring the node sequence,

Determining whether the newly obtained node bundle exists between the previously stored node bundles;

If it is determined that the newly acquired node bundle exists between the previously stored node bundles, extracting only the newly acquired node bundle existing between the existing acquired node bundles; And

And selecting only the extracted newly acquired node bundle as the node sequence.