KR20210117797A - Server and method for generating map based on object data and robot moving based on object map - Google Patents

Abstract

The present invention relates to a server for generating a map based on object information, which comprises: a pose information collecting unit for collecting pose information at a specific location within a specific place from a moving robot traveling in the specific location; a basic pose graph generating unit for generating a basic pose graph including a plurality of basic nodes located in the specific place on the basis of the collected pose information on the moving robot and at least one basic edge for connecting the basic nodes; an image receiving unit for receiving an image photographed by a camera from the moving robot; a layered pose graph generating unit for generating at least one layered pose graph for an object located in the specific place on the basis of the received image and the basic pose graph; and a map generating unit for generating a map on the specific place on the basis of the basic pose graph and the layered pose graph. Therefore, semantic-based spatial intelligence is applied to a plurality of moving robots.

Description

A server and method for generating a map based on object information, and a mobile robot that drives according to the map

The present invention relates to a server and method for generating a map based on object information, and to a mobile robot that runs according to the map.

SLAM (Simulation Language for Alternative Modeling) is an event-oriented or process-oriented simulation language, and is a language that expresses a system as an interconnected network structure using standard symbols and branches of nodes. Models of SLAM include discrete models using events or processes, continuous models including differential equations, and mixed models using all factors.

Recently, SLAM technology has been applied to mobile robots, allowing the mobile robot to move around an unknown environment and create an accurate map of the environment without external help only with sensors attached to the mobile robot. have.

In relation to this SLAM technology, Korean Patent Registration No. 10-1739996, which is a prior art, discloses a mobile robot and a method for recognizing a location and creating a map of the mobile robot.

Conventional SLAM technology focused only on 'driving', so driving was performed regardless of 'meaning'. The meaning-based driving technology refers to a driving technology that calculates what an object is in the surrounding environment and performs an action on it, rather than calculating the similarity of the surrounding environment and detecting obstacles.

However, in the case of meaning-based driving technology, it has a disadvantage in that it is difficult to detect changes in the surrounding environment other than the path to be driven by the mobile robot, and to respond according to the detected environmental change. In addition, when an object irrelevant to the driving of the mobile robot is created and destroyed after the map is generated, it has a disadvantage in that it is difficult for the mobile robot to respond.

Collecting pose information at a specific location in a specific location from a mobile robot traveling in a specific location, and at least connecting a plurality of basic nodes located in a specific location and a plurality of basic nodes based on the collected pose information of the mobile robot It is intended to provide a server, a method and a mobile robot for generating a basic pose graph including one basic edge.

An object of the present invention is to provide a server, a method, and a mobile robot that receive an image captured by a camera from a mobile robot and generate at least one layered pose graph for an object located in a specific place based on the received image and a basic pose graph.

An object of the present invention is to provide a server, a method, and a mobile robot that generate a map for a specific place based on a basic pose graph and at least one layered pose graph.

However, the technical problems to be achieved by the present embodiment are not limited to the technical problems described above, and other technical problems may exist.

As a means for achieving the above-described technical problem, an embodiment of the present invention provides a pose information collecting unit that collects pose information at a specific location within the specific location from a mobile robot traveling in a specific location, the collection A basic pose graph for generating a basic pose graph including a plurality of basic nodes located in the specific place and at least one basic edge connecting between the plurality of basic nodes based on the pose information of the mobile robot A generation unit, an image receiving unit that receives an image captured by a camera from the mobile robot, and a layered pose generating at least one layered pose graph for an object located in the specific place based on the received image and the basic pose graph It is possible to provide a map generating server including a graph generating unit and a map generating unit generating a map for the specific place based on the basic pose graph and the at least one layered pose graph.

Another embodiment of the present invention provides a measuring unit that measures pose information at a specific position with respect to a mobile robot traveling in a specific place, a photographing unit that takes an image of the specific position using a camera, and the measured specific position a transmission unit that transmits the pose information and the captured image to a map generating server in a plurality of base nodes and the plurality of base nodes located in a specific place based on the pose information of the mobile robot, comprising: a receiving unit receiving a map for A basic pose graph including at least one basic edge connecting between nodes is generated, and at least one layered pose graph for an object located in the specific place is generated based on the image and the basic pose graph and, based on the basic pose graph and the at least one layered pose graph, it is possible to provide a mobile robot in which a map for the specific place is generated.

Another embodiment of the present invention includes the steps of collecting pose information at a specific location within the specific location from a mobile robot traveling in a specific location, and within the specific location based on the collected pose information of the mobile robot. generating a basic pose graph including a plurality of located basic nodes and at least one basic edge connecting the plurality of basic nodes; receiving an image captured by a camera from the mobile robot , generating at least one layered pose graph for an object located in the specific place based on the received image and the basic pose graph, and the specific location based on the basic pose graph and the at least one layered pose graph and generating a map for

The above-described problem solving means are merely exemplary, and should not be construed as limiting the present invention. In addition to the exemplary embodiments described above, there may be additional embodiments described in the drawings and detailed description.

According to any one of the above-described problem solving means of the present invention, it is possible to provide a server, a method, and a mobile robot that allow the mobile robot to travel through a map generated based on the basic pose graph and the layered pose graph in a specific place. .

It is possible to provide a server, a method, and a mobile robot that generate at least one layered pose graph for each object or each color of an object based on the basic pose graph and provide control information for driving based on the layered pose graph.

When a control command for a specific object is received from a user, a server, a method, and a mobile robot that provide control information for reaching a specific object to a mobile robot based on the control command for the specific object may be provided.

To provide a server, method, and mobile robot that not only utilizes mobile robots in the field of spatial intelligence using meaning-based driving technology, but also applies meaning-based spatial intelligence to multiple mobile robots through the application of server-based spatial intelligence. can

1 is a block diagram of a map generating system according to an embodiment of the present invention.
2 is a block diagram of a map generating server according to an embodiment of the present invention.
3 is an exemplary diagram illustrating a basic pose graph according to an embodiment of the present invention.
4A and 4B are exemplary views for explaining a process of generating a layered pose graph for each object according to an embodiment of the present invention.
5 is an exemplary diagram for explaining a process of updating a layered pose graph according to an embodiment of the present invention.
6 is an exemplary view for explaining a process of deriving control information related to a path to be driven by the mobile robot according to an embodiment of the present invention.
7 is a flowchart of a method of generating a map based on object information in a map generating server according to an embodiment of the present invention.
8 is a block diagram of a mobile robot according to an embodiment of the present invention.
9 is a flowchart of a method of driving according to a map generated based on object information in a mobile robot according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art can easily implement them. However, the present invention may be embodied in many different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part is "connected" with another part, this includes not only the case of being "directly connected" but also the case of being "electrically connected" with another element interposed therebetween. . Also, when a part "includes" a component, it means that other components may be further included, rather than excluding other components, unless otherwise stated, and one or more other features However, it is to be understood that the existence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded in advance.

In this specification, a "part" includes a unit realized by hardware, a unit realized by software, and a unit realized using both. In addition, one unit may be implemented using two or more hardware, and two or more units may be implemented by one hardware.

Some of the operations or functions described as being performed by the terminal or device in the present specification may be instead performed by a server connected to the terminal or device. Similarly, some of the operations or functions described as being performed by the server may also be performed in a terminal or device connected to the server.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram of a map generating system according to an embodiment of the present invention. Referring to FIG. 1 , a map generating system 1 may include a map generating server 110 and a mobile robot 120 . The map generating server 110 and the mobile robot 120 exemplarily show components that can be controlled by the map generating system 1 .

Each component of the map generating system 1 of FIG. 1 is generally connected through a network. For example, as shown in FIG. 1 , the map generating server 110 may be connected to the mobile robot 120 at the same time or at a time interval.

A network refers to a connection structure that enables information exchange between each node, such as terminals and servers, and includes a local area network (LAN), a wide area network (WAN), and the Internet (WWW: World). Wide Web), wired and wireless data communication networks, telephone networks, wired and wireless television networks, and the like. Examples of wireless data communication networks include 3G, 4G, 5G, 3rd Generation Partnership Project (3GPP), Long Term Evolution (LTE), World Interoperability for Microwave Access (WIMAX), Wi-Fi, Bluetooth communication, infrared communication, ultrasound Communication, Visible Light Communication (VLC), LiFi, etc. are included, but are not limited thereto.

The map generation server 110 may collect pose information at a specific location within the specific location from the mobile robot 120 traveling in the specific location 130 .

The map generation server 110 provides a basic pose graph ( pose graph) can be created.

The map generation server 110 may receive an image captured by a camera from the mobile robot 120 .

The map generation server 110 may recognize an object located in a specific place from the received image using a predefined object dictionary, and extract color information related to the recognized object.

The map generation server 110 may generate at least one layered pose graph for an object located in a specific place based on the received image and the basic pose graph. For example, the map generation server 110 may generate a layer corresponding to the extracted object by reflecting color information related to the extracted object. Specifically, the map generation server 110 may generate a layer corresponding to the extracted object by using color information assigned to a class of the extracted object. Here, the layered pose graph may include a plurality of sub-nodes and at least one sub-edge connecting the plurality of sub-nodes based on the plurality of basic nodes and at least one basic edge.

The map generation server 110 may generate at least one layered pose graph on a layer based on whether the same class as the object class exists in at least two or more frames included in the image.

For example, when an object of the same class as the class of the object exists, the map generation server 110 connects between the first sub-node and the first sub-node in which the existence of the object is reflected on the layer based on the basic pose graph. A connecting first sub-edge may be generated.

For another example, when another object of a class different from the class of the object exists, the map generation server 110 may generate a new layer corresponding to the other object by reflecting color information related to the other object. For example, the map generation server 110 may generate a layer corresponding to another object by using color information assigned to a class of the other object. Thereafter, the map generation server 110 may generate a second sub-node connecting the second sub-node and the second sub-node in which the existence of another object is reflected on the new layer created based on the basic pose graph. .

The map generation server 110 may generate a map for a specific place based on the basic pose graph and at least one layered pose graph. In this case, the map generation server 110 may perform optimization such that at least one sub-node and sub-edge unnecessary for the generated at least one layered pose graph are removed.

The map generating server 110 may provide the mobile robot 120 with control information related to the path on which the mobile robot 120 will travel based on the generated map for a specific place.

When the map generating server 110 receives the control command input from the mobile robot 120, the map generation server 110 derives a specific object and color information related to the specific object from the received control command, and corresponds to the derived specific object-related color information. layers can be derived. For example, the map generation server 110 may derive a layer corresponding to color information assigned to a class of a specific extracted object. Thereafter, the map generation server 110 may derive a list of a specific sub-node included in a layer corresponding to color information related to a specific object and a specific sub-edge connecting between the specific sub-nodes.

The map generation server 110 may generate control information corresponding to a path for reaching a specific object based on the derived list, and may provide the generated control information to the mobile robot 120 .

The mobile robot 120 may measure pose information at a specific location with respect to the mobile robot 120 traveling in a specific location. For example, the mobile robot 120 scans a space in a specific place 130 using a LiDAR sensor built in the mobile robot 120 , and the mobile robot 120 detects the scanned space. Pose information including two-dimensional coordinates and an angle at a specific location can be measured. The LiDAR sensor is a sensor that can detect the distance to an object and various physical properties by shining a laser on the target. Such a lidar sensor is used for autonomous driving of the mobile robot 120 .

The mobile robot 120 may take an image of a specific location using a camera.

The mobile robot 120 may transmit the measured pose information and the captured image at the specific location to the map generating server 110 .

The mobile robot 120 may receive, from the map generation server 110 , a map for a specific place generated based on pose information at a specific location and object information included in an image.

The mobile robot 120 may receive control information related to a path on which the mobile robot 120 will travel based on a map for a specific place generated from the map generating server 110 .

The mobile robot 120 may receive a control command for a specific object from a user.

The mobile robot 120 may transmit the input control command to the map generation server 110 . In this case, the mobile robot 120 may receive control information related to a specific object based on a control command from the map generation server 110 .

The mobile robot 120 may control to travel in a specific place based on the received map. For example, the mobile robot 120 may be controlled to travel to a specific object in a specific place 130 .

The mobile robot 120 may be configured to include a lidar sensor, an ultrasonic sensor, a depth camera, an IMU, a 5G modem, a wheel encoder, a motor, and the like.

2 is a block diagram of a map generating server according to an embodiment of the present invention. Referring to FIG. 2 , the map generation server 110 includes a pose information collection unit 210 , a basic pose graph generation unit 220 , an image reception unit 230 , an object extraction unit 240 , and a layered pose graph generation unit 250 . ), a map generation unit 260 , a control information providing unit 270 , a control command receiving unit 280 , and a derivation unit 290 .

The pose information collecting unit 210 may collect pose information at a specific location within the specific location 130 from the mobile robot 120 traveling in the specific location 130 . Here, the pose information may include coordinate information and direction information of the mobile robot 120 at a specific location. In the pose information Pose=(x, h), for example, x may represent a point in a three-dimensional real space, and h may represent a Hamilton quaternary, so that a three-dimensional direction may be indicated.

The basic pose graph generator 220 includes a plurality of basic nodes located in a specific place 130 based on the collected pose information of the mobile robot 120 and at least one basic edge connecting the plurality of basic nodes. You can create a basic pose graph that does The process of generating the basic pose graph will be described in detail with reference to FIG. 3 .

3 is an exemplary diagram illustrating a basic pose graph according to an embodiment of the present invention. Referring to FIG. 3 , the basic pose graph generating unit 220 includes a plurality of basic nodes located in a specific place 130 based on the pose information and at least one basic edge connecting between the plurality of basic nodes. The pose graph 300 may be generated.

For example, the basic pose graph generating unit 220 may generate a first basic node ( 301 ) and the second basic node 302 , and a first basic edge 310 connecting between the first basic node 301 and the second basic node 302 may be generated. By performing this process, the basic pose graph generating unit 220 generates the basic pose graph 300 as a geographic graph indicating an area in which the mobile robot 120 can travel within a specific place 130 . can

The pose of the basic pose graph 300 may represent a movement path for detection of the surrounding environment at each position and direction. In addition, the basic pose graph 300 has a characteristic that is instantaneously invariant as it is indicated whether it is recognized as the same place along the basic edge.

Returning to FIG. 2 , the image receiving unit 230 may receive an image captured by the camera from the mobile robot 120 .

The object extraction unit 240 may recognize an object located in a specific place from the received image using a predefined object dictionary, and extract color information related to the recognized object. For example, the object extraction unit 240 may extract color information assigned to a class of the recognized object.

The layered pose graph generator 250 may generate at least one layered pose graph for an object located in the specific place 130 based on the received image and the basic pose graph. Here, the layered pose graph may include a plurality of sub-nodes and at least one sub-edge connecting the plurality of sub-nodes based on the plurality of basic nodes and at least one basic edge.

The layered pose graph generator 250 may generate at least one layered pose graph on a layer based on whether the same class as the object class exists in at least two or more frames included in the image. Here, one sub-edge is recognized as the same position in a specific place 130 through at least two key frames (for example, a frame at a specific time point in which a specific object exists in the image and disappears), the mobile robot 120 Due to this physical property, the object can be continuously observed while following the sub-edge. In addition, as the mobile robot 120 moves vertically through each layer, semantic-based map information is sequentially accumulated, so that the mobile robot 120 understands the flow of meaning for the object-based map. be able to

If seven classes of different objects are detected in at least two or more frames included in the image, the layered pose graph generating unit 250 additionally creates seven layers on the lowest layer corresponding to the basic pose graph. You can create a layered pose graph for each layer. In this case, the map may be composed of a total of 8 layers including the basic pose graph and 7 layered pose graphs.

That is, the layered pose graph generating unit 250 may generate a layered pose graph after generating a layer for each class based on the number of classes. Here, since the maximum number of classes may be set, N points of each layer corresponding to the base node may also be maximum. For example, when two classes are recognized at a location corresponding to the A base node, two nodes may be generated for each layer at a location corresponding to the A base node.

The layered pose graph generating unit 250 connects between the first sub-node and the first sub-node in which the existence of the object is reflected on the layer based on the basic pose graph when an object of the same class as the class of the object exists. A first sub-edge may be generated.

When an object (eg, a pencil) corresponding to a specific class exists while the mobile robot 120 is driving, the layered pose graph generating unit 250 generates a first sub-node on a layer corresponding to the corresponding class. can do. Thereafter, as the mobile robot 120 travels, the pencil was recognized before, but if there is a key frame in which the object disappears, such as the pencil does not exist thereafter, the layered pose graph generation unit 250 is displayed on the corresponding layer. No more first sub-nodes may be created in .

When an object corresponding to the same class is recognized by the two first sub-nodes, the layered pose graph generator 250 may connect the two first sub-nodes to the first sub-edges.

If a class can be found using two first sub-nodes even if co-visibility does not exist within a specific place 130, the layered pose graph generator 250 moves to the first sub-edge. By connecting, it is possible to create a semantic graph different from the basic pose graph. Here, the semantic graph is not generated once, unlike the basic pose graph, and may be initially generated when the initial map is generated.

When another object of a class different from the class of the object exists, the layered pose graph generator 250 may generate a new layer corresponding to the other object by reflecting color information related to the other object. For example, the layered pose graph generator 250 may generate a new layer corresponding to another object by using color information assigned to a class of the other object.

The layered pose graph generator 250 may generate a second sub-node in which the existence of another object is reflected on a new layer generated based on the basic pose graph and a second sub-edge connecting between the second sub-nodes. .

The process of generating the layered pose graph for each object will be described in detail with reference to FIGS. 4A and 4B .

4A and 4B are exemplary views for explaining a process of generating a layered pose graph for each object according to an embodiment of the present invention.

Referring to FIG. 4A , the basic pose graph generating unit 220 includes a plurality of basic nodes located in a specific place 130 and at least one basic edge connecting the plurality of basic nodes to the basic pose graph 400 . can create In this case, the basic pose graph generator 220 may represent each basic node and each basic edge of the basic pose graph 400 in 'black'.

The object extraction unit 240 recognizes a first object located in a specific place from the image as a ballpoint pen, and the layered pose graph generation unit 250 reflects the color information related to the extracted first object to correspond to the extracted first object. layer can be created. For example, when the color information assigned to the class of the ballpoint pen is 'green', the first layer for the 'ballpoint pen' may be generated so that 'green' is reflected in the sub-node and sub-edge.

When an object of the same class as the class of the first object corresponding to the 'ballpoint pen' exists, the layered pose graph generation unit 250 reflects the existence of the first object on the layer based on the basic pose graph 400. A first sub-edge connecting the first sub-node and the first sub-node may be generated. For example, the layered pose graph generating unit 250 determines whether an object of the same class as the first object 'ballpoint pen' exists on the first layer 410 based on the basic pose graph. It is possible to generate a first sub-node reflecting the existence of a first sub-node and a first sub-edge connecting between the first sub-nodes.

When a second object corresponding to a class different from that of the first object corresponding to the 'ballpoint pen' exists, the layered pose graph generating unit 250 reflects color information related to the second object to correspond to the second object. A second layer 420 that is a new layer may be created. For example, the object extraction unit 240 recognizes the second object located in the specific place 130 from the image as a 'note', and the layered pose graph generation unit 250 obtains color information related to the extracted second object. A layer corresponding to the first object extracted by reflection may be generated. For example, when the color information assigned to the class of the note is 'red', the second layer 420 for the 'note' may be created so that 'red' is reflected in the sub node and the sub edge. .

The layered pose graph generating unit 250 is configured to connect a second sub-node in which the existence of the second object is reflected on the second layer 420 generated based on the basic pose graph 400 and the second sub-node to the second sub-node. You can create 2 sub-edges.

Through this process, the layered pose graph generator 250 places the basic pose graph at the bottom as the 0th layer, and places the first layer 410 in which the 'green' color is reflected on the 0th layer, and the first The second layer 420 in which the 'red' color is reflected may be positioned on the layer 410 .

Referring to FIG. 4B , in the 0th layer corresponding to the basic pose graph, the basic edge has a property of co-visibility through at least two or more frames. Due to this property, according to the class of each object, both the corresponding object is visible at both nodes of the corresponding object, so that the color of the same object can be expressed in both nodes. For example, if the first object corresponding to the ballpoint pen is located at F3 in the first layer 410, as both the first objects are visible at both nodes F2 and F4 of F3, the color of the first object is at both nodes. can be expressed For another example, when the second object corresponding to the note is located at F2 in the second layer 420, the color of the second object changes to both nodes as the second object is visible at both nodes F1 and F3 of F2. can be expressed in Here, a plurality of components connected from a specific basic node through each layer may exist.

However, when the key frame is changed according to the movement of the mobile robot 120, the object disappears from the image, so that each layer can be expressed in a color of each layer in the form of a partial graph unconnected from the 0th layer. Here, when the generated partial graph is projected on the uppermost layer, the basic node can be seen.

Returning to FIG. 2 again, the map generator 260 may generate a map for a specific place 130 based on the basic pose graph and at least one layered pose graph. The map generator 260 may generate a map by estimating pose information and color information of the mobile robot 120 using Equation 1 below.

Referring to Equation 1, since the map generator 260 handles color information in an independent space by adding one dimension to the existing vector space, the mobile robot ( 120), a map may be generated by estimating the pose information and color information.

The map generator 260 may update the sub-node and sub-edge by detecting a change in the surrounding environment from the image captured by the mobile robot 120 while driving.

5 is an exemplary diagram for explaining a process of updating a layered pose graph according to an embodiment of the present invention.

Referring to FIG. 5 , the layered pose graph generator 250 may generate an initial layered pose graph 500 for 'pencil'.

Thereafter, when the 'pencil' is newly recognized at the position corresponding to the sub-node A 510, the layered pose graph generating unit 250 updates the map by additionally connecting the sub-edges between the sub-node A 510 and the neighboring node. can do.

The map generator 260 may update the map based on the basic pose graph and the updated layered pose graph.

Then, when the layered pose graph generation unit 250 recognizes the disappearance of the 'pencil' at the position corresponding to the sub-node B 520, the sub-edge between the sub-node B 520 and the neighboring sub-node is deleted to create a map. can be updated.

The map generator 260 may additionally update the map based on the basic pose graph and the additionally updated layered pose graph.

In the conventional map generation technology, when there is no problem in the driving of the mobile robot 120 after the map is generated, the map is not regenerated or updated.

However, in the present invention, the mobile robot 120 runs to scan the upper layer, and the sub-node of the upper layer matches the base node of the basic pose graph corresponding to the lowest layer, so the upper layer can be updated at any time. has a

Through this process, the map generator 260 checks the state of the semantic-based map while moving the layer horizontally or vertically along the connected component according to the driving of the mobile robot 120 , and updates it. can do. Here, the state of each layer is stored in a memory and a recordable HDD, and as the mobile robot 120 moves, a change in the corresponding node may be recognized.

The map generator 260 may perform optimization such that at least one unnecessary sub-node and sub-edge are removed from the generated at least one layered pose graph. For example, the map generator 260 may perform optimization for error correction by loop closure and deletion of unnecessary sub-nodes and sub-edges. In this case, the graph error caused by the judgment error of the same place may be improved through re-examination of the semantic segmentation layer.

Referring briefly to FIG. 4B , the map generator 260 uses the layered pose graph corresponding to the first layer 410 and the layered pose graph corresponding to the second layer 420 to check the same point according to the object comparison. can

At this time, the graphs of the first layer 410 and the second layer 420 may be sequentially checked, and the leftmost frame of the first layer 410 in which green is reflected is not visible, but on the contrary, the second frame in which red is reflected is not visible. The rightmost node of the layer 420 may not be visible.

Through this process, reexamination of each layer may be performed for error correction.

The control information providing unit 270 may provide the mobile robot 120 with control information related to a path on which the mobile robot 120 will travel based on the generated map for a specific place. For example, the control information providing unit 270 may provide the mobile robot 120 with control information related to the path on which the mobile robot 120 will travel based on the basic pose graph. As another example, the control information providing unit 270 may provide the mobile robot 120 with control information related to a path that the mobile robot 120 will travel to reach a specific object based on the layered pose graph. A process of providing control information will be described in detail with reference to FIG. 6 .

6 is an exemplary view for explaining a process of deriving control information related to a path to be driven by the mobile robot according to an embodiment of the present invention.

The control command receiving unit 280 may receive a control command input from the mobile robot 120 . For example, the control command receiving unit 280 may receive a control command, “Robot, come to the ballpoint pen” from the mobile robot 120 .

The derivation unit 290 may derive color information corresponding to a specific object and a class of the specific object from the received control command, and may derive a layer corresponding to the derived color information related to the specific object. For example, the derivation unit 290 derives the object included in the control command as 'ballpoint pen' and derives the color corresponding to the class of 'ballpoint pen' as 'red', so that the color of 'red' is A reflected layer can be derived.

The derivation unit 290 may derive a list of a specific sub-node included in a layer corresponding to color information related to a specific object and a specific sub-edge connecting between the specific sub-nodes. For example, the derivation unit 290 derives a list of a specific sub-node corresponding to the 'red' layer and a specific sub-edge connecting between the specific sub-nodes, and the shortest distance node from the derived list to the 'ballpoint pen' can be derived.

Referring to FIG. 6 , the derivation unit 290 may derive a movement path for the mobile robot 120 to reach the ballpoint pen based on the actual position of the ballpoint pen recognized by the mobile robot 120 . For example, α and β may be derived using q for _{pose (x 0} , y ₀ , z ₀ , q) from the current position 600 of the mobile robot 120 using a spherical coordinate system. In this case, the pose 610 of the ballpoint pen may be (x, y, z, q).

Through this process, the control information providing unit 270 generates control information corresponding to the path 620 for reaching a specific object based on the derived list, and provides the generated control information to the mobile robot 120 . can do. For example, the control information providing unit 270 may generate control information corresponding to the path 620 of the shortest node to the derived 'ballpoint pen', and provide the generated control information to the mobile robot 120 . have.

7 is a flowchart of a method of generating a map based on object information in a map generating server according to an embodiment of the present invention. The method for generating a map based on object information in the map generating server 110 shown in FIG. 7 includes steps processed in time series by the map generating system 1 according to the embodiment shown in FIGS. 1 to 6 . include Accordingly, even if omitted below, the method for generating a map based on object information in the map generating server 110 according to the embodiment shown in FIGS. 1 to 6 is also applied.

In step S710 , the map generation server 110 may collect pose information at a specific location within a specific location from the mobile robot 120 traveling in a specific location.

In step S720, the map generation server 110 provides a basic pose graph ( pose graph) can be created.

In step S730 , the map generation server 110 may receive an image captured by the camera from the mobile robot 120 .

In operation S740, the map generation server 110 may generate at least one layered pose graph for an object located in a specific place based on the received image and the basic pose graph.

In step S750, the map generation server 110 may generate a map for a specific place based on the basic pose graph and at least one layered pose graph.

In the above description, steps S710 to S750 may be further divided into additional steps or combined into fewer steps, according to an embodiment of the present invention. In addition, some steps may be omitted if necessary, and the order between the steps may be switched.

8 is a block diagram of a mobile robot according to an embodiment of the present invention. Referring to FIG. 7 , the mobile robot 120 may include a measuring unit 810 , a photographing unit 820 , a transmitting unit 830 , a receiving unit 840 , a control unit 850 , and an input unit 860 .

The measurement unit 810 may measure pose information at a specific location with respect to the mobile robot 120 traveling in a specific location. For example, the measurement unit 810 scans a space in a specific place 130 using a lidar sensor provided in the mobile robot 120 , and poses including coordinates and angles in two dimensions with respect to the scanned space. information can be measured.

The photographing unit 820 may photograph an image of a specific location using a camera.

The transmitter 830 may transmit the measured pose information and the captured image at the specific location to the map generating server 110 .

The receiver 840 may receive, from the map generating server 110 , a map for a specific place generated based on pose information at a specific location and object information included in an image. Here, the map for the specific place 130 includes a plurality of basic nodes located in the specific place 130 based on the pose information of the mobile robot 120 and at least one basic edge connecting the plurality of basic nodes. It may be generated based on a basic pose graph and at least one layered pose graph for an object located in a specific place 130 .

The controller 850 may control to drive in a specific place based on the received map. For example, the controller 850 may control the mobile robot 120 to drive the first node and the second node in one or both directions based on the basic edge connected between the first node and the second node. As another example, the controller 850 may control the mobile robot 120 to drive the first sub-node and the second sub-node in one or both directions based on the sub-edge connected between the first and second sub-nodes. can

The input unit 860 may receive a control command for a specific object from the user. For example, the input unit 860 may receive a control command such as “come to the ballpoint pen” from the user.

The transmitter 830 may transmit the input control command to the map generating server 110 .

The receiver 840 may receive control information related to a path on which the mobile robot 120 will travel based on the map for a specific place generated from the map generating server 110 .

The controller 850 may control the mobile robot 120 to travel in a specific place based on the received control information. For example, when a control command such as “come to the ballpoint pen” is received from the input unit 860 , the receiving unit 840 receives the control information including the driving path derived from the layer related to the 'ballpoint pen' from the map generation server 110 . , and the controller 850 may control the mobile robot 120 to run toward the 'ballpoint pen' based on the control information.

9 is a flowchart of a method of driving according to a map generated based on object information in a mobile robot according to an embodiment of the present invention. The method of driving according to the map generated based on the object information in the mobile robot 120 shown in FIG. 9 is time-series processed by the map generating system 1 according to the embodiment shown in FIGS. 1 to 8 . includes steps. Accordingly, even if omitted below, the method of driving according to the map generated based on the object information in the mobile robot 120 according to the embodiment shown in FIGS. 1 to 8 is also applied.

In step S910 , the mobile robot 120 may measure pose information at a specific location with respect to the mobile robot 120 traveling in a specific location.

In step S920 , the mobile robot 120 may capture an image of a specific location using a camera.

In step S930 , the mobile robot 120 may transmit the measured pose information and the captured image at the specific location to the map generating server 110 .

In operation S940 , the mobile robot 120 may receive a map for a specific place generated based on pose information at a specific location and object information included in the image from the map generating server 110 .

In step S950 , the mobile robot 120 may control to travel in a specific place based on the received map.

In the above description, steps S910 to S950 may be further divided into additional steps or combined into fewer steps, according to an embodiment of the present invention. In addition, some steps may be omitted if necessary, and the order between the steps may be switched.

The method for generating a map based on the object information in the map generating server described through FIGS. 1 to 9 and the method for driving according to the map generated based on the object information in the mobile robot are a computer stored in a medium executed by a computer. It may also be implemented in the form of a recording medium including a program or instructions executable by a computer. In addition, the method of generating a map based on the object information in the map generating server described through FIGS. 1 to 9 and the method of driving according to the map generated based on the object information in the mobile robot are in a medium executed by a computer. It may also be implemented in the form of a stored computer program.

Computer-readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. Also, computer-readable media may include computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.

The above description of the present invention is for illustration, and those of ordinary skill in the art to which the present invention pertains can understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a dispersed form, and likewise components described as distributed may be implemented in a combined form.

The scope of the present invention is indicated by the following claims rather than the above detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. do.

110: map generation server
120: mobile robot
210: pose information collection unit
220: basic pose graph generator
230: video receiver
240: object extraction unit
250: Layered pose graph generation unit
260: map generator
270: control information providing unit
280: control command receiving unit
290: derivation part
810: measurement unit
820: filming unit
830: transmission unit
840: receiver
850: control unit
860: input unit

Claims (20)

In the server for generating a map based on object information,
a pose information collecting unit that collects pose information at a specific location within the specific location from a mobile robot traveling in a specific location;
Basic for generating a basic pose graph including a plurality of basic nodes located in the specific place and at least one basic edge connecting between the plurality of basic nodes based on the collected pose information of the mobile robot pose graph generator;
an image receiving unit for receiving an image captured by a camera from the mobile robot;
a layered pose graph generator configured to generate at least one layered pose graph for an object located in the specific place based on the received image and the basic pose graph; and
A map generator for generating a map for the specific place based on the basic pose graph and the at least one layered pose graph
Which includes, the map generation server.
The method of claim 1,
The layered pose graph generating unit generates a layered pose graph including a plurality of sub-nodes and at least one sub-edge connecting between the plurality of sub-nodes based on the plurality of basic nodes and the at least one basic edge In, map generation server.
3. The method of claim 2,
Further comprising an object extraction unit for recognizing an object located in the specific place from the received image using a predefined object dictionary, and extracting color information related to the recognized object,
The layered pose graph generation unit to generate a layer corresponding to the extracted object by reflecting the color information related to the extracted object, map generation server.
4. The method of claim 3,
The layered pose graph generating unit generates the at least one layered pose graph on the layer based on whether an object of the same class as the class of the object exists in at least two or more frames included in the image. creation server.
5. The method of claim 4,
When an object of the same class as the class of the object exists, the layered pose graph generating unit connects between a first sub-node in which the existence of the object is reflected on the layer based on the basic pose graph and the first sub-node A map generating server that generates a first sub-edge to connect.
6. The method of claim 5,
The layered pose graph generating unit creates a new layer corresponding to the other object by reflecting color information related to the other object when there is another object of a class different from the class of the object, and based on the basic pose graph, and generating a second sub-node in which the existence of the other object is reflected on the created new layer and a second sub-edge connecting the second sub-node.
The method of claim 1,
The map generation server of claim 1, further comprising a control information providing unit providing control information related to a path to be driven by the mobile robot to the mobile robot based on the generated map for the specific place.
8. The method of claim 7,
a control command receiving unit for receiving a control command from the mobile robot; and
and a derivation unit for deriving a specific object and color information related to the specific object from the received control command, and deriving a layer corresponding to the derived color information related to the specific object.
9. The method of claim 8,
The derivation unit derives a list of a specific sub-node included in a layer corresponding to the color information related to the specific object and a specific sub-edge connecting between the specific sub-node,
The control information providing unit generates control information corresponding to a path for reaching the specific object based on the derived list, and provides the generated control information to the mobile robot.
3. The method of claim 2,
The map generation server of claim 1, wherein the map generation unit optimizes the at least one sub-node and sub-edge unnecessary for the generated at least one layered pose graph.
In a mobile robot that travels along a map generated based on object information,
a measurement unit for measuring pose information at a specific location with respect to the mobile robot traveling in a specific location;
a photographing unit for photographing an image of the specific location using a camera;
a transmission unit for transmitting the measured pose information and the photographed image at the specific location to a map generating server;
a receiver configured to receive a map for the specific place generated based on the pose information at the specific location and the object information included in the image from the map generating server; and
A control unit for controlling to drive the specific place based on the received map,
A basic pose graph including a plurality of basic nodes located in a specific place and at least one basic edge connecting between the plurality of basic nodes is generated based on the pose information of the mobile robot,
At least one layered pose graph for an object located in the specific place is generated based on the image and the basic pose graph,
The mobile robot, wherein the map for the specific place is generated based on the basic pose graph and the at least one layered pose graph.
12. The method of claim 11,
Further comprising an input unit for receiving a control command for a specific object,
The transmitting unit will transmit the input control command to the map generating server, the mobile robot.
13. The method of claim 12,
Wherein the receiving unit receives the control information related to the path to be driven by the mobile robot based on the generated map for the specific place from the map generating server, the mobile robot.
A method for generating a map based on object information in a map generating server, the method comprising:
collecting pose information at a specific location within the specific location from a mobile robot traveling in a specific location;
Generating a basic pose graph including a plurality of basic nodes located in the specific place and at least one basic edge connecting between the plurality of basic nodes based on the collected pose information of the mobile robot ;
receiving an image captured by a camera from the mobile robot;
generating at least one layered pose graph for an object located in the specific place based on the received image and the basic pose graph; and
generating a map for the specific place based on the basic pose graph and the at least one layered pose graph
A method for generating maps, comprising:
15. The method of claim 14,
The step of generating the layered pose graph comprises:
Generating a layered pose graph including a plurality of sub-nodes and at least one sub-edge connecting between the plurality of sub-nodes based on the plurality of basic nodes and the at least one basic edge, How to create a map.
15. The method of claim 14,
recognizing an object located in the specific place from the received image using a predefined object dictionary, and extracting color information related to the recognized object; and
The method further comprising the step of generating a layer corresponding to the extracted object by reflecting color information related to the extracted object.
14. The method of claim 13,
The step of generating the layered pose graph comprises:
The method comprising the step of generating the at least one layered pose graph on the layer based on whether an object of the same class as the class of the object exists in at least two or more frames included in the image .
18. The method of claim 17,
The step of generating the layered pose graph comprises:
When an object of the same class as the object class exists, a first sub-node in which the existence of the object is reflected on the layer based on the basic pose graph and a first sub-edge connecting the first sub-node generating a map.
18. The method of claim 17,
The step of generating the layered pose graph comprises:
generating a new layer corresponding to the other object by reflecting color information related to the other object when another object of a class different from the class of the object exists; and
and generating a second sub-node connecting the second sub-node and a second sub-node in which the existence of the other object is reflected on the created new layer based on the basic pose graph, How to create a map.
15. The method of claim 14,
The method further comprising the step of providing control information related to a path for the mobile robot to travel to the mobile robot based on the generated map for the specific place.

Images