KR20230053430A - Method of pose estimation and robot system using the same method - Google Patents

Abstract

The present invention relates to a method for measuring a pose of a robot and a robot system using the same. The method for measuring the pose of a robot according to the present invention comprises the steps of: moving at least one of a first robot and a second robot so that the first robot and the second robot are adjacent to each other; outputting a visual marker on a display of the second robot; photographing, by the first robot, the second robot; detecting the visual marker included in an image photographed by the first robot and calculating the pose of the visual marker included in the image; and calculating the pose of the second robot using coordinate data of the second robot and the pose of the visual marker included in the image. Thus, the present invention can provide the method and system for quickly and accurately measuring the pose of a robot.

Description

Method of pose estimation and robot system using the same method}

The present invention relates to a method for measuring a posture of a robot and a robot system using the same, and relates to a method for measuring a posture of an opponent robot for interaction between robots, and a robot system for performing the interaction.

As technology develops, various service devices are appearing, and in particular, technology development for robots performing various tasks or services has been actively conducted recently.

Furthermore, recently, with the development of artificial intelligence technology, cloud technology, etc., it has become possible to control robots more precisely and safely, and accordingly, the utilization of robots is gradually increasing. In particular, due to the development of technology, robots have reached a level where they can safely coexist with humans in an indoor space.

Accordingly, recently, robots have been replacing human tasks or tasks, and in particular, various methods for robots to directly provide services to people in indoor spaces are being actively researched.

For example, robots provide route guidance services in public places such as airports, stations, and department stores, and robots provide serving services in restaurants. Furthermore, robots are providing a delivery service in which mails and couriers are delivered in office spaces, co-living spaces, and the like. In addition, robots provide various services such as cleaning services, crime prevention services, and logistics handling services. The type and range of services provided by robots will increase exponentially in the future, and the level of service provision is expected to continue to evolve.

These robots provide various services not only in outdoor spaces but also in indoor spaces of buildings (or buildings) such as offices, apartments, department stores, schools, hospitals, amusement facilities, etc. In this case, robots provide It is controlled to provide various services while moving through the indoor space.

On the other hand, in order for robots to provide various services or live in an indoor space, there is a need for robots to perform tasks in collaboration with each other. For example, Republic of Korea Patent Publication No. 10-2010-0086093 (Self-driving Swarm Robot Position Control System) discloses a system in which a master robot autonomously navigates in a cluster together with a plurality of subordinate robots. As described above, with the gradual development of driving technology, recently, efforts to implement various services by controlling interaction between robots have been actively progressed.

Therefore, for robot interaction to provide a higher level of service using robots, essential research that can accurately and quickly measure the position or posture between robots is required.

The present invention provides a method and system for quickly and accurately measuring the posture of a robot.

More specifically, the present invention provides a method for measuring a posture of a robot using a display mounted on the robot and a visual marker.

In order to solve the problems described above, the method for measuring the posture of a robot and the robot system using the same according to the present invention use a process in which a visual marker is output to a display of an opponent robot and the robot photographs it to recognize the posture of the opponent robot. .

Specifically, the method for measuring the posture of a robot of the present invention includes the steps of moving at least one of the first robot and the second robot so that the first robot and the second robot are adjacent to each other, and a visual marker on the display of the second robot. outputting, photographing the second robot by the first robot, detecting a visual marker included in an image captured by the first robot, and determining a pose of the visual marker included in the image. and calculating the posture of the second robot using the coordinate data of the second robot and the posture of the visual marker included in the image.

In one embodiment of the present invention, the method of measuring the posture of the robot may further include monitoring a request signal requesting an output of the visual marker by the second robot.

The request signal may be transmitted from a server communicating with the first robot or from the first robot to the second robot.

The outputting of the visual marker may include turning off the output of the image information and outputting the visual marker or displaying the visual marker together with the image information when the request signal is received in a state in which image information is output to the display. can be printed out.

In the outputting of the visual marker, when the request signal is received in a state in which the display is deactivated, the display may be activated and the visual marker may be output.

On the other hand, the present invention is a step of moving a robot in a specific space where a display is arranged, a step of displaying image information on the display, and a request for the robot to request the output of a visual marker to a control system that controls the display. Transmitting a signal, controlling the display so that the control system outputs the visual marker in response to the request signal, and determining the current location of the robot by using the visual marker output on the display. Disclosed is a method for measuring a position of a robot including the step of estimating.

In addition, the present invention includes a first robot having a camera, interacting with the first robot, and including a second robot having a display, outputting a visual marker to the display of the second robot, A first robot photographs the second robot, detects a visual marker included in an image captured by the first robot, calculates a pose of the visual marker included in the image, and calculates the coordinates of the second robot. A plurality of pluralities characterized in that the posture of the second robot is estimated using data and the posture of the visual marker included in the image, and the operation of the first robot is controlled using the estimated posture of the second robot. Discloses a robot system in which a robot performs an interaction.

A method for measuring a posture of a robot and a robot system using the same according to the present invention detects a marker output on a display, measures the posture of the marker, compares the posture of the robot with the shape data of the robot, converts the posture into the posture of the robot, recognizes the posture of the other robot, , it is possible to quickly and accurately measure the posture of the opponent robot in a simple way.

In this way, since the accurate posture of the robot is quickly measured through the new process, human-level services are possible in various robot services, such as delivery, serving, and customer service.

In addition, since the present invention accurately and accurately measures the posture of the robot using the visual marker, it is possible to measure the posture of the other robot using a display for interaction with a human. In this case, by outputting a visual marker from a display fixed to a specific space, such as a signage, an effect of accurately and quickly measuring the position of a robot that has photographed the visual marker can be exerted.

1 is a conceptual diagram for explaining the interaction of robots according to the present invention.
2a and 2b each show an example of robots presented in the concept of FIG. 1 .
3 is a flowchart for explaining a method for measuring a posture of a robot according to the present invention.
FIG. 4 is a conceptual diagram illustrating the concept of the robot of FIG. 2a photographing a visual marker to measure the posture of the robot of FIG. 2b.
5 is a conceptual diagram for detecting a pose of a visual marker, and FIG. 6 is a conceptual diagram illustrating an example of transforming a posture of a robot.
7A and 7B are conceptual diagrams illustrating another embodiment of the present invention.
8 is image data representing actual data implemented using the process of the present invention.
9 is a conceptual diagram for measuring the position of a robot according to another embodiment of the present invention.

Hereinafter, the embodiments disclosed in this specification will be described in detail with reference to the accompanying drawings, but the same reference numerals will be assigned to the same or similar components regardless of reference numerals, and overlapping descriptions thereof will be omitted. The suffixes "module" and "unit" for components used in the following description are given or used together in consideration of ease of writing the specification, and do not have meanings or roles that are distinct from each other by themselves. In addition, in describing the embodiments disclosed in this specification, if it is determined that a detailed description of a related known technology may obscure the gist of the embodiment disclosed in this specification, the detailed description thereof will be omitted. In addition, the accompanying drawings are only for easy understanding of the embodiments disclosed in this specification, the technical idea disclosed in this specification is not limited by the accompanying drawings, and all changes included in the spirit and technical scope of the present invention , it should be understood to include equivalents or substitutes.

Terms including ordinal numbers, such as first and second, may be used to describe various components, but the components are not limited by the terms. These terms are only used for the purpose of distinguishing one component from another.

It is understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, but other elements may exist in the middle. It should be. On the other hand, when an element is referred to as “directly connected” or “directly connected” to another element, it should be understood that no other element exists in the middle.

Singular expressions include plural expressions unless the context clearly dictates otherwise.

In this application, terms such as "comprise" or "have" are intended to designate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or more other features It should be understood that the presence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded.

The present invention relates to a method for measuring a posture of a robot and a robot system using the same, and more specifically, to a method for measuring a posture of a robot for interaction between robots and a robot system for performing the interaction.

Here, interaction can be defined as a form of communication between humans and humans, humans and materials, humans and systems, and systems and systems. Specifically, it is a compound word of Inter (mutual) and action (action), which means an action in which two or more objects affect each other, and can mean all actions that have interactive and real-time characteristics of influence. .

In addition, in the present specification, a robot is i) a 　 machine with functions similar to those of a person, or 　ii) programmable with one or a plurality of 　 computer programs and automatically performing a complex series of actions. refers to mechanical devices. As another example, a robot may mean a machine that has a form and has the ability to think on its own. Furthermore, the robot described in this specification may be in the form of an android robot resembling a human in appearance and behavior, a humanoid robot having a body structure similar to a human, and a cyborg having a body part modified into a machine. can be implemented as

As technology advances, the use of robots is gradually increasing. Conventional robots have been utilized in special industrial fields (eg, industrial automation related fields), but are gradually transforming into service robots capable of performing useful tasks for humans or facilities.

In this case, the service provided by the robot may be implemented within a building. As such an example, the robots according to the present invention may be controlled based on at least one of a cloud server and a control unit provided in the robot itself, so as to travel in a building or provide a service corresponding to an assigned task.

The building according to the present invention is a building in which robots and people coexist, and the robots are made to avoid obstacles such as people, objects used by people (eg, strollers, carts, etc.), and animals. It may be configured to output notification information related to driving. The driving of the robot may be made to avoid an obstacle based on at least one of a cloud server and a control unit provided in the robot. The cloud server controls the robot so that the robot avoids obstacles and moves within the building based on information received through various sensors (eg, camera (image sensor), proximity sensor, infrared sensor, etc.) provided in the robot. can be performed. However, the present invention is not necessarily limited thereto, and the service or process described in the present invention may be applied outdoors.

Meanwhile, the robots of the present invention may be configured to provide services to people or target objects present in a building.

The type of service provided by the robot may be different for each robot. That is, various types of robots may exist depending on the purpose, the robot may have a different structure for each purpose, and a program suitable for the purpose may be installed in the robot.

For example, delivery to buildings, logistics, guidance, interpretation, parking assistance, security, crime prevention, security, security, cleaning, quarantine, disinfection, laundry, beverage preparation, food preparation, serving, fire suppression, medical assistance and entertainment services. Robots providing at least one of the services may be deployed. Services provided by robots may be various other than the examples listed above.

Meanwhile, the cloud server may assign an appropriate task to the robots in consideration of the use of each robot, and control the robots to perform the assigned task.

At least some of the robots described in the present invention may drive or perform missions under the control of the cloud server 20, and in this case, the amount of data processed by the robot itself to drive or perform missions may be minimized. there is. In the present invention, such a robot may be referred to as a brainless robot. These brainless robots may depend on the control of the cloud server for at least some control in performing actions such as driving, performing missions, performing charging, waiting, and cleaning within a building.

However, in this specification, the brainless robots are not classified and named, but all are unified and named as “robots”.

In addition, in the present invention, various infrastructures (or facility infrastructures) may be provided in buildings in order to implement services provided by robots.

Infrastructure or facilities Infrastructure is a facility provided in a building for service provision, robot movement, function maintenance, cleanliness maintenance, and the like, and its types and forms can be very diverse. For example, infrastructure provided in a building may be diverse, such as mobile facilities (eg, robot moving passages, elevators, escalators, etc.), charging facilities, communication facilities, cleaning facilities, structures (eg, stairs, etc.), and the like. there is. In this way, the infrastructure or facility infrastructure is defined as an infrastructure used by robots to perform assigned tasks.

In the present invention, the robots are made to cooperate with each other in order to perform assigned tasks. The collaboration is implemented through interaction between robots, and the present invention proposes a method for measuring the posture of a robot for this purpose.

As such an example, as shown in FIG. 1 , robots may cooperate with each other to implement a service such as delivering an item (eg, mail, courier, etc.) to a user or serving food.

A robot providing such a service may be configured such that the first robot 100 delivers an object to the second robot 200 in order to implement the service, for example, to deliver an object to another robot for delivery, or Picking up from the other robot, or, as another example, an operation of delivering the tray to another robot after serving food, and the like are possible.

More specifically, the first robot 100 is configured to pick up an object 10 such as a delivery item or a tray at a fixed location, and the second robot 200 mainly travels to transport the object 10. It is done. In this way, a first task may be assigned to the first robot 100, and a second task may be assigned to the second robot 200. For example, the task assigned to the second robot 200 may be a task of receiving a product from a user and transferring it to the first robot 100 in order to provide a corresponding service (product delivery). The task assigned to the first robot 100 may be a task of picking up the object 10 transported by the first robot 100 and loading it into a loading box.

In this case, the second robot 200 travels inside the building and moves to a specific location 11 adjacent to the first robot 100 while accommodating the object 10 delivered by the user. The first robot 100 recognizes when the second robot 200 approaches the specific location 11, picks up the object 10 transferred by the first robot 100, and loads it (not shown). ) is loaded into In this way, the first robot 100 and the second robot 200 cooperate with each other to complete a task in a specific section for delivery of the object 10, and may be referred to as one robot system.

Meanwhile, in order for the first robot 100 to accurately pick up the object 10 transported by the second robot 200, the first robot 100 accurately detects the posture of the second robot 200. shall. In particular, referring to FIGS. 2A and 2B together with FIG. 1 , the first robot 100 and the second robot 200 may have different shapes. That is, the shapes of the first robot 100 and the second robot 200 are made differently to suit each task, and in this case, there is a need to more easily and accurately measure the posture of the other robot to perform the task. It happens.

Hereinafter, the robots presented in the concept of FIG. 1 will be described in more detail with reference to FIGS. 2A and 2B.

2a and 2b each show an example of robots presented in the concept of FIG. 1 .

Referring to FIG. 2A , for example, the first robot 100 may include at least one of a body unit 110, a sensing unit, and a communication unit.

In order to pick up the object 10, an end effector 111, a manipulator 112, an actuator (not shown), and the like may be provided in the body part.

The end effector 111 is an end device of a robot arm designed to interact with the environment, and may be a gripper, a pin cell, or a scalpel. Such an end effector 111 may be provided separately from the robot and become a robot peripheral device or accessory.

The manipulator 112 may be a device that incorporates links, joints, and other structural elements and mechanisms into the body and arms of the robotic device. In the case of an articulated robot, a manipulator 112 having a plurality of joints performing rotational movement of a motor may be provided.

In this case, the manipulator 112 and the end effector 111 may form an arm with six or more degrees of freedom and a hand with one or more degrees of freedom. In addition, in order to implement the robot of the present invention as a dual-armed robot or a robot having several arms, the arms and hands may exist in several pairs.

In addition, the actuator is a device that converts electrical, chemical, or thermal energy into rotational or linear motion, and may be a motor, a pneumatic cylinder, or an artificial muscle.

In this way, the first robot 100 picks up and moves the object 10 by controlling the end effector 111 , the manipulator 112 , and the actuator.

The sensing unit may include one or more sensors for sensing at least one of information within the robot (particularly, a driving state of the robot), environment information surrounding the robot, location information of the robot, and user information.

For example, the sensing unit includes a camera (image sensor, 121), a proximity sensor, a bio sensor, an infrared sensor, a laser scanner (lidar sensor), an RGBD sensor, a geomagnetic sensor, an ultrasonic sensor, an inertial sensor, a UWB sensor, a microphone, and the like. can do.

The camera 121 may be disposed at an upper end of the body of the robot to photograph the surrounding environment mainly downward. In this case, the camera 121 is formed to enable tilting, etc., so that the shooting angle can be adjusted.

The communication unit of the robot is configured to transmit and receive wireless signals to and from the robot in order to perform wireless communication between a robot and a communication facility in a building, between a robot and another robot, or between a robot and a cloud server. As such an example, the communication unit may include a wireless Internet module, a short-distance communication module, a location information module, and the like.

Referring to FIG. 2B , for example, the second robot 200 may include at least one of a body unit 210, a driving unit 220, a sensing unit 230, a communication unit, and a display 240.

The body part 210 includes a case (casing, housing, cover, etc.) constituting an external appearance. In this embodiment, the case may be divided into a plurality of parts, and various electronic components are embedded in the space formed by the case. In this case, the body part 210 may be formed in different forms according to various services exemplified in the present invention. For example, in the case of a robot providing a delivery service, a container for storing the object 10 may be provided on the upper part of the body part 210 .

The driving unit 220 is configured to perform a specific operation according to a control command transmitted from the cloud server or the control unit of the second robot 200 .

The driving unit 220 provides a means for moving the body of the robot within a specific space in relation to driving. More specifically, the drive unit 220 includes a motor and a plurality of wheels, which are combined to perform functions of driving, changing directions, and rotating the robot R. However, the present invention is not necessarily limited thereto, and the drive unit 220 may include at least one of an end effector, a manipulator, and an actuator to perform other operations other than driving, such as pickup.

Similar to the first robot 100, the sensing unit 230 senses at least one of information within the robot (particularly, driving state of the robot), environment information surrounding the robot, location information of the robot, and user information. It may include one or more sensors for

For self-driving, the second robot 200 may perform image-based positioning (Visual Localization) through a process of acquiring images taken through the camera while moving in a specific space and then comparing them with 3D map data. there is.

In addition, the communication unit, similar to the first robot 100, to perform wireless communication between a robot and a communication facility in a building, between a robot and another robot, or between a robot and a cloud server, a wireless signal from a robot is made to transmit and receive As such an example, the communication unit may include a wireless Internet module, a short-distance communication module, a location information module, and the like.

The display 240 may be disposed upward from the front or upper surface of the second robot 200, and images and graphic objects may be output. In the present invention, the type of display 240 is not limited. The display 240 operates as a monitor and may be made of a touch screen. In this case, the display 240 may perform both a role of outputting information and a role of receiving information.

Meanwhile, as described with reference to FIG. 1 , the first robot 100 and the second robot 200 may each have a controller or be controlled by a cloud server. For example, a cloud server or a controller of the second robot 200 may control the display 240 to output image information. The image information may include not only information provided to the user, but also information provided to the other robot, for example, the first robot 100 .

In the present invention, a visual marker is output to the display 240 of the second robot 200, and the first robot 100 photographs it to measure the attitude of the opponent robot. Interaction between (100) and the second robot (200) is implemented. Hereinafter, with reference to FIGS. 3 to 9 , a process of measuring a posture of a robot to implement such an interaction will be described in detail.

3 is a flowchart for explaining a method for measuring a posture of a robot according to the present invention.

In the method for measuring the posture of a robot of the present invention, the first robot 100 photographs the visual marker output from the second robot 200, and the first robot 100 recognizes the posture of the second robot 200. A new posture measurement method is proposed. At this time, the posture of the robot may include a coordinate indicating a position and a direction indicating an orientation.

In the present invention, a method of measuring the posture of a partner robot by exemplifying a pair of robots (the first robot 100 and the second robot 200) will be described, but is not limited thereto. For example, one robot may photograph visual markers of a plurality of robots, and a plurality of robots may photograph visual markers of one robot.

Referring to FIG. 3 , in the posture measurement method of a robot of the present invention, at least one of the first robot 100 and the second robot 200 is positioned so that the first robot 100 and the second robot 200 are adjacent to each other. The step of moving (S111) may proceed first.

For example, the first robot 100 is fixed at a specific location, and the second robot 200 transports an object or the like to move near the first robot 100, or the first robot 100 and the second robot 100 The robots 200 may move to a specific point and be positioned adjacent to each other at the specific point.

At this time, the first robot 100 may include a camera 121 and the second robot 200 may include a display 240 . However, since the first robot 100 and the second robot 200 exemplified in the present invention are named for convenience of description, the first robot 100 has a display, and the second robot 200 It is also possible to provide this camera. Furthermore, it is also possible that both the first robot 100 and the second robot 200 have cameras and displays.

For example, the second robot 200 may output various image information to people around it using the display 240 . The image information may be a graphic object for interaction with a person or image information for advertising.

Next, a step of requesting the output of the visual marker from the second robot 200 (S112) may proceed.

Requesting the output of the visual marker from the second robot 200 may be performed by the first robot 100 or a server communicating with the first robot 100, for example, a cloud server. Specifically, in order for the first robot 100 to perform a task, the request signal of the visual marker is transmitted to the second robot 200, or the server transmits the request signal to the second robot 200. can transmit

As another example, if the cloud server determines that the posture data of the second robot 200 is necessary for collaboration between the first robot 100 and the second robot 200, the second robot 200 A request signal requesting the output of the visual marker may be transmitted to .

In this case, the method for measuring the posture of a robot according to the present invention may further include monitoring a request signal for requesting an output of the visual marker by the second robot 200 .

When a request signal is transmitted in the requesting step (S112), the second robot 200 may periodically monitor the request signal in order to receive the transmitted request signal and recognize the output request of the visual marker. there is.

Monitoring of the request signal may be performed when the second robot 200 moves to a specific location. For example, when the second robot 200 moves to deliver an object to the first robot 100 and reaches the specific location, the control unit of the second robot 200 determines this and signals the request. monitoring can begin.

Next, a step of outputting a visual marker to the display 240 of the second robot 200 (S113) may proceed.

Upon receiving the request signal, the second robot 200 displays the visual marker using the display 240 .

The visual marker may be, for example, a visual fiducial marker. The visual fiducial marker is a mark created in an image to be used as a reference or measurement point, and may be a publicly available marker such as Apriltag or Aruco that anyone can freely use. As another example, the visual marker is an artificial marker, and may be a marker of various types directly designed by a user.

At this time, in the step of outputting the visual marker (S113), when the request signal is received while the display 240 is deactivated, the display 240 can be activated and the visual marker can be output. As a specific example, the display 240 may output the visual marker while being activated in response to the request signal in a deactivated state.

As another example, in the step of outputting the visual marker (S113), if the request signal is received while the image information is output to the display 240, the output of the image information is turned off and the visual marker is output, or , The visual marker may be output together with the image information. That is, it is also possible to output an image or the like while the display 240 is activated, stop the output of the image according to the request signal, and output the visual marker.

Meanwhile, when a specific time elapses after the visual marker is output to the display 240 of the second robot 200, it is also possible to end the output of the visual marker. In this case, the specific time may be set to be longer than a control execution time for the first robot 100 to photograph the second robot 200 .

Next, the first robot 100 photographs the second robot 200 (S114).

The first robot 100 photographs the second robot 200 using a camera 121 .

For example, the second robot 200 transmits a feedback signal indicating that the visual marker has been output to the first robot 100, and the first robot 100 responds to the feedback signal to the first robot 100. 2 The robot 200 can be photographed. As another example, the feedback signal may be transmitted from the second robot 200 to a cloud server, and a control command to photograph the second robot 200 may be generated and transmitted to the first robot 100 in the cloud server. there is.

In this case, the camera 121 may be a sensor mounted on the first robot 100, but the present invention is not necessarily limited thereto. As such an example, a camera may be disposed in a space where a robot is located. The number of cameras disposed in the space is not limited, and a plurality of cameras may be disposed in the space. The types of cameras disposed in the space may vary, and for example, the camera disposed in the space may be a closed circuit television (CCTV).

Furthermore, the communication facility of the building can perform direct communication with the camera disposed in the space. In addition, communication facilities of the building may be configured to communicate with a video control system that controls the camera. At this time, the camera may be controlled by the video control system to detect a situation in which the second robot 200 arrives at the specific location and photograph the first robot 100 . In this case, the video control system may be provided as a separate control system or configured as part of the cloud server described above.

Referring to FIG. 3 , a step of detecting a visual marker included in an image captured by the first robot 100 and calculating a pose of the visual marker included in the image (S115) proceeds.

Here, as shown, the visual marker may include at least one April tag. The April tag is a mark serving as a visual reference and may be composed of two-dimensional information. A plurality of April tags may be combined to form the visual marker.

These markers are detected from the image captured by the first robot 100, and the posture of the visual marker is calculated using the detected visual markers. The posture detection of the visual marker may be performed using a Perspective-n-Point (PnP) method.

The PnP method may be a technique for estimating a camera pose (eg, a camera position, angle, and direction) using a perspective-n-point algorithm. Using this, the posture of the visual marker, which is the input image, is calculated with respect to the camera coordinate system.

Next, a step of calculating the posture of the second robot 200 using the coordinate data of the second robot 200 and the posture of the visual marker included in the image (S115) may proceed.

For example, the posture of the visual marker with respect to the camera coordinate system may be converted into the posture of the second robot 200 using shape data of the second robot 200, for example, CAD model data.

As another example, it is also possible to convert a marker pose with respect to the camera coordinate system to a robot pose using a calibration method. In this case, the calibration method between the camera and wheel odometry may be used.

As described above, in the present invention, when the partner robot outputs a visual marker, it is photographed to measure the pose of the partner robot, and interaction with the partner robot can be performed using the pose.

Hereinafter, each step of the method for measuring the posture of a robot according to the present invention will be described in detail with examples.

4 is a conceptual diagram showing the concept of the robot of FIG. 2a photographing a visual marker to measure the pose of the robot of FIG. 2b, FIG. 5 is a conceptual diagram of detecting the pose of the visual marker, and FIG. 6 is a conversion into the pose of the robot. 7a and 7b are conceptual diagrams illustrating another embodiment of the present invention, and FIG. 8 is image data representing actual data implemented using the process of the present invention.

First, referring to FIG. 4 , the second robot 200 outputs a visual marker 250 and the first robot 100 photographs the output visual marker 250 .

To make photographing of the visual marker 250 easier and more accurate, the display 240 of the second robot 200 outputting the visual marker 250 is placed on the front or upper surface of the second robot 200. It is disposed upward, and the first robot 100 may include a camera 121 that photographs the second robot 200 downward.

In order to implement this relative position, the display 240 of the second robot 200 may have a view area 241 in the form of an inclined plane with respect to the vertical direction. Meanwhile, the camera 121 of the first robot 100 may be disposed at an upper end of the body of the first robot 100 and disposed downward to have an angle of view toward the view area 241 . In this case, the viewing angle may be a field of view (FoV) of the camera 121 .

In addition, in order for the first robot 100 to more accurately photograph the second robot 200, the position and direction of the first robot 100 and the second robot 200 may be controlled so as to face each other. .

At this time, the visual marker 250 output to the display 240 may be output in an actual size at the center of the display 240 . In this case, the visual marker 250 may be output in an actual size using the pixel pitch (mm) of the display 240 and the size (pixel) of the marker image. As an example, the visual marker 250 The width (w) of may be 175 mm and the height (h) may be 91 mm.

First, the first robot 100 may detect the visual marker 250 included in the photographed image. In this case, the control unit of the first robot 100 may specify at least one marker reference point included in the visual marker 250 based on a predetermined marker detection algorithm (or program). The marker reference point may refer to at least one corner point corresponding to each corner included in the visual marker 250 .

Specifically, the control unit of the first robot 100 recognizes the visual marker 250 from the captured image, and selects at least one marker reference point (P, or corner point, a, b, c, d, e, f, g, h) can be specified (extracted).

Further, the control unit of the first robot 100 may specify the marker coordinate system {W} of the visual marker 250 based on the specified marker reference point P. That is, the marker coordinate system {W} degree-of-freedom information (or six-degree-of-freedom posture (POSE)) for the camera coordinate system {C} can be measured.

The marker coordinate system {W} may include a 3-dimensional axis with one of the specified marker reference points (eg, a, b, c, d, e, f, g, h) as the origin, corresponding to “” You can set the reference point of the marker to be used as the origin.

Next, the control unit of the first robot 100 may estimate the posture of the visual marker 250 . To this end, in the step of calculating the pose of the visual marker included in the image (S115), the camera 121 photographing the second robot 200 using the detected visual marker 250 The 6 degree of freedom posture of the visual marker 250 with respect to the coordinate system of can be estimated.

As shown, the camera 121 may have a camera coordinate system {C}. The camera coordinate system {C} may be a three-dimensional coordinate system having a specific point of the camera 121 as an origin. Also, as illustrated, the marker coordinate system {W} set based on the marker included in the image captured through the camera 121 may be set as another reference coordinate system. In this case, the marker coordinate system {W} may be a three-dimensional coordinate system having a specific point of a marker included in the image as an origin.

In this case, the visual marker 250 may have a reference coordinate system at the point of the marker, and degree of freedom information of the marker may be obtained by reflecting the relative positional relationship between the reference coordinate system of the camera 121 and the reference coordinate system of the marker. can

In this way, based on the camera coordinate system {C} of the camera 121, the degree of freedom information (six degree of freedom posture) of the marker coordinate system {W} can be extracted. Through this, the control unit can extract the degree of rotation and translation (translation) of the visual marker 250 or the marker coordinate system {W} included in the captured image with respect to the camera coordinate system {C}. there is.

For example, by specifying the pixel coordinates corresponding to the marker reference point and using the pixel coordinates corresponding to the marker reference point and the camera coordinate system {C} of the camera 121, the camera coordinate system {C} The position of the degree of freedom of the marker coordinate system {W} can be specified.

More specifically, the controller 130 may calculate PnP in order to extract degree of freedom information (or relative positional relationship) of the marker coordinate system {W} with respect to the camera coordinate system {W}.

Meanwhile, the relative positional relationship of the marker coordinate system {W} with respect to the camera coordinate system {W} may mean the degree to which the marker coordinate system {W} is rotated and translated with respect to the camera coordinate system {W}. . Information according to the degree of rotation and transformation may correspond to degree of freedom information (DOF posture) of the marker coordinate system {W} with respect to the camera coordinate system {C}.

Pixel coordinates (u, v) corresponding to at least one marker reference point of the visual marker 250, in order to extract the degree of freedom information (DOF posture) of the marker coordinate system {W} with respect to the camera coordinate system {C} By substituting the 3D coordinates (x, y, z) of the PnP equation, the degree of freedom information (DOF posture) of the marker coordinate system {W} with respect to the camera coordinate system {C} can be extracted.

Specifically, the 6 degree of freedom posture of the marker coordinate system with respect to the camera coordinate system is calculated, which can be calculated by defining the pixel coordinates of the corner point of the April tag as the following equation (1) using the PnP equation. .

In this case, the left side is the pixel coordinates of the corner point of the April tag, the first matrix on the right side is for the camera parameters, the second matrix is the 6 degree of freedom pose of the marker coordinate system with respect to the camera coordinate system, and the third matrix is the real world It is about the coordinate value of . Also, since the April tag provides a corner point in an image using an open source, pixel coordinates of the corner point can be calculated.

After the posture of the visual marker 250 is measured, the posture is converted into the posture of the second robot 200 .

First, as shown in FIG. 6 , the posture of the visual marker 250 may be converted into the posture of the robot using the CAD model, which is the shape data of the second robot 200 . For example, the 3D coordinates (x, y, z) of the corner point of the visual marker 250 with respect to the reference coordinate system of the CAD model may be extracted. For this coordinate conversion, shape data of the first robot 100 or the cloud server described above may store the shape data of the second robot 200 .

Furthermore, as shown, the robot coordinate system {O} set based on the robot may be set as one of the reference coordinate systems. The robot coordinate system {O} may be a three-dimensional coordinate system having a specific point of the second robot 200 as an origin.

Furthermore, the degree of freedom information of the second robot 200 can be acquired by reflecting the relative positional relationship between the reference coordinate system of the marker and the reference coordinate system of the second robot 200 . That is, in the present invention, based on the relative positional relationship between the reference coordinate system of the camera 121, the reference coordinate system of the marker, and the reference coordinate system of the robot, information on the degree of freedom of at least one of the camera 121, the marker, and the robot is collected. It is possible.

The coordinate conversion can be calculated by defining the pixel coordinates of the corner point of the April tag as the following Equation (2) using the PnP equation.

In this case, the left side is the pixel coordinates of the corner point of the April tag, the first matrix on the right side is for camera parameters, and the second matrix is the robot coordinate system for the camera coordinate system (ie, the coordinate system of the second robot 200). ), and the third matrix is for the coordinate values of the real world.

As shown in Figure 6. The coordinate values of x, y, and z for the corner point Pf at the lower right of the visual marker 250 are calculated based on the coordinate system of the robot reference, and the posture of the second robot 200 can be defined from this. there is. In this way, the 3D coordinates of the point of the visual marker 250 included in the image relative to the reference coordinate system of the CAD model of the second robot 200 are extracted.

Converting the posture of the visual marker 250 into the posture of the robot can be done in another method. Referring to FIGS. 7, 8A, and 8B , a marker posture with respect to a camera coordinate system may be converted into a robot posture using a calibration method.

As such an example, the visual marker 250 included in the image uses odometry information about the camera 121 of the first robot 100 and the wheel 220 of the second robot 200. ) may be converted into the posture of the second robot 200 .

In this case, in a system that uses a camera mounted on a mobile robot to explore the surroundings, in order to ensure an appropriate operating function, the internal and external parameters of the camera, the relative pose between the camera and the frame of the mobile robot, the Calibration is performed to estimate driving parameters. At this time, the odometry (mileage measurement) method is applied, and the odometry is to reconstruct the mobile robot configuration (ie, position and direction) depending on the measurement of the encoder on the wheel 220, in a known configuration. Starting with, the current position and direction of the robot are calculated by time integration of the vehicle speed corresponding to the measured wheel 220 speed. In the present invention, this method can be applied to the relationship between the reference coordinate system of the second robot 200, the corner point of the visual marker 250, and the wheel odometry to calculate the robot posture.

For example, if the wheel odometry is defined as ^Oi T _Oj , the relationship of Equation (3) below is established, and the posture of the visual marker 250 may be defined by Equation (4) below.

The posture of the visual marker 250 may be converted into the posture of the robot using Equations (3) and (4).

According to the method described above, the first robot 100 can measure the posture of the second robot 200 using the visual marker 250 . As such an example, (a), (b), and (c) of FIG. 8 show an example of projecting a CAD model (Mesh) of a robot onto an image based on the measured posture, respectively.

In this way, after measuring the posture of the second robot 200, the first robot 100 calculates the position of the transferred object of the second robot 200, picks up the object, and performs the given task. .

According to the process described above, the method for measuring the posture of a robot according to the present invention and the robot system using the same recognize the posture of the opponent robot, and through this, can quickly and accurately measure the posture of the opponent robot in a simple way.

On the other hand, the process described above can be modified as a method of locating one's own position, not the other robot's. Hereinafter, this modified example will be described with reference to FIG. 9 .

9 is a conceptual diagram for measuring the position of a robot according to another embodiment of the present invention.

It is very difficult to estimate the exact pose information of the camera of the robot because there is a large change over time in the real environment of the indoor space. For example, textureless areas, changing appearance caused by changes in content in signage, etc., changes in crowdedness or occlusions over time, dynamic Due to a dynamic environment or the like, estimation of a camera pose for an image becomes difficult.

The posture measurement using the visual marker described above with reference to FIGS. 1 to 8 may also be applied to camera pose estimation of a robot traveling in an indoor space.

Referring to (a) of FIG. 9 , first, the robot 300 may move in the indoor space 20 where the display 410 is disposed. In this case, the robot 300 may be the second robot 200 of the above-described example.

The display 410 exemplifies signage, but other types of displays are also possible. At this time, image information may be displayed on the display 410, and the image information may be advertisement information, guide information, video content, and the like.

While the robot 300 travels in the indoor space 20 , the display 410 may be detected using a camera 321 mounted on the robot 300 . When the display 410 is detected, the robot 300 transmits a request signal requesting output of the visual marker 420 to a control system that controls the display 410 . At this time, the control system may monitor reception of the request signal.

The control system may be a control system of a building having the indoor space, or may be an external server such as a cloud server.

Referring to (b) of FIG. 9 , the control system controls the display 410 to output the visual marker 420 in response to the request signal.

The visual marker 420 may be the visual marker of the examples described with reference to FIGS. 1 to 8 , and may be output to the display 410 together with image information output from the display 410 . As such an example, the visual marker 420 may be overlapped with the image information and output at a lower portion of one side of the display 410 .

Next, the robot 300 estimates the current location of the robot 300 using the visual marker 420 output to the display 410 . Also, in this case, the pose of the robot 300 may also be estimated. Estimation of the current position and pose of the robot 300 may be performed through image-based positioning using a feature map of the indoor space.

An accurate position of the robot 300 may be estimated using the position and the estimated pose. In this case, the robot 300 may execute various services based on the location.

As described in this example, the present invention outputs a visual marker on a display fixed to a specific space, such as a signage, so that the position of the robot that has photographed the visual marker can be accurately and quickly measured.

The method for measuring the posture of a robot described above and the robot system using the same are not limited to the configuration and method of the above-described embodiments, but the above embodiments are all or all of the embodiments so that various modifications can be made. Some of them may be selectively combined and configured.

Claims (14)

moving at least one of the first robot and the second robot so that the first robot and the second robot are adjacent to each other;
outputting a visual marker to a display of the second robot;
photographing the second robot by the first robot;
detecting a visual marker included in an image captured by the first robot and calculating a pose of the visual marker included in the image; and
and calculating a posture of the second robot using coordinate data of the second robot and a posture of a visual marker included in the image. According to claim 1,
The method of measuring the posture of a robot further comprising monitoring a request signal requesting an output of the visual marker by the second robot. According to claim 2,
The request signal is transmitted from a server communicating with the first robot or from the first robot to the second robot. According to claim 2,
The step of outputting the visual marker,
When the request signal is received while image information is output to the display, the output of the image information is turned off and the visual marker is output, or the visual marker is output together with the image information. measurement method. According to claim 2,
The step of outputting the visual marker,
and activating the display and outputting the visual marker when the request signal is received while the display is inactive. According to claim 1,
and ending the output of the visual marker when a specific time elapses after outputting the visual marker to the display of the second robot. According to claim 1,
The display is disposed upward from the front or upper surface of the second robot, and the first robot includes a camera that photographs the second robot downward. According to claim 1,
In the step of calculating the pose of the visual marker included in the image,
The method of measuring the posture of the robot, characterized in that the 6 degree of freedom posture of the visual marker with respect to the coordinate system of the camera photographing the second robot is estimated using the detected visual marker. According to claim 1,
In the step of calculating the posture of the second robot,
3D coordinates of a point of a visual marker included in the image with respect to a reference coordinate system of the CAD model of the second robot are extracted. According to claim 1,
In the step of calculating the posture of the second robot,
Converting the posture of the visual marker included in the image to the posture of the second robot using odometry information about the camera of the first robot and the wheel of the second robot How to measure posture. Moving the robot in a specific space where the display is arranged;
displaying image information on the display;
transmitting, by the robot, a request signal requesting output of a visual marker to a control system that controls the display;
controlling the display so that the control system outputs the visual marker in response to the request signal; and
and estimating, by the robot, a current position of the robot using a visual marker output on the display. According to claim 11,
The method of measuring the position of the robot further comprising the step of the control system monitoring the reception of the request signal. A first robot equipped with a camera;
A second robot that interacts with the first robot and has a display;
A visual marker is output to the display of the second robot, and the first robot photographs the second robot;
A visual marker included in an image captured by the first robot is detected, a pose of the visual marker included in the image is calculated, and the coordinate data of the second robot and the pose of the visual marker included in the image are calculated. Estimating the posture of the second robot using
A robot system in which a plurality of robots perform interactions, characterized in that for controlling the operation of the first robot using the estimated posture of the second robot. According to claim 13,
The second robot is controlled to deliver an object to the first robot,
The robot system in which a plurality of robots perform interaction, characterized in that the first robot moves an end effector to a set position of the second robot using the estimated posture of the second robot.

Images