JP7843842B2 - Robot posture measurement method and robot system in which multiple robots interact - Google Patents

Description

This invention relates to a method for measuring the posture of a robot and a robot system using this method, and more particularly to a method for measuring the posture of another robot for interaction between robots, and a robot system for performing such interaction.

With technological advancements, a wide variety of service devices have emerged, and in particular, recent technological development has been actively focused on robots that perform diverse tasks or services.

Furthermore, recent advancements in artificial intelligence and cloud technologies have made it possible to control robots more precisely and safely, leading to a gradual increase in their utilization. In particular, technological advancements have brought robots to a level where they can safely coexist with humans in indoor spaces.

Therefore, in recent years, robots have been replacing human tasks or work, and various methods of robots directly providing services to humans, particularly in indoor spaces, are being actively researched.

For example, robots provide navigation services in public places such as airports, train stations, and department stores, and serve food in restaurants. Furthermore, in office spaces and shared living spaces, robots provide delivery services for mail and packages. In addition, robots offer a variety of other services, including cleaning, security, and logistics processing. The types and scope of services offered by robots are expected to increase dramatically in the future, and the level of service provision is also expected to continue to develop.

Such robots provide a variety of services not only in outdoor spaces but also in indoor spaces of buildings (or complexes) such as offices, apartments, department stores, schools, hospitals, and amusement facilities. In this case, the robots are controlled to move around the indoor spaces of the building and provide a variety of services.

On the other hand, for robots to provide diverse services or live in indoor spaces, they need to cooperate with each other to perform tasks. For example, Korean Patent Publication No. 10-2010-0086093 (Autonomous Mobile Cluster Robot Position Control System) discloses a system in which a master robot forms a cluster with multiple subordinate robots and operates autonomously. Thus, gradually developing from mobile robot technology, recent efforts are actively being made to control the interaction between robots to realize diverse services.

Therefore, essential research is needed to enable robots to accurately and quickly measure their position and orientation, in order to provide higher-level services using robots through robot interaction.

This invention provides a method and system for rapidly and accurately measuring the posture of a robot.

More specifically, the present invention provides a method for measuring the posture of a robot using a display and visual markers attached to the robot.

To solve the above problems, the robot posture measurement method and robot system using the present invention employ a process in which a visual marker is output to the display of the target robot, and the robot photographs it to recognize the target robot's posture.

Specifically, the robot posture measurement method of the present invention includes the steps of: moving at least one of the first robot and the second robot so that they are adjacent to each other; outputting a visual marker to the display of the second robot; having the first robot photograph the second robot; detecting the visual marker included in the image taken by the first robot and calculating the posture (pose) of the visual marker included in the image; and calculating the posture of the second robot using the coordinate data of the second robot and the posture of the visual marker included in the image.

In one embodiment of the present invention, the robot posture measurement method may further include the step of monitoring a request signal in which the second robot requests the output of a visual marker.

The request signal may be transmitted from the server communicating with the first robot, or from the first robot to the second robot.

The step of outputting a visual marker may involve, upon receiving a request signal while image information is being output to the display, either turning off the output of the image information and outputting the visual marker, or outputting the visual marker together with the image information.

The step of outputting a visual marker may, if the request signal is received while the display is not activated, activate the display and output the visual marker.

On the other hand, the present invention discloses a robot position measurement method that includes the steps of: the robot moving in a specific space where a display is arranged; image information being displayed on the display; the robot transmitting a request signal to a server communicating with the robot or a control system controlling the display to request the output of a visual marker; the control system controlling the display in response to the request signal to output a visual marker; and the robot estimating its current position using the visual marker output on the display.

Furthermore, the present invention discloses a robot system in which multiple robots interact, comprising: a first robot equipped with a camera; a second robot that interacts with the first robot and is equipped with a display; the second robot outputs a visual marker to its display; the first robot photographs the second robot; the first robot detects the visual marker contained in the image captured by the first robot; the second robot calculates the pose of the visual marker contained in the image; the second robot estimates its pose using its coordinate data and the pose of the visual marker contained in the image; and the first robot controls its movement using the estimated pose of the second robot.

The robot posture measurement method and robot system using the robot posture measurement method according to the present invention detect a marker output to a display, measure the marker posture, compare it with the robot's shape data, convert it into the robot's posture, and recognize the posture of the target robot. This allows for quick and accurate measurement of the target robot's posture in a simple manner.

In this way, by rapidly measuring the precise robot posture through this new process, it becomes possible to provide human-like service in a variety of robotic services, such as delivery, food service, and customer service.

Furthermore, this invention enables accurate and precise measurement of a robot's posture using visual markers, by utilizing a display designed for human interaction to measure the posture of the target robot. In this case, visual markers can be output on a display fixed in a specific space, such as a digital signage screen, and the position of the robot capturing the visual marker can be measured accurately and quickly.

This is a conceptual diagram illustrating the interaction of the robot according to the present invention. This figure shows an example of a robot based on the concept in Figure 1. This figure shows an example of a robot based on the concept in Figure 1. This is a flowchart illustrating the robot posture measurement method of the present invention. This is a conceptual diagram illustrating the idea that the robot in Figure 2a takes a visual marker to measure the posture of the robot in Figure 2b. This is a conceptual diagram for detecting the pose of a visual marker. This is a conceptual diagram illustrating an example of converting a robot's posture. This is a conceptual diagram illustrating another embodiment of the present invention. This is a conceptual diagram illustrating another embodiment of the present invention. This is image data showing actual data realized using the process of the present invention. This is a conceptual diagram illustrating how to measure the position of a robot in another embodiment of the present invention.

The embodiments disclosed herein will be described in detail below with reference to the accompanying drawings. Regardless of the reference numerals used in the drawings, identical or similar components will be given the same reference numerals, and redundant descriptions will be omitted. The suffixes "module" and "part" used for components in the following description are added or mixed solely for the purpose of facilitating the creation of the specification and do not have any distinct meaning or role in themselves. Furthermore, in describing the embodiments disclosed herein, if it is determined that a specific description of related prior art may obscure the gist of the embodiments disclosed herein, such detailed description will be omitted. The accompanying drawings are provided to facilitate the understanding of the embodiments disclosed herein and should be understood not as limiting the technical ideas disclosed herein, but as including all modifications, equivalents, and substitutions that fall within the concept and technical scope of the present invention.

Terms including ordinal numbers, such as "first," "second," etc., can be used to describe various components, but the components themselves are not limited by these terms. These terms are used solely for the purpose of distinguishing one component from another.

When one component is said to be "connected" or "linked" to another component, it should be understood that it may be directly connected or linked to the other component, but that other components may also exist between them. On the other hand, when one component is said to be "directly connected" or "directly linked" to another component, it should be understood that there are no other components between them.

Unless otherwise indicated by the context, singular expressions include plural forms.

In this application, terms such as "includes" or "has" should be understood to indicate the presence of features, figures, steps, actions, components, parts, or combinations thereof described in the specification, without prejudice to the possibility of the presence or addition of one or more other features, figures, steps, actions, components, parts, or combinations thereof.

This invention relates to a method for measuring the posture of a robot and a robot system using the same, and more specifically, to a method for measuring the posture of robots for interaction between robots, and to a robot system for performing such interaction.

Here, interaction can be defined as a type of communication that can occur between humans, between humans and matter, between humans and systems, and between systems. Specifically, it is a compound word of "inter" (mutual) and "action" (movement, operation), meaning actions in which two or more objects influence each other, and can encompass all actions characterized primarily by bidirectionality and real-time nature of the influence.

Furthermore, in this specification, a robot means i) a machine having functions similar to those of a human, or ii) a mechanical device that can be programmed with one or more computer programs and automatically performs a complex series of tasks. As another example, a robot may mean a machine that has form and the ability to think for itself. Moreover, the robots described herein can be realized in forms such as android robots that resemble humans in appearance and behavior, humanoid robots with a body structure similar to that of a human, and cyborgs whose bodies have been modified with machinery.

As technology advances, the use of robots is gradually increasing. Traditionally, robots have been used in specialized industrial fields (for example, industrial automation), but they are gradually transforming into service robots capable of performing useful tasks for humans and equipment.

In this case, the services provided by the robot can be implemented within the building. As an example, the robot according to the present invention may be controlled based on at least one of a cloud server and a control unit provided within the robot itself, and may be configured to travel within the building or provide services corresponding to assigned tasks.

The building in this invention is a building where robots and humans coexist. The robot is configured to navigate while avoiding obstacles such as people, items used by people (e.g., strollers and carts), and animals, and may optionally be configured to output notification information related to the robot's movement. Such robot movement may be performed to avoid obstacles based on at least one of a cloud server and a control unit provided in the robot. The cloud server can control the robot to move within the building while avoiding obstacles based on information received through various sensors provided in the robot (e.g., cameras (image sensors), proximity sensors, infrared sensors, etc.). However, this invention is not necessarily limited thereto, and the services and processes described in this invention can also be applied outdoors.

On the other hand, the robot of the present invention may be configured to provide services to people or target objects present within a building.

The types of services provided by robots can vary from robot to robot. In other words, robots come in diverse types depending on their intended use, each with a different structure, and each robot is equipped with a program tailored to that use.

For example, a robot may be deployed to provide at least one of the following services: delivery to buildings, logistics operations, guidance, interpretation, parking assistance, security, crime prevention, guarding, public safety, cleaning, disease prevention, disinfection, washing, beverage production, food production, serving, fire suppression, medical assistance, and entertainment services. The services provided by the robot may be diverse beyond the examples listed above.

On the other hand, the cloud server can consider the specific purpose of each robot, assign appropriate tasks to them, and control the robots to ensure that the assigned tasks are performed.

At least some of the robots described in this invention can operate or perform tasks under the control of a cloud server , in which case the amount of data processed by the robot itself for operation or task performance can be minimized. In this invention, such robots may be referred to as brainless robots. Such brainless robots may rely on the control of a cloud server for at least some of their operations when performing activities such as moving around a building, performing tasks, charging, waiting, and cleaning.

However, in this specification, instead of distinguishing and naming unintelligible robots, all of them will be collectively referred to as "robots."

Furthermore, in this invention, the building may be provided with a variety of infrastructure (or equipment infrastructure) to perform the services offered by the robot.

Infrastructure or facility infrastructure refers to facilities provided within a building for purposes such as providing services, facilitating robot movement, maintaining functionality, and maintaining cleanliness. Its types and forms are extremely diverse. For example, building infrastructure can include a wide variety of items such as mobility equipment (e.g., robot pathways, elevators, escalators), charging facilities, communication equipment, cleaning facilities, and structures (e.g., stairs). Thus, the aforementioned infrastructure or facility infrastructure is defined as the infrastructure used by robots to perform their assigned tasks.

In this invention, robots are configured to cooperate with each other to perform assigned tasks. This cooperation is achieved through interaction between robots, and this invention provides a method for measuring the posture of the robots for this purpose.

As an example, as shown in Figure 1, robots can cooperate with each other to deliver goods (e.g., mail, packages, etc.) to users or to perform services such as serving meals.

A robot providing such a service may be configured so that the first robot 100 transmits items to the second robot 200 in order to perform the service. For example, this could involve transmitting items to the other robot for delivery, picking them up from the other robot, or, as another example, transmitting a tray to another robot after serving food.

More specifically, the first robot 100 is configured to pick up items 10, such as delivery goods or trays, from a fixed position, while the second robot 200 is configured to primarily move and transport the items 10. Thus, the first robot 100 may be assigned a first task, and the second robot 200 may be assigned a second task. For example, the task assigned to the second robot 200 might be to receive goods from the user and transport them to the first robot 100 in order to provide a corresponding service (item delivery). The task assigned to the first robot 100 might be to pick up the items 10 transported by the first robot 100 and load them into a loading box.

In this case, the second robot 200 travels within the building, carrying the item 10 transmitted by the user, and moves to a specific location 11 adjacent to the first robot 100. The first robot 100 recognizes when the second robot 200 approaches the specific location 11, picks up the item 10 transported by the first robot 100, and loads it into a loading box (not shown). In this way, the first robot 100 and the second robot 200 cooperate to complete a task in a specific section for the delivery of the item 10, and can be called a single robotic system.

On the other hand, in order for the first robot 100 to accurately pick up the item 10 transferred by the second robot 200, the first robot 100 must accurately detect the orientation of the second robot 200. In particular, referring to Figures 1, 2a, and 2b, the first robot 100 and the second robot 200 may have different shapes. That is, the shapes of the first robot 100 and the second robot 200 are different to suit their respective tasks, and in such cases, it is necessary to measure the orientation of the other robot more easily and accurately in order to perform the task.

The robot presented in the concept of Figure 1 will be explained in more detail below with reference to Figures 2a and 2b.

Figures 2a and 2b show examples of robots presented using the concept in Figure 1.

Referring to Figure 2a, for example, the first robot 100 may include at least one of the following: a body section 110, a sensing section, and a communication section.

The body portion may be equipped with an end effector 111, a manipulator 112, an actuator (not shown), etc., for picking up the item 10.

The end effector 111 is an end device of a robot arm designed to interact with the environment, and may be a gripper, pincer, scalpel, etc. Such an end effector 111 may be provided separately from the robot and used as a peripheral device or accessory for the robot.

The manipulator 112 may be a device that integrates links, joints, and other structural elements and mechanisms with the main body and arms of the robotic device. In the case of a multi-joint robot, the manipulator 112 may have multiple joints that perform rotational motion of motors.

In this case, the manipulator 112 and end effector 111 can form an arm with six or more degrees of freedom and a hand with one or more degrees of freedom. Furthermore, to realize the robot of the present invention as a two-armed robot or a robot with multiple arms, the arms and hands may exist in multiple pairs.

Furthermore, the actuator is a device that converts electrical, chemical, or thermal energy into rotational or linear motion, and may be a motor, pneumatic cylinder, artificial muscle, or the like.

In this manner, the first robot 100 controls the end effector 111, manipulator 112, and actuator to pick up and move the item 10.

The sensing unit may include one or more sensors for sensing at least one of the following: information within the robot (particularly the robot's operating state), information about the surrounding environment of the robot, the robot's position information, and user information.

For example, the sensing unit may include a camera (image sensor) 121, a proximity sensor, a biosensor, an infrared sensor, a laser scanner (LiDAR sensor), an RGBD sensor, a geomagnetic sensor, an ultrasonic sensor, an inertial sensor, a UWB sensor, a microphone, and the like.

The camera 121 may be positioned at the upper end of the robot's body and configured to primarily capture the surrounding environment downwards. In this case, the camera 121 may be configured to allow tilting, etc., and the shooting angle may be adjusted.

The robot's communication unit is configured to transmit and receive wireless signals for wireless communication between the robot and the building's communication equipment, between the robot and other robots, or between the robot and a cloud server. For example, the communication unit may include a wireless internet module, a short-range communication module, a location information module, and the like.

Referring to Figure 2b, for example, the second robot 200 may include at least one of the following: a body unit 210, a drive unit 220, a sensing unit 230, a communication unit, and a display 240.

The body portion 210 includes a case (casing, housing, cover, etc.) that forms the external appearance. In this embodiment, the case can be divided into multiple parts, and various electronic components are incorporated into the space formed by the case. In this case, the body portion 210 may be configured in different forms depending on the various services exemplified in the present invention. For example, in the case of a robot that provides delivery services, a storage box for storing items 10 may be provided on the upper part of the body portion 210.

The drive unit 220 is configured to perform specific actions based on control commands transmitted from the cloud server or the control unit of the second robot 200.

The drive unit 220 provides a means for the robot's body to move within a specific space in connection with its movement. More specifically, the drive unit 220 includes a motor and a plurality of wheels, which, in combination, perform the functions of moving, changing direction, and rotating the robot . However, the present invention is not necessarily limited thereto, and the drive unit 220 may include at least one of an end effector, a manipulator, and an actuator to perform other actions other than movement, such as picking up objects.

The sensing unit 230, similar to the first robot 100, may include one or more sensors for sensing at least one of the following: information within the robot (particularly the robot's operating state), information about the surrounding environment, the robot's position information, and user information.

The second robot 200 can perform visual localization by moving through a specific space, acquiring images with the camera, and then comparing them with three-dimensional map data.

Furthermore, similar to the first robot 100, the communication unit is configured to allow the robot to send and receive wireless signals for wireless communication between the robot and the building's communication equipment, between the robot and other robots, or between the robot and a cloud server. As an example, the communication unit may include a wireless internet module, a short-range communication module, a location information module, and the like.

The display 240 may be positioned facing upwards on the front or top surface of the second robot 200, and may output images and graphic objects. In this invention, there are no limitations on the type of display 240. The display 240 may function as a monitor and be configured as a touchscreen. In this case, the display 240 can perform both the role of outputting information and the role of receiving information.

On the other hand, as explained with reference to Figure 1, the first robot 100 and the second robot 200 each have their own control units or can be controlled by a cloud server. For example, the cloud server or the control unit of the second robot 200 can control the display 240 to output image information. This image information may include not only information provided to the user, but also information provided to the other robot, such as the first robot 100.

In this invention, interaction between the first robot 100 and the second robot 200 is realized by outputting a visual marker to the display 240 of the second robot 200, and the first robot 100 photographing it to measure the posture of the other robot. The process of measuring the posture of the robots in order to realize such interaction will be described in more detail below with reference to Figures 3 to 8 .

Figure 3 is a flowchart illustrating the robot posture measurement method of the present invention.

In the present invention, a novel method for measuring the posture of a robot is presented, in which the first robot 100 photographs a visual marker output from the second robot 200, and the first robot 100 recognizes the posture of the second robot 200. In this case, the posture of the robot may include coordinates indicating position and directions indicating orientation.

In this invention, a method for measuring the posture of an opposing robot is described using a pair of robots (first robot 100 and second robot 200) as an example, but the invention is not limited to this. For example, one robot may photograph the visual markers of multiple robots, or multiple robots may photograph the visual markers of one robot.

Referring to Figure 3, in the robot posture measurement method of the present invention, the step (S111) in which at least one of the first robot 100 and the second robot 200 moves so that they are adjacent to each other may be performed first.

For example, the first robot 100 may be fixed in a specific position, and the second robot 200 may move to the vicinity of the first robot 100 by transporting an object or the like, or the first robot 100 and the second robot 200 may move to a specific location so that they are adjacent to each other at that specific location.

In this case, the first robot 100 may be equipped with a camera 121, and the second robot 200 may be equipped with a display 240. However, the first robot 100 and the second robot 200 exemplified in this invention are named for convenience of explanation, and the first robot 100 may be equipped with a display, and the second robot 200 may be equipped with a camera. Furthermore, it is also possible for both the first robot 100 and the second robot 200 to be equipped with both a camera and a display.

For example, the second robot 200 can output various image information to people in its vicinity using the display 240. This image information may include graphic objects for human interaction or video information for advertisements, etc.

Next, the second robot 200 may be requested to output a visual marker (S112).

Requesting the output of a visual marker from the second robot 200 can be done by the first robot 100, or by a server communicating with the first robot 100, such as a cloud server. Specifically, the first robot 100 may transmit a visual marker request signal to the second robot 200 in order to perform a task, or the server may transmit the request signal to the second robot 200.

As another example, for the cooperation between the first robot 100 and the second robot 200, if the cloud server determines that the posture data of the second robot 200 is necessary, it can transmit a request signal to the second robot 200 requesting the output of a visual marker. For example, the cloud server may sense that the first robot 100 is approaching the second robot 200 and transmit a request signal to the second robot 200 requesting the output of a visual marker, and the second robot 200 may receive the request signal and output the visual marker.

In this case, the robot posture measurement method of the present invention may further include the step of monitoring a request signal that the second robot 200 requests the output of the visual marker.

When the request signal is transmitted in the request step (S112), the second robot 200 can periodically monitor the request signal in order to receive the transmitted request signal and recognize the request to output the visual marker.

The monitoring of the request signal may be performed when the second robot 200 moves to a specific location. For example, when the second robot 200 moves to the specific location to deliver an item to the first robot 100, the control unit of the second robot 200 can determine this and start monitoring the request signal.

Next, a step (S113) may be performed in which a visual marker is output to the display 240 of the second robot 200.

Upon receiving the request signal, the second robot 200 displays the visual marker using the display 240.

The aforementioned visual marker may, for example, be a Visual Reference Marker. This Visual Reference Marker is a mark generated for use as a reference or measurement point in an image, and may be a publicly available marker, such as Apriltag or Aruco, that can be freely used by anyone. Another example is that the visual marker is an Artificial Marker, which may be a variety of markers directly designed by the user.

In this case, the step of outputting the visual marker (S113) can be performed by activating the display 240 and outputting the visual marker if the request signal is received while the display 240 is not activated. As a specific example, the display 240 may be activated in response to the request signal while not activated, and then output the visual marker.

As another example, in the step of outputting the visual marker (S113), if the request signal is received while image information is being output to the display 240, the output of the image information can be turned off and the visual marker can be output, or the visual marker can be output together with the image information. That is, it is also possible to output an image or the like while the display 240 is activated, and then, upon receiving the request signal, stop the output of the image and output the visual marker.

On the other hand, it is also possible to terminate the output of the visual marker on the display 240 of the second robot 200 after a certain amount of time has elapsed since outputting the visual marker. In this case, the specific time may be set to be longer than the time it takes for the first robot 100 to perform control to photograph the second robot 200.

Next, step (S114) is performed in which the first robot 100 photographs the second robot 200.

The first robot 100 uses the camera 121 to photograph the second robot 200.

For example, the second robot 200 may transmit a feedback signal to the first robot 100 indicating that it has outputted the visual marker, and the first robot 100 may photograph the second robot 200 in response to the feedback signal. Alternatively, the feedback signal may be transmitted from the second robot 200 to a cloud server, which may generate a control command to photograph the second robot 200 and transmit it to the first robot 100.

In this case, the camera 121 may be a sensor attached to the first robot 100, but the present invention is not necessarily limited to this. As an example, cameras may be placed in the space where the robot is located. There is no limit to the number of cameras placed in the space; multiple cameras may be placed in the space. The types of cameras placed in the space are diverse; for example, the cameras placed in the space may be CCTV (closed circuit television) cameras.

Furthermore, the building's communication equipment can communicate directly with cameras placed in the aforementioned space. The building's communication equipment may also be configured to communicate with a video control system that controls the cameras. In this case, the camera can be controlled by the video control system to detect when the second robot 200 arrives at the specific location and to photograph the first robot 100. In this case, the video control system may be provided as a separate control system or may be configured as part of the cloud server described above.

Referring to Figure 3, step (S115) is performed to detect visual markers included in the image captured by the first robot 100 and to calculate the pose of the visual markers included in the image.

Here, the visual marker may include at least one April tag, as shown in the figure. The April tag may consist of two-dimensional information as a visual reference. Multiple April tags may be combined to constitute the visual marker.

Such markers are detected from images captured by the first robot 100, and the pose of the visual markers is calculated using the detected visual markers. The detection of the pose of the visual markers may be performed using the PnP (Perspective-n-Point) method.

The aforementioned PnP method is a technique that estimates camera pose (e.g., camera position, angle, and direction) using the perspective-n-point algorithm. Using this, the orientation of the visual marker, which is the input image, is calculated relative to the camera coordinate system.

Next, a step ( S116 ) may be performed to calculate the posture of the second robot 200 using the coordinate data of the second robot 200 and the posture of the visual marker included in the image.

For example, the pose of the visual marker relative to the camera coordinate system can be converted to the pose of the second robot 200 using the shape data of the second robot 200, such as CAD model data.

As another example, it is also possible to convert the marker orientation relative to the camera coordinate system to the robot orientation using a calibration method. In this case, the calibration method may be a calibration method between the camera and wheel odometry.

As described above, in this invention, when the opposing robot outputs a visual marker, it is photographed to measure the opposing robot's posture, and interaction with the opposing robot can be performed using this posture. For example, the first robot 100 and the second robot 200 can either transmit an item to the other, or both robots can interact with each other, such as shaking hands.

The following describes each step of the robot posture measurement method of the present invention in more detail, with examples.

Figure 4 is a conceptual diagram showing the robot in Figure 2a capturing a visual marker to measure the posture of the robot in Figure 2b; Figure 5 is a conceptual diagram for detecting the pose of the visual marker; Figure 6 is a conceptual diagram showing an example of conversion to the robot's posture; Figures 7a and 7b are conceptual diagrams showing other embodiments of the present invention; and Figure 8 is image data showing actual data realized using the process of the present invention.

First, referring to Figure 4, the second robot 200 outputs a visual marker 250, and the first robot 100 photographs the outputted visual marker 250.

To facilitate and accurately capture the visual marker 250, the display 240 of the second robot 200, which outputs the visual marker 250, may be positioned upward on the front or top surface of the second robot 200, and the first robot 100 may be equipped with a camera 121 that captures the second robot 200 facing downward.

To achieve such relative positions, the display 240 of the second robot 200 may have a view area 241 in the form of a surface inclined with respect to the vertical. On the other hand, the camera 121 of the first robot 100 may be positioned at the upper end of the body of the first robot 100 and angled downwards to have a field of view toward the view area 241. In this case, the field of view may be the angle of view (FoV, Field of View) of the camera 121.

Furthermore, in order for the first robot 100 to more accurately photograph the second robot 200, the positions and orientations of the first robot 100 and the second robot 200 can be controlled so that they face each other.

In this case, the visual marker 250 output to the display 240 can be output to the center of the display 240 at a predetermined size. In this case, the visual marker 250 can be output to a predetermined size using the pixel pitch (Pixel Pitch, mm) of the display 240 and the size of the marker image (pixels). For example, the width w of the visual marker 250 may be 175 mm and the height h may be 91 mm.

First, the first robot 100 can detect the visual markers 250 included in the captured image. In this case, the control unit of the first robot 100 can identify at least one marker reference point included in the visual markers 250 based on a pre-configured marker detection algorithm (or program). The marker reference point may represent at least one corner point corresponding to each corner included in the visual markers 250.

Specifically, the control unit of the first robot 100 can recognize the visual marker 250 from the captured image and identify (extract) at least one marker reference point (P, or corner point, a, b, c, d, e, f, g, h) corresponding to each corner of the recognized marker.

Furthermore, the control unit of the first robot 100 can determine the marker coordinate system {W} of the visual marker 250 based on the identified marker reference point P. That is, it can measure the degrees of freedom information (or 6-degree-of-freedom posture) of the marker coordinate system {W} relative to the camera coordinate system {C}.

The marker coordinate system {W} may include three-dimensional axes with any one of the specified marker reference points (e.g., a, b, c, d, e, f, g, h) as the origin, or it may have a marker reference point corresponding to a specific point on the camera as its origin.

Next, the control unit of the first robot 100 can estimate the orientation of the visual marker 250. Therefore, in the step of calculating the orientation of the visual marker included in the image (S115), the detected visual marker 250 can be used to estimate the 6-degree-of-freedom orientation of the visual marker 250 relative to the coordinate system of the camera 121 that captured the second robot 200.

Such a camera 121 may be configured to have a camera coordinate system {C}, as shown in the figure. The camera coordinate system {C} may be a three-dimensional coordinate system with a specific point on the camera 121 as its origin. Alternatively, as shown in the figure, a marker coordinate system {W} may be set as another reference coordinate system, based on markers included in the image captured by the camera 121. In this case, the marker coordinate system {W} may be a three-dimensional coordinate system with a specific point on the markers included in the image as its origin.

In this case, the visual marker 250 may have a reference coordinate system at the marker's point, and the marker's degrees of freedom information can be obtained by reflecting the relative positional relationship between the camera 121's reference coordinate system and the marker's reference coordinate system.

In this way, the degree of freedom information (6 degrees of freedom orientation) of the marker coordinate system {W} can be extracted using the camera coordinate system {C} of the camera 121 as a reference. This allows the control unit to extract the degree to which the visual marker 250 or the marker coordinate system {W} contained in the captured image has been rotated and transformed (translationally moved) relative to the camera coordinate system {C}.

For example, by identifying the pixel coordinates corresponding to the marker reference point, and using the pixel coordinates corresponding to the marker reference point and the camera coordinate system {C} of the camera 121, the degree of freedom orientation of the marker coordinate system {W} relative to the camera coordinate system {C} can be determined.

More specifically, the control unit may calculate PnP in order to extract the degrees of freedom information (or relative positional relationship) of the marker coordinate system {W} with respect to the camera coordinate system { C }.

On the other hand, the relative positional relationship of the marker coordinate system { W } with respect to the camera coordinate system { C } can indicate the degree to which the marker coordinate system {W} has been rotated and transformed with respect to the camera coordinate system {C}. This information, based on the degree of rotation and transformation, can correspond to the degrees of freedom information (degrees of freedom orientation) of the marker coordinate system {W} with respect to the camera coordinate system {C}.

To extract the degrees of freedom information (degrees of freedom orientation) of the marker coordinate system {W} relative to the camera coordinate system {C} , the pixel coordinates (u, v) corresponding to at least one marker reference point of the visual marker 250 and the three-dimensional coordinates (x, y, z) of the camera can be substituted into the PnP equation, thereby extracting the degrees of freedom information (degrees of freedom orientation) of the marker coordinate system {W} relative to the camera coordinate system {C}.

Specifically, the 6-degree-of-freedom orientation of the marker coordinate system relative to the camera coordinate system is calculated. This can be calculated by defining the pixel coordinates of the corner points of the April tag using the PnP equation in the following equation (1).

In this case, the left-hand side represents the pixel coordinates of the corner points of the April Tag, the first matrix on the right-hand side corresponds to the camera parameters, the second matrix represents the 6-degree-of-freedom orientation of the marker coordinate system relative to the camera coordinate system, and the third matrix corresponds to the real-world coordinate values. Furthermore, since the April Tag provides corner points in the video using open-source software, the pixel coordinates of these corner points can be calculated.

When the orientation of the visual marker 250 is measured, that orientation is converted into the orientation of the second robot 200.

First, as shown in Figure 6, the pose of the visual marker 250 can be converted to the robot's pose using the CAD model, which is the shape data of the second robot 200. For example, the three-dimensional coordinates (x, y, z) of the corner points of the visual marker 250 relative to the reference coordinate system of the CAD model can be extracted. For such coordinate conversion, the shape data of the second robot 200 may be stored in the first robot 100 or the cloud server described above.

Furthermore, as shown in the figure, a robot coordinate system {O}, which is set with the robot as the reference, may be set as one of the reference coordinate systems. The robot coordinate system {O} may be a three-dimensional coordinate system with a specific point on the second robot 200 as its origin.

Furthermore, it goes without saying that the degrees of freedom information of the second robot 200 is obtained by reflecting the relative positional relationship between the marker's reference coordinate system and the second robot 200's reference coordinate system. In other words, in this invention, it is possible to collect degrees of freedom information for at least one of the camera 121, the marker, and the robot based on the mutually relative positional relationship between the camera 121's reference coordinate system, the marker's reference coordinate system, and the robot's reference coordinate system.

The aforementioned coordinate transformation can be calculated by defining the pixel coordinates of the corner points of the April tag using the PnP equation and the following equation (2).

In this case, the left-hand side represents the pixel coordinates of the corner points of the April tag, the first matrix on the right-hand side represents the camera parameters, the second matrix represents the 6-degree-of-freedom orientation of the robot coordinate system (i.e., the coordinate system of the second robot 200) relative to the camera coordinate system, and the third matrix represents the real-world coordinate values.

As shown in Figure 6, the posture of the second robot 200 can be defined by calculating the x, y, and z coordinate values for the corner point Pf at the lower right end of the visual marker 250, using the robot's reference coordinate system as a reference. In this method, the 3D coordinates of the points of the visual marker 250 included in the image are extracted relative to the reference coordinate system of the CAD model of the second robot 200.

It is also possible to convert the orientation of the visual marker 250 to the robot's orientation by other methods. Referring to Figures 7a , 7b , and 8 , the marker orientation relative to the camera coordinate system can be converted to the robot's orientation using a calibration method.

As an example, the odometry information of the camera 121 of the first robot 100 and the wheel 220 of the second robot 200 can be used to convert the orientation of the visual marker 250 included in the image into the orientation of the second robot 200.

In this case, to ensure proper operation in a system that uses a camera mounted on a mobile robot to explore its surroundings, corrections are made to estimate the internal and external parameters of the camera, the relative pose between the camera and the mobile robot's frame, and the mobile robot's movement parameters. At this time, an odometry method (distance measurement) is applied. Odometry involves reconstructing the mobile robot's configuration (i.e., position and orientation) based on the encoder measurements at the wheel 220. Starting from a known configuration, the robot's current position and orientation are calculated by the time integral of the vehicle speed corresponding to the measured speed of the wheel 220. In this invention, the robot's posture can be calculated by applying such a method to the relationship between the reference coordinate system of the second robot 200, the corner points of the visual marker 250, and the wheel odometry.

For example, if we define the wheel odometry as ^Oi T _Oj , the relationship in equation (3) below holds, and the orientation of the visual marker 250 can be defined by equation (4) below.

Using equations (3) and (4), the orientation of the visual marker 250 can be converted to the robot's orientation.

According to the method described above, the first robot 100 can measure the posture of the second robot 200 using the visual marker 250. As an example of this, Figures 8(a), (b), and (c) show examples where the robot's CAD model (Mesh) is projected onto the image based on the measured posture.

In this way, after measuring the posture of the second robot 200, the first robot 100 calculates the position of the object transported by the second robot 200, picks up the object, and then performs the assigned task.

Through the process described above, the robot posture measurement method and robot system using the same according to the present invention can recognize the posture of an opposing robot, thereby enabling rapid and accurate measurement of the opposing robot's posture in a simple manner.

On the other hand, the process described above may be modified to measure the position of the robot itself, rather than the position of the target robot. Such modifications will be explained below with reference to Figure 9.

Figure 9 is a conceptual diagram illustrating how to measure the position of a robot in another embodiment of the present invention.

In real-world indoor environments, significant changes occur over time, making it extremely difficult to accurately estimate the camera pose of a robot. For example, factors such as textureless areas, changes in appearance due to content changes on signage, changes in crowding and occlusion over time, and dynamic environmental changes all make it challenging to estimate the camera pose in an image.

Referring to Figures 1 through 8, the posture measurement method using the visual markers described above can also be applied to estimating the camera pose of a robot moving through an indoor space.

Referring to Figure 9(a), the robot 300 can first move within the indoor space 20 where the display 410 is located. In this case, the robot 300 may be the second robot 200 illustrated above.

Although a digital signage display is given as an example of the display 410, other types of displays are also possible. In this case, image information may be displayed on the display 410, and the image information may be advertising information, informational information, video content, etc.

When the robot 300 is moving through the indoor space 20, the display 410 can be detected using the camera 321 attached to the robot 300. Upon detection of the display 410, the robot 300 transmits a request signal to the control system controlling the display 410, requesting the output of the visual marker 420. At this time, the control system can monitor the reception of the request signal.

The control system may be a control system for a building having an indoor space, or it may be an external server such as a cloud server.

As another example, the system may be equipped with a separate server that operates the robot. This server can detect when the robot approaches the display and transmit a request signal to the control system requesting the output of the visual marker 420.

Referring to Figure 9(b), in response to the request signal, the control system controls the display 410 to output the visual marker 420.

The visual marker 420 may be an exemplary visual marker as described with reference to Figures 1 to 8, and may be output to the display 410 together with the image information output from the display 410. As an example, the visual marker 420 may be output overlapping the image information at the lower end of one side of the display 410.

Next, the robot 300 estimates its current position using the visual marker 420 output to the display 410. In this case, the robot 300's pose can also be estimated. The estimation of the robot 300's current position and pose may be performed by visual localization using the indoor space feature map.

Using this position and estimated pose, the precise position of the robot 300 can be estimated. In this case, the robot 300 can perform various services based on the position.

As illustrated in this example, the present invention can output a visual marker on a display fixed in a specific space, such as a signage screen, and accurately and quickly measure the position of a robot that has captured the visual marker.

The robot posture measurement method and robot system using the same described above are not limited to the configuration and methods of the above embodiments. The above embodiments can also be configured by selectively combining all or part of each embodiment to allow for various modifications.

Claims (13)

A step in which at least one of the first robot and the second robot moves so that the first robot and the second robot are adjacent to each other,
The steps include outputting a visual marker to the display of the second robot,
The first robot photographs the second robot,
The steps include detecting visual markers included in an image captured by the first robot and calculating the pose of the visual markers included in the image,
The step of calculating the pose of the second robot using the pose of the visual marker included in the image relative to the camera coordinate system of the first robot ,
A method for measuring the posture of a robot. The robot posture measurement method according to claim 1, further comprising the step of monitoring a request signal indicating that the second robot requests the output of the visual marker. The robot posture measurement method according to claim 2, characterized in that the request signal is transmitted from a server communicating with the first robot or from the first robot to the second robot. The step of outputting the aforementioned visual marker is:
When the request signal is received while image information is being output to the display, the output of the image information is turned off and the visual marker is output, or the visual marker is output together with the image information.
The method for measuring the posture of a robot according to claim 2. The step of outputting the aforementioned visual marker is:
The display is characterized by activating the display and outputting the visual marker when it receives the request signal while the display is not activated.
The method for measuring the posture of a robot according to claim 2. The robot posture measurement method according to claim 1, characterized in that the output of the visual marker is terminated after a specific time has elapsed since the output of the visual marker to the display of the second robot. The robot posture measurement method according to claim 1, characterized in that the display is positioned facing upward on the front or top surface of the second robot, and the first robot is equipped with a camera that photographs the second robot facing downward. The robot posture measurement method according to claim 1, characterized in that, in the step of calculating the posture of the visual marker contained in the image, the six-degree-of-freedom posture of the visual marker with respect to the coordinate system of the camera that photographed the second robot is estimated using the detected visual marker. The robot posture measurement method according to claim 1, characterized in that, in the step of calculating the posture of the second robot, the 3D coordinates of the points of the visual markers included in the image are extracted relative to the reference coordinate system of the CAD model of the second robot. The robot posture measurement method according to claim 1, characterized in that, in the step of calculating the posture of the second robot, the posture of the visual marker included in the image is converted to the posture of the second robot using the camera of the first robot and odometry information for the wheels of the second robot. The first robot is equipped with a camera,
The system includes a second robot that interacts with the first robot and is equipped with a display,
The second robot outputs a visual marker to its display, and the first robot photographs the second robot.
The first robot detects the visual markers contained in the image it captures, calculates the pose of the visual markers contained in the image, and estimates the pose of the second robot using the pose of the visual markers contained in the image relative to the camera coordinate system of the first robot .
The operation of the first robot is controlled using the estimated posture of the second robot.
A robotic system in which multiple robots interact with each other. The first robot and the second robot are controlled to interact with each other.
The first robot is characterized by moving the end effector to a set position of the second robot using the estimated posture of the second robot.
A robot system in which a plurality of robots interact as described in claim 11. A robot system in which a plurality of robots interact, as described in claim 11, characterized in that the second robot receives a request signal from the first robot or from a server that communicates with the first robot, requesting the output of the visual marker.