KR102190743B1 - AUGMENTED REALITY SERVICE PROVIDING APPARATUS INTERACTING WITH ROBOT and METHOD OF THEREOF - Google Patents

Abstract

The present disclosure relates to providing an augmented reality service capable of interacting with a robot, and an augmented reality that can conveniently implement augmented reality based on location and direction information obtained from a robot and a mobile device, and control the robot. To provide content to users. An augmented reality service providing apparatus that interacts with a robot according to an embodiment of the present disclosure extracts the location information of the robot from a communication unit communicating with the robot, a camera taking an image, a display unit displaying an image, and an image captured by the camera. , When the direction information acquired by the robot is received through the communication unit, spatial information about the position and posture of the robot is generated based on the extracted position information and the received direction information, and the robot is tracked using the generated spatial information. And a processor for controlling the display unit to determine a position where the virtual object is to be displayed, to set a restricted area in which the robot cannot move, and to display an augmented reality image obtained by combining the virtual object with the captured image.

Description

Device and method for providing augmented reality service that interacts with a robot {AUGMENTED REALITY SERVICE PROVIDING APPARATUS INTERACTING WITH ROBOT and METHOD OF THEREOF}

The present disclosure relates to an apparatus and method for providing an augmented reality service that interacts with a robot, and more particularly, to realize augmented reality by fusion of information obtained from a robot and a mobile device, and to control the robot within the implemented augmented reality screen. It relates to an augmented reality service providing apparatus and method that can be.

Augmented Reality (AR) is a technology that synthesizes and provides virtual image information in reality. The unique features of augmented reality that are differentiated from virtual reality (VR), which are often mentioned along with augmented reality, are the combination of a real image and a virtual image, the ability to interact in real time, and the three-dimensional It can be said that it lies in the space. Augmented reality technology can be used to augment virtual objects or information useful to users in the reality they see, so it is augmented in various industries such as entertainment, education, telemedicine, broadcasting, architecture, design, and manufacturing processes. The demand for services using reality is increasing.

In augmented reality, since it is necessary to place a virtual image at an exact location, it is important to obtain location information of an actual object. Based on the method of acquiring location information, augmented reality technology can be largely classified into a marker method and a markerless method. The marker method is a method of attaching a marker serving as a reference coordinate to an actual object in advance and obtaining location information using the recognized marker as a reference point. The marker method is cumbersome in that a marker must be prepared and attached in advance, and there is a problem that augmented reality cannot be realized without attaching a marker. In addition, it is possible to implement augmented reality without markers using motion capture equipment, lidar equipment, etc., but it is unreasonable to be generally used because it uses expensive equipment. In addition, the markerless method using a beacon has a problem that it is difficult to use in a high-speed Real Time System (RTS) due to a slow communication speed.

The present disclosure is to solve the above-described problem, by fusion of information obtained from each of the devices that provide an augmented reality image and a robot that exists in the image to implement a markerless augmented reality, and robot control content on the augmented reality screen It is an object of the present invention to provide an apparatus and method for providing an augmented reality service that interacts with a robot capable of providing.

An augmented reality service providing apparatus for interacting with a robot according to an embodiment of the present disclosure for achieving the above object includes a communication unit for communicating with the robot, a camera for photographing an image, a display unit for displaying an image, and an image captured by the camera. When the location information of the robot is extracted from and the direction information obtained by the robot is received through the communication unit, spatial information regarding the location and posture of the robot is provided based on the extracted location information and the received direction information. An augmented reality that tracks the robot using the generated spatial information, determines a location where a virtual object is to be displayed, sets a restricted area where the robot cannot move, and synthesizes the virtual object with the captured image It may include a processor that controls the display unit to display an image.

And the extracted position information is a scalar value indicating two-dimensional coordinate information of the robot, the received direction information is a vector value indicating the yaw direction information of the robot, through the inertial measurement unit attached to the robot It may be acquired.

In addition, further comprising a user interface unit for receiving a user command for controlling the operation of the robot, the processor may control the communication unit to transmit a control signal corresponding to the user command input through the user interface to the robot. have.

In addition, the processor may control the communication unit to transmit a control signal for stopping the movement after moving to the boundary of the set restricted area when a user command for moving the robot into the set restricted area is input. .

Further, the processor further includes an output unit for feeding back a notification signal, and the processor controls each of the virtual objects to move according to a preset algorithm, and provides a notification signal when the position of the robot overlaps the position of the virtual object. have.

In addition, when the terrain information obtained through the sensor attached to the robot is received through the communication unit, the processor may update the restricted area based on the received terrain information.

Meanwhile, in an augmented reality service providing method for interacting with a robot according to an embodiment of the present disclosure for achieving the above object, the step of photographing an image including a robot and extracting the location information of the robot from the photographed image, the Receiving the direction information obtained by the robot, generating spatial information about the position and posture of the robot based on the extracted position information and the obtained direction information, and the robot using the generated spatial information Tracking, determining a location where the virtual object is to be displayed, setting a restricted area in which the robot cannot move, and displaying an augmented reality image obtained by synthesizing the virtual object with the captured image.

And the extracted position information is a scalar value indicating two-dimensional coordinate information of the robot, the received direction information is a vector value indicating the yaw direction information of the robot, through the inertial measurement unit attached to the robot It may be acquired.

In addition, when a user command for controlling the operation of the robot is input, transmitting a control signal corresponding to the input user command to the robot.

And the step of transmitting the control signal to the robot, when a user command for moving the robot into the set restricted area is input, a control signal for stopping the movement after moving to the border of the set restricted area is sent to the robot. Can be transmitted.

The method may further include controlling the virtual object to move according to a preset algorithm, and providing a notification signal when the position of the robot overlaps the position of the virtual object.

In addition, in the step of setting the restricted area, when the terrain information acquired through a sensor attached to the robot is received, the restricted area may be updated based on the received terrain information.

According to various embodiments of the present disclosure as described above, it is possible to easily implement markerless augmented reality through information detected by a robot that can be controlled by a user in an augmented reality screen without separate equipment, and simply manipulate virtual objects. It is possible to provide a service that can control a robot that is a real object, not an augmented reality service, on an augmented reality screen.

1 is a conceptual diagram of an augmented reality service providing system according to an embodiment of the present disclosure,
2 is a schematic block diagram for explaining the configuration of an augmented reality service providing apparatus according to an embodiment of the present disclosure;
3 is a detailed block diagram for explaining the configuration of an augmented reality service providing apparatus according to an embodiment of the present disclosure;
4 is a block diagram illustrating a configuration of a robot that interacts with an augmented reality service providing apparatus according to an embodiment of the present disclosure;
5 is a system diagram for a method of providing an augmented reality service according to an embodiment of the present disclosure;
6 and 7 are diagrams illustrating an augmented reality implementation screen according to various embodiments of the present disclosure;
8 is a flowchart illustrating a method of providing an augmented reality service that interacts with a robot according to an embodiment of the present disclosure.

Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In describing the present disclosure, if it is determined that a detailed description of a related known function or configuration may unnecessarily obscure the subject matter of the present disclosure, a detailed description thereof will be omitted. In addition, terms to be described later are terms defined in consideration of functions in the present disclosure and may vary according to a user, operator, or custom. Therefore, the definition should be made based on the contents throughout this specification.

Terms including ordinal numbers, such as first and second, may be used to describe various elements, but the elements are not limited by terms. The terms are only used for the purpose of distinguishing one component from another component. For example, without departing from the scope of the present disclosure, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element. The term and/or includes a combination of a plurality of related items or any one of a plurality of related items.

The terms used in the present specification are used to describe embodiments, and are not intended to limit and/or limit the present disclosure. Singular expressions include plural expressions unless the context clearly indicates otherwise. In the present specification, terms such as include or have are intended to refer to the presence of features, numbers, actions, components, parts, or a combination thereof described in the specification, but one or more other features, numbers, actions, and components It is to be understood that the possibility of the presence or addition of, parts or combinations thereof is not preliminarily excluded.

In an embodiment, the'module' or'unit' performs at least one function or operation, and may be implemented as hardware or software, or a combination of hardware or software. In addition, a plurality of'modules' or a plurality of'units' may be integrated into at least one module and implemented as at least one processor except for the'module' or'unit' that needs to be implemented with specific hardware.

Hereinafter, the present disclosure will be described in detail with reference to the accompanying drawings.

1 is a conceptual diagram of an augmented reality service providing system 1000 according to an embodiment of the present disclosure. Referring to FIG. 1, an augmented reality service providing system 1000 may include an augmented reality service providing apparatus 100 and a robot 200. The augmented reality service providing device 100 photographs a general image, generates an augmented reality image, and communicates with the robot 200 to receive direction information, sensor information, etc., and is implemented as any electronic device capable of transmitting a control signal. It is possible. For example, the augmented reality service providing device 100 may be implemented as a smart phone, tablet, PDA, smart glass, or the like. The robot 200 may perform various tasks according to a control signal received from the augmented reality service providing apparatus 100. In addition, the robot 200 may acquire direction information, obstacle information, terrain information, and the like by mounting various sensors.

The augmented reality service providing system 1000 fuses information that can be obtained from the camera 110 of the augmented reality service providing device 100 and the inertial measurement unit 210 of the robot 200 without a separate marker. You can track your location. The movement of the robot 200 on the 2D plane is determined through three pieces of information (eg, x coordinate information, y coordinate information, and yaw direction information in a Cartesian coordinate system). Among the three pieces of information, a two-dimensional coordinate value for the position of the robot 200 is a scalar value, and a value for the attitude or direction of the robot 200 is a vector value. Spatial information on a two-dimensional plane determined by these three pieces of information is sometimes referred to as'accurate location including object direction information'. According to an embodiment of the present disclosure, the augmented reality service providing apparatus 100 and the robot 200 divide and obtain spatial information composed of three pieces of information necessary for location tracking of the robot 200 on a two-dimensional plane. Specifically, 2D coordinate information (x coordinates and y coordinates) of the robot 200 may be extracted based on the image captured by the camera 110 of the augmented reality service providing apparatus 100. In addition, information on the yaw direction may be obtained from the inertial measurement unit 210 attached to the robot 200. The direction information obtained from the robot 200 is transmitted to the augmented reality service providing apparatus 100. The augmented reality service providing apparatus 100 having all three information elements of spatial information necessary to track the position of the robot 200 determines the generation position of the virtual object, and the virtual object is displayed in the image including the robot 200. Can be added to generate an augmented reality image. The augmented reality service providing apparatus 100 may provide a user interface through which a user can control the robot 200 displayed on an augmented reality image.

The augmented reality service providing system 1000 according to an embodiment of the present disclosure has an effect of generating an augmented reality image through real-time communication with the robot 200 without expensive motion capture equipment or a separate marker or beacon. . In addition, a user can obtain a new user experience through contents that control the robot 200, which is a real object, in an augmented reality image in which a real object and a virtual object are mixed.

Meanwhile, in describing the embodiment of FIG. 1, the augmented reality service providing system 1000 has been described as including one augmented reality service providing device 100 and one robot 200, but a plurality of augmented reality services The providing device 100 may interact with one robot 200, or may interact with a plurality of robots 200 with one augmented reality service providing device 100.

2 is a schematic block diagram illustrating the configuration of an augmented reality service providing apparatus 100 according to an embodiment of the present disclosure. Referring to FIG. 2, the augmented reality service providing apparatus 100 may include a camera 110, a communication unit 120, a display unit 130, and a processor 140.

The camera 110 may take an image. For example, the camera 110 may capture an image including the robot 200. The camera 110 may be implemented as a lens or as an image sensor such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS). In addition, the camera 110 may be implemented with a plurality of photographing elements. For example, the first photographing element may be a wide-angle lens capable of photographing a bright image with a wide angle of view due to a short focal length, and the second photographing element may be a telephoto lens capable of photographing a dark image having a narrow angle of view due to a long focal length. As another example, the plurality of photographing elements may be photographing elements having different shutter speeds (exposure times).

The communication unit 120 may communicate with an external device such as the robot 200 using a wireless network. For example, the communication unit 120 may receive direction information of the robot 200 from the robot 200 and transmit a signal for controlling the movement of the robot 200 to the robot 200. The communication unit 120 may use various methods such as Bluetooth, Zigbee communication, WiFi, infrared (InfraRed, IR) communication, and Near Field Communication (NFC) as a wireless communication method.

The display unit 130 may display an image captured by the camera 110 and an augmented reality image generated by the processor 140. In addition, the display unit 130 may display a user interface window for selecting various functions provided by the augmented reality service providing apparatus 100. For example, the display unit 130 may display a user interface window for controlling the operation of the robot 200. The display unit 130 may be implemented in various forms such as a liquid crystal display (LCD), an organic electroluminescent diode (OLED), and a plasma display panel (PDP).

The display unit 130 may be implemented in the form of a touch screen forming a layered structure together with a touch pad, and the touch screen may be configured to detect a touch input location, area, and even a pressure of the touch input. In this case, the display unit 130 may also perform the function of the user interface unit 150 to be described later.

The processor 140 may recognize the initial position of the object, track the movement of the object, and render an augmented reality image. In addition, since the processor 140 can control the movement of the robot 200, it is possible to provide content for the robot 200 to interact with virtual objects in an augmented reality image.

According to an embodiment of the present disclosure, the processor 140 may determine a position and a posture of the robot 200 in a two-dimensional plane using position information and direction information of the robot 200. The processor 140 may extract location information of the robot 200 (eg, two-dimensional coordinate information such as x and y coordinate information in a Cartesian coordinate system) from an image captured by the camera 110. For example, the processor 140 may pre-process the image captured by the camera 110 and then recognize the robot 200 using a color extraction algorithm. In addition, the processor 140 may recognize the shape of the robot 200 using a Hough Circle or Edge Detection algorithm. Through this image processing method, the processor 140 may obtain the two-dimensional position coordinates of the robot 200. The direction information of the robot 200 is obtained from the inertial measurement unit 210 mounted on the robot 200. For example, the direction information of the robot 200 may be information on a yaw direction, which is rotation information in the z direction in an orthogonal coordinate system. When the direction information of the robot 200 is received through the communication unit 120, the processor 140 creates spatial information about the position and posture of the robot 200 by fusing the obtained two-dimensional position information and the received direction information. can do. Further, the processor 140 may track the robot 200 using the generated spatial information.

According to an embodiment of the present disclosure, the processor 140 may render an augmented reality image based on the tracked position of the robot 200. The processor 140 may determine a location in which the virtual object is to be displayed. The processor 140 may generate an augmented reality image by synthesizing a virtual object with the captured image, and control the display 130 to display the generated augmented reality image. In addition, the processor 140 may set an area of the augmented reality image as an area in which the robot 200 can move. In other words, the processor 140 may set a restricted area in which the robot 200 cannot move. Separating and setting the area in which the robot 200 can move and the area in which it cannot move is advantageous in providing an augmented reality service or content in which the robot 200, which is a real object, and a virtual object interact. A more detailed operation of the processor 140 will be described later.

3 is a detailed block diagram illustrating a configuration of an augmented reality service providing apparatus 100 according to an embodiment of the present disclosure. 3, the augmented reality service providing apparatus 100 includes a camera 110, a communication unit 120, a display unit 130, a processor 140, a user interface unit 150, an output unit 160, and a memory. It may include 170. In addition to the components shown in the embodiment of FIG. 3, the augmented reality service providing apparatus 100 may include various configurations such as a power supply unit (not shown), a sensor unit (not shown), and a voice input unit (not shown). In addition, it goes without saying that the augmented reality service providing apparatus 100 is not necessarily limited to being implemented including all of the configurations shown in FIG. 3. The descriptions of the camera 110, the communication unit 120, and the display unit 130 are omitted because they overlap with those described in FIG. 2.

The user interface unit 150 may receive a control command or the like from a user. For example, the user interface unit 150 may receive a user command for controlling the operation of the robot 200, and the processor 140 transmits a control signal corresponding to the input user command to the robot 200 The communication unit 120 can be controlled so as to be. The user interface unit 150 may be implemented in the form of a touch screen, and in this case, the display unit 130 may also perform the function of the user interface unit 150. The user interface unit 150 may be implemented in the form of a voice input device that recognizes a user's voice.

The output unit 160 may feed back a notification signal, a warning signal, and a guide signal to the user. For example, the output unit 160 may provide feedback to the user in the form of displaying a notification message or the like. In this case, since a message is displayed through the display unit 130, the display unit 130 may also serve as the function of the output unit 160. As another example, the output unit 160 may feed back a notification signal to the user through sound, vibration, lighting, or the like.

The memory 170 may store various modules, software, and data for driving the augmented reality service providing apparatus 100. For example, the memory 170 includes software for augmented reality rendering, augmented reality content data including 3D modeling data of a virtual object, a control command for controlling the robot 200, a communication protocol with the robot 200, etc. Can be saved.

The memory 170 is a storage medium for storing various programs required to operate the augmented reality service providing device 100, and may be implemented in the form of a flash memory, a hard disk drive (HDD), a solid state drive (SSD), and the like. I can. For example, the memory 170 temporarily stores a program for performing an operation of the augmented reality service providing device 100, a ROM for storing an application, and data according to the operation of the augmented reality service providing device 100 It can have a RAM for. In addition, the memory 170 may further include an Electrically Erasable and Programmable ROM (EEPROM) for storing various reference data.

The processor 140 may control the above-described configurations of the augmented reality service providing apparatus 100. For example, the processor 140 may control the communication unit 120 to transmit a control signal to the robot 200 and receive direction information, sensor information, and the like from the robot 200.

The processor 140 may be implemented as a single processor (eg, one CPU or one AP) to perform an object tracking operation such as the robot 200, an augmented reality image generation operation, and a robot 200 control operation. Alternatively, it may be implemented with a plurality of processors and IPs that perform specific functions. In the embodiment of FIG. 3, it is illustrated that the processor 140 includes an augmented reality module 141, an image processing module 143, and a robot control module 145. The augmented reality module 141 may set a virtual space in which an augmented reality content such as a virtual object is displayed in an actual image captured by the camera 110. The augmented reality module 141 may receive and use data necessary to create an augmented reality image from the outside. For example, the augmented reality module 141 may receive 3D modeling data of a virtual object generated externally and spatial information data calculated by the image processing module 143 and use it to create an augmented reality image. The image processing module 143 may process an image captured by the camera 110 to recognize the position of the robot 200 and recognize a two-dimensional plane on which the robot 200 is located. For example, the image processing module 143 may recognize the robot using a color extraction algorithm or the like, and recognize the shape of the robot using a Hough Circle algorithm, an Edge Detection algorithm, or the like. The robot control module 145 may control the communication unit 120 to transmit a control signal to the robot 200 based on a predetermined motion algorithm for the movement of the robot 200. For example, when a user command is input through the user interface unit 150, the robot control module 145 may generate a control signal corresponding to the input user command as string data based on an operation algorithm.

4 is a block diagram illustrating a configuration of a robot 200 that interacts with the augmented reality service providing apparatus 100 according to an embodiment of the present disclosure. Referring to FIG. 4, the robot 200 may include an inertial measurement unit 210, a sensor unit 220, a communication unit 230, a driving unit 240, a power supply unit 250, and a processor 260. In addition to the components shown in the embodiment of FIG. 4, the robot 200 may include various components such as a memory (not shown). In addition, it goes without saying that the robot 200 is not necessarily limited to being implemented including all of the configurations shown in FIG. 4.

The inertial measurement unit 210 may detect direction information of the robot 200. The inertial measurement unit 210 may detect direction information of the robot 200 at preset time intervals. The inertial measurement unit 210 may be implemented as an inertial measurement unit (IMU) including an acceleration sensor, an angular velocity sensor, and a geomagnetic sensor.

The acceleration sensor detects the amount of change in velocity per unit time. The acceleration sensor can be implemented in three axes. When implemented as a three-axis acceleration sensor, the acceleration sensors include X, Y, and Z-axis acceleration sensors disposed in different directions and perpendicular to each other. The acceleration sensor converts the output values of each of the X, Y, and Z axis acceleration sensors into digital values and provides them to the preprocessor. In this case, the preprocessor may include a chopping circuit, an amplifying circuit, a filter, and an A/D converter. Accordingly, the electrical signal output from the 3-axis acceleration sensor is chopped, amplified, filtered, and then converted into a digital voltage value. The angular velocity sensor is a component that senses the angular velocity by detecting the amount of change in a preset direction of the robot 200 during a unit time. As for the angular velocity sensor, a gyroscope having three axes can be used. In addition to the six axes of these inertial sensors, a geomagnetic sensor can be used. The geomagnetic sensor is a sensor that can detect the azimuth angle by detecting the flow of a magnetic field. The geomagnetic sensor may detect an azimuth coordinate of the robot 200 and detect a direction in which the robot 200 is placed based on the azimuth coordinate. The geomagnetic sensor can detect geomagnetism by measuring a voltage value induced by the geomagnetism using a flux gate or the like. The geomagnetic sensor can be implemented in two or three axes.

The sensor unit 220 may detect a topography or environment around the robot 200 or a state of the robot 200 itself. The sensor unit 220 may include various sensors such as an obstacle sensor 221 and an image sensor 223. In the present specification, the sensor unit 220 may be defined as a concept excluding sensors included in the inertial measurement unit 210.

The obstacle sensor 221 may detect an obstacle that obstructs the movement of the robot 200. For example, the obstacle sensor 221 may include a non-contact detection sensor and a contact collision/bumper sensor capable of identifying walls, gaps, columns, thresholds, mounds, etc., through which the robot 200 cannot pass.

The image sensor 223 may generate environmental information by photographing the surroundings of the robot 200. Also, the image sensor 223 may determine the location of the robot 200 by photographing the ceiling. In addition, the image sensor 223 may photograph objects located in the moving direction of the robot 100. For example, the image sensor 223 may be implemented as a Charge Coupled Device (CCD) device or a Complementary Metal Oxide Semiconductor (CMOS) device.

The communication unit 230 may transmit and receive data, control signals, and the like with an external device. For example, the communication unit 230 may transmit direction information of the robot 200 to the augmented reality service providing apparatus 100. In addition, the communication unit 230 may receive a control signal from the augmented reality service providing apparatus 100. The communication unit 230 may use various methods such as Bluetooth, Zigbee communication, WiFi, infrared (InfraRed, IR) communication, and Near Field Communication (NFC) as a wireless communication method.

The driving unit 240 may drive the robot 200. For example, the driving unit 240 may move the robot 200 to a position to perform the task under the control of the processor 260. In this case, the driving unit 240 may include at least one wheel in contact with the floor, a motor providing power to the wheel, and a driver to control the motor. As another example, the driving unit 240 may perform an operation to perform a task. In the case of an object movement task, the driving unit 240 may include a motor for performing an operation of picking up an object.

The power supply unit 250 supplies power required for driving the robot 200. For example, the power supply unit 250 may be implemented as a battery capable of charging and discharging. The processor 260 may control the driving unit 240 to move to the charging station when the remaining power of the robot 200 falls below a preset level or when a task is completed. The power supply unit 250 may be charged in any of a contact type and a non-contact type charging method.

The memory (not shown) may store direction information of the robot 200, map information around the robot 200, and various types of information sensed by the sensor unit 220. Various programs required to operate the robot 200 may be stored in the memory (not shown).

The processor 260 may control the above-described components of the robot 200. When a control signal is received from the augmented reality service providing apparatus 100 through the communication unit 230, the processor 260 may control the driving unit 240 so that the robot 200 performs an operation according to the received control signal. . The processor 260 may control the inertial measurement unit 210 and the sensor unit 220 to acquire direction information, terrain information, and the like of the robot 200 at each preset period. The processor 260 may control the communication unit 230 to transmit periodically acquired direction information, terrain information, and the like to the augmented reality service providing apparatus 100. Through this, the augmented reality service providing apparatus 100 may update the moving direction of the robot 200, the terrain around the robot 200, and the state of the robot 200.

5 is a system diagram for a method of providing an augmented reality service according to an embodiment of the present disclosure. The augmented reality service providing system 1000 according to an embodiment of the present disclosure may include an augmented reality service providing device 100 and a robot 200.

Referring to FIG. 5, the augmented reality service providing apparatus 100 may capture an image including the robot 200 (S505). This is because an augmented reality image will be generated by rendering and synthesizing a virtual object around the robot 200 or the robot 200. The augmented reality service providing apparatus 100 may extract the location information of the robot 200 by analyzing the captured image (S510). The augmented reality service providing apparatus 100 may pre-process the captured image and recognize the robot 200 from the image based on a color or shape. For example, the augmented reality service providing apparatus 100 may convert an image acquired through an RGB channel into an HSV scale and recognize the robot 200 from a captured image using a color extraction algorithm. In addition, the augmented reality service providing apparatus 100 may extract the position coordinates of the robot 200 in a two-dimensional plane using an algorithm capable of recognizing the shape of the robot 200. For example, when using a Cartesian coordinate system, the augmented reality service providing apparatus 100 may acquire two scalar values, an x coordinate and a y coordinate value.

The augmented reality service providing system 1000 according to an embodiment of the present disclosure also uses the robot 200 that can be manipulated as part of the augmented reality content as a means for acquiring information for implementing augmented reality. Through this, markerless augmented reality can be easily implemented without additional equipment. The robot 200 may acquire its own direction information, which is a vector value, through the inertial measurement unit 210 attached to the robot 200 (S515). The direction information of the robot 200 sensed through an angular velocity sensor, a geomagnetic sensor, etc. is transmitted to the augmented reality service providing apparatus 100 (S520). Two of the three pieces of information for determining the position and posture of the robot 200 in the two-dimensional plane are information obtained by the augmented reality service providing apparatus 100, and the other one is information obtained by the robot 200.

The augmented reality service providing apparatus 100 that has received the direction information from the robot 200 may create spatial information on the position and posture of the robot by fusing the 2D coordinate information obtained by itself and the received direction information ( S525). In addition, the augmented reality service providing apparatus 100 may track the robot 200 using the generated spatial information (S530). The augmented reality service providing apparatus 100 may determine a virtual object to be placed according to the contents of the augmented reality content to be provided. In addition, since the position of the robot 200 included in the image has been determined, the augmented reality service providing apparatus 100 may determine a position in which the virtual object is to be displayed based on the position or posture of the robot 200 (S535). In addition, the augmented reality service providing apparatus 100 may set a restricted area on the screen in which the robot 200 cannot move (S540). In the embodiment of FIG. 5, steps S535 and S540 are shown to be sequentially performed, but step S540 may be performed prior to step S535, or steps S535 and S540 may be performed in parallel. The augmented reality service providing apparatus 100 may provide restricted area information to the robot 200 (S545). The robot 200 may store the received restricted area information and use it when performing a moving task.

The augmented reality service providing apparatus 100 may generate an augmented reality image by synthesizing a virtual object at a determined location and display the generated augmented reality image (S550). When content capable of controlling the robot 200 is provided in the augmented reality image, a user interface for receiving a user command for controlling the robot 200 may be displayed in one area of the augmented reality image. When the user inputs a user command through the user interface (S555), the augmented reality service providing apparatus 100 may transmit a preset control signal corresponding to the user command to the robot 200 (S560). The robot 200 may perform a preset operation in the augmented reality image according to the received control signal (S565).

The augmented reality service providing system 1000 as described above uses the actual robot 200 constituting a part of the augmented reality content as a device for acquiring direction information, which is one of the spatial information, and is augmented without a separate marker or beacon. Reality service can be provided. In addition, the augmented reality service providing system 1000 may provide a new user experience to a user by providing an augmented reality service capable of not only interacting with the robot 200 that is a real object as well as a virtual object.

6 is a diagram illustrating an example of an augmented reality service provided by the augmented reality service providing apparatus 100. An augmented reality image including the robot 200 is displayed on the screen of the augmented reality service providing apparatus 100 implemented as a mobile device. The position and posture of the robot 200 are recognized based on information respectively acquired by the augmented reality service providing apparatus 100 and the robot 200. In the screen of FIG. 6, a rectangular virtual box is displayed at the recognized position of the robot 200. In addition, virtual objects such as animals and shells are displayed around the robot 200. Since the augmented reality service providing device 100 and the robot 200 periodically update 2D coordinate information and direction information, respectively, the augmented reality service providing device 100 can track the robot 200 in real time. For example, the augmented reality service providing apparatus 100 may provide an augmented reality game content that moves a shell, which is a virtual object, to a location where the robot 200 is based on the location of the robot 200 tracked in real time. . As another example, the virtual object may move according to a preset algorithm. If it is determined that the positions of the virtual object and the robot 200 overlap, the augmented reality service providing apparatus 100 may output a notification signal.

As illustrated in FIG. 6, a user interface 610 for receiving a user command for manipulating the robot 200 may be displayed in an area of the screen where an augmented reality image is displayed. In FIG. 6, a boundary 620 of a restricted area in a rectangular shape is displayed around the robot 200. When a user command for moving the robot 200 beyond the boundary 620 of the restricted area into the restricted area is input, the augmented reality service providing device 100 is a control signal that allows the robot 200 to move only to the boundary of the restricted area. Can be transmitted to the robot 200.

7 is a diagram illustrating an example of an augmented reality image for explaining setting and updating a restricted area. As described above, the augmented reality service providing apparatus 100 may determine the location of the virtual object based on the location of the robot 200. In addition, the augmented reality service providing apparatus 100 may set an area in which the robot 200 is movable and augmented reality content is provided. In other words, it is possible to set a restricted area in which the robot 200 cannot move and augmented reality content is not provided. The robot 200 cannot move beyond the boundary line 710 of the restricted area shown in FIG. 7, and an augmented reality content that interacts with the robot 200 is provided only within the boundary line 710 of the restricted area. The first restricted area is determined according to the position of the robot 200 in the image. In response to content provided inside the boundary line 710 of the restricted area, background screen content may be provided outside the boundary line 710 of the restricted area. For example, a virtual background object corresponding to the spectator seat and spectators is synthesized outside the boundary line 710 of the restricted area, and the robot 200 and the virtual object play sports inside the boundary line 710 of the restricted area. Can be provided.

The robot 200 includes a sensor unit 220 including a sensor capable of detecting an obstacle as well as an inertial measurement unit 210 that senses a direction. The robot 200 provides an augmented reality service and may acquire topographic information by detecting nearby obstacles, walls, etc. using sensors attached during operation. The augmented reality providing apparatus 100 receiving the terrain information obtained from the robot 200 may update the restricted area. As an example of FIG. 7, the robot 200 provides an augmented reality service and may detect an obstacle 720 or a wall 730 inside the boundary 710 of the restricted area while moving. The robot 200 transmits topographic information about the detected obstacle 720 and the position and shape of the wall 730 to the augmented reality service providing device 100, and the augmented reality service providing device 100 is the obstacle 720 Alternatively, the area in which the wall 730 exists may be additionally included as a restricted area. If the robot 200 detects that the obstacle 720 has been removed while providing the augmented reality service, the augmented reality service providing apparatus 100 may exclude the area where the obstacle 720 was placed from the restricted area. The augmented reality service providing apparatus 100 may determine a background object to be displayed outside the boundary line 710 of the restricted area based on the updated shape of the restricted area. For example, if the robot 200 detects a lot of obstacles and the restricted area is updated in a narrow and long shape, the augmented reality service providing apparatus 100 creates a background object in the shape of a canyon outside the boundary line 710 of the restricted area. It can be synthesized.

8 is a flowchart illustrating a method of providing an augmented reality service that interacts with a robot according to an embodiment of the present disclosure. Referring to FIG. 8, the augmented reality service providing apparatus 100 may capture an image including the robot 200 and extract location information from the captured image (S810). The location information may be composed of two scalar values representing coordinates in a two-dimensional plane of the robot 200. The augmented reality service providing apparatus 100 may extract 2D coordinate information where the robot 200 is located by using an algorithm that distinguishes the color and shape of the robot 200 from the background.

Further, the augmented reality service providing apparatus 100 may receive direction information acquired by the robot 200 from the robot 200 (S820). The robot 200 may detect the direction it is facing through the inertial measurement unit 210. For example, the robot 200 may acquire direction information on how much it is inclined in the yaw direction, which is a vector value. The robot 200 may transmit the acquired direction information to the augmented reality service providing apparatus 100.

The augmented reality service providing apparatus 100 fuses the direction information received from the robot 200 and the 2D coordinate information obtained through image processing to provide spatial information on the position and posture of the robot 200 in the 2D plane. Can be generated (S830). In addition, the augmented reality service providing apparatus 100 may track the robot 200 based on the generated spatial information. In addition, the augmented reality service providing apparatus 100 may determine a virtual object to be rendered based on the augmented reality content to be provided. In addition, the augmented reality service providing apparatus 100 may determine a location in which the virtual object is to be displayed based on the location of the robot 200. In addition, the augmented reality service providing apparatus 100 may set a restricted area in an area of the augmented reality image (S840). Augmented reality content is provided in an area other than the restricted area, and the robot 200 may move under the control of the augmented reality service providing apparatus 100. The augmented reality service providing apparatus 100 may display an augmented reality image by synthesizing a virtual object (S850). In addition to the virtual object, a user interface for controlling the robot 200, a boundary of a restricted area, etc. may be included in the augmented reality image and displayed. Descriptions of various embodiments of the other augmented reality service providing method will be omitted since they overlap with the description of the augmented reality service providing apparatus 100.

The methods described above may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the present disclosure, or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -Includes hardware devices specially configured to store and execute program instructions such as magneto-optical media, SSD, ROM, RAM, flash memory, etc. Examples of the program instructions include not only machine language codes such as those produced by a compiler, but also language codes that can be executed by a computer using an interpreter or the like. The hardware device described above may be configured to operate as one or more software modules to perform the operation of the present disclosure, and vice versa.

Although the present disclosure as described above has been described by the limited embodiments and drawings, the present disclosure is not limited to the above embodiments, and various modifications and variations from these descriptions are those of ordinary skill in the field to which the present disclosure belongs. This is possible. Therefore, the scope of the present disclosure is limited to the described embodiments and should not be determined, and should be determined by the claims to be described later as well as those equivalent to the claims.

100: augmented reality service providing device 110: camera
120: communication unit 130: display unit
140: processor 150: user interface unit
160: output unit 170: memory
200: robot 210: inertial measurement unit
220: sensor unit 230: communication unit
240: driving unit 250: power supply
260: processor 1000: augmented reality service providing system

Claims (12)

In an augmented reality service providing device that interacts with a robot,
A communication unit communicating with the robot;
A camera that takes an image;
A display unit that displays an image; And
When the position information of the robot is extracted from the image captured by the robot by the camera, and the direction information acquired by the robot is received through the communication unit, the robot based on the extracted position information and the received direction information Generates spatial information about the position and posture of the robot, tracks the robot using the generated spatial information, determines a position where a virtual object is to be displayed, sets a restricted area in which the robot cannot move, and sets the captured image A processor that controls the display unit to display an augmented reality image obtained by synthesizing the virtual object in
The restricted area is an augmented reality service providing apparatus which is an area other than an area in which an augmented reality image interacting with the robot is provided. The method of claim 1,
The extracted location information is a scalar value representing two-dimensional coordinate information of the robot,
The received direction information is a vector value indicating information on a yaw direction of the robot, and is obtained through an inertial measurement unit attached to the robot. The method of claim 1,
A user interface unit receiving a user command for controlling the operation of the robot; further comprising,
The processor,
An augmented reality service providing apparatus for controlling the communication unit to transmit a control signal corresponding to a user command input through the user interface unit to the robot. In an augmented reality service providing device that interacts with a robot,
A communication unit communicating with the robot;
A camera that takes an image;
A display unit that displays an image;
A user interface unit receiving a user command for controlling the operation of the robot; And
When the position information of the robot is extracted from the image captured by the robot by the camera, and the direction information acquired by the robot is received through the communication unit, the robot based on the extracted position information and the received direction information Generates spatial information about the position and posture of the robot, tracks the robot using the generated spatial information, determines a position where a virtual object is to be displayed, sets a restricted area in which the robot cannot move, and sets the captured image A processor for controlling the display unit to display an augmented reality image obtained by synthesizing the virtual object in
The processor is an augmented reality that controls the communication unit to transmit a control signal to the robot to stop the movement after moving to the boundary of the set restricted area when a user command for moving the robot into the set restricted area is input. Service provision device. In an augmented reality service providing device that interacts with a robot,
A communication unit communicating with the robot;
A camera that takes an image;
A display unit that displays an image;
An output unit for feeding back a notification signal; And
When the position information of the robot is extracted from the image captured by the robot by the camera, and the direction information acquired by the robot is received through the communication unit, the robot based on the extracted position information and the received direction information Generates spatial information about the position and posture of the robot, tracks the robot using the generated spatial information, determines a position where a virtual object is to be displayed, sets a restricted area in which the robot cannot move, and sets the captured image A processor for controlling the display unit to display an augmented reality image obtained by synthesizing the virtual object in
The processor, an augmented reality service providing apparatus that controls the virtual object to move according to a preset algorithm, and controls the output unit to provide a notification signal when the position of the robot overlaps the position of the virtual object. In an augmented reality service providing device that interacts with a robot,
A communication unit communicating with the robot;
A camera that takes an image;
A display unit that displays an image; And
When the position information of the robot is extracted from the image captured by the robot by the camera, and the direction information acquired by the robot is received through the communication unit, the robot based on the extracted position information and the received direction information Generates spatial information about the position and posture of the robot, tracks the robot using the generated spatial information, determines a position where a virtual object is to be displayed, sets a restricted area in which the robot cannot move, and sets the captured image A processor that controls the display unit to display an augmented reality image obtained by synthesizing the virtual object in
The processor, when receiving the terrain information obtained through the sensor attached to the robot through the communication unit, an augmented reality service providing apparatus for updating the restricted area based on the received terrain information. In the method of providing an augmented reality service that interacts with a robot,
Capturing an image including the robot with a camera located outside the robot, and extracting location information of the robot from the captured image;
Receiving direction information acquired by the robot;
Generating spatial information about the position and posture of the robot based on the extracted position information and the obtained direction information;
Tracking the robot using the generated spatial information, determining a position where a virtual object is to be displayed, and setting a restricted area in which the robot cannot move; And
Including; displaying an augmented reality image obtained by synthesizing a virtual object with the captured image,
The augmented reality service providing method in which the restricted area is an area other than an area in which an augmented reality image interacting with the robot is provided. The method of claim 7,
The extracted location information is a scalar value representing two-dimensional coordinate information of the robot,
The received direction information is a vector value indicating information on a yaw direction of the robot, and is obtained through an inertial measurement unit attached to the robot. The method of claim 7,
When a user command for controlling the operation of the robot is input, transmitting a control signal corresponding to the input user command to the robot. In the method of providing an augmented reality service that interacts with a robot,
Capturing an image including the robot and extracting location information of the robot from the captured image;
Receiving direction information acquired by the robot;
Generating spatial information about the position and posture of the robot based on the extracted position information and the obtained direction information;
Tracking the robot using the generated spatial information, determining a position where a virtual object is to be displayed, and setting a restricted area in which the robot cannot move;
Displaying an augmented reality image obtained by synthesizing a virtual object with the captured image; And
When a user command to move the robot into the set restricted area is input, transmitting a control signal to stop the movement after moving to the boundary of the set restricted area to the robot. In the method of providing an augmented reality service that interacts with a robot,
Capturing an image including the robot and extracting location information of the robot from the captured image;
Receiving direction information acquired by the robot;
Generating spatial information about the position and posture of the robot based on the extracted position information and the obtained direction information;
Tracking the robot using the generated spatial information, determining a position where a virtual object is to be displayed, and setting a restricted area in which the robot cannot move;
Displaying an augmented reality image obtained by synthesizing a virtual object with the captured image;
Controlling the virtual object to move according to a preset algorithm; And
Providing a notification signal when the location of the robot overlaps the location of the virtual object; augmented reality service providing method comprising a. In the method of providing an augmented reality service that interacts with a robot,
Capturing an image including the robot and extracting location information of the robot from the captured image;
Receiving direction information acquired by the robot;
Generating spatial information about the position and posture of the robot based on the extracted position information and the obtained direction information;
Tracking the robot using the generated spatial information, determining a position where a virtual object is to be displayed, and setting a restricted area in which the robot cannot move; And
Including; displaying an augmented reality image obtained by synthesizing a virtual object with the captured image,
The step of setting the restricted area,
When the terrain information acquired through the sensor attached to the robot is received, the augmented reality service providing method of updating the restricted area based on the received terrain information.

Images