KR102639276B1 - Remote control device and system for remotely controlling robot using various work contents in virtual space - Google Patents

Abstract

The present invention discloses a remote control device and system that remotely controls a robot using various work contents in virtual space. The remote control device of the present invention detects an object using the received information when it receives image information and 3D spatial information from the communication unit and robot that communicates with the robot and the user terminal, and determines the robot's movement path based on the detected object. It generates content that displays a user interface, object, and movement path that can be controlled by selecting the task of the robot on the user terminal based on image information, and includes a control unit that transmits the generated content to the user terminal. .

Description

Remote control device and system for remotely controlling robot using various work contents in virtual space}

The present invention relates to remote control technology for robots, and more specifically, using various work contents in virtual space that support smooth robot control by expressing the type and area of work objects in real images in virtual space from a remote location. This relates to remote control devices and systems that control robots remotely.

Disasters such as forest fires, earthquakes, and tsunamis themselves cause great damage to people, but not only do they cause secondary damage due to damage to industrial facilities, but the secondary damage often causes casualties.

For example, a magnitude 9.0 earthquake that occurred in Japan's East Japan Sea on March 11, 2011 generated a tsunami of more than 14 meters, causing many casualties and damaging the Fukushima nuclear power plant, causing serious damage from radioactive leaks.

In such a disaster situation, the Fukushima nuclear power plant was damaged, causing serious damage in the form of a radioactive leak.

In the case of the Fukushima nuclear power plant accident, the direct involvement of humans in tasks such as assessing the condition of the nuclear power plant or making repairs to block radiation leaks poses a significant risk factor, so mobile robots are needed to replace people. Robots perform various functions in disaster situations, such as preventing the spread of disaster, minimizing damage, and handling accidents.

Meanwhile, in the case of robots controlled in situations where it is difficult for humans to be involved, most robots are controlled remotely by humans. Therefore, a remote coordinator installs a camera on the robot so that it can be adjusted while checking the robot's surrounding environment, and controls the robot while watching the captured images transmitted by the camera. However, this control has many limitations in remote control by the coordinator, and the control itself is difficult.

Korean Patent Publication No. 10-1478908 (2015.01.02.)

The technical problem to be achieved by the present invention is a remote control device and system that remotely controls a robot using various work contents in virtual space that represent the type and area of the work object in the actual image in virtual space to make work in remote locations easier. The purpose is to provide.

Another technical problem to be achieved by the present invention is to provide a remote control device and system that remotely controls a robot by providing various work contents in a virtual space where the user can selectively recognize only the desired objects.

In order to achieve the above object, a remote control device that remotely controls a robot using various work contents in a virtual space according to the present invention includes a communication unit that communicates with the robot and a user terminal, and image information and 3D spatial information from the robot. When received, an object can be detected using the received information, a movement path of the robot can be created based on the detected object, and a task of the robot can be selected and controlled from the user terminal based on the image information. It includes a control unit that generates content that displays a user interface, the object, and the movement path, and transmits the generated content to the user terminal.

In addition, the control unit is characterized in that it detects the object using a deep learning-based artificial intelligence model.

In addition, the control unit is characterized in that it generates the content so that only desired objects are selectively displayed according to a user input related to object selection received from the user terminal.

In addition, the control unit is characterized in that it generates the content so that only objects related to the task are selectively displayed according to a user input related to task selection received from the user terminal.

In addition, the control unit generates the content so that the relative distance between the object and the robot is displayed using the 3D spatial information.

In addition, the control unit further receives posture information of the work tool mounted on the robot, and generates the content so that guidelines for manipulating the object are displayed through an extension line when the work tool is directed to the object. Do it as

In addition, the control unit is characterized in that it generates the content so that first-person image information and third-person robot status are simultaneously expressed.

The remote control system according to the present invention is capable of moving and supports a robot that performs a preset task, a user terminal that remotely controls the robot, and the user terminal remotely controls the robot between the robot and the user terminal. It includes a remote control device, wherein the remote control device receives image information and 3D spatial information from a communication unit that communicates with the robot and the user terminal, and detects an object using the received information. , A user interface that generates a movement path of the robot based on the detected object and can select and control the robot's work at the user terminal based on the image information, and where the object and the movement path are displayed. It is characterized by comprising a control unit that generates content and transmits the generated content to the user terminal.

The remote control device and system of the present invention that controls a robot remotely using various work contents in a virtual space expresses the type and area of the work object in the actual image in the virtual space to make work in remote areas easier, thereby providing information on the object. can be expressed diversely and accurately.

Additionally, work convenience can be maximized by providing a separate user interface (UI) so that users can selectively recognize only the objects they want.

1 is a configuration diagram for explaining a remote control system according to an embodiment of the present invention.
Figure 2 is a block diagram for explaining a remote control device according to an embodiment of the present invention.
FIG. 3 is a block diagram for explaining the control unit of FIG. 2.
FIG. 4 is a block diagram for explaining the virtual environment creation unit of FIG. 3.
Figure 5 is a diagram for explaining content displayed on a user interface in a virtual space according to an embodiment of the present invention.
FIG. 6 is a diagram for explaining how the content of FIG. 5 is output from the HMD.

Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings. First, it should be noted that when adding reference numerals to components in each drawing, the same components are given the same reference numerals as much as possible even if they are shown in different drawings. Additionally, in describing the present invention, if a detailed description of a related known configuration or function is determined to be obvious to those skilled in the art or may obscure the gist of the present invention, the detailed description will be omitted.

1 is a configuration diagram for explaining a remote control system according to an embodiment of the present invention.

Referring to FIG. 1, the remote control system 400 expresses the type and area of the work object of the actual image in a virtual space to facilitate work in a remote location. The remote control system 400 controls the robot remotely by providing various work contents in a virtual space where the user can selectively recognize only the desired objects. The remote control system 400 includes a remote control device 100, a robot 200, and a user terminal 300.

The remote control device 100 supports the user terminal 300 to remotely control the robot 200 between the robot 200 and the user terminal 300. The remote control device 100 generates content to be output from the user terminal 300 based on the image information and 3D spatial information received from the robot 200 and transmits the generated content to the user terminal 300. Here, the content is content that supports the robot 200 to perform various tasks in a virtual space, and includes a user interface that allows selection of tasks, objects, movement paths, etc. according to user input. The remote control device 100 may be a server computer including a desktop, server computer, cluster computer, etc.

The robot 200 communicates with the remote control device 100 and moves or performs tasks according to user input received from the remote control device 100. To this end, the robot 200 includes a moving means that enables movement, and robot joints, arms, etc. to perform preset tasks. Preset tasks may include object manipulation using work tools, object movement, etc. In addition, the robot 200 can collect three-dimensional spatial information by measuring image information and depth information, including a stereo camera and LiDAR, and joint information that detects the state of each joint. can also be collected. The robot 200 transmits the collected information to the remote control device 100.

The user terminal 300 communicates with the remote control device 100, receives content related to the robot 200 from the remote control device 100, and outputs the received content. When a user input corresponding to output content is input, the user terminal 300 transmits the input user input to the remote control device 100 so that the robot 200 is controlled according to the user input. The user terminal 300 may be a personal computing system including a Head Mounted Display (HMD), smartphone, laptop, desktop, tablet PC, handheld PC, etc., and is preferably an HMD specialized for outputting virtual space. You can.

Meanwhile, the remote control system 400 includes a communication network 450 for communicating between the remote control device 100, the robot 200, and the user terminal 300. The communication network 450 may be composed of a backbone network and a subscriber network. The backbone network can be composed of one of the X.25 network, Frame Relay network, ATM network, MPLS (Multi Protocol Label Switching) network, and GMPLS (Generalized Multi Protocol Label Switching) network, or multiple integrated networks. Subscriber networks include FTTH (Fiber To The Home), ADSL (Asymmetric Digital Subscriber Line), cable network, zigbee, Bluetooth, and Wireless LAN (IEEE 802.11b, IEEE 802.11a, IEEE 802.11g, IEEE 802.11n). ), Wireless Hart (ISO/IEC62591-1), ISA100.11a (ISO/IEC 62734), CoAP (Constrained Application Protocol), MQTT (Message Queuing Telemetry Transport), WIBro (Wireless Broadband), Wimax, 3G, HSDPA (High Speed Downlink Packet Access), 4G and 5G. In some embodiments, the communication network 450 may be an Internet network or a mobile communication network. Additionally, the communication network 450 may include any wireless or wired communication method that is well known or will be developed in the future.

Figure 2 is a block diagram for explaining a remote control device according to an embodiment of the present invention.

Referring to Figures 1 and 2, the remote control device 100 includes a communication unit 10 and a control unit 30, and may further include a storage unit 50.

The communication unit 10 communicates with the robot 200 and the user terminal 300. The communication unit 10 receives image information, 3D spatial information, joint information, etc. from the robot 200 and transmits a user input for controlling the robot to the robot 200. The communication unit 10 transmits content related to the robot to the user terminal 300 and receives user input for controlling the robot from the user terminal 300.

The control unit 30 performs overall control of the remote control device 100. When the control unit 30 receives image information and 3D spatial information from the robot 200, it detects an object using the received information. At this time, the control unit 30 may detect the object by performing deep learning-based object segmentation (instance segmentation). The control unit 30 creates a movement path of the robot 200 based on the detected object. Additionally, the control unit 30 generates content that displays a user interface, objects, and movement paths that can be controlled by selecting a task for the robot on the user terminal 300. The control unit 30 transmits the generated content to the user terminal 300. When a user input corresponding to content transmitted from the user terminal 300 is received, the control unit 30 controls the content according to the user input, and when a user input for controlling the robot 200 is received, the control unit 30 controls the user input to the robot 200. ) is transmitted. Through this, the control unit 30 can enable the robot 200 to operate according to user input.

The storage unit 50 stores a program or algorithm for the remote control device 100 to operate. The storage unit 50 stores image information, 3D spatial information, and joint information received from the robot 200. The storage unit 50 stores content generated by the control unit 30. Additionally, the storage unit 50 stores user input received from the user terminal 300. The storage unit 50 includes a flash memory type, hard disk type, multimedia card micro type, card type memory (for example, SD or XD memory, etc.), Random Access Memory (RAM), Static Random Access Memory (SRAM), Read-Only Memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Programmable Read-Only Memory (PROM), magnetic memory, It may include at least one storage medium of a magnetic disk and an optical disk.

FIG. 3 is a block diagram for explaining the control unit of FIG. 2, FIG. 4 is a block diagram for explaining the virtual environment creation unit of FIG. 3, and FIG. 5 shows a user interface in a virtual space according to an embodiment of the present invention. This is a diagram for explaining content, and FIG. 6 is a diagram for explaining how the content in FIG. 5 is output from the HMD.

1 to 5, the control unit 30 includes an object detection unit 31, a movement path creation unit 33, and a virtual environment creation unit 35.

The object detection unit 31 detects an object using image information and 3D spatial information from the robot 200. The object detection unit 31 detects the object area using , implemented based on deep learning, and recognizes the object included in the detected object area. Here, the artificial intelligence model may include, but is not limited to, R-CNN, Fast R-CNN, Faster R-CNN, R-FCN, YOLO, SSD, etc.

The movement path generator 33 generates a movement path along which the robot 200 moves according to the user input received from the user terminal 300. That is, the movement path creation unit 33 can create a movement path that moves from the current location of the robot 200 to the destination included in the user input along an optimal path. For example, the movement path generator 33 may generate a movement path so that the robot 200 moves to the final destination while avoiding various obstacles in the environment where the robot 200 is located. At this time, the movement path creation unit 33 may generate a movement path by applying SLAM (Simultaneous Localization And Mapping) technology based on the detected object and 3D spatial information.

The virtual environment creation unit 35 generates content that displays a user interface, objects, and movement paths that can select and control robot tasks based on image information. The virtual environment creation unit 35 may generate content so that only desired objects are selectively displayed according to user input related to object selection received from the user terminal 300. The virtual environment creation unit 35 may generate content so that only objects related to the task are selectively displayed according to a user input related to task selection received from the user terminal 300. For example, if the task is firefighting, you can create content so that only objects related to firefighting, such as fire extinguishers, hoses, taps, etc., are displayed. The virtual environment creation unit 35 may generate content to display the relative distance between the object and the robot 200 using 3D spatial information. In addition, the virtual environment creation unit 35 further receives posture information of the work tool mounted on the robot 200 from the robot 200, and when the work tool is directed to the object, a guideline for manipulating the object is displayed through an extension line. You can create content as much as possible. Here, the virtual environment creation unit 35 can generate content to simultaneously express first-person image information and third-person robot status. In order to provide such content, the virtual environment creation unit 35 may include a module that provides an individual user interface, preferably a robot status display UI 41, an object detection display UI 43, and a movement path display UI. (45), including the task type selection UI (47). For example, the virtual environment creation unit 35 displays a UI for selecting a task type based on image information, a UI for selecting an object recognition object, and a UI for selecting output data, and displays three-dimensional robot posture, movement path, etc. Content that displays relative distances, objects, object manipulation guidelines, etc. can be created (Figure 4). In particular, when the user terminal 300 is an HMD, the virtual environment creation unit 35 generates content to match the HMD on which the content is output (FIG. 5), so that the user can easily and accurately recognize the current state of the robot. help.

Although the preferred embodiments have been described and illustrated above to illustrate the technical idea of the present invention, the present invention is not limited to the configuration and operation as shown and described, and does not depart from the scope of the technical idea. Those skilled in the art will appreciate that many changes and modifications are possible to the present invention. Accordingly, all such appropriate changes, modifications and equivalents shall be considered to fall within the scope of the present invention.

10: Department of Communications
30: control unit
31: object detection unit
33: Movement path creation unit
35: Virtual environment creation unit
41: Robot status display UI
43: Object detection display UI
45: Movement path display UI
47: Task type selection UI
50: storage unit
100: remote control device
200: Robot
300: User terminal
400: remote control system
450: communication network

Claims (8)

A communication unit that communicates with the robot and the user terminal; and
When image information and 3D spatial information are received from the robot, an object is detected using the received information, a movement path of the robot is generated based on the detected object, and the user terminal is based on the image information. It includes a user interface capable of selecting and controlling the robot's tasks, a control unit that generates content displaying the object and the movement path, and transmits the generated content to the user terminal;
The control unit,
A remote control device for remotely controlling a robot using various work contents in a virtual space, characterized in that the content is generated so that only desired objects are selectively displayed according to a user input related to object selection received from the user terminal. According to clause 1,
The control unit,
A remote control device that remotely controls a robot using various work contents in a virtual space, characterized by detecting the object using a deep learning-based artificial intelligence model. delete According to clause 1,
The control unit,
A remote control for controlling a robot remotely using various task contents in a virtual space, characterized in that the content is generated so that only objects related to the task are selectively displayed according to a user input related to task selection received from the user terminal. Control device. According to clause 1,
The control unit,
A remote control device for remotely controlling a robot using various work contents in a virtual space, characterized in that the content is generated to display the relative distance between the object and the robot using the 3D spatial information. According to clause 1,
The control unit,
Further receiving posture information of the work tool mounted on the robot, and generating the content so that guidelines for manipulating the object are displayed through an extension line when the work tool is directed to the object. A remote control device that remotely controls a robot using various work contents. According to clause 1,
The control unit,
A remote control device that remotely controls a robot using various work contents in a virtual space, characterized in that the content is generated to simultaneously express first-person image information and third-person robot status. A robot that can move and perform preset tasks;
a user terminal that remotely controls the robot; and
Includes; a remote control device between the robot and the user terminal that supports the user terminal to remotely control the robot;
The remote control device is,
A communication unit that communicates with the robot and the user terminal; and
When image information and 3D spatial information are received from the robot, an object is detected using the received information, a movement path of the robot is generated based on the detected object, and the user terminal is based on the image information. It includes a user interface capable of selecting and controlling the robot's tasks, a control unit that generates content displaying the object and the movement path, and transmits the generated content to the user terminal;
The control unit,
Characterized in controlling the robot remotely using various work contents in a virtual space, characterized in that the content is generated so that only desired objects are selectively displayed according to user input related to object selection received from the user terminal. Remote control system.

Images