KR102422355B1 - An remote control system based on augmented reality for underater robot sterilization robot based on 3d environmental recognition - Google Patents

Abstract

The present invention relates to an underwater robot control system, and more particularly, to a robot control system capable of remotely controlling an underwater robot using augmented reality technology.

Description

AR-based underwater robot remote control system {AN REMOTE CONTROL SYSTEM BASED ON AUGMENTED REALITY FOR UNDERATER ROBOT STERILIZATION ROBOT BASED ON 3D ENVIRONMENTAL RECOGNITION}

The present invention relates to an underwater robot control system, and more particularly, to a robot control system capable of remotely controlling an underwater robot using augmented reality technology.

In the case of work that requires progress underwater, it takes a lot of time and money for a person such as a diver to proceed, and furthermore, there are many risks. have.

In the case of underwater robots that currently perform tasks on behalf of humans in the water, unintegrated information is displayed on multiple monitors, the multi-joint robot arm is controlled with a non-intuitive control device, and the handling convenience is improved. In the case of , there is a problem that the structure is excessively complicated.

Furthermore, since it is operated in such a way that a plurality of workers cooperate to control one underwater robot, there is a problem that the operation efficiency is reduced and the training time for adjusting the underwater robot is excessively long.

Therefore, there is an urgent need to develop a control system for efficient operation and operation of underwater robots.

On the other hand, the following prior art document discloses a technology related to an underwater robot for irradiating an underwater tunnel and a method for controlling an underwater robot, but the technical gist of the present invention is not disclosed.

Republic of Korea Patent Publication No. 20-2019-0136384

The augmented reality-based underwater robot remote control system according to an embodiment of the present invention aims to solve the following problems in order to solve the above problems.

It is to provide an underwater robot remote control system that can increase the work efficiency of workers at the robot's work site by building an intuitive interface so that human movements are directly applied to robot control.

In addition, by integrating various information required for robot operation into augmented reality and providing the operator with an underwater robot remote control system, only one or a few workers can control the underwater robot.

The problems to be solved of the present invention are not limited to those mentioned above, and other problems not mentioned will be clearly understood by those of ordinary skill in the art from the following description.

An augmented reality-based underwater robot remote control system according to an embodiment of the present invention includes: an underwater robot including a camera for detecting underwater environmental information, and a manipulator having an end working part, a cylinder actuator, and a joint; an adjustment device provided to be operable by a user's hand to adjust the position and direction of the end working part; Rendering and outputting virtual information including the manipulator, the virtual manipulator, and a virtual working path of the manipulator to the underwater environment information detected by the camera through a display provided therein, provided so that the user can wear it VR devices; and a control module configured to generate a driving command for the cylinder actuator based on an operation result of the adjustment device to drive the cylinder actuator.

Preferably, the camera is a 3DoF active camera, and the camera rotates based on rotation of the VR device.

The control module may include: an end working part target moving speed calculating unit configured to calculate a target moving speed of the end working part based on the target position information of the end working part received from the adjusting device; a target rotational angular velocity calculating unit for calculating a target rotational angular velocity of the joint based on the target moving velocity of the end work unit; a cylinder rod target speed calculation unit for calculating a target speed of the cylinder rod of the cylinder actuator based on the target rotational angular speed; a joint angle calculation unit for calculating the actual angle of the joint by detecting the driving condition of the manipulator driven by the cylinder rod target speed calculating unit in real time; It is preferable to include; an actual position calculation unit for calculating the actual position of the end work part based on the actual angle of the joint.

It is preferable that the actual angle of the joint is fed back to at least one of the target rotational angular velocity calculating unit and the cylinder rod target speed calculating unit.

It is preferable that the actual position of the end working part is fed back to the target moving speed calculating unit of the end working part.

The user is configured to select a driving mode of the augmented reality-based underwater robot remote control system using the control device, and the driving mode is preferably at least one of a training mode, a control mode, and an environment detection mode.

When the user selects the training mode as the driving mode, the virtual manipulator and the virtual working path of the manipulator are rendered and output on the display of the VR device on the underwater environment information detected by the camera, wherein the control module includes: Simulation control is performed with the virtual manipulator based on the target position information of the end work unit received from the adjustment device, and based on the actual angle of the joint calculated in the simulation control process and the actual position of the end work unit, the It is preferable to further include a controllability determination unit that determines in real time whether the operating point of the underwater robot is within the working range of the manipulator.

When the user selects the control mode as the driving mode, the underwater environment information detected by the camera and the manipulator are output to the display of the VR device, and the control module, the target rotational angular velocity of the joint and the actual joint a joint angle prediction unit for predicting the future angle of the joint in real time based on the angle; and a collision occurrence determination unit configured to determine in real time whether a collision has occurred within the manipulator within a preset time based on the future angle of the joint.

When the user selects the environment detection mode as the driving mode, the control module detects in real time the load amount of the motor driving the manipulator while driving the end work unit at a speed less than a preset speed, and determines the position of the end work unit and the motor. It is preferable to further include an environmental information generating unit that generates environmental information around the underwater robot based on the load amount of .

When the user selects the environment detection mode as the driving mode, the display of the VR device includes the manipulator, the virtual manipulator, and a virtual work path of the manipulator in the underwater environment information generated by the environment information generating unit. It is preferable that the information is rendered and output.

The augmented reality-based underwater robot remote control system according to an embodiment of the present invention utilizes augmented reality technology to provide integrated information to the actual work site through a VR device, thereby providing information integrated into augmented reality by one coordinator It has the advantage of being able to remotely control the underwater robot with an intuitive control device based on the

In addition, there is an effect that the coordinator can easily control the camera and the articulated robot arm due to the construction of an intuitive control interface.

Effects of the present invention are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those of ordinary skill in the art from the following description.

1 is a schematic diagram of an augmented reality-based underwater robot remote control system according to an embodiment of the present invention.
2 is a conceptual diagram for explaining generation of an image output through a display of a VR device in an augmented reality-based underwater robot remote control system according to an embodiment of the present invention.
3 is a view for explaining the control of the cylinder actuator of the underwater robot in the augmented reality-based underwater robot remote control system according to an embodiment of the present invention.
4 is a block diagram illustrating a basic control module in an augmented reality-based underwater robot remote control system according to an embodiment of the present invention.
5 is a block diagram illustrating a control algorithm in a training mode in an augmented reality-based underwater robot remote control system according to an embodiment of the present invention.
6 is a block diagram for explaining a control algorithm in a control mode in an augmented reality-based underwater robot remote control system according to an embodiment of the present invention.

A preferred embodiment according to the present invention will be described in detail with reference to the accompanying drawings, but the same or similar components are assigned the same reference numerals regardless of reference numerals, and redundant description thereof will be omitted.

In addition, in the description of the present invention, if it is determined that a detailed description of a related known technology may obscure the gist of the present invention, the detailed description thereof will be omitted. In addition, it should be noted that the accompanying drawings are only for easy understanding of the spirit of the present invention, and should not be construed as limiting the spirit of the present invention by the accompanying drawings.

Hereinafter, the configurations of the augmented reality-based underwater robot remote control system according to an embodiment of the present invention will be described with reference to FIGS. 1 to 7 .

The augmented reality-based underwater robot remote control system according to an embodiment of the present invention includes an underwater robot 100, an adjustment device 200, a VR device 300 and a control module 400 as shown in FIG. configured to do

The underwater robot 100 is configured to include a camera 110 and a manipulator 120, wherein the camera 110 performs a function of detecting underwater environmental information, and rotates the space of Yaw, Pitch, and Roll. It is preferably a stereo camera that can sense, and is a possible 3DoF (Degree of Freedom) camera, which is configured to rotate based on the rotation of the VR device 300 , which will be described later.

The manipulator 120 is configured to include an end work part 121, a cylinder actuator 122, and a joint 123, and is driven based on an operation of the adjustment device 200 to be described later, details of which will be described later. do.

The adjustment device 200 is configured to be operable by a user's hand in order to adjust the position and direction of the end operation part 121 .

The VR device 300 is formed so that a user can wear it, and the manipulator 120, the virtual manipulator, and the manipulator 120 are based on underwater environmental information detected by the camera 110 through a display provided therein. It is configured to render and output virtual information including a virtual working path.

That is, as shown in FIG. 2 , the manipulator information and the 3D environment information are rendered on the stereo image provided by the 3-DOF active camera 110 so that the real environment information and the virtual information are three-dimensionally visualized on the VR device 300 . do.

The control module 400 generates a driving command of the cylinder actuator 122 based on the operation result of the above-described adjustment device 200 to perform a function of driving the cylinder actuator, and this control module 400 may be configured separately. Or it may be provided inside the underwater robot 100.

The control element of this cylinder actuator 122 is the joint angle (q _k ) and the speed (d _k ) of the rod (122a ₎ as shown in FIG. And the equations for the speed (d _k ) of the rod 122a are the same as the following Equations (1) and (2), respectively.

<Formula (1)>

<Formula (2)>

The control module 400 of the augmented reality-based underwater robot remote control system according to an embodiment of the present invention is based on the target position information of the end work unit 121 generated through the adjustment device 200, the joint 123 ) and the target speed of the cylinder rod 122a are calculated, and the driving of the manipulator 120 is controlled based on the calculation.

To this end, the control module 400 includes a target movement speed calculation unit 410 for the end work part, a target rotation angular speed calculation unit 420, a target cylinder rod speed calculation unit 430, and a joint angle calculation unit as shown in FIG. It is configured to include a 440 and an actual position calculation unit 450 .

The end working part target moving speed calculation unit 410 has a function of calculating the target moving speed of the end working part 121 based on the target position information of the end working part 121 received from the adjusting device 200 . carry out

The target rotational angular velocity calculation unit 420 performs a function of calculating the target rotational angular velocity of the joint 123 based on the target movement speed of the end operation unit 121 calculated by the end operation unit target movement speed calculation unit 410 . do.

The cylinder rod target speed calculation unit 430 is a function of calculating the target speed of the cylinder rod 122a of the cylinder actuator 122 based on the target rotational angular speed of the joint 123 calculated by the target rotational angular speed calculation unit 420 carry out

The operation of the manipulator 120 driven by the cylinder rod target speed calculation unit 430 of the joint angle calculation unit 440 is detected in real time to calculate the actual angle of the joint 123 .

The actual position calculation unit 450 performs a function of calculating the actual position of the end work unit 121 based on the actual angle of the joint 123 calculated by the joint angle calculation unit 440 .

The actual position of the end work unit 121 calculated by the actual position calculating unit 450 is configured to be fed back to the end work unit target moving speed calculating unit 410 .

On the other hand, the augmented reality-based underwater robot remote control system according to an embodiment of the present invention has various driving modes, and may be configured so that the user selects the driving mode by using the control device 200 .

In particular, the driving mode is preferably composed of at least one of a training mode, a control mode, and an environment detection mode.

The training mode is a mode for determining whether work is possible before the actual manipulator 120 is driven, and performs a function of controlling the virtual manipulator through simulation and providing the result as 3D visualization and haptic to the user.

At this time, as shown in FIG. 5 , the control module 400 performs simulation control with the virtual manipulator based on the target position information of the end work part received from the adjustment device 200 , as shown in FIG. 5 . Based on the actual angle of the joint 123 and the actual position of the end work part 121 calculated in the simulation control process, it is determined whether the operating point of the underwater robot 100 is within the working range of the manipulator 120 in real time. It is configured to further include a controllability determination unit 460 to determine.

When the controllability determination unit 460 determines that the operating point of the underwater robot 100 is not within the working range of the manipulator 120 , a warning message indicating that the working range is out of the working range is displayed on the display of the VR device 300 . It is preferably configured to be output.

The control mode is a mode in which the actual manipulator 120 is driven by the manipulation of the control device 200 by the user, and a collision test is performed through simulation of future motion according to the control command of the robot and the result is provided to the user. is composed

In this case, if the control command by the current user's operation is continued, it is determined in real time whether a collision occurs within a preset time T and has a function of notifying the user of the possibility of a collision.

That is, when the user selects the control mode as the driving mode, the underwater environment information detected by the camera 110 and the manipulator 120 are output on the display of the VR device 300 .

At this time, the control module 400 is a joint angle prediction unit that predicts the future angle of the joint 123 in real time based on the target rotational angular velocity of the joint 123 and the actual angle of the joint 123 as shown in FIG. 6 . It is configured to further include a 470 and a collision occurrence determination unit 480 which determines in real time whether a collision has occurred in the manipulator 120 within a preset time based on the future angle of the joint 123 .

In particular, when the collision occurrence determination unit 480 determines that a collision within the manipulator 120 occurs within a preset time, the display of the VR device 300 indicates that a collision within the manipulator 120 occurs within a preset time. It is preferable to be configured to output a warning message of

The environment detection mode is a mode that can be utilized when the underwater robot 100 performs an operation in an environment in which the field of view is not properly secured, and when the detection of the environmental information by the camera 110 is impossible.

That is, when the user selects the environment detection mode, the control module 400 drives the end work unit 121 at a speed less than a preset speed while detecting the load amount of the motor driving the manipulator 120 in real time, and the end work It may be configured to further include an environmental information generating unit that generates environmental information around the underwater robot 100 based on the position of the part 121 and the load amount of the motor.

In particular, when the manipulator 120 moves in one direction and comes into contact with an obstacle such as a rock in the water, the manipulator 120 is no longer able to move in that direction. It is possible to recognize that there is an obstacle at the corresponding position, and in this state, the environmental information generating unit is based on the position of the end work unit 121 and the load amount of the motor obtained while moving the manipulator 120 along the surface of the obstacle. After identifying the shape of the obstacle, it is possible to create it.

At this time, on the display of the VR device 300, the virtual information including the manipulator, the virtual manipulator, and the virtual work path of the manipulator is rendered on the underwater environmental information generated by the environment information generating unit, so that the camera 110 is output. It is possible to acquire underwater environmental information even in a situation where it is impossible to detect underwater environmental information.

In particular, if the moving speed of the end work unit 121 in the environment detection mode is set too fast, the end work unit 121 may be damaged due to a collision with the obstacle upon reaching the obstacle, and the moving speed of the end work unit 121 If is set too slowly, the time required to detect the underwater environment is too long. Therefore, it is necessary to determine the moving speed of the end work unit 121 in consideration of the durability of the manipulator 120 including the end work unit 121. will be.

The embodiments described in this specification and the accompanying drawings are merely illustrative of some of the technical ideas included in the present invention. Therefore, since the embodiments disclosed in the present specification are for explanation rather than limitation of the technical spirit of the present invention, it is obvious that the scope of the technical spirit of the present invention is not limited by these embodiments. Modifications and specific embodiments that can be easily inferred by a person of ordinary skill in the art within the scope of the technical idea included in the specification and drawings of the present invention are included in the scope of the present invention. will have to be interpreted.

100: underwater robot
200: adjuster
300: VR device
400: control module

Claims (10)

An underwater robot including a camera for detecting underwater environmental information, and a manipulator having an end work part, a cylinder actuator, and a joint;
an adjustment device provided to be operable by a user's hand to adjust the position and direction of the end working part;
Rendering and outputting virtual information including the manipulator, the virtual manipulator, and a virtual working path of the manipulator to the underwater environment information detected by the camera through a display provided therein, provided so that the user can wear it VR devices; and
a control module for generating a driving command for the cylinder actuator based on an operation result of the adjusting device to drive the cylinder actuator;
including,
The user is configured to select a driving mode of the augmented reality-based underwater robot remote control system using the control device,
The driving mode is at least one of a training mode, a control mode, and an environment detection mode,
When the user selects the environment detection mode as the driving mode, the control module detects in real time the load amount of the motor driving the manipulator while driving the end work unit at a speed less than a preset speed, and determines the position of the end work unit and the motor. Augmented reality-based underwater robot remote control system comprising an environment information generating unit for generating environmental information around the underwater robot based on the load of
The method according to claim 1,
The camera is a 3DoF active camera, and an augmented reality-based underwater robot remote control system in which the camera rotates based on the rotation of the VR device.
The method according to claim 1, wherein the control module,
a target moving speed calculating unit for the end working part for calculating a target moving speed of the end working part based on the target position information of the end working part received from the adjusting device;
a target rotational angular velocity calculating unit for calculating a target rotational angular velocity of the joint based on the target moving velocity of the end work unit;
a cylinder rod target speed calculation unit for calculating a target speed of the cylinder rod of the cylinder actuator based on the target rotational angular speed;
a joint angle calculation unit for calculating the actual angle of the joint by detecting the driving condition of the manipulator driven by the cylinder rod target speed calculating unit in real time;
an actual position calculation unit for calculating the actual position of the end work part based on the actual angle of the joint;
Augmented reality-based underwater robot remote control system further comprising a.
4. The method of claim 3,
The actual angle of the joint is fed back to at least one of the target rotational angular velocity calculating unit and the cylinder rod target speed calculating unit, an augmented reality-based underwater robot remote control system.
4. The method of claim 3,
The actual position of the end work unit is an augmented reality-based underwater robot remote control system that is fed back to the end work unit target movement speed calculation unit.
delete The method according to claim 1,
When the user selects the training mode as the driving mode, the virtual manipulator and the virtual working path of the manipulator are rendered and output on the display of the VR device on the underwater environment information detected by the camera,
The control module performs simulation control with the virtual manipulator based on the target position information of the end work part received from the adjustment device, and the actual angle of the joint calculated in the simulation control process and the actual position of the end work part Augmented reality-based underwater robot remote control system further comprising a; a controllability determination unit for determining in real time whether the action point of the underwater robot is within the working range of the manipulator based on the
The method according to claim 1,
When the user selects the control mode as the driving mode, the underwater environment information detected by the camera and the manipulator are output to the display of the VR device,
The control module is
a joint angle prediction unit for predicting a future angle of the joint in real time based on the target rotational angular velocity of the joint and the actual angle of the joint; and
a collision occurrence determination unit configured to determine in real time whether a collision has occurred in the manipulator within a preset time based on the future angle of the joint;
Augmented reality-based underwater robot remote control system further comprising a.
delete The method according to claim 1,
When the user selects the environment detection mode as the driving mode, the display of the VR device includes the manipulator, the virtual manipulator, and a virtual work path of the manipulator in the underwater environment information generated by the environment information generating unit. An augmented reality-based underwater robot remote control system that is output by rendering information.

Images