KR102277393B1 - Apparatus and method for training - Google Patents

Abstract

A training device is provided. The training device may include: an acquisition unit configured to acquire an actual real-time image captured by an actual robot; and a simulation unit for generating a disaster environment in which virtual disaster information is added to the real-time image.

Description

TRAINING DEVICES AND METHOD FOR TRAINING

The present invention relates to an apparatus and method for training a pilot to operate a disaster response robot.

A disaster response robot may be used for rescue and search in a complex disaster environment such as fire, agricultural smoke, collapse, etc., where it is difficult for firefighters to be directly deployed.

In an environment where disaster response robots are deployed, professional robot pilots control the robot using a remote controller based on limited video and audio information.

In order to accurately control the robot in the field, pilots are performing repetitive robot control training in similar disaster environments or virtual reality-based training systems. Therefore, a problem of cost consumed in simulating a disaster environment for repetitive training and a problem of similarity to reality arise in the process of simulating the movement of a real robot in the training system.

Korean Patent Publication No. 1875170 discloses a tangible fire drill simulation system.

Korean Patent Publication No. 1875170

An object of the present invention is to provide a training apparatus and a training method that provide a robot control environment similar to an actual disaster situation.

The training apparatus of the present invention includes: an acquisition unit for acquiring an actual real-time image taken by an actual robot; and a simulation unit for generating a disaster environment in which virtual disaster information is added to the real-time image.

The training method of the present invention includes an acquisition step of acquiring a real-time image captured by the robot in real time, inertial measurement information and mileage information of the robot, and environment information in which the robot is disposed; a calculation step of calculating position information and direction information of the robot using at least one of the inertia measurement information and the travel distance information; creating a virtual space using the environment information, and matching the robot on the virtual space using the position information of the robot; extracting a matching area matching the real-time image in the virtual space using the position information and the direction information of the robot; determining disaster information that needs to be shaped in the matching area, and shaping the disaster information in the matching area; generating a disaster environment in which virtual disaster information is added to the real-time image by integrating the real-time image and the matching area; It may include; providing the disaster environment to the control unit of the robot.

The training apparatus and training method of the present invention can implement augmented reality that tracks the actual accident scene. Augmented reality allows pilots to conduct pilot training of robots in training situations similar to real-life accident sites.

In an actual disaster situation, the pilot can control the robot by understanding the surrounding environment only with the real-time video (including voice) recorded by the robot. In order to simulate a real situation, the training device of the present invention may project and shape an actual disaster situation on a real-time image captured by the robot.

According to the present invention, there can be provided a disaster robot training system that acquires the motion or movement of the robot through a motor control command of the real robot and camera image information, and creates a disaster environment such as fire and agricultural activities based on augmented reality.

According to the training apparatus and method of the present invention, the cost problem spent on simulating a disaster environment can be reduced. In addition, according to the present invention, since augmented reality is implemented using the movement of the real robot, the problem of similarity in reality for simulating the actual movement of the robot can be naturally solved.

According to the present invention, since training is possible for various topographical features, the training effect of the disaster response robot can be improved, and the usability of the robot can be improved. According to the present invention, various disaster response situations can be integrated into augmented reality in the real-time camera image of the robot affected by the real-time robot state and environment of the disaster response robot. Since the training of the pilot is carried out using an image in which an actual camera image and a disaster response situation are integrated into augmented reality, a training system that simulates a disaster situation at a low cost can be provided.

1 is a schematic diagram illustrating a training system of the present invention.
2 is a schematic diagram showing a training device of the present invention.
3 is a schematic diagram showing the operation of the training device of the present invention.
4 is a flowchart illustrating a training method of the present invention.
5 is a diagram illustrating a computing device according to an embodiment of the present invention.

Hereinafter, with reference to the accompanying drawings, the embodiments of the present invention will be described in detail so that those of ordinary skill in the art to which the present invention pertains can easily implement them. However, the present invention may be embodied in many different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

In the present specification, duplicate descriptions of the same components will be omitted.

Also, in this specification, when it is mentioned that a certain element is 'connected' or 'connected' to another element, it may be directly connected or connected to the other element, but another element in the middle. It should be understood that there may be On the other hand, in this specification, when it is mentioned that a certain element is 'directly connected' or 'directly connected' to another element, it should be understood that the other element does not exist in the middle.

In addition, the terms used herein are used only to describe specific embodiments, and are not intended to limit the present invention.

Also, in this specification, the singular expression may include the plural expression unless the context clearly dictates otherwise. Also, in this specification, terms such as 'include' or 'have' are only intended to designate that the features, numbers, steps, operations, components, parts, or combinations thereof described in the specification exist, and one or more It should be understood that the existence or addition of other features, numbers, steps, operations, components, parts or combinations thereof is not precluded in advance.

Also in this specification, the term 'and/or' includes a combination of a plurality of listed items or any of a plurality of listed items. In this specification, 'A or B' may include 'A', 'B', or 'both A and B'.

Also, in this specification, detailed descriptions of well-known functions and configurations that may obscure the gist of the present invention will be omitted.

1 is a schematic diagram illustrating a training system of the present invention.

The training system shown in FIG. 1 may include a robot 10 , a control unit 30 , and a training server 100 . The training server 100 may exist independently of the robot 10 and the steering unit 30 . Alternatively, the training server 100 may be integrally formed with the robot 10 or integrally formed with the control unit 30 .

The robot 10 may search a set area, rescue people, or suppress a disaster when a disaster situation such as fire, agricultural smoke, gas pollution, or flooding occurs. In particular, the robot 10 may be put into a place difficult for firefighters to enter.

The control unit 30 may remotely control the robot 10 . The control unit 30 may include a communication module communicating with the robot 10 , a handle operated by the pilot, and a display unit viewed by the pilot.

In a situation in which the robot 10 or the disaster environment is not visible to the naked eye, the pilot may use the display unit to determine the position of the robot 10 or recognize the disaster environment, and to control the robot 10 according to the recognition content.

The pilot may perform the robot 10 operation training in a state in which the robot 10 is put into the terrain where disaster situations such as fire, flooding, agricultural smoke, gas pollution, etc. do not occur. However, according to the training that does not reflect the disaster situation, it is difficult to respond to changes in the surroundings that change every moment in the actual disaster situation.

In order to improve the training effect, a real disaster situation may be realized by lighting a real landmark, and the training may proceed. In this case, the training effect is very high, but the training is limited to one-off training due to the loss of the features. Therefore, repeated training is impossible.

The training server 100 may provide the pilot with a training environment similar to an actual disaster situation by using augmented reality.

2 is a schematic diagram showing a training device of the present invention.

The training apparatus shown in FIG. 2 may include the training server 100 of FIG. 1 .

The training apparatus of the present invention may include an acquisition unit 110 and a simulation unit 130 .

The acquisition unit 110 may acquire a real-time image captured by the actual robot 10 .

The simulation unit 130 may create a disaster environment in which virtual disaster information is added to a real-time image.

For example, in the case of fire-related training, if an actual flame image or a smoke image is overlaid and displayed on a real-time image, it may be similar to an image of an actual fire scene. An image manipulated similarly to an image of an actual fire scene may correspond to a disaster environment. Alternatively, a state in which a virtual disaster is given to a geographical feature in which a disaster has not occurred may correspond to a disaster environment. The disaster environment needs to be perceived visually or acoustically by the pilot conducting the training. Accordingly, the disaster environment may include image data in which flames, water, smoke, etc. that the pilot can see with his/her eyes are visually shaped. Alternatively, the disaster environment may include voice (acoustic) data in which the sound of burning, water, and the like, which can be heard by the pilot, is aurally and aurally shaped. The disaster environment may include both image data and voice data.

The acquisition unit 110 may include a first acquirer 111 , a second acquirer 112 , and a third acquirer 113 .

The first acquisition unit 111 may acquire an actual real-time image captured by the actual robot 10 .

The second acquisition unit 112 may acquire inertia measurement information and travel distance information of the robot 10 . The second acquirer 112 may calculate the current position and direction of the robot 10 by using at least one of inertia measurement information and travel distance information.

The second acquisition unit 112 ultimately aims to acquire the current position (latitude position, longitude position, height above sea level, etc.) of the robot 10 and the optical axis direction of the camera mounted on the robot 10 . If the robot 10 can determine its own location using a GPS module or the like, the second acquisition unit 112 may directly acquire location information from the robot 10 .

However, when the topographical features of the disaster environment are indoors, underground, tunnels, etc., the robot 10 cannot directly determine its own location due to the non-operation of the GPS module. The robot 10 may generate inertia measurement information and mileage information in preparation for various situations in which it is difficult to determine its own position. Corresponding information may be provided to the control unit 30 , and may be provided to the second acquisition unit 112 in a training situation. The second acquirer 112 may calculate the current position of the robot 10 by using inertia measurement information and travel distance information that can determine speed, acceleration, angular velocity, direction, gravity, and the like.

The third acquisition unit 113 may acquire environment information in which the robot 10 is disposed. The environment information may include various types of information about the terrain features to which the robot 10 is put. For example, the environmental information may include the height, length, width, material, indoor structure, indoor shape, types of various facilities disposed in the terrain feature, the material of the facility, the location of the facility, and the like of the terrain feature.

A first generation unit 131 , a second generation unit 132 , and a third generation unit 133 may be provided in the simulation unit 130 .

The first generator 131 may generate a virtual space by using the environment information.

The second generator 132 may generate disaster information according to the virtual space. The second generation unit 132 may virtualize the entire feature on which the robot 10 is inserted. In order to reduce the load of the second generator 132 , only a portion of the feature may be shaped into a virtual space. In this case, the second generator 132 may generate a virtual space according to at least the current position and direction of the robot 10 . The disaster information may include visually visible image data such as flame, water, and smoke, and audio data such as audible burning sound and water sound. It may be difficult to shape the disaster information as it is in the environment information acquired by the third acquisition unit 113 . In addition, some environmental information may become an obstacle in adding disaster information to real-time images in the future. Therefore, it is preferable to separately create a virtual space in which graphic processing is easy. The second generator 132 may generate disaster information in a virtual space according to the current position and direction of the robot 10 .

The third generator 133 may create an augmented reality-based disaster environment in which a real-time image and disaster information are matched. The third generator 133 may perform a synchronization process of matching the real-time image with the disaster information using the current position and direction of the robot 10 and integrating the matched real-time image with the disaster information. When the synchronization process is completed, virtual disaster information such as a flame image may be overlaid and displayed on an actual real-time image. A real-time image displayed overlaid with disaster information may correspond to a disaster environment.

The robot 10 may be put into a feature in which a disaster does not actually occur, and may take a real-time image of the feature in which a disaster does not occur and provide it to the control unit 30 . The real-time image may be processed by the training device of the present invention and turned into a disaster environment. A disaster environment in which a real-time image is reversed may be provided to the control unit 30 and displayed through the display unit. The pilot watches the image in which the disaster exists through the display unit, and can control the robot 10 according to the situation displayed on the image.

The robot 10 may correspond to the control unit 30 1:1. Accordingly, when the control unit 30 for controlling the robot 10 is provided, the robot 10 may basically transmit a real-time image to the control unit 30 .

In order to create a disaster environment necessary for training, a real-time image of the robot 10 must be acquired by the acquisition unit 110 . When the training device is provided separately from the control unit 30 , the subject that can provide the real-time image to the acquisition unit 110 is the robot 10 or the control unit 30 that receives the real-time image from the robot 10 . can be Considering the robot 10 that can be put into a place where transmission and reception of radio waves is difficult, whether the real-time image transmitted from the robot 10 normally arrives at the control unit 30 may also be an important training point. In order to simulate the communication state of the real environment as it is, the acquisition unit 110 may communicate with the control unit 30 .

Once the real-time image ① transmitted from the robot 10 is received by the communication module provided in the control unit 30, the acquisition unit 110 can intercept the real-time image ① output from the communication module toward the display unit (interrupt) . The acquisition unit 110 that intercepts the real-time image ① transmitted from the robot 10 to the control unit 30 may transmit the real-time image ① to the simulation unit 130 .

The simulation unit 130 may create a disaster environment ② by adding disaster information to the real-time image ①. The simulation unit 130 may transmit the disaster environment ② to the control unit 30 instead of the real-time image ① to be transmitted to the control unit 30 . The disaster environment ② transmitted to the control unit 30 instead of the real-time image ① is displayed through the display unit, and the pilot can watch an image in which disaster information such as flames are displayed similarly to the actual disaster situation.

If the communication state is not considered during training, the acquisition unit 110 may directly communicate with the robot 10 and receive a real-time image from the robot 10 . In this case, the communication channel for real-time images formed between the robot 10 and the control unit 30 may be deactivated.

According to the training apparatus of the present invention, the real-time image generated by the robot 10 is not directly transmitted to the control unit 30 , but passes through the acquisition unit 110 and the simulation unit 130 . Even if the training device is integrally formed with the robot 10 or the control unit 30, at least the simulation unit 130 is additionally passed through. The time the real-time image is displayed on the display unit may be delayed by the amount of time delayed while passing through the simulation unit 130 . In order to provide a realistic training environment, it is desirable that the delay time be minimized. In order to minimize the delay time, the faster the processing speed of the simulation unit 130, the better.

Hereinafter, a method for the simulation unit 130 to realistically implement virtual disaster information will be described.

The acquisition unit 110 may acquire position information of the robot 10 . In this case, the training device may be provided with a simulation unit 135 that detects a feature existing in the location information and simulates a process of spreading a disaster to the feature.

In an actual disaster situation, flames are basically propagated according to the laws of physics. Flames generated at the first location do not propagate to the second location without any justification. The simulation unit 135 may be used to simulate a disaster propagation process that follows a real disaster situation. The simulation unit 135 may use the location information of the robot 10 to determine the topographic features around the robot 10 . The simulation unit 135 may be provided with a model function in which a process of propagating flame, water, smoke, etc. is mathematically modeled. The simulation unit 135 may grasp information such as a structure, material, and facility of a feature together, input this information into a model function, and simulate a disaster propagation process using a result output from the model function. The simulation unit 135 may shape a disaster propagation process over time, and information in which the disaster propagation process is shaped may correspond to disaster settlement.

The simulation unit 130 may receive the simulation result of the simulation unit 135 and use it as disaster information. The disaster information generated by inputting the simulation result of the simulation unit 135 may have a propagation process and appearance that follow the actual disaster. The simulation unit 130 may extract and use the image data included in the disaster information from the pre-stored actual flame image (including video), water image (including video), and smoke image (including video).

The operation of the simulation unit 130 will be described in order as follows.

The simulation unit 130 may create a virtual space in which the robot 10 follows the inputted feature.

The simulation unit 130 may simulate in real time a process of propagation of a disaster in a virtual space.

The simulation unit 130 may shape a disaster on a virtual space in real time according to the simulation result.

3 is a schematic diagram showing the operation of the training device of the present invention.

The simulation unit 130 may extract the matching area m matching the real-time image ① in the virtual space i in real time. In order to extract the matching area m, the acquisition unit 110 may acquire position information and direction information of the robot 10 . When inertia measurement information and travel distance information are obtained from the robot 10 , the acquisition unit 110 may analyze the inertia measurement information and the like to calculate position information and direction information of the robot 10 . The simulation unit 130 may extract the matching area m matching the real-time image ① by analyzing the location information and direction information based on the feature.

On the other hand, it is better to exclude the method of extracting the matching area through image analysis of the real-time image ①. Because, it may be difficult to analyze the real-time image on which disaster information is projected. In addition, a photo of a specific geographical feature may be hung on the wall of the building in which the robot 10 is inserted. According to the image analysis technique, when the robot 10 takes a picture, analysis and simulation of a specific geographical feature in the picture are performed erratically. As a result, extraction of the matching area is very difficult.

According to the present invention, the relative position between the robot 10 and the feature can be determined by analyzing the location information and direction information of the robot 10 and the information of the feature. Using the identified relative position, the matching area m matching the real-time image ① can be accurately extracted.

The simulation unit 130 may shape a disaster by targeting the entire terrain in which the robot 10 is inserted. Alternatively, in order to improve the processing speed, the simulation unit 130 may shape the disaster only for a part of the feature. In this case, it is advantageous to set the portion to be shaped based on the real-time image.

For example, the simulation unit 130 may shape a disaster f (image, voice, etc.) such as a flame on a simulation area s larger than the current matching area m in real time. In this case, all of the current matching area m may be included in the simulation area s.

The simulation unit 130 may reflect the disaster f being shaped on the simulation area according to a change in the position or direction of the robot 10 in the real-time image ①. The object to be photographed by the camera of the robot 10 may change from moment to moment by the control of the pilot. By the pilot's control, the robot 10 takes a picture of an object while drawing a continuous movement line. Therefore, when the simulation of the surrounding area larger than the image currently being photographed by the robot 10 is performed, disaster f such as a flame image can be provided seamlessly to the display unit that the pilot sees.

The simulation unit 130 may integrate the virtual disaster f into the real-time image ① to create a disaster environment ② used for training, and provide it to the control unit 30 .

4 is a flowchart illustrating a training method of the present invention.

The training method of FIG. 4 may be performed by the training apparatus shown in FIG. 2 .

The training method of the present invention includes an acquisition step (S 510), a calculation step (S 520), a matching step (S 530), an extraction step (S 540), a shaping step (S 550), a generating step (S 560), a providing step (S 570) may be included.

The obtaining step (S 510) may be a step of obtaining a real-time image captured by the robot 10 in real time, inertial measurement information and mileage information of the robot 10, and environment information in which the robot 10 is disposed. . The acquiring step S510 may be performed by the second acquiring unit 112 and the third acquiring unit 113 of the acquiring unit 110 .

The calculating step ( S520 ) may be a step of calculating location information and direction information of the robot 10 using at least one of inertia measurement information and travel distance information. The calculation step ( S520 ) may be performed by the second acquirer 112 .

The matching step ( S530 ) may be a step of creating a virtual space using environment information and matching the robot 10 on the virtual space using location information of the robot 10 . The matching step ( S540 ) may be performed by the first generator 131 of the simulation unit 130 .

The extraction step ( S540 ) may be a step of extracting a matching region matching a real-time image in a virtual space using the location information and direction information of the robot 10 . The extraction step ( S540 ) may be performed by the first generator 131 of the simulation unit 130 .

In the shaping step (S550), disaster information that needs to be shaped in the matching area may be determined, and the disaster information may be shaped in the matching area. The shaping step ( S550 ) may be performed by the second generation unit 132 and the simulation unit 135 of the simulation unit 130 .

The generating step ( S560 ) may be a step of generating a disaster environment in which virtual disaster information is added to the real-time image by integrating the real-time image and the matching area. The generation step ( S560 ) may be performed by the third generation unit 133 of the simulation unit 130 .

The providing step ( S570 ) may be a step of providing a disaster environment to the control unit 30 of the robot 10 . The providing step ( S570 ) may be performed by the third generating unit 133 .

5 is a diagram illustrating a computing device according to an embodiment of the present invention. The computing device TN100 of FIG. 5 may be a device (eg, a training device, etc.) described herein.

In the embodiment of FIG. 5 , the computing device TN100 may include at least one processor TN110 , a transceiver device TN120 , and a memory TN130 . In addition, the computing device TN100 may further include a storage device TN140 , an input interface device TN150 , an output interface device TN160 , and the like. Components included in the computing device TN100 may be connected by a bus TN170 to communicate with each other.

The processor TN110 may execute a program command stored in at least one of the memory TN130 and the storage device TN140. The processor TN110 may mean a central processing unit (CPU), a graphics processing unit (GPU), or a dedicated processor on which methods according to an embodiment of the present invention are performed. The processor TN110 may be configured to implement procedures, functions, methods, and the like described in connection with an embodiment of the present invention. The processor TN110 may control each component of the computing device TN100 .

Each of the memory TN130 and the storage device TN140 may store various information related to the operation of the processor TN110 . Each of the memory TN130 and the storage device TN140 may be configured as at least one of a volatile storage medium and a non-volatile storage medium. For example, the memory TN130 may include at least one of a read only memory (ROM) and a random access memory (RAM).

The transceiver TN120 may transmit or receive a wired signal or a wireless signal. The transceiver TN120 may be connected to a network to perform communication.

On the other hand, the embodiment of the present invention is not implemented only through the apparatus and/or method described so far, and a program for realizing a function corresponding to the configuration of the embodiment of the present invention or a recording medium in which the program is recorded may be implemented. And, such an implementation can be easily implemented by those skilled in the art from the description of the above-described embodiments.

Although the embodiment of the present invention has been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements by those skilled in the art using the basic concept of the present invention as defined in the following claims are also presented. It belongs to the scope of the invention.

10...Robot 30...Control Unit
100...training server 110...acquisition unit
111...First Acquisition Unit 112...Second Acquisition Unit
113...3rd Acquisition Unit 130...Simulation Unit
131...first generation unit 132...second generation unit
133...the third generation part 135...the simulation part

Claims (8)

an acquisition unit for acquiring an actual real-time image taken by an actual robot;
Includes; a simulation unit for generating a disaster environment in which virtual disaster information is added to the real-time image;
When a control unit for controlling the robot is provided, the robot transmits the real-time image to the control unit,
the acquisition unit intercepts the real-time image transmitted from the robot to the control unit and transmits it to the simulation unit;
and the simulation unit transmits the disaster environment to the control unit instead of the real-time image to be transmitted to the control unit.
delete delete delete an acquisition unit for acquiring an actual real-time image taken by an actual robot;
Includes; a simulation unit for generating a disaster environment in which virtual disaster information is added to the real-time image;
The simulation unit creates a virtual space in which the robot follows the inputted feature,
The simulation unit simulates in real time the process of propagation of a disaster in the virtual space,
The simulation unit shapes the disaster in real time on the virtual space according to the simulation result,
The simulation unit extracts a matching area matching the real-time image in the virtual space in real time,
The simulation unit is a training device for creating the disaster environment by overlaying the disaster shaped in real time on the matching area on the real-time image.
6. The method of claim 5,
The acquisition unit acquires the position information and direction information of the robot,
The simulation unit analyzes the location information and the direction information based on the feature, and extracts the matching area matching the real-time image.
6. The method of claim 5,
The simulation unit shapes the disaster in real time on a simulation area larger than the current matching area,
The simulation area includes all of the current matching area,
The simulation unit is a training device for reflecting the disaster being shaped on the simulation area in the real-time image according to a change in position or direction of the robot.
A training method performed by a training device, comprising:
generating a virtual space in which the robot follows the input feature;
simulating in real time a process of propagating a disaster in the virtual space;
shaping the disaster on the virtual space in real time according to the simulation result;
extracting a matching area matching the real real-time image captured by the robot in real time in the virtual space;
creating a disaster environment by overlaying a disaster shaped in real time on the matching area on the real-time image;
A training method comprising

Images