KR20180135761A - Methods and apparatus for military training based on mixed reality - Google Patents

Abstract

The present invention relates to a method and apparatus for a mixed reality-based military drill. According to the present invention, in an actual simulated military drill, an optimal virtual entity is created and arranged by analyzing a real image of a scene that is input in real time from a camera worn by a user, information on life and death, location, and others is synchronized with other user by using SLAM, etc., and a virtual image is generated by controlling a state and motion of the virtual entity based on difficulty of the drill and identified sensor information. A three-dimensional image obtained by synthesizing the virtual image generated according to the present invention on the real image is outputted to the user′s HMD in real time, thereby enhancing a sense of reality of the military drill through on-site training. In addition, it is possible to avoid costs and inconvenience of preparing virtual entities such as enemy, weapons of enemy, military vehicles, covers, shelters, obstacles, and others in the real environment, thereby performing a simulated military drill in efficient and flexible situations.

Description

≪ Desc / Clms Page number 1 > METHODS AND APPARATUS FOR MILITARY TRAINING BASED ON MIXED REALITY < RTI ID =

The present invention relates to a mixed reality based military training method and apparatus, and more particularly, to a mixed reality based military training method and apparatus, And more particularly, to a hybrid reality-based military training method and apparatus capable of performing simulated military training in a more efficient and flexible situation by generating a virtual image for controlling the state and movement of virtual presence in real time based on the recognized sensor information.

Training methods using virtual reality are widely used in simulated military training. The HMD (Head Mounted Display) is mainly used in the indoor environment. The visual field of the user is a virtual environment or a virtual image is added to a real image stored in the server in advance. These training methods in the indoor are effective in terms of time and cost, but there is a limit to obtaining the effect of training in actual outdoor military training field. That is, the reality of the actual terrain is greatly reduced, and a large number of users are constrained to perform operations while actually moving. Outdoors, there is a way to wear MILES equipment and train with mock weapons. However, the outdoor military training scene is a cost and a hassle to prepare a virtual existence, that is, an enemy, a military vehicle, a cover, a cover, and an obstacle all at once.

Therefore, it is possible to conduct simulation while observing actual terrain at actual training site of outdoor, but by analyzing the topographical information of the scene and appropriately arranging virtual existence, it is possible to perform simulation training of more efficient and flexible situation. Although these mixed-image-based application technologies are known, many military operations are performed on a team-by-team basis. Therefore, during training, team members share life and death information and location information More realistic training can be performed. Therefore, in order to share such team information, it is possible to use sharing and synchronization methods of position information. For example, when using SLAM (simultaneous localization and mapping) method used in robotics, etc., simultaneous position estimation and mapping This is possible.

KR 10-1653735.

 Moon, Hyunsoo et al., "Simultaneous Position Estimation and Mapping of a Robot Robot", The Korea Institute of Intelligent Systems, 2011, Vol. 21, No. 3, pp. 296-301.

Disclosure of Invention Technical Problem [8] Accordingly, the present invention has been made keeping in mind the above problems occurring in the prior art, and it is an object of the present invention to provide a method and apparatus for analyzing real- A virtual image in which the state and motion of a virtual existence is controlled in real time based on the sensor information is generated and the generated virtual image is synthesized with the input real image and output to the user's HMD in real time, And to provide a mixed reality-based military training method and apparatus capable of simulated military training.

According to an aspect of the present invention, there is provided a mixed reality reality military training apparatus for use in an actual training field, comprising: a team information synchronization unit for synchronizing team information; A sensor information recognition unit for recognizing information input from a simulated weapon used by a user or a sensor attached to the user's body; A virtual presence creator for analyzing a 3D real image of the training field input in real time and arranging a virtual presence based on the result of the analysis; And a virtual image generation unit for generating a 3D virtual image based on at least one of the synchronized team information, the recognized sensor information, and the generated virtual presence.

The team information may include at least one of life information and location information of the user and other users.

The team information synchronization unit may perform simultaneous position estimation and map creation using the SLAM.

The virtual presence may be at least one of virtual concealment, cover, enemy force, obstacle, weapon, and vehicle generated according to a predetermined training difficulty based on a result of analyzing the real image.

The virtual image generation unit may control the status or movement of the virtual presence in real time based on a predetermined training difficulty, the synchronized team information, and the recognized sensor information.

The real image may be input from a camera worn by the user during training.

The generated virtual image may be synthesized with the real image and displayed through the user's HMD.

According to another aspect of the present invention, there is provided a mixed reality based military training method for use in an actual training field, comprising: synchronizing team information; Recognizing information input from a simulated weapon used by a user or a sensor attached to the user's body; Analyzing a 3D real-life image of the training field input in real time, and arranging a virtual entity based on the analyzed result; And generating a 3D virtual image based on at least one of the synchronized team information, the recognized sensor information, and the generated virtual presence.

The team information may include at least one of life information and location information of the user and other users.

The step of synchronizing the team information can perform simultaneous position estimation and mapping using SLAM.

The virtual presence may be at least one of virtual concealment, cover, enemy force, obstacle, weapon, and vehicle generated according to a predetermined training difficulty based on a result of analyzing the real image.

The step of generating the virtual image may include controlling in real time the state or movement of the virtual presence based on the predetermined training difficulty, the synchronized team information, and the recognized sensor information.

The mixed reality-based military training method may further include displaying the generated virtual image on the real image through the user's HMD.

According to still another preferred embodiment, there is provided a computer-readable recording medium on which a program for executing the above-described respective methods is recorded.

As described above, according to the present invention, in an actual simulated military training field, an actual virtual image of a scene input from a user in real time is analyzed to generate an optimal virtual presence, and SLAM or the like is used Synchronizes life and death information and location information with other users, generates a virtual image by controlling the state and motion of virtual existence in real time based on the training difficulty and the recognized sensor information, and outputs the generated virtual image to the real image By outputting the synthesized 3D image to the user's HMD in real time, it enhances the sense of reality by training in the field, and it prepares all the virtual existence such as enemy army, enemy army, military vehicle, cover, The cost and hassle to be done are eliminated, so that more efficient and flexible mock military training is possible. There is an effect.

1 is a view showing a real image, a virtual image, and a composite image according to an embodiment of the present invention.
FIG. 2 is a diagram illustrating equipment for mixed reality based military training according to an embodiment of the present invention.
3 is a block diagram illustrating a configuration of a mixed reality based military training apparatus according to an embodiment of the present invention.
4 is a view showing a mixed reality based military training system according to another embodiment of the present invention.
5 is a flowchart illustrating a specific example of a mixed reality based military training method according to another embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily carry out the present invention. In the following description, only parts necessary for understanding the operation according to the embodiment of the present invention are shown and described, and the other parts of the drawings and descriptions are omitted so as not to obscure the gist of the present invention. The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein.

In addition, terms and words used in the following description and claims should not be construed to be limited to ordinary or dictionary meanings, but are to be construed in a manner consistent with the technical idea of the present invention As well as the concept.

Throughout the specification, when a part is referred to as being "connected" to another part, it is not limited to the case where it is "directly connected," but is "electrically connected" Or wirelessly or wirelessly over a wired / wireless network. Also, when a part is referred to as "including " an element, it does not exclude other elements unless specifically stated otherwise.

For simplicity of explanation, one or more methods are shown and described herein as a series of steps, for example in the form of a flowchart or a flowchart, but the present invention is not limited by the order of the steps, As it can be done in a different order than that shown and described herein or concurrently with other steps. Furthermore, not all illustrated steps may have to be implemented in accordance with the present invention.

1 is a view showing a real image, a virtual image, and a composite image according to an embodiment of the present invention.

Referring to FIG. 1 (a), it is a real scene image input from a 3D camera worn by a user in an actual simulated military training site, and is a real field scene displayed at the user's view angle. Since the 3D camera needs to photograph a real image of the front including the user's view, it is preferably, but not necessarily, attached to the user's HMD or smart glass.

Referring to FIG. 1 (b), a 3D virtual image added to a real image includes a virtual image that is not present in a real image, that is, a virtual object such as an enemy, enemy weapon, military vehicle, concealed object, Objects can be included.

The virtual presence can be effectively arranged by recognizing and analyzing the position and size of the terrain or obstacle displayed on the real image. For example, it can be configured to have various effects such as displaying an enemy hiding behind a real object such as a hill or a tree, disposing obstacles that are not included in the real image, or displaying a military vehicle from afar. This eliminates the cost and hassle of preparing a virtual presence in a real environment, thus creating a more flexible training environment. In addition, it is also possible to set a predetermined difficulty level to adjust the speed or the number of moving enemy vehicles or vehicles.

Referring to FIG. 1 (c), a 3D image synthesized with the virtual image of (a) and the virtual image of (b) is an image in which a virtual presence is added to an actual front scene. By displaying the synthesized image on the user's HMD, the user can perform the training with the image added with the virtual presence in the actual environment.

FIG. 2 is a diagram illustrating equipment for mixed reality based military training according to an embodiment of the present invention.

Referring to FIG. 2, a mixed reality-based military training apparatus according to an embodiment of the present invention includes a military training apparatus 10, a 3D camera 11, an HMD 21, a gesture sensor 35, ), And the like.

The military training device 10 is preferably constructed in the form of a wearable or shoulderable backpack or bag that can be worn on the body to enable dynamic military operations training such as running or collapsing, .

The 3D camera 11 is preferably attached to a user's HMD or a smart glass because it needs to photograph a real image in front of the user, and it is more preferable that the 3D camera 11 is capable of photographing a wider range than a human's view.

The HMD 21 is a face wearable display device which is often used in a virtual reality, and matches a real image and a virtual image in a three-dimensional virtual space and transmits the same to a user. A stereoscopic display technique is used to display stereoscopic images by presenting different image information to the right and left eyes of a person.

The gesture sensor 35 is a sensor attached to the user's body, and transmits the attitude information of the user in real time, and may include one or more such as an arm or a leg.

The simulated weapon 37 may be a weapon such as a gun or a grenade, and is equipped with a sensor, so that it can produce the same effect as a real weapon. For example, if you trigger a fictitious enemy, you can summarize the information received from the sensor and make the fictitious existence disappear from the virtual image if you hit a virtual enemy. In addition, a predetermined vibration device may be provided in the simulated weapon 37 so that the user can recognize the vibration effect such as triggering, thereby further enhancing the sense of reality.

3 is a block diagram illustrating a configuration of a mixed reality based military training apparatus according to an embodiment of the present invention.

3, a military training apparatus 10 according to the present invention includes a virtual presence generating unit 13, a sensor information recognizing unit 15, a team information synchronizing unit 17, and a virtual image generating unit 19, And may further include an image synthesizing unit 23. The image synthesizing unit 23 may be a microcomputer.

The virtual presence generation unit 13 can generate a virtual presence by analyzing the 3D real-life image of the training scene input from the camera 11 in real time. The virtual presence generation unit 12 recognizes and analyzes the position and size of the terrain or obstacles displayed on the real image, analyzes the virtual concealment, cover, enemy, obstacle, weapon, vehicle, Can be deployed. For example, it can be configured to have various effects such as displaying an enemy hiding behind a real object such as a hill or a tree, disposing obstacles that are not included in the real image, or displaying a military vehicle from afar.

The sensor information recognizing unit 15 can receive the information inputted from the simulated weapon 37 used by the user or the sensor 35 attached to the user's body to recognize the posture of the user or the state of the simulated weapon. The recognition result of the sensor information recognition unit 15 may be used for control of virtual presence in the form of a vector or coordinate, or may be used for a virtual image screen configuration.

The team information synchronization unit 17 shares life and death information and location information with another user who is a team member who performs the training.

The team information synchronization unit 17 may be implemented using a SLAM system used in robotics or the like. The implementation of SLAM is very diverse, such as using EKF (Extended Kalman Filter) or SURF (Speeded Up Robust Features) algorithm. In order to recognize the surrounding environment and to measure distance information, ultrasonic sensor, laser sensor, And can form strategic formation, collision avoidance, obstacle avoidance, location estimation, and map creation through signal transmission between users. According to the synchronized team information, an efficient operation can be performed. For example, it is possible to develop situations such that a virtual enemy group displayed in front of one user is removed by another user.

The virtual image generation unit 19 generates a virtual image based on the team information synchronized by the team information synchronization unit 17, the virtual presence generated by the virtual presence generation unit 13, the information recognized by the sensor information recognition unit 15, A 3D virtual image can be generated based on the difficulty level and the like. In addition, the virtual image generating unit 19 can control the state, motion, and number of the virtual presence in real time based on the mentioned information.

The image synthesizing unit 23 synthesizes the generated virtual image with the real image, and the synthesized 3D image can be output through the HMD 21 of the user.

4 is a view showing a mixed reality based military training system according to another embodiment of the present invention.

Referring to FIG. 4, the mixed reality-based military training apparatus 10 according to the present invention can be worn by a plurality of users at the same time. In this way, when a military operation training is performed by a team, Lt; / RTI >

Information related to the presence of the terrain that is visible in front and information related to the virtual presence are stored in the military training apparatus 10 carried by each user and common information such as position information and life and death information between users is stored in the server 50 .

For example, the simultaneous position estimation and mapping information implemented by the SLAM method and the like are stored in the server 50, and each user accesses the server by wireless communication to make the communication. When a map is created, the position information can be shared by only the coordinate values of each user, so that wireless transmission can be transmitted and received at a high speed.

5 is a flowchart illustrating a specific example of a mixed reality based military training method according to another embodiment of the present invention.

Referring to FIG. 5, in step S102, a live-view image is input in real time from a 3D camera attached to a user's body at an actual training site.

At the same time, team information such as life and death information and position information can be synchronized in real time (S104).

At the same time, the state of the simulated weapon, the attitude of the user, and the like can be recognized in real time by receiving information input from the simulated weapon sensor 37 or the sensor 35 attached to the user's body (S106).

In step S108, the location and size of the terrain or obstacle displayed in the real image inputted in step S102 are recognized and analyzed, and based on the analyzed result, the virtual concealment, cover, enemy, obstacle, It is possible to effectively create a virtual existence.

In step S110, based on the predetermined training difficulty, the team information synchronized in step S104, and the sensor information recognized in step S106, the virtual presence state or movement can be controlled in real time.

In step S112, a virtual image can be generated based on virtual presence, the training difficulty, the team information synchronized in step S104, and the sensor information recognized in step S106, which are generated in step S108 and controlled in step S110.

The virtual image generated in step S112 is combined with the real image input in step S102 (S114), and the synthesized 3D image can be output through the user's HMD 21 (S116).

As described above, according to the embodiments, in the actual simulated military training field, the real image of the scene input from the camera in real time is analyzed from the camera worn by the user, and the optimal virtual presence is arranged and generated. Synchronizes life and death information and location information with other users, generates a virtual image by controlling the state and motion of a virtual existence in real time based on the training difficulty and the recognized sensor information, By outputting the 3D image synthesized in the image to the user's HMD in real time, it is possible to improve the reality by training in the field, and realize the virtual existence such as enemy army, enemy army, military vehicle, cover, It eliminates the cost and hassle of having to prepare everything so that you can do mock military training in a more efficient and flexible situation. .

In addition, embodiments of the mixed reality based military training method described above may be implemented in the form of computer program instructions that may be executed through various computer components. Furthermore, the computer program embodied may be recorded on a computer-readable recording medium. The recording medium mentioned above may be, but is not necessarily, a ROM, a magnetic disk or a compact disk, an optical disk, or the like.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed embodiments, but, on the contrary, . Accordingly, the true scope of the present invention should be determined by the technical idea of the appended claims.

10: Military training device
11: 3D camera
13:
15: Sensor information recognition unit
17: Team information synchronization section
19: virtual image generation unit
21: HMD
23:
35: Gesture sensor
37: Mock weapon
50: Server

Claims (14)

As a mixed reality based military training device used in actual training field,
A team information synchronization unit for synchronizing team information;
A sensor information recognition unit for recognizing information input from a simulated weapon used by a user or a sensor attached to the user's body;
A virtual presence creator for analyzing a 3D real image of the training field input in real time and arranging a virtual presence based on the result of the analysis; And
A virtual image generation unit for generating a 3D virtual image based on at least one of the synchronized team information, the recognized sensor information, and the generated virtual presence;
Based military training device.
The method according to claim 1,
Wherein the team information includes at least one of life information and location information of the user and other users.
The method according to claim 1,
Wherein the team information synchronization unit performs simultaneous localization and mapping using simultaneous localization and mapping (SLAM).
The method according to claim 1,
The virtual presence means that,
Based on a result of analyzing the real image, at least one of virtual concealment, cover, enemy force, obstacle, weapon, and vehicle generated according to a predetermined training degree of difficulty.
The method according to claim 1,
Wherein the virtual image generating unit comprises:
Wherein the control unit controls in real time the status or movement of the virtual presence based on the predetermined training difficulty, the synchronized team information, and the recognized sensor information.
The method according to claim 1,
Wherein the real image is input from a camera worn by the user during training.
The method according to claim 1,
Wherein the generated virtual image is synthesized with the real image and displayed through a user's HMD.
As a mixed reality based military training method used in actual training field,
Synchronizing the team information;
Recognizing information input from a simulated weapon used by a user or a sensor attached to the user's body;
Analyzing a 3D real-life image of the training field input in real time, and arranging a virtual entity based on the analyzed result; And
Generating a 3D virtual image based on at least one of the synchronized team information, the recognized sensor information, and the generated virtual presence;
Based military training method.
9. The method of claim 8,
Wherein the team information includes at least one of life information and location information of the user and other users.
10. The method of claim 9,
Wherein the step of synchronizing the team information comprises simultaneous position estimation and mapping using SLAM.
9. The method of claim 8,
The virtual presence means that,
Based on a result of analyzing the real image, at least one of virtual concealment, cover, enemy, obstacle, weapon, and vehicle generated according to a predetermined training difficulty.
9. The method of claim 8,
Wherein the step of generating the virtual image comprises:
Controlling in real time the state or movement of the virtual entity based on a predetermined training difficulty, the synchronized team information, and the recognized sensor information;
Based military training method.
9. The method of claim 8,
Synthesizing the generated virtual image with the real image and displaying the virtual image through a user's HMD;
Based military training method. ≪ RTI ID = 0.0 > 8. < / RTI >
A computer-readable recording medium on which a program for executing the method according to any one of claims 8 to 13 is recorded.

Images