KR20230154488A - Submarine combat system using VR and control method of submarine combat system using VR - Google Patents

Abstract

The present invention relates to a submarine combat system using VR that can efficiently operate the combat system within a submarine with spatial constraints using VR and voice recognition, and a method of controlling the submarine combat system using VR. A submarine combat system using VR according to an embodiment of the present invention is a submarine combat system system in which a plurality of console devices are installed, is connected to a network with a plurality of console devices, and receives console screen information from the plurality of console devices, A control device that converts this into VR screen information; VR equipment that displays VR screen information converted from a control device; and a headset that inputs voice information and provides it as a control device,
The control device inputs voice information input through the headset, and when the input voice information is command information to be transmitted to each console device, it converts it into text information and provides it to each console device, and the input voice information is used in VR. When the video control information is for display on the device, the console screen information received from a plurality of console devices is synthesized based on the video control information and transmitted to the VR device,
When acceleration information is input through a VR device, the control device performs image synthesis based on the input acceleration information on the console screen information received from a plurality of console devices and transmits it to the VR device.

Description

Submarine combat system using VR and control method of submarine combat system using VR {Submarine combat system using VR and control method of submarine combat system using VR}

The present invention relates to a submarine combat system using VR that can efficiently operate the combat system within a submarine with spatial constraints using VR and voice recognition, and a method of controlling the submarine combat system using VR.

The content described in this section simply provides background information for this embodiment and does not constitute prior art.

VR is a technology that allows people to have a realistic experience in a virtual world, and utilizes an HMD, a display device mounted on the head. VR has been introduced and has various applications in computer games, medicine, aviation, and the military. In particular, in the military field, it is applied to military training and simulated combat training, and is very useful in that it allows soldiers to train for combat with a realistic feel.

VR has the characteristic of not requiring a specific space. Therefore, even if there are space constraints, it is possible to achieve the desired goal using VR.

Korean Patent Publication No. 10-2137872 (title of the invention: AR VR MR engagement simulation training system using operational theme content) is a simulation training system that includes a server storing programs for simulation training, and a simulation training program on the server. It includes a database that stores the necessary data and a display that allows you to set up a program for simulated training through a server, and a motion controller that outputs simulated training videos and controls them to enable training through the server. The motion controller outputs simulated training videos. It further includes wearable VR and a remote control that controls training according to simulated training videos output through VR, and a howitzer that can be trained to fire guided missiles and artillery shells using VR and remote control. . To improve piloting skills, AR and motion recognition technology can be applied to provide a combat simulation training system using AR VR MR that allows various training methods through operational theme content.

In other words, Korean Patent Publication No. 10-2137872 (title of the invention: AR VR MR engagement simulation training system using operational theme content) is a simulation training system using AR VR MR, which allows setting up a program for simulation training. We are starting to construct a motion controller that controls simulation training video output and training through a display and server.

Meanwhile, the combat system deployed on submarines has many limitations due to limited space. For example, in the case of a submarine commander, there is a need to directly and comprehensively check multiple console screens placed at various locations within the submarine and issue command orders in a unified manner, but due to limited space and distance constraints within the submarine, multiple console screens are It was very difficult to check in real time.

Therefore, the present invention proposes a configuration that utilizes VR in a combat system applied to a limited space, such as a submarine, to enable efficient surveillance and operation.

Korean Patent Publication No. 10-2137872 (Title of invention: AR VR MR combat simulation training system using operational theme content)

Therefore, the purpose of the present invention is to provide a submarine combat system using VR that can promote efficient operation of the combat system using VR and voice recognition in a combat system used within a limited space such as a submarine, and a submarine combat system using VR. It provides a control method for the system.

Another object of the present invention is to increase the sonar sound wave listening efficiency of sonar in a combat system used within a limited space such as a submarine, and to provide submarine combat using VR that can promote efficient monitoring and operation of the combat system according to sonar operation. It provides a control method for a submarine combat system using a system and VR.

In order to solve the above technical problem, the submarine combat system system using VR according to an embodiment of the present invention is a submarine combat system system installed with a plurality of console equipment, connected to a network with a plurality of console equipment, and a plurality of consoles. A control device that receives console screen information from the equipment and converts it into VR screen information; VR equipment that displays VR screen information converted from a control device; and a headset that inputs voice information and provides it as a control device,

The control device inputs voice information input through the headset, and when the input voice information is command information to be transmitted to each console device, it converts it into text information and provides it to each console device, and the input voice information is used in VR. When the video control information is for display on the device, the console screen information received from a plurality of console devices is synthesized based on the video control information and transmitted to the VR device,

When acceleration information is input through a VR device, the control device performs image synthesis based on the input acceleration information on the console screen information received from a plurality of console devices and transmits it to the VR device.

According to one embodiment, the control device includes a voice receiver that receives voice information input through a headset and analyzes whether it is command information to be transmitted to each console device or image control information for display on VR equipment; A voice text transmission unit that converts command information provided by the voice receiver into text information and transmits it to a plurality of console devices; It is characterized by including a video control unit that selects console screen information to be displayed on the VR equipment based on video control information provided by the audio receiver or acceleration information provided by the VR equipment.

According to one embodiment, the voice receiver includes an FFT conversion unit that performs FFT conversion on the received voice information; A feature point extraction unit that extracts feature points from the FFT-converted signal using CNN; a generator that analyzes voice information received using the feature point extraction signal and generates command information or image control information.

According to one embodiment, the video control unit includes a generator that generates video selection information for selecting console screen information to be displayed on VR equipment based on video control information provided by the audio receiver.

According to one embodiment, the image control unit is characterized by including a movement direction confirmation unit that inputs acceleration information provided by VR equipment and generates screen movement direction information.

According to one embodiment, the control device includes a console screen receiver that receives console screen information from a plurality of console devices; Video synthesis that converts a plurality of console screen information received from the console screen receiver into window screen information so that it can be displayed using VR equipment, and selects console screen information to be displayed on VR equipment under the control of the video control unit. wealth; and a VR transmitter that converts the output signal of the video synthesis unit into a VR signal and transmits it to the VR equipment.

According to one embodiment, a plurality of console devices includes a console screen transmitter that transmits a console screen according to the process and results of each function to the console screen receiver; A text receiving unit that receives text information transmitted from the voice text transmitting unit; It is characterized by including a text display unit that displays text information received from the text receiver on each console screen.

According to one embodiment, the video synthesis unit configures a plurality of console screens as window screens in the initial state and displays the entire video on the VR equipment.

According to one embodiment, the video synthesis unit divides the plurality of console screens into upper and lower parts in the initial state and displays the entire video on the VR equipment.

According to one embodiment, the image synthesis unit is characterized in that, in the initial state, it controls the display of a window screen at a preset specific position among the entire video display screen to be selected.

According to one embodiment, the image composition unit displays and controls the selection of another window screen based on the image selection information of the image control unit in the initially selected window screen.

According to one embodiment, the image synthesis unit controls the display of the selected window screen as the entire screen of the VR equipment based on the image selection information of the image control unit.

According to one embodiment, when the video synthesis unit controls the display of the selected window screen as the entire screen of the VR equipment, the selected console information is displayed as text on a portion of the screen of the VR equipment.

According to one embodiment, the video synthesis unit controls the display of command information to be delivered to a plurality of console devices at a preset position on the display screen of the VR equipment as text according to voice information input through a headset.

According to one embodiment, the video synthesis unit is characterized in that it controls deletion of command information to be transmitted to a plurality of console devices displayed at preset positions on the display screen of the VR equipment according to voice information input through the headset.

According to one embodiment, the video synthesis unit changes and controls command information to be transmitted to a plurality of console devices displayed at preset positions on the display screen of the VR equipment according to voice information input through a headset.

According to one embodiment, command information to be transmitted to a plurality of console devices displayed at preset positions on the display screen of the VR equipment is transmitted as text to a plurality of console devices through a voice text transmitter according to voice information input through a headset. Do it as

According to one embodiment, command information transmitted as text to a plurality of console devices is displayed in text in a window displayed on the entire screen information of the VR equipment, and when a confirmation signal is received from the corresponding console equipment, the text displayed in the window disappears. .

According to one embodiment, at least one of the plurality of console devices is for sonar linkage and control, and command information transmitted from the control device is displayed as text on the display screen of the console device.

In order to solve the above technical problem, the submarine combat system system using VR according to an embodiment of the present invention is a submarine combat system system in which a plurality of console equipment is installed, is connected to a network with a plurality of console equipment, and includes a plurality of consoles. A control device that receives console screen information from the equipment and converts it into VR screen information; VR equipment that displays VR screen information converted from a control device; and a headset that inputs voice information and provides it as a control device,

The control device includes a voice receiver that receives voice information input through a headset and analyzes whether it is command information to be transmitted to each console device or VR screen control information for display on the VR device; A voice text transmission unit that converts command information provided by the voice receiver into text information and transmits it to each console device; a video control unit that selects console screen information to be displayed on the VR equipment based on VR screen control information provided by the audio receiver and/or acceleration information provided by the VR equipment; A console screen receiver that receives console screen information from a plurality of console devices; A plurality of console screen information received from the console screen receiver is converted into small window screen information to be displayed using VR equipment, and the video is configured by selecting console screen information to be displayed on VR equipment under the control of the video control unit. synthesis section; And a VR transmitter that converts the output signal of the video synthesis unit into a VR signal and transmits it to the VR equipment,

The plurality of console devices includes a console screen transmitter that transmits a console screen according to the process and results of each function to the console screen receiver; A text receiving unit that receives text information transmitted from the voice text transmitting unit; It is characterized by including a text display unit that displays text information received from the text receiver on each console screen.

In order to solve the above technical problems, the control method of the submarine combat system system using VR according to an embodiment of the present invention uses the submarine combat system system using VR according to any one of claims 1 to 19, Receiving console screen information from a plurality of console devices at the control device and converting it into VR screen information; A step of inputting voice information input through a headset from the control device, and when the input voice information is command information to be transmitted to each console device, converting it into text information and providing it to each console device; When the audio information input from the control device through the headset is video control information for display on the VR equipment, video synthesis is performed based on the video control information on the console screen information received from multiple console devices and transmitted to the VR equipment. steps; And when acceleration information is input from the control device through the VR equipment, a step of performing image synthesis of the console screen information received from a plurality of console devices based on the input acceleration information and transmitting it to the VR equipment. .

The submarine combat system system using VR and the control method of the submarine combat system system using VR according to an embodiment of the present invention utilize VR in a combat system operated in a place where there are limited space and distance constraints, such as a submarine. Thus, information on multiple console screens can be provided to the commander in real time. Therefore, it becomes possible for a submarine or ship commander to check multiple console screens in real time and issue commands in a unified way, regardless of space constraints.

In addition, the present invention can convert command information to be transmitted to each console device through a headset and VR equipment worn by the commander into text information and provide it to a plurality of console devices, and the plurality of console devices can convert the received text information into text information. can be displayed on the console screen. Therefore, the present invention allows a manager who operates console equipment for sonar linkage and control, such as a sonar detector, to visually check the commander's command information on the console screen while completely focusing his hearing on sonar sound detection.

Figure 1 is an illustration of a plurality of console devices installed in a typical submarine.
Figure 2 shows a conceptual diagram of a submarine combat system using VR according to an embodiment of the present invention.
Figure 3 shows a detailed configuration diagram of a submarine combat system using VR according to an embodiment of the present invention.
Figure 4 shows a detailed configuration diagram of the voice receiver 160 in the submarine combat system using VR according to an embodiment of the present invention.
Figure 5 shows a detailed configuration diagram of the image control unit 180 in the submarine combat system using VR according to an embodiment of the present invention.
Figures 6 to 8 show examples of video display screens displayed on the VR equipment 300.
9 to 11 show other examples of video display screens displayed on the VR equipment 300.
Figures 12 to 14 show other examples of video display screens displayed on the VR equipment 300.

Hereinafter, embodiments disclosed in the present specification will be described in detail with reference to the attached drawings. However, identical or similar components will be assigned the same reference numbers regardless of reference numerals, and duplicate descriptions thereof will be omitted. The suffixes "part" and "unit", "device" and "system", etc. for the components used in the following description are given or used interchangeably only considering the ease of writing the specification, and in themselves have distinct meanings or It doesn't have a role.

Additionally, in describing the embodiments disclosed in this specification, if it is determined that detailed descriptions of related known technologies may obscure the gist of the embodiments disclosed in this specification, the detailed descriptions will be omitted. In addition, the attached drawings are only for easy understanding of the embodiments disclosed in this specification, and the technical idea disclosed in this specification is not limited by the attached drawings, and all changes included in the spirit and technical scope of the present invention are not limited. , should be understood to include equivalents or substitutes.

Terms containing ordinal numbers, such as first, second, etc., may be used to describe various components, but the components are not limited by the terms. The above terms are used only for the purpose of distinguishing one component from another.

When a component is said to be "connected" or "connected" to another component, it is understood that it may be directly connected to or connected to the other component, but that other components may exist in between. It should be. On the other hand, when it is mentioned that a component is “directly connected” or “directly connected” to another component, it should be understood that there are no other components in between.

Singular expressions include plural expressions unless the context clearly dictates otherwise.

In this application, terms such as “comprise” or “have” are intended to designate the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, but are not intended to indicate the presence of one or more other features. It should be understood that this does not exclude in advance the possibility of the existence or addition of elements, numbers, steps, operations, components, parts, or combinations thereof.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. It is obvious to those skilled in the art that the present invention can be embodied in other specific forms without departing from the spirit and essential features of the present invention.

Submarines are a fleet of ships that can operate under water, and their scope of use is so wide that they can perform a wide variety of missions, including enemy reconnaissance and torpedo attacks, but also include meteorological and oceanographic surveys and human rescue. Due to its nature, submarines have a limited internal space and are equipped with various components, including combat systems, within that limited space.

As shown in Figure 1, a typical submarine combat system system has multiple multi-function control console devices deployed, and allows a specific administrator to manage each console device. And the commander, who commands the submarine as a whole, directly monitors multiple console screens separately from the managers of each console device and issues command orders in a unified manner.

However, due to the structure and distance constraints of the submarine's internal space, the distance between multiple console screens and the commander may be more than a certain distance, and there may be cases where the console screen is blocked by the relevant manager. Because of these limitations, a problem arises in that it takes a certain amount of time for the submarine commander to recognize the entire situation, which becomes a problem in the submarine's combat system system, which must be accompanied by prompt and accurate command.

In addition, the console equipment that performs the sonar sound detection function, which is one of the important functions of the submarine, and the corresponding manager (sonic searcher) have to focus entirely on the sonar sound detection by hearing, but there is a problem of having to listen to the commander's commands at the same time. .

Therefore, in the present invention, in a combat system that has spatial constraints such as a submarine, a plurality of console screens are provided to the commander in real time using a VR screen, and the commander can enlarge, reduce, or display the screen of interest according to the situation. It allows rapid operation of the combat system by allowing the commander's commands to be selected and transmitted as text through multiple console devices.

In addition, the present invention allows the console equipment and sonar that perform the sonar sound detection function to focus entirely on sonar sound detection by hearing, and the commands of the submarine commander are displayed in text on the screen of the console equipment through text transmission. It is characterized by controlling it so that it can be done.

In addition, the present invention is characterized in that the submarine's combat system uses voice and VR acceleration information to control the VR screen display provided to the commander and to control the text display of command information on a plurality of console devices.

Figure 2 shows a conceptual diagram of a submarine combat system using VR according to an embodiment of the present invention.

In the illustrated embodiment, five multi-function control console devices are installed. Each console device (200, 210, 220, 230, 240) is placed within the submarine and controls surveillance and operation to achieve the objectives of survivability and stealth of the submarine. Each console device (200, 210, 220, 230, 240) uses single and multiple acoustic/non-acoustic sensors to estimate target motion information, target maneuver analysis technology for submarines, sonar linkage and control technology, heavy torpedo control technology, and multiple sensor complexes in the underwater environment. It is equipped with navigation technology, sonar environment analysis processing and analysis technology, and target maneuver analysis technology. To this end, each console equipment (200, 210, 220, 230, 240) has features such as sonar sensor signal reception and display processing, target management and engagement support through target information fusion processing, armament operation, tactical navigation support, and operation support.

Each of the console devices 200, 210, 220, 230, and 240 is connected to the control device 100 through a network. Each of the console devices (200, 210, 220, 230, and 240) provides console screen information according to the performance of each function to the control device 100 through the network, and the control device 100 converts the received console screen information into a VR screen to provide VR equipment. Provided as (300).

The control device 100 is connected to the commander's headset 400 and controls the input and output of voice information. If the input voice information is command information to be transmitted to each console device, it is converted into text information and transmitted to the corresponding console device. to provide.

And when acceleration information is recognized based on the commander's movement in the VR equipment 300, the recognized acceleration information is provided to the control device 100. At this time, the control device 100 controls the image displayed on the VR equipment 300 based on the recognized VR acceleration information.

In the illustrated embodiment, the VR equipment 300 is configured to provide the commander with screen information about a plurality of console equipment (200, 210, 220, 230, 240), and in one embodiment of the present invention, there are two divided upper and lower for each console equipment. It is displayed as a window, and consists of a total of 10 windows.

In the illustrated embodiment, the headset 400 and the VR equipment 300 are displayed separately, but an integrated head-mounted display (HMD) can be used. And the VR equipment 300 is equipped with a gyro sensor to detect the commander's movements.

Figure 3 shows a detailed configuration diagram of a submarine combat system using VR according to an embodiment of the present invention.

A plurality of console devices (200, 210, 220, 230, 240) are connected to the control device 100 through a wired or wireless network. Although not shown, multiple console devices are wired or wirelessly connected to multiple sensors, devices, and weapons mounted inside and outside the submarine in order to perform their unique functions. That is, the plurality of console devices 200, 210, 220, 230, and 240 perform unique functions assigned to them under the management of the submarine combat system system, and display the process and results of performing the functions on the console screen.

For example, console equipment that receives and displays sonar sensor signals displays the performance process and results of the corresponding function on the console screen. Console equipment that performs target management and engagement support through target information fusion processing displays the performance process and results of the corresponding functions on the console screen. The console equipment that controls armament operation displays the performance process and results of the corresponding function on the corresponding console screen. Console equipment that supports tactical navigation support and operation displays the performance process and results of the corresponding function on the console screen.

In this way, screen information displayed according to the process and results of performing each function in each console device is transmitted to the control device 100 through the network. Therefore, each console device includes a console screen transmitter 250 that transmits console screen information.

Additionally, each console device includes a text receiver 260 that receives text information transmitted from the control device 100. It also includes a text display unit 270 that displays the received text information in text at a preset location on the console screen so that the manager of the console device can quickly recognize it.

Typically, the text display unit 270 is set to a certain area of the console screen, and the console screen commonly uses an area that displays the execution process and results of each function described above. Therefore, it is preferable that the text on the text display unit 270 is displayed only for a set short period of time so as not to interfere with the display of the corresponding function. Alternatively, it is desirable to set the text display on the text display unit 270 to disappear as soon as the administrator of the corresponding console screen confirms it. To this end, the corresponding console equipment can be configured to transmit a text confirmation signal to the control device 100.

In addition, the text displayed on the text display unit 270 is displayed by converting the commander's voice command information into text information, and it is desirable for the manager of the corresponding console equipment to quickly recognize it visually. In particular, if the console equipment in question is a console equipment that performs sonar sensor signal reception and display processing, in order to detect and identify sonar targets, the manager (sonic searcher) of the console equipment must focus all of his hearing on sonar target detection. . Therefore, when the command text information commanded by the commander is displayed on the console equipment while the manager of the console equipment is fully focused aurally on sonar target detection, the manager immediately recognizes the command visually and performs control accordingly. It becomes possible to do so.

And the control device 100 is connected wired or wirelessly to the headset 400 worn by the commander. And the control device 100 is connected to the VR equipment 300, which is an image display device, by wire or wirelessly.

The control device 100 includes a console screen receiver 140 that receives console screen information from a plurality of console devices, and can convert the plurality of console screen information received from the console screen receiver into window screen information and display it using VR equipment. A video synthesis unit 170 that selects and configures console screen information to be displayed on the VR equipment under the control of the video control unit, and a VR transmitter 190 that converts the output signal of the video synthesis unit into a VR signal and transmits it to the VR equipment. Includes.

In addition, the control device 100 includes a voice receiver 160 that receives voice information input through the headset 400 and analyzes whether it is command information to be transmitted to each console device or image control information for display on VR equipment. , a voice text transmitter 150 that converts command information provided by the voice receiver into text information and transmits it to a plurality of console devices, VR equipment based on video control information provided by the voice receiver or acceleration information provided by the VR equipment It includes an image control unit 180 that selects console screen information to be displayed.

And, if the voice information analyzed by the voice receiver 160 is video control information for controlling the image to be displayed on the VR equipment 300, the control device 100 controls the video synthesizer based on this signal. Select the output signal of (170).

Additionally, the VR equipment 300 detects acceleration information according to the commander's movement using a gyro sensor and provides the corresponding acceleration information to the video control unit 180. At this time, the image control unit 180 selects the output signal of the image synthesis unit 170 based on the detected acceleration information.

And the VR image signal transmitted from the VR transmitter 190 is visually provided to the submarine commander through the VR equipment 300.

Figure 4 shows a detailed configuration diagram of the voice receiver 160 in the submarine combat system using VR according to an embodiment of the present invention.

The voice receiver 160 receives voice information from the headset 400. The received voice information is subjected to FFT conversion and feature point extraction through the FFT converter 161 and the feature point extractor 162. Then, the extracted feature points are compared with previously stored audio information to determine whether it is command information for the system or image control information for selection and control of the displayed image, and generate command information or image control information. Includes part 163.

Here, the feature point extraction unit 162 extracts feature points from the voice signal converted through the FFT converter 161 using a CNN (Convolution Neural Network). In addition, comparison with pre-stored voice information using the extracted feature points is signal processed based on the meaning of the word or/and the current display state of the screen. For example, with the console screen at the top of number 4 currently selected, 'Zoom to the top of console number 4' and 'Zoom to console' change the console screen at the top of number 4 to VR based on the meaning of the word or (and) the display state of the current screen. It can be recognized as the same command to display the equipment 300 in full screen size. Similarly, when a specific console screen is enlarged and selected, 'console entire' and 'console reduction' are selected for all console screens on the entire screen of the VR equipment 300 based on the meaning of the words or (and) the current display state of the screen. It can be recognized as the same command to display as a small window screen. As such, in the present invention, for signal processing of input voice information, it is possible to perform signal processing based on the meaning of the word or/and the current display state of the screen in addition to the predetermined word form.

Figure 5 shows a detailed configuration diagram of the image control unit 180 in the submarine combat system using VR according to an embodiment of the present invention.

The video control unit 180 selects and controls the video on the console screen displayed on the VR equipment 300 based on video control information input through the voice receiver 160 or acceleration information input through the VR equipment 300. For this purpose, the VR equipment 300 is configured to detect the movement of the submarine system tube and provide acceleration information according to the detected value to the image control unit 180.

The acceleration information input to the image control unit 180 is input to the moving direction confirmation unit 182, and the moving direction confirmation unit 182 recognizes and outputs direction information from the input acceleration information. The direction information output from the moving direction confirmation unit 182 or the image control information output from the voice receiver 160 is input to the image selection information generation unit 181. The image selection information generator 181 outputs image selection information for selecting an image to be output through the VR equipment 300 to the image synthesis unit 170 based on the input information.

Next, we will look at the operation process of the submarine combat system using VR and voice recognition according to an embodiment of the present invention.

Figures 6 to 8 show examples of video display screens displayed on the VR equipment 300.

In the initial state, the system manager's VR equipment (300) displays screen information about all console equipment (200, 210, 220, 230, 240) in the submarine. To this end, the control device 100 receives screen information of all console devices (200, 210, 220, 230, and 240) through the console screen receiver 140, composites the screen information into multiple small windows within one screen in the video synthesis unit 170, and transmits the screen information to the VR transmitter. It is transmitted to VR equipment (300) through (190).

Figure 6 shows the state of the video screen displayed on the VR equipment 300 in the initial state, and the top of console screen 5 is selected in the initial state. That is, the image synthesis unit 170 can configure each of the plurality of console screens as window screens in the initial screen state of the VR equipment 300 and control the entire image to be displayed. In addition, in the initial screen state, a plurality of console screens can be divided into upper and lower sections and controlled to display the entire VR equipment video. Additionally, in the initial state, the image synthesis unit 170 can control the display of a window screen at a preset specific position (position 5-1) among all video display screens in a selected state.

In this state where it is displayed on the VR equipment 300, if the commander turns his head to the right or gives a voice command “top of console 4”, the top window of console 4 is selected as shown in FIG. In other words, the image synthesis unit 170 selects another window screen 4-1 as shown in FIG. 7 based on the image selection information of the image control unit 180 from the initially selected window screen 5-1. You can control the display.

At this time, the video display control of the VR equipment 300 is performed by an image synthesis unit under the control of the video control unit 180 that recognizes the video control information through voice recognition shown in FIG. 4 or the acceleration information based on the commander's movement shown in FIG. 5. Control is performed through the image selection information in (170).

And when the system manager commands 'Enlarge the top of console 4' or 'Enlarge the console' by voice, the voice information is recognized by the video control unit 180 through the voice receiver 160, and the currently selected upper window 4 is displayed. As shown in Figure 8, the screen size is converted to the full screen of the VR equipment 300. Additionally, the selected console screen information, such as 'top 4', is also displayed in the upper right corner of the VR screen.

9 to 11 show other examples of video display screens displayed on the VR equipment 300.

As shown in FIG. 8, when a specific console screen is enlarged and the commander inputs the command 'full console' or 'reduce console', all console devices (200, 210, 220, 230, 240) in the submarine are displayed in the commander's VR equipment (300). Screen information about is displayed. The display screen of the VR equipment 300 at this time is shown in FIG. 9. In this case, since the top of console No. 4 was selected in the step before the command, the top of console No. 4 is also selected in the display state of FIG. 9.

Next, when the commander inputs a command to be transmitted to the console equipment, the command is displayed at the bottom of the display screen of the VR equipment 300, as shown in FIG. 10. For example, when the system manager inputs the command 'Confirm delivery target number 1', the text to be delivered is displayed at the bottom of the screen, such as 'Confirm target number 1'. That is, the image synthesis unit 170 can display and control command information to be transmitted to a plurality of console devices at a preset position on the display screen of the VR equipment 300 as text according to voice information input through the headset 400. . At this time, the command information can be transmitted to the video synthesis unit 170 through the voice receiver 160 and the video control unit 180.

And when the commander orders 'cancel delivery', as shown in FIG. 11, the text message to be displayed is deleted. That is, the image synthesis unit 170 can control deletion of command information to be transmitted to a plurality of console devices displayed at preset positions on the display screen of the VR equipment 300 according to voice information input through the headset. Control at this time can also be transmitted to the video synthesis unit 170 through the voice receiver 160 and the video control unit 180.

Figures 12 to 14 show other examples of video display screens displayed on the VR equipment 300.

FIG. 12 shows an example in which the video synthesis unit 170 changes and controls command information to be transmitted to a plurality of console devices displayed at preset positions on the display screen of the VR device according to voice information input through a headset. Figure 12 is an example of what is displayed on the VR equipment when the commander changes the display screen from "Confirm delivery target number 1" to "Confirm delivery target number 2," as shown in Figure 10.

That is, the video synthesis unit 170 analyzes audio information input through the headset 400 through the audio receiver 160 and the video control unit 180, and synthesizes the video under the control of the video control unit 180. It is possible to change and control command information to be displayed on the VR screen configured in the unit 170.

In addition, the video synthesis unit 170 transmits command information to a plurality of console devices displayed at preset positions on the display screen of the VR equipment 300 through the voice text transmission unit 150 according to voice information input through the headset. Text is sent to multiple console devices. That is, the command information displayed at the preset position at the bottom of the display screen of the VR equipment 300, as shown in FIG. 12, "Check target number 2", is a plurality of consoles based on the voice information instructing the commander to transmit the command, "Okay". transmitted to the equipment.

And as shown in FIG. 13, the command information sent as text to a plurality of console devices is displayed in text in the window displayed on the entire screen information of the VR equipment, and as shown in FIG. 14, a confirmation signal is received from the corresponding console equipment. You can control the text displayed in the window to disappear. That is, when the plurality of console devices that have received text information confirm the text information, they can provide a text confirmation signal (control information) to the control device 100. The control device 100 can control the text to disappear from the window corresponding to the console that received the text confirmation signal from the VR display screen information of the video synthesis unit 170.

The submarine combat system using VR of the present invention described above is implemented in a computing environment including a general computing device. The computing environment may include computing devices. A computing device may be any type of computing device that transmits and receives signals to and from other terminals. In an embodiment of the present invention, the control device 100 can be replaced with a computing device, and software that converts the received console screen image into a VR screen image can be installed in the control device 100.

A computing device includes at least one processor (or control module), a computer-readable storage medium, and a communication bus. The processor may control the computing device to operate according to the previously mentioned embodiments. The processor may execute one or more programs stored in a computer-readable storage medium. One or more programs may include one or more computer-executable instructions, which, when executed by a processor, may be configured to cause a computing device to perform operations according to example embodiments.

A computer-readable storage medium is configured to store computer-executable instructions, computer-executable instructions, program code, program data, and/or other suitable form of information. A program stored on a computer-readable storage medium includes a set of instructions executable by a processor. In one embodiment, the computer-readable storage medium includes memory (volatile memory, such as random access memory, non-volatile memory, or a suitable combination thereof), one or more magnetic disk storage devices, optical disk storage devices, flash memory devices. , other types of storage media that can be accessed by a computing device and store desired information, or a suitable combination thereof.

A communication bus interconnects various other components of a computing device, including a processor and computer-readable storage media.

A computing device may also include one or more input/output interfaces and one or more communication interfaces that provide an interface for one or more input/output devices. The input/output interface and communication interface are connected to a communication bus. An input/output device (not shown) may be connected to other components of the computing device through an input/output interface. Exemplary input/output devices include input devices such as pointing devices (such as a mouse or trackpad), keyboards, touch input devices (such as a touchpad or touch screen), voice or sound input devices, various types of sensor devices, and/or imaging devices; and/or output devices such as display devices, printers, speakers, and/or network cards. An exemplary input/output device may be a component constituting a computer device and may be included within the computing device, or may be connected to the computing device as a separate device that is distinct from the computing device.

Operations according to the present embodiments may be implemented in the form of program instructions that can be performed through various computer means and recorded on a computer-readable medium. Computer-readable media refers to any media that participates in providing instructions to a processor for execution. Computer-readable media may include program instructions, data files, data structures, or combinations thereof. For example, there may be magnetic media, optical recording media, memory, etc. A computer program may be distributed over networked computer systems so that computer-readable code can be stored and executed in a distributed manner. Functional programs, codes, and code segments for implementing this embodiment can be easily deduced by programmers in the technical field to which this embodiment belongs.

The above detailed description should not be construed as restrictive in any respect and should be considered illustrative. The scope of the present invention should be determined by reasonable interpretation of the appended claims, and all changes within the equivalent scope of the present invention are included in the scope of the present invention.

100: control device, 140: console screen receiver, 150: voice text transmission unit, 160: voice receiver, 161: FFT conversion unit, 162: feature point extraction unit, 163: command information/video control information generation unit, 170: video synthesis unit , 180: video control unit, 181: video selection information generation unit, 182: movement direction confirmation unit, 190: VR transmitter, 200 ~ 240: console equipment, 250: console screen transmitter, 260: text receiver, 270: text display unit, 300 : VR equipment, 400 : Headset

Claims (21)

In a submarine combat system system equipped with a plurality of console equipment,
A control device connected to a network with a plurality of console devices, receiving console screen information from the plurality of console devices, and converting it into VR screen information;
VR equipment that displays VR screen information converted from a control device; and
Includes a headset that inputs voice information and provides it as a control device,
The control device inputs voice information input through the headset, and when the input voice information is command information to be transmitted to each console device, it converts it into text information and provides it to each console device, and the input voice information is used in VR. When the video control information is for display on the device, the console screen information received from a plurality of console devices is synthesized based on the video control information and transmitted to the VR device,
When acceleration information is input through VR equipment, the control device performs image synthesis based on the input acceleration information on the console screen information received from a plurality of console equipment and transmits it to the VR equipment. A submarine combat system system using VR. In claim 1,
The control device includes a voice receiver that receives voice information input through a headset and analyzes whether it is command information to be transmitted to each console device or video control information for display on VR equipment;
A voice text transmission unit that converts command information provided by the voice receiver into text information and transmits it to a plurality of console devices;
A submarine combat system system using VR that includes a video control unit that selects console screen information to be displayed on VR equipment based on video control information provided by the voice receiver or acceleration information provided by VR equipment. In claim 2,
The voice receiver includes an FFT conversion unit that performs FFT conversion on the received voice information;
A feature point extraction unit that extracts feature points from the FFT-converted signal using CNN;
A submarine combat system using VR that includes a generator that analyzes voice information received using a feature point extraction signal and generates command information or image control information. In claim 3,
The video control unit is a submarine combat system using VR that includes a generator that generates video selection information to select console screen information to be displayed on VR equipment based on video control information provided by the voice receiver. In claim 4,
A submarine combat system using VR that includes a video control unit that inputs acceleration information provided by VR equipment and a movement direction confirmation unit that generates screen movement direction information. In claim 5,
The control device includes a console screen receiver that receives console screen information from a plurality of console devices;
Video synthesis that converts a plurality of console screen information received from the console screen receiver into window screen information so that it can be displayed using VR equipment, and selects console screen information to be displayed on VR equipment under the control of the video control unit. wealth; and
A submarine combat system system using VR that includes a VR transmitter that converts the output signal of the video synthesis unit into a VR signal and transmits it to VR equipment. In claim 6,
The plurality of console devices includes a console screen transmitter that transmits a console screen according to the process and results of each function to the console screen receiver;
A text receiving unit that receives text information transmitted from the voice text transmitting unit;
A submarine combat system using VR that includes a text display unit that displays text information received from the text receiver on each console screen. In claim 7,
The video synthesis unit is a submarine combat system using VR that configures multiple console screens as window screens in the initial state and displays the entire video on VR equipment. In claim 8,
The video synthesis unit is a submarine combat system using VR that divides multiple console screens into upper and lower sections in the initial state and displays the entire video on VR equipment. In claim 9,
The video synthesis unit is a submarine combat system using VR that controls the display of a window screen at a preset specific location in the selected state among the entire video display screen in the initial state. In claim 10,
The video synthesis unit is a submarine combat system using VR that displays and controls the selection of other window screens based on the video selection information of the video control unit from the initially selected window screen. In claim 11,
The video synthesis unit is a submarine combat system using VR that controls the display of the selected window screen as the full screen of the VR equipment based on the video selection information of the video control unit. In claim 12,
A submarine combat system using VR that displays and controls the selected console information as text on a portion of the screen of the VR equipment when the video synthesis unit controls the display of the selected window screen as the entire screen of the VR equipment. In claim 13,
The video synthesis unit is a submarine combat system system using VR that displays and controls command information to be transmitted to a plurality of console devices at a preset location on the display screen of the VR device as text according to voice information input through a headset. In claim 14,
The video synthesis unit is a submarine combat system system using VR that deletes and controls command information to be transmitted to a plurality of console devices displayed at preset positions on the display screen of the VR device according to voice information input through the headset. In claim 15,
The video synthesis unit is a submarine combat system system using VR that changes and controls command information to be transmitted to a plurality of console devices displayed at preset positions on the display screen of the VR device according to voice information input through a headset. In claim 16,
A submarine combat system using VR in which command information to be transmitted to a plurality of console devices displayed at preset positions on the display screen of the VR equipment is sent as text to a plurality of console devices through a voice text transmitter according to the voice information input through the headset. . In claim 17,
A submarine combat system using VR in which command information sent as text to multiple console devices is displayed in a window displayed on the full screen information of the VR device, and when a confirmation signal is received from the corresponding console device, the text displayed in the window disappears. In claim 18,
A submarine combat system using VR in which at least one console device among a plurality of console devices is for sonar linkage and control, and command information transmitted from the control device is displayed in text on the display screen of the corresponding console device. In a submarine combat system system equipped with a plurality of console equipment,
A control device connected to a network with a plurality of console devices, receiving console screen information from the plurality of console devices, and converting it into VR screen information;
VR equipment that displays VR screen information converted from a control device; and
Includes a headset that inputs voice information and provides it as a control device,
The control device includes a voice receiver that receives voice information input through a headset and analyzes whether it is command information to be transmitted to each console device or VR screen control information for display on the VR device;
A voice text transmission unit that converts command information provided by the voice receiver into text information and transmits it to each console device;
a video control unit that selects console screen information to be displayed on the VR equipment based on VR screen control information provided by the audio receiver and/or acceleration information provided by the VR equipment;
A console screen receiver that receives console screen information from a plurality of console devices;
A plurality of console screen information received from the console screen receiver is converted into small window screen information to be displayed using VR equipment, and the video is configured by selecting console screen information to be displayed on VR equipment under the control of the video control unit. synthesis section; and
It includes a VR transmitter that converts the output signal of the video synthesis unit into a VR signal and transmits it to the VR equipment,
The plurality of console devices includes a console screen transmitter that transmits a console screen according to the process and results of each function to the console screen receiver;
A text receiving unit that receives text information transmitted from the voice text transmitting unit;
A submarine combat system using VR that includes a text display unit that displays text information received from the text receiver on each console screen. Using the submarine combat system system using VR according to any one of claims 1 to 20,
Receiving console screen information from a plurality of console devices at the control device and converting it into VR screen information;
A step of inputting voice information input through a headset from the control device, and when the input voice information is command information to be transmitted to each console device, converting it into text information and providing it to each console device;
When the audio information input from the control device through the headset is video control information for display on the VR equipment, video synthesis is performed based on the video control information on the console screen information received from multiple console devices and transmitted to the VR equipment. steps; and
When acceleration information is input from the control device through the VR device, submarine combat using VR includes performing image synthesis on the console screen information received from a plurality of console devices based on the input acceleration information and transmitting it to the VR device. Control method of a system system.