CN114743433A

CN114743433A - Multi-channel alarm presenting method and device for simulating threats in flight training environment

Info

Publication number: CN114743433A
Application number: CN202210328837.3A
Authority: CN
Inventors: 姚乃明; 陈辉; 郭雯丽; 刘璐; 王宏安; 庄迁政; 李晨; 曾伟明
Original assignee: Institute of Software of CAS; China Aeronautical Radio Electronics Research Institute
Current assignee: Institute of Software of CAS; China Aeronautical Radio Electronics Research Institute
Priority date: 2021-12-23
Filing date: 2022-03-30
Publication date: 2022-07-12
Anticipated expiration: 2042-03-30
Also published as: CN114743433B

Abstract

The invention provides a multi-channel alarm presenting method and device for simulating threats in a flight training environment, and relates to the technical field of man-machine interaction and the related technical field of multi-mode presentation. Calculating the visual field range of the user; dividing the threat target into a threat target in the user visual field range and a threat target outside the user visual field range according to whether the threat target is positioned in the user visual field range or not; for threat targets in the user visual field range, pushing and displaying threat information to a user in a user visual field central area; and for threat targets outside the visual field range of the user, prompting threat information to the user by using an audio-visual dual-channel presentation method. The method and the device can present threat target information to the user in real time along with the change of the head pose of the user, improve the perception capability of the user on the environment situation, shorten the discovery and cognition time of the user on the threat target, and improve the interaction efficiency of the user in the simulated flight training process.

Description

Multi-channel alarm presenting method and device for simulating threats in flight training environment

Technical Field

The invention relates to the technical field of human-computer interaction and the related technical field of multi-mode presentation, in particular to a multi-channel alarm presentation method and an electronic device for simulating a threat target in a flight training environment.

Background

The simulated flight training is a complex man-machine interaction task and has high requirements on the perception and processing capability of users. In the simulated flight training environment, the user faces various threats, if the threat warning system can prompt the user to threaten the information of a target in a natural interaction mode, the system is greatly beneficial to improving the perception capability of the user to the surrounding situation, so that the user can quickly make a response and take proper countermeasures, thereby assisting the user to finish a training task with high quality and high efficiency. Threat object presentation techniques have wide application in a variety of scenarios. For example, when flight training is simulated, the system can inform the user of the position, the type, the threat level and the like of the threat target by presenting threat target information such as enemy aircraft, enemy warships and the like, assist the user to master more comprehensive environment situation information, and improve the success rate and the completion efficiency of a training task. For example, in a simulated driving training environment, the system can improve the comprehensive perception capability of the user on the road surface situation by presenting threat targets influencing driving safety, such as adjacent vehicles, pedestrians and the like with collision risks, and assist the user in taking corresponding safety measures, so that the purpose of safe driving is achieved. When presenting relevant threat information, the system needs to be performed without affecting the user's completion of the primary task. Most existing threat target presentation methods adopt a mode of directly presenting threat data, and neglect integral perception taking a user body as a center, so that the presentation effect of threat target information is not natural enough, and a user is easy to generate higher cognitive load.

The existing threat alarm presenting method can be divided into two types, wherein the first type is two-dimensional overlook image presentation, and the second type is three-dimensional stereo model presentation. The two-dimensional overhead image presentation method is a view obtained by orthographic projection from top to bottom by taking a carrier where a user is located as a center; the three-dimensional threat presenting method is characterized in that the warning information is presented in the position of the threat relative to the user by taking the user as the center. The two-dimensional top-view threat presentation method has the following defects: 1) threat alert location information loss: because threat information in a three-dimensional space needs to be presented in a two-dimensional plane, the presentation of part of position information of threat alarm has limitation; 2) with higher cognitive load: the method comprises the steps that a two-dimensional overlook image is adopted to present threat alarm information, and when a user perceives the position of the threat alarm information, space perception with isohexia as the center is used; the main scene task of the user is a spatial perception centered on the user's body. For example, in simulated flight training, if a user perceives a threat target using a two-dimensional top-view threat information presentation method, the user needs to perform conversion from the alien-center spatial coordinates to the body-center spatial coordinates, which causes additional cognitive load. The three-dimensional threat presentation method has the following defects: 1) the visible area of existing display devices is limited: the effective presentation area of the devices commonly used in the market today, such as the virtual reality head-mounted display device, the augmented reality head-mounted display device, the vehicle-mounted or airborne head-up display, is too small to support the user to fully observe the environmental information. 2) Complex three-dimensional model manipulation: generally, a user needs to drag, zoom and translate the three-dimensional presentation model to completely and accurately recognize the alarm information presented by the model, but it is difficult for the user to correctly adjust the presentation visual angle of the three-dimensional model while driving a vehicle.

In summary, the existing presentation methods and systems cannot fully present the current situation to the user, resulting in a situation that the user does not perceive the situation fully, and the user has a higher cognitive load in the corresponding task scenario. How to design a user-centered adaptive threat object presentation method and provide a high-efficiency and low-load threat presentation system for users becomes an urgent problem to be solved.

Disclosure of Invention

In order to overcome the defects of the prior art, the invention provides a multi-channel alarm presenting method and an electronic device for a threat target in a simulated flight training environment.

In order to achieve the purpose, the technical scheme adopted by the invention is as follows:

a multi-channel alarm presenting method for simulating threats in a flight training environment comprises the following steps:

1) calculating the visual field range of the user;

2) dividing the threat target into a threat target in the user visual field range and a threat target outside the user visual field range according to whether the threat target is positioned in the user visual field range or not;

3) for threat targets in the user visual field range, pushing and displaying threat information to a user in a user visual field central area;

4) and for threat targets outside the visual field range of the user, prompting threat information to the user by using an audio-visual dual-channel presentation method.

Further, the user field of view is calculated from the user's current head pose and field of view.

Further, the method for acquiring the current head posture of the user comprises the following steps: the method comprises the steps of acquiring 6 degree-of-freedom attitude parameters of a head of a user in a three-dimensional space by using a pose sensor in the head-mounted device, wherein the 6 degree-of-freedom attitude parameters comprise X, Y, Z three-axis positions with the pose sensor as a coordinate origin, a heading angle, a pitch angle and a roll angle.

Further, the visual range refers to a maximum display range provided by a head-mounted display device used by a user.

Furthermore, the central area of the user field of view is a circular area defined by taking the center of the user field of view as a center of a circle and taking one fifth of the height range of the field of view as a radius.

Further, the audiovisual dual-channel presentation method comprises the following steps: the method includes presenting one or more pointers representing the location of the threat target to a user via a head mounted display, and presenting a warning alert tone via a head mounted stereo headset, the warning alert tone characterizing the location of the threat target and threat level information via a sound characteristic.

Further, for a plurality of threat objects outside the visual field range of the user, the warning prompt tones of the threat objects are played in turn in a virtual surround sound mode.

Further, the threat objects in the visual field range of the user are displayed through the solid line icons and are displayed at the positions of the threat objects.

Further, for the threat target out of the visual field of the user, the display is carried out through a dotted line icon, and the offset direction and the distance of the position where the threat target is actually located relative to the current position are displayed through an arrow icon.

A multi-channel warning presenting device simulating threats in a flight training environment comprises head pose acquisition equipment, user input equipment, graphic display equipment, audio output equipment, a memory and a processor, wherein the head pose acquisition equipment is head-mounted equipment with a pose sensor, the graphic display equipment is a head-mounted display, the audio output equipment is used for outputting warning prompt tones, a computer program is stored in the memory, and the processor realizes the steps of the method when executing the computer program.

Compared with the prior art, the invention has the beneficial effects that:

1) the method uses a double-channel presentation method of auditory sense and visual sense, fully calls a plurality of sensory abilities of the user, can effectively help the user to quickly discover and recognize threat targets outside the visual field range, improves the completion efficiency of training tasks, and reduces the cognitive load of the user in the training process.

2) The invention provides a natural and effective presentation scheme for the threat targets in the visual field range and out of the visual field range respectively, and the two schemes can help the user to quickly distinguish the urgency degree of the threat targets and have the beneficial effect of reducing and enhancing the complexity of the display interface.

Drawings

FIG. 1 is a schematic diagram of a multi-channel warning presentation device for simulating threats in a flight training environment according to an embodiment of the invention.

FIG. 2 is a schematic diagram of an adjustment interface with an exemplary user initial field of view.

FIG. 3 is a flowchart of a multi-channel alarm presenting method for simulating threats in a flight training environment according to an embodiment of the present invention.

FIG. 4 is a schematic diagram of a multi-channel warning presentation device for simulating threats in a flight training environment according to another embodiment of the invention.

Detailed Description

The present invention will be described in detail with reference to the accompanying drawings. It should be noted that the described embodiments are for illustrative purposes only and are not limiting on the scope of the invention.

The embodiment discloses a multi-channel alarm presenting method and device for simulating threats in a flight training environment, and provides a three-dimensional threat presenting method taking a user as a core. For example, in a simulated flight training scenario, a threat to the left rear of the user's field of view may be mapped to the left inside the user's field of view, with a specially designed icon indicating that the threat is not in the orientation corresponding to the icon in the field of view, but rather is located to the left rear outside the field of view, with the presentation of more visual elements and auditory channels indicating the specific orientation of the threat, so that the user can directly and naturally perceive the threat in the scene.

Fig. 1 shows a multi-channel alert presentation apparatus that simulates a threat in a flight training environment, the apparatus 101 being any device with computing capabilities, such as a personal computer, a server, a smart phone, a game console, an embedded computer, and the like. In the present embodiment, the apparatus 101 is depicted as an on-board embedded computer in simulated flight training, and in other embodiments, the apparatus 101 may be any suitable type of device.

Device 101 may have input and output functionality such as touch, keyboard and mouse or other physical interaction interfaces, gestures, eye movements, voice, and any other configuration that allows communication between device 101 and its user. In addition, the apparatus 101 should have a user head pose capturing function, which can be provided by the apparatus 101 or transmitted to the apparatus 101 after being captured by other devices; device 101 has spatial audio output functionality that may be provided by device 101 or through an external device such as a sound card, headphones, or surround sound that supports virtual surround sound.

The graphical display device 102 may be any head mounted device with display capabilities including, but not limited to: a head-mounted augmented reality or virtual reality screen, a large screen, a spherical projection screen, or the like. Wherein the threat information is presented in a current field of view 103 of the user, the field of view 103 being calculated from a combination of the head pose of the user and the initial field of view. Wherein the icon of the threat 104 is shown outside the field of view, indicating that the threat object is in the upper left position; since the positions of the icons of the threats 104 are obtained by a relative head pose mapping method, the positions where the threats 104 are actually located are not in the positions where the icons of the threats 104 are located, and are indicated by dotted lines. The arrow icon 105 is used to indicate the offset direction and distance of the position in which the threat 104 is actually located relative to the current position. Threat 106 icon presents threats that are within the user's field of view. The icon of the threat 106 represents an airborne threat and the icon borders are drawn in solid lines to indicate that the threat represented by the threat 106 icon is in a corresponding position within the field of view. In addition, the devices 101 will take turns playing audio alerts for out-of-view threats, and in the exemplary scenario shown in fig. 1, virtual surround sound alerts corresponding to

threats

104 and 107 will take turns.

FIG. 2 illustrates an exemplary user initial field of view adjustment interface. Wherein the field of view 103 is determined by the top left icon 201, the top right icon 202 and the bottom icon 203. The user adjusts the positions of the icon 201, the icon 202, and the icon 203 through the input device of the apparatus 101 as described above so that the visual field range 103 coincides with the user's real visual field range.

FIG. 3 illustrates an exemplary process by which threat information is presented to a user by way of a multi-channel alert presentation approach that simulates a threat in a flight training environment. Before turning to the description of fig. 3, it should be noted that the flowchart of fig. 3 is described as a sample with reference to the construction shown in fig. 1, 2, but the process of fig. 3 may be performed in any system and is not limited to the scenario shown in fig. 1, 2. Further, the flow chart in FIG. 3 illustrates an example, the process of which is performed in a particular order as indicated by the arrows, but the stages shown in these figures can be adapted in any order or in any combination or sub-combination.

At 301, a current head pose of the user is obtained; at 302, the head pose information is combined with the initial field of view of the user to obtain a real-time field of view range for the user. At 304, threat alert information (block 303) is divided into two parts, based on the user's real-time field of view (block 302), threat within the field of view (block 304) and threat outside the field of view (block 306), respectively. Threat information in the visual field (block 304), and directly displaying icons in corresponding directions in the visual field range of the user, namely, in-situ push display (block 308); threat information outside the horizon (block 306) is prompted by auditory channels such as spatial audio, vocalization, voice broadcast (block 309), and further visually presented after being mapped into the user's horizon by a mapping method based on the user's head pose (block 307) (block 310).

That is, at block 308, the user can intuitively perceive the situation information within the visual field range without the need of spatial coordinate transformation from the alien perception to the ontology perception, thereby completing the perception and evaluation of the situation with lower cognitive load. After coordinate mapping at block 307, threats outside the user's field of view are mapped to the user's field of view and a push display on the visual channel is performed (block 310), thereby allowing the user to fully utilize the visual channel to perceive threat information outside the user's field of view without performing additional visual searches, thereby reducing the associated burden. To further reduce cognitive load, block 309 presents threat information outside the user's field of view to the user through the auditory channel: the load on the user perception of spatial mapping of the threat outside the field of view in block 307 is compensated for by direct audio rendering.

FIG. 4 shows a multi-channel warning presentation device for simulating threats in a flight training environment according to another embodiment of the invention. The apparatus 400 includes one or more processors 404 and one or more data remembrance components, namely a memory 405. The apparatus 400 may be any device with some computing capability, such as a head-mounted hybrid-implementation all-in-one machine, a situation computer, a mobile notebook, a server, a tablet computer, a set-top box, an embedded computer, a smart mobile phone, and so on. Typical examples of processor(s) 404 are microprocessors, such as those in personal computers, mobile laptops, servers, head-mounted hybrid-implementation kiosk devices. Data storage component(s) 405 are components capable of short-term or long-term storage of data, including, for example, hard disks, removable disks, memory, cache memory, tapes, etc., and are examples of device-scale storage media. The apparatus 400 may include a user input device 407 and an output device 406, such as a wearable mixed reality all-in-one machine, including the user input device 407, such as a gesture, eye movement, voice, and other collection devices, and an augmented reality screen and a dual-channel headset as the output device 406; the apparatus 400 may also be connected to one or more user input devices 407 and output devices 406 by external connection or modification, such as a personal computer connected to a mouse and a keyboard as the user input devices 407, and a head-mounted virtual reality display including a headset as the output device 406. It should be noted that the output device 406 mentioned above is merely an example, and any device or devices capable of display, sound output, etc., including but not limited to headphones, stereos, etc., as well as head-mounted augmented reality displays, display screens, large screens, marquees, etc., are within the scope of the inventive subject matter. Also, the above-mentioned user input devices are merely examples, and any device or devices capable of supporting user input are within the scope of the present subject matter, including but not limited to keyboards, mice, eye trackers, gesture capture devices, wearable physiological information capture devices, myoelectric capture devices, brain-computer interfaces, and the like.

Software computer programs utilized by the apparatus 400 may be stored in the data remembrance component(s) 405 and executed on the processor(s) 404. An example of such software is adaptive threat presentation software 403 that may be implemented in conjunction with some or all of the functionality described above in connection with fig. 1-3, but any type of software could be used.

It should be noted that the interfaces depicted in fig. 1, 2, and the system architecture depicted in fig. 4 are merely examples. Any suitable design can visually, audibly convey the status of the threat in the current situation, and adapt based on the user's head pose. Any such design is within the scope of the subject matter of this disclosure. For example, the interface described in the present invention is applied to a simulated flight training scenario.

Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example for implementing the claims. The protection scope of the present invention shall be subject to the claims.

Claims

1. A multi-channel alarm presenting method for simulating threats in a flight training environment is characterized by comprising the following steps:

calculating the visual field range of the user;

dividing the threat target into a threat target in the user visual field range and a threat target outside the user visual field range according to whether the threat target is positioned in the user visual field range or not;

for threat targets in the user visual field range, pushing and displaying threat information to a user in a user visual field central area;

and for threat targets outside the visual field range of the user, prompting threat information to the user by using an audio-visual dual-channel presentation method.

2. The method of claim 1, wherein the user field of view is calculated based on a current head pose of a user and a field of view.

3. The method of claim 2, wherein the current head pose of the user is obtained by: the method comprises the steps of acquiring 6 degree-of-freedom attitude parameters of a head of a user in a three-dimensional space by using a pose sensor in the head-mounted device, wherein the 6 degree-of-freedom attitude parameters comprise X, Y, Z three-axis positions with the pose sensor as a coordinate origin, a heading angle, a pitch angle and a roll angle.

4. The method of claim 2, wherein the visual range refers to a maximum display range provided by a head mounted display device used by a user.

5. The method of claim 1, wherein the central region of the user's field of view is a circular region defined by a center of the user's field of view and a radius of one fifth of the height of the field of view.

6. The method of claim 1, wherein the audiovisual dual channel presentation method is: the method includes presenting one or more pointers representing the location of the threat target to a user via a head mounted display, and presenting a warning alert tone via a head mounted stereo headset, the warning alert tone characterizing the location of the threat target and threat level information via a sound characteristic.

7. The method of claim 6, wherein for a plurality of threat objects outside the user's field of view, the alert tones of the plurality of threat objects are played in turn by means of virtual surround sound.

8. The method of claim 1, wherein the threat objects within the user's field of view are displayed by solid line icons and at the location of the threat objects.

9. The method of claim 1, wherein threat objects outside the user's field of view are displayed by dashed line icons and the direction and distance of the offset of the orientation in which the threat object is actually located from the current orientation is displayed by arrow icons.

10. A multi-channel warning presentation apparatus simulating threats in a flight training environment, comprising a head pose acquisition device, a user input device, a graphic display device, an audio output device, a memory and a processor, wherein the head pose acquisition device is a head-mounted device with a pose sensor, the graphic display device is a head-mounted display, the audio output device is used for outputting warning sounds, the memory stores a computer program, and the processor implements the steps of the method of claim 1 when executing the computer program.