KR20180134310A - Appratus for controlling integrated supervisory of pilots status and method for guiding task performance ability of pilots using the same - Google Patents

Abstract

According to one aspect of the present invention, a method of deriving a pilot′s mission performance capability performed by an integrated supervisory and control apparatus of a pilot state for evaluating a mission performance capability necessary for piloting a piloted object based on state information of the pilot comprises: a signal measuring step of measuring surrounding environment information in conjunction with a control apparatus of the piloted object and measuring at least one biological signals including an electroencephalography signal, an eye movement signal, a skin electrical activity signal, and a respiration measurement signal through a sensor device mounted on the pilot; an information calculating step of reflecting the pilot′s environment information in real time into an arousal state evaluation model for evaluation of the pilot′s state, analyzing the at least one biological signals to extract individual feature information, inputting the extracted individual feature information into the arousal state evaluation model and calculating convergence feature information through a correlation analysis with the pilot′s arousal state; and an arousal state evaluation and feedback providing step of evaluating the pilot′s real-time arousal state level using the convergence feature information, and providing an arousal deriving feedback signal in accordance with the evaluated arousal state level.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an integrated monitoring and control apparatus for a pilot state, and a method for deriving a mission performance capability of the pilot using the same. 2. Description of the Related Art [0002]

The present invention relates to an integrated monitoring and control apparatus for a pilot state in which a pilot's awakening state is maintained by predicting the possibility of continuous piloting mission by monitoring the arousal state of the pilot reflecting the surrounding environment information affecting the bio signal of the pilot, And a method for deriving the ability of the pilots to carry out their mission.

Recently, as the technology of manned / unmanned airplanes develops, the role of pilots becomes more and more important. However, aircraft pilots are faced with complexity of mission such as 3D space rotation, increase of mission duration due to long flight, strong noise and vibration And fatigue are increasing, and attention is lowered, which is often exposed to environments where it is difficult to concentrate on the main mission, flight.

 Although the market for unmanned aerial vehicles is rapidly growing, the accident rate is 10 to 100 times higher than that of manned aviation equipment, and 38% of the causes of accidents are known to be caused by human factors.

In order to prevent manned / unmanned aircraft accidents due to changes in the perceived state of pilots, the need for objective evaluation of pilots' physical and mental conditions is increasing. Therefore, there is a need for a method of objectively evaluating the pilot's awakening status in order to perform safety tasks and prevent accidents of manned / unmanned aircraft pilots and operators.

Conventionally, the pilot's subjective state and self-report questionnaire were used to evaluate the arousal status of the pilot, which is unreliable for use as an objective indicator due to individual differences, and it is difficult to accurately grasp the arousal status of the pilot. It is also very important to determine whether to start and stop the flight according to the pilot status. However, in the case of the Air Force, it is difficult to decide to cancel the flight according to the subjective judgment of the individual due to the nature of the military field, and there is a problem that the possibility of an accident is high compared to a civil aircraft.

In the past, a method of evaluating the arousal state of a pilot using an objective biochemical test method has been proposed. However, biochemical testing methods using serum, urine, adrenaline, cortisol, etc. disturbs the attention of the pilots and it is impossible to monitor the pilot status in real time during flight.

In order to solve such problems, Korean Patent Registration No. 10-1663922 (Pilot Condition Monitoring System) and Korean Patent Laid-Open No. 10-2017-0000630 (Pilot Biometric Information ) Discloses a technique capable of detecting a waking state during flight using bio-signals of a pilot such as brain waves, respiration, pulse, skin electrical activity, electrocardiogram, and the like.

However, the conventional technique for evaluating a pilot's arousal plaque using bio-signals has a problem in that it is difficult to apply to the actual flying environment because the surrounding environment factors are not taken into consideration for the careless recognition state. In other words, in the actual flying environment, there is a high possibility that bio-signals of different patterns are generated due to weather deterioration such as fog or snow or high altitude. Therefore, when the pilot's awakening state is detected using the existing learned model, The change is not reflected and it becomes difficult to accurately recognize the pilot's condition.

The conventional technique of evaluating the awakening state of the pilot using the multiple bio-signals analyzes the awakening state of the pilot by independently analyzing each bio-signal without considering the correlation between the multiple bio-signals, There is a limit in understanding.

In the case of large aircraft, it is very important to prevent accidents by precisely understanding the state of pilots because large accidents can be caused by the pilot status. Therefore, there is a need for an apparatus that can initially determine whether the condition and mental state of the pilot is capable of carrying out a flight mission and continuously monitor the condition.

Korean Patent No. 10-1663922 (Pilot Condition Monitoring System)

Korean Patent Publication No. 10-2017-0000630 (aviation system for coordinating mission according to pilot bio information)

In order to solve the above problems, an embodiment of the present invention provides a method for evaluating an arousal state of a pilot using a variety of bio-signals of a pilot, And to provide a pilot monitoring system capable of maintaining the alert state of the pilot through sensory stimulation and a method of deriving a pilot's mission performance capability using the same.

It should be understood, however, that the technical scope of the present invention is not limited to the above-described technical problems, and other technical problems may exist.

According to an aspect of the present invention, there is provided a method for deriving a mission performance capability of a pilot according to an aspect of the present invention, including the steps of: A method for deriving a mission performance capability of a pilot performed by an integrated supervisory control device, comprising the steps of: measuring peripheral environment information in connection with a control device of the steering control object; A signal measurement step of measuring at least one or more biological signals including an electrical activity signal and a breathing measurement signal; The control environment information is reflected to an arousal state evaluation model for state evaluation of pilots in real time, and the individual feature information is extracted by analyzing the at least one bio-signal to input the extracted individual feature information into the arousal state evaluation model An information calculating step of calculating convergence feature information through correlation analysis with a pilot's arousal state; And evaluating an awake state level of a real-time pilot using the fusion feature information, and providing an arousal-inducing feedback signal according to the evaluated arousal state level.

 According to another aspect of the present invention, there is provided an integrated monitoring and controlling apparatus for a pilot state, comprising: a memory for storing a program for evaluating and deriving a mission performance capability necessary for adjusting a steering target based on state information of a pilot; And a processor for executing the program, wherein the processor collects steering environment change information in cooperation with the control device of the steering target object by execution of the program, and transmits the EEG signal through the sensor device mounted on the pilot At least one biological signal including at least one biological signal including at least one biological signal, an eye movement signal, a skin electrical activity signal, and a respiration measurement signal, reflects the manipulation environment information in real time to an arousal state evaluation model for evaluating the state of the pilot, Extracting the individual feature information, inputting the extracted individual feature information to the arousal state evaluation model, and calculating the fusion feature information through correlation analysis with the arousal state of the pilot, Evaluating the level of arousal status of the pilot, And provides an arousal-induced feedback signal according to the state level.

According to any one of the above-described objects of the present invention, the awakening state of the pilot can be objectively evaluated by reflecting the surrounding environment information in real time and analyzing the arousal state of the pilot considering the correlation between the multiple bio- The ability to predict performance can be quantitatively predicted to provide feedback appropriate to current pilot awakening conditions.

In addition, the present invention can quantitatively predict the likelihood of a pilot's mission input and the possibility of continuous mission execution, provide feedback necessary to induce arousal while varying intensity and intensity according to the pilot's arousal state level, And can contribute to the efficient operation of the steering body.

FIG. 1 is a block diagram illustrating an integrated monitoring and controlling apparatus for a pilot state according to an embodiment of the present invention. Referring to FIG.
Figure 2 is a block diagram illustrating a processor that is part of Figure 1;
FIG. 3 is a flowchart illustrating a method of deriving a mission performance capability of a pilot according to an exemplary embodiment of the present invention.
Fig. 4 is a diagram for explaining an overall flow for evaluating and inducing the pilot arousal state by Fig. 3; Fig.
FIG. 5 is a graph illustrating a process of generating a variable through the canonical correlation analysis of FIG.
6 is a view for explaining a multi-bio signal characteristic fusion process through canonical correlation analysis of the present invention.
FIG. 7 is a diagram for explaining an arousal state evaluation model according to an embodiment of the present invention.
8 is a diagram illustrating a neurofeedback flow for deriving a pilot awakening state according to an embodiment of the present invention.
9 is a view for explaining a scenario of evaluating a pilot's awakening state using the virtual environment of the head-mounted display device.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings, which will be readily apparent to those skilled in the art. The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. In order to clearly illustrate the present invention, parts not related to the description in the drawings are omitted, and like parts are denoted by similar reference numerals throughout the specification. In the following description with reference to the drawings, the same reference numerals will be used to designate the same names, and the reference numerals are merely for convenience of description, and the concepts, features, and functions Or the effect is not limited to interpretation.

Throughout the specification, when a part is referred to as being "connected" to another part, it includes not only "directly connected" but also "electrically connected" with another part in between . Also, when a component is referred to as "comprising ", it is understood that it may include other components as well as other components, But do not preclude the presence or addition of a feature, a number, a step, an operation, an element, a component, or a combination thereof.

Herein, the term " part " or " module " means a unit realized by hardware or software, a unit realized by using both, and a unit realized by using two or more hardware Or two or more units may be realized by one hardware.

FIG. 1 is a block diagram illustrating an integrated monitoring and controlling apparatus for a pilot state according to an embodiment of the present invention. Referring to FIG.

Referring to FIG. 1, an integrated supervisory control apparatus 100 of a pilot state includes a communication module 110, a memory 120, a processor 130, and a database 140.

The communication module 110 provides a communication interface to a user terminal in cooperation with a communication network. The communication module 110 can receive a data request transmitted from the user terminal and transmit data to the user terminal as a response thereto. Here, the communication module 110 may be a device including hardware and software necessary to transmit / receive a signal such as a control signal or a data signal through a wired / wireless connection with another network device.

The memory 120 records a program for performing a method of deriving a mission performance capability of a pilot. The memory 120 also functions to temporarily or permanently store data processed by the processor 130. Here, the memory 120 may include volatile storage media or non-volatile storage media, but the scope of the present invention is not limited thereto.

Processor 130 controls the entire process of providing a pilot's ability to derive mission performance. Each step performed by the processor 130 will be described later with reference to FIG. 2 and FIG.

Here, the processor 130 may include any kind of device capable of processing data, such as a processor. Herein, the term " processor " may refer to a data processing apparatus embedded in hardware, for example, having a circuit physically structured to perform a function represented by a code or an instruction contained in the program. As an example of the data processing apparatus built in hardware, a microprocessor, a central processing unit (CPU), a processor core, a multiprocessor, an application-specific integrated circuit (ASIC) circuit, and a field programmable gate array (FPGA), but the scope of the present invention is not limited thereto.

Figure 2 is a block diagram illustrating a processor that is part of Figure 1;

2, the processor 130 includes a peripheral environment collecting unit 131, a biological signal collecting unit 132, a feature information extracting unit 133, a signal converging unit 134, an arousal state evaluating unit 135, And a feedback control unit 136.

The surrounding environment collection unit 131 collects surrounding environment information such as light amount information, flight information, and noise information in association with the control device 200 of the steering target object. When the ambient environment information is measured using sensors such as a light amount sensor, a speed sensor, a noise sensor, and the like installed on the steering target object, the surrounding environment detector collects the ambient environment information Environmental information is inputted.

The biological signal collecting unit 132 collects at least one biological signal including an EEG signal, an eye movement signal, a skin electrical activity signal, and a respiration measurement signal through a sensor device 300 mounted on a pilot.

The sensor device 300 may be used for a headset or a headband having a built-in microchip for brain wave measurement for extracting an EEG signal from a channel such as Cz, c3, c4, Oz, 03, A smart band which is a wearable device for measuring an infective sensor, a heart rate, a blood pressure, a pulse, and respiration, and a plurality of sensors for measuring flickering or eye movement components attached to the eyeball in a horizontal or vertical direction around the eyeball .

The EEG signals are electrical signals according to the electrical activity of the synapses. The characteristics of the bands according to the EEG frequency and the characteristics in the time domain are changed according to the user's intention and state. EEG signals are divided into delta wave, theta wave, alpha wave, beta wave and gamma wave depending on the frequency and voltage, and the user's state information is displayed according to each signal range. For example, when measuring the mental cognitive status of a person, bandwidth filtering is applied at 1 to 30 Hz, including delta, theta, alpha, and gamma.

The eye movement signal is a signal generated by the electrical activity of the muscles around the eye during eye movement. It is measured by attaching a plurality of sensors around the eyeball, and in order to eliminate the noise of the signal caused by the movement of the body or the head, Apply bandwidth filtering of ~ 10 Hz. The noise-canceled eye movement signal enables the pilot's eye flicker to be measured according to fatigue.

The feature information extractor 133 selects a variable having a high mutual information amount among a plurality of bio-signals and extracts feature information highly related to the pilot arousal state for each bio-signal.

The signal fusion unit 134 calculates the fusion feature information by combining the extracted individual feature information in consideration of the correlation between the multiple bio-signals.

The arousal state evaluating unit 135 classifies the arousal state of the pilot through the classifier robust to the surrounding environment by reflecting the surrounding environment information in real time.

The feedback control unit 136 provides appropriate feedback according to the evaluated pilot's arousal state level so that the pilot can continuously maintain the optimal arousal state. For example, in order to improve the concentration of pilots with high fatigue, stimulation of awake EEG such as Steady State Visually Evoked Potential (SSVEP) is presented. In addition, the feedback control unit 136 may derive the arousal state of the pilot through the sensory stimulation output device 400 including the neuro feedback system that visualizes the EEG activation for the current arousal state of the pilot, thereby inducing the arousal state of the pilot himself / herself .

The sensory stimulation output device 400 includes a head-mounted display, a head-up mount, a visual output device using an instrument panel of a steering object, etc., When a sense induction feedback signal is inputted, an auditory output device such as a headphone or a speaker of a controlled object that changes a sound intensity according to a pilot's arousal level, Such as smart bands or chairs of the steering object, which include a tactile output device.

Meanwhile, the database 140 stores collected environment information, bio-signals, a pilot condition evaluation result, and the like.

FIG. 3 is a flowchart illustrating a method for deriving a mission performance capability of a pilot according to an embodiment of the present invention. FIG. 4 is a view for explaining an overall flow for evaluating and inducing a pilot awakening state by FIG. Is a graph for explaining the process of generating a variance through the canonical correlation analysis of FIG.

3 and 4, the method of deriving the mission performance capability of the pilot collects various bio-signals such as brain waves, eye movement, skin electrical activity, respiration, etc. through the sensor device 300 attached to the body of the pilot S310).

The processor 130 performs a signal preprocessing process for removing noise components due to head and body movements in the collected bio-signals, and analyzes the bio-signals that have undergone the signal preprocessing process so as to best represent the arousal state of the pilot Information is extracted (S320).

The processor 130 determines a time-frequency component of the noise signal due to the movement of the muscles caused by the movement of the head and the body, linearly separates other biological signals including the eye movement signal and the captured signal, Remove.

The processor 130 analyzes the biological signals from which the captured signals are removed in the time-space frequency domain. The physiological signals such as heart rate, blood pressure, and pulse can be quantitatively evaluated, and the dermatological activity signal, the eye movement signal, Generates a moving window of about 1500 ms and then moves the moving window in real time and extracts an amplitude value for each channel as characteristic information within each section.

According to the level of the arousal state of the pilot, the changes between the biological signals have a close correlation. Accordingly, the processor 130 performs canonical correlation analysis to measure correlation coefficients between multivariate measures, and uses characteristic correlation methods to maximize the correlation coefficient between multivariate values using canonical correlation analysis, (S330). ≪ / RTI >

In general, the correlation analysis method quantifies the relationship between the variables and the variance. When the correlation is analyzed through the correlation analysis between the specific bio-signal (variance X) and the pilot's arousal state (variance Y) It is possible to predict how the other variable Y changes as it changes.

For example, since heart rate and pilot fatigue state are negatively correlated, fatigue is reduced when the pilot's heart rate rises. Pilot fatigue is reduced when the heart rate rises based on a heart rate of 80 bpm (bit per minute), and fatigue is increased when the heart rate is lowered on the basis of 80 bpm.

Canonical Correlation Analysis (CCA) can use the correlation coefficient of the linear combination of variables belonging to each group of variables to determine the association between groups of variables so that it is possible to maximize the correlation between variance X and variance Y . That is, the variance U and the variance V capable of maximizing the relationship between the variance X and the variance Y are generated and expressed as U = aX, V = bY. Here, a and b represent weights that maximize the correlation between the linear combinations.

6 is a view for explaining a multi-bio signal characteristic fusion process through canonical correlation analysis of the present invention.

Referring to FIG. 6, X = {x1, x2, x3, ... , xn}, Y = {y1, y2, t3, ... , ym}, as shown in FIG. 5, a = {a1, a2, a3, ... , an}, b = {b1, b2, b3, ... , bm}, so U = aX = {(a1, x1), (a2, x2), (a3, x3), ... , (an, xn)}, V = bY = {(b1, y1), (b2, y2), (b3, , (bn, ym)}. Here, x1, x2, ... , xn is a biological signal such as heart rate information, EEG feature information, respiration information, and skin electrical activity, and y1, y2, ... , and ym can be defined as a pilot's awakening status level or awakening status level.

When recognizing a specific awakening state, the degree of contribution of each bio-signal to the awakening state may be different. However, if correlation analysis is performed using a simple linear combination or the like, the characteristic information of the multi- The overall evaluation performance may be degraded.

Accordingly, the processor 130 reflects the bio-signal having a high correlation with a specific arousal state in the arousal state evaluation model in consideration of the relationship between the multiple bio-signals and the specific arousal state.

For example, when estimating the fatigue (Karolinska Sleepiness Scale, KSS) of pilot awakening with EEG Power Spectrum Density (PSD), Galvanic Skin Response (GSR) and inspiratory volume (RESP), PSD 120 dB, GSR 100 μS, KSS was 100 for 100 mL, PSD was 300 dB, GSR was -200 μS, and volume was 200 mL. KSS was 9 for PSD of 200 dB, GSR of -250 μS and respiration of 150 mL.

PSD 250 dB, GSR 200 μS, and respiratory volume 160 mL. When all the multiple bio-signals are linearly combined, the performance of the arousal state evaluation due to the GSR, which is presumed not to be correlated with the KSS, There is a possibility of degradation.

Table 1 shows VEC_n (w_1, w_2, w_3) weighting vectors for predicting KSS values according to weight vectors, and weight vectors with high weights of EEG correlated with KSS are Vec_n = PSD * w_1 + GSR * w_2 + RESP * w_3.

[Table 1]

As shown in Table 1, in the case of Vec_3, the sum is linearly increased according to the KSS value. Thus, as the weights of the EEGs, which are highly correlated with the KSS, are higher than when the weights are constant for each variable, the correlation of the summations increases. In addition to the canonical correlation analysis method, the processor 130 can maximize the relationship between the variables by using methods for calculating weights.

Referring again to FIG. 3, since the bio-signal of the pilot itself may vary greatly depending on the surrounding environment, the processor 130 quantitatively measures the surrounding environment information that affects the bio-signal and reflects the measured information in the arousal state evaluation model S340).

In other words, a bio-signal generated at night where the peripheral vision is darkened due to a decrease in the light amount and a bright environment surrounding it may cause a large difference due to various environmental factors. Accordingly, the processor 130 collects the light amount information measured through the optical sensor, the flight information measured through the speed sensor, and the like, and calculates the peripheral environment information so that the performance deviation of the arousal state evaluation model can be minimized Is input as a bias of the arousal state evaluation model.

FIG. 7 is a diagram for explaining an arousal state evaluation model according to an embodiment of the present invention.

Referring to FIG. 7, the arousal state evaluation model is a multilayer artificial neural network structure, and the multilayer artificial neural network is a network of neurons composed of an input layer, a hidden layer, and an output layer. In this arousal state evaluation model, a process in which a value obtained by multiplying an input signal of each neuron by a weight is transmitted to a next neuron is repeated, and a final output value is given.

The arousal state evaluation model is a Recurrent Convolutional Neural Network (RCNN) model, which uses the following Equation 1, and the surrounding information is input to the bias as shown in Equation (2) to be a classifier robust to changes in the surrounding environment do.

[Equation 1]

&Quot; (2) "

This arousal state estimation model reflects the contextual information by inserting a circulating neural network structure between convolutional layers so as to detect signal changes over time between spatial feature extraction.

Referring again to FIG. 3, the processor 130 evaluates the awake state level of the real-time pilot by analyzing the arousal state through the fusion feature information in which the feature information of various bio-signals are fused using the arousal state evaluation model (S350) . The processor 130 evaluates the arousal state of the pilot at intervals of several ms using an Area Under Curve (AUC) score, and calculates the individual evaluation accuracy by statistical techniques such as t-test and p-value.

Pilot arousal status includes, but is not limited to, attention, fatigue, mental workload, and unconsciousness. Here, fatigue affects the performance of flight tasks by directly inducing physical condition changes such as drowsiness, and it can be measured and evaluated by using the speed of cognitive response to specific situations and subjective indicators. Attention can be measured and evaluated by using other factors besides flight mission, such as radio communication or response to specific visual stimuli, and mental workloads can be measured and evaluated by aircraft Which can be measured and evaluated by using the difficulty level and accuracy of the instrument panel.

In this way, the processor can set the awake state level suitable for the mission performance related to the steering object step by step based on objective indicators including the results of correlation analysis and subjective indicators including the results of fatigue measurement or questionnaire results.

In accordance with the evaluation result of the awakening state, the processor 130 determines that the pilot is in a state suitable for performing the mission when the awake state of the current pilot is high, and when the awake state of the pilot is low, And outputs an induced feedback signal to increase the arousal state of the pilot (S360, S370).

For example, the processor 130 generates a sensory-induced feedback signal to increase fatigue for a long period of time, such as flying, and to induce an arousal state of the pilot with reduced concentration. At this time, the processor 130 includes an awake stimulation stimulus such as a State Visually Evoked Potential (SSVEP), a Steady State Auditory Evoked Potential (SSAEP), and a Steady State Somatosensory Evoked Potential (SSSEP). The neurofeedback system includes a system for training the pilot to control the brain-wave generating ability by himself in real time while confirming the degree of EEG activation related to the degree of awakening.

8 is a diagram illustrating a neurofeedback flow for deriving a pilot awakening state according to an embodiment of the present invention.

As shown in FIG. 8, the processor transmits the sensed feedback signal to the neurofeedback system to induce the arousal of the pilot and acquires a real-time EEG signal (S810 and S820). The processor 130 measures the EEG signals in theta and alpha bands of the frontal lobe and the occiput lobe to extract the feature values and then inputs the extracted feature values to the learned classifier through the arousal state evaluation model S830, S840).

The processor 130 acquires the classification result in the classifier (S850), visualizes and outputs the current arousal level of the pilot, that is, the concentration level of the pilot, and controls the pilot himself to adjust the brain waves in operation S860. The processor 130 generates a sensed feedback signal for inducing the awake state of the real-time pilot according to the current state of the pilot's arousal state, and transmits the sensed feedback signal to the neuro feedback system.

In addition to the neuro feedback system, the sensory stimulation output device 400 can provide visual stimulation within a range that does not interfere with the pilot's mission performance by using a wearable device such as a smart watch or a display means within the controlled object. In addition, the sensory stimulation output apparatus 400 can provide stimulation in various forms such as auditory stimulation such as a warning sound, tactile stimulation such as vibration, and the like.

The processor 130 may provide a personalized optimizer classifier for the pilot's awakening state using data stored in the database 140 and may provide an improved classifier in the new pilot. At this time, the database () stores a plurality of bio-signals, individual feature information of bio-signals, fusion characteristic information, a level of arousal state, various variables during a feedback control process, and the like.

Pilot arousal status can be evaluated in a virtual environment other than actual flight, ie, a simulation environment, a head mount display, or a head up mount enhancement / virtual reality environment.

9 is a view for explaining a scenario of evaluating a pilot's awakening state using the virtual environment of the head-mounted display device.

As shown in FIG. 9, the awakening state of the pilot can be evaluated by using a head mount display device capable of constructing a visual environment similar to an actual flying environment anytime and anywhere.

9 (a) is a screen recognized by the pilot, and FIG. 9 (b) is a screen provided to the user by the head-mounted display device, and the position of the pilot eye is currently adjusted / State information, and the awakening state can be evaluated using various bio-signals and surrounding information of the pilot based on the current adjustment / operation state information.

 As described above, the present invention can be utilized as a method for objectively evaluating the arousal state of pilots of various maneuvering objects such as public transportation, automobiles, etc. in addition to manned / unmanned airplanes. In addition, it is possible to perform task scheduling based on the evaluation result of the arousal state of the pilot, and it is possible to improve the mission performance efficiency and the success rate through the optimal task assignments for each pilot.

The method of deriving the mission performance capability of the pilot according to the embodiment of the present invention described above may be implemented in the form of a recording medium including instructions executable by a computer such as a program module executed by a computer. Such a recording medium includes a computer-readable medium, which may be any available medium that can be accessed by a computer, and includes both volatile and non-volatile media, both removable and non-removable media. The computer-readable medium may also include computer storage media, which may be volatile and non-volatile, implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data , Both removable and non-removable media.

It will be understood by those of ordinary skill in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present invention as defined by the following claims. will be. It is therefore to be understood that the above-described embodiments are illustrative in all aspects and not restrictive. For example, each component described as a single entity may be distributed and implemented, and components described as being distributed may also be implemented in a combined form.

Furthermore, while the methods and systems of the present invention have been described in terms of specific embodiments, some or all of those elements or operations may be implemented using a computer system having a general purpose hardware architecture.

It is intended that the present invention covers the modifications and variations of this invention provided they come within the scope of the appended claims and their equivalents. .

100: Integrated supervisory control device of pilot state 110: Communication module
120: memory 130: processor
131: Ambient environment information collecting unit 132: Biological signal collecting unit
133: Feature information collecting unit 134: Signal fusion unit
135: Arousal state evaluation unit 136: Feedback control unit
200: Control device of the steering object 300: Sensor device
400: Sensory stimulation output device

Claims (14)

A method of inducing a pilot's mission performance performed by an integrated supervisory control apparatus in a pilot state for evaluating a mission performance capability necessary for coordination of a pilot object based on state information of the pilot,
A signal for measuring at least one living body signal including an EEG signal, an eye movement signal, a skin electric activity signal, and a respiration measurement signal through a sensor device mounted on the pilot, Measuring step;
The control environment information is reflected to an arousal state evaluation model for state evaluation of pilots in real time, and the individual feature information is extracted by analyzing the at least one bio-signal to input the extracted individual feature information into the arousal state evaluation model An information calculating step of calculating convergence feature information through correlation analysis with a pilot's arousal state; And
Evaluating an awake state level of a real-time pilot using the convergence feature information, and providing an arousal-inducing feedback signal in accordance with the evaluated arousal state level, the method comprising: Induction method. The method according to claim 1,
Wherein the information calculating step defines the weight of the awake state evaluation model as a weight and a bias form, inputs the steering environment information as a bias, and defines a degree of contribution to the arousal state of the pilot according to the bio- A method of deriving pilots' ability to perform mission. 3. The method of claim 2,
The information calculating step may include:
Wherein the at least one biometric signal is set as a variable X and the arousal state of the pilot is set as a variable Y. The correlation between the variance X and the variance Y is calculated using a canonical correlation analysis using correlation coefficients of linear combinations of variables belonging to each variable group Canonical correlation analysis) to calculate the mission performance capability of the pilot. The method of claim 3,
Wherein the information calculating step sets the weights a and b to maximize the correlation between the linear combinations of the variance X and the variance Y to produce a variance of U = aX and V = The method of deriving the task performance ability of. The method according to claim 1,
The arousal state evaluation and feedback providing step may include setting an awake state level suitable for the mission performance ability related to the steered object based on an objective indicator including the analysis result of the correlation and a subjective indicator including a fatigue measurement result or a survey result A method of deriving a pilot's mission performance ability. The method according to claim 1,
The aural condition evaluation and feedback providing step may include a step of, when the arousal induction feedback signal is generated according to the arousal state level, the bio-signal having a high correlation with the correlation coefficient generated by the correlation analysis result and the arousal state of the pilot, Wherein the pilot is generated by the pilot. The method according to claim 1,
Wherein the arousal state evaluation and feedback providing step includes:
Wherein the awakening state is derived from the arousal state of the pilot through the sensory stimulation output device that is operated by the arousal induction feedback signal and induces the arousal state of the pilot. 8. The method of claim 7,
Wherein the awakening induction feedback signal is a simultaneous output or a sequential output of the sensory stimulation including tactile stimulation, auditory stimulation, and visual stimulation of the sensory stimulation output device according to a feedback reaction of the pilot. 9. The method of claim 8,
Wherein the arousal induction feedback step signal is a step of outputting a stimulus output in the order of tactile stimulation, auditory stimulation, and visual stimulation by setting different intensity and time differences according to the arousal state level of the pilot. A memory in which a program for evaluating and guiding mission performance performance necessary for adjustment of a steering target object based on state information of a pilot is recorded; And
And a processor for executing the program,
The processor, by executing the program,
Collecting the steering environment change information in association with the control device of the steering object and collecting at least one or more biological signals including an EEG signal, an eye movement signal, a skin electrical activity signal, and a respiration measurement signal through a sensor device mounted on the pilot The control environment information is reflected to an arousal state evaluation model for evaluating the state of a pilot in real time, and the individual feature information is extracted by analyzing the at least one bio-signal to input the extracted individual feature information into the arousal state evaluation model And the augmented state of the pilot is evaluated by using the convergence feature information, and an arousal-induced feedback signal according to the evaluated arousal state level is provided The integrated monitoring and control device of the pilot state, 11. The method of claim 10,
The processor comprising:
A peripheral environment information collecting unit for collecting peripheral environment information including light amount information and flight information by communicating with the control device of the steering object;
A biological signal collecting unit for collecting at least one biological signal including an EEG signal, an eye movement signal, a skin electrical activity signal, and a respiration measurement signal through a sensor device mounted on the pilot;
A feature information extracting unit for extracting feature information for each bio-signal by analyzing the noise components of the at least one bio-signal in a time-space frequency domain;
A signal fusion unit for combining the extracted individual feature information into fusion feature information in consideration of a correlation between multiple bio-signals;
An arousal state evaluating unit for evaluating the arousal state level of the pilot using the fusion feature information; And
And a feedback control unit for generating and outputting a sensed feedback signal for providing sensory stimulation including visual stimulation, auditory stimulation, and tactile stimulation in accordance with the assessed pilot's aural condition level. 12. The method of claim 11,
Wherein the signal fusion unit sets the at least one or more biological signals as the variable X and the arousal state of the pilot as the variable Y and uses the correlation between the variance X and the variance Y as the correlation coefficient of the linear combination of the variables belonging to each variable group And performing a canonical correlation analysis to calculate convergence feature information. 12. The method of claim 11,
Wherein the arousal state evaluating unit classifies the arousal state level in real time through the arousal state evaluation model and uses an Area Under Curve (AUC) score in consideration of the sensitivity and the specificity as an evaluation scale, . 12. The method of claim 11,
Further comprising a sensory stimulation output device for simultaneously outputting or sequentially outputting sensory stimulation including tactile stimulation, auditory stimulation, and visual stimulation in accordance with the sensory feedback feedback signal.

Images