CN114327046A - Multi-mode man-machine interaction and state intelligent early warning method, device and system - Google Patents
Multi-mode man-machine interaction and state intelligent early warning method, device and system Download PDFInfo
- Publication number
- CN114327046A CN114327046A CN202111452903.XA CN202111452903A CN114327046A CN 114327046 A CN114327046 A CN 114327046A CN 202111452903 A CN202111452903 A CN 202111452903A CN 114327046 A CN114327046 A CN 114327046A
- Authority
- CN
- China
- Prior art keywords
- state
- display item
- early warning
- human
- hand
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Abstract
The method comprises a multi-channel man-machine interaction process, particularly a state intelligent early warning process, and can acquire physical and mental state index data of an operator from multi-modal acquisition equipment in real time; inputting the body and mind state index data acquired from different acquisition devices into corresponding prediction models to obtain the state stage of corresponding prediction targets; and triggering corresponding early warning operation according to the state stage of each predicted target. The method enriches the non-contact instruction input mode of natural human-computer interaction, can effectively reduce human errors in the human-computer interaction process, improves the safety, and simultaneously improves the system interaction performance, the human-computer matching capability and the human-computer interaction efficiency.
Description
Technical Field
The application relates to the technical field of human-computer interaction, in particular to a method, a device and a system for multi-mode human-computer interaction and intelligent state early warning.
Background
The development trend of man-machine interaction is to enable an operator to get rid of any form of interactive interface to the greatest extent, the mode of inputting information becomes simpler and more random, the requirements of the operator can be captured visually, directly and comprehensively by means of the fusion of artificial intelligence and big data, and the execution and feedback can be smoothly carried out according to the intention of the operator.
At present, human-computer interaction includes touch instruction input through mouse click buttons or touch screens and non-contact instruction input through gestures, voice and the like, and corresponding operations are performed on 2D or 3D display items in a screen through the instruction input; compared with partial energy of the operating personnel dispersed by contact type instruction input, the hands are not only liberated by non-contact type instruction input, interaction convenience is improved, and the natural interaction mode of instruction input through voice, gestures and the like of a person is collected, so that the interaction experience feeling of the intelligent device can be improved.
However, as the system performance is intelligentized and complicated, the requirement for optimal human-computer matching is higher and higher, and the current natural interaction non-contact mode has a single instruction input mode, the human-computer interaction efficiency needs to be further improved.
Disclosure of Invention
In order to enrich the non-contact mode instruction input mode and improve the user experience, the application provides a method, a device and a system for multi-mode man-machine interaction and intelligent state early warning.
In a first aspect, the method for multimodal human-computer interaction and intelligent state early warning provided by the application adopts the following technical scheme:
a multi-mode man-machine interaction and state intelligent early warning method executes any one or multiple man-machine interaction flows synchronously, wherein the man-machine interaction flows comprise a brain-machine interaction flow, an eye movement interaction flow, a myoelectricity interaction flow, a gesture interaction flow, a voice interaction flow, an emotion interaction flow and a state intelligent early warning flow; wherein:
the eye movement interaction process comprises the following steps: determining a corresponding operation display item according to the eye movement data acquired in real time, and triggering a corresponding instruction for operating the operation display item;
the gesture interaction process comprises the following steps: judging a current gesture according to the gesture parameters acquired in real time, and triggering a corresponding instruction according to a gesture judgment result;
the state intelligent early warning process comprises the following steps:
acquiring physical and mental state index data of an operator from multi-modal acquisition equipment in real time;
obtaining a state stage corresponding to a prediction target according to the body and mind state index data acquired from different acquisition equipment;
and triggering corresponding early warning operation according to the state stage of each predicted target.
Through adopting above-mentioned technical scheme, can receive the action and the physical and mental state index data of the operating personnel who gathers through different equipment simultaneously, and through obtaining corresponding instruction or triggering corresponding incident to action and physical and mental state index data analysis, watch the discernment through eye movement, gesture identification promotes traditional mouse control, and combine physical and mental state index data feedback early warning real-time feedback operating personnel physical and mental state not only realize carrying out human-computer interaction through operating personnel physical and mental state, and the authenticity and the experience that reach the experiment that can be better are felt, thereby will control personnel's experience sense datamation.
Optionally, the determining, according to the eye movement data obtained in real time, a corresponding operation display item, and triggering a corresponding instruction for operating the operation display item, includes:
acquiring eye movement data in real time; the eye movement data comprise a first coordinate of the eye fixation position of the user on the eye tracker and a corresponding fixation effective value;
converting the first coordinate obtained each time into a second coordinate of a first screen where an operation display item is located, and searching a corresponding operation display item on the first screen according to the second coordinate;
judging the operation effectiveness of the operation display item according to the effective value; and triggering a corresponding instruction for operating the operation display item when the operation validity is valid.
Optionally, the finding a corresponding operation display item on the first screen according to the second coordinate includes:
acquiring coordinates of each display item on the first screen;
judging whether the second coordinate is in the first area of the display item, and if the second coordinate is in the first area of the display item, judging the display item as a corresponding operation display item;
the first area is an area which takes the coordinate of the display item as the center of a circle and takes a preset first value as a radius.
Optionally, the valid value is a duration of gazing at the first coordinate position;
the judging the operation validity of the operation display item according to the valid value comprises the following steps:
judging whether the effective value is larger than a preset threshold value or not, and if so, judging that the operation effectiveness of the operation display item is effective; and if the operation validity of the operation display item is less than or equal to the preset threshold, the operation validity of the operation display item is invalid.
Optionally, the gesture parameters include:
IsLeft, if the left-hand parameter is in an effective state, the left hand is used for making the current gesture;
isright, if the right-hand parameter is in a valid state, it is the right hand that makes the current gesture;
the sample grabbing strength parameter hand, GrabStrength, is between 0 and 1; when the sample grabbing strength parameter is 0, the hand state is a fully unfolded state; when the sample grabbing strength parameter is 1, the hand state is a fist making state;
palm normal, for judging the palm center vertical direction, wherein the palm center vertical direction parameter is an abscissa X, which represents that the palm center vertical direction is horizontal; the palm center vertical direction parameter is a vertical coordinate Y which represents that the palm center vertical direction is vertical;
palm velocity, the hand movement direction parameter being abscissa X, which represents the hand movement direction as left and right, and the hand movement direction parameter being abscissa Y, which represents the hand movement direction as up and down.
Optionally, the determining a current gesture according to the gesture parameter and triggering a corresponding instruction according to a gesture determination result includes:
judging whether the current gesture is made by the left hand or the right hand according to the left hand parameter and the right hand parameter to obtain a first result;
judging the hand state according to the sample grabbing strength parameter to obtain a second result;
judging the palm center vertical direction according to the palm center vertical direction parameters to obtain a third result;
judging the hand moving direction according to the hand moving direction parameter to obtain a fourth result;
obtaining a current gesture according to the first result, the second result, the third result and the fourth result;
and triggering an instruction corresponding to the current gesture according to the current gesture.
Optionally, the obtaining the state phase of the corresponding prediction target according to the physical and mental state index data obtained from different acquisition devices includes:
and inputting the body and mind state index data acquired from different acquisition equipment into corresponding prediction models to obtain the state stage of the corresponding prediction target.
Optionally, the training method of the prediction model includes the following steps:
collecting a sample data set of the prediction target; the sample data set comprises a plurality of sample data, and each sample data comprises the body and mind state index data corresponding to a person in different state stages of the prediction target;
classifying each sample data in the sample data set according to the state stage, and respectively labeling;
dividing the classified and labeled sample data set into a training set and a test set;
training the constructed prediction model through the training set, and obtaining a final prediction model after iteration of a preset training period;
inputting each sample data in the test set into the trained prediction model in sequence to obtain a corresponding test result; the test result comprises a state stage of output of the prediction model after each sample data in the test set is input into the trained prediction model;
calculating the evaluation index of the prediction model after the training according to the test result; and finishing training when the evaluation index meets the requirement.
Optionally, the obtaining the state phase of the corresponding prediction target according to the physical and mental state index data obtained from different acquisition devices includes:
judging the relation between the body and mind state index data acquired by the corresponding acquisition equipment and the corresponding state preset value;
if the physical and mental state index data are all larger than the state preset value, the state stage corresponding to the predicted target is a first state stage;
if the physical and mental state index data are all smaller than the state preset value, the state stage corresponding to the predicted target is a second state stage;
and if the physical and mental state index data are not completely larger or not smaller than the state preset value, the state stage corresponding to the prediction target is a third state stage.
Optionally, the early warning operation includes:
outputting the early warning state of the state stage in a mode of displaying different state stages by frames with different colors;
outputting the early warning state of the state stage in a mode of playing different state stages by voice;
and outputting the early warning state of the state stage by sending an early warning message.
In a second aspect, the application provides a device for multimodal human-computer interaction and intelligent state early warning, which adopts the following technical scheme.
A multi-modal human-computer interaction and state intelligent early warning device comprises: the system comprises a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein the processor executes the program to realize the method for multimodal human-computer interaction and intelligent state early warning according to the first aspect.
By adopting the technical scheme, the behavior and the physical and mental state index data of the operator collected by different devices can be received at the same time, the corresponding instruction is obtained or the corresponding event is triggered by analyzing the behavior and the physical and mental state index data, especially, the physical and mental state of the operator is fed back by combining the feedback and early warning of the physical and mental state index data, the human-computer interaction is realized through the physical and mental state of the operator, the authenticity and the experience feeling of the experiment can be better achieved, and the experience feeling of an operator is digitalized.
In a third aspect, the application provides a system for multimodal human-computer interaction and intelligent state early warning, which adopts the following technical scheme.
A multi-mode human-computer interaction and state intelligent early warning system comprises a multi-mode acquisition device and a multi-mode human-computer interaction and state intelligent early warning device in the second aspect;
the multi-mode acquisition device acquires human-computer interaction data of an operator; the human-computer interaction data comprises cranial nerve signals, eye movement data, myoelectric signals, gesture parameters, voice signals, facial expression data and physical and mental state index data;
the multi-mode human-computer interaction and state intelligent early warning device acquires the human-computer interaction data from the multi-mode acquisition device and executes a corresponding human-computer interaction process by utilizing the human-computer interaction data; the human-computer interaction flow comprises a brain-computer interaction flow, an eye movement interaction flow, a myoelectricity interaction flow, a gesture interaction flow, a voice interaction flow, an emotion interaction flow and a state intelligent early warning flow.
Through adopting above-mentioned technical scheme, can carry out human-computer interaction through multiple mode, and receive the action and the physical and mental state index data of the operating personnel who gathers through different equipment simultaneously, and through obtaining corresponding instruction or triggering corresponding incident to action and physical and mental state index data analysis, especially combine the real-time feedback operating personnel physical and mental state of body and mental state index data feedback early warning real-time feedback operating personnel state, not only realize carrying out human-computer interaction through operating personnel state of mind, and the authenticity and the experience that reach the experiment that can be better are felt, thereby will control personnel's experience sense datamation.
In summary, the present application includes at least one of the following beneficial technical effects:
1. meanwhile, behavior and physical and mental state index data of an operator, which are acquired through different devices, are received, corresponding instructions are obtained through analyzing the behavior and physical and mental state index data or corresponding events are triggered, the traditional mouse control is improved through eye movement fixation recognition and gesture recognition, and the physical and mental state of the operator is fed back in real time through feedback and early warning of the physical and mental state index data, so that not only is human-computer interaction realized through the physical and mental state of the operator, but also the authenticity and experience of an experiment can be better achieved, and the experience of an operator is digitalized;
2. the attention point of the gaze is detected in real time through eye movement fixation identification, the instruction is input through eye movement, the purpose of freeing both hands is achieved, and the corresponding instruction of the operation display item is triggered and operated only when the operation effectiveness is effective when man-machine interaction is carried out through eye movement, so that instruction misjudgment is avoided, the reliability of non-contact instruction input is improved, and the user experience is improved;
3. when the command is input through the gesture, the current gesture is judged through a plurality of results obtained by different hand parameters, so that the accuracy of gesture recognition is effectively improved, recognition errors are reduced, command misjudgment is avoided, and a corresponding event is ensured to be triggered.
Drawings
Fig. 1 is a flowchart of a method for multimodal human-computer interaction and intelligent early warning of states according to an embodiment of the present application;
fig. 2 is a flowchart of step S201 provided in an embodiment of the present application;
fig. 3 is a block diagram of a multi-modal human-computer interaction and state intelligent warning apparatus provided in the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is further described in detail below with reference to fig. 1-3 and the embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.
The method and the device realize multisource heterogeneous human-computer interaction, can not only input instructions through a traditional mouse keyboard, but also simultaneously receive behavior and physical and mental state index data of operators acquired through different devices, and obtain corresponding instructions or trigger corresponding events through analyzing the behavior and physical and mental state index data, for example, the operation function of clicking events is finished through watching targets and gestures by eyes, the original mouse click is replaced, and the corresponding target function is controlled by assisting a voice command to replace the original keyboard control; the body and mind state index data real-time feedback function not only realizes human-computer interaction through the body and mind state of an operator, but also can display the heart rate state and the tension degree of the operator on an interface through corresponding visual colors, numerical values or other modes, so that the experience feeling of the human-computer interaction operator is visualized in a datamation mode.
The application discloses a multi-mode man-machine interaction and state intelligent early warning method, which comprises but is not limited to the following man-machine interaction processes:
a voice interaction process; touch interactive process; a gesture interaction process; a brain-computer interaction flow; eye movement interaction flow; an emotion interaction flow; myoelectricity interaction flow;
each man-machine interaction process can be executed independently, and various man-machine interaction processes can be executed synchronously.
Referring to fig. 1, as an embodiment of a method for multimodal human-computer interaction and intelligent early warning of states, the method includes the following steps:
step S100, starting a program, and then simultaneously executing a personal human-computer interaction process, step S200 (eye movement interaction process), step S300 (gesture interaction process), step S400 (voice interaction process), step S500 (state intelligent early warning process, i.e. process for performing human-computer interaction through body and mind state index data), and step S600 (other human-computer interaction mode reserved ports).
Step S200, acquiring eye movement data from eye movement detection equipment (such as a Tobi eye movement instrument) in real time; the eye movement data comprise a first coordinate of the eye fixation position of the user on the eye tracker and a corresponding fixation effective value; then step S201 is performed.
In this embodiment, the effective value is the duration of the gaze first coordinate location; that is, the time during which the first coordinate position is not changed, in order to improve the reliability of the eye movement input command in the present embodiment, the judging the validity of the operation on the operation display item according to the valid value includes:
judging whether the effective value is greater than a preset threshold (generally 1.5S), if so, judging that the operation effectiveness of the operation display item is effective; and if the operation validity of the operation display item is less than or equal to the preset threshold, the operation validity of the operation display item is invalid.
Step S201, determining a corresponding operation display item according to the eye movement data acquired in real time, and triggering a corresponding instruction of the operation display item.
Step S300, acquiring gesture parameters from a gesture recognition device (such as leapfunction gesture recognition) in real time, and then executing step S301;
and S301, judging the current gesture according to the gesture parameters, and triggering a corresponding instruction according to the gesture judgment result. In this embodiment, the gesture parameters include:
IsLeft is the left hand parameter, if the left hand parameter is in the valid state, the left hand is the left hand for making the current gesture; isright, if the right-hand parameter is in an active state, it is the right hand that makes the current gesture; judging whether the current gesture is made by the left hand or the right hand according to the left hand parameter and the right hand parameter to obtain a first result;
the sample grabbing strength parameter hand, GrabStrength, is between 0 and 1; when the sample grabbing strength parameter is 0, the hand state is a fully unfolded state; when the sample grabbing strength parameter is 1, the hand state is a fist making state; judging the hand state according to the sample grabbing strength parameter to obtain a second result;
palm normal, for judging the palm center vertical direction, wherein the palm center vertical direction parameter is an abscissa X, which represents that the palm center vertical direction is horizontal; the palm center vertical direction parameter is a vertical coordinate Y which represents that the palm center vertical direction is vertical; judging the palm center vertical direction according to the palm center vertical direction parameters to obtain a third result;
palm velocity for determining a hand movement direction, the hand movement direction parameter being an abscissa X representing that the hand movement direction is left and right, the hand movement direction parameter being an abscissa Y representing that the hand movement direction is up and down; judging the hand moving direction according to the hand moving direction parameter to obtain a fourth result;
obtaining a current gesture according to the first result, the second result, the third result and the fourth result; and then triggering an instruction corresponding to the current gesture according to the current gesture, and judging the current gesture according to a plurality of results obtained by different hand parameters, so that instruction misjudgment can be avoided, the reliability of non-contact instruction input is improved, accurate triggering of a corresponding event is ensured, and the user experience is improved.
In this embodiment, the current gesture obtained according to the first result, the second result, the third result and the fourth result includes, but is not limited to, the following:
the vertical direction of the palm center is horizontal when the fist is held by the right hand and moved left and right;
the vertical direction of the palm center is vertical when the fist is held by the right hand and moved left and right;
the vertical direction of the palm center of the hand-held fist at the right side is horizontal;
the vertical direction of the palm center of the hand-held fist is vertical;
the right hand is unfolded, moves left and right, and the vertical direction of the palm center is horizontal;
the vertical direction of the palm center is vertical when the right hand is unfolded, moves left and right;
the right hand is unfolded to move up and down, and the palm center is horizontal in the vertical direction;
the right hand is unfolded, moves up and down, and the vertical direction of the palm center is vertical;
the left and right movement of the left fist and the right palm center is horizontal in the vertical direction;
the vertical direction of the left-right moving palm center of the left-hand fist is vertical;
the left hand is clenched to move the palm center up and down, and the vertical direction is horizontal;
the vertical direction of the palm center of the left hand clenching the fist to move up and down is vertical;
the left hand is unfolded, the left and right hands move, and the palm center is horizontal in the vertical direction;
the vertical direction of the palm center is vertical when the left hand is unfolded, the left hand is moved, the right hand is moved and the palm center is moved;
the left hand is unfolded to move the palm center up and down, and the vertical direction is horizontal;
the left hand is unfolded, the vertical direction of the palm center is vertical when the palm center moves up and down.
Triggering an instruction corresponding to the current gesture according to the current gesture, for example, controlling the rotation of the operation display item through the left and right movement of a fist held by a right hand; the palm of the right hand is unfolded to move up and down to control the operation display item to move up and down; the palm of the right hand is unfolded to control the operation display item to move left and right; controlling the zooming function of the topographic map by making a fist and extending the palm of the left hand, amplifying the 2D map when detecting that the left hand is unfolded, and shrinking the 2D map when making the fist; the vertical direction of the palm center of the right hand is horizontal, the left-right movement controls the left-right movement of the 2D map, the vertical direction of the palm center is vertical, and the up-down movement controls the up-down movement of the 2D map; and the left and right movement under the fist state of the right hand controls the 3D camera to rotate left and right.
Step S400, detecting a voice signal sent by radio equipment in real time, and then executing step S401;
step S401, analyzing the voice signal, and judging whether the voice signal is a voice command (identified by a keyword); if the voice signal is a voice command, executing step S402; otherwise, step S400 is executed.
Step S402, obtaining a corresponding instruction according to the voice command; and triggering a corresponding event according to the corresponding instruction.
S500, acquiring physical and mental state index data of an operator in real time from multi-modal acquisition equipment; the physical and mental state index data includes, but is not limited to, electroencephalogram, physiological data (heart rate value, blood pressure, etc.), eye movement, near-infrared data, etc. of the operator, and then step S501 is executed.
S501, obtaining a state stage corresponding to a prediction target according to the body and mind state index data acquired from different acquisition equipment; in this embodiment, one or more prediction targets may be provided, and one prediction target is provided with a plurality of state stages, and each state stage corresponds to one physical and mental state index data value; the prediction model can output the state stage of the prediction target corresponding to the input physical and mental state index data according to the input physical and mental state index data.
And step S502, triggering corresponding early warning operation according to the state stage of each predicted target output by the prediction model. In this embodiment, the physical and mental state index data of the operator may be directly obtained in real time according to step S500 to trigger the corresponding pre-warning operation.
Wherein the early warning operations include, but are not limited to:
(1) outputting the early warning state of the state stage in a mode of displaying different state stages by frames with different colors;
(2) outputting the early warning state of the state stage in a mode of playing different state stages by voice;
(3) and outputting the early warning state of the state stage by sending an early warning message.
In this embodiment, step S501 includes at least two implementations.
(1) And inputting the body and mind state index data acquired from different acquisition equipment into corresponding prediction models to obtain the state stage of the corresponding prediction target. The training method of the prediction model comprises the following steps:
firstly, collecting a sample data set of a prediction target; the method comprises the steps that a sample data set comprises a plurality of sample data, wherein each sample data comprises body and mind state index data corresponding to a person in different state stages of a prediction target;
secondly, classifying each sample data in the sample data set according to the state stage, and respectively labeling;
thirdly, dividing the classified and labeled sample data set into a training set and a test set;
fourthly, training the constructed prediction model through a training set, and obtaining a final prediction model after iterating a preset training period (the training period is one period when all sample data in the training set is input into the prediction model);
fifthly, inputting each sample data in the test set into the trained prediction model in sequence to obtain a corresponding test result; the test result comprises a state stage of output of the prediction model after each sample data in the test set is input into the prediction model obtained by training;
sixthly, calculating the evaluation index of the prediction model after the training according to the test result; and finishing the training when the evaluation index meets the requirement, adjusting the sample set and optimizing the parameters if the evaluation index does not meet the requirement, and repeating the second step to the fifth step.
The intelligent state early warning process is explained in detail below by taking a prediction target as an operator stress state as an example; the physical and mental state index data used for judging the pressure state are electroencephalogram and physiological data, and the pressure state comprises a high-pressure state, a baseline state and a low-pressure state; and the early warning operation adopts a mode of displaying different state stages by frames with different colors to output the early warning state of the state stage.
(1) Collecting a sample data set of a prediction target;
inducing values of the electroencephalogram and physiological data of the sampling object in the low-voltage state and the high-voltage state according to the subjective questionnaire and the state scene (the low-voltage state and the high-voltage state); taking a rest state for a preset time (such as 5 minutes) as a baseline state, and recording the values of the data electroencephalogram and physiological data at the moment; storing the electroencephalogram and physiological data corresponding to the low-pressure state, the high-pressure state and the baseline state of each sampling object, and completing the collection of a sample data set of the pressure state; the more sample objects are collected, the more accurate the model prediction result is trained.
(2) And training the prediction model according to the second step to the sixth step to obtain the final prediction model.
(3) Carrying out intelligent state early warning by using the trained prediction model;
collecting values of electroencephalogram and physiological data of an operator from electroencephalogram and physiological detection equipment in real time;
inputting the electroencephalogram and physiological data into a prediction model, outputting a corresponding state stage by the prediction model, and storing the values of the electroencephalogram and physiological data and the corresponding state stage into a database;
triggering corresponding early warning operation according to the corresponding state stage, and specifically comprising:
if the high-voltage state is detected, displaying the high-voltage state by using a first color frame, such as red;
if the baseline state is the baseline state, displaying the baseline state by a second color frame, such as blue;
if the voltage is in a low voltage state, displaying the low voltage state, such as green, by using a third color frame;
and displaying the warning in a visual way by using different colors of a frame of the human-computer interaction interface, and simultaneously sending a prediction result (state stage) to the communication link equipment.
(2) And obtaining the state stage corresponding to the prediction target in a mode of presetting a threshold. This approach includes, but is not limited to, the following steps:
judging the relation between the corresponding physical and mental state index data acquired from the corresponding acquisition equipment and the state preset value;
if the physical and mental state index data are larger than the state preset value, the state stage corresponding to the predicted target is a first state stage;
if the physical and mental state index data are smaller than the state preset value, the state stage corresponding to the predicted target is a second state stage;
and if the body and mind state index data are not completely larger or not smaller than the state preset value, the state stage corresponding to the prediction target is a third state stage.
Taking the prediction target as the pressure state of the operator as an example, the intelligent state early warning process is explained in detail; the physical and mental state index data used for judging the pressure state are electroencephalogram and physiological data, and the pressure state comprises a high-pressure state (a first state stage), a low-pressure state (a second state stage) and a baseline state (a third state stage);
judging the size relation between the electroencephalogram and physiological data and the preset values of the corresponding states of the electroencephalogram and physiological data;
if the electroencephalogram and physiological data are both larger than the preset values of the corresponding states of the electroencephalogram and physiological data, the state is a high-voltage state;
if the electroencephalogram and physiological data are smaller than the preset values of the corresponding states of the electroencephalogram and physiological data, the state is a low-voltage state;
and if the electroencephalogram and physiological data are not completely larger or not smaller than the preset value of the corresponding state of the electroencephalogram and physiological data, the corresponding state is the baseline state.
Step S600, detecting the state of the reserved interface in real time, and then executing step S601.
Step S601, judging whether the state of the reserved interface is in-place state; and when the reserved interface is in the on-position state, indicating that the current reserved interface is accessed into corresponding behavior or physiological acquisition equipment, and executing the step S602, otherwise, executing the step S600.
Step S602, behavior or physiological data acquired by the access acquisition equipment is received through the reserved interface, and a corresponding instruction is obtained or a corresponding event is triggered according to the behavior or physiological data analysis.
The embodiment of the application realizes multisource heterogeneous human-computer interaction, the behavior and the physical and mental state index data of the operators collected through different devices can be received simultaneously, corresponding instructions are obtained through analyzing the behavior and the physical and mental state index data or corresponding events are triggered, the traditional mouse control is improved through eye movement fixation recognition, gesture recognition and voice recognition, human-computer interaction is realized through the physical and mental state of the operators through the feedback of the physical and mental state index data and early warning and real-time feedback of the physical and mental state of the operators, the authenticity and experience of experiments can be better achieved, and the experience of operators is digitalized.
In another embodiment of the present application, as shown in fig. 2, step S201 includes the following steps:
step S2011, the first coordinate in the eye movement data acquired each time is converted into a second coordinate of the first screen where the operation display item is located, and the corresponding operation display item is found on the first screen according to the second coordinate.
In this embodiment, finding the corresponding operation display item on the first screen according to the second coordinate includes:
acquiring coordinates of each display item on a first screen;
judging whether the second coordinate is in the first area of the display item, and if the second coordinate is in the first area of the display item, judging the display item as a corresponding operation display item;
the first area is an area which takes the coordinate of the display item as the center of a circle and takes a preset first value as a radius.
Step S2012, judging the operation effectiveness of the operation display item according to the effective value in the eye movement data; if yes, executing step S2013; otherwise, step S200 is executed.
And step S2013, triggering a corresponding instruction of the operation display item.
In the embodiment, when human-computer interaction is carried out through eye movement, the corresponding instruction of the operation display item is triggered and operated only when the operation effectiveness is effective, so that instruction misjudgment is avoided, the reliability of non-contact instruction input is improved, the accurate triggering of the corresponding event is ensured, and the user experience is improved.
The embodiment of the application also discloses a multi-mode man-machine interaction and state intelligent early warning device which is deployed in a man-machine interaction system or a probabilistic interaction man-machine engineering optimization design technology demonstration system; specifically, the apparatus includes: one or more processors and memory, as shown in FIG. 2, take the example of a processor 200 and memory 100. The processor 200 and the memory 100 may be connected by a bus or other means, such as by a bus connection for example.
The memory 100 is a non-transitory computer readable storage medium, and can be used to store a non-transitory software program and a non-transitory computer executable program, such as a method of multimodal human-machine interaction and intelligent warning of state in the embodiments of the present application. The processor 200 implements a method of multimodal human-machine interaction and intelligent warning of states in the embodiment of the present application by running a non-transitory software program and instructions stored in the memory 100.
The memory 100 may include a storage program area and a storage data area, wherein the storage program area may store an operating device, an application program required for at least one function; the data storage area can store data and the like required by executing the multi-modal human-computer interaction and state intelligent early warning method in the embodiment. Further, the memory 100 may include high speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, the memory optionally includes memory located remotely from the processor, and these remote memories may be connected to the terminal over a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
Non-transitory software programs and instructions required for implementing the method for multimodal human-machine interaction and intelligent warning of status in the foregoing embodiments are stored in a memory, and when being executed by one or more processors, the method for multimodal human-machine interaction and intelligent warning of status in the foregoing embodiments is performed, for example, the method steps S100 to S600, S200 to S201, S300 to S301, S400 to S402, S500 to S502, S600 to S602, and S2011 to S2013 in fig. 1 described above are performed.
In addition, the application also provides a multi-mode human-computer interaction and state intelligent early warning system, which comprises a multi-mode acquisition device and the multi-mode human-computer interaction and state intelligent early warning device provided by the embodiment of the application;
the multi-mode acquisition device acquires human-computer interaction data of an operator; the human-computer interaction data comprises, but is not limited to, cranial nerve signals, eye movement data, myoelectric signals, gesture parameters, voice signals, facial expression data and physical and mental state index data; correspondingly, the multi-mode acquisition device comprises but is not limited to a cranial nerve signal detection device, an eye tracker, a myoelectricity monitoring device, a gesture recognition device, a voice recognition device (capable of acquiring voice signals including voice information and a voice emotion label), a facial expression recognition device and a multi-mode acquisition device for acquiring physical and mental state index data;
the multimode man-machine interaction and state intelligent early warning device acquires man-machine interaction data from the multimode acquisition device and executes a corresponding man-machine interaction process by utilizing the man-machine interaction data; the human-computer interaction process comprises but is not limited to a brain-computer interaction process, an eye movement interaction process, a myoelectricity interaction process, a gesture interaction process, a voice interaction process, an emotion interaction process and a state intelligent early warning process; for example, the emotion labels are obtained according to the voice signals, the gesture parameters and the facial expression data, and then corresponding events are triggered or corresponding instructions are issued according to the emotion labels.
The foregoing is a preferred embodiment of the present application and is not intended to limit the scope of the application in any way, and any features disclosed in this specification (including the abstract and drawings) may be replaced by alternative features serving equivalent or similar purposes, unless expressly stated otherwise. That is, unless expressly stated otherwise, each feature is only an example of a generic series of equivalent or similar features.
Claims (12)
1. A multi-mode man-machine interaction and state intelligent early warning method is characterized in that any one or multiple man-machine interaction flows are executed synchronously, and the man-machine interaction flows comprise a brain-machine interaction flow, an eye movement interaction flow, a myoelectricity interaction flow, a gesture interaction flow, a voice interaction flow, an emotion interaction flow and a state intelligent early warning flow; wherein:
the eye movement interaction process comprises the following steps: determining a corresponding operation display item according to the eye movement data acquired in real time, and triggering a corresponding instruction for operating the operation display item;
the gesture interaction process comprises the following steps: judging a current gesture according to the gesture parameters acquired in real time, and triggering a corresponding instruction according to a gesture judgment result;
the state intelligent early warning process comprises the following steps:
acquiring physical and mental state index data of an operator from multi-modal acquisition equipment in real time;
obtaining a state stage corresponding to a prediction target according to the body and mind state index data acquired from different acquisition equipment;
and triggering corresponding early warning operation according to the state stage of each predicted target.
2. The method according to claim 1, wherein the determining a corresponding operation display item according to the eye movement data acquired in real time and triggering a corresponding instruction for operating the operation display item comprises:
acquiring eye movement data in real time; the eye movement data comprise a first coordinate of the eye fixation position of the user on the eye tracker and a corresponding fixation effective value;
converting the first coordinate obtained each time into a second coordinate of a first screen where an operation display item is located, and searching a corresponding operation display item on the first screen according to the second coordinate;
judging the operation effectiveness of the operation display item according to the effective value; and triggering a corresponding instruction for operating the operation display item when the operation validity is valid.
3. The method according to claim 2, wherein the finding the corresponding operation display item on the first screen according to the second coordinate comprises:
acquiring coordinates of each display item on the first screen;
judging whether the second coordinate is in the first area of the display item, and if the second coordinate is in the first area of the display item, judging the display item as a corresponding operation display item;
the first area is an area which takes the coordinate of the display item as the center of a circle and takes a preset first value as a radius.
4. The method according to claim 2, characterized in that said valid value is the duration of the gaze of said first coordinate position;
the judging the operation validity of the operation display item according to the valid value comprises the following steps:
judging whether the effective value is larger than a preset threshold value or not, and if so, judging that the operation effectiveness of the operation display item is effective; and if the operation validity of the operation display item is less than or equal to the preset threshold, the operation validity of the operation display item is invalid.
5. The method of claim 1, wherein the gesture parameters comprise:
IsLeft, if the left-hand parameter is in an effective state, the left hand is used for making the current gesture;
isright, if the right-hand parameter is in a valid state, it is the right hand that makes the current gesture;
the sample grabbing strength parameter hand, GrabStrength, is between 0 and 1; when the sample grabbing strength parameter is 0, the hand state is a fully unfolded state; when the sample grabbing strength parameter is 1, the hand state is a fist making state;
palm normal, for judging the palm center vertical direction, wherein the palm center vertical direction parameter is an abscissa X, which represents that the palm center vertical direction is horizontal; the palm center vertical direction parameter is a vertical coordinate Y which represents that the palm center vertical direction is vertical;
palm velocity, the hand movement direction parameter being abscissa X, which represents the hand movement direction as left and right, and the hand movement direction parameter being abscissa Y, which represents the hand movement direction as up and down.
6. The method according to claim 5, wherein the determining a current gesture according to the gesture parameters and triggering a corresponding instruction according to a gesture determination result comprises:
judging whether the current gesture is made by the left hand or the right hand according to the left hand parameter and the right hand parameter to obtain a first result;
judging the hand state according to the sample grabbing strength parameter to obtain a second result;
judging the palm center vertical direction according to the palm center vertical direction parameters to obtain a third result;
judging the hand moving direction according to the hand moving direction parameter to obtain a fourth result;
obtaining a current gesture according to the first result, the second result, the third result and the fourth result;
and triggering an instruction corresponding to the current gesture according to the current gesture.
7. The method of claim 1, wherein the obtaining the state phase of the corresponding predicted target according to the physical and mental state index data obtained from different acquisition devices comprises:
and inputting the body and mind state index data acquired from different acquisition equipment into corresponding prediction models to obtain the state stage of the corresponding prediction target.
8. The method of claim 7, wherein the training method of the predictive model comprises the steps of:
collecting a sample data set of the prediction target; the sample data set comprises a plurality of sample data, and each sample data comprises the body and mind state index data corresponding to a person in different state stages of the prediction target;
classifying each sample data in the sample data set according to the state stage, and respectively labeling;
dividing the classified and labeled sample data set into a training set and a test set;
training the constructed prediction model through the training set, and obtaining a final prediction model after iteration of a preset training period;
inputting each sample data in the test set into the trained prediction model in sequence to obtain a corresponding test result; the test result comprises a state stage of output of the prediction model after each sample data in the test set is input into the trained prediction model;
calculating the evaluation index of the prediction model after the training according to the test result; and finishing training when the evaluation index meets the requirement.
9. The method of claim 1, wherein the obtaining the state phase of the corresponding predicted target according to the physical and mental state index data obtained from different acquisition devices comprises:
judging the relation between the body and mind state index data acquired by the corresponding acquisition equipment and the corresponding state preset value;
if the physical and mental state index data are all larger than the state preset value, the state stage corresponding to the predicted target is a first state stage;
if the physical and mental state index data are all smaller than the state preset value, the state stage corresponding to the predicted target is a second state stage;
and if the physical and mental state index data are not completely larger or not smaller than the state preset value, the state stage corresponding to the prediction target is a third state stage.
10. The method of claim 1, wherein the pre-warning operation comprises:
outputting the early warning state of the state stage in a mode of displaying different state stages by frames with different colors;
outputting the early warning state of the state stage in a mode of playing different state stages by voice;
and outputting the early warning state of the state stage by sending an early warning message.
11. A multi-modal human-computer interaction and state intelligent early warning device comprises: memory, processor and computer program stored on the memory and executable on the processor, wherein the processor implements the method of multimodal human-machine interaction and intelligent warning of status as claimed in any one of claims 1 to 10 when executing the program.
12. A multi-modal human-computer interaction and state intelligent early warning system, which is characterized by comprising a multi-modal acquisition device and the multi-modal human-computer interaction and state intelligent early warning device as claimed in claim 11;
the multi-mode acquisition device acquires human-computer interaction data of an operator; the human-computer interaction data comprises cranial nerve signals, eye movement data, myoelectric signals, gesture parameters, voice signals, facial expression data and physical and mental state index data;
the multi-mode human-computer interaction and state intelligent early warning device acquires the human-computer interaction data from the multi-mode acquisition device and executes a corresponding human-computer interaction process by utilizing the human-computer interaction data; the human-computer interaction flow comprises a brain-computer interaction flow, an eye movement interaction flow, a myoelectricity interaction flow, a gesture interaction flow, a voice interaction flow, an emotion interaction flow and a state intelligent early warning flow.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111452903.XA CN114327046B (en) | 2021-11-30 | 2021-11-30 | Method, device and system for multi-mode human-computer interaction and intelligent state early warning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111452903.XA CN114327046B (en) | 2021-11-30 | 2021-11-30 | Method, device and system for multi-mode human-computer interaction and intelligent state early warning |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114327046A true CN114327046A (en) | 2022-04-12 |
| CN114327046B CN114327046B (en) | 2023-03-17 |
Family
ID=81048139
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111452903.XA Active CN114327046B (en) | 2021-11-30 | 2021-11-30 | Method, device and system for multi-mode human-computer interaction and intelligent state early warning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114327046B (en) |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104808909A (en) * | 2015-04-14 | 2015-07-29 | 联想(北京)有限公司 | Information processing method and electronic device |
| CN106128032A (en) * | 2016-07-05 | 2016-11-16 | 北京理工大学珠海学院 | A kind of fatigue state monitoring and method for early warning and system thereof |
| JP2017204082A (en) * | 2016-05-10 | 2017-11-16 | 株式会社デンソー | Operation device for vehicle |
| CN109213323A (en) * | 2018-08-28 | 2019-01-15 | 北京航空航天大学青岛研究院 | A method of screen Attitude estimation is realized based on eye movement interaction technique |
| CN109276237A (en) * | 2018-11-14 | 2019-01-29 | 郑州大学 | A kind of patients with cerebral apoplexy balanced capacity evaluation and test of Omni-mobile and training system |
| CN110236812A (en) * | 2019-07-16 | 2019-09-17 | 哈尔滨理工大学 | A kind of multi-modal Human-computer interactive intelligent wheelchair system with sign monitoring function |
| CN110286763A (en) * | 2019-06-21 | 2019-09-27 | 济南大学 | A kind of navigation-type experiment interactive device with cognitive function |
| CN110390457A (en) * | 2018-04-23 | 2019-10-29 | 宁波欧依安盾安全科技有限公司 | A kind of safe competency realtime qualification and early warning system |
| CN111190484A (en) * | 2019-12-25 | 2020-05-22 | 中国人民解放军军事科学院国防科技创新研究院 | Multi-mode interaction system and method |
| CN111290575A (en) * | 2020-01-21 | 2020-06-16 | 中国人民解放军空军工程大学 | Multichannel interactive control system of air defense anti-pilot weapon |
| CN112120716A (en) * | 2020-09-02 | 2020-12-25 | 中国人民解放军军事科学院国防科技创新研究院 | Wearable multi-mode emotional state monitoring device |
-
2021
- 2021-11-30 CN CN202111452903.XA patent/CN114327046B/en active Active
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104808909A (en) * | 2015-04-14 | 2015-07-29 | 联想(北京)有限公司 | Information processing method and electronic device |
| JP2017204082A (en) * | 2016-05-10 | 2017-11-16 | 株式会社デンソー | Operation device for vehicle |
| CN106128032A (en) * | 2016-07-05 | 2016-11-16 | 北京理工大学珠海学院 | A kind of fatigue state monitoring and method for early warning and system thereof |
| CN110390457A (en) * | 2018-04-23 | 2019-10-29 | 宁波欧依安盾安全科技有限公司 | A kind of safe competency realtime qualification and early warning system |
| CN109213323A (en) * | 2018-08-28 | 2019-01-15 | 北京航空航天大学青岛研究院 | A method of screen Attitude estimation is realized based on eye movement interaction technique |
| CN109276237A (en) * | 2018-11-14 | 2019-01-29 | 郑州大学 | A kind of patients with cerebral apoplexy balanced capacity evaluation and test of Omni-mobile and training system |
| CN110286763A (en) * | 2019-06-21 | 2019-09-27 | 济南大学 | A kind of navigation-type experiment interactive device with cognitive function |
| CN110236812A (en) * | 2019-07-16 | 2019-09-17 | 哈尔滨理工大学 | A kind of multi-modal Human-computer interactive intelligent wheelchair system with sign monitoring function |
| CN111190484A (en) * | 2019-12-25 | 2020-05-22 | 中国人民解放军军事科学院国防科技创新研究院 | Multi-mode interaction system and method |
| CN111290575A (en) * | 2020-01-21 | 2020-06-16 | 中国人民解放军空军工程大学 | Multichannel interactive control system of air defense anti-pilot weapon |
| CN112120716A (en) * | 2020-09-02 | 2020-12-25 | 中国人民解放军军事科学院国防科技创新研究院 | Wearable multi-mode emotional state monitoring device |
Non-Patent Citations (1)
| Title |
|---|
| 王巍: "《Leap Motion人机交互应用开发》", 30 November 2015 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114327046B (en) | 2023-03-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104504390B (en) | A kind of user on the network's state identification method and device based on eye movement data | |
| CN107097227B (en) | human-computer cooperation robot system | |
| EP2552304B1 (en) | Brain-computer interface system and method | |
| US20210109598A1 (en) | Systems, methods and devices for gesture recognition | |
| Ktena et al. | A virtual reality platform for safe evaluation and training of natural gaze-based wheelchair driving | |
| CN111712192A (en) | Brain-computer interface with adaptation for high speed, accurate and intuitive user interaction | |
| CN111104820A (en) | Gesture recognition method based on deep learning | |
| CN108703824B (en) | Bionic hand control system and control method based on myoelectricity bracelet | |
| CN103336581A (en) | Human eye movement characteristic design-based human-computer interaction method and system | |
| CN111898407B (en) | Human-computer interaction operating system based on human face action recognition | |
| CN111290575A (en) | Multichannel interactive control system of air defense anti-pilot weapon | |
| CN110970130A (en) | Data processing method for attention defect hyperactivity disorder | |
| CN110658742A (en) | Multi-mode cooperative control wheelchair control system and method | |
| Fahim et al. | A visual analytic in deep learning approach to eye movement for human-machine interaction based on inertia measurement | |
| CN109976525A (en) | A kind of user interface interaction method, apparatus and computer equipment | |
| CN104866164A (en) | Human-machine interaction system and method based on blink signal mode detection | |
| Çığ et al. | Gaze-based prediction of pen-based virtual interaction tasks | |
| CN114821753B (en) | Eye movement interaction system based on visual image information | |
| CN106681509A (en) | Interface operating method and system | |
| Weisz et al. | Assistive grasping with an augmented reality user interface | |
| CN113974589B (en) | Multi-modal behavior paradigm evaluation optimization system and cognitive ability evaluation method | |
| Park et al. | Korean finger number gesture recognition based on CNN using surface electromyography signals | |
| CN114327046B (en) | Method, device and system for multi-mode human-computer interaction and intelligent state early warning | |
| Zhang et al. | Research on multi-modal interactive control for quadrotor uav | |
| Huang et al. | Real-time precise human-computer interaction system based on gaze estimation and tracking |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |