CN113349781A - Electroencephalogram capture method and system for typical mental state of operator - Google Patents
Electroencephalogram capture method and system for typical mental state of operator Download PDFInfo
- Publication number
- CN113349781A CN113349781A CN202110560459.7A CN202110560459A CN113349781A CN 113349781 A CN113349781 A CN 113349781A CN 202110560459 A CN202110560459 A CN 202110560459A CN 113349781 A CN113349781 A CN 113349781A
- Authority
- CN
- China
- Prior art keywords
- electroencephalogram
- computer unit
- operator
- mental state
- cooperative robot
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000006996 mental state Effects 0.000 title claims abstract description 174
- 238000000034 method Methods 0.000 title claims abstract description 49
- 230000001939 inductive effect Effects 0.000 claims abstract description 43
- 230000033001 locomotion Effects 0.000 claims abstract description 39
- 230000008569 process Effects 0.000 claims abstract description 26
- 230000005540 biological transmission Effects 0.000 claims abstract description 20
- 230000006870 function Effects 0.000 claims abstract description 12
- 238000012544 monitoring process Methods 0.000 claims abstract description 7
- 238000011156 evaluation Methods 0.000 claims description 24
- 210000004556 brain Anatomy 0.000 claims description 20
- 210000001652 frontal lobe Anatomy 0.000 claims description 10
- 210000000869 occipital lobe Anatomy 0.000 claims description 10
- 210000001152 parietal lobe Anatomy 0.000 claims description 10
- 210000003478 temporal lobe Anatomy 0.000 claims description 10
- 210000003710 cerebral cortex Anatomy 0.000 claims description 6
- 230000009471 action Effects 0.000 claims description 5
- 238000004891 communication Methods 0.000 claims description 5
- 238000005070 sampling Methods 0.000 claims description 5
- 238000012546 transfer Methods 0.000 claims description 5
- 230000002457 bidirectional effect Effects 0.000 claims description 4
- 238000004364 calculation method Methods 0.000 claims description 3
- 238000010586 diagram Methods 0.000 description 10
- 238000004590 computer program Methods 0.000 description 9
- 230000006698 induction Effects 0.000 description 7
- 101150083764 KCNK9 gene Proteins 0.000 description 6
- 230000001976 improved effect Effects 0.000 description 6
- 230000002452 interceptive effect Effects 0.000 description 6
- 238000002474 experimental method Methods 0.000 description 5
- 238000012545 processing Methods 0.000 description 5
- 230000007246 mechanism Effects 0.000 description 4
- 230000004044 response Effects 0.000 description 4
- 230000000638 stimulation Effects 0.000 description 4
- 230000008901 benefit Effects 0.000 description 3
- 230000008859 change Effects 0.000 description 3
- 239000000463 material Substances 0.000 description 3
- 238000011160 research Methods 0.000 description 3
- 238000012360 testing method Methods 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 230000000763 evoking effect Effects 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 239000013589 supplement Substances 0.000 description 2
- 230000003044 adaptive effect Effects 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 230000033228 biological regulation Effects 0.000 description 1
- 230000004397 blinking Effects 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 238000004070 electrodeposition Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000036632 reaction speed Effects 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
- 230000004936 stimulating effect Effects 0.000 description 1
- 230000002123 temporal effect Effects 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Images
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/16—Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state
- A61B5/18—Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state for vehicle drivers or machine operators
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/16—Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state
- A61B5/165—Evaluating the state of mind, e.g. depression, anxiety
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/24—Detecting, measuring or recording bioelectric or biomagnetic signals of the body or parts thereof
- A61B5/25—Bioelectric electrodes therefor
- A61B5/251—Means for maintaining electrode contact with the body
- A61B5/256—Wearable electrodes, e.g. having straps or bands
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/24—Detecting, measuring or recording bioelectric or biomagnetic signals of the body or parts thereof
- A61B5/25—Bioelectric electrodes therefor
- A61B5/279—Bioelectric electrodes therefor specially adapted for particular uses
- A61B5/291—Bioelectric electrodes therefor specially adapted for particular uses for electroencephalography [EEG]
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/24—Detecting, measuring or recording bioelectric or biomagnetic signals of the body or parts thereof
- A61B5/316—Modalities, i.e. specific diagnostic methods
- A61B5/369—Electroencephalography [EEG]
- A61B5/372—Analysis of electroencephalograms
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/24—Detecting, measuring or recording bioelectric or biomagnetic signals of the body or parts thereof
- A61B5/316—Modalities, i.e. specific diagnostic methods
- A61B5/369—Electroencephalography [EEG]
- A61B5/377—Electroencephalography [EEG] using evoked responses
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
- A61B5/7235—Details of waveform analysis
- A61B5/7264—Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/74—Details of notification to user or communication with user or patient ; user input means
- A61B5/7475—User input or interface means, e.g. keyboard, pointing device, joystick
-
- A—HUMAN NECESSITIES
- A63—SPORTS; GAMES; AMUSEMENTS
- A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
- A63F13/20—Input arrangements for video game devices
- A63F13/21—Input arrangements for video game devices characterised by their sensors, purposes or types
- A63F13/212—Input arrangements for video game devices characterised by their sensors, purposes or types using sensors worn by the player, e.g. for measuring heart beat or leg activity
-
- A—HUMAN NECESSITIES
- A63—SPORTS; GAMES; AMUSEMENTS
- A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
- A63F13/20—Input arrangements for video game devices
- A63F13/24—Constructional details thereof, e.g. game controllers with detachable joystick handles
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/015—Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
-
- A—HUMAN NECESSITIES
- A63—SPORTS; GAMES; AMUSEMENTS
- A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- A63F2300/00—Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game
- A63F2300/10—Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game characterized by input arrangements for converting player-generated signals into game device control signals
- A63F2300/1012—Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game characterized by input arrangements for converting player-generated signals into game device control signals involving biosensors worn by the player, e.g. for measuring heart beat, limb activity
-
- A—HUMAN NECESSITIES
- A63—SPORTS; GAMES; AMUSEMENTS
- A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- A63F2300/00—Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game
- A63F2300/10—Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game characterized by input arrangements for converting player-generated signals into game device control signals
- A63F2300/1043—Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game characterized by input arrangements for converting player-generated signals into game device control signals being characterized by constructional details
Landscapes
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Heart & Thoracic Surgery (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Public Health (AREA)
- Molecular Biology (AREA)
- Veterinary Medicine (AREA)
- Pathology (AREA)
- Animal Behavior & Ethology (AREA)
- Surgery (AREA)
- Medical Informatics (AREA)
- Psychiatry (AREA)
- Multimedia (AREA)
- Psychology (AREA)
- Human Computer Interaction (AREA)
- Artificial Intelligence (AREA)
- Child & Adolescent Psychology (AREA)
- Social Psychology (AREA)
- Educational Technology (AREA)
- Signal Processing (AREA)
- Physiology (AREA)
- General Engineering & Computer Science (AREA)
- Developmental Disabilities (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Theoretical Computer Science (AREA)
- Hospice & Palliative Care (AREA)
- Dermatology (AREA)
- Cardiology (AREA)
- General Physics & Mathematics (AREA)
- Neurosurgery (AREA)
- Neurology (AREA)
- Evolutionary Computation (AREA)
- Fuzzy Systems (AREA)
- Mathematical Physics (AREA)
- Measurement And Recording Of Electrical Phenomena And Electrical Characteristics Of The Living Body (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
The invention discloses an electroencephalogram capture method and system for typical mental states of an operator.A electroencephalogram signal acquisition module acquires electroencephalogram signals of typical mental states of the operator by triggering an acquisition mode and a timing acquisition mode through a capture instruction, and comprises a first computer unit and electroencephalogram acquisition equipment connected with the first computer unit; the virtual cooperative robot operation module controls the virtual cooperative robot to complete typical mental state inducing tasks and continuously induces an operator to generate a plurality of typical mental states, and the typical mental state inducing tasks comprise a second computer unit used for simulating the movement of the cooperative robot and control instruction sending equipment connected with the second computer unit; the first computer unit is connected with the second computer unit through a data transmission network, and the functions of monitoring the dynamic process of the virtual cooperative robot operated by an operator and adaptively acquiring the electroencephalogram signals of the typical mental state of the operator are realized. The invention improves the flexibility of the electroencephalogram acquisition system of the mental state of an operator and the effectiveness of electroencephalogram data.
Description
Technical Field
The invention belongs to the technical field of brain-computer interfaces, and particularly relates to an electroencephalogram capture method and system for typical mental states of an operator.
Background
In the human-machine system, mental state fluctuations are easily generated by an operator influenced by internal or external factors while performing an operation task, which may reduce the reaction speed and operation accuracy of the operator, and thus it is important to recognize the mental state of the operator while working. Physiological signal-based mental state identification methods are widely focused on due to their objectivity, and electroencephalogram signals are effective means for identifying mental states due to their high temporal resolution.
The acquisition of the electroencephalogram signals under different mental states is the basis for researching the electroencephalogram response mechanism of mental state change and identifying the mental state of an operator based on the electroencephalogram signals. On one hand, the existing mental state related electroencephalogram signal acquisition is mainly based on two modes, firstly, the electroencephalogram signal under continuous stimulation of a picture or audio and video mode is acquired, and the ecological effectiveness of the electroencephalogram signal applied to a human-computer system is low; and secondly, electroencephalogram signals of an operator during physical operation are collected, and the system running cost is high under the mode. On the other hand, in an actual human-machine system, mental state fluctuation of an operator does not exist all the time, and most of the existing electroencephalogram signal acquisition systems acquire and store electroencephalogram data for a full time under the stimulation of an inducing material. The method lacks a sorting mechanism, and electroencephalogram signals in a non-target mental state can be acquired and stored mistakenly, so that the reliability of acquired electroencephalogram data is reduced.
Disclosure of Invention
The technical problem to be solved by the present invention is to provide an electroencephalogram capture method and system for an operator in a typical mental state, to dynamically and adaptively capture an electroencephalogram signal of the operator in the typical mental state in a continuous operation process, to improve flexibility of electroencephalogram signal acquisition and reliability of the acquired and stored electroencephalogram signal, and to meet experimental requirements of acquiring the electroencephalogram signal of the operator in the typical mental state in the operation process.
The invention adopts the following technical scheme:
an electroencephalogram capture system of a typical mental state of an operator, comprising:
the electroencephalogram signal acquisition module is used for acquiring electroencephalogram signals of an operator in a typical mental state by triggering acquisition modes through a capture instruction and comprises a first computer unit for acquiring the electroencephalogram signals and electroencephalogram acquisition equipment connected with the first computer unit;
the virtual cooperative robot operation module is used for controlling the virtual cooperative robot to complete typical mental state inducing tasks and continuously inducing an operator to generate a plurality of typical mental states, and comprises a second computer unit used for simulating the movement of the cooperative robot and a virtual cooperative robot control instruction sending device connected with the second computer unit;
the first computer unit is connected with the second computer unit through a data transmission network, and the functions of monitoring the dynamic process of the virtual cooperative robot operated by an operator and adaptively acquiring the electroencephalogram signals of the typical mental state of the operator are realized.
Specifically, the first computer unit comprises an electroencephalogram signal capturing discriminator and a typical mental state subjective evaluation scale; the electroencephalogram signal capturing discriminator receives the operation log information of the second computer unit through a data transmission network and sends a capturing instruction to the electroencephalogram acquisition equipment to trigger electroencephalogram signal acquisition; typical mental state subjective rating scales are used to determine typical mental state categories.
Further, typical mental state subjective rating scales include concentration state, non-concentration state, stress state, fatigue state, active state, and passive state.
Specifically, the electroencephalogram signal acquisition mode of the electroencephalogram signal acquisition module for acquiring the electroencephalogram signal of the operator in the typical mental state further comprises a timing acquisition mode, and the typical mental state electroencephalogram signal of the operator in a time period without obvious operation log information is acquired in the timing acquisition operation process.
Specifically, the virtual cooperative robot control instruction sending device is connected with the second computer unit in a USB wireless transmission manner, and is configured to send the virtual cooperative robot control instruction, and the virtual cooperative robot receives and executes the virtual cooperative robot control instruction and generates operation result log information at the same time.
Further, the virtual cooperative robot control instructions include instructions for controlling the robot chassis, and instructions for movement or motion of the robot arm and the gripper.
Specifically, the electroencephalogram acquisition equipment comprises an electroencephalogram cap, electrodes of the electroencephalogram cap are arranged in frontal lobe, occipital lobe, parietal lobe and temporal lobe areas of a brain, and the sampling frequency of the electroencephalogram acquisition equipment is 1000 Hz.
The invention also provides a brain wave capturing method of the typical mental state of an operator, which utilizes a brain wave capturing system of the typical mental state of the operator and comprises the following steps:
s1, collecting brain electrical signals of frontal lobe, occipital lobe, parietal lobe and temporal lobe areas of cerebral cortex by brain electrical collecting equipment;
s2, controlling the virtual cooperative robot to move through the game handle to complete a mental state inducing task, wherein the mental state inducing task is one or a combination of track tracking, object block grabbing and object block transferring, and corresponding operation log information is generated when a control instruction related to the mechanical arm or the clamping jaw is executed in the process of completing the inducing task;
s3, when the mental state inducing task is to grab or transfer the object block, the operation log information generated in the step S2 is transmitted from the second computer unit to the first computer unit through the data transmission network, a capture instruction is adopted to trigger the acquisition mode, and when the first computer unit identifies the operation log information corresponding to the typical mental state, the capture instruction is sent to acquire the electroencephalogram signal in the typical mental state;
s4, when the mental state inducing task is track tracking, the electroencephalogram signals in the step S1 are collected at regular time and stored;
s5, acquiring and storing the labels of the electroencephalogram signals acquired in the step S3 and the step S4 through the typical mental state subjective evaluation scale of the first computer unit, and completing the electroencephalogram capture of the typical mental state of the operator.
Specifically, in step S2, the virtual cooperative robot operating module includes 5 mental state inducing tasks, and the controlling of the virtual cooperative robot by the gamepad specifically includes:
the game handle is connected to the second computer unit through the wireless USB receiver, the virtual cooperative robot runs in the ROS environment of a robot operating system of the second computer unit, the game handle is operated to send a control instruction to the virtual cooperative robot in the second computer unit, the virtual cooperative robot comprises a chassis, a mechanical arm and a clamping jaw, the control instruction for controlling the linear velocity or the angular velocity of the motion of the chassis is sent to the virtual cooperative robot through a rocker of the game handle, and the chassis executes forward, backward, left-turn and right-turn motions according to the control instruction to realize parallel motions of movement and rotation; and sending a control instruction for controlling the opening or closing of the clamping jaw and the movement or rotation of the tail end of the mechanical arm along the direction X, Y, Z under a Cartesian coordinate system to the virtual cooperative robot through keys of the game handle, wherein the clamping jaw or the mechanical arm generates stepping movement according to the key control instruction.
Specifically, step S3 specifically includes:
s301, taking the first computer unit running MATLAB as an ROS node, and under the same network, realizing communication connection between the first computer unit and the node in the ROS network running on the second computer unit, and realizing bidirectional information transmission between the first computer unit and the second computer unit;
s302, in an ROS Master operated by a second computer unit, a node/rosout is a publisher of a topic/rosout _ agg, and the/Msg information of the topic/rosout _ agg is a control instruction received by a virtual cooperative robot, a calculation result generated by the received instruction and a robot instruction execution result, so that a first computer unit node operated with MATLAB subscribes the topic/rosout _ agg and reads message contents circularly to acquire real-time log information of operation;
and S303, when the electroencephalogram signal capturing discriminator operated on the first computer unit identifies specific operation log information related to the typical mental state of an operator, generating a capturing instruction, wirelessly transmitting the electroencephalogram data acquired by the electroencephalogram acquisition equipment to the first computer unit through a data transmission network, and storing the electroencephalogram data, wherein the electroencephalogram acquisition equipment acquires the electroencephalogram signals with the storage time of 5-60 seconds.
Compared with the prior art, the invention has at least the following beneficial effects:
the invention discloses an electroencephalogram capture system for a typical mental state of an operator, and aims to obtain electroencephalogram data of the operator in the typical mental state in the operation process, realize dynamic monitoring of the operation process of the operator, adaptively acquire electroencephalogram signals of the operator in the process of continuously operating a virtual cooperative robot to complete a mental state induction task, and obtain a function of a typical mental state label of the electroencephalogram data. On one hand, the virtual cooperative robot operation module provides a virtual interactive operation scene to realize the acquisition of tested electroencephalogram data in the interactive operation process, compared with the acquisition of pictures or video mode stimulation, interactive induction task operation restores the influence factors of mental state fluctuation of an operator in an actual operation environment to a higher degree, and lays a research foundation for a mental state monitoring algorithm to better meet the actual working condition; compared with the operation of a real robot, the virtual interactive operation has the advantage that the cost is greatly reduced. On the other hand, in the actual human-computer system, the mental state fluctuation of an operator does not exist all the time, so that in addition to the timing acquisition mode, a capture instruction trigger acquisition mode is designed in the electroencephalogram acquisition module. The capture instruction triggers the acquisition mode to monitor the operation process of an operator, triggers typical mental state electroencephalogram signal acquisition according to the actual condition of the mental state inducing task, and improves the inducing efficiency of the stimulating material and the reliability of stored electroencephalogram data.
Further, the electroencephalogram signal capture discriminator is used for receiving and recognizing the operation log information from the second computer unit and generating an electroencephalogram signal capture instruction. The virtual cooperative robot has the advantages that correct operation prompts and wrong operation prompts are timely provided according to the actual condition that an operator operates the virtual cooperative robot to complete a mental state inducing task, the operator is induced to generate an active state or a passive state by combining operation results, the correct or wrong operation results of the operator are collected, and electroencephalogram signals in a typical mental state after prompting are received. The typical mental state subjective evaluation scale is used for acquiring mental state labels of electroencephalogram data. The individual difference caused by factors such as personality, personality and experience in the induction of the mental state is considered, the typical mental state subjective evaluation scale is set, the electroencephalogram data label deviation caused by the individual difference can be avoided, and the accuracy of the acquired electroencephalogram data label is improved. The typical mental state subjective evaluation scale is used as a ring of man-machine interaction and is positioned in the first computer unit of the electroencephalogram acquisition module, so that the automation degree of the experiment is improved. The existing mental state evaluation method has two modes of subjective evaluation and objective evaluation. The subjective evaluation is to obtain the current mental state of the tested object based on the subjective evaluation mode, and can provide basis for researching the relation between the mental state and the electroencephalogram response.
Further, concentration, non-concentration, tension, fatigue, active and passive states are matched to mental state evoked task planning; more fundamentally, concentration, non-concentration, tension, fatigue, active and passive states occur frequently in real man-machine systems and are closely related to the operator's decision and performance, and when the operator is in a poor mental state, it is easy for insufficient or even wrong operation to occur. The duration of attention becomes low when the operator is tired, and the judgment and operation capabilities become poor; the reaction is slow due to excessive pressure and stress. In the non-attentive state and the passive state, the operator is less attentive than in the active state; the state of concentration, non-concentration, tension, fatigue, active and passive describe the mental state from different perspectives, essentially covering all possible categories during the operation of mental state inducing tasks; in addition, the evaluation mode of the discrete mental state is simpler and more practical than the evaluation based on the dimension model.
Furthermore, the timing acquisition mode is used as a supplement of a capture instruction triggering acquisition mode and is used for acquiring the electroencephalogram signals of the operator in a typical mental state in a time period without obvious operation log information. The timing acquisition mode is set, the efficiency of the system is improved, more electroencephalogram data can be acquired, and the timing acquisition mode is mainly used for acquiring electroencephalogram data in a concentration state, a non-concentration state, a tension state and a fatigue state.
Furthermore, the virtual cooperative robot control instruction sending equipment is connected with the second computer unit through the USB ultra-miniature receiver, is plug-and-play, and adopts a 2.4G game level wireless technology to realize high-speed data transmission.
Furthermore, the virtual cooperative robot consists of a chassis, a mechanical arm and a clamping jaw. In order to control the virtual cooperative robot to complete the tasks of moving, grabbing and the like, a command for controlling the movement of a chassis of the robot, a mechanical arm movement command and a clamping jaw movement command are designed. The moving task is realized by controlling the movement of the chassis, and the tasks such as grabbing and the like are realized by the cooperative movement of the mechanical arm and the clamping jaw.
Furthermore, the electrodes of the brain electric cap are arranged in the frontal lobe, occipital lobe, parietal lobe and temporal lobe areas of the brain, 30 brain electric electrodes are arranged in total, the sampling frequency is 1000Hz, and the collected brain electric signals have high time resolution and relatively high spatial resolution.
An electroencephalogram capture method for typical mental states of an operator is characterized in that an acquisition mode triggered by a capture instruction and a timing acquisition mode are combined to acquire an electroencephalogram signal of the operator when the mental state task is completed by controlling a virtual cooperative robot to move through a game handle, and a mental state label corresponding to stored electroencephalogram data is acquired.
Further, when the virtual cooperative robot is controlled to move by the gamepad in the step S2, the rocker of the gamepad is used for changing the linear velocity and angular velocity of the chassis movement, so as to realize stepless speed regulation; the linear velocity and the angular velocity of the chassis can be adjusted at the same time, and the parallel motion of the movement and the rotation of the robot is realized; the motion mode of the mechanical arm and the clamping jaw is stepping motion, the mechanical arm is enabled to approach to a grabbing target by adjusting the pose of the tail end of the mechanical arm, and finally the clamping jaw is closed to complete grabbing.
Further, step S3 realizes bidirectional information transmission between the first computer unit and the second computer unit based on ROS-MATLAB communication, and realizes a function that the first computer unit reads real-time log information of the virtual cooperative robot operated by the operator, and acquires electroencephalogram signals of the operator in a typical mental state based on an electroencephalogram signal capture discriminator.
In conclusion, the method and the device realize the function of dynamically monitoring the operation process of the operator, adaptively acquiring the electroencephalogram signals of the operator in a typical mental state during the process of continuously operating the virtual cooperative robot to complete the mental state induction task in the interactive environment, and acquiring the data tags through the subjective evaluation scale.
The technical solution of the present invention is further described in detail by the accompanying drawings and embodiments.
Drawings
FIG. 1 is a continuous induction and dynamic adaptive capture system of brain electrical signals of typical mental states of an operator;
FIG. 2 is a diagram illustrating mental state inducing tasks in the virtual cooperative robot manipulator module;
fig. 3 is a schematic diagram of the channel position of the electroencephalogram acquisition device.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It will be understood that the terms "comprises" and/or "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
It is also to be understood that the terminology used in the description of the invention herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in the specification of the present invention and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.
It should be further understood that the term "and/or" as used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.
Various structural schematics according to the disclosed embodiments of the invention are shown in the drawings. The figures are not drawn to scale, wherein certain details are exaggerated and possibly omitted for clarity of presentation. The shapes of various regions, layers and their relative sizes and positional relationships shown in the drawings are merely exemplary, and deviations may occur in practice due to manufacturing tolerances or technical limitations, and a person skilled in the art may additionally design regions/layers having different shapes, sizes, relative positions, according to actual needs.
Referring to fig. 1, the present invention provides an electroencephalogram capture system for typical mental states of an operator, comprising:
the electroencephalogram signal acquisition module comprises a first computer unit for acquiring electroencephalogram signals, and electroencephalogram acquisition equipment and a data transmission network which are connected with the first computer unit;
the virtual cooperative robot operation module comprises a second computer unit used for simulating the movement of the cooperative robot and a virtual cooperative robot control instruction sending device connected with the second computer unit.
The electroencephalogram signal acquisition module acquires and stores electroencephalogram signals of an operator in a typical mental state in two modes, namely a capture instruction triggering acquisition mode and a timing acquisition mode.
The timing acquisition mode is a supplement of an instruction triggering acquisition mode, typical mental state electroencephalogram signals of an operator at a time period without obvious operation log information are acquired in the timing acquisition operation process, and the working efficiency of the system is improved.
In the electroencephalogram signal acquisition module, the acquisition command triggering acquisition mode is realized by an electroencephalogram signal acquisition discriminator in a first computer unit; the electroencephalogram signal capturing discriminator receives operation log information from a second computer unit in the virtual cooperative robot operation module through a data transmission network; when specific operation information related to certain typical mental states is identified, the electroencephalogram signal capturing discriminator sends a capturing instruction to trigger electroencephalogram signal acquisition, and the functions of monitoring the dynamic process of an operator operating the virtual cooperative robot and adaptively acquiring the electroencephalogram signals of the typical mental states of the operator are achieved.
The first computer unit is provided with a typical mental state subjective evaluation scale. The labels of the electroencephalogram data obtained by the two electroencephalogram signal acquisition modes in the electroencephalogram signal acquisition module, namely the category of the typical mental state, are determined by a typical mental state subjective evaluation scale.
The subjective evaluation scale is an evaluation scale containing six typical mental states, and an operator completes subjective evaluation of the mental states according to the content of the scale. The six typical mental states are states that are easily generated in actual operations and may affect the quality of operations, including a state of concentration, a state of non-concentration, a state of tension, a state of fatigue, an active state, and a passive state.
The electroencephalogram acquisition equipment comprises an electroencephalogram cap, electrodes of the electroencephalogram cap are arranged in frontal lobe, occipital lobe, parietal lobe and temporal lobe areas of a brain, and the sampling frequency of the electroencephalogram acquisition equipment is 1000 Hz.
In the virtual cooperative robot operation module, a second computer unit is connected with a virtual cooperative robot control instruction sending device in a USB wireless transmission mode, an operator manually operates and controls the virtual cooperative robot instruction sending device, and sends a control instruction to the virtual cooperative robot in the second computer unit to control the virtual cooperative robot to complete a typical mental state inducing task and continuously induce the virtual cooperative robot to generate a plurality of typical mental states.
The control instruction of the virtual cooperative robot comprises an instruction for controlling the movement or action of a robot chassis, a robot arm and a clamping jaw, namely an instruction for controlling the linear velocity and the angular velocity of the movement of the chassis, an instruction for adjusting the terminal pose of the mechanical arm and an instruction for controlling the opening and closing of the clamping jaw.
The virtual cooperative robot in the second computer unit receives and executes the control instruction, and at the same time, generates operation result log information. The operation result log information is result information generated after the virtual cooperative robot receives and executes the control instruction, such as the control instruction is successfully executed, the control instruction cannot be executed, and the like.
The invention relates to an electroencephalogram capture method for typical mental states of an operator, which is characterized in that the operator operates a virtual cooperative robot through a game handle according to task requirements under an operation module of the virtual cooperative robot, real-time operation log information is generated in the operation process, an electroencephalogram signal acquisition module receives the real-time operation log information from the operation module of the virtual cooperative robot, whether a capture instruction is generated or not is judged according to the operation log information, and after the capture instruction is generated, electroencephalogram signals of the operator are acquired and stored; the method comprises the following steps:
s1, collecting brain electrical signals of frontal lobe, occipital lobe, parietal lobe and temporal lobe areas of cerebral cortex by brain electrical collecting equipment;
the virtual cooperative robot operator wears an electroencephalogram cap, sits in front of the second computer unit, the head is 40-100 cm away from the display of the second computer unit, 34 electroencephalogram caps (comprising 2 reference electrodes, 2 electro-ocular electrodes, the rest 30 electroencephalogram electrodes arranged according to the international 10-20 standard, the specific electrode positions are shown in figure 3) are adopted to collect electroencephalogram signals of the frontal lobe, occipital lobe, parietal lobe and temporal lobe areas of the cerebral cortex of the operator, and the sampling frequency of the electroencephalogram amplifier is 1 kHz.
S2, controlling the virtual cooperative robot to move through the game handle to complete a mental state inducing task, wherein the mental state inducing task is one or a combination of track tracking, object block grabbing and object block transferring, and corresponding operation log information is generated when a control instruction related to the mechanical arm or the clamping jaw is executed in the process of completing the inducing task;
and the virtual cooperative robot operator controls the virtual cooperative robot to move by using the gamepad under a virtual cooperative robot operation module (a second computer unit and an external operation environment space thereof) according to task requirements so as to complete an operation task, wherein the operation task is selected from one or more of track tracking, object block grabbing and object block transferring.
Referring to fig. 2, there are 5 mental state inducing tasks in the virtual cooperative robot operating module, and the specific requirements of each task are as follows:
in Task 0, an operator needs to control the virtual cooperative robot to walk in a rectangular room for one turn clockwise, without limiting the motion trajectory of the virtual cooperative robot.
In Task 1, the operator needs to control the virtual cooperative robot to walk along two sides of the triangle, namely a and c.
In Task 2, the operator needs to control the virtual cooperative robot to grab the blue small object block on the desktop, and then place the small object block in the white tray.
Task3 is a time-limited Task, and the operator needs to control the virtual cooperative robot to go out of the maze within 1 minute.
In contrast to Task 2, in Task 4 the small pieces are replaced with one cola bottle and the tray is placed on another table.
In the electroencephalogram capture system of the typical mental state of the operator, the selected typical mental state is a positive state, a negative state, a concentration state, a non-concentration state, a tension state and a fatigue state which are frequently appeared in a man-machine system and influence the operation quality. The order of execution of the mental state induction Task requests is Task 1 → Task 2 → Task3 → Task 0 → Task 1 → Task 2 → Task3 → Task 0 → Task 4. Since Task 1 is the first Task of the whole experiment, and Task 1 has high requirement on the walking track of the virtual cooperative robot, Task 1 is used to induce the attentive state of the operator. The Task 2 is used for inducing the operator to generate a positive state and a negative state, when the operator executes the Task 2, once an error operation occurs in the Task execution process, the experimental guidance interface running in the first computer unit gives an error operation prompt to the operator according to the operation log information received from the second computer unit, wherein the error operation prompt comprises a voice prompt and a text prompt, and the error operation refers to a fault such as collision after the virtual cooperative robot executes an action. On the contrary, when the Task 2 in the second computer unit is successfully completed, the experiment guide interface gives the operator a correct operation prompt, and the correct operation prompt comprises a voice prompt and a text prompt. The wrong operation cues are used to induce a negative state and the correct operation cues are used to induce an positive state. Since the track of Task 0 is monotonous, simple, and lengthy, and Task 0 has no requirement for the walking track of the virtual cooperative robot, Task 0 is used to induce the non-concentration state of the operator. Task3 is used to induce the operator to be stressed because Task3 is set to a time-limited Task, and the operator needs to control the virtual collaborative robot to go out of the maze within 1 minute. The Task 4 is placed at the last in the sequence of execution of mental state inducing tasks, and the Task 4 is used to induce fatigue state to the operator through the first two rounds of mental state inducing tasks.
In order to complete the above task, the specific implementation manner of operating the virtual cooperative robot by the operator in the virtual cooperative robot operating module is as follows:
the game handle is connected to the second computer unit through the wireless USB receiver, the virtual cooperative robot runs in the ROS environment of the robot operating system of the second computer unit, an operator operates the game handle to send a control instruction to the virtual cooperative robot in the second computer unit, and the virtual cooperative robot consists of a chassis, a mechanical arm and a clamping jaw. An operator sends linear velocity or angular velocity analog quantity for controlling the chassis to move to the virtual cooperative robot through a rocker of the gamepad, and the chassis executes forward, backward, left-turn and right-turn movements according to a control instruction and can also realize parallel movement of movement and rotation; an operator sends a control instruction for controlling the opening or closing of the clamping jaws and the movement or rotation of the tail end of the mechanical arm along the direction X, Y, Z under a Cartesian coordinate system to the virtual cooperative robot through the key operation of the game handle, and the clamping jaws or the mechanical arm perform stepping movement according to the key control instruction (switching value).
A virtual cooperative robot is a kind of simulation of a real cooperative robot. In the experiment stage, the motion and the action of the real cooperative robot in the actual operation are replaced by controlling the motion and the action of the virtual cooperative robot in the virtual cooperative robot operation module, and the typical mental state of an operator is induced by combining the operation environment and the operation result of the virtual cooperative robot, so that the experiment cost is reduced.
S3, when the mental state inducing task is to grab or transfer the object block, the log information of the operation generated in the step S2 is transmitted to the first computer unit from the second computer unit through TCP/IP communication (here, the local area network is 5GHz Wi-Fi) of the same local area network, and in the capture instruction triggering collection mode, when the first computer unit identifies the log information of the operation related to the typical mental state, the capture instruction is sent to collect and store the electroencephalogram signals of 30 channels of the operator in the typical mental state;
the method specifically comprises the following steps:
s301, capturing electroencephalogram signals of operators in typical mental states is achieved based on MATLAB. In order to read log information of the operation of the virtual cooperative robot in the ROS environment running on the second computer unit in real time, the first computer unit running with MATLAB is used as an ROS node, so that communication connection between the first computer unit and the node in the ROS network running on the second computer unit is realized in the same network, and bidirectional information transmission between the first computer unit and the second computer unit is finally realized;
s302, in an ROS Master running on a second computer unit, a node/rosout is a publisher of a topic/rosout _ agg, and the/Msg information of the topic/rosout _ agg is a control instruction received by the virtual cooperative robot, a calculation result generated by the received instruction and a j robot instruction execution result. Enabling a first computer unit node running MATLAB to subscribe the topic/rosout _ agg and circularly read the message content to acquire real-time log information of operation;
and S303, when the electroencephalogram signal capturing discriminator operated on the first computer unit identifies specific operation log information related to the typical mental state of an operator, generating a capturing instruction, and enabling the electroencephalogram data of 30 channels acquired by the electroencephalogram acquisition equipment to be wirelessly transmitted to the first computer unit through a data transmission network and stored. In the system, after an electroencephalogram signal capturing instruction is generated, electroencephalogram signals with the storage time of 5-60 seconds are collected by electroencephalogram collecting equipment.
S4, when the mental state inducing task is track tracking, the electroencephalogram signals in the step S1 are collected at regular time and stored;
in the timing acquisition mode, when an operator operates the virtual cooperative robot to complete a mental state inducing task, electroencephalogram signals of the virtual cooperative robot are acquired at regular time. The Task of timing acquisition comprises Task 0, Task 1, Task3 and Task 4.
S5, acquiring and storing the labels of the electroencephalogram signals acquired in the step S3 and the step S4 through the typical mental state subjective evaluation scale of the first computer unit, and completing electroencephalogram capture of the typical mental state of an operator; the label is stored as the name of the electroencephalogram data file, and the mental state label is obtained through the naming mode of the electroencephalogram data file;
and S6, after acquiring and storing electroencephalogram data of an operator in a certain typical mental state, circulating the step S2 to the step S5, continuously transmitting the operated log information from the second computer unit to the MATLAB on the first computer unit in real time along with the continuous operation of the operator on the task, judging whether the operated log information is target operation log information related to the typical mental state or not by the MATLAB, and if so, sending a capture instruction to acquire and store electroencephalogram signals of the operator, otherwise, not storing the electroencephalogram data at the moment.
In yet another embodiment of the present invention, a terminal device is provided that includes a processor and a memory for storing a computer program comprising program instructions, the processor being configured to execute the program instructions stored by the computer storage medium. The Processor may be a Central Processing Unit (CPU), or may be other general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf Programmable gate array (FPGA) or other Programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc., which is a computing core and a control core of the terminal, and is adapted to implement one or more instructions, and is specifically adapted to load and execute one or more instructions to implement a corresponding method flow or a corresponding function; the processor provided by the embodiment of the invention can be used for the operation of the electroencephalogram capture method of the typical mental state of an operator, and comprises the following steps:
the brain electrical acquisition equipment acquires brain electrical signals of frontal lobe, occipital lobe, parietal lobe and temporal lobe areas of cerebral cortex; the virtual cooperative robot is controlled by the game handle to move to complete an operation task, the category of the mental state inducing task is one or a combination of a plurality of track tracking, object block grabbing and object block transferring, and in the process of completing the inducing task, corresponding operation log information is generated when a control instruction related to the mechanical arm or the clamping jaw is executed; when the mental state inducing task is to grab or transfer an object block, the generated operation log information is transmitted to the first computer unit through the second computer unit, a capture instruction triggering acquisition mode is adopted, and when the first computer unit identifies the operation log information corresponding to the typical mental state, a capture instruction is sent to acquire an electroencephalogram signal in the typical mental state; when the mental state inducing task is track tracking, the electroencephalogram signals collected at fixed time are stored; and acquiring and storing the acquired electroencephalogram signal label through the typical mental state subjective evaluation scale of the first computer unit, and completing electroencephalogram capture of the typical mental state of the operator.
In still another embodiment of the present invention, the present invention further provides a storage medium, specifically a computer-readable storage medium (Memory), which is a Memory device in a terminal device and is used for storing programs and data. It is understood that the computer readable storage medium herein may include a built-in storage medium in the terminal device, and may also include an extended storage medium supported by the terminal device. The computer-readable storage medium provides a storage space storing an operating system of the terminal. Also, one or more instructions, which may be one or more computer programs (including program code), are stored in the memory space and are adapted to be loaded and executed by the processor. It should be noted that the computer-readable storage medium may be a high-speed RAM memory, or may be a non-volatile memory (non-volatile memory), such as at least one disk memory.
One or more instructions stored in the computer readable storage medium can be loaded and executed by the processor to realize the corresponding steps of the electroencephalogram capture method related to the typical mental state of the operator in the embodiment; one or more instructions in the computer-readable storage medium are loaded by the processor and perform the steps of:
the brain electrical acquisition equipment acquires brain electrical signals of frontal lobe, occipital lobe, parietal lobe and temporal lobe areas of cerebral cortex; the virtual cooperative robot is controlled by the game handle to move to complete an operation task, the category of the mental state inducing task is one or a combination of a plurality of track tracking, object block grabbing and object block transferring, and in the process of completing the inducing task, corresponding operation log information is generated when a control instruction related to the mechanical arm or the clamping jaw is executed; when the mental state inducing task is to grab or transfer an object block, the generated operation log information is transmitted to the first computer unit through the second computer unit, a capture instruction triggering acquisition mode is adopted, and when the first computer unit identifies the operation log information corresponding to the typical mental state, a capture instruction is sent to acquire an electroencephalogram signal in the typical mental state; when the mental state inducing task is track tracking, the electroencephalogram signals collected at fixed time are stored; and acquiring and storing the acquired electroencephalogram signal label through the typical mental state subjective evaluation scale of the first computer unit, and completing electroencephalogram capture of the typical mental state of the operator.
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. The components of the embodiments of the present invention generally described and illustrated in the figures herein may be arranged and designed in a wide variety of different configurations. Thus, the following detailed description of the embodiments of the present invention, presented in the figures, is not intended to limit the scope of the invention, as claimed, but is merely representative of selected embodiments of the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
According to the invention, a plurality of operators are tested according to the steps S1-S6, electroencephalogram signals of the operators in a typical mental state when the virtual cooperative robot in the virtual cooperative robot operation module is operated by the game handle are recorded in the test, and the operators are prevented from blinking eyes as much as possible in the test process. The result shows that the electroencephalogram capture system in the typical mental state of the operator can dynamically and adaptively acquire electroencephalogram data of the operator in different typical mental states, the reliability of the electroencephalogram data is high, and the electroencephalogram capture system is expressed in that the subjective evaluation is basically consistent with the design target of the evoked task and the separability of the acquired electroencephalogram data is high.
The invention has the beneficial effects that:
firstly, the interactive virtual cooperative robot operation task restores the influence factors of the mental state fluctuation of the operator in the actual operation environment to a higher degree, and has higher ecological efficiency and lower cost.
And secondly, the capture instruction triggers the acquisition mode to improve the induction efficiency of the stimulation material and the reliability of the stored electroencephalogram data, and meanwhile, the system efficiency is improved in a timing acquisition mode in an auxiliary mode and more electroencephalogram data are acquired.
Thirdly, the typical mental state subjective evaluation scale covers several mental states which often appear in a man-machine system and influence the decision and the behavior of an operator, so that the EEG data label deviation caused by individual difference is avoided, and the accuracy of the acquired EEG data label and the automation degree of a test system are improved. The method lays a good research foundation for researching the electroencephalogram response mechanism of mental state change, mental state identification and the relation between the mental state and the operation quality.
In summary, the electroencephalogram capture method and system for the typical mental state of the operator enable the operator to operate the virtual cooperative robot through the game handle according to a certain operation flow under the virtual cooperative robot operation module on the computer, monitor the operation condition of the virtual cooperative robot on the task in real time, and collect and store electroencephalogram data under the typical mental state based on the operation log information. The electroencephalogram data acquisition system can realize dynamic self-adaptive acquisition of electroencephalogram data in a typical mental state in an operation process, improve the flexibility of the electroencephalogram acquisition system and the effectiveness of the electroencephalogram data, and provide a research basis for researching an electroencephalogram response mechanism of mental state change of an operator, mental state identification, a correlation relationship between the mental state and operation quality and an operation compensation strategy.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The above-mentioned contents are only for illustrating the technical idea of the present invention, and the protection scope of the present invention is not limited thereby, and any modification made on the basis of the technical idea of the present invention falls within the protection scope of the claims of the present invention.
Claims (10)
1. An electroencephalogram capture system of a typical mental state of an operator, comprising:
the electroencephalogram signal acquisition module is used for acquiring electroencephalogram signals of an operator in a typical mental state by triggering acquisition modes through a capture instruction and comprises a first computer unit for acquiring the electroencephalogram signals and electroencephalogram acquisition equipment connected with the first computer unit;
the virtual cooperative robot operation module is used for controlling the virtual cooperative robot to complete typical mental state inducing tasks and continuously inducing an operator to generate a plurality of typical mental states, and comprises a second computer unit used for simulating the movement of the cooperative robot and a virtual cooperative robot control instruction sending device connected with the second computer unit;
the first computer unit is connected with the second computer unit through a data transmission network, and the functions of monitoring the dynamic process of the virtual cooperative robot operated by an operator and adaptively acquiring the electroencephalogram signals of the typical mental state of the operator are realized.
2. The electroencephalogram capture system for typical mental states of an operator according to claim 1, wherein the first computer unit comprises an electroencephalogram capture discriminator and a typical mental state subjective rating scale; the electroencephalogram signal capturing discriminator receives the operation log information of the second computer unit through a data transmission network and sends a capturing instruction to the electroencephalogram acquisition equipment to trigger electroencephalogram signal acquisition; typical mental state subjective rating scales are used to determine typical mental state categories.
3. The electroencephalograph capture system of typical mental states of an operator according to claim 2, wherein the typical mental state subjective rating scale includes a state of concentration, a state of non-concentration, a state of stress, a state of fatigue, a state of aggressiveness, and a state of negativity.
4. The electroencephalogram capture system for typical mental states of an operator according to claim 1, wherein the mode of acquiring electroencephalogram signals of the operator in the typical mental states by the electroencephalogram signal acquisition module further comprises a timing acquisition mode, and the typical mental state electroencephalogram signals of the operator in a time period without obvious operation log information in the timing acquisition operation process.
5. The electroencephalogram capture system for typical mental states of operators according to claim 1, wherein the virtual cooperative robot control instruction sending device is connected with the second computer unit through a USB wireless transmission mode and is used for sending the virtual cooperative robot control instructions, and the virtual cooperative robot receives and executes the virtual cooperative robot control instructions and simultaneously generates operation result log information.
6. The electroencephalograph capture system of typical mental states of an operator according to claim 5, wherein the virtual collaborative robot control instructions include instructions for controlling a chassis of the robot, and instructions for movement or action of the robot arm and the jaws.
7. The electroencephalogram capture system for typical mental states of operators according to claim 1, wherein the electroencephalogram acquisition equipment comprises an electroencephalogram cap, electrodes of the electroencephalogram cap are arranged in frontal lobe, occipital lobe, parietal lobe and temporal lobe areas of a brain, and the sampling frequency of the electroencephalogram acquisition equipment is 1000 Hz.
8. An electroencephalogram capture method of typical mental states of an operator, characterized in that the electroencephalogram capture system of typical mental states of an operator according to claim 1 is utilized, and comprises the following steps:
s1, collecting brain electrical signals of frontal lobe, occipital lobe, parietal lobe and temporal lobe areas of cerebral cortex by brain electrical collecting equipment;
s2, controlling the virtual cooperative robot to move through the game handle to complete a mental state inducing task, wherein the mental state inducing task is one or a combination of track tracking, object block grabbing and object block transferring, and corresponding operation log information is generated when a control instruction related to the mechanical arm or the clamping jaw is executed in the process of completing the inducing task;
s3, when the mental state inducing task is to grab or transfer the object block, the operation log information generated in the step S2 is transmitted from the second computer unit to the first computer unit through the data transmission network, a capture instruction is adopted to trigger the acquisition mode, and when the first computer unit identifies the operation log information corresponding to the typical mental state, the capture instruction is sent to acquire the electroencephalogram signal in the typical mental state;
s4, when the mental state inducing task is track tracking, the electroencephalogram signals in the step S1 are collected at regular time and stored;
s5, acquiring and storing the labels of the electroencephalogram signals acquired in the step S3 and the step S4 through the typical mental state subjective evaluation scale of the first computer unit, and completing the electroencephalogram capture of the typical mental state of the operator.
9. The method according to claim 8, wherein in step S2, the virtual cooperative robot operation module includes 5 mental state inducing tasks, and the controlling of the virtual cooperative robot by the game pad includes:
the game handle is connected to the second computer unit through the wireless USB receiver, the virtual cooperative robot runs in the ROS environment of a robot operating system of the second computer unit, the game handle is operated to send a control instruction to the virtual cooperative robot in the second computer unit, the virtual cooperative robot comprises a chassis, a mechanical arm and a clamping jaw, the control instruction for controlling the linear velocity or the angular velocity of the motion of the chassis is sent to the virtual cooperative robot through a rocker of the game handle, and the chassis executes forward, backward, left-turn and right-turn motions according to the control instruction to realize parallel motions of movement and rotation; and sending a control instruction for controlling the opening or closing of the clamping jaw and the movement or rotation of the tail end of the mechanical arm along the direction X, Y, Z under a Cartesian coordinate system to the virtual cooperative robot through keys of the game handle, wherein the clamping jaw or the mechanical arm generates stepping movement according to the key control instruction.
10. The method according to claim 8, wherein step S3 is specifically:
s301, taking the first computer unit running MATLAB as an ROS node, and under the same network, realizing communication connection between the first computer unit and the node in the ROS network running on the second computer unit, and realizing bidirectional information transmission between the first computer unit and the second computer unit;
s302, in an ROS Master operated by a second computer unit, a node/rosout is a publisher of a topic/rosout _ agg, and the/Msg information of the topic/rosout _ agg is a control instruction received by a virtual cooperative robot, a calculation result generated by the received instruction and a robot instruction execution result, so that a first computer unit node operated with MATLAB subscribes the topic/rosout _ agg and reads message contents circularly to acquire real-time log information of operation;
and S303, when the electroencephalogram signal capturing discriminator operated on the first computer unit identifies specific operation log information related to the typical mental state of an operator, generating a capturing instruction, wirelessly transmitting the electroencephalogram data acquired by the electroencephalogram acquisition equipment to the first computer unit through a data transmission network, and storing the electroencephalogram data, wherein the electroencephalogram acquisition equipment acquires the electroencephalogram signals with the storage time of 5-60 seconds.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110560459.7A CN113349781A (en) | 2021-05-21 | 2021-05-21 | Electroencephalogram capture method and system for typical mental state of operator |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110560459.7A CN113349781A (en) | 2021-05-21 | 2021-05-21 | Electroencephalogram capture method and system for typical mental state of operator |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113349781A true CN113349781A (en) | 2021-09-07 |
Family
ID=77526713
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110560459.7A Pending CN113349781A (en) | 2021-05-21 | 2021-05-21 | Electroencephalogram capture method and system for typical mental state of operator |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113349781A (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105563495A (en) * | 2016-02-01 | 2016-05-11 | 浙江大学 | Mechanical arm system controlled on basis of refined motor imagination electroencephalogram signals and method |
| CN105955486A (en) * | 2016-05-16 | 2016-09-21 | 西北工业大学 | Method for assisting teleoperation based on visual stimulation of brainwaves |
| CN112101285A (en) * | 2020-09-25 | 2020-12-18 | 西安交通大学 | Induction paradigm design and electroencephalogram identification method for typical mental state of remote operator of explosive-handling robot |
-
2021
- 2021-05-21 CN CN202110560459.7A patent/CN113349781A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105563495A (en) * | 2016-02-01 | 2016-05-11 | 浙江大学 | Mechanical arm system controlled on basis of refined motor imagination electroencephalogram signals and method |
| CN105955486A (en) * | 2016-05-16 | 2016-09-21 | 西北工业大学 | Method for assisting teleoperation based on visual stimulation of brainwaves |
| CN112101285A (en) * | 2020-09-25 | 2020-12-18 | 西安交通大学 | Induction paradigm design and electroencephalogram identification method for typical mental state of remote operator of explosive-handling robot |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112990074B (en) | VR-based multi-scene autonomous control mixed brain-computer interface online system | |
| Cao et al. | A hybrid brain computer interface system based on the neurophysiological protocol and brain-actuated switch for wheelchair control | |
| CN109605385B (en) | Rehabilitation assisting robot driven by hybrid brain-computer interface | |
| CA2882968C (en) | Facilitating generation of autonomous control information | |
| CN108297098A (en) | The robot control system and method for artificial intelligence driving | |
| CN101362334A (en) | Intelligent robot and operation method thereof | |
| CN106737760A (en) | A kind of human-like intelligent robot and man-machine communication's system | |
| CN109521880B (en) | Teleoperation robot system and method based on mixed bioelectricity signal driving | |
| CN110154056A (en) | Service robot and its man-machine interaction method | |
| Zhang et al. | A simple platform of brain-controlled mobile robot and its implementation by SSVEP | |
| CN114327048B (en) | Mechanical arm control method and system based on electroencephalogram signals and electrooculogram signals | |
| CN109215774A (en) | Real time remote diagnosis by feeling the pulse system | |
| CN106272446A (en) | The method and apparatus of robot motion simulation | |
| CN205750354U (en) | A kind of expression robot | |
| CN106502416B (en) | A kind of driving simulation system and its control method of intelligent recognition bimanual input | |
| CN113349781A (en) | Electroencephalogram capture method and system for typical mental state of operator | |
| CN108008810A (en) | A kind of confirmation method and system based on Mental imagery | |
| CN105182817A (en) | Intelligent shopping guide robot for detecting heart rate and electrocardiogram | |
| Meier et al. | Synchronized multimodal recording of a table setting dataset | |
| CN110640757A (en) | Multi-mode interaction method applied to intelligent robot and intelligent robot system | |
| CN110502120A (en) | It is dynamic to catch the virtual newscaster's system and realization side that data and deliberate action data combine | |
| CN106997449A (en) | Robot and face identification method with face identification functions | |
| CN204807972U (en) | Intelligent shopping guide robot of detectable rhythm of heart electrocardiogram | |
| CN112947752A (en) | Cooperative man-machine interaction control method based on intelligent equipment | |
| CN117524417A (en) | Autism rehabilitation training system, method, equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |