CN109407847A - A kind of multimodal human-computer interaction and man-machine environment test data synchronization system and method - Google Patents
A kind of multimodal human-computer interaction and man-machine environment test data synchronization system and method Download PDFInfo
- Publication number
- CN109407847A CN109407847A CN201811289880.3A CN201811289880A CN109407847A CN 109407847 A CN109407847 A CN 109407847A CN 201811289880 A CN201811289880 A CN 201811289880A CN 109407847 A CN109407847 A CN 109407847A
- Authority
- CN
- China
- Prior art keywords
- data
- module
- project
- equipment
- analysis
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/015—Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/013—Eye tracking input arrangements
Abstract
The present invention provides a kind of multimodal human-computer interaction and man-machine environment test data synchronization system and method, which includes: the back-stage management subsystem of human-computer interaction interface and control human-computer interaction interface；Project management module, experimental design module, synchronous data sampling module and Data Synchronization Analysis module are provided on human-computer interaction interface, project management module is embedded with project and creates region, task-driven region and item list area；Experimental design module is for providing experimental design interface；Synchronous data sampling module carries out multi-channel data synchronous acquisition for controlling synchronous data sampling equipment, and the data of acquisition are stored, and synchronous data sampling equipment includes physiological data monitoring device and/or eye movement data monitoring device；The data that the Data Synchronization Analysis module is used to acquire data synchronous acquisition equipment synchronize analysis.
Description
Technical field
The present invention relates to field of human-computer interaction, in particular to a kind of multimodal human-computer interaction and man-machine environment test data are same
Walk system and method.
Background technique
Human-computer interaction (Human-Computer Interaction, write a Chinese character in simplified form HCI), which refers to, uses certain between people and computer
Kind conversational language, the information exchanging process between people and computer with certain interactive mode, to complete to determine task.
In field of scientific studies such as psychological study, laboratory applications research, military project research, car steering researchs, often
It needs to simulate practical application scene to acquire the related data of people, therefore, realizes scientific research using human-computer interaction technology, be
A kind of research direction of great researching value.
Currently, any system or machine, even state-of-the-art artificial intelligence system at present, uses and ensures and always want
It is completed under certain environmental conditions directly or indirectly through people, therefore in order to make system or machine that there is good guarantee
Property, applicability and maximize play its efficiency, it is necessary to consider human factor, make one, machine, environment cooperate, be mutually adapted.Not
Coming the fields such as artificial intelligence and man-machine coordination, man-machine hybrid intelligent, intelligent human-machine interaction in human-computer interaction research field all needs
Human factor research is carried out, and is made one by the design of " human-centred ", the collaborative work of intelligence system, environment, it is mutually suitable
It answers.
Human Engineering (ergonomics) is the interactive scientific domain between a research people and system other elements, is
One is designed theory, principle, data, method to promote human welfare and optimize the profession of total system performance.People because
There are three professional research fields for engineering: physiology user's factor, cognition user's factor and tissue user's factor, wherein physiology user's factor is paid close attention to
It is people's human anatomy, anthropometry, physiology and biomechanical characterization when carrying out physiological activity.Recognize user's factor concern
Be cognitive process, such as perception, memory, reasoning and response process, they affect the interaction of people Yu system other elements.Phase
Closing project includes Mental Workload, decision process, skilled operation, human-computer interaction, human reriability, operating pressure and training etc., this
A little aspects all affect the design of man-machine system.Tissue user's factor is concerned with the optimization of sociotechnical system, including tissue
Structure, policy and process.
With the development of new science and technology, such as virtual/augmented reality interacts skill with emulation technology, eye movement tracking with eye control
Art, EEG measuring and brain-machine interaction interfacing, human-computer interaction and artificial intelligence, Cognitive Neuroscience and big data, cloud meter
The technologies such as calculation go deep into mixing together, and single pass human-computer interaction and single-channel data carry out human factor analysis and be no longer satisfied
Scientific research and application development need.
Single channel man-machine interactive system is only capable of being monitored and analyzed for a physical signs at present, monitoring and analysis
Principle are as follows: by collection mediums such as sensors, the single physical signs (for example, to skin electricity) of human body is monitored, monitoring knot
Fruit is uploaded to computer, carries out data analysis to collected physical signs by computer program, finally obtains experimental result,
The experimental result is used for the technical support etc. of related scientific research project.But this single channel monitors the physical signs list not only monitored
One, it can not achieve and monitoring is synchronized to multinomial physical signs, and be also left to be desired to the analysis of monitoring result.
Summary of the invention
In consideration of it, the embodiment of the invention provides a kind of multimodal human-computer interactions and man-machine environment test data synchronization system
And method, to eliminate or improve one or more defects existing in the prior art.
Technical scheme is as follows:
According to one aspect, the present invention provides a kind of multimodal human-computer interaction and man-machine environment test data synchronization system multi-pass
Road human-computer interaction and man-machine environment test data synchronization system, the system include: human-computer interaction interface and control human-computer interaction circle
The back-stage management subsystem in face；
Be provided on the human-computer interaction interface project management module, experimental design module, synchronous data sampling module and
Data Synchronization Analysis module, wherein
The project management module includes the following region being arranged on interactive interface:
Project creates region, for creating experimental project and storing newly-built experimental project information；
Task-driven region, for importing experimental project from local or cloud；
Item list area, for showing the experimental project list established in predetermined amount of time；
The experimental design module carries out experimental design for providing experimental design interface；
The synchronous data sampling module be used for by the back-stage management subsystem controls synchronous data sampling equipment into
Row multi-channel data synchronous acquisition, and the data of acquisition are stored, the synchronous data sampling equipment includes physiological data
Monitoring device and/or eye movement data monitoring device；
The data that the Data Synchronization Analysis module is used to acquire data synchronous acquisition equipment synchronize analysis.
Preferably, the Data Synchronization Analysis module includes: Physiological Analysis module, eye movement data analysis module and comprehensive system
Count analysis module；The Physiological Analysis module is for being counted and being divided to the data that the physiological data monitoring device acquires
Analysis；The data that the eye movement data analysis module is used to acquire the eye movement data monitoring device carry out statistics and analysis；Institute
State data and/or eye movement data monitoring of the comprehensive statistics analysis module for acquiring based on the physiological data monitoring device
The data of equipment acquisition carry out comprehensive analysis.
Preferably, be additionally provided on the human-computer interaction interface: data readback module is used to play back and checks the project
The experimental project that management module, experimental design module and/or synchronous data sampling module have been established or imported, and experiment is provided
Project editing interface.
Preferably, the synchronous data sampling equipment further includes at least one of following equipment: motion capture equipment, view
Frequency equipment, face acquisition equipment, movement monitoring device, eeg monitoring equipment and expression monitoring device；The human-computer interaction interface
On be additionally provided with at least one of lower module: behavioural analysis module, brain electricity analytical module, action behavior module and expression point
Analyse module；The behavioural analysis module is used for the motion capture equipment, video equipment and/or face acquisition equipment acquisition
Data carry out statistics and analysis；The brain electricity analytical module is for being counted and being divided to the data that eeg monitoring equipment acquires
Analysis；The motion analysis module is used to carry out statistics and analysis to the data of movement monitoring device acquisition；The Expression analysis mould
The data that block is used to acquire expression monitoring device carry out statistics and analysis；The comprehensive statistics analysis module is used for based on described
Face described in motion capture equipment, the video equipment acquires the number of the data of equipment acquisition, the movement monitoring device acquisition
According to the data of, eeg monitoring equipment acquisition, data, the physiological data monitoring device of expression monitoring device acquisition
The data of acquisition and/or the data of eye movement data monitoring device acquisition carry out comprehensive analysis.
Preferably, the project management module further includes the software information display area being arranged on interactive interface, is used for
Display system software essential information.
Preferably, when the newly-built region of the project is chosen, passes through newly-built information. dialog typing and create experimental project letter
It ceases, and newly-built experimental project is associated with to the path of setting；The item list area is chosen time display and experiment item has been established
Mesh list, this has been established experimental project list and is provided with project status function button, and associated with project data.
The present invention provides a kind of multimodal human-computer interaction and man-machine environment test data synchronous method according to another aspect,
Method includes the following steps:
Multimodal human-computer interaction interface is provided, the human-computer interaction interface is associated with back-stage management subsystem, described man-machine
Project management module, experimental design module, synchronous data sampling module and Data Synchronization Analysis module are provided on interactive interface,
Wherein, the project management module includes: that project creates region, task-driven region and item list area；
Region is created by the project of the project management module to create experimental project and store newly-built experimental project letter
Breath by task-driven region to import experimental project from local or cloud, and shows pre- timing by item list area
Between the experimental project list established in section；
Experimental design interface is provided by the experimental design module, carries out experimental design；
By the synchronous data sampling module come by the back-stage management subsystem controls synchronous data sampling equipment into
Row multi-channel data synchronous acquisition, and the data of acquisition are stored, the synchronous data sampling equipment includes physiological data
Monitoring device and/or eye movement data monitoring device；
Analysis is synchronized by the data that the Data Synchronization Analysis module acquires data synchronous acquisition equipment.
Preferably, the Data Synchronization Analysis module includes: Physiological Analysis module, eye movement data analysis module and comprehensive system
Count analysis module；The method also includes: the number that the physiological data monitoring device is acquired by the Physiological Analysis module
According to progress statistics and analysis；The data that the eye movement data monitoring device acquires are carried out by the eye movement data analysis module
Statistics and analysis；The data that are acquired by the comprehensive statistics analysis module based on the physiological data monitoring device and/or
The data of the eye movement data monitoring device acquisition carry out comprehensive analysis.
Preferably, data readback module and behavioural analysis module are additionally provided on the human-computer interaction interface；The method
Further include: the experimental project that the project management module has been established or imported is checked by data readback module playback,
And provide experimental project editing interface；It is united by the data that the behavioural analysis module acquires the motion capture equipment
Meter and analysis；The comprehensive statistics analysis module be also used to based on the motion capture equipment acquire data, the physiology number
Comprehensive analysis is carried out according to the data of monitoring device acquisition and/or the data of eye movement data monitoring device acquisition.
Preferably, data readback module and behavioural analysis module are additionally provided on the human-computer interaction interface；The data
Synchronous acquisition equipment further includes at least one of following equipment: motion capture equipment, face acquisition equipment, is moved video equipment
Make monitoring device, eeg monitoring equipment and expression monitoring device；The method also includes:
Check that the project management module, experimental design module and/or data are same by data readback module playback
The experimental project that step acquisition module has been established or imported, and experimental project editing interface is provided；Pass through the behavioural analysis mould
Block carries out statistics and analysis to the data of the motion capture equipment, video equipment and/or face acquisition equipment acquisition；Pass through institute
It states brain electricity analytical module and statistics and analysis is carried out to the data that eeg monitoring equipment acquires；By the motion analysis module to dynamic
The data for making monitoring device acquisition carry out statistics and analysis；The number that expression monitoring device is acquired by the Expression analysis module
According to progress statistics and analysis；The comprehensive statistics analysis module is used to be based on the motion capture equipment, video equipment institute
The number that data, the eeg monitoring equipment of the data, the movement monitoring device acquisition of stating facial acquisition equipment acquisition acquire
According to the data of, expression monitoring device acquisition, the data of physiological data monitoring device acquisition and/or the eye movement data
The data of monitoring device acquisition carry out comprehensive analysis.
Preferably, when the newly-built region of the project is chosen, passes through newly-built information. dialog typing and create experimental project letter
It ceases, and newly-built experimental project is associated with to the path of setting；The item list area is chosen time display and experiment item has been established
Mesh list, this has been established experimental project list and is provided with project status function button, and associated with project data.
Preferably, the method also includes: pass through the project management module be arranged on interactive interface software letter
Cease display area display system software essential information.
According to another aspect, the present invention also provides a kind of computer readable storage medium, it is stored thereon with computer program,
The step of program realizes method as previously described when being executed by processor.
Multimodal human-computer interaction of the invention can be realized with man-machine environment test data synchronization system and method to multinomial
Index synchronizes monitoring, and can carry out data analysis to the monitoring result of the monitoring device to match with this system, together
Shi Shixian carries out data analysis to the monitoring result of existing monitoring device.
Attendant advantages, purpose and feature of the invention will will be partly illustrated in the following description, and will be right
Partly become obvious after research hereafter in those of ordinary skill in the art, or can practice according to the present invention and obtain
Know.Objectives and other advantages of the present invention can be by specifically noting in written explanation and its claims and attached drawing
Structure is realized and is obtained.
It will be appreciated by those skilled in the art that can be not limited to the objects and advantages that the present invention realizes above specific
It is described, and the above and other purpose that the present invention can be realized will be more clearly understood according to following detailed description.
Detailed description of the invention
The drawings described herein are used to provide a further understanding of the present invention, constitutes part of this application, not
Constitute limitation of the invention.In the accompanying drawings:
Fig. 1 is that multimodal human-computer interaction and the structure of man-machine environment test data synchronization system are shown in one embodiment of the invention
It is intended to；
Fig. 2 is the structure of multimodal human-computer interaction and man-machine environment test data synchronization system in another embodiment of the present invention
Schematic diagram；
Fig. 3 be another embodiment of the present invention in multimodal human-computer interaction with it is man-machine in man-machine environment test data synchronization system
The schematic diagram of interactive interface.
Specific embodiment
To make the objectives, technical solutions, and advantages of the present invention clearer, right below with reference to embodiment and attached drawing
The present invention is described in further details.Here, exemplary embodiment and its explanation of the invention is used to explain the present invention, but simultaneously
It is not as a limitation of the invention.
Here, it should also be noted that, in order to avoid having obscured the present invention because of unnecessary details, in the accompanying drawings only
Show with closely related structure and/or processing step according to the solution of the present invention, and be omitted little with relationship of the present invention
Other details.
It should be emphasized that term "comprises/comprising" refers to the presence of feature, element, step or component when using herein, but simultaneously
It is not excluded for the presence or additional of one or more other features, element, step or component.
Here, it should also be noted that, if term " connection " can not only refer to herein directly to be connected without specified otherwise
It connects, can also indicate being indirectly connected with there are intermediary.
Hereinafter, the embodiment of the present invention will be described with reference to the drawings.In the accompanying drawings, identical appended drawing reference represents identical
Or similar component or same or like step.
As shown in Figure 1, the system includes: back-stage management subsystem 100 and human-computer interaction interface 110, wherein back-stage management
Subsystem 100 is used to control the display of human-computer interaction interface.In an embodiment of the present invention, back-stage management subsystem 100 can be with
It is realized by the processor of operation computer software or microprocessor etc., back-stage management subsystem 100 connects data communication module
120, for being communicated via data communication module 120 with wireless sensor and other external device (ED)s etc., back-stage management subsystem
System 100 is also connected with server 130, which can be local server or cloud server.Back-stage management subsystem 100 with
Wireless mode connects the first synchronous data sampling equipment 140, the second synchronous data sampling equipment 150 and third synchronous data sampling
Equipment 160, to acquire different monitoring data respectively, to synchronize processing；It can also be using the first number of wired mode connection
According to synchronous acquisition equipment 140, the second synchronous data sampling equipment 150 and third synchronous data sampling equipment 160, to adopt respectively
Collect different monitoring data, to synchronize processing.As an example, the first synchronous data sampling equipment 140, the second data are same
Step acquisition equipment 150 and third synchronous data sampling equipment 160 can be motion capture equipment, physiological data monitoring device respectively
With eye movement data monitoring device, but the present invention is not limited thereto, further includes more or fewer synchronous data sampling equipment.
In an embodiment of the present invention, project management module 111, experimental design module are provided on human-computer interaction interface
112, synchronous data sampling module 113 and Data Synchronization Analysis module.Wherein, Data Synchronization Analysis module can include: physiology point
Analyse module 116, eye movement data analysis module 117 and comprehensive statistics analysis module 118 etc., in another embodiment, human-computer interaction
Data readback module 114 and behavioural analysis module 116 are also provided on interface.Fig. 3 is shown in one embodiment of the invention
The schematic example of human-computer interaction interface.
Project management module 111 is used for: being shown to the essential information of this system, grassroot project, lead-in item and query term
Mesh list etc. is managed.In the present embodiment, project is often referred to experimental project.Project management module includes being arranged in interactive interface
Upper multiple regions specifically include software information region, grassroot project region, lead-in item region and item list area.Work as boundary
When " project management " above face is chosen, as shown in figure 3, can show these areas of project management module 111 on interface
Domain.The position arrangement of each region is merely illustrative on interactive interface in Fig. 3, can also be changed.
There is referred to as system essential information display area in software information region, for showing this system essential information, for example (,) it is aobvious
Show version number, issue date, softdog residue pot life etc..
Project creates region for grassroot project and stores grassroot project information, and grassroot project can establish (storage) at this
Ground (the machine) also can establish on Cloud Server, establishes and claims the machine project in local project, establish the item in Cloud Server
Mesh claims cloud project.It, can be newly-built in fact by newly-built information. dialog typing when newly-built the machine project is selected in the newly-built region of project
Project information is tested, and newly-built experimental project is associated with to the path of setting.As an example, after clicking newly-built the machine project, it can bullet
Information. dialog is created out, and the grassroot project item of information that can input information, such as project name, project institute are equipped in the dialog box
Want storing path, confirmation, cancellation etc., wherein Path selection button can be equipped with by the wanted storing path of project, to carry out project guarantor
Deposit the selection in path.After the content for completing the dialog box is filled in and confirmed, item preview module can be entered, item preview module can
The corresponding record list of current project is in human-computer interaction interface in bulleted list for generating for the project management module
Upper carry out preview, and the project relative recording in the record list of preview is edited.Such as: it clicks project and creates in region
Newly-built the machine project, pop-up creation information. dialog inputs respective name in the project name item of information in dialog box,
The project path to be saved can be selected according to experiment actual demand, after clicking confirmation, successfully create a new project；This
Outside, newly-built cloud project also may be selected, the process for creating cloud project is similar with creation the machine project, and only store path is different,
This is repeated no more.
Task-driven region is used to import experimental project from local or cloud, i.e., for import established the machine project or
Cloud project.For example, when clicking importing the machine project in task-driven region Path selection dialog box can be popped up, selection will be led
Position corresponding to the machine project entered can complete the importing of the machine project.Cloud project can also be imported similarly.
Item list area is used to show the experimental project list established in predetermined amount of time.For example, item list area
History before can be used for showing this week and this week creates bulleted list, is also possible to the short interior history creation of longer or shorter time
Bulleted list.The experimental project list that has been established shown in item list area is provided with project status function button, and with
Project data is associated.It is somebody's turn to do as an example, the destination item in the bulleted list shown in double-click item list area can enter
The project page of project, wherein the symbol button that can click change, the symbol can be respectively arranged on each project name of this week list
Number button show that the fixation project that the symbol button becomes searching convenient for user after being clicked is shown before being clicked for on-fixed project
Show, on-fixed project, which is shown, show with fixed project with switching over again to symbol button progress same operation (clicking).To non-
After fixterm purpose symbol button is clicked, item list area automatically generates fixed project team and on-fixed project team.As
The project that fixed project is shown, can be automatically located in the top of item list area, when fixed project is more than two, solid
Determine to arrange downwards in project display group by the sequence of the last opening time, to facilitate experimenter quickly to find this
Mesh；As the project that on-fixed project is shown, can arrange downwards in order, when on-fixed project is more than two, non-solid
Determine in project display group by the last opening time sequence arrangement downwards.In addition, clicking the entry name in bulleted list by right key
Claim the edit operation of file where can deleting in list area to the project and open it.
In project management module after typing content, item preview interface can be entered, for synchronous to human-machine interaction data
The list that project management in system generates carries out preview.It may be selected to close project management module after finished item preview, and can
Into experimental design module.
Experimental design module 112 is for providing experimental design interface, and to carry out experimental design, specific design method is according to real
The actual demand tested determines, the experimental design data including experimental design plan are produced after the completion of experimental design.It completes
After the operation of experimental design module, synchronous data sampling module 113 can be entered.
Synchronous data sampling module 113 is for the information in information and experimental design based on the typing of project management module institute
Carry out synchronous data sampling and/or editor.Project management module, the information of experimental design module institute typing for example including but it is unlimited
In the project information of creation, picture stimulant, visual transmission object etc..
Matched multiple synchronous data sampling equipment are equipped with during synchronous data sampling, matched synchronous data sampling is set
Standby may include the equipment for monitoring the various parameters of testee, such as store and emit wired/line sensor of data, movement
Capture equipment (such as motion capture recorder), heart rate detection sensor, eye movement data monitor, video equipment, face acquisition are set
At least two kinds of equipment in standby, movement monitoring device, eeg monitoring equipment and expression monitoring device etc., it is possible to use existing skill
Other synchronous data sampling equipment in art.Specifically, synchronous data sampling module 113 can be used for being based on generating in experimental design
Planning of experiments carry out multi-channel data synchronous acquisitions by the multiple synchronous data sampling equipment of back-stage management subsystem controls, and
The data of acquisition are stored, synchronous data sampling equipment preferably at least includes raw for detecting at least one of testee
It manages the physiological data monitoring device of parameter (such as heart rate, skin electricity situation) and the eye movement data monitoring for monitoring eye movement data is set
It is standby, as an example, synchronous data sampling equipment may include physiological data monitoring device, eye movement data monitoring device and for catching
Catch the motion capture equipment of testee's movement.
The data that behavioural analysis module 115 is used to acquire motion capture equipment carry out statistics and analysis, tested to judge
The movement of examination person.
The data that Physiological Analysis module 116 is used to acquire physiological data monitoring device carry out statistics and analysis, with judgement
The physiological status of testee.
The data that eye movement data analysis module 117 is used to acquire eye movement data monitoring device carry out statistics and analysis.
Comprehensive statistics analysis module is used to carry out comprehensive analysis: motion capture based on at least 2 class data in following data
The data of the data of equipment acquisition, the data of physiological data monitoring device acquisition and the acquisition of eye movement data monitoring device.For example, base
It can be used for comprehensive analysis testee whether happy, anxiety, fear etc. in the heart rate data and eye movement data measured.Based on multinomial
The comprehensive statistics and analysis of the data of acquisition, the more accurate or more fully state judgement of available testee.
In alternative embodiment of the present invention, synchronous data sampling equipment includes at least one of following equipment: movement is caught
Catch equipment, video equipment, face acquisition equipment, movement monitoring device, eeg monitoring equipment and expression monitoring device.
In an embodiment of the present invention, Data Synchronization Analysis module may also include that behavioural analysis module, brain electricity analytical mould
At least one of block, action behavior module and Expression analysis module.At this point, being additionally provided on human-computer interaction interface with lower module
At least one of: behavioural analysis module, brain electricity analytical module, action behavior module and Expression analysis module.
At this point, behavioural analysis module is used for the number to motion capture equipment, video equipment and/or face acquisition equipment acquisition
According to progress statistics and analysis.The data that brain electricity analytical module is used to acquire eeg monitoring equipment carry out statistics and analysis.Movement
Analysis module is used to carry out statistics and analysis, in one embodiment of the invention, motion capture to the data of movement monitoring device acquisition
Equipment can be identical or different equipment with movement monitoring device.Expression analysis module is used for the number acquired to expression monitoring device
According to statistics and analysis is carried out, in an embodiment of the present invention, expression monitoring device can be same equipment with face acquisition equipment
Or distinct device.
Comprehensive statistics analysis module is used for based on motion capture equipment, video equipment and/or face acquisition equipment acquisition
Data, the data of movement monitoring device acquisition, the data of eeg monitoring equipment acquisition, the data of expression monitoring device acquisition, life
Manage two or more kinds of data in the data of data monitoring equipment acquisition and/or the data of eye movement data monitoring device acquisition
Carry out comprehensive analysis.
In attached drawing as shown in Figure 2, other than with the module in Fig. 1, further it is shown that brain electricity analytical module 119,
Motion analysis module 121 and Expression analysis module 122.
In an alternative embodiment of the invention, after synchronous data sampling module completes synchronous data sampling, data can be entered
Playback module, for project (such as project management module, experimental design module and/or the synchronous data sampling module to record
The experimental project for having been established or having imported) play back and checks, and experimental project editor circle can be provided according to experiment specific requirements
Face, to be edited.To select which data to carry out comprehensive analysis to.
Multimodal human-computer interaction as above and man-machine environment test data synchronization system of the invention can be realized to multinomial finger
Mark synchronizes monitoring, and can carry out data analysis to the monitoring result of the monitoring device to match with this system, simultaneously
It realizes and data analysis is carried out to the monitoring result of existing monitoring device.
In the preferred embodiment of the present invention, more or fewer modules can be selected based on actual conditions, as long as can be realized
The synchronous detection of multinomial data, within the scope of the present invention.
It correspondingly, can the present invention also provides a kind of multimodal human-computer interaction and man-machine environment test data synchronous method
Data (multichannel) of the synchronous acquisition from multiple synchronous data sampling equipment, and realize the synthesis point to testee's state
Analysis.The method of the present invention includes following steps:
Multimodal human-computer interaction interface is provided, the human-computer interaction interface is associated with back-stage management subsystem, described man-machine
Project management module, experimental design module, synchronous data sampling module and Data Synchronization Analysis module are provided on interactive interface,
Wherein, the project management module includes: that project creates region, task-driven region and item list area；
Region is created by the project of the project management module to create experimental project and store newly-built experimental project letter
Breath by task-driven region to import experimental project from local or cloud, and shows pre- timing by item list area
Between the experimental project list established in section；
Experimental design interface is provided by the experimental design module, carries out experimental design；
By the synchronous data sampling module come by the back-stage management subsystem controls synchronous data sampling equipment into
Row multi-channel data synchronous acquisition, and the data of acquisition are stored, the synchronous data sampling equipment includes physiological data
Monitoring device and/or eye movement data monitoring device；
Analysis is synchronized by the data that the Data Synchronization Analysis module acquires data synchronous acquisition equipment.
Preferably, Data Synchronization Analysis module can include: Physiological Analysis module, eye movement data analysis module and comprehensive statistics
Analysis module；
The method may also include that
Statistics and analysis is carried out to the data that physiological data monitoring device acquires by Physiological Analysis module；
Statistics and analysis is carried out to the data that the eye movement data monitoring device acquires by eye movement data analysis module；
The data and/or the eye movement acquired by comprehensive statistics analysis module based on the physiological data monitoring device
The data of data monitoring equipment acquisition carry out comprehensive analysis.
Preferably, be additionally provided on human-computer interaction interface data readback module, behavioural analysis module, brain electricity analytical module,
Action behavior module and Expression analysis module；Synchronous data sampling equipment further includes at least one of following equipment: movement is caught
Catch equipment, video equipment, face acquisition equipment, movement monitoring device, eeg monitoring equipment and expression monitoring device；
At this point, the method also includes:
Check that the project management module, experimental design module and/or data are synchronized and adopted by the playback of data readback module
The experimental project that collection module has been established or imported, and experimental project editing interface is provided；
By behavioural analysis module to motion capture equipment, video equipment and/or face acquisition equipment acquisition data into
Row statistics and analysis；
Statistics and analysis is carried out to the data that eeg monitoring equipment acquires by brain electricity analytical module；
Statistics and analysis is carried out by data of the motion analysis module to movement monitoring device acquisition；
Statistics and analysis is carried out to the data that expression monitoring device acquires by Expression analysis module；
Comprehensive statistics analysis module is used for based on motion capture equipment, video equipment and/or face acquisition equipment acquisition
Data, the data of movement monitoring device acquisition, the data of eeg monitoring equipment acquisition, the data of expression monitoring device acquisition, life
The data of the data and/or the acquisition of eye movement data monitoring device of managing the acquisition of data monitoring equipment carry out comprehensive analysis.
The invention further relates to storage mediums, can store computer program code thereon, when program code is performed
It may be implemented the various embodiments of method of the invention, which can be tangible media, such as CD, USB flash disk, soft
Disk, hard disk etc..
It should be clear that the invention is not limited to specific configuration described above and shown in figure and processing.
For brevity, it is omitted here the detailed description to known method.In the above-described embodiments, several tools have been described and illustrated
The step of body, is as example.But method process of the invention is not limited to described and illustrated specific steps, this field
Technical staff can be variously modified, modification and addition after understanding spirit of the invention, or suitable between changing the step
Sequence.
Those of ordinary skill in the art should be it will be appreciated that each example described in conjunction with embodiment disclosed herein
The component part of property, system and method can be realized with the combination of hardware, software or the two.Specifically actually with hardware also
It is software mode to execute, the specific application and design constraint depending on technical solution.Professional technician can be to every
A specific application uses different methods to achieve the described function, but this realizes it is not considered that beyond of the invention
Range.When realizing in hardware, it may, for example, be electronic circuit, specific integrated circuit (ASIC), firmware appropriate, insert
Part, function card etc..When being realized with software mode, element of the invention is used to execute program or the generation of required task
Code section.Perhaps code segment can store in machine readable media program or the data-signal by carrying in carrier wave is passing
Defeated medium or communication links are sent." machine readable media " may include any medium for capableing of storage or transmission information.
The example of machine readable media includes electronic circuit, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), soft
Disk, CD-ROM, CD, hard disk, fiber medium, radio frequency (RF) link, etc..Code segment can be via such as internet, inline
The computer network of net etc. is downloaded.
It should also be noted that, the exemplary embodiment referred in the present invention, is retouched based on a series of step or device
State certain methods or system.But the present invention is not limited to the sequence of above-mentioned steps, that is to say, that can be according in embodiment
The sequence referred to executes step, may also be distinct from that the sequence in embodiment or several steps are performed simultaneously.
Software can be placed in random access memory (RAM), memory, read-only memory (ROM), electrically programmable ROM, electric erasable
Any other form of depositing well known in programming ROM, register, hard disk, moveable magnetic disc, CD-ROM or technical field
In storage media.
In the present invention, feature embodiment description and/or illustrated for one can be in one or more other realities
It applies in mode and uses in the same manner or in a similar manner, and/or combine with the feature of other embodiments or replace other
The feature of embodiment.
The foregoing is only a preferred embodiment of the present invention, is not intended to restrict the invention, for the skill of this field
For art personnel, the embodiment of the present invention can have various modifications and variations.All within the spirits and principles of the present invention, made
Any modification, equivalent substitution, improvement and etc. should all be included in the protection scope of the present invention.
Claims (11)
1. a kind of multimodal human-computer interaction and man-machine environment test data synchronization system, which is characterized in that the system includes: man-machine
The back-stage management subsystem of interactive interface and control human-computer interaction interface；
Project management module, experimental design module, synchronous data sampling module and data are provided on the human-computer interaction interface
Synchronization Analysis module, wherein
The project management module includes the following region being arranged on interactive interface:
Project creates region, for creating experimental project and storing newly-built experimental project information；
Task-driven region, for importing experimental project from local or cloud；
Item list area, for showing the experimental project list established in predetermined amount of time；
The experimental design module carries out experimental design for providing experimental design interface；
The synchronous data sampling module is used to carry out by the back-stage management subsystem controls synchronous data sampling equipment more
Channel data synchronous acquisition, and the data of acquisition are stored, the synchronous data sampling equipment includes physiological data monitoring
Equipment and/or eye movement data monitoring device；
The data that the Data Synchronization Analysis module is used to acquire data synchronous acquisition equipment synchronize analysis.
2. system according to claim 1, which is characterized in that the Data Synchronization Analysis module includes: Physiological Analysis mould
Block, eye movement data analysis module and comprehensive statistics analysis module；
The data that the Physiological Analysis module is used to acquire the physiological data monitoring device carry out statistics and analysis；
The data that the eye movement data analysis module is used to acquire the eye movement data monitoring device carry out statistics and analysis；
The data and/or the eye movement number that the comprehensive statistics analysis module is used to acquire based on the physiological data monitoring device
Comprehensive analysis is carried out according to the data of monitoring device acquisition.
3. system according to claim 1, which is characterized in that be additionally provided on the human-computer interaction interface:
Data readback module is used to play back and checks the project management module, experimental design module and/or synchronous data sampling
The experimental project that module has been established or imported, and experimental project editing interface is provided.
4. system according to claim 1, which is characterized in that the synchronous data sampling equipment further includes in following equipment
At least one: motion capture equipment, video equipment, face acquisition equipment, movement monitoring device, eeg monitoring equipment and expression
Monitoring device；
Be additionally provided on the human-computer interaction interface at least one of lower module: behavioural analysis module, brain electricity analytical module,
Action behavior module and Expression analysis module；
The behavioural analysis module is used for the number to the motion capture equipment, video equipment and/or face acquisition equipment acquisition
According to progress statistics and analysis；
The data that the brain electricity analytical module is used to acquire eeg monitoring equipment carry out statistics and analysis；
The motion analysis module is used to carry out statistics and analysis to the data of movement monitoring device acquisition；
The data that the Expression analysis module is used to acquire expression monitoring device carry out statistics and analysis；
The comprehensive statistics analysis module is used to adopt based on the motion capture equipment, the video equipment and/or the face
Collect the data of equipment acquisition, the data of the movement monitoring device acquisition, the data of eeg monitoring equipment acquisition, the table
Data, the data and/or the eye movement data monitoring device of physiological data monitoring device acquisition of feelings monitoring device acquisition
The data of acquisition carry out comprehensive analysis.
5. system according to claim 1, it is characterised in that:
The project management module further includes the software information display area that interactive interface is arranged in and is, is used for display system software
Essential information；
When newly-built the machine project is selected in the newly-built region of the project, passes through newly-built information. dialog typing and create experimental project
Information, and newly-built experimental project is associated with to the path of setting；
The experimental project list that has been established shown in the item list area is provided with project status function button, and with item
Mesh number is according to associated.
6. a kind of multimodal human-computer interaction and man-machine environment test data synchronous method, which is characterized in that this method includes following
Step:
Multimodal human-computer interaction interface is provided, the human-computer interaction interface is associated with back-stage management subsystem, the human-computer interaction
Project management module, experimental design module, synchronous data sampling module and Data Synchronization Analysis module are provided on interface,
In, the project management module includes: that project creates region, task-driven region and item list area；
Region is created by the project of the project management module to create experimental project and store newly-built experimental project information, is led to
It crosses task-driven region to import experimental project from local or cloud, and is shown in predetermined amount of time by item list area
The experimental project list of foundation；
Experimental design interface is provided by the experimental design module, carries out experimental design；
It is carried out by the synchronous data sampling module by the back-stage management subsystem controls synchronous data sampling equipment more
Channel data synchronous acquisition, and the data of acquisition are stored, the synchronous data sampling equipment includes physiological data monitoring
Equipment and/or eye movement data monitoring device；
Analysis is synchronized by the data that the Data Synchronization Analysis module acquires data synchronous acquisition equipment.
7. according to the method described in claim 6, it is characterized in that, the Data Synchronization Analysis module includes: Physiological Analysis mould
Block, eye movement data analysis module and comprehensive statistics analysis module；
The method also includes:
Statistics and analysis is carried out to the data that the physiological data monitoring device acquires by the Physiological Analysis module；
Statistics and analysis is carried out to the data that the eye movement data monitoring device acquires by the eye movement data analysis module；
The data and/or the eye movement acquired by the comprehensive statistics analysis module based on the physiological data monitoring device
The data of data monitoring equipment acquisition carry out comprehensive analysis.
8. according to the method described in claim 6, it is characterized in that, being additionally provided with data readback mould on the human-computer interaction interface
Block, behavioural analysis module, brain electricity analytical module, action behavior module and Expression analysis module；The synchronous data sampling equipment
It further include at least one of following equipment: motion capture equipment, video equipment, face acquisition equipment, movement monitoring device, brain
Pyroelectric monitor equipment and expression monitoring device；
The method also includes:
Check that the project management module, experimental design module and/or data are synchronized and adopted by data readback module playback
The experimental project that collection module has been established or imported, and experimental project editing interface is provided；
By the behavioural analysis module to the number of the motion capture equipment, video equipment and/or face acquisition equipment acquisition
According to progress statistics and analysis；
Statistics and analysis is carried out to the data that eeg monitoring equipment acquires by the brain electricity analytical module；
Statistics and analysis is carried out by data of the motion analysis module to movement monitoring device acquisition；
Statistics and analysis is carried out to the data that expression monitoring device acquires by the Expression analysis module；
The comprehensive statistics analysis module is used to adopt based on the motion capture equipment, the video equipment and/or the face
Collect the data of equipment acquisition, the data of the movement monitoring device acquisition, the data of eeg monitoring equipment acquisition, the table
Data, the data and/or the eye movement data monitoring device of physiological data monitoring device acquisition of feelings monitoring device acquisition
The data of acquisition carry out comprehensive analysis.
9. according to the method described in claim 6, it is characterized by:
When the newly-built region of the project is chosen, experimental project information is created by newly-built information. dialog typing, and will create
Experimental project is associated with the path of setting；
The item list area is chosen time display and experimental project list has been established, this has been established experimental project list and is provided with
Project status function button, and it is associated with project data.
10. according to the method described in claim 6, it is characterized in that, the method also includes:
It is basic by the software information display area display system software of the project management module being arranged on interactive interface
Information.
11. a kind of computer readable storage medium, is stored thereon with computer program, which is characterized in that the program is by processor
It is realized when execution such as the step of any one of claim 6 to 10 the method.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811289880.3A CN109407847A (en) | 2018-10-31 | 2018-10-31 | A kind of multimodal human-computer interaction and man-machine environment test data synchronization system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811289880.3A CN109407847A (en) | 2018-10-31 | 2018-10-31 | A kind of multimodal human-computer interaction and man-machine environment test data synchronization system and method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109407847A true CN109407847A (en) | 2019-03-01 |
Family
ID=65471030
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811289880.3A Pending CN109407847A (en) | 2018-10-31 | 2018-10-31 | A kind of multimodal human-computer interaction and man-machine environment test data synchronization system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109407847A (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104616136A (en) * | 2015-02-15 | 2015-05-13 | 王任直 | Clinical data management system |
| CN105117116A (en) * | 2015-05-06 | 2015-12-02 | 湖南工学院 | System and method for acquiring behavior data generated by executing control task |
| CN106443032A (en) * | 2016-09-30 | 2017-02-22 | 内蒙古蒙牛乳业（集团）股份有限公司 | Data acquisition method and device |
| WO2017112679A1 (en) * | 2015-12-22 | 2017-06-29 | University Of Florida Research Foundation, Inc. | Multimodal closed-loop brain-computer interface and peripheral stimulaiton for neuro-rehabilitation |
| CN107137096A (en) * | 2017-06-22 | 2017-09-08 | 中国科学院心理研究所 | A kind of multi-modal physiology and behavioral data merge acquisition system |
| CN107753013A (en) * | 2017-11-09 | 2018-03-06 | 广州众健医疗科技有限公司 | A kind of cardiac electric data sampler data processing method |
-
2018
- 2018-10-31 CN CN201811289880.3A patent/CN109407847A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104616136A (en) * | 2015-02-15 | 2015-05-13 | 王任直 | Clinical data management system |
| CN105117116A (en) * | 2015-05-06 | 2015-12-02 | 湖南工学院 | System and method for acquiring behavior data generated by executing control task |
| WO2017112679A1 (en) * | 2015-12-22 | 2017-06-29 | University Of Florida Research Foundation, Inc. | Multimodal closed-loop brain-computer interface and peripheral stimulaiton for neuro-rehabilitation |
| CN106443032A (en) * | 2016-09-30 | 2017-02-22 | 内蒙古蒙牛乳业（集团）股份有限公司 | Data acquisition method and device |
| CN107137096A (en) * | 2017-06-22 | 2017-09-08 | 中国科学院心理研究所 | A kind of multi-modal physiology and behavioral data merge acquisition system |
| CN107753013A (en) * | 2017-11-09 | 2018-03-06 | 广州众健医疗科技有限公司 | A kind of cardiac electric data sampler data processing method |
Non-Patent Citations (2)
| Title |
|---|
| 九天科技: "《精通微信应用、技巧与营销》", 30 June 2014, 中国铁道出版社 * |
| 邓奕等: "《Altium Designer原理图与PCB设计》", 30 September 2015, 华中科技大学出版社 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105431851A (en) | A healthcare decision support system for tailoring patient care | |
| CN101385641A (en) | Wave analysis method and device of physiological parameter | |
| CN107657990A (en) | A kind of auxiliary of operation record typing supports system and method | |
| Caballero et al. | Visual analytics for evaluating clinical pathways | |
| Krutanard et al. | Discovering organizational process models of resources in a hospital using Role Hierarchy Miner | |
| CN112927793A (en) | Consciousness disturbance electroencephalogram data analysis system based on Web | |
| JP2023526412A (en) | Information processing method, electronic device, and computer storage medium | |
| CN109549655A (en) | A kind of Experiment of Psychology and physiological monitoring system and its application method | |
| CN109243549A (en) | A kind of intelligent follow-up method, device and server | |
| CN109299345B (en) | Project preview system and method for multi-channel human-computer environment test data synchronization platform | |
| CN109407847A (en) | A kind of multimodal human-computer interaction and man-machine environment test data synchronization system and method | |
| Fernandez-Llatas et al. | Interactive process mining in surgery with real time location systems: Interactive trace correction | |
| Shrawankar et al. | Cloud Computing Technologies for Smart Agriculture and Healthcare | |
| CN113974589B (en) | Multi-modal behavior paradigm evaluation optimization system and cognitive ability evaluation method | |
| Majumder et al. | A smart cyber-human system to support mental well-being through social engagement | |
| CN108765413A (en) | Method, apparatus and computer-readable medium for image classification | |
| Cernea | User-Centered Collaborative Visualization | |
| Voßkühler | OGAMA description (for Version 2.5) | |
| CN109471532A (en) | Multichannel man-machine environment test data synchronous platform experimental design work system and method | |
| Jovic et al. | Use case diagram based scenarios design for a biomedical time-series analysis web platform | |
| Keskin | Exploring the cognitive processes of map users employing eye tracking and EEG | |
| Dominic et al. | Combining predictive analytics and artificial intelligence with human intelligence in iot-based image-guided surgery | |
| KR102380027B1 (en) | Patient health examination terminal using bio big-data visualization method | |
| Planke et al. | Online Multimodal Inference of Mental Workload for Cognitive Human Machine Systems. Computers. 2021; 10: 81 | |
| Čok et al. | Integrating the Kansei Engineering into the Design Golden Loop Development Process |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |