CN113657243B - Test method for performance influence of non-contact bionic remote control gesture - Google Patents
Test method for performance influence of non-contact bionic remote control gesture Download PDFInfo
- Publication number
- CN113657243B CN113657243B CN202110927934.XA CN202110927934A CN113657243B CN 113657243 B CN113657243 B CN 113657243B CN 202110927934 A CN202110927934 A CN 202110927934A CN 113657243 B CN113657243 B CN 113657243B
- Authority
- CN
- China
- Prior art keywords
- task
- gesture
- remote control
- matrix
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Landscapes
- Eye Examination Apparatus (AREA)
- Measurement Of The Respiration, Hearing Ability, Form, And Blood Characteristics Of Living Organisms (AREA)
Abstract
The invention provides a test method aiming at the performance influence of a non-contact bionic remote control gesture, which adopts a performance influence test system comprising a bionic unmanned aerial/underwater carrier, a seat conforming to human engineering, a large-screen display, a Leap Motion gesture recognition system, a Tobii Pro Nano eye tracker, a Biopac MP150 physiological recorder and computing equipment to test four nerve indexes of effective reaction duration, blinking duration, saccular movement length and average ineffective shrinkage; the method comprises the specific steps of preheating, instantaneous step, rest and recheck, test starting, cyclic judgment and data processing. The method can reflect the degree of influence on the neuropsychological/physiological performance of the measured gesture combination under the load brought to an operator by long-time special remote control work, so that related workers can compare and select from a large number of gestures in a gesture library, and further arrange the high-performance gesture combination.
Description
Technical Field
The invention relates to the technical field of gesture recognition, in particular to a method for testing the performance influence of a non-contact bionic remote control gesture.
Background
Unmanned aerial/underwater remote control vehicles play an important role in modern industrial society. In recent years, the industry of unmanned aerial/underwater vehicles is vigorously developed, and people are excited; this dramatic increase in industrial scale is not only in yield, quality, but also in product variety. Nowadays, bionics remote control vehicles are not only popular with consumers in the civil market, but also enjoy irreplaceable dominance in high-risk extreme environmental specialties (e.g., anti-terrorist investigation in complex terrains, investigation and repair of submerged pipelines, rescue from nuclear accidents, etc.).
In order to adapt to various high-risk extreme environments, stricter requirements are certainly made on various aspects of the special bionic unmanned aerial/underwater vehicle; the control link of the bionic unmanned vehicle is the field in which many scholars and institutions are always studying the attack and the customs.
From the point of view of human-computer interaction engineering, in the control link of the unmanned vehicle, 3 participating objects (namely the unmanned vehicle, the control end and the operator) are in total; these three participating subjects constitute a neuroreflex circuit. In the prior art, there are various and well-established schemes for the interaction loop between the control end and the unmanned vehicle. In recent years, many scholars and organizations have also begun to try to make more and more innovative studies in the interaction loop between the control end and the operator. At present, the main implementation means of the interaction ring between the control end and the operator are divided into two remote control methods, namely traditional contact (laptop) and innovative non-contact (non-laptop). For example: for traditional contact control instruments such as push rods and handles, various studies of scholars show that: the handle with good human engineering shape and size can bring better neuropsychological/physiological performance to the control of the unmanned carrier, so that neuropsychological/physiological load of an operator when the operator executes special work tasks is reduced, and the task success rate of special work is improved. However, gesture-based contactless control has some irreplaceable distinctions relative to traditional contact control. From a biomechanical point of view, the human hand and wrist have 24 degrees of freedom in total, that is, the two hands and the two wrists have 48 degrees of freedom in total; the huge number of joint degrees of freedom can provide more gesture choices theoretically, so that the non-contact control mainly based on gesture control can not only undertake the unmanned aerial/underwater vehicle remote control work which can be performed by the traditional contact control, but also can rapidly send dozens of different commands through different gestures so as to be competent for special work tasks in various long-time and high-risk extreme environments. In addition, the learners show that the non-contact control mainly based on gesture control can better help the operator to establish proprioception (preproception) because the different bionic unmanned vehicle motion remote control logics are greatly different from the traditional contact remote control modes (such as a handle, a rocker and other control ends).
Currently, the common non-contact control instruments mainly based on gesture control mainly include an optical gesture recognition instrument mainly based on Leap Motion and a wearable glove. Compared with the wearable glove, the Leap Motion can additionally identify the degree of freedom of the movement of the hand driven by the arm and elbow joints, thereby providing more gestures for selection, and even finishing the joint gesture of common identification of both hands; the wearable glove cannot measure the relative position between the two hands, and can only independently measure the motion tracks of joints of each finger and each wrist; therefore, the optical gesture recognition apparatus based on Leap Motion is currently favored by researchers. For example: the application publication number CN 104793738A and the patent name 'a non-contact computer control method based on Leap Motion', disclose a gesture recognition method based on Leap Motion; patent publication No. CN104007819A, patent name "gesture recognition method and device, and a Leap Motion somatosensory control system" disclose a three-dimensional model operation gesture library which can be constructed and is suitable for Leap Motion, and a corresponding gesture recognition method; patent publication No. CN 106598227A and patent name "gesture recognition method based on Leap Motion and Kinect" disclose a gesture recognition method that combines gesture feature information obtained by two sensors, namely Leap Motion and Kinect, to collect a plurality of samples for each gesture to be recognized to form a training sample set, and further to recognize the gesture; patent publication No. CN105389539A, patent name "a three-dimensional gesture attitude estimation method and system based on depth data" discloses a method and system for realizing three-dimensional gesture attitude estimation by using regression algorithm with the aid of 3-dimensional gesture data taken as an auxiliary.
However, the above patents and the prior art neglect to pay attention to the operator itself and put more attention to the contactless control end (i.e., Leap Motion); there is also a "central nervous-executive tissue-organ interaction loop" within the operator's nervous system and cognitive system; therefore, how to scientifically select, arrange and combine different operation gestures, so as to bring better neuropsychological/physiological performance to the control of the bionic unmanned vehicle, reduce neuropsychological/physiological load of an operator when the operator executes a special work task, and improve the task success rate of special work.
Disclosure of Invention
In view of the problems in the prior art, an object of the present invention is to provide a method for testing the performance impact of a non-contact bionic remote control gesture, so as to obtain a "performance impact index" of the neuropsychological/physiological performance impact of a non-contact bionic unmanned aerial/underwater vehicle remote control gesture combination, thereby reflecting the measured degree of the neuropsychological/physiological performance impact on the gesture combination under the load of an operator caused by long-time special remote control work, so that related workers can compare and select from a large number of gestures in a gesture library, and then arrange a high-performance gesture combination.
The purpose of the invention is realized by the following technical scheme:
a test method for the performance influence of non-contact bionic remote control gestures is characterized in that:
testing by adopting a performance influence testing system, wherein the performance influence testing system comprises a bionic unmanned aerial/underwater vehicle, an ergonomic seat, a large-screen display, a Leap Motion gesture recognition system, a Tobii Pro Nano eye tracker, a Biopac MP150 physiological recorder and computing equipment;
the testing method comprises the steps of selecting N operators for testing, and testing four nerve indexes of effective reaction time length, blinking time length, saccular movement length of eyeballs and average ineffective contraction quantity of the operators;
the effective reaction time long finger is extracted by a Leap Motion gesture recognition system after the gesture change requirement appears and until the gesture change is completed, and the time length required by a new gesture to be successfully recognized by the Leap Motion gesture recognition system is required; the blinking time length refers to the time length of the eye in a complete eye-opening-blinking pulse and in a closed state of the eye, and is extracted and calculated by a Tobii Pro Nano eye movement instrument; the saccular motion length of the eyeball refers to the saccadic length of the eyeball and is extracted and calculated by a Tobii Pro Nano eye tracker; the ineffective contraction quantity refers to a parameter for measuring the intensity of conjugated contraction (Colinkage) of a pair of antagonistic muscles (agonist-antagonist muscles), and is extracted by a Biopac MP150 physiological recorder and then calculated by a calculating device;
conjugated contraction, also known as conjugation activation (coactivation), refers to a phenomenon of "simultaneous activation" of antagonistic muscle pairs (agonist-agonist muscles) around joints in response to central nervous requirements in situations where the central nervous system requires the execution of organized organs to accurately stabilize a static posture or a dynamic posture change. Therefore, when the operator performs the gesture transition between the gesture a and the gesture B, in order to keep the gesture stable in the transition process, the severity of the conjugate contraction is low, and then the physiological load on the operator caused by the gesture combination a-B is relatively small; conversely, if the severity of the conjugate contraction induced during the transition of the gesture combination a-B is high, then it places a relatively high physiological burden on the operator.
All the selected operators should receive not less than 20 academic Leap Motion non-contact unmanned vehicle remote control training before formally starting the experiment; in addition to the theoretical courses, at least 16 actual remote trainings at the time of learning should be set to help the operators to form subjective motility (video practice) and body perception of external extension. Moreover, after the actual remote control training, the operators should memorize and be familiar with the transformation modes of all the different gesture combinations to be tested. Research shows that the study and adaptation of a new hand to a control mode of the new hand can influence the objective evaluation of the performance of the control mode; the operator's skill is therefore a variable that must be controlled.
The test method comprises the following specific steps:
s001, preheating: preheating for 10min, enabling an operator to remotely control the non-contact type remote control unmanned aerial/underwater vehicle to freely move around a test field, and simultaneously confirming that all gesture combinations to be tested can be successfully recognized by a Leap Motion gesture recognition system;
s002, instantaneous step: entering a remote control task cycle; the remote control task cycle is M cycles which are not completely repeated by adopting gesture combination, the task time of each incomplete repetition is 10min, and the total time of the remote control task cycle is not less than 3.5 h;
s003, rest and recheck: the rest and recheck are carried out for 5min between two incomplete repeated tasks, an operator checks and confirms whether all the test equipment works normally or not, whether the condition of the operator is abnormal or not is confirmed at the same time, and other personnel check and confirm whether the site occupied by the test is abnormal or not;
s004, starting testing: simultaneously starting a Leap Motion gesture recognition system, a Tobii Pro Nano eye tracker and a Biopac MP150 physiological recorder and carrying out a task stage with the duration of 10 min; in the task stage, all remote control gesture combinations to be tested, namely k groups, need to be used at least 1 time; the task stage comprises a main line task and a special task;
the main line task is that an operator remotely controls the bionic unmanned aerial/underwater vehicle, moves to a certain specified target position from a starting point, and simultaneously, a measured gesture combination is a gesture combination related to the motion attitude control of the bionic unmanned aerial/underwater vehicle; the special task means that an operator remotely controls the bionic unmanned aerial/underwater vehicle to arrive at a designated place or in front of an encountered target, remote control instructions such as 'taking a picture', 'infrared imaging', 'radioactivity measurement record' and the like are sent out through Leap Motion, and the gesture combination measured by the special task is a gesture combination which is irrelevant to the Motion attitude control of the bionic unmanned aerial/underwater vehicle and sends out a functional instruction;
s005, cyclic judgment: judging whether the remote control task cycle is performed for M times and exceeds the total task time; if yes, go to step S006; if not, returning to the step S003 to continue the incomplete repeated remote control task cycle;
s006, data processing: after the test is finished, the test data are stored and transferred into the computing equipment, the data are processed through the information processing software, and the neuropsychological/physiological performance influence index P of each gesture combination tested in each test process is obtained through calculation, so that different remote control gestures can be evaluated conveniently.
For further optimization, the bionic unmanned aerial/underwater vehicle is at least provided with a camera on the frontal plane for displaying the self-centered visual environment of the unmanned aerial/underwater vehicle; the ergonomic chair does not obstruct the free and flexible movable elbow and wrist support of the upper limb, and is convenient for an operator to carry out long-time non-contact remote control experiments.
For further optimization, the Biopac MP150 physiological recorder collects 10 antagonistic muscles of an operator participating in a remote control task, two electrodes are attached to each muscle and connected with a signal collecting end of the Biopac MP150 physiological recorder, and one electrode is attached to a skin proximal bone (such as a bone bulge of an elbow and a wrist) of two upper limbs of the operator and connected with a grounding end of the Biopac MP150 physiological recorder.
For further optimization, the specific calculation steps of the average invalid shrinkage are as follows:
EC(t)＝Contractionhigher(t)-Contractionlower(t)；
in the formula, tecThe effective co-collaboration duration represents the total duration of action potentials simultaneously appearing on a pair of relevant antagonistic muscles in the process from gesture change to gesture change completion; contractionhigher(t) is the instantaneous muscle high electrical signal, representing the instant of muscle signal acquisition, measured as the instantaneous electrical signal value of a higher electrical signal in a bundle of antagonistic muscles; contractionlower(t) instantaneous muscle low electrical signal, indicating a moment of muscle signal acquisition, measured as the instantaneous electrical signal value of a bundle of muscles of a pair of antagonistic muscles having a lower electrical signal; ec (t) is the instantaneous effective contraction quantity, representing the electric signal difference between the two muscle bundles constituting a pair of antagonist meats, at a moment of muscle signal acquisition; WC is the ineffective shrinkage;
in order to measure the ineffective contraction quantity of all the participated antagonistic muscle pairs in the gesture combination transition process, the average ineffective contraction quantity is obtained through the ineffective contraction quantity, and specifically, the average ineffective contraction quantity is as follows:
wherein n is the number of antagonistic muscle pairs participating in the remote control of the non-contact vehicle.
For further optimization, the method for obtaining the performance impact index P in step S006 specifically includes:
in long-time special work, different remote control gesture combination changes can bring neuropsychological/physiological performance influences to an operator, so that in M times of incomplete repeated remote control tasks, an optimal state task and a fatigue state task are selected to form a contrast group, data of the optimal state task and data of the fatigue state task are transferred to computing equipment for processing and analysis, and the method specifically comprises the following steps:
s101, performing Fourier transform FFT on the EMG signal collected by the Biopac MP150 and used for calculating the average invalid contraction quantity, and thus filtering out the interference factors of the inharmonic frequencies;
s102, extracting four nerve index parameters from the acquired optimal state task data and fatigue state task data according to different remote control gesture combinations;
s103, performing matrix fitting on four nerve index parameters of N selected operators, specifically:
in the formula, B is a data matrix of the optimal state task; bi1Effective reaction time under the optimal state task; bi2The blink time length under the task in the best state is set; bi3The saccular motion length of the eyeball under the optimal state task; bi4The average invalid shrinkage under the best state task; f is a data matrix of the fatigue state task; f. ofi1Effective reaction time under the fatigue state task; f. ofi2The blinking time length of the task in the fatigue state is shown; f. ofi3The saccular motion length of the eyeball under the fatigue state task; f. ofi4The average ineffective shrinkage under the fatigue state task;
s104, solving weight coefficients a, b, c and d of the four nerve index parameters by adopting a principal component analysis method; the method specifically comprises the following steps:
s201, firstly, carrying out standardization processing on the matrix F in the step S103 to eliminate the difference between the dimension and the magnitude, so that four nerve index parameters in the matrix F have comparability;
s202, subtracting the average value of the column where each element in the F matrix is located from each element in the F matrix, and dividing the average value by the standard deviation of the sample of the column where each element is located, so that the average value of each column of the original matrix F is 0 and the variance is 1, and obtaining a matrix A:
A＝[aij]N×4；
in the formula, SjRepresents the sample standard deviation; f. ofijRepresenting the data in matrix F;
represents the average value of the column in which the matrix F is located; namely:
s203, obtaining a covariance matrix C according to the normalized matrix A:
in the formula: k ═ 1, 2, 3, 4]；m＝[1，2，3，4]；
Respectively are the average values of the kth column and the mth column in the matrix A; and when k is equal to m,
s204, four eigenvalues lambda of the covariance matrix C are obtained through calculation1、λ2、λ3、λ4And arranging the four characteristic values in a descending order (such as lambda)1≥λ2≥λ3≥λ4) Simultaneously calculating to obtain eigenvectors corresponding to the four eigenvalues
Thus, a new matrix V is obtained:
s205, obtaining a calculation matrix R through the matrix A and the new matrix V:
calculating that each row of the matrix R is equivalent to the matrix A, namely all rows of the matrix after the fatigue state task data is standardized, namely the projection of a vector formed by four nerve index parameters on a principal component coordinate axis; the vectors formed by the projections are principal component score vectors, and the median or mode obtained for each item of the vectors is the weight coefficients a, b, c and d of the four nerve index parameters;
s105, obtaining a performance influence index P of the corresponding gesture combination through a weighted multivariate function formula:
in the formula, σ (X) and σ (Y) represent normal standard deviations; x is fijThe data are fatigue state task data; y is bijThe task data is the best state; p is an element of [ 0; 1]。
For further optimization, the method for obtaining the performance impact index P in step S006 further includes:
s106, calculating according to the step S105 to obtain each measured gesture combination and the corresponding performance influence index P, and then arranging the performance influence indexes P of the measured k to the gesture combinations in a descending order, so that the neuropsychological/physiological performance of the non-contact remote control gesture combination after long-time special remote control work is obtained.
For further optimization, the N is not less than 15 persons.
For further optimization, the number of M is not less than 16.
For further optimization, the information processing software in step S006 is Matlab or software similar to Matlab.
The invention has the following technical effects:
the invention integrates 4 important parameters (namely four nerve index parameters) in 3 different fields of neurology, cognitive psychology and physiology into a performance influence index P aiming at the neuropsychological/physiological performance influence of the Leap Motion non-contact bionic unmanned aerial/underwater vehicle remote control gesture combination by a weighted multivariate function formula, and the performance influence index P is directly and reasonably shown: under the load brought to an operator by long-time special remote control work, the measured influence degree of the neuropsychological physiological performance of the gesture combination is reduced, so that related workers can compare and select a large number of gestures in a gesture library, high-performance gesture combinations are arranged, the task success rate of the special work is improved, and the development of non-contact remote control gestures is facilitated. Meanwhile, the method can carry out reasonable individualized adjustment according to different types of special remote control tasks and different styles of bionic unmanned aerial/underwater vehicles and the weight coefficient solved by principal component analysis according to specific conditions; through principal component analysis, a method for solving the weight coefficient of the cross-domain and cross-disciplinary weighted multivariate function expression is provided, and the solution is quicker and more convenient. By simulating long-time high-load special work tasks in the real world, an incompletely repeated cyclic experiment is provided, and a comparison group of an optimal state and a fatigue state is selected, so that the influence of the neuropsychological/physiological performance of the non-contact remote control gesture combination after long-time special remote control work is more visual and closer to the real situation.
Drawings
FIG. 1 is a schematic diagram of a neuroreflex circuit of three participating objects in human-computer interaction engineering.
FIG. 2 is a schematic structural diagram of experimental equipment used in the system of the present invention.
Fig. 3 is a schematic diagram of a specific placement position of the electrode of the system of the present invention.
FIG. 4 is a flow chart of the testing method of the present invention.
Wherein, 10, control end-unmanned vehicle interactive ring; 20. an operator-control interaction loop; 30. central nerve-executive tissue organ interaction ring; 40. tobii Pro Nano eye tracker; 50. a Leap Motion gesture recognition system; 60. biopac MP150 physiological recorder.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Example (b):
as shown in fig. 1 to 4, a method for testing the performance influence of a non-contact bionic remote control gesture is characterized in that:
the performance impact testing system is adopted for testing, and comprises a bionic unmanned aerial/underwater vehicle, an ergonomic seat, a large-screen display, a Leap Motion gesture recognition system, a Tobii Pro Nano eye tracker, a Biopac MP150 physiological recorder and computing equipment;
the bionic unmanned aerial/underwater vehicle is at least provided with a camera on the frontal plane for displaying the visual environment of the unmanned aerial/underwater vehicle with the self as the center; the chair conforming to the ergonomics does not obstruct the free and flexible movable elbow and wrist support of the upper limb, and is convenient for an operator to carry out long-time non-contact remote control experiments;
the testing method comprises the steps of selecting N operators for testing, and testing four nerve indexes of effective reaction duration, blinking duration, saccular movement length of eyeballs and average ineffective contraction quantity of the operators; n is not less than 15 people, namely N is not less than 15.
In the effective reaction time, the length of time required by the new gesture to be successfully recognized by the Leap Motion gesture recognition system from the appearance of the gesture change requirement to the completion of the gesture change is extracted by the Leap Motion gesture recognition system; the blinking time refers to the time length of the eye in a complete 'eye opening-blinking' pulse and the eye in a closed state, and is extracted and calculated by a Tobii Pro Nano eye movement instrument; the saccular movement length of the eyeball refers to the saccadic length of the eyeball and the pupil, and is extracted and calculated by a Tobii Pro Nano eye tracker; the ineffective contraction quantity refers to a parameter for measuring the intensity of conjugated contraction (Coconstriction) of a pair of antagonistic muscles (agonst-antanst muscles), and is extracted by a Biopac MP150 physiological recorder and then calculated by a calculating device; the Biopac MP150 physiological recorder collects 10 antagonistic muscles (2 x5 pairs of antagonistic muscles in both hands) of an operator participating in a remote control task, and the measured muscle information is shown in table 1 below; two electrodes are attached to each muscle bundle and connected with the signal acquisition end of the Biopac MP150 physiological recorder, one electrode is attached to the skin near bones of two upper limbs of an operator (such as the bony prominences of the elbow and wrist) and connected with the grounding end of the Biopac MP150 physiological recorder, and the positions where the electrodes are attached are shown in fig. 3 and table 2.
Table 1:
table 2:
conjugated contraction, also known as conjugation activation (coactivation), refers to a phenomenon of "simultaneous activation" of antagonistic muscle pairs (agonist-agonist muscles) around joints in response to central nervous requirements in situations where the central nervous system requires the execution of organized organs to accurately stabilize a static posture or a dynamic posture change. Therefore, when the operator performs the gesture transition between the gesture a and the gesture B, in order to keep the gesture stable in the transition process, the severity of the conjugate contraction is low, and then the physiological load on the operator caused by the gesture combination a-B is relatively small; conversely, if the severity of the conjugate contraction induced during the transition of the gesture combination a-B is high, then it places a relatively high physiological burden on the operator.
The specific calculation steps of the average invalid shrinkage are as follows:
EC(t)＝Contractionhigher(t)-Contractionlower(t)；
in the formula, tecThe effective co-collaboration duration represents the total duration of action potentials simultaneously appearing on a pair of relevant antagonistic muscles in the process from gesture change to gesture change completion; contractionhigher(t) is the instantaneous muscle high electrical signal, indicating a brake of muscle signal acquisition, measured higher in a pair of antagonistic musclesInstantaneous electrical signal value of a bundle of muscles of the frontal electrical signal; contractionlower(t) instantaneous muscle low electrical signal, indicating a moment of muscle signal acquisition, measured as the instantaneous electrical signal value of a bundle of muscles of a pair of antagonistic muscles having a lower electrical signal; ec (t) is the instantaneous effective contraction quantity, representing the electric signal difference between the two muscle bundles constituting a pair of antagonist meats, at a moment of muscle signal acquisition; WC is the ineffective shrinkage;
in order to measure the ineffective contraction quantity of all the participated antagonistic muscle pairs in the gesture combination transition process, the average ineffective contraction quantity is obtained through the ineffective contraction quantity, and specifically, the average ineffective contraction quantity is as follows:
wherein n is the number of antagonistic muscle pairs participating in the remote control of the non-contact vehicle.
All the selected operators should receive not less than 20 academic Leap Motion non-contact unmanned vehicle remote control training before formally starting the experiment; in addition to the theoretical courses, at least 16 actual remote trainings at the time of learning should be set to help the operators to form subjective motility (video practice) and body perception of external extension. Moreover, after the actual remote control training, the operators should memorize and be familiar with the transformation modes of all the different gesture combinations to be tested. Research shows that the study and adaptation of a new hand to a control mode of the new hand can influence the objective evaluation of the performance of the control mode; the operator's skill is therefore a variable that must be controlled.
The test method comprises the following specific steps:
s001, preheating: preheating for 10min, enabling an operator to remotely control the non-contact type remote control unmanned aerial/underwater vehicle to freely move around a test field, and simultaneously confirming that all gesture combinations to be tested can be successfully recognized by a Leap Motion gesture recognition system;
s002, instantaneous step: entering a remote control task cycle; the remote control task cycle is M cycles which are performed by adopting gesture combination and incomplete repetition, the task time of each incomplete repetition is 10min, M is not less than 16 times (namely M is not less than 16), and the total time of the remote control task cycle is not less than 3.5 h;
s003, rest and recheck: the rest and recheck are carried out for 5min between two incomplete repeated tasks, an operator checks and confirms whether all the test equipment works normally or not, whether the condition of the operator is abnormal or not is confirmed at the same time, and other personnel check and confirm whether the site occupied by the test is abnormal or not;
s004, starting testing: simultaneously starting a Leap Motion gesture recognition system, a Tobii Pro Nano eye tracker and a Biopac MP150 physiological recorder and carrying out a task stage with the duration of 10 min; in the task stage, all remote control gesture combinations to be tested, namely k groups, need to be used at least 1 time; the task stage comprises a main line task and a special task;
the remote control gesture combination conversion refers to a process of converting an original gesture A into a target gesture B, and the gesture combination is named as an A-B combination; likewise, there is another possibility of a combination of gesture A and gesture B, namely a B-A combination. The neuropsychological/physiological performance impact of the A-B combination and the B-A combination transformation process is not necessarily equal; thus A-B and B-A are different combinations of remote control gestures.
The main line task is that an operator remotely controls the bionic unmanned aerial/underwater vehicle, moves to a certain specified target position from a starting point, and simultaneously, a measured gesture combination is a gesture combination related to the motion attitude control of the bionic unmanned aerial/underwater vehicle; the special task means that an operator remotely controls the bionic unmanned aerial/underwater vehicle to arrive at a designated place or in front of an encountered target, remote control instructions such as 'taking a picture', 'infrared imaging', 'radioactivity measurement record' and the like are sent out through Leap Motion, and the gesture combination measured by the special task is a gesture combination which is irrelevant to the Motion attitude control of the bionic unmanned aerial/underwater vehicle and sends out a functional instruction;
s005, cyclic judgment: judging whether the remote control task cycle is performed for M times and exceeds the total task time; if yes, go to step S006; if not, returning to the step S003 to continue the incomplete repeated remote control task cycle;
s006, data processing: after the test is finished, storing the test data and transferring the test data into computing equipment, processing the data through information processing software and calculating to obtain the respective neuropsychological/physiological performance influence index P of the gesture combination tested in each test process so as to evaluate different remote control gestures; matlab or Matlab-like software is adopted as the information processing software.
The method for obtaining the performance impact index P comprises the following steps:
in long-time special work, different remote control gesture combination changes can bring neuropsychological/physiological performance influence to an operator, therefore, in M times of incomplete repeated remote control tasks, an optimal state task and a fatigue state task are selected to form a control group (the optimal state task is selected from 1 st or 2 nd remote control tasks, the second-selected optimal state task is preferably selected from the 2 nd remote control task, the fatigue state task is data of the neuropsychological/physiological performance in a worse time period in M times of experiments performed by the operator), and the data of the optimal state task and the fatigue state task are transferred to a computing device for processing and analysis,
the neuropsychological and physiological performance difference of the operator in the optimal state task and the fatigue state task is caused by the workload of the operator brought by a long-time special remote control task. By studying different combinations of telepresence gestures, the degree of correlation of the operator's neuropsychological/physiological performance between the best-state task and the tired-state task can be known. For example, the measured degree of correlation of the neuropsychological and physiological performances of the gesture combination A-B in the two states is high, which indicates that the fatigue state of the gesture combination A-B after long-time high-load special work can still keep the gesture change process similar to the optimal state. This also indicates that under long-term high-load special work, the neuropsychological performance of the gesture combination a-B is less affected by the workload; on the contrary, the measured degree of correlation between the neuropsychological/physiological performances of the gesture combination C-D in the two states is low, which indicates that the fatigue state of the gesture combination C-D after long-time high-load special work cannot keep the gesture change process similar to the optimal state, namely indicates that the neuropsychological performance of the gesture combination C-D is greatly influenced by the work load under long-time high-load special work. It can therefore be concluded that: gesture combination A-B is preferred over gesture combination C-D. In fact, in the experimental process of verifying the feasibility of the invention, the inventor finds that in the 16 th incompletely repeated task (namely, the fatigue state task), the transition motion track of the gesture combination C-D with the lower correlation degree of the optimal state and the fatigue state is completely different from that of the standard state.
S101, performing Fourier transform FFT on the EMG signal collected by the Biopac MP150 and used for calculating the average invalid contraction quantity, and thus filtering out the interference factors of the inharmonic frequencies;
s102, extracting four nerve index parameters from the acquired optimal state task data and fatigue state task data according to different remote control gesture combinations;
s103, performing matrix fitting on four nerve index parameters of N selected operators, specifically:
in the formula, B is a data matrix of the optimal state task; bi1Effective reaction time under the optimal state task; bi2The blink time length under the task in the best state is set; bi3The saccular motion length of the eyeball under the optimal state task; bi4The average invalid shrinkage under the best state task; f is a data matrix of the fatigue state task; f. ofi1Effective reaction time under the fatigue state task; f. ofi2The blinking time length of the task in the fatigue state is shown; f. ofi3The saccular motion length of the eyeball under the fatigue state task; f. ofi4The average ineffective shrinkage under the fatigue state task;
s104, solving weight coefficients a, b, c and d of the four nerve index parameters by adopting a principal component analysis method; the method specifically comprises the following steps:
s201, firstly, carrying out standardization processing on the matrix F in the step S103 to eliminate the difference between the dimension and the magnitude, so that four nerve index parameters in the matrix F have comparability;
s202, subtracting the average value of the column where each element in the F matrix is located from each element in the F matrix, and dividing the average value by the standard deviation of the sample of the column where each element is located, so that the average value of each column of the original matrix F is 0 and the variance is 1, and obtaining a matrix A:
A＝[aij]N×4；
in the formula, SjRepresents the sample standard deviation; f. ofijRepresenting the data in matrix F;
represents the average value of the column in which the matrix F is located; namely:
s203, obtaining a covariance matrix C according to the normalized matrix A:
in the formula: k ═ 1, 2, 3, 4]；m＝[1，2，3，4]；
Respectively are the average values of the kth column and the mth column in the matrix A; and when k is equal to m,
s204, four eigenvalues lambda of the covariance matrix C are obtained through calculation1、λ2、λ3、λ4And arranging the four characteristic values in a descending order (such as lambda)1≥λ2≥λ3≥λ4) Simultaneously calculating to obtain eigenvectors corresponding to the four eigenvalues
Thus, a new matrix V is obtained:
s205, obtaining a calculation matrix R through the matrix A and the new matrix V:
calculating that each row of the matrix R is equivalent to the matrix A, namely all rows of the matrix after the fatigue state task data is standardized, namely the projection of a vector formed by four nerve index parameters on a principal component coordinate axis; the vectors formed by the projections are principal component score vectors, and the median or mode obtained for each item of the vectors is the weight coefficients a, b, c and d of the four nerve index parameters;
s105, obtaining a performance influence index P of the corresponding gesture combination through a weighted multivariate function formula:
in the formula, σ (X) and σ (Y) represent normal standard deviations; x is fijThe data are fatigue state task data; y is bijThe task data is the best state; p is an element of [ 0; 1]。
And S106, calculating to obtain each detected gesture combination and the corresponding performance influence index P according to the step S105, and then arranging the performance influence indexes P of the detected k to the gesture combinations according to the sequence from large to small, so as to obtain the neuropsychological/physiological performance of the non-contact remote control gesture combination after long-time special remote control work.
In order to facilitate observation and analysis, the ordinal performance influence index P can be graded and colored, so that the neuropsychological/physiological performance of the non-contact remote control gesture combination after long-time special remote control work becomes clear at a glance.
For example: the value range of the performance impact index P is [ 0.9; 1], then the first stage: excellent, the color is green; the range of the performance impact index P is [ 0.75; 0.9), then the second stage: generally, the color is yellow; the value range of the performance impact index P is [ 0.5; 0.75), then third stage: poor, orange in color; the value range of the performance impact index P is [ 0; 0.5), then fourth stage: in a poor condition, the color was red.
Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that changes, modifications, substitutions and alterations can be made in these embodiments without departing from the principles and spirit of the invention, the scope of which is defined in the appended claims and their equivalents.
Claims (3)
1. A test method for the performance influence of non-contact bionic remote control gestures is characterized in that:
testing by adopting a performance influence testing system, wherein the performance influence testing system comprises a bionic unmanned aerial/underwater vehicle, an ergonomic seat, a large-screen display, a Leap Motion gesture recognition system, a Tobii Pro Nano eye tracker, a Biopac MP150 physiological recorder and computing equipment;
the testing method comprises the steps of selecting N operators for testing, and testing four nerve indexes of effective reaction time length, blinking time length, saccular movement length of eyeballs and average ineffective contraction quantity of the operators;
the effective reaction time long finger is extracted by a Leap Motion gesture recognition system after the gesture change requirement appears and until the gesture change is completed, and the time length required by a new gesture to be successfully recognized by the Leap Motion gesture recognition system is required; the blinking time length refers to the time length of the eye in a complete eye-opening-blinking pulse and in a closed state of the eye, and is extracted and calculated by a Tobii Pro Nano eye movement instrument; the saccular motion length of the eyeball refers to the saccadic length of the eyeball and is extracted and calculated by a Tobii Pro Nano eye tracker; the ineffective contraction quantity refers to a parameter for measuring the intensity of conjugated contraction of a pair of antagonistic muscles, is extracted by a Biopac MP150 physiological recorder, and is subsequently calculated by a calculating device;
the test method comprises the following specific steps:
s001, preheating: preheating for 10min, enabling an operator to remotely control the non-contact type remote control unmanned aerial/underwater vehicle to freely move around a test field, and simultaneously confirming that all gesture combinations to be tested can be successfully recognized by a Leap Motion gesture recognition system;
s002, instantaneous step: entering a remote control task cycle; the remote control task cycle is M cycles which are not completely repeated by adopting gesture combination, the task time of each incomplete repetition is 10min, and the total time of the remote control task cycle is not less than 3.5 h;
s003, rest and recheck: the rest and recheck are carried out for 5min between two incomplete repeated tasks, an operator checks and confirms whether all the test equipment works normally or not, whether the condition of the operator is abnormal or not is confirmed at the same time, and other personnel check and confirm whether the site occupied by the test is abnormal or not;
s004, starting testing: simultaneously starting a Leap Motion gesture recognition system, a Tobii Pro Nano eye tracker and a Biopac MP150 physiological recorder and carrying out a task stage with the duration of 10 min; in the task stage, all remote control gesture combinations to be tested, namely k groups, need to be used at least 1 time; the task stage comprises a main line task and a special task;
the main line task is that an operator remotely controls the bionic unmanned aerial/underwater vehicle, moves to a certain specified target position from a starting point, and simultaneously, a measured gesture combination is a gesture combination related to the motion attitude control of the bionic unmanned aerial/underwater vehicle; the special task means that an operator remotely controls the bionic unmanned aerial/underwater vehicle to arrive at a designated place or in front of an encountered target, and sends out remote control commands of 'taking pictures', 'infrared imaging' and 'radioactivity measurement record' through Leap Motion, and the gesture combination measured by the special task is a gesture combination which is irrelevant to the Motion attitude control of the bionic unmanned aerial/underwater vehicle and sends out a functional command;
s005, cyclic judgment: judging whether the remote control task cycle is performed for M times and exceeds the total task time; if yes, go to step S006; if not, returning to the step S003 to continue the incomplete repeated remote control task cycle;
s006, data processing: after the test is finished, storing the test data and transferring the test data into computing equipment, processing the data through information processing software and calculating to obtain the respective neuropsychological/physiological performance influence index P of the gesture combination tested in each test process so as to evaluate different remote control gestures;
the method for obtaining the performance impact index P specifically comprises the following steps:
selecting an optimal state task and a fatigue state task to form a control group in M times of incomplete repeated remote control tasks, and transferring data of the optimal state task and the fatigue state task to computing equipment for processing and analysis, wherein the method specifically comprises the following steps:
s101, performing Fourier transform FFT on the EMG signal collected by the Biopac MP150 and used for calculating the average invalid contraction quantity, and thus filtering out the interference factors of the inharmonic frequencies;
s102, extracting four nerve index parameters from the acquired optimal state task data and fatigue state task data according to different remote control gesture combinations;
s103, performing matrix fitting on four nerve index parameters of N selected operators, specifically:
in the formula, B is a data matrix of the optimal state task; bi1Effective reaction time under the optimal state task; bi2The blink time length under the task in the best state is set; bi3The saccular motion length of the eyeball under the optimal state task; bi4The average invalid shrinkage under the best state task; f is a data matrix of the fatigue state task; f. ofi1Effective reaction time under the fatigue state task; f. ofi2The blinking time length of the task in the fatigue state is shown; f. ofi3The saccular motion length of the eyeball under the fatigue state task; f. ofi4The average ineffective shrinkage under the fatigue state task;
s104, solving weight coefficients a, b, c and d of the four nerve index parameters by adopting a principal component analysis method; the method specifically comprises the following steps:
s201, firstly, carrying out standardization processing on the matrix F in the step S103 to eliminate the difference between the dimension and the magnitude, so that four nerve index parameters in the matrix F have comparability;
s202, subtracting the average value of the column where each element in the F matrix is located from each element in the F matrix, and dividing the average value by the standard deviation of the sample of the column where each element is located, so that the average value of each column of the original matrix F is 0 and the variance is 1, and obtaining a matrix A:
A＝[aij]N×4；
in the formula, SjRepresents the sample standard deviation; f. ofijRepresenting the data in matrix F;
represents the average value of the column in which the matrix F is located; namely:
s203, obtaining a covariance matrix C according to the normalized matrix A:
in the formula: k ═ 1, 2, 3, 4]；m＝[1，2，3，4]；
Respectively are the average values of the kth column and the mth column in the matrix A; and when k is equal to m,
s204, four covariance matrixes C are obtained through calculationCharacteristic value lambda1、λ2、λ3、λ4And arranging the four eigenvalues in descending order, and simultaneously calculating to obtain eigenvectors corresponding to the four eigenvalues
Thus, a new matrix V is obtained:
s205, obtaining a calculation matrix R through the matrix A and the new matrix V:
calculating that each row of the matrix R is equivalent to the matrix A, namely all rows of the matrix after the fatigue state task data is standardized, namely the projection of a vector formed by four nerve index parameters on a principal component coordinate axis; the vectors formed by the projections are principal component score vectors, and the median or mode obtained for each item of the vectors is the weight coefficients a, b, c and d of the four nerve index parameters;
s105, obtaining a performance influence index P of the corresponding gesture combination through a weighted multivariate function formula:
in the formula, σ (X) and σ (Y) represent normal standard deviations; x is fijThe data are fatigue state task data; y is bijThe task data is the best state; p is an element of [ 0; 1]。
2. The method for testing the performance influence of the non-contact bionic remote control gesture according to claim 1, wherein the method comprises the following steps: the Biopac MP150 physiological recorder collects 10 pairs of antagonistic muscles of an operator participating in a remote control task, two electrodes attached to each muscle are connected with a signal collecting end of the Biopac MP150 physiological recorder, and one electrode is attached to the skin near bones of two upper limbs of the operator and connected with a grounding end of the Biopac MP150 physiological recorder.
3. The method for testing the performance influence of the non-contact bionic remote control gesture according to any one of claims 1 or 2, characterized by comprising the following steps: the specific calculation steps of the average invalid shrinkage are as follows:
EC(t)＝Contractionhigher(t)-Contractionlower(t)；
in the formula, tecThe effective co-collaboration duration represents the total duration of action potentials simultaneously appearing on a pair of relevant antagonistic muscles in the process from gesture change to gesture change completion; contractionhigher(t) is the instantaneous muscle high electrical signal, representing the instant of muscle signal acquisition, measured as the instantaneous electrical signal value of a higher electrical signal in a bundle of antagonistic muscles; contractionlower(t) instantaneous muscle Low electric Signal, indicating a brake of muscle Signal acquisition, of a pair of antagonistic muscles measured, of the bundle of muscles having the lower electric SignalAn instantaneous electrical signal value; ec (t) is the instantaneous effective contraction quantity, representing the electric signal difference between the two muscle bundles constituting a pair of antagonist meats, at a moment of muscle signal acquisition; WC is the ineffective shrinkage;
in order to measure the ineffective contraction quantity of all the participated antagonistic muscle pairs in the gesture combination transition process, the average ineffective contraction quantity is obtained through the ineffective contraction quantity, and specifically, the average ineffective contraction quantity is as follows:
wherein n is the number of antagonistic muscle pairs participating in the remote control of the non-contact vehicle.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110927934.XA CN113657243B (en) | 2021-08-13 | 2021-08-13 | Test method for performance influence of non-contact bionic remote control gesture |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110927934.XA CN113657243B (en) | 2021-08-13 | 2021-08-13 | Test method for performance influence of non-contact bionic remote control gesture |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113657243A CN113657243A (en) | 2021-11-16 |
| CN113657243B true CN113657243B (en) | 2022-04-12 |
Family
ID=78479662
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110927934.XA Active CN113657243B (en) | 2021-08-13 | 2021-08-13 | Test method for performance influence of non-contact bionic remote control gesture |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113657243B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113970968B (en) * | 2021-12-22 | 2022-05-17 | 深圳市心流科技有限公司 | Intelligent bionic hand action pre-judging method |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104793738A (en) * | 2015-03-17 | 2015-07-22 | 上海海洋大学 | Non-contact type computer operating method based on Leap Motion |
| EP3432792A4 (en) * | 2016-03-22 | 2019-11-27 | LifeSignals, Inc. | Systems and methods for physiological signal collection |
| US10842407B2 (en) * | 2018-08-31 | 2020-11-24 | Facebook Technologies, Llc | Camera-guided interpretation of neuromuscular signals |
| CN109933206B (en) * | 2019-03-26 | 2022-03-08 | 长安大学 | Finger non-contact drawing method and system based on Leap Motion |
| CN113100769A (en) * | 2021-04-21 | 2021-07-13 | 北京理工大学 | Physiological index-based working state evaluation method and system for unmanned aerial vehicle operator |
-
2021
- 2021-08-13 CN CN202110927934.XA patent/CN113657243B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN113657243A (en) | 2021-11-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107378944B (en) | Multidimensional surface electromyographic signal artificial hand control method based on principal component analysis method | |
| US11069148B2 (en) | Visualization of reconstructed handstate information | |
| KR20200024324A (en) | Armband to track hand motion using electrical impedance measurement background | |
| Peterson et al. | A penalized time-frequency band feature selection and classification procedure for improved motor intention decoding in multichannel EEG | |
| JP2020034849A (en) | Work support device, work support method, and work support program | |
| WO2015093224A1 (en) | Movement analysis device, movement analysis method, and movement analysis program | |
| US10849532B1 (en) | Computer-vision-based clinical assessment of upper extremity function | |
| Ison et al. | Simultaneous myoelectric control of a robot arm using muscle synergy-inspired inputs from high-density electrode grids | |
| CN113657243B (en) | Test method for performance influence of non-contact bionic remote control gesture | |
| CN109126045A (en) | intelligent motion analysis and training system | |
| KR100994408B1 (en) | Method and device for deducting pinch force, method and device for discriminating muscle to deduct pinch force | |
| CN109585018A (en) | Information processing method, information processing unit and physiological detection equipment | |
| Robson et al. | Creating a virtual perception for upper limb rehabilitation | |
| Huang et al. | Performances of surface EMG and ultrasound signals in recognizing finger motion | |
| CN110675936B (en) | Fitness compensation assessment method and system based on OpenPose and binocular vision | |
| Sheng et al. | A hand gesture recognition using single-channel electrodes based on artificial neural network | |
| Weisz et al. | Grasping with your face | |
| US20240061512A1 (en) | Computer-implemented method and apparatus for determining reaction time processes | |
| Chen | Design and evaluation of a human-computer interface based on electrooculography | |
| Kahol et al. | The effect of fatigue on cognitive and psychomotor skills of surgical residents | |
| Cho et al. | Estimation of Forearm Pose Based on Upper Arm Deformation Using a Deep Neural Network | |
| Sharma et al. | Assessing the impact of muscular fatigue on myoelectric signals using myo armband | |
| Pan et al. | Artificial Intelligence Hand Spatial Position Predictor Based on Data Gloves and Jetson Xavier NX | |
| Zhou et al. | JISAP: Joint Inference for Surgeon Attributes Prediction during Robot-Assisted Surgery | |
| Mussa-Ivaldi et al. | The remapping of space in motor learning and human–machine interfaces |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |