CN105759677A - Multimode behavior analysis and monitoring system and method adapted to visual terminal operation post - Google Patents
Multimode behavior analysis and monitoring system and method adapted to visual terminal operation post Download PDFInfo
- Publication number
- CN105759677A CN105759677A CN201510145902.9A CN201510145902A CN105759677A CN 105759677 A CN105759677 A CN 105759677A CN 201510145902 A CN201510145902 A CN 201510145902A CN 105759677 A CN105759677 A CN 105759677A
- Authority
- CN
- China
- Prior art keywords
- data
- behavior
- modal
- post
- analysis
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000004458 analytical method Methods 0.000 title claims abstract description 64
- 238000000034 method Methods 0.000 title claims abstract description 60
- 238000012544 monitoring process Methods 0.000 title claims abstract description 27
- 230000000007 visual effect Effects 0.000 title abstract description 12
- 230000006399 behavior Effects 0.000 claims abstract description 100
- 210000000707 wrist Anatomy 0.000 claims abstract description 26
- 230000004044 response Effects 0.000 claims abstract description 9
- 230000008569 process Effects 0.000 claims description 37
- 210000001217 buttock Anatomy 0.000 claims description 25
- 230000003542 behavioural effect Effects 0.000 claims description 22
- 238000001514 detection method Methods 0.000 claims description 10
- 238000000605 extraction Methods 0.000 claims description 7
- 230000011273 social behavior Effects 0.000 claims description 7
- 230000009471 action Effects 0.000 claims description 6
- 238000013480 data collection Methods 0.000 claims description 6
- 230000004927 fusion Effects 0.000 claims description 6
- 238000009826 distribution Methods 0.000 claims description 5
- 210000003414 extremity Anatomy 0.000 claims description 5
- 230000008921 facial expression Effects 0.000 claims description 5
- 230000003945 visual behavior Effects 0.000 claims description 5
- 238000013481 data capture Methods 0.000 claims description 4
- 231100001261 hazardous Toxicity 0.000 claims description 3
- 238000011156 evaluation Methods 0.000 abstract description 3
- 230000000638 stimulation Effects 0.000 abstract description 3
- 230000008901 benefit Effects 0.000 abstract description 2
- 210000001508 eye Anatomy 0.000 description 17
- 238000005516 engineering process Methods 0.000 description 9
- 210000003128 head Anatomy 0.000 description 9
- 238000004422 calculation algorithm Methods 0.000 description 7
- 230000008859 change Effects 0.000 description 5
- 230000001360 synchronised effect Effects 0.000 description 4
- 238000004364 calculation method Methods 0.000 description 3
- 238000004891 communication Methods 0.000 description 3
- 230000036541 health Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000011002 quantification Methods 0.000 description 2
- PXFBZOLANLWPMH-UHFFFAOYSA-N 16-Epiaffinine Natural products C1C(C2=CC=CC=C2N2)=C2C(=O)CC2C(=CC)CN(C)C1C2CO PXFBZOLANLWPMH-UHFFFAOYSA-N 0.000 description 1
- 241001269238 Data Species 0.000 description 1
- 210000005252 bulbus oculi Anatomy 0.000 description 1
- 230000006835 compression Effects 0.000 description 1
- 238000007906 compression Methods 0.000 description 1
- 239000012141 concentrate Substances 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 239000006185 dispersion Substances 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 210000004247 hand Anatomy 0.000 description 1
- 229940056582 human hair preparation Drugs 0.000 description 1
- 230000006698 induction Effects 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 238000007689 inspection Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000003340 mental effect Effects 0.000 description 1
- 230000005693 optoelectronics Effects 0.000 description 1
- 238000004445 quantitative analysis Methods 0.000 description 1
- 238000013139 quantization Methods 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 230000008929 regeneration Effects 0.000 description 1
- 238000011069 regeneration method Methods 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 230000007480 spreading Effects 0.000 description 1
- 238000003892 spreading Methods 0.000 description 1
- 238000007619 statistical method Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 230000002618 waking effect Effects 0.000 description 1
- 238000005303 weighing Methods 0.000 description 1
Landscapes
- Alarm Systems (AREA)
Abstract
The invention discloses a multimode behavior analysis and monitoring system and method adapted to a visual terminal operation post. The system comprises an eye tracking photoelectric sensor, a face tracking photoelectric sensor, a trunk tracking photoelectric sensor, a wrist piezoelectric transducer, an ancon piezoelectric transducer, a hip piezoelectric transducer, a back piezoelectric transducer, a visual terminal, an operation keyboard, a multichannel information acquisition and processing device, a feedback control and interface apparatus and an alarm device. The method comprises the following steps of: 1, human behavior data acquisition; 2, machine response behavior data acquisition; 3, behavior analysis; 4, feedback control and interface; and 5, alarm execution. The system and method have the advantages of realizing normative evaluation and risk identification of visual terminal operation post work behaviors and realize the purpose of effective behavior control by giving a risk alarm to personnel through a perceptible feedback mode, such as visual, audio and touch stimulation modes.
Description
Technical field
The present invention relates to a kind of multi-modal behavior analysis being suitable to vision terminal job post and monitoring system and method, belong to ergonomics, human engineering, intelligence system and industrial control system technical field.
Background technology
At present, traditional vision terminal (VDT-VisualDisplayTerminal) operation technique is the general name that the visual display terminal utilizing computer system carries out data, word, Image Information Processing work, is a kind of technology type based on mental work.Vision terminal job technology is then the technology of the full-time core medium obtained for job information with visual display terminal.The feature of this technology is: personnel are realized jointly cooperating with computer (or machine) completing technology task by the mode that visual attention location screen and hands behaviour's input equipment (such as keyboard, mouse) combine.
Today in industrial automation, intelligent very fast development, many relations country, the great public safety of social production system in, the post having significant proportion is vision terminal job technology, and the mission critical post that some vision terminal job post is system, job task difficulty action accomplishment is directly connected to the reliability of whole system.The system running state in visual information monitoring post, the real-time induction system of the energy in such as, scheduling post, operational chain of command in public transportation system is monitored post, is also had the safety inspection ray image identification post in high safety grade place, these broadly fall into typical visual display terminal operation post category, and bear mission critical.It is based on the critical great attention in this kind of post, receive much concern about problems such as the job performance in post, workload, work quality and security risks that thus post introduces to system, particularly in the real-time identification of post very dangerous behavior and the meaning of system risk strick precaution aspect is more great.
At present, the existing correlation technique about driver behaviorist risk identification in driving safety field.The such as patent No. is a kind of method that the patent of invention of CN101565036B provides anti-fatigue-driving, image information by Real-time Collection driver, analyse whether as same driver, monitor continuous driving time and eye open and-shut mode, the sight line dispersion of driver and stare position to analyze the driving condition of driver, taking to report to the police and brake measure for fatigue state.
Such as, application number be 201310248710.1 application for a patent for invention provide a kind of vehicle-mounted real-time intelligent fatigue monitoring and auxiliary device, driver fatigue state is detected by image collection, driver status is carried out parametrization and fatigue state is carried out classification, take to interfere the related measure of instruction according to classification, also support the tired baseline of user is carried out personal settings and adjustment simultaneously.
Above-mentioned visible, according to driving behavior feature, the fatigue state of driver is the emphasis identification of driving safety, anti-norm condition.And fatigue state cannot be limited to for the behavior analysis in vision terminal job post, risk identification and intelligent monitoring, because this post personnel also have most to be because the attention irrational distribution that level of skill causes not to the risk that system introduces except fatigue, it is impossible to timely discovery situation also makes fast reaction.Therefore the behavior analysis for vision terminal job post from the angle of people-machine cooperating process quantitative analysis Yu assessment, should form the intelligence control system that multi-modal information is comprehensively analyzed, people-machine behavior synchronization is monitored, take into account risk prompting and performance evaluation.
In sum, at present, the behavior analysis for visual task post not yet develops with monitoring systems technology and method, does not all have the System and method for of behavior analysis and the monitoring systems technology being proposed for visual task post both at home and abroad.
Summary of the invention
It is an object of the invention to provide a kind of multi-modal behavior analysis being suitable to vision terminal job post that can overcome above-mentioned technical problem and monitoring system and method, the present invention is by the data acquisition of the human body behavior of multiple modalities with machine respondent behavior, and in conjunction with people-machine cooperative behaviors comprehensive quantification analysis, realize the standardization to post work behavior to assess and risk identification, and by appreciable feedback system, for instance vision, hearing, touch feels that personnel are carried out the risk warning purpose to reach the effective management and control of behavior by stimulation mode.
The invention provides a kind of multi-modal behavior analysis being suitable to vision terminal job post and monitoring system, the multi-modal behavior analysis being suitable to vision terminal job post of the present invention and monitoring system are the quantization monitoring system systems of the people-machine cooperating process being core with operation keyboard with vision terminal, and its function is human body operation behavior data snooping, machine respondent behavior data acquisition, people-machine behavioral data analysis, Risk-warning feedback control.
The multi-modal behavior analysis being suitable to vision terminal job post of the present invention and monitoring system include: eye tracing photoelectric sensor, feature tracking photoelectric sensor, trunk tracing photoelectric sensor, wrist piezoelectric transducer, ancon piezoelectric transducer, buttocks piezoelectric transducer, back piezoelectric transducer, vision terminal, operation keyboard, multi-channel information collection and the equipment of process, feedback control and interface arrangement and alarm appliance.
Described eye tracing photoelectric sensor, feature tracking photoelectric sensor, trunk tracing photoelectric sensor, wrist piezoelectric transducer, ancon piezoelectric transducer, buttocks piezoelectric transducer, back piezoelectric transducer and vision terminal, operation keyboard are all connected with the equipment of process with multi-channel information collection with independent data cable.
Described multi-channel information collection is connected with interface arrangement with the equipment of process and feedback control.
Described feedback control is connected with interface arrangement and alarm appliance.
Wherein said eye tracing photoelectric sensor, feature tracking photoelectric sensor, trunk tracing photoelectric sensor, wrist piezoelectric transducer, ancon piezoelectric transducer, buttocks piezoelectric transducer, back piezoelectric transducer are used for human body operation behavior data snooping；Described vision terminal, operation keyboard are for machine respondent behavior data acquisition；Described multi-channel information collection is analyzed for people-machine behavioral data with the equipment of process；Described feedback control and interface arrangement, alarm appliance are for Risk-warning feedback control.
Described eye tracing photoelectric sensor is used for obtaining the real-time sight line drop point site data of people；Described feature tracking photoelectric sensor is used for obtaining the real-time expression shape change data of people；Described trunk tracing photoelectric sensor for coordinating the acquisition real-time head pose data of people with feature tracking photoelectric sensor；Described wrist piezoelectric transducer is for obtaining the pressure delta data that people's wrist acts on desktop；Described ancon piezoelectric transducer is for obtaining the pressure delta data that people's ancon acts on armrest；Described buttocks piezoelectric transducer is for obtaining the pressure delta data that people's buttocks acts on seat horizontal supporting surface；Described back piezoelectric transducer acts on the pressure delta data on the vertical supporting surface of seat for obtaining people back.
Described multi-channel information collection and process equipment for collect the incoming multi-modal information of described eye tracing photoelectric sensor, feature tracking photoelectric sensor, trunk tracing photoelectric sensor, wrist piezoelectric transducer, ancon piezoelectric transducer, buttocks piezoelectric transducer, back piezoelectric transducer and vision terminal, operation keyboard and be analyzed, decision-making.
Described feedback control and interface arrangement, for receiving the result of decision that multi-channel information collection and process equipment draw, carry out information with industry standard interface form and alarm appliance and external system or equipment mutual；Described alarm appliance is used for showing warning message and running state information being back to feedback control and interface arrangement.
Described eye tracing photoelectric sensor, feature tracking photoelectric sensor, trunk tracing photoelectric sensor, wrist piezoelectric transducer, ancon piezoelectric transducer, buttocks piezoelectric transducer, back piezoelectric transducer and vision terminal, operation keyboard are all connected with the equipment of process with multi-channel information collection with independent data cable, and all data paths are distributed parallel communication channel.Described eye tracing photoelectric sensor, feature tracking photoelectric sensor and trunk tracing photoelectric sensor obtain the real-time sight line drop point of people, expression shape change and head pose information respectively and are sent to multi-channel information collection and process equipment；Described wrist piezoelectric transducer, ancon piezoelectric transducer, buttocks piezoelectric transducer and back piezoelectric transducer obtain the force data of the wrist of health when people is present, elbow, buttocks, loin and corresponding weight-bearing surface respectively and are sent to multi-channel information collection and process equipment；Meanwhile, the content information synchronous driving of described vision terminal demonstration to multi-channel information collection with process equipment；The command signal synchronous driving that described operated key hair updo goes out is to multi-channel information collection and process equipment；Described multi-channel information collection is concentrated with the equipment of process and is received above-mentioned incoming multi-modal information, time in addition, empty registration carry out depth analysis, makes warning decision-making, and decision information is sent to feedback control and interface arrangement；Described feedback control and interface arrangement receive the warning decision information that multi-channel information collection is incoming with the equipment of process, are compiled into control instruction and are sent to alarm appliance by standard interface；Described alarm appliance receives the instruction that feedback control sends with interface arrangement, completes to pass on to people the task of appreciable information warning (feel such as vision, hearing, touch and stimulate), and the completion status of warning task is back to feedback control and interface arrangement；Meanwhile, also to support that system carries out information by industry standard interface and external equipment or system mutual for described feedback control and interface arrangement.
Present invention also offers a kind of multi-modal behavior analysis being suitable to vision terminal job post and monitoring method, the method comprises the following steps:
(1) human body behavioral data obtains；
(2) machine respondent behavior data acquisition；
(3) behavior analysis；
(4) feedback control and interface；
(5) warning performs.
Described step (1) human body behavioral data obtains the quantized data obtained for describing post human users's behavior；Described step (2) machine respondent behavior data acquisition obtains the quantized data performing operational order situation for describing post equipment；Described step (3) behavior analysis realizes the assessment of the standardization to post people-machine cooperating process and real-time risk identification and provides warning decision-making；It is mutual that described step (4) feedback control and interface carry out information according to warning decision-making and report to the police execution and external system or equipment；Described step (5) is reported to the police and is performed to realize can give post personnel feedback by sensible form, to assist it that self duty is adjusted.
Described step (1) human body behavioral data obtains and specifically includes following steps:
(11) visual behaviour data acquisition；
(12) facial expression data collection；
(13) head pose data acquisition；
(14) limbs pressure data gathers；
(15) multi-modal human body behavioural information merges.
Wherein, described step (11) visual behaviour data acquisition obtains the sight line drop point site sequence of post personnel；
Described step (12) facial expression data collection obtains the dynamic expression intensity index sequence of post personnel；
Described step (13) head pose data acquisition obtains the face orientation direction sequence of post personnel；
The pressure data collection of described step (14) limbs obtains the wrist of post personnel, elbow, the back of the body, buttocks pressure distribution data sequence；
Above-mentioned steps (11) to (15) is parallel process of real-time data acquisition, and the multi-modal human body behavioural information of the data that above-mentioned steps (11) to (14) gathers incoming step (15) simultaneously merges；Merged by step (15) multi-modal human body behavioural information and the human body behavior description data of multiple different modalities are carried out sequential registration and alignment of data, then integral data is sent to described step (3) behavior analysis.
Described step (2) machine respondent behavior data acquisition comprises the following steps:
(21) area-of-interest-detection is answered；
(22) operation response detection；
(23) machine respondent behavior registration.
Wherein, step (21) answers area-of-interest-detection acquisition vision terminal to show the zone position information should paid close attention in content in real time；Step (22) operation response detection obtains the machine response data to human users's action.The data that above-mentioned steps (21), step (22) gather each incoming step (23) machine respondent behavior registration；By step (23) machine respondent behavior registration, the machine respondent behavior of above-mentioned steps (21), step (22) is described data and carry out dimension fusion, then pass to described step (3) behavior analysis.
Described step (3) behavior analysis comprises the following steps:
(31) people-machine cooperative behaviors is analyzed；
(32) the code of conduct knowledge base；
(33) work behavior standardization assessment；
(34) real-time risk identification；
(35) remind and warning decision-making.
Wherein, described step (31) people-machine cooperative behaviors analysis receives above-mentioned steps (15) multi-modal human body behavioural information simultaneously and merges the multi-modal data stream incoming with step (23) machine respondent behavior registration and carry out data fusion, and the data after merging are respectively sent to described step (32) the code of conduct knowledge base, the assessment of step (33) work behavior standardization and step (34) risk identification in real time；In step (32) the code of conduct knowledge base, first incoming multi-modal data is carried out feature extraction and analysis, choose suitable the code of conduct priori according to analysis result and be respectively sent to the assessment of step (33) work behavior standardization and step (34) risk identification in real time, the feature of extraction is updated in knowledge base as new knowledge, for the self study of the code of conduct priori selection rule simultaneously；Assess in step (33) work behavior standardization, according to the priori that step (32) the code of conduct knowledge base provides, the compliance of behavior is carried out Quantitative marking；In step (34) risk identification in real time, according to the priori that step (32) the code of conduct knowledge base provides, the data characteristics characterizing hazardous act is identified；The analysis result incoming step (35) simultaneously of the assessment of step (33) work behavior standardization and step (34) risk identification in real time is reminded and warning decision-making；Reminded by step (35) and with warning decision-making, the state need to pointed out or warn is made a policy, and decision information is sent to step (4) feedback control and interface.
Described step (4) feedback control and interface comprise the following steps:
(41) controlling alarm；
(42) interface.
Wherein, described step (41) controlling alarm receives above-mentioned steps (35) and reminds the decision information incoming with warning decision-making, information being compiled into control instruction and is sent to step (5) warning execution by step (42) interface, receiving step (5) warning simultaneously performs the instruction execution state information fed back；Step (42) interface provides multiple industry standard interface agreement to meet different requirements analysis.
Described step (5) is reported to the police and is performed, receive the control instruction that above-mentioned steps (4) feedback control is incoming with interface, complete to make people's appreciable mode task to personnel's feedback alarm information by instruction, and by instruction completion status information back to step (4) feedback control and interface.
It is an advantage of the invention that the data acquisition by the human body behavior of multiple modalities Yu machine respondent behavior, and in conjunction with people-machine cooperative behaviors comprehensive quantification analysis, achieve the assessment of the standardization to vision terminal job post work behavior and risk identification, and by appreciable feedback system, as vision, hearing, touch feels that personnel are carried out the risk warning purpose to reach the effective management and control of behavior by stimulation mode.
Accompanying drawing explanation
Fig. 1 is the structural representation of a kind of multi-modal behavior analysis being suitable to vision terminal job post of the present invention and monitoring system；
Fig. 2 is the schematic flow sheet of a kind of multi-modal behavior analysis being suitable to vision terminal job post of the present invention and monitoring method.
Detailed description of the invention
Below in conjunction with accompanying drawing, embodiments of the present invention are described in detail.As it is shown in figure 1, a kind of multi-modal behavior analysis being suitable to vision terminal job post of the present invention includes with monitoring system:
111 eye tracing photoelectric sensors, are used for obtaining the real-time sight line drop point site data of people；
112 feature tracking photoelectric sensors, are used for obtaining the real-time expression shape change data of people；
113 trunk tracing photoelectric sensors, for coordinating the acquisition real-time head pose data of people with 112 feature tracking photoelectric sensors；
114 wrist piezoelectric transducers, for obtaining the pressure delta data that people's wrist acts on desktop；
115 ancon piezoelectric transducers, for obtaining the pressure delta data that people's ancon acts on armrest；
116 buttocks piezoelectric transducers, for obtaining the pressure delta data that people's buttocks acts on seat horizontal supporting surface；
117 back piezoelectric transducers, act on the pressure delta data on the vertical supporting surface of seat for obtaining people back；
121 vision terminals, vision terminal job nucleus equipment；
122 operation keyboards, vision terminal job nucleus equipment；
123 multi-channel information collections and the equipment of process, for collecting, process, analyze 111 eye tracing photoelectric sensors, 112 feature tracking photoelectric sensors, 113 trunk tracing photoelectric sensors, 114 wrist piezoelectric transducers, 115 ancon piezoelectric transducers, 116 buttocks piezoelectric transducers, 117 back piezoelectric transducers, 121 vision terminals, the 122 operation incoming multi-modal informations of keyboards, and make warning decision-making according to analyzing result；
131 feedback control and interface arrangement, be used for according to warning decision making feedback control instruction, and it is mutual to carry out information by industry standard interface agreement and external system or equipment.
132 alarm appliances, for waking up to Crinis Carbonisatus optical signals to warn and running status feed back to 131 feedback control and interface arrangement.
Described 111 eye tracing photoelectric sensors, 112 feature tracking photoelectric sensors, 113 trunk tracing photoelectric sensors, 114 wrist piezoelectric transducers, 115 ancon piezoelectric transducers, 116 buttocks piezoelectric transducers, 117 back piezoelectric transducers, 121 vision terminals, 122 operation keyboards are all connected with the equipment of process with 123 multi-channel information collections with data cable.
Described 131 feedback control are connected with interface arrangement and 132 alarm appliances.
Described 111 eye tracing photoelectric sensors, 112 feature tracking photoelectric sensors, 113 trunk tracing photoelectric sensors, 114 wrist piezoelectric transducers, 115 ancon piezoelectric transducers, 116 buttocks piezoelectric transducers, 117 back piezoelectric transducers, 121 vision terminals, 122 operation keyboards are all connected with interface arrangement with 131 feedback control with data cable, and all data paths are distributed parallel communication channel.Specifically, described 111 eye tracing photoelectric sensors, 112 feature tracking photoelectric sensors and 113 trunk tracing photoelectric sensors obtain the real-time sight line drop point of people, expression shape change and head pose information respectively and are sent to 123 multi-channel information collections and process equipment；Described 114 wrist piezoelectric transducers, 115 ancon piezoelectric transducers, 116 buttocks piezoelectric transducers and 117 back piezoelectric transducers obtain the force data in the wrist of health when people is present, elbow, buttocks, loin and face of weighing accordingly respectively and are sent to 123 multi-channel information collections and process equipment；Meanwhile, the content information synchronous driving of described 121 vision terminal demonstration to 123 multi-channel information collections with process equipment；The command signal synchronous driving that described 122 operated key hair updos go out is to 123 multi-channel information collections and process equipment；Described 123 multi-channel information collections and the equipment of process concentrate the multi-modal information that the described 111 eye tracing photoelectric sensors of reception, 112 feature tracking photoelectric sensors, 113 trunk tracing photoelectric sensors, 114 wrist piezoelectric transducers, 115 ancon piezoelectric transducers, 116 buttocks piezoelectric transducers, 117 back piezoelectric transducers, 121 vision terminals, 122 operation keyboards are incoming, time in addition, empty registration carry out depth analysis, make warning decision-making and decision information be sent to 131 feedback control and interface arrangement；Described 131 feedback control and interface arrangement receive the decision information that 123 multi-channel information collections are incoming with process equipment, it is compiled into control instruction and by standard interface, instruction is sent to 132 alarm appliances, supporting to carry out information with industry standard interface protocol form and external system or equipment mutual simultaneously；Described 132 alarm appliances receive the controlling alarm instruction incoming from interface arrangement of 131 feedback control and send the optical signal of different wave length and the different frequencys and pass on the information warning of different levels to people, and instruction execution state feeds back to 123 multi-channel information collections and process equipment simultaneously.
As in figure 2 it is shown, a kind of multi-modal behavior analysis being suitable to vision terminal job post of the present invention comprises the following steps with monitoring method:
Step 21, is obtained the quantized data for describing post human users's behavior by human body behavioral data acquiring unit；Described human body behavioral data acquiring unit comprises 111 eye tracing photoelectric sensors in Fig. 1,112 feature tracking photoelectric sensors, 113 trunk tracing photoelectric sensors, 114 wrist piezoelectric transducers, 115 ancon piezoelectric transducers, 116 buttocks piezoelectric transducers, 117 back piezoelectric transducers.
Step 22, is obtained the quantized data performing operational order situation for describing post equipment by machine respondent behavior data capture unit；Described machine respondent behavior data capture unit comprises 121 vision terminals in Fig. 1,122 operation keyboards.
Step 23, is realized the standardization to post people-machine cooperating process by behavior analysis unit and assesses and real-time risk identification, make warning decision-making；Described behavior analysis unit comprises 123 multi-channel information collections in Fig. 1 and process equipment.
Step 24, is realized decision information being compiled into control instruction and instruction being spread out of by industry standard interface by feedback control and interface unit, and provides information mutual for external system or equipment；Described feedback control and interface unit comprise 131 feedback control in Fig. 1 and interface arrangement.
Step 25, is performed feedback control instruction by warning performance element, by instruction to make the appreciable mode of people to personnel's feedback alarm information, assists it that self duty is adjusted, instruction execution state is back to step 24 simultaneously；Described warning performance element comprises 132 alarm appliance in Fig. 1.
Described step 21 includes: obtained the sight line drop point site sequence of post personnel by step 211 visual behaviour data acquisition module；The dynamic expression intensity index sequence of post personnel is obtained by step 212 facial expression data acquisition module；The face orientation direction sequence of post personnel is obtained by step 213 head pose data acquisition module；And obtained the wrist of post personnel, elbow, the back of the body, buttocks pressure distribution data sequence by step 214 limbs pressure data acquisition module, above-mentioned steps 211, step 212, step 213 and step 214 are parallel process of real-time data acquisition, the multi-modal human body behavioural information Fusion Module of data incoming step 215 simultaneously that each step gathers.
In step 211, utilize high speed optoelectronic sensor, 111 eye tracing photoelectric sensor in Fig. 1, obtain personnel's eye video image, sample rate and be not limited within the scope of 30Hz-60Hz, obtain sight line drop point site sequence by the eyeball posture analysis algorithm based on high-speed video.
In the step 212, utilize photoelectric sensor, 112 feature tracking photoelectric sensor in Fig. 1, acquisition personnel's face video image, sample rate and be not limited within the scope of 15Hz-25Hz, by obtaining dynamic expression intensity index sequence based on the face feature point extraction algorithm of video and expression space affine model.
In step 213, utilize binocular photoelectric sensor group, 112 feature tracking photoelectric sensors and 113 trunk tracing photoelectric sensors in Fig. 1, obtain personnel's face and (containing head, cervical region) video image above the waist simultaneously, sample rate all and be not limited within the scope of 15Hz-25Hz, obtain face orientation direction sequence by the head pose analysis based on stereoscopic vision.
In step 214, utilize distributed pressure sensor net, 114 wrist piezoelectric transducers, 115 ancon piezoelectric transducers, 116 buttocks piezoelectric transducers, 117 back piezoelectric transducers in Fig. 1, acquisition personnel are the pressure distribution data sequence at buttocks and back when operating keyboard or wrist and ancon and sitting posture during mouse, sample rate for and be not limited to 5Hz.
In step 215, the human body behavior description data carrying out multiple modalities carry out sequential registration.Owing to the sequential of each modal data incoming parallel is inconsistent, so each modal data need to be fitted regeneration by same sample rate in this step, completion is because of the low data room caused of sensor self sample rate, it is ensured that alignment of data.Then the multi-modal data after registration is integrated into human body behavioural information and is sent to step 23.
Described step 22 includes: is answered area-of-interest-detection module to obtain vision terminal by step 221 and shows the zone position information should paid close attention in content in real time；Operated response detection module by step 222 and obtain the machine response data to human users's action.Above-mentioned steps 221 is process of real-time data acquisition, step 222 is the triggering non-real-time data of event, data each incoming step 223 machine respondent behavior registration module that two steps gather.
In step 221, film recording is utilized to obtain vision terminal demonstration content, namely personnel can watch the synchrodata of content, sample rate is 10Hz-15Hz, in conjunction with relevant priori, answer the position sequence of region-of-interest by obtaining personnel in dynamic process based on the related content track algorithm of video analysis.Specific algorithm repeats no more herein.
In step 222, utilize keyboard, mouse signal to catch acquisition personnel's key command coding signal, obtained the data stream of personnel's trigger action instruction by known decoding algorithm.
In step 223, utilize absolute time tag that sparse triggering command data incoming for step 222 are imported the incoming sample rate of step 221 in the continuous data stream of 10Hz-15Hz, by Data Dimensionality Reduction compression algorithm by regular for two modal datas for low-dimensional blended data, and it is sent to step 23 as machine respondent behavior information.
Described step 23 includes: the multi-modal data stream incoming with step 223 by step 231 people-machine cooperating analysis module receiving step 215 simultaneously also carries out data fusion, and the data after merging are respectively sent to step 232 the code of conduct knowledge base, step 233 work behavior standardization evaluation module and step 234 risk identification module in real time；Chosen suitable the code of conduct knowledge by step 232 the code of conduct knowledge base according to data characteristics and be respectively sent to step 233 and step 234；The compliance of behavior is carried out Quantitative marking by the priori provided according to step 232 the code of conduct knowledge base by step 233；The data characteristics characterizing hazardous act is identified by the priori provided according to step 232 the code of conduct knowledge base by step 234；The analysis result incoming step 235 simultaneously of step 233 and step 234 is reminded and warning decision-making module, step 235 state need to pointed out or warn is made warning decision-making.
In step 231, the multi-modal data stream that receiving step 215 is incoming with step 223 simultaneously, first deadline pairing, namely the machine respondent behavior information that step 223 is incoming is carried out content understanding, obtain rational minimum analysis time window, utilize minimum analysis time window that the human body behavioural information of the incoming high sampling rate of step 215 is carried out segmentation and matched with machine respondent behavior；Complete space pairing again, carry out space pairing by the sight line drop point site sequence in human body behavioural information with the region-of-interest position sequence of answering in machine respondent behavior information；When utilizing, the result of empty pairing carry out statistical analysis, obtain ASSOCIATE STATISTICS amount array (as density and the divergence of vision attention, the degree of stability of operating attitude, operation machine frequent degree) realize the quantificational description of people-machine cooperative behaviors.
In step 232, first incoming multi-modal data is carried out feature extraction and analysis, choose suitable the code of conduct priori according to analysis result and be respectively sent to step 233 and step 234, the feature of extraction is updated in knowledge base as new knowledge, for the self study of the code of conduct priori selection rule simultaneously.Concrete self-learning algorithm repeats no more herein.
In step 233, in the code of conduct priori provide ASSOCIATE STATISTICS amount array incoming for step 231 and step 232, pattern lack of standardization carries out Pattern similarity calculating, sends result of calculation to step 235.
In step 234, in the code of conduct priori provide ASSOCIATE STATISTICS amount array incoming for step 231 and step 232, limit risk carries out Pattern similarity calculating, sends result of calculation to step 235.
In this step 235, the limit risk matching similarity that first analytical procedure 234 is incoming, if the similarity of certain pattern is higher than set alarm threshold value, then generates warning message and be sent to step 24；If limit risk does not mate, then the pattern match similarity lack of standardization that analytical procedure 233 is incoming, all Pattern similarities lack of standardization are weighted, if value of calculation is higher than set prompting threshold value, then generates prompting message and be sent to step 24.
Described step 24 includes: the warning decision information incoming by step 241 alarm control module receiving step 235, generates control instruction and is sent to step 242 interface module；Step 242 interface module realizes the spreading out of of control instruction, the reception of respective feedback information and provides information mutual outside system or equipment.
In step 241, decision information incoming for step 235 is compiled into corresponding controlling instruction code and spreads and give step 242.
In step 242, control instruction code stream incoming for step 241 is packaged into industry standard interface protocol form data, is sent to step 25 feedback information that receiving step 25 is passed back；Thering is provided simultaneously and carry out information alternately with industry standard interface protocol form and external system or equipment (i.e. step 30), described industry standard interface agreement includes: RS232, RJ45, USB wired data transfer agreement, ZigBee, bluetooth, Wi-Fi home control network communication protocol.
Described step 25, the normal data that receiving step 242 is incoming, corresponding control instruction is obtained through decoding, by instruction execution action, appreciable signal is sent to people, such as light, sound, vibration, to realize warning and to assist people to adjust the purpose of self duty, instruction execution state information feedback is sent to step 242 simultaneously.
The above; being only the specific embodiment of the present invention, but protection scope of the present invention is not limited thereto, any those familiar with the art is in scope disclosed by the invention; the change that can readily occur in or replacement, all should be encompassed in the protection domain of the claims in the present invention.
Claims (8)
1. the multi-modal behavior analysis being suitable to vision terminal job post and monitoring system, it is characterized in that, including: eye tracing photoelectric sensor, feature tracking photoelectric sensor, trunk tracing photoelectric sensor, wrist piezoelectric transducer, ancon piezoelectric transducer, buttocks piezoelectric transducer, back piezoelectric transducer, vision terminal, operation keyboard, multi-channel information collection and the equipment of process, feedback control and interface arrangement and alarm appliance；
Described eye tracing photoelectric sensor, feature tracking photoelectric sensor, trunk tracing photoelectric sensor, wrist piezoelectric transducer, ancon piezoelectric transducer, buttocks piezoelectric transducer, back piezoelectric transducer and vision terminal, operation keyboard are all connected with the equipment of process with multi-channel information collection with independent data cable；
Described multi-channel information collection is connected with interface arrangement with the equipment of process and feedback control；
Described feedback control is connected with interface arrangement and alarm appliance.
2. the multi-modal behavior analysis being suitable to vision terminal job post and monitoring method, it is characterised in that comprise the following steps:
(1) human body behavioral data acquiring unit；
(2) machine respondent behavior data capture unit；
(3) behavior analysis unit；
(4) feedback control and interface unit；
(5) warning performance element.
3. be suitable to multi-modal behavior analysis and the monitoring method in vision terminal job post as claimed in claim 2, it is characterised in that:
Described step (1) human body behavioral data acquiring unit obtains the quantized data for describing post human users's behavior；Described step (2) machine respondent behavior data capture unit obtains the quantized data performing operational order situation for describing post equipment；Described step (3) behavior analysis unit realizes the assessment of the standardization to post people-machine cooperating process and real-time risk identification and provides warning decision-making；It is mutual that described step (4) feedback control and interface unit carry out information according to warning decision-making and warning performance element and external system or equipment；Described step (5) warning performance element realizes can give post personnel feedback by sensible form, to assist it that self duty is adjusted.
4. be suitable to multi-modal behavior analysis and the monitoring method in vision terminal job post as claimed in claim 2, it is characterised in that:
Described step (1) human body behavioral data obtains and specifically includes following steps:
(11) visual behaviour data acquisition；
(12) facial expression data collection；
(13) head pose data acquisition；
(14) limbs pressure data gathers；
(15) multi-modal human body behavioural information merges；
Wherein, described step (11) visual behaviour data acquisition obtains the sight line drop point site sequence of post personnel；
Described step (12) facial expression data collection obtains the dynamic expression intensity index sequence of post personnel；
Described step (13) head pose data acquisition obtains the face orientation direction sequence of post personnel；
The pressure data collection of described step (14) limbs obtains the wrist of post personnel, elbow, the back of the body, buttocks pressure distribution data sequence；
Above-mentioned steps (11) to (15) is parallel process of real-time data acquisition, and the multi-modal human body behavioural information of the data that above-mentioned steps (11) to (14) gathers incoming step (15) simultaneously merges；Merged by step (15) multi-modal human body behavioural information and the human body behavior description data of multiple different modalities are carried out sequential registration and alignment of data, then integral data is sent to described step (3) behavior analysis.
5. be suitable to multi-modal behavior analysis and the monitoring method in vision terminal job post as claimed in claim 2, it is characterised in that:
Described step (2) machine respondent behavior data acquisition comprises the following steps:
(21) area-of-interest-detection is answered；
(22) operation response detection；
(23) machine respondent behavior registration；
Wherein, step (21) answers area-of-interest-detection acquisition vision terminal to show the zone position information should paid close attention in content in real time；Step (22) operation response detection obtains the machine response data to human users's action, the data that above-mentioned steps (21), step (22) gather each incoming step (23) machine respondent behavior registration；By step (23) machine respondent behavior registration, the machine respondent behavior of above-mentioned steps (21), step (22) is described data and carry out dimension fusion, then pass to described step (3) behavior analysis.
6. be suitable to multi-modal behavior analysis and the monitoring method in vision terminal job post as claimed in claim 2, it is characterised in that:
Described step (3) behavior analysis comprises the following steps:
(31) people-machine cooperative behaviors is analyzed；
(32) the code of conduct knowledge base；
(33) work behavior standardization assessment；
(34) real-time risk identification；
(35) remind and warning decision-making；
Wherein, described step (31) people-machine cooperative behaviors analysis receives above-mentioned steps (15) multi-modal human body behavioural information simultaneously and merges the multi-modal data stream incoming with step (23) machine respondent behavior registration and carry out data fusion, and the data after merging are respectively sent to described step (32) the code of conduct knowledge base, the assessment of step (33) work behavior standardization and step (34) risk identification in real time；In step (32) the code of conduct knowledge base, first incoming multi-modal data is carried out feature extraction and analysis, choose suitable the code of conduct priori according to analysis result and be respectively sent to the assessment of step (33) work behavior standardization and step (34) risk identification in real time, the feature of extraction is updated in knowledge base as new knowledge, for the self study of the code of conduct priori selection rule simultaneously；Assess in step (33) work behavior standardization, according to the priori that step (32) the code of conduct knowledge base provides, the compliance of behavior is carried out Quantitative marking；In step (34) risk identification in real time, according to the priori that step (32) the code of conduct knowledge base provides, the data characteristics characterizing hazardous act is identified；The analysis result incoming step (35) simultaneously of the assessment of step (33) work behavior standardization and step (34) risk identification in real time is reminded and warning decision-making；Reminded by step (35) and with warning decision-making, the state need to pointed out or warn is made a policy, and decision information is sent to step (4) feedback control and interface.
7. be suitable to multi-modal behavior analysis and the monitoring method in vision terminal job post as claimed in claim 2, it is characterised in that:
Described step (4) feedback control and interface comprise the following steps:
(41) controlling alarm；
(42) interface；
Wherein, described step (41) controlling alarm receives above-mentioned steps (35) and reminds the decision information incoming with warning decision-making, information being compiled into control instruction and is sent to step (5) warning execution by step (42) interface, receiving step (5) warning simultaneously performs the instruction execution state information fed back；Step (42) interface provides multiple industry standard interface agreement to meet different requirements analysis.
8. be suitable to multi-modal behavior analysis and the monitoring method in vision terminal job post as claimed in claim 2, it is characterised in that:
Described step (5) is reported to the police and is performed, receive the control instruction that above-mentioned steps (4) feedback control is incoming with interface, complete to make people's appreciable mode task to personnel's feedback alarm information by instruction, and by instruction completion status information back to step (4) feedback control and interface.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510145902.9A CN105759677B (en) | 2015-03-30 | 2015-03-30 | A kind of multi-modal behavioural analysis and monitoring system and method suitable for vision terminal job post |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510145902.9A CN105759677B (en) | 2015-03-30 | 2015-03-30 | A kind of multi-modal behavioural analysis and monitoring system and method suitable for vision terminal job post |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105759677A true CN105759677A (en) | 2016-07-13 |
| CN105759677B CN105759677B (en) | 2018-11-20 |
Family
ID=56341761
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510145902.9A Active CN105759677B (en) | 2015-03-30 | 2015-03-30 | A kind of multi-modal behavioural analysis and monitoring system and method suitable for vision terminal job post |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105759677B (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108482426A (en) * | 2018-04-27 | 2018-09-04 | 大连奇辉计算机网络有限公司 | A kind of button tracing system to relay system interlocking consing |
| CN111126771A (en) * | 2019-11-26 | 2020-05-08 | 公安部第一研究所 | Safety inspector image recognition quality supervision and guarantee system and method based on regional attention prediction |
| CN111562267A (en) * | 2020-05-29 | 2020-08-21 | 重庆施鲁逊智能科技有限公司 | Visual detection system for assembling automobile fuse box |
| WO2021068638A1 (en) * | 2019-10-12 | 2021-04-15 | 中国海洋大学 | Interactive intenstive learning method that combines tamer framework and facial expression feedback |
| CN115941514A (en) * | 2022-09-29 | 2023-04-07 | 国网浙江省电力有限公司桐乡市供电公司 | Unmanned aerial vehicle self-inspection system and method based on 5G slicing technology |
| CN116452051A (en) * | 2023-04-19 | 2023-07-18 | 浙江乐恒动力科技股份有限公司 | Interactive digital production management system and management method |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102982316A (en) * | 2012-11-05 | 2013-03-20 | 安维思电子科技（广州）有限公司 | Driver abnormal driving behavior recognition device and method thereof |
| CN103268495A (en) * | 2013-05-31 | 2013-08-28 | 公安部第三研究所 | Human body behavioral modeling identification method based on priori knowledge cluster in computer system |
| CN103347446A (en) * | 2010-12-10 | 2013-10-09 | Tk控股公司 | System for monitoring a vehicle driver |
| CN103632543A (en) * | 2013-11-04 | 2014-03-12 | 安徽博仑特智能科技有限公司 | Intelligent cloud vehicle management system device |
| CN204440098U (en) * | 2015-03-30 | 2015-07-01 | 公安部第一研究所 | A kind of multi-modal behavioural analysis and supervisory system being suitable for vision terminal job post |
-
2015
- 2015-03-30 CN CN201510145902.9A patent/CN105759677B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103347446A (en) * | 2010-12-10 | 2013-10-09 | Tk控股公司 | System for monitoring a vehicle driver |
| CN102982316A (en) * | 2012-11-05 | 2013-03-20 | 安维思电子科技（广州）有限公司 | Driver abnormal driving behavior recognition device and method thereof |
| CN103268495A (en) * | 2013-05-31 | 2013-08-28 | 公安部第三研究所 | Human body behavioral modeling identification method based on priori knowledge cluster in computer system |
| CN103632543A (en) * | 2013-11-04 | 2014-03-12 | 安徽博仑特智能科技有限公司 | Intelligent cloud vehicle management system device |
| CN204440098U (en) * | 2015-03-30 | 2015-07-01 | 公安部第一研究所 | A kind of multi-modal behavioural analysis and supervisory system being suitable for vision terminal job post |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108482426A (en) * | 2018-04-27 | 2018-09-04 | 大连奇辉计算机网络有限公司 | A kind of button tracing system to relay system interlocking consing |
| CN108482426B (en) * | 2018-04-27 | 2023-05-30 | 大连鼎汉奇辉电子系统工程有限公司 | Key tracking system for operating relay interlocking control console |
| WO2021068638A1 (en) * | 2019-10-12 | 2021-04-15 | 中国海洋大学 | Interactive intenstive learning method that combines tamer framework and facial expression feedback |
| CN111126771A (en) * | 2019-11-26 | 2020-05-08 | 公安部第一研究所 | Safety inspector image recognition quality supervision and guarantee system and method based on regional attention prediction |
| CN111126771B (en) * | 2019-11-26 | 2023-06-20 | 公安部第一研究所 | Safety inspector identification picture quality supervision and guarantee system and method based on regional attention prediction |
| CN111562267A (en) * | 2020-05-29 | 2020-08-21 | 重庆施鲁逊智能科技有限公司 | Visual detection system for assembling automobile fuse box |
| CN111562267B (en) * | 2020-05-29 | 2023-07-21 | 重庆施鲁逊智能科技有限公司 | Visual inspection system for automobile fuse box assembly |
| CN115941514A (en) * | 2022-09-29 | 2023-04-07 | 国网浙江省电力有限公司桐乡市供电公司 | Unmanned aerial vehicle self-inspection system and method based on 5G slicing technology |
| CN116452051A (en) * | 2023-04-19 | 2023-07-18 | 浙江乐恒动力科技股份有限公司 | Interactive digital production management system and management method |
| CN116452051B (en) * | 2023-04-19 | 2023-09-12 | 浙江乐恒动力科技股份有限公司 | Interactive digital production management system and management method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105759677B (en) | 2018-11-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105759677A (en) | Multimode behavior analysis and monitoring system and method adapted to visual terminal operation post | |
| Wang et al. | From personalized medicine to population health: a survey of mHealth sensing techniques | |
| CN104834902A (en) | Safety check graph discrimination system including video analysis and graph discrimination method | |
| CN103340637A (en) | System and method for driver alertness intelligent monitoring based on fusion of eye movement and brain waves | |
| CN103699226A (en) | Tri-modal serial brain-computer interface method based on multi-information fusion | |
| CN106293102A (en) | A kind of robot affective interaction method based on user mood change emotion | |
| Tavakoli et al. | Multimodal driver state modeling through unsupervised learning | |
| CN103144098A (en) | Household service robot | |
| CN108968989A (en) | One kind based on it is psychologic stress training system and its application method | |
| CN204926116U (en) | Picture system is declared in safety inspection containing video analysis | |
| Yi et al. | [Retracted] Home Interactive Elderly Care Two‐Way Video Healthcare System Design | |
| WO2007085303A1 (en) | Information overlay system | |
| CN114237391A (en) | Urban rail transit dispatching virtual training test system and method thereof | |
| CN104257349A (en) | Mirror and method for monitoring health of user through integration of image processing and heart rate characteristics | |
| CN204440098U (en) | A kind of multi-modal behavioural analysis and supervisory system being suitable for vision terminal job post | |
| Garrido et al. | Automatic detection of falls and fainting | |
| CN107736893A (en) | mental emotion monitoring system based on mobile device | |
| CN103366153A (en) | Semantic cognitive facial feature identification method | |
| Du et al. | Personalize wayfinding information for fire responders based on virtual reality training data | |
| CN103258107A (en) | Monitoring method and assistant monitoring system | |
| Zhou et al. | Driver fatigue tracking and detection method based on OpenMV | |
| CN106843478A (en) | The method and system of multiple computers are controlled by single input equipment automatic conversion | |
| WO2023286105A1 (en) | Information processing device, information processing system, information processing program, and information processing method | |
| CN110135744A (en) | Construction worker's safety behavior is accustomed to evaluation method | |
| Mubashira et al. | A Comprehensive Study on Human Interaction with IoT Systems |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |