WO2018229592A1 - Procédé et système d'évaluation et de contrôle de conformité au moyen de la détection d'émotion - Google Patents

Procédé et système d'évaluation et de contrôle de conformité au moyen de la détection d'émotion Download PDF

Info

Publication number
WO2018229592A1
WO2018229592A1 PCT/IB2018/053968 IB2018053968W WO2018229592A1 WO 2018229592 A1 WO2018229592 A1 WO 2018229592A1 IB 2018053968 W IB2018053968 W IB 2018053968W WO 2018229592 A1 WO2018229592 A1 WO 2018229592A1
Authority
WO
WIPO (PCT)
Prior art keywords
subject
task
data
compliance
concept
Prior art date
Application number
PCT/IB2018/053968
Other languages
English (en)
Inventor
Yuen Lee Viola LAM
Johan Matthijs DOLSMA
Original Assignee
Lam Yuen Lee Viola
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US15/647,272 external-priority patent/US20180232567A1/en
Application filed by Lam Yuen Lee Viola filed Critical Lam Yuen Lee Viola
Priority to JP2019600105U priority Critical patent/JP3223411U/ja
Priority to CN201880003286.5A priority patent/CN109716382A/zh
Priority to US16/313,895 priority patent/US11475788B2/en
Publication of WO2018229592A1 publication Critical patent/WO2018229592A1/fr
Priority to US17/960,835 priority patent/US20230105077A1/en

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
    • G06Q50/10—Services
    • G06Q50/20—Education
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
    • G06Q10/063—Operations research, analysis or management
    • G06Q10/0639—Performance analysis of employees; Performance analysis of enterprise or organisation operations
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09B—EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
    • G09B7/00—Electrically-operated teaching apparatus or devices working with questions and answers
    • G09B7/02—Electrically-operated teaching apparatus or devices working with questions and answers of the type wherein the student is expected to construct an answer to the question which is presented or wherein the machine gives an answer to the question presented by a student
    • G09B7/04—Electrically-operated teaching apparatus or devices working with questions and answers of the type wherein the student is expected to construct an answer to the question which is presented or wherein the machine gives an answer to the question presented by a student characterised by modifying the teaching programme in response to a wrong answer, e.g. repeating the question, supplying a further explanation

Definitions

  • the present invention relates generally to methods and systems for providing and delivery of compliance administration and monitoring in the contexts of educational programmes and training, including corporate training, academic tutoring, in-class and out-of-class learnings, medical treatment and health improvement programmes, sport training, fitness and lifestyle programmes, correctional service and rehabilitation programmes, as well as governmental law and regulatory enforcement, and behavioral standards for individuals.
  • the present invention relates to methods and systems for administering, evaluating and monitoring compliance with task performance requirements in an action programme.
  • Compliance means conforming to certain task performance specifications as required by an action programme by a subject enrolled therein.
  • Conventional compliance evaluation and monitoring techniques focus on providing one-time testing with clearly defined passing criteria to subjects in an action programme. Many of these conventional techniques focus only on obtaining pass/fail indicators as effectively as possible without any capability for predicting the subject's progress or guiding the subject towards total compliance. Further, these traditional compliance evaluation and monitoring methods often rely on one-time or sparse, at best, manual administrations of tests. This can hardly provide assurance for compliance on a continuous basis.
  • the present invention provides a method and system for administering, evaluating, and monitoring a subject's compliance with task performance requirements within an action programme using one or more of sensing of the subject's gestures, emotions, and movements, speech and voice recognition, behavior pattern recognition, and quantitative measurements of questionnaire results and task performances.
  • the subject within the action programme is to perform certain tasks according to performance specifications.
  • the method and system evaluate and monitor the compliance of the subject by periodically administering one or more questionnaires and analyzing the subject's responses to the questionnaires; and by continuously monitoring the subject's performance of one or more tasks under performance specification requirements.
  • the method and system estimates the affective state and cognitive state of the subject by image and/or video capturing and analyzing the subject's facial expression, eye movements, point-of-gaze, and head pose; and physiologic detection, such as tactile pressure exerted on a tactile sensing device, subject's handwriting, tone of voice, and speech clarity during the subject is responding to the questionnaire or during a sampling time window when the subject is performing the task procedure.
  • the image or video capture can be accomplished by using built-in or peripheral cameras in desktop computers, laptop computers, tablet computers, and/or smartphones used by the subject in responding to the questionnaire, and/or other optical sensing devices placed and installed in the environments within which the subject performs the tasks in the action programme.
  • the captured images and/or videos are then analyzed using machine vision techniques. For example, stalled eye movements, out-of-focus point-of-gaze, and a tilted head pose are signals indicating lack of interest and attention toward, and/or lack of knowledge in the subject matters being presented in the questionnaire and task procedural instructions, untruthfulness in answering the questionnaire, or lack of skill/knowledge in the tasks at hands; while a strong tactile pressure detected is a signal indicating anxiety, lack of confidence, and/or frustration in the subject matters being presented in the questionnaire and task procedural instructions or of the tasks at hands; either could represent a tendency of low level of compliance or noncompliance.
  • selected performance data and behavioral data from the subject are also collected in determining the subject's comprehension of and level of engagement in the materials presented in the questionnaires and task procedural instructions for tasks required to be performed in the action programme.
  • selected performance data and behavioral data include, but not limited to, correctness of answers to questions in the questionnaire, number of successful and unsuccessful attempts, closeness of the subject's answers to model answers, number of toggling between given answer choices, and response speed to questions of certain types, and subject matters. For example, the subject's excessive toggling between given choices and slow response speed in answering a question indicate doubts and hesitations on the answer to the question.
  • the affective state and cognitive state estimation and performance data are primarily used in gauging the subject's level of compliance with performance specifications of tasks in an action programme. While a single estimation is used in providing a snapshot assessment of the subject's progress toward total compliance in her task performance and prediction of the subject's eventual achievable level of compliance, multiple estimations are used in providing an assessment history and trends of the subject's progress. Furthermore, the estimated affective states and cognitive states of the subject are used in the modeling of the compliance programme in terms of choice of methods of compliance evaluation and monitoring, and instruction delivery and administration.
  • the method and system provide a mechanism for delivering and managing interactive and adaptive compliance questionnaire and task procedural instructions.
  • the mechanism logically structures the questionnaire and task procedural instructions materials and the delivery mechanism data for evaluating and monitoring compliance in an action programme as Domain Knowledge, and its data are stored in a Domain Knowledge repository.
  • a Domain Knowledge repository comprises one or more Concept objects and one or more Task objects.
  • Each Concept object comprises one or more Knowledge and Skill items.
  • Knowledge and Skill items are ordered by task performance specification complexity/difficulty/stringency levels, and two or more Knowledge and Skill items can be linked to form a Curriculum.
  • a Curriculum defined by the present invention may be the equivalence of the operation manual/standard and there is one-to-one relationship between a Knowledge and Skill item and a task performance specification in the operation manual/standard.
  • the Concept objects can be linked to form a logical tree data structure for used in a Task selection process.
  • Each Task object has various task procedural instruction materials.
  • Each Task object is associated with one or more Concept objects in a Curriculum.
  • a Task object can be classified as: Basic Task, Interactive Task, or Task with an Underlying Cognitive or Expert Model.
  • Each Basic Task comprises one or more operation notes, task procedural instructions, illustrations, test questions and answers designed to assess whether the subject has read all the materials.
  • Cognitive or Expert Model comprises one or more task procedures each comprises one or more instructional steps designed to guide the subject in completing the task procedure according to performance specification. Each step provides an answer, common misconceptions, and hints. The steps are in the order designed to follow the delivery flow of a task procedure. This allows a tailored scaffolding (e.g. providing guidance and/or hints) for each task based on a point in a task procedure executed.
  • a tailored scaffolding e.g. providing guidance and/or hints
  • the mechanism for delivering and managing interactive and adaptive compliance questionnaires and instructions logically builds on top of the Domain
  • the system executes each of one or more of the Task objects associated with a Curriculum in a Domain Knowledge in a work session for a subject.
  • the system measures the subject's performance and obtain the subject's performance metrics in each Task such as: the numbers of successful and unsuccessful attempts to complete the instructional steps in the Task, number of hints requested, and the time spent in completing the Task.
  • the performance metrics obtained, along with the information of the Task object, such as its specification complexity/difficulty/stringency level, are fed into a logistic regression mathematical model of each Concept object associated with the Task object.
  • the advantages of the Subject Model include that the execution of the Task objects can adapt to the changing ability of the subject. For non-limiting example, following the Subject Model, the system can estimate the compliance level achievable by the subject, estimate how much performance improvement can be expected for a next Task, and provide a prediction of the subject's level of compliance in a future point of time. These data are then used in the Training Model and enable hypothesis testing to make further improvement to the system, evaluate compliance officer quality and compliance questionnaire and task procedural instruction material quality.
  • the system receives the data collected from the execution of the Task objects under the Subject Model and the Domain Knowledge for making decisions on the instruction delivery strategy and providing feedbacks to the subject and compliance officer.
  • the system is mainly responsible for executing the folio wings:
  • the system's trainer module matches the current affective state of the subject with the available states in the pedagogical agent. Besides providing the affective state information, text messages can be sent to the system's communication module for rendering the pedagogical agent in a user interface.
  • the system estimates the probability of the subject achieving a target compliance level in a task associated with the task performance specification materials in each Concept. Based on a predetermined threshold (e.g. 95%), the compliance officer can decide when a Concept is mastered.
  • a predetermined threshold e.g. 95%
  • the method and system for administering, evaluating, and monitoring a subject's compliance with task performance requirements within an action incorporate machine learning techniques that are based on interlinking models of execution comprising: a Domain Model, an Assessment Model, a Learner Model, a Deep Learner Model, one or more Motivational Models, a Transition
  • the interlinking models of execution is purposed for driving, inducing, or motivating certain desirable actions, behavior, and/or outcome from the subject. These certain desirable actions and/or outcome can be, as non-limiting examples, mastering certain task procedures, adopting certain desirable behaviors, achieving certain job assignment goals, making certain purchases, and conducting certain commercial activities. Therefore, these interlinking models of execution are also applicable in the fields of corporate training and commercial retailing and trading.
  • FIG. 1 depicts a schematic diagram of a system for administering, evaluating, and monitoring a subject's compliance with task performance requirements within an action programme in accordance to one embodiment of the present invention
  • FIG. 2 depicts a logical data flow diagram of the system
  • FIG. 3 depicts an activity diagram of a method for administering, evaluating, and monitoring a subject's compliance with task performance requirements within an action programme to one embodiment of the present invention
  • FIG. 4 depicts a flow diagram of an iterative machine learning workflow used by the system in calculating a probability of achieving a target compliance level by the subject;
  • FIG. 5 illustrates a logical data structure used by the system in accordance to one embodiment of the present invention
  • FIG. 6A depicts a logical block diagram of interlinking models of execution in accordance to one aspect of the present invention.
  • FIG. 6B shows the logical components details of the interlinking models of execution.
  • the method and system for administering, evaluating, and monitoring a subject's compliance with task performance requirements within an action programme use a combination of sensing of the subject's gestures, emotions, and movements, and quantitative measurements of questionnaire results and task performances.
  • the method and system estimate the affective state and cognitive state of the subject by image and/or video capturing and analyzing the subject's facial expression, eye movements, point-of-gaze, and head pose, and haptic feedback, such as tactile pressure exerted on a tactile sensing, subject's handwriting, tone of voice, and speech clarity during the subject is responding to the questionnaire or during a sampling time window when the subject is performing the task procedure.
  • the image or video capture can be performed by using built-in or peripheral cameras in desktop computers, laptop computers, tablet computers, and/or smartphones used by the subject, and/or other optical sensing devices placed and installed in the environments within which the subject performs the tasks in the action programme.
  • the captured images and/or videos are then analyzed using machine vision techniques. For example, stalled eye movements, out-of-focus point-of-gaze, and a tilted head pose are signals indicating lack of interest and attention, and/or lack of knowledge in the subject matters being presented in questionnaire, untruthfulness in answering the questionnaire, or lack of skill / knowledge in the tasks at hands; while a strong tactile pressure detected is a signal indicating anxiety, lack of confidence, and/or frustration in the subject matters being presented in questionnaire or of the tasks at hands; either could represent a tendency of low level of compliance or noncompliance.
  • selected performance data and behavioral data from the subject are also collected in the affective state and cognitive state estimation.
  • These selected performance data and behavioral data include, but not limited to, number of successful and unsuccessful attempts to task procedural step completions, speed in completing task procedures, correctness of answers to questions in the questionnaire, number of successful and unsuccessful attempts to questions, closeness of the subject's answers to model answers, toggling between given answer choices, and response speed to test questions of certain types, subject matters, and/or task performance specification complexity/difficulty/stringency levels, working steps toward a solution, the subject's handwriting, tone of voice, and speech clarity.
  • the subject's excessive toggling between given choices and slow response speed in answering a test question indicating doubts and hesitations on the answer to the question.
  • the subject's intermediate working steps toward completing a task procedural step are captured for matching with the model solution and in turn provides insight to the subject's understanding of the task procedural instruction and task performance specification materials.
  • the system for administering, evaluating, and monitoring a subject's compliance with task performance requirements within an action programme comprises a sensor handling module implemented by a combination of software and firmware executed in general purposed and specially designed computer processors.
  • the sensor handling module manages the various sensors employed by the system.
  • the sensor handling module is in electrical and/or data communications with various electronic sensing devices including, but not limited to, optical and touch sensing devices; input devices including, but not limited to, keyboard, mouse, pointing device, stylus, and electronic pen; image capturing devices; and cameras.
  • input sensory data are continuously collected at various sampling rates and averages of samples of input sensory data are computed.
  • a reference rate is chosen (e.g. 5Hz).
  • a slower sampling rate input sensory data is interpolated with zero order hold and then sampled at the reference rate.
  • a higher sampling rate input sensory data is subsampled at the reference rate.
  • a trace of the last few seconds is kept in memory after which the average is calculated. Effectively this produces a moving average of an input sensory data and acts as a low-pass filter to remove noise.
  • a low-cost optical sensor built-in in a computing device e.g. subject facing camera in a tablet computer
  • images are obtained from the sensor.
  • Each image is then processed by face/eye tracking and analysis systems known in the art.
  • the three- dimensional (3D) head orientation is measured in Euler angles (pitch, yaw, and roll).
  • a 3D vector is assumed from the origin of the optical sensor to the center of the pupil of the user, secondly, a 3D vector is determined from the center of the eye -ball to the pupil.
  • a calibration step helps to compensate for offsets (subject position behind the screen, camera position relative to the screen). Using this data, the planar coordinate of the gaze on the computer screen can be derived.
  • the images and/or videos captured as mentioned above are processed to identify key landmarks on the face such as eyes, tip of the nose, corners of the mouth.
  • the regions between these landmarks are then analyzed and classified into facial expressions such as: attention, brow furrow, brow raise, cheek raise, chin raise, dimpler (lip corners tightened and pulled inwards), eye closure, eye widen, inner brow raise, jaw drop, lid tighten, lip corner depression, lip press, lip pucker (pushed forward), lip stretch, lip such, mouth open, nose wrinkle, smile, smirk, upper lip raise.
  • the system comprises a wearable device to measure physiologic parameters not limiting to: heart rate, electro dermal activity (EDA) and skin temperature.
  • This device is linked wirelessly to the client computing device (e.g. tablet computer or laptop computer).
  • the heart rate is derived from observations of the blood volume pulse.
  • the EDA measures skin conductivity as an indicator for sympathetic nervous system arousal. Based on this, features related to stress, engagement, and excitement can be derived.
  • Another approach is to use vision analysis techniques to directly measure the heart rate based on the captured images. This method is based on small changes in light absorption by the veins in the face, when the amount of blood varies due to the heart rate.
  • test answers may be written on a dedicated note paper using a digital pen and receive commands such as 'task procedural step completed'.
  • the written answer is then digitized on the fly and via an intelligent optical character recognition engine, the system can evaluate the content written by the subject and provide any necessary feedback to guide the subject when needed.
  • OCR OCR after the tasks has been completed. The paper is scanned using a copier and the digitized image is fed to OCR software.
  • the system comprises one or more voice recording devices for recording the subject's speech during a compliance evaluation and monitoring session.
  • the subject's speech is then digitized on the fly and via an intelligent voice recognition engine, the system can evaluate the content spoken by the subject and provide any necessary feedback to guide the subject when needed.
  • the substantive content of the subject's speech is recognized for verbal commands related to a task procedure and/or verbal answers to questionnaire test questions for further compliance analysis.
  • the subject's voice and speech clarity are recognized as input to the affective state and cognitive state estimation.
  • a pedagogical agent may be non- human animated character with human traits implemented by a combination of software and/or firmware running in one or more general purposed computer processors and/or specially configured computer processors. It can display the basic emotions by selecting from a set of animations (e.g. animated GIFs), or by using scripted geometric transformation on a static image displayed to the subject in a user interface. Another method is to use SVG based animations.
  • the animation can be annotated with text messages (e.g. displayed in a balloon next to the animation). The text messages are generated by and received from the trainer module of the system. The subject's responses to the pedagogical agent are received by the system for estimating the subject's affective state.
  • the affective state and cognitive state estimation and performance data are primarily used in gauging the subject's level of compliance with performance specifications of tasks in an action programme. While a single estimation is used in providing a snapshot assessment of the subject's progress toward total compliance in her task performance and prediction of the subject's eventual achievable level of compliance, multiple estimations are used in providing an assessment history and trends of the subject's progress. Furthermore, the estimated affective states and cognitive states of the subject are used in the modeling of the compliance programme in terms of choice of methods of compliance evaluation and monitoring, and instruction delivery and administration.
  • the method and system logically structure the compliance questionnaire and task procedural instruction materials, and the delivery mechanism in a compliance programme as Domain Knowledge 500.
  • a Domain Knowledge 500 comprises one or more Concept objects 501 and one or more Task objects 502.
  • Each Concept object 501 comprises one or more Knowledge and Skill items 503.
  • the Knowledge and Skill items 503 are ordered by task performance specification complexity/difficulty/stringency levels, and two or more Concept objects 501 can be grouped to form a Curriculum.
  • a Curriculum defined by the present invention may be the equivalence of the operation manual/standard and there is one-to-one relationship between a
  • the Concept objects can be linked to form a logical tree data structure (Knowledge Tree) such that Concept objects having Knowledge and Skill items that are fundamental and/or basic in a topic are represented by nodes closer to the root of the logical tree and Concept objects having Knowledge and
  • Skill items that are more advance and branches of some common fundamental and/or basic Knowledge and Skill items are represented by nodes higher up in different branches of the logical tree.
  • Each Task object 502 has various compliance questionnaire and task procedural instruction content materials 504, and is associated with one or more Concept objects 501 in a Curriculum. The associations are recorded and can be looked up in a question matrix 505.
  • a Task object 502 can be classified as: Basic Task, Interactive Task, or Task with an Underlying Cognitive or Expert Model.
  • Each Basic Task comprises one or more operation notes, task procedural instructions (e.g. video clips and other multi-media content), test questions and answers designed to assess whether the subject has read all the materials.
  • Each Interactive Task with an Underlying Cognitive or Expert Model comprises one or more problem-solving exercises each comprises one or more steps designed to guide the subject in deriving the solutions to problems. Each step provides an answer, common misconceptions, and hints. The steps are in the order designed to follow the delivery flow of a task procedure. This allows a tailored scaffolding (e.g. providing guidance and/or hints) for each task based on a point in a task procedure executed.
  • a Task object gathers a set of compliance questionnaire and task procedural instruction materials (e.g. operation notes and illustrations) relevant in the achievement of a compliance level.
  • a Task can be one of the following types:
  • Reading Task operation notes or illustrations to introduce a new topic without grading, required to be completed before proceeding to a Practice Task is allowed;
  • Practice Task a set of questions from one topic to practice on questions from a new topic until a threshold is reached (e.g. five consecutive successful attempts without hints, or achieve an understanding level of 60% or more);
  • Mastery Challenge Task selected questions from multiple topics to let the subject achieves mastery (achieve an understanding level of 95% or more) on a topic, and may include pauses to promote retention of knowledge (e.g. review opportunities for the subjects); or
  • Group Task a set of questions, problem sets, and/or problem-solving exercises designed for peer challenges to facilitate more engagement from multiple subjects in a focus group, maybe ungraded.
  • the Domain Knowledge, its constituent Task objects and Concept objects, Knowledge and Skill items and Curriculums contained in each Concept object, operation notes, illustrations, test questions and answers in each Task object are data entities stored a relational database accessible by the system (a Domain Knowledge repository).
  • a Domain Knowledge repository One or more of Domain Knowledge repositories may reside in third-party systems accessible by the system for administering, evaluating, and monitoring a subject's compliance with task performance requirements within an action programme.
  • the mechanism for delivering and managing interactive and adaptive compliance questionnaires and task procedural instructions logically builds on top of the Domain Knowledge two models of operation: Subject Model and Training Model.
  • Subject Model Under the Subject Model, the system executes each of one or more of the Task objects associated with a Curriculum in a Domain Knowledge for a subject. During the execution of the Task objects, the system measures the subject's performance and obtain the subject's performance metrics in each Task such as: the numbers of successful and unsuccessful attempts to questions in the
  • the performance metrics obtained, along with the information of the Task object, such as its specification complexity/difficulty/stringency level, are fed into a logistic regression mathematical model of each Concept object associated with the Task object. This is also called the knowledge trace of the subject, which is the calculation of the probability of the subject achieving a target compliance level in a task associated with the Concept object.
  • the calculation of a probability of achieving a target compliance level uses a time- based moving average of subject's answer scores to questions in the questionnaires with lesser weight on older attempts, the number of successful attempts, number of failed attempts, success rate (successful attempts over total attempts), time spent, and task performance specification complexity/difficulty/stringency level.
  • the calculation of a probability of achieving a target compliance level uses a time -based moving average of subject's completion of task procedural steps with lesser weight on older attempts, the number of successful attempts, number of failed attempts, success rate (successful attempts over total attempts), time spent, and task performance specification complexity/difficulty/stringency level.
  • the system calculates the probability of the subject achieving a target compliance level in a task associated with the task performance specification in the Concept object using an iterative machine learning workflow to fit mathematical models on to the collected data (subject's performance metrics and information of the Task) including, but not limited to, a time -based moving average of subject's answer scores to questions in the questionnaires with lesser weight on older attempts, the number of successful attempts, number of failed attempts, success rate (successful attempts over total attempts), time spent, topic difficulty, and question difficulty.
  • FIG. 4 depicts a flow diagram of the aforesaid iterative machine learning workflow.
  • data is collected (401), validated and cleansed (402); then the validated and cleansed data is used in attempting to fit a mathematical model (403); the mathematical model is trained iteratively (404) in a loop until the validated and cleansed data fit the mathematical model; then the mathematical model is deployed (405) to obtain the probability of the subject achieving a target compliance level in a task associated with the task performance specification in the Concept object; the fitted mathematical model is also looped back to and used in the step of validating and cleansing of the collected data.
  • the knowledge trace of the subject is used by the system in driving Task compliance questionnaire and task procedural instruction material items selection, driving Task object selection, and driving compliance questionnaire and task procedural instruction material ranking.
  • the advantages of the Subject Model include that the execution of the Task objects can adapt to the changing ability of the subject. For non-limiting example, under the Subject Model the system can estimate the compliance level achievable by the subject, estimate how much performance improvement can be expected for the next Task, and provide a prediction of the subject's level of compliance in a future point of time. These data are then used in the Training Model and enable hypothesis testing to make further improvement to the system, evaluate compliance officer quality and compliance questionnaire and task procedural instruction material quality.
  • the system's trainer module receives the data collected from the execution of the Task objects under the Subject Model and the Domain Knowledge for making decisions on the compliance questionnaire and task procedural instruction delivery strategy and providing feedbacks to the subject and compliance officer.
  • the system for administering, evaluating, and monitoring a subject's compliance with task performance requirements within an action programme comprises a trainer module implemented by a combination of software and firmware executed in general purposed and specially designed computer processors.
  • the trainer module resides in one or more server computers.
  • the trainer module is primarily responsible for executing the machine instructions corresponding to the carrying-out of the activities under the Training Model.
  • the trainer module executes the folio wings:
  • feedback is provided as a function of the current affective state of the subject. For example, this can be an encouraging, empathetic, or challenging message selected from a generic list, or it is a dedicated hint from the Domain Knowledge.
  • the system estimates the probability of the subject achieving a target compliance level in a task associated with the task performance specification materials in each Concept. Based on a predetermined threshold (e.g. 95%), the compliance officer can decide when a Concept is mastered.
  • a predetermined threshold e.g. 95%
  • the system for administering, evaluating, and monitoring a subject's compliance with task performance requirements within an action programme further comprises a communication module implemented by a combination of software and firmware executed in general purposed and specially designed computer processors.
  • a communication module implemented by a combination of software and firmware executed in general purposed and specially designed computer processors.
  • one part of the communication module resides and is executed in one or more server computers
  • other part of the communication module resides and is executed in one or more client computers including, but not limited to, desktop computers, laptop computers, tablet computers, smartphones, and other mobile computing devices, among which some are dedicated for use by the subjects and others by compliance officer.
  • the communication module comprises one or more user interfaces designed to present relevant data from the Domain Knowledge and materials generated by the system operating under the Subject Model and Training Model to the subjects and the compliance officers.
  • the user interfaces are further designed to facilitate user interactions in capturing user input (textual, gesture, image, and video inputs) and displaying feedback including textual hints and the simulated pedagogical agent's actions.
  • Another important feature of the communication module is to provide an on-screen (the screen of the computing device used by a subject) planar coordinates and size of a visual cue or focal point for the current Task object being executed.
  • FIG. 2 depicts a logical data flow diagram of the system for administering, evaluating, and monitoring a subject's compliance with task performance requirements within an action programme in accordance to various embodiments of the present invention.
  • the logical data flow diagram illustrates how the major components of the system work together in a feedback loop in the execution during the Subject Model and Training Model.
  • a suitable series of tasks is selected by the subject in an action programme.
  • This series of tasks corresponds directly to a Curriculum object, which is a set of linked Concept objects in the Domain Knowledge 202, and constitutes the target compliance level 201 for this subject.
  • the system's trainer module selects and retrieves from the Domain Knowledge 202 a suitable Concept object and the associated first Task object. Entering the Subject Model, the Task object data is retrieved from the Domain Knowledge 202 .
  • the system renders the Task object data (e.g. operation notes) on the user interface for the subject, and the subject starts working on the task. Meanwhile, the system manages the compliance evaluation and monitoring process 203 by collecting affective state sensory data including, but not limited to, point-of-gaze, emotion, and physiologic data, and cognition state data via Task questions and answers and the subject's behavioral-analyzing interactions with the user interface (204). After analyzing the collected affective state sensory data and cognition state data, the compliance state 205 is updated. The updated compliance state 205 is compared with the target compliance level 201. The determined knowledge/skill gap or the fit of the task procedural instruction delivery strategy 206 is provided to the Training Model again, completing the loop.
  • affective state sensory data including, but not limited to, point-of-gaze, emotion, and physiologic data
  • cognition state data via Task questions and answers
  • the subject's behavioral-analyzing interactions with the user interface (204) After analyzing the collected affective state sensory data and cognition state data, the compliance state 205 is
  • FIG. 3 depicts an activity diagram illustrating in more details the execution process of the system for administering, evaluating, and monitoring a subject's compliance with task performance requirements within an action programme under the Subject Model and Training Model.
  • the execution process is as follows:
  • a subject logs into the system via her computing device running a user interface rendered by the system's communication module.
  • the subject select a Curriculum presented to her in the user interface.
  • the system's trainer module running in a server computer, selects and requests from the Domain Knowledge repository one or more Task objects associated with the Curriculum selected.
  • the system evaluates the Knowledge Tree and finds the Concept Knowledge and Skills that the subject has not yet learned and/or been evaluated as close to the root (fundamental) of the Knowledge Tree as possible.
  • This process is executed by the system's recommendation engine, which can be implemented by a combination of software and firmware executed in general purposed and specially designed computer processors.
  • the recommendation engine can recommend Practice Tasks, and at lower rate Mastery Challenge Tasks.
  • the system further comprises a recommendation engine for recommending the task performance specification materials (e.g. topic) to be learned next in a Curriculum.
  • the recommendation engine recommends the next Task to be executed by the system under the Training Model.
  • the subject's negative emotion can be eased by recognizing the difficult / unfamiliar topics (from the affective state data estimated during the execution of certain Task) and recommending the next Task of a different / more familiar topic; and recommending the next Task of a difficult / unfamiliar topic when subject's emotion state is detected position.
  • the recommendation engine can select the next Task of higher difficulty when the estimated affective state data shows that the subject is unchallenged. This allows the matching of Tasks with the highest compliance level gains. This allows the clustering of Tasks based on similar performance data and/or affective state and cognitive state estimation. This also allows the matching of subject peers with similar compliance level accomplishment to form focus groups.
  • Task objects are found, their data are retrieved and are sent to the subject's computing device for presentation in the system's communication module user interface.
  • the subject selects a Task object to begin the compliance evaluation and monitoring session.
  • the system's trainer module retrieves from the Domain Knowledge repository the next item in the selected Task object for rendering in the system's communication module user interface.
  • the system's communication module user interface renders the item (compliance questionnaire question and/or task procedure instruction) in the selected Task object.
  • a camera for capturing the subject's face is activated.
  • virtual assistant may be presented in the form of guidance and/or textual hint displayed in the system's communication module user interface.
  • the subject submits an attempt answer and/or an attempt command for completing a task procedural step.
  • the attempt answer and/or attempt command is graded and the grade is displayed to the subject in the system's communication module user interface. 313. The attempt answer and/or attempt command and grade is also stored by the system for further analysis.
  • the attempt answer and/or attempt command and grade are used in calculating the probability of the subject's understanding of the Concept associated with the selected Task object and the probability of the subject achieving a target compliance level in the task.
  • the system's trainer module selects and requests the next Task based on the calculated probability of the subject's understanding of the associated Concept and the probability of the subject achieving a target compliance level in the task, and repeat the steps from step 303.
  • the system's trainer module retrieves the next item in the selected Task and repeat the steps from step 306.
  • the system for administering, evaluating, and monitoring a subject's compliance with task performance requirements within an action programme further comprises an administration module that takes information from the compliance officers, subjects, and Domain Knowledge in offering assistance with the operation of face-to-face compliance evaluation and monitoring process across multiple physical facilities as well as online, remote evaluation and monitoring.
  • the administration module comprises a constraint- based scheduling algorithm that determines the optimal scheduling of compliance evaluation and monitoring sessions while observing constraints such compliance officers' certification, travelling distance for subjects and compliance officers, first-come-first-served, composition of the compliance officers group based on compliance level achievement progress and training strategy.
  • the scheduling algorithm can select subjects with complementary skill sets so that they can help each other and form focus groups.
  • An in-person face-to-face compliance evaluation and monitoring session may comprise a typical flow such as: subjects check in, perform a small task to evaluate the cognitive state of the subjects, and the results are presented on the compliance officer's user interface dashboard directly after completion.
  • the session then continues with explanation of a new task performance specification by the compliance officer, here the compliance officer receives assistance from the system's pedagogical agent with pedagogical goals and hints. After the explanation, the subjects may engage in the new task in which the system provides as much scaffolding as needed.
  • the system's trainer module decides how to continue the compliance evaluation and monitoring session with a few options: e.g. recommend to form focus groups each with subjects who have achieved similar compliance levels in prior tasks completed.
  • the compliance evaluation and monitoring session is concluded by checking out.
  • the attendance data is collected for billing purposes and for compliance certification purposes.
  • the method and system for administering, evaluating, and monitoring a subject's compliance with task performance requirements within an action programme comprise a mechanism for delivering and managing interactive and adaptive compliance questionnaire and task procedural instructions.
  • the mechanism logically structures compliance questionnaire and task procedural instruction materials and the delivery mechanism data in a compliance programme as a Domain Knowledge, with its constituent Concept objects and Task objects having Knowledge and Skill items, and training materials respectively that are relevant to the concerned industry or trade.
  • the system's estimation of the subjects' affective states and cognitive states can be used in driving the selection and presentment of survey questions. This in turn enables more accurate and speedy survey results procurements from the subjects.
  • the system's estimation of the employee subjects' affective states and cognitive states on duty continuously allows an employer to gauge the skill levels, engagement levels, and interests of the employees and in turn provides assistance in work and role assignments.
  • the method and system for method and system for administering, evaluating, and monitoring a subject's compliance with task performance requirements within an action incorporate machine learning techniques that are based on interlinking models of execution comprising: a Domain Model, an Assessment Model, a Learner Model, a Deep Learner Model, one or more Motivational Models, a Transition Model, and a Pedagogical Model.
  • the interlinking models of execution is purposed for driving, inducing, or motivating certain desirable actions, behavior, and/or outcome from the subject.
  • These certain desirable actions and/or outcome can be, as non-limiting examples, learning certain subject matters, achieving certain academic goals, achieving certain career goals, completing certain job assignments, making certain purchases, and conducting certain commercial activities.
  • These interlinking models of execution together form a machine learning feedback loop comprising the continuous tracking and assessment of learning progress of the subject under the Assessment Model, driving the learning activities under the Learner Model, motivating the subject under the Deep Learner Model and Motivation Operational Model, and selecting and re- selecting knowledge space items under the Domain Model and Transition Model, and delivering the knowledge space items and activities from one knowledge state to the next under the Pedagogical Model.
  • the electronic embodiments disclosed herein may be implemented using general purpose or specialized computing devices, computer processors, or electronic circuitries including but not limited to application specific integrated circuits (ASIC), field programmable gate arrays (FPGA), and other programmable logic devices configured or programmed according to the teachings of the present disclosure.
  • ASIC application specific integrated circuits
  • FPGA field programmable gate arrays
  • Computer instructions or software codes running in the general purpose or specialized computing devices, computer processors, or programmable logic devices can readily be prepared by practitioners skilled in the software or electronic art based on the teachings of the present disclosure.
  • All or portions of the electronic embodiments may be executed in one or more general purpose or computing devices including server computers, personal computers, laptop computers, mobile computing devices such as smartphones and tablet computers.
  • the electronic embodiments include computer storage media having computer instructions or software codes stored therein which can be used to program computers or microprocessors to perform any of the processes of the present invention.
  • the storage media can include, but are not limited to, floppy disks, optical discs, Blu-ray Disc, DVD, CD-ROMs, and magneto-optical disks, ROMs, RAMs, flash memory devices, or any type of media or devices suitable for storing instructions, codes, and/or data.
  • Various embodiments of the present invention also may be implemented in distributed computing environments and/or Cloud computing environments, wherein the whole or portions of machine instructions are executed in distributed fashion by one or more processing devices interconnected by a communication network, such as an intranet, Wide Area Network (WAN), Local Area Network (LAN), the Internet, and other forms of data transmission medium.
  • a communication network such as an intranet, Wide Area Network (WAN), Local Area Network (LAN), the Internet, and other forms of data transmission medium.

Landscapes

  • Business, Economics & Management (AREA)
  • Engineering & Computer Science (AREA)
  • Human Resources & Organizations (AREA)
  • Theoretical Computer Science (AREA)
  • Educational Administration (AREA)
  • Tourism & Hospitality (AREA)
  • Economics (AREA)
  • General Physics & Mathematics (AREA)
  • Strategic Management (AREA)
  • Physics & Mathematics (AREA)
  • Entrepreneurship & Innovation (AREA)
  • Educational Technology (AREA)
  • Development Economics (AREA)
  • Marketing (AREA)
  • General Business, Economics & Management (AREA)
  • Game Theory and Decision Science (AREA)
  • Operations Research (AREA)
  • Quality & Reliability (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Primary Health Care (AREA)
  • Electrically Operated Instructional Devices (AREA)
  • Measurement Of The Respiration, Hearing Ability, Form, And Blood Characteristics Of Living Organisms (AREA)
  • Eye Examination Apparatus (AREA)

Abstract

La présente invention concerne un système de gestion, d'évaluation et de contrôle de la conformité d'un sujet à des exigences se rapportant à l'exécution de tâches dans le cadre d'un programme d'actions, comprenant des capteurs optiques destinés à capturer l'expression faciale, les mouvements des yeux, le point de regard et la position de tête de sujet d'un sujet lors d'une séance d'évaluation et de contrôle de conformité ; un référentiel de données de connaissances de domaine (500) comprenant des entités de données de concept (501), comportant chacune des éléments de contenu de connaissances et de compétences (503), et des entités de données de tâche (502), comportant chacune des éléments matériels de contenu de conférence (504) ; un module de sujet configuré pour estimer l'état affectif et l'état cognitif du sujet au moyen de données sensorielles collectées à partir des capteurs optiques ; et un module de formateur configuré pour sélectionner une entité de données de tâche en vue de leur distribution et de leur présentation au sujet après chaque achèvement d'une entité de données de tâche, sur la base d'une probabilité de compréhension par le sujet des éléments de contenu de connaissances et de compétences de l'entité de données de concept associée et d'une probabilité que le sujet atteigne un niveau de conformité cible.
PCT/IB2018/053968 2017-06-15 2018-06-04 Procédé et système d'évaluation et de contrôle de conformité au moyen de la détection d'émotion WO2018229592A1 (fr)

Priority Applications (4)

Application Number Priority Date Filing Date Title
JP2019600105U JP3223411U (ja) 2017-06-15 2018-06-04 感情検出を用いて追随性を評価及び監視するためのシステム
CN201880003286.5A CN109716382A (zh) 2017-06-15 2018-06-04 使用情绪检测评估和监测遵从性的方法和系统
US16/313,895 US11475788B2 (en) 2017-06-15 2018-06-04 Method and system for evaluating and monitoring compliance using emotion detection
US17/960,835 US20230105077A1 (en) 2017-06-15 2022-10-06 Method and system for evaluating and monitoring compliance, interactive and adaptive learning, and neurocognitive disorder diagnosis using pupillary response, face tracking emotion detection

Applications Claiming Priority (12)

Application Number Priority Date Filing Date Title
US201762520542P 2017-06-15 2017-06-15
US62/520,542 2017-06-15
US15/647,272 US20180232567A1 (en) 2017-02-14 2017-07-12 Interactive and adaptive training and learning management system using face tracking and emotion detection with associated methods
US15/647,272 2017-07-12
US201762594557P 2017-12-05 2017-12-05
US62/594,557 2017-12-05
US201862622888P 2018-01-27 2018-01-27
US62/622,888 2018-01-27
US201862627734P 2018-02-07 2018-02-07
US62/627,734 2018-02-07
US201862646365P 2018-03-21 2018-03-21
US62/646,365 2018-03-21

Related Parent Applications (1)

Application Number Title Priority Date Filing Date
US15/647,272 Continuation-In-Part US20180232567A1 (en) 2017-02-14 2017-07-12 Interactive and adaptive training and learning management system using face tracking and emotion detection with associated methods

Related Child Applications (2)

Application Number Title Priority Date Filing Date
US16/313,895 A-371-Of-International US11475788B2 (en) 2017-06-15 2018-06-04 Method and system for evaluating and monitoring compliance using emotion detection
US17/960,835 Continuation-In-Part US20230105077A1 (en) 2017-06-15 2022-10-06 Method and system for evaluating and monitoring compliance, interactive and adaptive learning, and neurocognitive disorder diagnosis using pupillary response, face tracking emotion detection

Publications (1)

Publication Number Publication Date
WO2018229592A1 true WO2018229592A1 (fr) 2018-12-20

Family

ID=64660048

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/IB2018/053968 WO2018229592A1 (fr) 2017-06-15 2018-06-04 Procédé et système d'évaluation et de contrôle de conformité au moyen de la détection d'émotion

Country Status (2)

Country Link
HK (1) HK1248064A2 (fr)
WO (1) WO2018229592A1 (fr)

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2001337590A (ja) * 2000-05-25 2001-12-07 Morigo Corp 通信教育システム，通信教育用管理サーバ，通信教育用コンピュータ端末システム
JP2002175307A (ja) * 2000-09-04 2002-06-21 Ntt Learning Systems Kk 電子解説方法
KR20100011280A (ko) * 2008-07-24 2010-02-03 포항공과대학교 산학협력단 표정 증폭을 이용한 미세 표정인식 방법 및 장치
KR20150002181A (ko) * 2013-06-28 2015-01-07 재단법인대구경북과학기술원 Ｍｄｍ을 이용한 등교 관리 시스템 및 그 동작 방법
KR20160030415A (ko) * 2014-09-10 2016-03-18 (주)지디아이코리아 육성 시뮬레이션을 통한 교육 관리 시스템 및 방법
CN106228293A (zh) * 2016-07-18 2016-12-14 重庆中科云丛科技有限公司 教学评估方法及系统
CN106919922A (zh) * 2017-03-07 2017-07-04 佛山市金蓝领教育科技有限公司 一种防作弊远程考试系统

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2001337590A (ja) * 2000-05-25 2001-12-07 Morigo Corp 通信教育システム，通信教育用管理サーバ，通信教育用コンピュータ端末システム
JP2002175307A (ja) * 2000-09-04 2002-06-21 Ntt Learning Systems Kk 電子解説方法
KR20100011280A (ko) * 2008-07-24 2010-02-03 포항공과대학교 산학협력단 표정 증폭을 이용한 미세 표정인식 방법 및 장치
KR20150002181A (ko) * 2013-06-28 2015-01-07 재단법인대구경북과학기술원 Ｍｄｍ을 이용한 등교 관리 시스템 및 그 동작 방법
KR20160030415A (ko) * 2014-09-10 2016-03-18 (주)지디아이코리아 육성 시뮬레이션을 통한 교육 관리 시스템 및 방법
CN106228293A (zh) * 2016-07-18 2016-12-14 重庆中科云丛科技有限公司 教学评估方法及系统
CN106919922A (zh) * 2017-03-07 2017-07-04 佛山市金蓝领教育科技有限公司 一种防作弊远程考试系统

Also Published As

Publication number Publication date
HK1248064A2 (zh) 2018-10-05

Similar Documents

Publication Publication Date Title
US20180232567A1 (en) Interactive and adaptive training and learning management system using face tracking and emotion detection with associated methods
US20200178876A1 (en) Interactive and adaptive learning, neurocognitive disorder diagnosis, and noncompliance detection systems using pupillary response and face tracking and emotion detection with associated methods
US11475788B2 (en) Method and system for evaluating and monitoring compliance using emotion detection
Sharma et al. Multimodal data capabilities for learning: What can multimodal data tell us about learning?
US10643487B2 (en) Communication and skills training using interactive virtual humans
JP3223411U (ja) 感情検出を用いて追随性を評価及び監視するためのシステム
Hasnine et al. Students’ emotion extraction and visualization for engagement detection in online learning
WO2019180652A1 (fr) Systèmes d'apprentissage interactifs, adaptatifs et motivationnels au moyen d'un suivi de visage et d'une détection d'émotion, et procédés associés
KR20210019266A (ko) 시선 추적을 이용한 머신 러닝 기반의 읽기 능력 진단 장치 및 방법
Sinha Enriching problem-solving followed by instruction with explanatory accounts of emotions
Chiu et al. A bayesian classification network-based learning status management system in an intelligent classroom
Buono et al. Assessing student engagement from facial behavior in on-line learning
US20230105077A1 (en) Method and system for evaluating and monitoring compliance, interactive and adaptive learning, and neurocognitive disorder diagnosis using pupillary response, face tracking emotion detection
Singh et al. Do I Have Your Attention: A Large Scale Engagement Prediction Dataset and Baselines
Sajja et al. Integrating AI and Learning Analytics for Data-Driven Pedagogical Decisions and Personalized Interventions in Education
US20230121592A1 (en) Using biometric data intelligence for education management
US20230127335A1 (en) Intelligent and adaptive measurement system for remote education
Utami et al. A Brief Study of The Use of Pattern Recognition in Online Learning: Recommendation for Assessing Teaching Skills Automatically Online Based
WO2018229592A1 (fr) Procédé et système d'évaluation et de contrôle de conformité au moyen de la détection d'émotion
Nakamura et al. Estimating learners’ subjective impressions of the difficulty of course materials in e-learning environments
Rajendran Enriching the student model in an intelligent tutoring system
Kai Modeling student affective state patterns during self-regulated learning in physics playground
Ting et al. Usability Evaluation of Facial Emotion Recognition System
Perera Multiparameter-based evaluation to identify the trustworthiness of the examinee during e-assessments
JP2023003516A (ja) 集中度判別プログラム

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18817464

Country of ref document: EP

Kind code of ref document: A1

WPC Withdrawal of priority claims after completion of the technical preparations for international publication

Ref document number: 62/622,888

Country of ref document: US

Date of ref document: 20190212

Free format text: WITHDRAWN AFTER TECHNICAL PREPARATION FINISHED

Ref document number: 62/627,734

Country of ref document: US

Date of ref document: 20190212

Free format text: WITHDRAWN AFTER TECHNICAL PREPARATION FINISHED

ENP Entry into the national phase

Ref document number: 2019600105

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2018817464

Country of ref document: EP

Effective date: 20200115

122 Ep: pct application non-entry in european phase

Ref document number: 18817464

Country of ref document: EP

Kind code of ref document: A1