Connect public, paid and private patent data with Google Patents Public Datasets

Scenario workflow based assessment system and method

Download PDF

Info

Publication number
US20050282141A1
US20050282141A1 US10870879 US87087904A US2005282141A1 US 20050282141 A1 US20050282141 A1 US 20050282141A1 US 10870879 US10870879 US 10870879 US 87087904 A US87087904 A US 87087904A US 2005282141 A1 US2005282141 A1 US 2005282141A1
Authority
US
Grant status
Application
Patent type
Prior art keywords
assessment
event
scenario
criteria
workflow
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Abandoned
Application number
US10870879
Inventor
Mark Falash
Kenneth Donovan
Peter Bilazarian
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Lockheed Martin Corp
Original Assignee
Lockheed Martin Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date

Links

Images

Classifications

    • GPHYSICS
    • G09EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09BEDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
    • G09B19/00Teaching not covered by other main groups of this subclass
    • GPHYSICS
    • G09EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09BEDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
    • G09B7/00Electrically-operated teaching apparatus or devices working with questions and answers

Abstract

A system and a method for automating performance assessments of an exercise or a training activity provide event assessment information in real-time to one or more evaluators in conjunction with unfolding events. The evaluators can wirelessly communicate assessment information to a database for after action review (AAR).

Description

    FIELD OF THE INVENTION
  • [0001]
    The invention pertains to systems and methods for assessing performance of participants during training exercises, or, rehearsing of missions. More particularly, the invention pertains to automated systems and methods to facilitate performance evaluation by providing real-time feedback to evaluators as an activity proceeds.
  • BACKGROUND OF THE INVENTION
  • [0002]
    The importance of training personnel to respond to events such as fires, violent domestic events, accidents or natural disasters (earthquakes, tornadoes, floods or the like) is well recognized. Similar comments apply to military training/mission rehearsal.
  • [0003]
    Training/rehearsal activities can last hours or days and can involve a large number of geographically dispersed participants. The value of collecting information as to how the exercise was carried out to facilitate an accurate and meaningful after-action review is also well known. One such system and method are disclosed in U.S. Pat. No. 6,106,297 issued Aug. 22, 2000, assigned to the assignee hereof and entitled “Distributed Interactive Simulation Exercise Manager System and Method”. The '297 patent is hereby incorporated by reference.
  • [0004]
    While the primary value of conducting a performance session, such as a training or exercise session, is an effective and accurate assessment, (the basis of measurable and verifiable feedback to the session audience or participant, the after-action review (AAR)) obtaining such assessments during such sessions can be difficult. A problem in efficiently assessing, or evaluating, performance during complex tasks is defining what is important to be assessed at any given time, and what assessment criteria should be used.
  • [0005]
    It has been known in prior art to define assessment criteria and guidance prior to the assessment session. The assessor is then required to monitor performance activities to determine what type of events are taking place, recall and apply the applicable assessment criteria and assessment guidance, and record the applicable assessment. This approach is labor intensive, particularly for complex tasks involving teams of several individuals, and teams in different locations.
  • [0006]
    There continues to be a need for improved, preferably real-time evaluation systems and methods. Preferably such systems and methods will be flexible and cost effective to implement so as to be usable to provide assessment information for a wide range of civilian and military exercises.
  • SUMMARY OF THE INVENTION
  • [0007]
    A method which embodies the invention includes defining a scenario workflow relative to a selected situation, establishing a set of assessment criteria relative to a plurality of workflow related events, carrying out the scenario; as the scenario proceeds, retrieving assessment criteria for at least one active event; and providing the returned assessment criteria to an evaluator.
  • BRIEF DESCRIPTION OF THE DRAWINGS
  • [0008]
    FIG. 1 is an over-view of a method in accordance with the invention;
  • [0009]
    FIG. 2 illustrates a system in accordance with the invention;
  • [0010]
    FIG. 3 illustrates an exemplary scenario event;
  • [0011]
    FIG. 4 illustrates an exemplary workflow event list for the event of FIG. 3;
  • [0012]
    FIG. 5 illustrates an exemplary event assessment criteria list;
  • [0013]
    FIG. 6 illustrates a displayed event list for activating an event;
  • [0014]
    FIG. 7A illustrates an exemplary assessment entering screen; and
  • [0015]
    FIG. 7B illustrates an exemplary assessment prompting display for an active event.
  • DETAILED DESCRIPTION OF THE EMBODIMENTS
  • [0016]
    While embodiments of this invention can take many different forms, specific embodiments thereof are shown in the drawings and will be described herein in detail with the understanding that the present disclosure is to be considered as an exemplification of the principles of the invention, and as a disclosure of the best mode of practicing the invention. It is not intended to limit the invention to the specific embodiment illustrated.
  • [0017]
    Systems and methods that embody the invention improve the efficiency of assessing performance during complex tasks, such as for distributed teams cooperating to achieve a common goal. Assessment accuracy is increased while reducing the work associated with recording behavior observations, preparing material for briefings and debriefings, and presenting feedback messages or comments during and after the assessment session.
  • [0018]
    Efficiency is increased because 1) the assessor is prompted during the assessment session with the applicable assessment criteria and guidance. An assessor can more quickly determine what assessment is needed; 2) the assessor is prompted with a tailored form for recording the assessment, so the assessor can record the assessment more quickly and accurately; 3) assessment responsibilities can be efficiently distributed and allocated across several assessors during the assessment session, so fewer assessors are needed; 4) expert subjective judgment guidance that is applicable to the assessment event can be captured from experts prior to the assessment session, and communicated to all assessors as required during the assessment session, so the assessors are not required to be experts.
  • [0019]
    Further, a method that embodies the invention automates performance assessment of an activity such as a training or a rehearsal exercise (the assessment session). The automation results in measurable or observable assessments of session events. In one embodiment, the method includes:
  • [0020]
    1) generating a scenario workflow which defines an event list or events that are expected to occur during an assessment session;
  • [0021]
    2) generating a workflow event assessment list which defines assessment criteria or guidance to be used during a respective event;
  • [0022]
    3) defining active events from the event list;
  • [0023]
    4) prompting the assessor regarding the applicable assessment criteria relative to a currently active event; and
  • [0024]
    5) recording the assessors' observations.
  • [0025]
    Alternative embodiments include:
  • [0026]
    a) Predefining the scenario workflow (e.g., a predefined script that always occurs in the same sequence).
  • [0027]
    b) The scenario workflow which defines an event list can be dynamically adjusted in response to the actions taken by the participants during the assessment session (e.g., a portion of the scenario workflow only occurs if the participants behave in a certain way referred to as a triggering behavior).
  • [0028]
    c) The scenario workflow which defines an event list can be dynamically adjusted by the evaluator based on accomplishment of assessment objectives during the assessment session (e.g., the evaluator may add scenario workflow to increase the workload during an assessment session—referred to as an “inject”).
  • [0029]
    d) The workflow event list criteria can be predefined (e.g., the criteria to be used for an event is always the same whenever that event occurs in the scenario workflow).
  • [0030]
    e) The workflow event list assessment criteria can be dynamically adjusted in response to the actions taken by the participants during the assessment session (e.g., if the participants or audience respond to a scenario situation).
  • [0031]
    f) The workflow event list assessment criteria can be dynamically adjusted by the evaluator based on accomplishment of assessment objectives during the assessment session (e.g., if the evaluator may determine that a report preparer has demonstrated mastery of the primary criteria of submitting a properly formatted report in a timely fashion. The assessment criteria for future reporting could be adjusted to record whether the report preparer consults all relevant information sources needed for a quality report).
  • [0032]
    g) Defining what event from the event list are currently applicable can be based on human observation.
  • [0033]
    h) Defining what events from the event list are currently applicable can be based on observation by a computer-based agent.
  • [0034]
    i) Prompting the assessor regarding the applicable assessment criteria can be predefined (e.g., the assessor is always provided the same cue or input screen each time the applicable event occurs).
  • [0035]
    j) Prompting the assessor regarding the applicable assessment criteria can be dynamically adjusted in response to the actions taken by the audience during the assessment session (e.g., if the participants are frequently performing one type of task (Task A) and infrequently performs another type of task (Task B), then the prompting means
  • [0036]
    k) Prompting the assessor regarding the applicable assessment criteria can be dynamically adjusted by the evaluator based on accomplishment of assessment objectives during the assessment session (e.g., if the evaluator determines that a report preparer has demonstrated mastery of the primary criteria of submitting a properly formatted report in a timely fashion. The assessment criteria for future reporting could be adjusted to record whether the report preparer consults all relevant information sources needed for a quality report).
  • [0037]
    In one system, that embodies this invention, the scenario workflow is defined to detail sequences or series of events that would result for a particular situation or course of action. The scenario workflow is then stored in a scenario workflow event database. The scenario workflow is analyzed to define what assessment criteria, if any, are applicable for each of the workflow events. These assessment criteria are added to the scenario workflow event database.
  • [0038]
    During the assessment session, the assessor or evaluator is provided an automated assessment device, or AAD. The AAD facilitates the use of hand-carried Tablet PCs by the evaluation team to coordinate evaluation responsibilities, display the pre-brief material and exercise status at any time during the exercise. As the scenario session proceeds, the scenario is monitored to determine what events are currently active.
  • [0039]
    By accessing the workflow event database, the system retrieves the details for the scenario event, including the assessment criteria list for that event. The evaluator is prompted with context-sensitive assessment criteria as key and critical events occur. As the evaluator records observations against each exercise goal, AAD provides immediate feedback of exercise goals that require additional attention. This significantly reduces the need for the evaluator to closely monitor actual scenario event execution while recording key observations. The AAD provides assistance to the evaluator to prepare the After Action Review (AAR).
  • [0040]
    Following the exercise, the evaluation team can quickly review evaluator comments grouped by exercise objectives and select candidates for discussion. During the review, the evaluator can use the AAD to identify and replay critical periods of audience actions that temporally relate to the evaluator's notes, while referencing the guidance provided during the exercise planning process. This automation and recall capability are not available using current manual data collection and synthesis techniques. The resultant After Action Reports provide invaluable lessons for the exercise participants, exercise coordinators and stakeholder agencies, including best practices that can be shared, thus, enhancing emergency preparedness.
  • [0041]
    FIG. 1 illustrates an overall view of a method 100 in accordance with the invention. In a step 102 a a scenario workflow is generated. In a step 102 b, a workflow event list is produced from the scenario workflow. The event list produced in step 102 b defines those events which are expected to occur during a scenario assessment session.
  • [0042]
    In step 104 a, event assessment criteria are assigned. In step 104 b an event assessment criteria list is generated. Those of skill will understand that the workflow event list step 102 b and event assessment criteria list step 104 b could be stored as known to those of skill in the art for subsequent use.
  • [0043]
    In step 106 a the scenario is initiated and the participants or audience participate in the ongoing scenario. Scenario workflow events unfold, step 106 b and the assessor 110 is available to observe the participants' or audience's performance in response to the events 106 b.
  • [0044]
    As the eventss unfold, step 106 b, active events are recognized, step 108 either automatically or by personnel associated with implementing the session.
  • [0045]
    Active events trigger a retrieval of evaluation or assessment information, step 116. This information is forwarded to the assessor 110, step 118.
  • [0046]
    The assessors' remarks, comments and evaluations are received step 120 and stored for subsequent use, step 122 during after action review. Step 118 can be repeated as appropriate to provide supplemental assessment information to assessor 110 in view of previously recorded assessments step 122.
  • [0047]
    FIG. 2 is a block diagram of a system 10 in accordance with the invention. System 10 represents one embodiment for implementing the methodology 100 of FIG. 1. As illustrated in FIG. 1, the training audience or participants 12 can communicate in a system 10 using, for example, personal computers which communicate via an intranet 12 a and server 12 b with the Internet 14. It will be understood that the participants 12 might participate in the subject scenario as multiple separate teams or as one organization working together to respond to the scenario. The participants or audience 12 can receive information via their personal computers and intranet 12 a, such as by e-mail, telephone or any other form of information providing systems, such as video, crisis information management system (CIMS) or the like.
  • [0048]
    The evaluators 110 can wirelessly communicate with a server associated with web 14 using wireless personal communication devices indicated generally at 20. This wireless communication capability makes it possible for the evaluators to readily move among the audience or participants during the exercise.
  • [0049]
    A plurality of web based servers 24 provides useful information for the exercise. An emergency scene simulator system 26 provides a realistic representation of the subject scenario which requires the attention and action of the training audience or participants 12. The e-mail, phone, crisis information management system 28 provides realistic communication and coordination capabilities for the audience/participants 12. The scenario log/replay system 30 records audience communications, scenario events, and emergency scene related simulation activity. System 32 stores and makes available scenario events, procedures, standards, including the workflow event list, the event assessment criteria lists and related assessment information.
  • [0050]
    Role players 34 interact with the ongoing scenario and participants in the session via one or more web based servers. They can communicate with the audience or participants and/or control the sequence of events, the emergency scene simulator 26 and any other event related information.
  • [0051]
    As noted above, the scenario workflow is defined to detail sequences or series of events that would result for a particular situation or course of action, step 102 a. A sample scenario is illustrated in FIG. 3 (a single event is shown for simplicity). The scenario events can then be stored in a database, such as database 32. It will be understood that the scenario events might be stored at a plurality of locations without departing from the spirit and scope of the invention.
  • [0052]
    For any given scenario event, there is a workflow procedure or event sequence, the workflow event list step 102 b, that the audience is expected to follow in response to the occurrence of the scenario events. The expected workflow event list 102 b for the sample scenario event in FIG. 3 is illustrated in FIG. 4. The scenario workflow event list 102 b can then be stored in a database such as database 32, FIG. 2.
  • [0053]
    The scenario workflow is analyzed to define what assessment criteria, if any, are applicable for each of the workflow events step 104 a. The assessment criteria list, step 104 b, is created and stored in a database such as database 32. These criteria include a description of the objective behavior that is expected of the audience, an event priority used to designate relative priority of events, an Evaluator Observation column to cue the assessor as to what should be observed, and a Performance Outcome note to define the criteria for successful performance by the participants or audience. These criteria which can be added to the event list 102 b from FIG. 4, are illustrated in FIG. 5, as the assessment criteria list step 104 b. The assessment criteria list can be stored in a database such as 32.
  • [0054]
    Event determining software can be used to define what events from the event list are currently active or applicable, step 108. As the scenario session proceeds, the scenario is monitored to determine what events are currently active. This can be accomplished by monitoring the scenario activity and looking for a match of the expected “Trigger” condition for those scenario events that are expected but have not yet occurred (status=“Pending”).
  • [0055]
    In one embodiment, active event identification can be accomplished by a human monitoring the event status see screen of FIG. 6, including all e-mail communication, as well as all voice between the role players and the training audience. When the monitor determines that an event had started, the monitor can designate that even as active (status =“Active”), using a PC-based workstation. This status of the events that are currently applicable would be recorded, for example in database 32, and transmitted over the network to all role players 34 and evaluators 110.
  • [0056]
    By assessing the workflow event database, the details for the active scenario event, including the assessment criteria list step 104 b for that event can be retrieved, for example from database 32. The evaluator 110 is prompted with context-sensitive assessment criteria as key and critical events occur via an automated assessment device, or AAD 20.
  • [0057]
    The AAD 20, for example, hand-carried Tablet-PCs, enables the evaluation team 110 to coordinate evaluation responsibilities, display lists on other exercise material and exercise status at any time during the exercise. FIGS. 7A and 7B are sample displays as might be presented to the evaluation team 110.
  • [0058]
    FIG. 7A illustrates an exemplary normal AAD display where the scenario event monitor is shown in the bottom half of the display. The evaluator records assessments in the top half of the display. As the evaluator records observations against each exercise goal, the AAD provides immediate feedback of exercise goals that require additional attention step 118. This significantly reduces the need for the evaluator to closely monitor actual scenario event execution while recording key observations.
  • [0059]
    FIG. 7B illustrates AAD operation when a critical event has become active. The evaluator is cued to this activity by the “alert” button, lower left comer. Upon selecting the alert button, an Alert Display that is appropriate for assessing the current active event is displayed. This event-appropriate cue enables extremely efficient assessment..
  • [0060]
    It will be understood that none of the details pertaining to communication via intranet 12 a or Internet 14 are limitations of the invention. Similarly, those of skill in the art will recognize that none of the size or location of the participants or audience 12, evaluator or assessment team(s) 110 or role players 34 are limitations of the invention.
  • [0061]
    From the foregoing, it will be observed that numerous variations and modifications may be effected without departing from the spirit and scope of the invention. It is to be understood that no limitation with respect to the specific apparatus illustrated herein is intended or should be inferred. It is, of course, intended to cover by the appended claims all such modifications as fall within the scope of the claims.

Claims (29)

1. A method comprising:
defining a scenario workflow relative to a selected situation;
storing the workflow;
establishing a set of assessment criteria relative to a plurality of workflow related events;
retrieving assessment criteria for at least one active event as the scenario proceeds; and
providing the retrieved assessment criteria to an evaluator.
2. A method as in claim 1 where the assessment criteria are provided to the evaluator at or about the time the active event occurs.
3. A method as in claim 2 where the criteria are communicated, at least in part, wirelessly.
4. A method as in claim 2 which includes storing real-time evaluation information relative to the active event.
5. A method as in claim 4 which includes providing supplemental information relative to evaluating the event, in response to previously stored observations.
6. A method as in claim 5 where the criteria are communicated, at least in part, wirelessly.
7. A method as in claim 5 where the supplemental information is provided in real-time as the event is taking place.
8. A method as in claim 7 which includes storing evaluation information during the event grouped at least by predetermined objectives.
9. A method as in claim 8 which includes replaying stored information relative to the event, in combination with temporally related, pre-stored, evaluation information.
10. A method as in claim 9 which includes preparing reports pertaining to scenario implementation.
11. A method as in claim 7 where the supplemental information is provided, in part, wirelessly to at least one evaluator.
12. A method as in claim 7 which includes carrying out a plurality of events related to the scenario workflow.
13. A method as in claim 12 which includes wirelessly receiving assessment criteria related to a plurality of workflow related events.
14. A method as in claim 13 which includes providing wireless receivers to the evaluators.
15. A system comprising:
a stored workflow event list;
a stored event assessment criteria list;
circuitry to provide information to an evaluator for assessing a plurality of events; and
circuitry for providing supplemental information to the evaluator.
16. A system as in claim 15 which includes circuitry to provide assessment criteria to a plurality of evaluators.
17. A system as in claim 16 which includes storage for a plurality of event assessments.
18. A system as in claim 17 which includes a plurality of evaluator units which provide context-sensitive assessment criteria.
19. A system as in claim 18 where at least some of the units communicate wirelessly.
20. A system as in claim 19 where the units include software enabling an evaluator to enter exercise related observations.
21. A system as in claim 20 where the units include software to receive and present feedback information to the evaluator relative to the exercise.
22. A system for evaluating performance during a scenario comprising:
first software for creating a plurality of scenario workflow events;
second software for creating assessment criteria for each of the workflow events;
software for determining at least one active event of an on-going scenario;
software for retrieving assessment criteria for the active event;
software for receiving event related assessment information from the evaluator; and
software providing for supplemental assessment information to an evaluator.
23. A system as in claim 22 where the first software creates and stores at least one workflow event list.
24. A system as in claim 23 where the second software creates and stores a workflow event assessment criteria list.
25. A system as in claim 22 which includes software for displaying assessment criteria for the active event.
26. A system as in claim 22 which includes software for displaying supplemental information concerning an on-going event to the evaluator.
27. A system as in claim 22 which includes wireless I/O devices for entering assessment information.
28. A system as in claim 19 which includes software for storing assessment information.
29. A system as in claim 28 for retrieving stored assessment information.
US10870879 2004-06-17 2004-06-17 Scenario workflow based assessment system and method Abandoned US20050282141A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US10870879 US20050282141A1 (en) 2004-06-17 2004-06-17 Scenario workflow based assessment system and method

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
US10870879 US20050282141A1 (en) 2004-06-17 2004-06-17 Scenario workflow based assessment system and method
GB0512298A GB2415823B (en) 2004-06-17 2005-06-16 Scenario workflow based assessment system and method
US11867328 US7991729B2 (en) 2004-06-17 2007-10-04 Scenario workflow based assessment system and method

Publications (1)

Publication Number Publication Date
US20050282141A1 true true US20050282141A1 (en) 2005-12-22

Family

ID=34862192

Family Applications (2)

Application Number Title Priority Date Filing Date
US10870879 Abandoned US20050282141A1 (en) 2004-06-17 2004-06-17 Scenario workflow based assessment system and method
US11867328 Active 2026-03-16 US7991729B2 (en) 2004-06-17 2007-10-04 Scenario workflow based assessment system and method

Family Applications After (1)

Application Number Title Priority Date Filing Date
US11867328 Active 2026-03-16 US7991729B2 (en) 2004-06-17 2007-10-04 Scenario workflow based assessment system and method

Country Status (2)

Country Link
US (2) US20050282141A1 (en)
GB (1) GB2415823B (en)

Cited By (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20070048710A1 (en) * 2005-08-09 2007-03-01 The University Of North Dakota Bioterrorism and disaster response system
US20070174093A1 (en) * 2005-09-14 2007-07-26 Dave Colwell Method and system for secure and protected electronic patient tracking
US20070260436A1 (en) * 2006-04-27 2007-11-08 Lockheed Martin Integrated Systems And Solutions System and method for evaluating system architectures
US20070297589A1 (en) * 2005-09-14 2007-12-27 Greischar Patrick J Method and system for data aggregation for real-time emergency resource management
US20080046285A1 (en) * 2006-08-18 2008-02-21 Greischar Patrick J Method and system for real-time emergency resource management
US20090099983A1 (en) * 2006-05-19 2009-04-16 Drane Associates, L.P. System and method for authoring and learning
US8577718B2 (en) 2010-11-04 2013-11-05 Dw Associates, Llc Methods and systems for identifying, quantifying, analyzing, and optimizing the level of engagement of components within a defined ecosystem or context
US20140315159A1 (en) * 2013-01-29 2014-10-23 Wells Fargo Bank, N.A. Banking Services Experience Center
US8952796B1 (en) 2011-06-28 2015-02-10 Dw Associates, Llc Enactive perception device
US8996359B2 (en) 2011-05-18 2015-03-31 Dw Associates, Llc Taxonomy and application of language analysis and processing
US9020807B2 (en) 2012-01-18 2015-04-28 Dw Associates, Llc Format for displaying text analytics results
US20160034834A1 (en) * 2014-08-01 2016-02-04 Ncino, Inc. Capturing evolution of a resource memorandum according to resource requests
US9269353B1 (en) 2011-12-07 2016-02-23 Manu Rehani Methods and systems for measuring semantics in communications
US9667513B1 (en) 2012-01-24 2017-05-30 Dw Associates, Llc Real-time autonomous organization

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130304792A1 (en) * 2012-05-11 2013-11-14 Outcome Logic, Inc. System and method for assessing, managing and recovering from emergencies
US8954579B2 (en) * 2012-08-21 2015-02-10 Microsoft Corporation Transaction-level health monitoring of online services

Citations (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5597312A (en) * 1994-05-04 1997-01-28 U S West Technologies, Inc. Intelligent tutoring method and system
US5815152A (en) * 1995-04-18 1998-09-29 Logical Software Solutions Corporation Method and apparatus for defining and evaluating a graphic rule
US5863208A (en) * 1996-07-02 1999-01-26 Ho; Chi Fai Learning system and method based on review
US6053737A (en) * 1997-11-04 2000-04-25 Northrop Grumman Corporation Intelligent flight tutoring system
US6101101A (en) * 1998-05-28 2000-08-08 Sampo Semiconductor Corporation Universal leadframe for semiconductor devices
US6106297A (en) * 1996-11-12 2000-08-22 Lockheed Martin Corporation Distributed interactive simulation exercise manager system and method
US6161101A (en) * 1994-12-08 2000-12-12 Tech-Metrics International, Inc. Computer-aided methods and apparatus for assessing an organization process or system
US6428323B1 (en) * 1999-08-30 2002-08-06 Carla M. Pugh Medical examination teaching system
US20030091968A1 (en) * 1996-05-08 2003-05-15 Gaumard Scientific, Inc. Interactive education system for teaching patient care
US6652283B1 (en) * 1999-12-30 2003-11-25 Cerego, Llc System apparatus and method for maximizing effectiveness and efficiency of learning retaining and retrieving knowledge and skills
US20040249678A1 (en) * 2003-06-03 2004-12-09 Henderson E. Devere Systems and methods for qualifying expected risk due to contingent destructive human activities

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5658161A (en) 1994-11-14 1997-08-19 School Dis. #1 In The City & County Of Denver, State Of Colorado Creative assessment method
US6101489A (en) 1998-12-22 2000-08-08 Ac Properties, B.V. System, method and article of manufacture for a goal based system utilizing a time based model
US6712638B2 (en) * 2002-02-26 2004-03-30 Palmone, Inc. Communication device for a handheld computer
WO2005036500A1 (en) * 2003-09-16 2005-04-21 Leonhardi Rudo Von Driver training system and method, and use thereof
US20090035736A1 (en) * 2004-01-16 2009-02-05 Harold Wolpert Real-time training simulation system and method

Patent Citations (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5597312A (en) * 1994-05-04 1997-01-28 U S West Technologies, Inc. Intelligent tutoring method and system
US6161101A (en) * 1994-12-08 2000-12-12 Tech-Metrics International, Inc. Computer-aided methods and apparatus for assessing an organization process or system
US5815152A (en) * 1995-04-18 1998-09-29 Logical Software Solutions Corporation Method and apparatus for defining and evaluating a graphic rule
US20030091968A1 (en) * 1996-05-08 2003-05-15 Gaumard Scientific, Inc. Interactive education system for teaching patient care
US5863208A (en) * 1996-07-02 1999-01-26 Ho; Chi Fai Learning system and method based on review
US6106297A (en) * 1996-11-12 2000-08-22 Lockheed Martin Corporation Distributed interactive simulation exercise manager system and method
US6053737A (en) * 1997-11-04 2000-04-25 Northrop Grumman Corporation Intelligent flight tutoring system
US6101101A (en) * 1998-05-28 2000-08-08 Sampo Semiconductor Corporation Universal leadframe for semiconductor devices
US6428323B1 (en) * 1999-08-30 2002-08-06 Carla M. Pugh Medical examination teaching system
US6652283B1 (en) * 1999-12-30 2003-11-25 Cerego, Llc System apparatus and method for maximizing effectiveness and efficiency of learning retaining and retrieving knowledge and skills
US20040249678A1 (en) * 2003-06-03 2004-12-09 Henderson E. Devere Systems and methods for qualifying expected risk due to contingent destructive human activities

Cited By (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20070048710A1 (en) * 2005-08-09 2007-03-01 The University Of North Dakota Bioterrorism and disaster response system
US20070174093A1 (en) * 2005-09-14 2007-07-26 Dave Colwell Method and system for secure and protected electronic patient tracking
US20090018869A1 (en) * 2005-09-14 2009-01-15 Patrick J Greischar Method and system for data aggregation for real-time emergency resource management
US8428961B2 (en) 2005-09-14 2013-04-23 Emsystem, Llc Method and system for data aggregation for real-time emergency resource management
US20070297589A1 (en) * 2005-09-14 2007-12-27 Greischar Patrick J Method and system for data aggregation for real-time emergency resource management
US20070260436A1 (en) * 2006-04-27 2007-11-08 Lockheed Martin Integrated Systems And Solutions System and method for evaluating system architectures
US20090099983A1 (en) * 2006-05-19 2009-04-16 Drane Associates, L.P. System and method for authoring and learning
US20080046285A1 (en) * 2006-08-18 2008-02-21 Greischar Patrick J Method and system for real-time emergency resource management
US8577718B2 (en) 2010-11-04 2013-11-05 Dw Associates, Llc Methods and systems for identifying, quantifying, analyzing, and optimizing the level of engagement of components within a defined ecosystem or context
US8996359B2 (en) 2011-05-18 2015-03-31 Dw Associates, Llc Taxonomy and application of language analysis and processing
US8952796B1 (en) 2011-06-28 2015-02-10 Dw Associates, Llc Enactive perception device
US9269353B1 (en) 2011-12-07 2016-02-23 Manu Rehani Methods and systems for measuring semantics in communications
US9020807B2 (en) 2012-01-18 2015-04-28 Dw Associates, Llc Format for displaying text analytics results
US9667513B1 (en) 2012-01-24 2017-05-30 Dw Associates, Llc Real-time autonomous organization
US20140315159A1 (en) * 2013-01-29 2014-10-23 Wells Fargo Bank, N.A. Banking Services Experience Center
US9772766B2 (en) * 2013-01-29 2017-09-26 Wells Fargo India Solutions Private Limited Banking services experience center
US9418116B2 (en) * 2014-08-01 2016-08-16 Ncino, Inc. Capturing evolution of a resource memorandum according to resource requests
US20160034834A1 (en) * 2014-08-01 2016-02-04 Ncino, Inc. Capturing evolution of a resource memorandum according to resource requests

Also Published As

Publication number Publication date Type
US7991729B2 (en) 2011-08-02 grant
GB2415823B (en) 2009-07-01 grant
GB0512298D0 (en) 2005-07-27 grant
GB2415823A (en) 2006-01-04 application
US20080026348A1 (en) 2008-01-31 application

Similar Documents

Publication Publication Date Title
Bray et al. Selection of salesmen by means of an assessment center.
Woodman et al. The role of team development in organizational effectiveness: A critical review.
US5590057A (en) Training and certification system and method
US5991595A (en) Computerized system for scoring constructed responses and methods for training, monitoring, and evaluating human rater's scoring of constructed responses
US7310626B2 (en) Electronic employee selection systems and methods
Ashby* Monitoring student retention in the Open University: definition, measurement, interpretation and action
Salas et al. Situation awareness in team performance: Implications for measurement and training
Staples et al. A self-efficacy theory explanation for the management of remote workers in virtual organizations
US20020055866A1 (en) Computer-implemented system for human resources management
Brewer All measures of performance are subjective: More evidence on US federal agencies
Cooke et al. Measuring team knowledge during skill acquisition of a complex task
Zaccaro Nonequivalent associations between forms of cohesiveness and group-related outcomes: Evidence for multidimensionality
US20030044760A1 (en) Method for improved administering of tests using customized user alerts
Nielsen et al. It's worth the hassle!: the added value of evaluating the usability of mobile systems in the field
Oser et al. Toward a definition of teamwork: An analysis of critical team behaviors
Staples et al. A self-efficacy theory explanation for the management of remote workers in virtual organizations
Brannick et al. The measurement of team process
Stout et al. Individual task proficiency and team process behavior: What's important for team functioning?
US20060072739A1 (en) Method and system for assessing and deploying personnel for roles in a contact center
Randel et al. Differences in expert and novice situation awareness in naturalistic decision making
Hirokawa et al. Task-group communication and decision-making performance
Cohen Defining and measuring effectiveness in public management
Brawley et al. Conducting team building research in the context of sport and exercise
US20020116242A1 (en) Emergency response synchronization matrix
US20050273381A1 (en) System and method for monitoring employee productivity, attendance and safety

Legal Events

Date Code Title Description
AS Assignment

Owner name: LOCKHEED MARTIN CORPORATION, FLORIDA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:FALASH, MARK D.;DONOVAN, KENNETH B.;BILAZARIAN, PETER;REEL/FRAME:015853/0575;SIGNING DATES FROM 20040813 TO 20040831