WO2015099581A2 - Method and system of employee competency assessment field of the invention - Google Patents

Method and system of employee competency assessment field of the invention Download PDF

Info

Publication number
WO2015099581A2
WO2015099581A2 PCT/RU2014/000995 RU2014000995W WO2015099581A2 WO 2015099581 A2 WO2015099581 A2 WO 2015099581A2 RU 2014000995 W RU2014000995 W RU 2014000995W WO 2015099581 A2 WO2015099581 A2 WO 2015099581A2
Authority
WO
WIPO (PCT)
Prior art keywords
expert
tasks
assessment
data
exercise
Prior art date
Application number
PCT/RU2014/000995
Other languages
French (fr)
Other versions
WO2015099581A3 (en
Inventor
Evgenij Valentinovich LUR`E
Original Assignee
Zakrytoe Aktsionernoe Obshchestvo "Ekopsi"
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Zakrytoe Aktsionernoe Obshchestvo "Ekopsi" filed Critical Zakrytoe Aktsionernoe Obshchestvo "Ekopsi"
Publication of WO2015099581A2 publication Critical patent/WO2015099581A2/en
Publication of WO2015099581A3 publication Critical patent/WO2015099581A3/en

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/10—Office automation; Time management
    • G06Q10/105—Human resources
    • G06Q10/1053—Employment or hiring
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
    • G06Q10/063—Operations research, analysis or management
    • G06Q10/0631—Resource planning, allocation, distributing or scheduling for enterprises or organisations
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/10—Office automation; Time management
    • G06Q10/105—Human resources

Definitions

  • This invention belongs to the methods and systems that allow assessing remotely personnel of entities. More specifically, the invention allows assessing competencies through remote observation of personnel behavior in simulated situations.
  • Personnel assessment is a critical element of an entity's (organizations) personnel management system. Personnel assessment allows assessing its quality from the standpoint of conformity to the entity's business goals and strategy.
  • Personnel assessment is performed during different phases of the personnel management system functioning for a variety of purposes, specifically:
  • This phase involves assessing the existing workforce capacity and shaping personnel retention requirements
  • This phase involves assessing the existing versus required personnel levels and determining the necessity of training specific employees, checking the conformity of employee career levels to their competency;
  • the remote assessment task is accomplished through indirect interaction between the assessee and the assessing expert: the assessee completes a questionnaire or undergoes online testing, fulfills a competitive analytical or creative assignment, which is further submitted to the expert for assessment.
  • the expert is a person specifically trained to observe, describe, classify and assess the behavior of the assessment event participant(s) in specific exercises, to interview and to align his/her assessments with other experts involved in the assessment event. The expert assesses a participant based on the findings.
  • Information reliability is the ability to be perceived correctly. Generally, information reliability is achieved through:
  • Information may be deemed complete if the minimal set of indicators it comprises is sufficient for making the right decision.
  • Key competencies of the participating employee are assessed using the competency assessment tool.
  • the developer determines key competencies out of any set of topics and well-developed questions and demonstrates tasks for the corresponding training goals according to the principal competencies. Participants, in their turn, accomplish demonstration tasks assessed by observers and answer questions. Grade is calculated based on assessment and answer, with a view to determining how well the participant demonstrates training goals and competency, and then communicated to the participant. When the participant's score drops below the existing threshold, it is recommended that the administrator prepare development plan for the participant. In some options, the participant is an expert in the discipline and checks connection between the training goals and questions, demonstration tasks.
  • the major drawbacks of the above invention are asynchronous interaction between the participant and the expert, hampering the assessment of the participant's behavior while being assessed, and the lack of interactive exercises, which allow assessing the task executing progress as well as the result, the flexibility of decisions, change of argumentation focus depending on the exercise statement and other task executing qualities assessed based on the behavior observed in solving various analytic cases or in different simulated situations in the course of role- playing.
  • Computer system for employee candidates and web-based interactive assessment software the method of assessing the employee candidate's competencies and skills, offering potential candidates a series of employee-customer scenarios and associated questions requiring an interactive answer, in which the type of answers allows employers to select more effectively potential employees who have the qualifications necessary for a given position.
  • the invention's drawback is the impossibility of interaction between one or more assessees and one or more assessing experts within a single scenario. As a consequence, nothing but the assignment product is assessed whereas the product of analytical activity is inherent in the presentation of results, thus increasing the influence of the assessee's presentation skill on other qualities being assessed.
  • This invention is aimed at eliminating the inherent drawbacks of the existing solutions.
  • the underlying approach offers the following benefits:
  • the technical result of the invention is an increase in the observed assessment parameters, causing more complete information received in assessing the participant, greater information reliability because of the possibility of monitoring the assignment progress, which allows reducing the likelihood of prompts, cheat sheets, technical tools being used to help accomplish the assignments; more objective assessment because of the possibility of identifying misprints, correct line of reasoning despite incorrect answer and similar situations.
  • An increase in the observed parameters is achieved by broadcasting video stream and ongoing (real-time) information about the progress of exercises from the participant in the course of assessment, including, but not limited to, the information about the current status of assignment, the text entered in the exercise input fields, the status of control elements (button pressed/released, flag selected, switch selected, etc.).
  • Some options of the invention offer feedback from the expert transmitted to the participant, which allows asking questions and finding out the progress of task executing by the participant.
  • the technical effects achieved when using the invention include greater efficiency of using time and technical resources of the system because of scheduling the accomplishment of exercises and commutations, thus shortening downtime and causing optimal software and hardware load.
  • the processing rate equals to the duration of assessment session multiplied by the number of participants.
  • the processing rate decreases logarithmically.
  • Another technical effect of the invention is enhanced efficiency of communication channels since video broadcasting is carried out as needed rather than on an ongoing basis.
  • the use of the tasks sequence development (planning) method increases the speed of scheduling and lowers the likelihood of planning errors.
  • the employee's competency assessment method consists of the following steps:
  • - at least one task is an exercise
  • At least one task is commutation with the expert.
  • video stream and information about the current exercise progress are transmitted from the assessee at least to one expert.
  • the employee's competency assessment method consists of the following steps:
  • At least one task is an exercise
  • video stream and information about the current exercise progress are transmitted from the assessee at least to one expert in the order for accomplishing commutations received during the previous step • Processing of the assessment results.
  • Data about the assessee comprise at least full name and data for authorization in the system, including at least login and password.
  • the assessing expert data comprise at least full name, data about the entity, the assessment system access rights, data for authorization in the system, including at least login and password.
  • Information about the current exercise progress comprises one of the following elements or their combination:
  • the assessment scenario is received from remote data storage or remote server.
  • the assessment scenario is received from local data storage.
  • An exercise may be performed in the form of a test, role-playing, analytic case, e- mail emulator.
  • the order for executing tasks may be changed by the expert.
  • the order for executing tasks by participants is established as follows: • At least the following is received:
  • At least one connection is made between each participant and the expert from the expert list for all tasks in the schedule, in which case the schedule of tasks must comprise at least two tasks with references to different experts from the expert list.
  • the above method for establishing the order of exercises to be followed by participants may undergo the following modification, in which the shaping of participants' task executing schedule occurs in two phases: a. A preliminary sequence of tasks is made, in which case the sequence for execution of tasks by participants in even positions on the list differs from the sequence for participants in odd positions on the list; b. The operator prepares an optimal schedule based on the preliminary sequence of tasks.
  • Such methods may be implemented as computer appliance.
  • the employee competency assessment system includes:
  • At least one task is an exercise
  • At least one task is commutation with the expert.
  • video stream and information about the current exercise progress are transmitted from the assessee at least to one expert.
  • the employee competency assessment system in one of the implementation options includes: a. one or more data storage devices;
  • At least one task is an exercise
  • At least one task is commutation with the expert.
  • o data at least about two assessees
  • video stream and information about the current exercise progress are transmitted from the assessee at least to one expert in the order for accomplishing commutations received during the previous step.
  • the assessee data comprise at least full name and data for authorization in the system, including at least login and password;
  • the assessing expert data comprise at least full name, data about the entity, the assessment system access rights, data for authorization in the system, including at least login and password;
  • Information about the current exercise progress comprises one of the following elements or their combination:
  • the assessment scenario is received from remote data storage or remote server;
  • the assessment scenario is received from local data storage
  • An exercise may be performed in the form of a test, role-playing, analytic case, e- mail emulator;
  • the task executing sequence may be changed by the expert
  • the sequence of task executing by the participants is formed as follows: a. At least the following is received:
  • At least one connection is made between each participant and the expert from the expert list for all tasks in the schedule, in which case the schedule must comprise at least two tasks with references to different experts from the expert list.
  • the shaping of participants' task executing schedule occurs in two phases:
  • a preliminary sequence of tasks is made, in which case the sequence for execution of tasks by participants in even positions on the list differs from the sequence for participants in odd positions on the list; b.
  • the operator prepares an optimal schedule based on the preliminary sequence of tasks.
  • Fig3. Chart of commutation between experts and participants in the 2 experts per 4 participants scenario.
  • Fig 5. Session run delay time screen.
  • Fig 6. Exercise screen.
  • Fig 7. Interface of the participant-expert interaction on the participant 's side.
  • Fig 8. Session completion screen.
  • Fig. 1 illustrates the functional chart of the assessment system.
  • Scenario creation tools (2) are used to create a scenario or a family of scenarios (5) for the project, sharing the same content and differing in the number of experts and assessees. Then one scenario is selected from those assigned to the project for carrying out the assessment session.
  • the session involves consecutive presentation of content (6) to the assessment event participants (9) and selective commutation of video communication (4) between the experts (1) and participants (9) according to the scenario.
  • Content is downloaded (6) from external (8) or internal (7) sources. If necessary, any phase of the session (exercisee or video conference) for a given participant may be extended or postponed by one of the assessing experts (1).
  • Fig. 2 illustrates the chart of commutation between experts and participants in the following scenario: 2 experts per 6 participants.
  • the Oy-axis shows the numbers of participants; the Ox-axis, the sequence numbers of video conferences.
  • the points on the graph correspond to each meeting between an expert and a participant.
  • the assessment session completion was followed by the joint work of experts to process and describe the results.
  • each expert conducted 21 video conferences and each participant was involved in 7 video conferences.
  • the ratio in the 6 participants per 2 experts scenario is 3 participants per expert during the session.
  • Fig. 3 illustrates the chart of commutation between experts and participants in the following scenario: 2 experts per 4 participants.
  • each expert dealt with 2 participants per session.
  • the axes of the graph are denominated similarly to the previous scenario option.
  • each expert conducted 14 video conferences and each participant was still involved in 7 video conferences, i.e. the assessment event content for each participant does not depend on a specific scenario - instead, the scenario changes exclusively the model of load allocation among the experts and determines the content presentation sequence and the model of video streams commutation.
  • Fig. 4 illustrates the schedule of a session involving three participants and two experts. The starting time of exercises for the third participant in the schedule is shifted relative to the first two in order to prevent interviews in the exercises of all the three participants from occurring simultaneously.
  • Fig. 5 shows the session run delay time screen.
  • the participant sees the session (10) schedule and the session run delay time indicator (1 1). Once the session starts, the screen is automatically updated and replaced by the participant's main screen.
  • Fig. 6 illustrates the exercise screen. While the participants is performing self- guided exercises and preparing for interactive exercises, the participant's interface displays the session schedule (10), highlighting the current exercise and the time remaining until the end of the current activity (14).
  • the major workspace (17) comprises the assignment text or the participant's response field. Buttons above the workspace are used to switch between the assignment text and response (15, 16).
  • the participant enters his/her answer the input text is displayed to the expert as entered.
  • the participant is able to see his/her own video stream (12).
  • the participant Upon expiry of the intended preparation or exercise completion time, the participant is automatically switched to the interview mode.
  • Fig. 7 illustrates the interface of the participant-expert interaction on the participant's side. Overall, the participant's interface remains the same; the only differences are newly-added video frame for the expert's video stream (13) and blocked text input in the response field. Upon expiry of the intended exercise completion or interviewing time, the screen shows the assignment and instructions for the next exercise.
  • Fig. 8 illustrates the session completion screen. Upon completion of the last exercise/interview, the participant is redirected to the session completion screen.
  • Fig. 9 illustrates the expert's pre-session screen.
  • the expert sees the time feed (20), highlighting the moments of interaction with the assessee (21).
  • the time feed may be rescaled.
  • the expert is able to select any event on the time feed, in which case space 26 will show the list of competencies (27, 27A) and their respective indicators (28) assessed in the selected exercise/interview. It is also possible to display on screen the documents related to the selected exercise/interview - fulfillment instructions, participant instructions, interview questions, assignment text, participant's response (which is empty before the exercise begins) and other documents by selecting the relevant point in the heading of space 26 or 30.
  • the session control panel (22) comprises two timers - timer from the beginning of the current activity (reset to zero before the session start), timer until the end of the current activity (before the session shows the time remaining until the session start) - and the "Start" and "Postpone” buttons.
  • the session is deemed to have begun after the "Start” is pressed; if the participant has already entered the session, the broadcasting of his/her video stream begins. Pushing the "Postpone” button will postpone the session start time.
  • Fig. 10 illustrates the expert's screen while the participant proceeds with the exercise.
  • the expert's interface looks similarly (Fig. 10) except that the message (25) is replaced by the participant's video frame (12) and the "Start" button (23) disappears from the session control panel.
  • the comment field (29) is activated in the space which specifies the competencies being assessed (30).
  • the comments added by the expert during the session are available while the results are being processed.
  • Fig. 11 illustrates the expert's screen during the interview. When the participant moves on to the interview, the expert's video frame (13) and video stream control buttons such as sound transmission disconnection (33) and video transmission disconnection (34) appear in the interface.
  • the session control panel (22) shows additional buttons for moving the current exercise to the session end (35) and for early exercise completion (36).
  • the expert's interface status becomes similar to the pre-session status (Fig. 9); after the "Start" button is pushed, the expert's interface commences the participant's preparation monitoring (Fig. 10).
  • Fig. 12 illustrates the expert's screen after the session is over.
  • the expert's interface looks as follows: the session control panel has two buttons - switch to results processing (37) and save grades assigned during the session subsequent processing (38).
  • the participant's video frame is replaced with a static message (25).
  • Fig. 13 illustrates the Inbasket exercise chart.
  • the input xml file presets the initial system status (3), the acceptable user actions (2) and the applicable processing rules (4). While doing the exercise, the assessee may only perform actions permitted by the exercise settings, namely: respond to letters, forward letters, complete tests, create calendar events, etc. Each action is accompanied by checking whether or not the rule related to the action being performed exists. If it does, the relevant scenario branch is activated; otherwise, the exercise scenario remains unchanged and the participant proceeds with the exercise.
  • the employee's competency assessment method consists of the following steps: • Obtainment of the assessment scenario comprising:
  • - at least one task is an exercise
  • At least one task is commutation with the expert.
  • video stream and information about the current exercise progress are transmitted from the assessee at least to one expert.
  • the assessment scenario comprises:
  • - at least one task is an exercise
  • At least one task is commutation with the expert.
  • video stream and information about the current exercise progress are transmitted from the assessee at least to one expert in the order of accomplishing commutations that was received during the previous step.
  • the above method may also be implemented as a software and hardware complex.
  • the employee's competency assessment system including:
  • video stream and information about the current exercise progress are transmitted from the assessee at least to one expert.
  • the system may undergo the following modifications: a. the assessment scenario comprises:
  • - at least one task is an exercise
  • At least one task is commutation with the expert.
  • video stream and information about the current exercise progress are transmitted from the assessee at least to one expert in the order of accomplishing commutations that was received during the previous step.
  • the method and the system may undergo the modifications described below.
  • the assessee data comprise at least a full name and data for authorization in the system, including at least a login and a password;
  • the assessing expert data comprise at least a full name, data about the entity, the assessment system access rights, data for authorization in the system, including at least a login and a password;
  • Information about the current exercise progress comprises one of the following elements or their combination:
  • the assessment scenario is received from remote data storage or remote server;
  • the assessment scenario is received from local data storage
  • An exercise may be performed in the form of a test, role-playing, analytic case, e-mail emulator;
  • the task executing sequence may be changed by the expert;
  • the order of exercises to be followed by participants is formed as follows: a. At least the following is received:
  • a task executing schedule is made for each participant; c. At least one connection is made between each participant and the expert from the expert list for all tasks in the schedule, in which case the schedule of tasks must comprise at least two tasks with references to different experts from the expert list.
  • the shaping of participants' task executing schedule occurs in two phases:
  • a preliminary sequence of tasks is made, in which case the sequence for execution of tasks by participants in even positions on the list differs from the sequence for participants in odd positions on the list;
  • the operator prepares an optimal schedule based on the preliminary sequence of tasks.
  • the assessment is to be preceded by the creation of accounts for assessees and the assessing experts (if not yet available), in which case the following data are to be preset for the assessee:
  • the data to be preset for the expert are as follows:
  • Data for authentication may include both common login/password pairs and authentication via digital certificate, smart card and other tools.
  • the assessment system access rights show which actions may be performed, which system sections are accessible, and which projects are available to the assessing expert. Inside the projects, the expert may be entitled to act as observer during the sessions, to conduct integration sessions, to change exercise instructions, etc. Following authentication, the assessee moves on to the session run delay time screen (Fig. 5); the expert, to the list of sessions that he/she is appointed to observe. From the list of sessions, he/she may navigate to the needed (up-to-date) session.
  • “Scenario” means data comprising the schedule of execution of exercises and commutations within the assessment session for all participants and all assessing experts. Such schedule is exemplified in Fig. 2.
  • a single set of exercises may correspond to several scenarios differing in the initial quantity of experts and/or assessees, which allows adjusting flexibly to the assessment environment.
  • “Assessment session” means a set of exercises performed by each assessee with a view to assessing his/her professional and personal qualities.
  • “Project” means a group of assessment sessions using the same set of exercises and tests for executing the same assessment task.
  • the schedule is created using special automatic and semi-automatic algorithms.
  • One of the semi-automatic scheduling options is the following algorithm:
  • the initial arrangement of exercises is preset by the simplest rearrangements (participant 1 is doing exercise A while participant 2 is doing exercise B and vice versa). Subsequent rearrangements in the schedule are carried out by the editor or another person in charge. That said, the algorithm checks and ensures that the observer does not work simultaneously with several participants or more than one expert work with one participant (in case of multiple experts in a scenario).
  • the algorithms used in certain invention options may include, but not be limited to, genetic scheduling algorithms.
  • the video streams commutation chart as exemplified by a fragment of the two experts per three participants scenario is shown in Fig. 3.
  • Fig. 3 The video streams commutation chart as exemplified by a fragment of the two experts per three participants scenario is shown in Fig. 3.
  • expert 1 is connected to participant 1; expert 2, to participant 2; meanwhile, participant 3 is still waiting for the assessment session to begin and is shown a message specifying the time remaining until connection of the expert.
  • participant 3 is still waiting for the assessment session to begin and is shown a message specifying the time remaining until connection of the expert.
  • participant 3 Upon completion of introduction and exercise 1, expert 1 is connected to participant 3, to whom this moment marks the beginning of the assessment session.
  • participants 1 and 2 are doing the exercise requiring no interaction with the expert and expert 2 is a having a break.
  • participants 1 and 2 are connected to expert 2 and expert 1 respectively.
  • participant 2 is shown a message asking to wait for connection to the expert.
  • both the participant and the expert need to authorize in the system, typically using the login/password pair. Then the participant connects to the assessment session and the expert gets the system interaction interface.
  • scenario is run on the user device, in which case content is presented according to the scenario.
  • Content includes exercises, textual instructions and video instructions concerning exercises, as well as commutations with the expert.
  • Scenario is the sequence in which exercises are to be performed by the user and the order of commutations with the expert. Scenario may be either stored in a local data storage on the user device or received from remote data storages and remote servers.
  • the system comprises different types of exercises (see the details below).
  • Exercise type Inbasket.
  • the assessee is able to perform the main actions in email client, namely: 1. To manage files and to move letters.
  • exercise interface allows integrating interfaces for completing questionnaires.
  • Exercise type case (analytic case).
  • the idea of the exercise is that the assessee is given information for analysis and the instruction comprising wordings of information handling goals and objectives.
  • information for analysis and analytical task are related to the professional field of the assessee or that of the position sought by the assessee.
  • the assessee analyzes independently the information received within the preparation time limits. Furthermore, he/she is allowed to write down his/her ideas, to mark the key points of the materials provided, to prepare the plan of his/her answer, etc. using the response field.
  • the participant communicates with the expert by providing him/her with the analysis results, answering questions, etc.
  • Exercise type role-playing.
  • the idea of the exercise is that the assessee is given the instruction comprising the description of the current situation and the assessee's position in a certain simulated situation related to his/her professional field or that of the position sought by the assessee.
  • the instruction also comprises the wordings of the assessee's goals and objectives in the upcoming interaction with other participants in the simulated situation.
  • a short time for preparation is directly followed by role-playing - the participant meets the expert acting as his/her opponents in the simulated situation.
  • Role-playing interaction is of limited duration. Once it is over, the participant and the expert proceed to discussing the quality of role-playing interaction, solutions and the degree of goals achievement by the participant.
  • the assessment session also includes the commutations of video connection between the assessee and the expert.
  • Video connection commutation allows the expert to conduct a structured interview with the assessee for the purpose of assessing behavioral manifestations of the assessed qualities in non-role-playing situations and the assessee's experience.
  • Video connection commutation is based on the following principle:
  • Video broadcasting may be suspended or terminated at the expert's sole discretion; (d) outside the video conference, video stream is broadcast from the assessee to the expert whose meeting comes first, specifically unless he/she is busy with another video conference;
  • all video streams are saved in the video storage (4) and are available for reproduction both in the course and upon completion of the assessment session.
  • the participant When entering the assessment session interface before it starts (Fig. 5), the participant sees the session schedule (10) and the indicator of the time remaining until the session beginning (11).
  • the session beginning is accompanied by the automatic screen update and replacement by the participant's main screen, which is implemented in two options - the participant's screen during self-directed work (Fig. 6) and the participant's screen during interaction with the expert (Fig. 7).
  • the participant's interface displays the session schedule (10), highlighting the current exercise and the time remaining until the end of the current activity (14).
  • the major workspace (17) comprises the assignment text or the participant's response field. Buttons above the workspace are used to switch between the assignment text and response (15, 16).
  • the participant When the participant enters his/her answer, the input text is displayed to the expert as entered. In the course of preparation, the participant is able to see his/her own video stream (12). Upon expiry of the intended preparation or exercise completion time, the participant is automatically switched to the interview mode (Fig. 7). Overall, the participant's interface remains the same; the only differences are a newly-added video frame for the expert's video stream (13) and a blocked text input in the response field. Upon expiry of the intended exercise completion or interviewing time, the screen shows the assignment and instructions for the next exercise (Fig. 6). Upon completion of the last exercise/interview, the participant is redirected to the session completion screen (Fig. 8).
  • the expert sees the time feed (20), highlighting the moments of interaction with the assessee (21).
  • the time feed may be rescaled.
  • the expert is able to select any event on the time feed, in which case space 26 will show the list of competencies (27, 27A) and their respective indicators (28) assessed in the selected exercise/interview. It is also possible to display on screen the documents related to the selected exercise/interview - fulfillment instructions, participant instructions, interview questions, assignment text, participant's response (which is empty before the exercise begins) and other documents by selecting the relevant point in the heading of space 26 or 30.
  • the participant's video frame is replaced by the message "Video will appear after the session start" (25).
  • the session control panel (22) comprises two timers - timer from the beginning of the current activity (reset to zero before the session start), timer until the end of the current activity (before the session shows the time remaining until the session start) - and the "Start" and "Postpone” buttons.
  • the session is deemed to have begun after the "Start” is pressed; if the participant has already entered the session, the broadcasting of his/her video stream begins. Pushing the "Postpone” button will postpone the session start time.
  • the expert's interface looks similarly (Fig. 10) except that the message (25) is replaced by the participant's video frame (12) and the "Start” button (23) disappears from the session control panel.
  • the comment field (29) is activated in the space which specifies the competencies being assessed (30).
  • the comments inserted by the expert during the session are available while the results are being processed.
  • the expert's video frame (13) and video stream control buttons such as sound transmission disconnection (33) and video transmission disconnection (34) pop up in the interface.
  • the session control panel (22) shows additional buttons for moving the current exercise to the session end (35) and for early exercise completion (36).
  • the expert's interface status becomes similar to the pre-session status (Fig. 9); after the "Start" button is pushed, the expert's interface commences the participant's preparation monitoring (Fig. 10).
  • the expert's interface looks as follows: the session control panel has two buttons— switch to results processing (37) and the saving of grades assigned during the session subsequent processing (38).
  • the participant's video frame is replaced with a static message (25).
  • the assessment session completion is followed by the assessment results processing and description step.
  • the system offers two methods for arranging the step, namely: the online method and the offline method.
  • the results of each participant are processed through the system web interface, in which case the processing of results may involve either one or two (or more) experts, working remotely or otherwise.
  • the results processing interface is used as a separable desktop; changes are synchronized automatically without any extra actions from users. It is possible to discuss the results online through video conferencing.
  • video connection organization video connection organization is similar to the broadcasting of stream video from the expert during the assessment session: video broadcasting may start and end at the expert's sole discretion; however, unlike video conference during the assessment session, video connection in the integration session is available throughout the processing and description of results.
  • all data about the participant's progress with exercises, as well as the comments given by the assessing experts may be downloaded for processing to a local application.
  • experts directly discuss the grade for each competency of each participant in the assessment event, adducing arguments in favor of any given grade, relying on the data stored in or downloaded from the system.
  • the progress of discussion is similar to the convergence of grades after the assessment center.
  • the final grade is recorded in the system through the online convergence interface.
  • the integration session products are grades for competencies and the generated automatic report. An integration session is deemed incomplete unless the final grade is missing at least for one competency.
  • This invention improves the efficiency, completeness and quality of remote assessment, thus optimizing the time spent idle by participants and ensuring that hardware resources are utilized efficiently.

Abstract

This invention belongs to the methods and systems which allow assessing remotely the personnel of entities. More specifically, the invention allows assessing competencies through remote observation of personnel behavior in simulated situations. The technical result of the invention (personnel assessment method) is an increase in the observed assessment parameters, causing more complete information received in assessing the participant, greater information reliability due to the possibility of monitoring the assignment accomplishment progress, which allows reducing the likelihood of prompts, cheat sheets, technical tools being used to help accomplish the assignments; more objective assessment due to the possibility of identifying misprints, correct line of reasoning despite incorrect answer and similar situations.

Description

METHOD AND SYSTEM OF EMPLOYEE COMPETENCY ASSESSMENT
FIELD OF THE INVENTION
This invention belongs to the methods and systems that allow assessing remotely personnel of entities. More specifically, the invention allows assessing competencies through remote observation of personnel behavior in simulated situations.
BACKGROUND OF THE INVENTION
Personnel assessment is a critical element of an entity's (organizations) personnel management system. Personnel assessment allows assessing its quality from the standpoint of conformity to the entity's business goals and strategy.
Personnel assessment is performed during different phases of the personnel management system functioning for a variety of purposes, specifically:
- in determining personnel requirements in the course of calendar planning. This phase involves assessing the existing workforce capacity and shaping personnel retention requirements;
- in selecting personnel with a view to determining the eligibility of candidates for vacant positions;
- in determining the need for personnel development. This phase involves assessing the existing versus required personnel levels and determining the necessity of training specific employees, checking the conformity of employee career levels to their competency;
- in personnel certification carried out on a regular basis for assessing workforce capacity level and developing a broad range of regulatory impacts, specifically:
• for taking measures related to remunerations, promotions, sanctions, dismissals;
• for personnel rotation planning;
• for succession pool planning;
• for developing employee motivation and stimulation.
Typically, the remote assessment task is accomplished through indirect interaction between the assessee and the assessing expert: the assessee completes a questionnaire or undergoes online testing, fulfills a competitive analytical or creative assignment, which is further submitted to the expert for assessment. The expert is a person specifically trained to observe, describe, classify and assess the behavior of the assessment event participant(s) in specific exercises, to interview and to align his/her assessments with other experts involved in the assessment event. The expert assesses a participant based on the findings. Such solution has the following drawbacks: first, inability to assess the assessee's communication skills and the associated competencies; second, the impossibility of expert assessment of behavior while executing a task; third, the impossibility of controlling whether or not the assessee fulfills assignments on his/her own.
The reliability, completeness and objectivity of the information received are critical to personnel assessment. Information reliability is the ability to be perceived correctly. Generally, information reliability is achieved through:
• specifying the time of events which details are delivered;
• comparing data received from different sources;
• timely discovery of misleading information;
· removal of distorted information, etc.
Information may be deemed complete if the minimal set of indicators it comprises is sufficient for making the right decision.
With standard testing techniques and remote personnel assessment, it is difficult to check the reliability of information received, the lack of third-party intervention when the participant fulfills an assignment (use of cheat sheets, wireless headphones and other technical tools).
Data completeness is crucial to assessment. It implies the availability of the entire necessary set of parameters being observed, ensuring the highest quality of assessment. When some parameters are missing, the quality and accuracy of assessment worsen greatly, for example, when assignments only include tests that show whether the participant answered any given assignment correctly or incorrectly, ignoring the possibility of misprint, accidental mistake and whether or not the test has been passed.
One of the existing methods is described in patent application US2012/007717 A1 "Competency assessment tool", March 29, 2012:
Key competencies of the participating employee are assessed using the competency assessment tool. The developer determines key competencies out of any set of topics and well-developed questions and demonstrates tasks for the corresponding training goals according to the principal competencies. Participants, in their turn, accomplish demonstration tasks assessed by observers and answer questions. Grade is calculated based on assessment and answer, with a view to determining how well the participant demonstrates training goals and competency, and then communicated to the participant. When the participant's score drops below the existing threshold, it is recommended that the administrator prepare development plan for the participant. In some options, the participant is an expert in the discipline and checks connection between the training goals and questions, demonstration tasks.
The major drawbacks of the above invention are asynchronous interaction between the participant and the expert, hampering the assessment of the participant's behavior while being assessed, and the lack of interactive exercises, which allow assessing the task executing progress as well as the result, the flexibility of decisions, change of argumentation focus depending on the exercise statement and other task executing qualities assessed based on the behavior observed in solving various analytic cases or in different simulated situations in the course of role- playing.
One of the existing methods is described in patent application US2003/0208752 "Employee candidate computer and web-based interactive assessment software and method of employee candidate assessment", November 6, 2003. Declared is the following:
Computer system for employee candidates and web-based interactive assessment software, the method of assessing the employee candidate's competencies and skills, offering potential candidates a series of employee-customer scenarios and associated questions requiring an interactive answer, in which the type of answers allows employers to select more effectively potential employees who have the qualifications necessary for a given position.
The invention's drawback is the impossibility of interaction between one or more assessees and one or more assessing experts within a single scenario. As a consequence, nothing but the assignment product is assessed whereas the product of analytical activity is inherent in the presentation of results, thus increasing the influence of the assessee's presentation skill on other qualities being assessed.
The above-mentioned technical solutions have the following drawbacks:
incomplete information received while assessing the participant, deteriorating the accuracy and quality of assessment, reduced information reliability, apparent from non-identifiability of the supporting materials and technical tools affecting the progress of assessment,
high level of subjectivity resulting from the impossibility of assignment progress monitoring, hampering the detection of correct solution despite incorrect response (such as misprints, etc.).
SUMMARY OF THE INVENTION
This invention is aimed at eliminating the inherent drawbacks of the existing solutions. The underlying approach offers the following benefits:
• effective remote assessment of multiple participants
· synchronous interaction between the participant and the expert that allows not only assessing the result but also using the information accompanying the progress of exercises, namely: behavior, task executing progress, the flexibility of decisions, etc.
• more accurate and complete assessment of the participant's competencies. The technical result of the invention (personnel assessment method) is an increase in the observed assessment parameters, causing more complete information received in assessing the participant, greater information reliability because of the possibility of monitoring the assignment progress, which allows reducing the likelihood of prompts, cheat sheets, technical tools being used to help accomplish the assignments; more objective assessment because of the possibility of identifying misprints, correct line of reasoning despite incorrect answer and similar situations.
An increase in the observed parameters is achieved by broadcasting video stream and ongoing (real-time) information about the progress of exercises from the participant in the course of assessment, including, but not limited to, the information about the current status of assignment, the text entered in the exercise input fields, the status of control elements (button pressed/released, flag selected, switch selected, etc.).
Some options of the invention offer feedback from the expert transmitted to the participant, which allows asking questions and finding out the progress of task executing by the participant.
The technical effects achieved when using the invention include greater efficiency of using time and technical resources of the system because of scheduling the accomplishment of exercises and commutations, thus shortening downtime and causing optimal software and hardware load. When the system is used by multiple participants and one expert, the processing rate equals to the duration of assessment session multiplied by the number of participants. When two or more experts are used by multiple participants, the processing rate decreases logarithmically.
Another technical effect of the invention is enhanced efficiency of communication channels since video broadcasting is carried out as needed rather than on an ongoing basis.
The use of the tasks sequence development (planning) method increases the speed of scheduling and lowers the likelihood of planning errors.
The employee's competency assessment method consists of the following steps:
• Obtainment of the assessment scenario comprising:
■ at least two tasks, in which case
- at least one task is an exercise;
- at least one task is commutation with the expert.
■ order of executing the above-mentioned tasks;
■ data about at least one assessee;
■ data about at least one assessing expert;
• Consecutive execution of tasks according to the assessment scenario, in which case
- if the task being executed is an exercise, video stream and information about the current exercise progress are transmitted from the assessee at least to one expert.
· Processing of the assessment results.
The employee's competency assessment method, according to one of implementation options, consists of the following steps:
• Obtainment of the assessment scenario comprising:
■ at least two tasks, in which case
at least one task is an exercise
- at least one task is commutation with the expert
■ data at least about two assessees
■ data at least about two assessing experts ■ order of executing the above-mentioned tasks for each participant
■ order of carrying out commutations between experts and participants
· Execution of tasks in the sequence received during the previous step for each participant, in which case
- if the task being executed is an exercise, video stream and information about the current exercise progress are transmitted from the assessee at least to one expert in the order for accomplishing commutations received during the previous step • Processing of the assessment results.
Data about the assessee comprise at least full name and data for authorization in the system, including at least login and password.
The assessing expert data comprise at least full name, data about the entity, the assessment system access rights, data for authorization in the system, including at least login and password.
Information about the current exercise progress comprises one of the following elements or their combination:
• text entered in the input field;
• status of control elements;
• status of the exercise being performed.
In certain implementation options, the assessment scenario is received from remote data storage or remote server.
In certain implementation options, the assessment scenario is received from local data storage.
An exercise may be performed in the form of a test, role-playing, analytic case, e- mail emulator.
In certain implementation options, the order for executing tasks may be changed by the expert.
In certain implementation options, the order for executing tasks by participants is established as follows: • At least the following is received:
- the list of participants comprising at least two participants;
- the list of tasks comprising at least two tasks;
- the list of experts comprising at least two experts;
• A task executing schedule is made for each participant;
• At least one connection is made between each participant and the expert from the expert list for all tasks in the schedule, in which case the schedule of tasks must comprise at least two tasks with references to different experts from the expert list.
The above method for establishing the order of exercises to be followed by participants may undergo the following modification, in which the shaping of participants' task executing schedule occurs in two phases: a. A preliminary sequence of tasks is made, in which case the sequence for execution of tasks by participants in even positions on the list differs from the sequence for participants in odd positions on the list; b. The operator prepares an optimal schedule based on the preliminary sequence of tasks.
Such methods may be implemented as computer appliance.
The employee competency assessment system includes:
a. one or more data storage devices;
b. one or more processors;
c. tools ensuring network interaction with remote computer systems;
d. at least one display;
e. at least one input device;
f. one or more computer programs,
where one or more computer programs are read from one or more data storage devices and run with one or more processors, with one or more programs comprising the following instructions:
■ Obtainment of the assessment scenario comprising:
o at least two tasks, in which case
at least one task is an exercise;
at least one task is commutation with the expert.
o order of executing the above-mentioned tasks; o data about at least one assessee;
o data about at least one assessing expert; ■ Consecutive execution of tasks according to the assessment scenario, in which case
if the task being executed is an exercise, video stream and information about the current exercise progress are transmitted from the assessee at least to one expert.
■ Processing of the assessment results.
The employee competency assessment system in one of the implementation options includes: a. one or more data storage devices;
b. one or more processors;
c. tools ensuring network interaction with remote computer systems; d. at least one display;
e. at least one input device;
f. one or more computer programs,
where one or more computer programs are read from one or more data storage devices and run with one or more processors, with one or more programs comprising the following instructions:
■ Obtainment of the assessment scenario comprising:
o at least two tasks, in which case
at least one task is an exercise;
at least one task is commutation with the expert.
o data at least about two assessees;
o data at least about two assessing experts;
o order of executing the above-mentioned tasks for each participant;
o order of carrying out commutations between experts and participants.
■ Execution of tasks in the sequence received during the previous step for each participant, in which case
if the task being executed is an exercise, video stream and information about the current exercise progress are transmitted from the assessee at least to one expert in the order for accomplishing commutations received during the previous step.
■ Processing of the assessment results.
The assessee data comprise at least full name and data for authorization in the system, including at least login and password;
The assessing expert data comprise at least full name, data about the entity, the assessment system access rights, data for authorization in the system, including at least login and password; Information about the current exercise progress comprises one of the following elements or their combination:
• text entered in the input field;
• status of control elements;
• status of the exercise being performed. In certain implementation options, the assessment scenario is received from remote data storage or remote server;
In certain implementation options, the assessment scenario is received from local data storage;
An exercise may be performed in the form of a test, role-playing, analytic case, e- mail emulator;
In certain implementation options, the task executing sequence may be changed by the expert;
In certain implementation options, the sequence of task executing by the participants is formed as follows: a. At least the following is received:
• the list of participants comprising at least two participants;
• the list of tasks comprising at least two tasks;
• the list of experts comprising at least two experts;
b. A task executing schedule is made for each participant;
c. At least one connection is made between each participant and the expert from the expert list for all tasks in the schedule, in which case the schedule must comprise at least two tasks with references to different experts from the expert list.
In certain invention implementation options, the shaping of participants' task executing schedule occurs in two phases:
a. A preliminary sequence of tasks is made, in which case the sequence for execution of tasks by participants in even positions on the list differs from the sequence for participants in odd positions on the list; b. The operator prepares an optimal schedule based on the preliminary sequence of tasks.
OVERVIEW OF CHARTS
Fig 1. - Functional chart of the assessment system.
Fig 2. - Chart of commutation between experts and participants in the 2 experts per 6 participants scenario.
Fig3. - Chart of commutation between experts and participants in the 2 experts per 4 participants scenario.
Fig 4. - 2 participants per 3 experts scenario.
Fig 5. - Session run delay time screen. Fig 6. - Exercise screen.
Fig 7. - Interface of the participant-expert interaction on the participant 's side.
Fig 8.—Session completion screen.
Fig 9 -Expert 's pre-session screen.
Fig 10 - Expert 's screen while the participant proceeds with the exercise.
Fig 11. -Expert 's screen during the interview.
Fig 12.—Expert 's screen after the session is over.
Fig 13.—Inbasket exercise chart. DETAILED DESCRIPTION OF DRAWINGS
Fig. 1 illustrates the functional chart of the assessment system. Scenario creation tools (2) are used to create a scenario or a family of scenarios (5) for the project, sharing the same content and differing in the number of experts and assessees. Then one scenario is selected from those assigned to the project for carrying out the assessment session. The session involves consecutive presentation of content (6) to the assessment event participants (9) and selective commutation of video communication (4) between the experts (1) and participants (9) according to the scenario. Content is downloaded (6) from external (8) or internal (7) sources. If necessary, any phase of the session (exercisee or video conference) for a given participant may be extended or postponed by one of the assessing experts (1).
Fig. 2 illustrates the chart of commutation between experts and participants in the following scenario: 2 experts per 6 participants. The Oy-axis shows the numbers of participants; the Ox-axis, the sequence numbers of video conferences. The points on the graph correspond to each meeting between an expert and a participant. The assessment session completion was followed by the joint work of experts to process and describe the results. As can be seen from the graph, each expert conducted 21 video conferences and each participant was involved in 7 video conferences. The ratio in the 6 participants per 2 experts scenario is 3 participants per expert during the session.
Fig. 3 illustrates the chart of commutation between experts and participants in the following scenario: 2 experts per 4 participants. In this scenario, each expert dealt with 2 participants per session. The axes of the graph are denominated similarly to the previous scenario option. As can be seen from the graph, each expert conducted 14 video conferences and each participant was still involved in 7 video conferences, i.e. the assessment event content for each participant does not depend on a specific scenario - instead, the scenario changes exclusively the model of load allocation among the experts and determines the content presentation sequence and the model of video streams commutation. Fig. 4 illustrates the schedule of a session involving three participants and two experts. The starting time of exercises for the third participant in the schedule is shifted relative to the first two in order to prevent interviews in the exercises of all the three participants from occurring simultaneously.
Fig. 5 shows the session run delay time screen. When entering the assessment session interface before it starts, the participant sees the session (10) schedule and the session run delay time indicator (1 1). Once the session starts, the screen is automatically updated and replaced by the participant's main screen.
Fig. 6 illustrates the exercise screen. While the participants is performing self- guided exercises and preparing for interactive exercises, the participant's interface displays the session schedule (10), highlighting the current exercise and the time remaining until the end of the current activity (14). The major workspace (17) comprises the assignment text or the participant's response field. Buttons above the workspace are used to switch between the assignment text and response (15, 16). When the participant enters his/her answer, the input text is displayed to the expert as entered. In the course of preparation, the participant is able to see his/her own video stream (12). Upon expiry of the intended preparation or exercise completion time, the participant is automatically switched to the interview mode.
Fig. 7 illustrates the interface of the participant-expert interaction on the participant's side. Overall, the participant's interface remains the same; the only differences are newly-added video frame for the expert's video stream (13) and blocked text input in the response field. Upon expiry of the intended exercise completion or interviewing time, the screen shows the assignment and instructions for the next exercise.
Fig. 8 illustrates the session completion screen. Upon completion of the last exercise/interview, the participant is redirected to the session completion screen.
Fig. 9 illustrates the expert's pre-session screen. When entering the assessment session interface before it starts (Fig. 9), the expert sees the time feed (20), highlighting the moments of interaction with the assessee (21). The time feed may be rescaled. The expert is able to select any event on the time feed, in which case space 26 will show the list of competencies (27, 27A) and their respective indicators (28) assessed in the selected exercise/interview. It is also possible to display on screen the documents related to the selected exercise/interview - fulfillment instructions, participant instructions, interview questions, assignment text, participant's response (which is empty before the exercise begins) and other documents by selecting the relevant point in the heading of space 26 or 30. Before the session starts, the participant's video frame is replaced by the message "Video will appear after the session start" (25). The session control panel (22) comprises two timers - timer from the beginning of the current activity (reset to zero before the session start), timer until the end of the current activity (before the session shows the time remaining until the session start) - and the "Start" and "Postpone" buttons. The session is deemed to have begun after the "Start" is pressed; if the participant has already entered the session, the broadcasting of his/her video stream begins. Pushing the "Postpone" button will postpone the session start time.
Fig. 10 illustrates the expert's screen while the participant proceeds with the exercise. After the session starts, the expert's interface looks similarly (Fig. 10) except that the message (25) is replaced by the participant's video frame (12) and the "Start" button (23) disappears from the session control panel. The comment field (29) is activated in the space which specifies the competencies being assessed (30). The comments added by the expert during the session are available while the results are being processed. Fig. 11 illustrates the expert's screen during the interview. When the participant moves on to the interview, the expert's video frame (13) and video stream control buttons such as sound transmission disconnection (33) and video transmission disconnection (34) appear in the interface. The session control panel (22) shows additional buttons for moving the current exercise to the session end (35) and for early exercise completion (36). When the participant moves on to the next exercise, the expert's interface status becomes similar to the pre-session status (Fig. 9); after the "Start" button is pushed, the expert's interface commences the participant's preparation monitoring (Fig. 10).
Fig. 12 illustrates the expert's screen after the session is over. Upon completion of all exercises/interviews in the session, the expert's interface looks as follows: the session control panel has two buttons - switch to results processing (37) and save grades assigned during the session subsequent processing (38). The participant's video frame is replaced with a static message (25).
Fig. 13 illustrates the Inbasket exercise chart. The input xml file presets the initial system status (3), the acceptable user actions (2) and the applicable processing rules (4). While doing the exercise, the assessee may only perform actions permitted by the exercise settings, namely: respond to letters, forward letters, complete tests, create calendar events, etc. Each action is accompanied by checking whether or not the rule related to the action being performed exists. If it does, the relevant scenario branch is activated; otherwise, the exercise scenario remains unchanged and the participant proceeds with the exercise.
DETAILED DESCRIPTION OF THE INVENTION
The general process of system functioning is outlined in Fig. 1.
The employee's competency assessment method consists of the following steps: • Obtainment of the assessment scenario comprising:
■ at least two tasks, in which case
- at least one task is an exercise;
- at least one task is commutation with the expert.
■ order of executing the above-mentioned tasks;
■ data about at least one assessee;
■ data about at least one assessing expert;
• Consecutive execution of tasks according to the assessment scenario, in which case
- if the task being executed is an exercise, video stream and information about the current exercise progress are transmitted from the assessee at least to one expert.
• Integration session.
If necessary, the method described above may be modified as follows:
a. the assessment scenario comprises:
• at least two tasks, in which case
- at least one task is an exercise;
- at least one task is commutation with the expert.
• data at least about two assessees;
• data at least about two assessing experts;
• order of executing the above-mentioned tasks for each participant;
• order of carrying out commutations between experts and participants.
b. execution of tasks in the sequence received during the previous step for each participant, in which case
if the task being executed is an exercise, video stream and information about the current exercise progress are transmitted from the assessee at least to one expert in the order of accomplishing commutations that was received during the previous step.
The above method may also be implemented as a software and hardware complex.
The employee's competency assessment system, including:
a. one or more data storage devices; b. one or more processors;
c. tools ensuring network interaction with remote computer systems; d. at least one display;
e. at least one input device;
f. one or more computer programs,
where one or more computer programs are read from one or more data storage devices and run with one or more processors, with one or more programs comprising the following instructions:
■ Obtainment of the assessment scenario comprising:
o at least two tasks, in which case
•S at least one task is an exercise;
• at least one task is commutation with the expert.
o order of executing the above-mentioned tasks; o data about at least one assessee;
o data about at least one assessing expert;
■ Consecutive execution of tasks according to the assessment scenario, in which case
if the task being executed is an exercise, video stream and information about the current exercise progress are transmitted from the assessee at least to one expert.
■ Processing of the assessment results.
In certain implementation options, the system may undergo the following modifications: a. the assessment scenario comprises:
• at least two tasks, in which case
- at least one task is an exercise;
- at least one task is commutation with the expert.
• data at least about two assessees;
• data at least about two assessing experts;
• order of executing the above-mentioned tasks for each participant;
• order of carrying out commutations between experts and participants. b. execution of tasks in the sequence received during the previous step for each participant, in which case
if the task being executed is an exercise, video stream and information about the current exercise progress are transmitted from the assessee at least to one expert in the order of accomplishing commutations that was received during the previous step.
In different implementation options, the method and the system may undergo the modifications described below.
The assessee data comprise at least a full name and data for authorization in the system, including at least a login and a password;
The assessing expert data comprise at least a full name, data about the entity, the assessment system access rights, data for authorization in the system, including at least a login and a password;
Information about the current exercise progress comprises one of the following elements or their combination:
• text entered in the input field;
· status of control elements;
• status of the exercise being performed.
In certain implementation options, the assessment scenario is received from remote data storage or remote server;
In certain implementation options, the assessment scenario is received from local data storage;
An exercise may be performed in the form of a test, role-playing, analytic case, e-mail emulator;
In certain implementation options, the task executing sequence may be changed by the expert; In certain implementation options, the order of exercises to be followed by participants is formed as follows: a. At least the following is received:
• the list of participants comprising at least two participants;
• the list of tasks comprising at least two tasks;
· the list of experts comprising at least two experts;
b. A task executing schedule is made for each participant; c. At least one connection is made between each participant and the expert from the expert list for all tasks in the schedule, in which case the schedule of tasks must comprise at least two tasks with references to different experts from the expert list.
In certain invention implementation options, the shaping of participants' task executing schedule occurs in two phases:
a. A preliminary sequence of tasks is made, in which case the sequence for execution of tasks by participants in even positions on the list differs from the sequence for participants in odd positions on the list;
b. The operator prepares an optimal schedule based on the preliminary sequence of tasks.
Below is a step-by-step description of the system and method operation.
Step O
The assessment is to be preceded by the creation of accounts for assessees and the assessing experts (if not yet available), in which case the following data are to be preset for the assessee:
• Full name;
· Data for authentication.
The data to be preset for the expert are as follows:
• Full name;
• Data about the entity;
• Assessment system access rights;
· Data for authentication.
Data for authentication may include both common login/password pairs and authentication via digital certificate, smart card and other tools.
The assessment system access rights show which actions may be performed, which system sections are accessible, and which projects are available to the assessing expert. Inside the projects, the expert may be entitled to act as observer during the sessions, to conduct integration sessions, to change exercise instructions, etc. Following authentication, the assessee moves on to the session run delay time screen (Fig. 5); the expert, to the list of sessions that he/she is appointed to observe. From the list of sessions, he/she may navigate to the needed (up-to-date) session.
In advance, before the assessment (assessment session) start, it may be necessary to create a project, using the scenario creation tool (3) and the assessment scenario or family of scenarios (2) sharing the same content and differing in the number of experts and assesses.
"Scenario" means data comprising the schedule of execution of exercises and commutations within the assessment session for all participants and all assessing experts. Such schedule is exemplified in Fig. 2.
A single set of exercises may correspond to several scenarios differing in the initial quantity of experts and/or assessees, which allows adjusting flexibly to the assessment environment.
"Assessment session" means a set of exercises performed by each assessee with a view to assessing his/her professional and personal qualities.
"Project" means a group of assessment sessions using the same set of exercises and tests for executing the same assessment task.
In certain invention implementation options, the schedule is created using special automatic and semi-automatic algorithms. One of the semi-automatic scheduling options is the following algorithm:
The initial arrangement of exercises is preset by the simplest rearrangements (participant 1 is doing exercise A while participant 2 is doing exercise B and vice versa). Subsequent rearrangements in the schedule are carried out by the editor or another person in charge. That said, the algorithm checks and ensures that the observer does not work simultaneously with several participants or more than one expert work with one participant (in case of multiple experts in a scenario).
Moreover, the algorithms used in certain invention options may include, but not be limited to, genetic scheduling algorithms.
The video streams commutation chart as exemplified by a fragment of the two experts per three participants scenario is shown in Fig. 3. According to the chart, after the scenario loading, as soon as the assessment session starts, expert 1 is connected to participant 1; expert 2, to participant 2; meanwhile, participant 3 is still waiting for the assessment session to begin and is shown a message specifying the time remaining until connection of the expert. Upon completion of introduction and exercise 1, expert 1 is connected to participant 3, to whom this moment marks the beginning of the assessment session. Meanwhile, participants 1 and 2 are doing the exercise requiring no interaction with the expert and expert 2 is a having a break. Having finished the exercise without the observer, participants 1 and 2 are connected to expert 2 and expert 1 respectively. Unless expert 1 has finished video conference with participant 3 by that time, participant 2 is shown a message asking to wait for connection to the expert.
STEP 1
To initiate the assessment session, both the participant and the expert need to authorize in the system, typically using the login/password pair. Then the participant connects to the assessment session and the expert gets the system interaction interface.
Within the session, scenario is run on the user device, in which case content is presented according to the scenario. Content includes exercises, textual instructions and video instructions concerning exercises, as well as commutations with the expert.
Scenario is the sequence in which exercises are to be performed by the user and the order of commutations with the expert. Scenario may be either stored in a local data storage on the user device or received from remote data storages and remote servers.
The system comprises different types of exercises (see the details below).
Exercise type: Inbasket. When performing an Inbasket exercise, the assessee is able to perform the main actions in email client, namely: 1. To manage files and to move letters.
2. To respond to a letter.
3. To move a letter to the calendar.
4. To create a letter.
5. To use the calendar and to view events.
6. To create an event.
7. To use the address directory.
Apart from standard email client functions, the exercise interface allows integrating interfaces for completing questionnaires. Exercise type: case (analytic case). The idea of the exercise is that the assessee is given information for analysis and the instruction comprising wordings of information handling goals and objectives. As a rule, information for analysis and analytical task are related to the professional field of the assessee or that of the position sought by the assessee. The assessee analyzes independently the information received within the preparation time limits. Furthermore, he/she is allowed to write down his/her ideas, to mark the key points of the materials provided, to prepare the plan of his/her answer, etc. using the response field. When the time given for information analysis and preparation is over, the participant communicates with the expert by providing him/her with the analysis results, answering questions, etc.
Exercise type: role-playing. The idea of the exercise is that the assessee is given the instruction comprising the description of the current situation and the assessee's position in a certain simulated situation related to his/her professional field or that of the position sought by the assessee. The instruction also comprises the wordings of the assessee's goals and objectives in the upcoming interaction with other participants in the simulated situation. A short time for preparation is directly followed by role-playing - the participant meets the expert acting as his/her opponents in the simulated situation. Role-playing interaction is of limited duration. Once it is over, the participant and the expert proceed to discussing the quality of role-playing interaction, solutions and the degree of goals achievement by the participant.
The assessment session also includes the commutations of video connection between the assessee and the expert. Video connection commutation allows the expert to conduct a structured interview with the assessee for the purpose of assessing behavioral manifestations of the assessed qualities in non-role-playing situations and the assessee's experience.
Video connection commutation is based on the following principle:
(a) video stream is broadcast from each assessee while performing exercises and commutations with the expert, with the exception of breaks inside the scenario;
(b) during the video conference, the participant's video stream is broadcast to the expert participating in the meeting according to the scenario;
(c) during the video conference, video broadcasting from the expert to the corresponding participant begins. Video broadcasting may be suspended or terminated at the expert's sole discretion; (d) outside the video conference, video stream is broadcast from the assessee to the expert whose meeting comes first, specifically unless he/she is busy with another video conference;
(e) in certain invention implementation options, all video streams are saved in the video storage (4) and are available for reproduction both in the course and upon completion of the assessment session.
When entering the assessment session interface before it starts (Fig. 5), the participant sees the session schedule (10) and the indicator of the time remaining until the session beginning (11). The session beginning is accompanied by the automatic screen update and replacement by the participant's main screen, which is implemented in two options - the participant's screen during self-directed work (Fig. 6) and the participant's screen during interaction with the expert (Fig. 7). While the participants is performing self-guided exercises and preparing for interactive exercises, the participant's interface displays the session schedule (10), highlighting the current exercise and the time remaining until the end of the current activity (14). The major workspace (17) comprises the assignment text or the participant's response field. Buttons above the workspace are used to switch between the assignment text and response (15, 16). When the participant enters his/her answer, the input text is displayed to the expert as entered. In the course of preparation, the participant is able to see his/her own video stream (12). Upon expiry of the intended preparation or exercise completion time, the participant is automatically switched to the interview mode (Fig. 7). Overall, the participant's interface remains the same; the only differences are a newly-added video frame for the expert's video stream (13) and a blocked text input in the response field. Upon expiry of the intended exercise completion or interviewing time, the screen shows the assignment and instructions for the next exercise (Fig. 6). Upon completion of the last exercise/interview, the participant is redirected to the session completion screen (Fig. 8).
When entering the assessment session interface before it starts (Fig. 9), the expert sees the time feed (20), highlighting the moments of interaction with the assessee (21). The time feed may be rescaled. The expert is able to select any event on the time feed, in which case space 26 will show the list of competencies (27, 27A) and their respective indicators (28) assessed in the selected exercise/interview. It is also possible to display on screen the documents related to the selected exercise/interview - fulfillment instructions, participant instructions, interview questions, assignment text, participant's response (which is empty before the exercise begins) and other documents by selecting the relevant point in the heading of space 26 or 30. Before the session starts, the participant's video frame is replaced by the message "Video will appear after the session start" (25). The session control panel (22) comprises two timers - timer from the beginning of the current activity (reset to zero before the session start), timer until the end of the current activity (before the session shows the time remaining until the session start) - and the "Start" and "Postpone" buttons. The session is deemed to have begun after the "Start" is pressed; if the participant has already entered the session, the broadcasting of his/her video stream begins. Pushing the "Postpone" button will postpone the session start time. After the session starts, the expert's interface looks similarly (Fig. 10) except that the message (25) is replaced by the participant's video frame (12) and the "Start" button (23) disappears from the session control panel. The comment field (29) is activated in the space which specifies the competencies being assessed (30). The comments inserted by the expert during the session are available while the results are being processed. When the participant moves on to the interview, the expert's video frame (13) and video stream control buttons such as sound transmission disconnection (33) and video transmission disconnection (34) pop up in the interface. The session control panel (22) shows additional buttons for moving the current exercise to the session end (35) and for early exercise completion (36). When the participant moves on to the next exercise, the expert's interface status becomes similar to the pre-session status (Fig. 9); after the "Start" button is pushed, the expert's interface commences the participant's preparation monitoring (Fig. 10). Upon completion of all exercises/interviews in the session, the expert's interface looks as follows: the session control panel has two buttons— switch to results processing (37) and the saving of grades assigned during the session subsequent processing (38). The participant's video frame is replaced with a static message (25).
STEP 2
The assessment session completion is followed by the assessment results processing and description step. The system offers two methods for arranging the step, namely: the online method and the offline method. In the first case, the results of each participant are processed through the system web interface, in which case the processing of results may involve either one or two (or more) experts, working remotely or otherwise. In case of 2 remote experts, the results processing interface is used as a separable desktop; changes are synchronized automatically without any extra actions from users. It is possible to discuss the results online through video conferencing. The principle of video connection organization video connection organization is similar to the broadcasting of stream video from the expert during the assessment session: video broadcasting may start and end at the expert's sole discretion; however, unlike video conference during the assessment session, video connection in the integration session is available throughout the processing and description of results. In case two (offline processing of results), all data about the participant's progress with exercises, as well as the comments given by the assessing experts may be downloaded for processing to a local application. In addition, experts directly discuss the grade for each competency of each participant in the assessment event, adducing arguments in favor of any given grade, relying on the data stored in or downloaded from the system. The progress of discussion is similar to the convergence of grades after the assessment center. The final grade is recorded in the system through the online convergence interface. The integration session products are grades for competencies and the generated automatic report. An integration session is deemed incomplete unless the final grade is missing at least for one competency.
This invention improves the efficiency, completeness and quality of remote assessment, thus optimizing the time spent idle by participants and ensuring that hardware resources are utilized efficiently.

Claims

1. The employee's competency assessment method, including the following steps:
• Obtainment of the assessment scenario comprising:
■ at least two tasks, in which case
- at least one task is an exercise;
- at least one task is commutation with the expert.
■ order of executing the above-mentioned tasks;
■ data about at least one assessee;
■ data about at least one assessing expert;
• Consecutive execution of tasks according to the assessment scenario, in which case
- if the task being executed is an exercise, video stream and information about the current exercise progress are transmitted from the assessee at least to one expert.
• Processing of the assessment results.
2. The method according to claim 1, wherein the assessee data comprise at least full name and data for authorization in the system, including at least a login and a password;
3. The method according to claim 1 , wherein the assessing expert data comprise at least full name, data about the entity, the assessment system access rights, data for authorization in the system, including at least a login and a password;
4. The method according to claim 1, wherein information about the current exercise progress comprises one of the following elements or their combination:
• text entered in the input field;
• status of control elements;
• status of the exercise being performed.
5. The method according to claim 1, wherein the assessment scenario is received from a remote data storage or a remote server;
6. The method according to claim 1, wherein the assessment scenario is received from a local data storage;
7. The method according to claim 1 , wherein an exercise may be performed in the form of a test, role-playing, analytic case, e-mail emulator;
8. The method according to claim 1, wherein the order for executing tasks may be changed by the expert;
9. The employee's competency assessment method, including the following steps:
• Obtainment of the assessment scenario comprising:
■ at least two tasks, in which case
- at least one task is an exercise;
- at least one task is commutation with the expert.
■ data at least about two assessees;
■ data at least about two assessing experts;
■ order of executing the above-mentioned tasks for each participant;
■ order of carrying out commutations between experts and participants.
• Execution of tasks in the sequence received during the previous step for each participant, in which case
- if the task being executed is an exercise, video stream and information about the current exercise progress are transmitted from the assessee at least to one expert in the order for accomplishing commutations received during the previous step.
• Processing of the assessment results.
10. The method according to claim 9, wherein the assessee data comprise at least full name and data for authorization in the system, including at least a login and a password;
11. The method according to claim 9, wherein the assessing expert data comprise at least full name, data about the entity, the assessment system access rights, data for authorization in the system, including at least a login and a password;
12. The method according to claim 9, wherein information about the current exercise progress comprises one of the following elements or their combination:
• text entered in the input field;
• status of control elements;
• status of the exercise being performed.
13. The method according to claim 9, wherein the assessment scenario is received from a remote data storage or a remote server;
14. The method according to claim 9, wherein the assessment scenario is received from a local data storage;
15. The method according to claim 9, wherein an exercise may be performed in the form of a test, role-playing, analytic case, e-mail emulator;
16. The method according to claim 9, wherein the order for executing tasks may be changed by the expert;
17. The method according to claim 9, wherein the order of exercises to be followed by participants is formed as follows:
a. At least the following is received:
• the list of participants comprising at least two participants;
• the list of tasks comprising at least two tasks;
• the list of experts comprising at least two experts;
b. A task executing schedule is made for each participant;
c. At least one connection is made between each participant and the expert from the expert list for all tasks in the schedule, in which case the schedule must comprise at least two tasks with references to different experts from the expert list.
18. The method according to claim 17, wherein the forming of participants' task executing schedule occurs in two phases:
a. A preliminary sequence of tasks is made, in which case the sequence for execution of tasks by participants in even positions on the list differs from the sequence for participants in odd positions on the list; b. The operator prepares an optimal schedule based on the preliminary sequence of tasks.
19. The method according to claim 9, wherein the preliminary task executing order to be followed by participants is formed according to claim 17 or 18;
20. The employee's competency assessment system, including:
a. one or more data storage devices;
b. one or more processors;
c. tools ensuring network interaction with remote computer systems; d. at least one display;
e. at least one input device;
f. one or more computer programs,
where one or more computer programs are read from one or more data storage devices and run with one or more processors, with one or more programs comprising the following instructions:
■ Obtainment of the assessment scenario comprising:
o at least two tasks, in which case
at least one task is an exercise; at least one task is commutation with the expert.
o order of executing the above-mentioned tasks; o data about at least one assessee;
o data about at least one assessing expert;
■ Consecutive execution of tasks according to the assessment scenario, in which case if the task being executed is an exercise, video stream and information about the current exercise progress are transmitted from the assessee at least to one expert.
■ Processing of the assessment results.
21. The system according to claim 20, wherein the assessee data comprise at least full name and data for authorization in the system, including at least a login and a password;
22. The system according to claim 20, wherein the assessing expert data comprise at least a full name, data about the entity, the assessment system access rights, data for authorization in the system, including at least a login and a password;
23. The system according to claim 20, wherein information about the current exercise progress comprises one of the following elements or their combination:
• text entered in the input field;
• status of control elements;
• status of the exercise being performed.
24. The system according to claim 20, wherein the assessment scenario is received from a remote data storage or a remote server;
25. The system according to claim 20, wherein the assessment scenario is received from a local data storage;
26. The system according to claim 20, wherein an exercise may be performed in the form of a test, role-playing, analytic case, e-mail emulator;
27. The system according to claim 20, wherein the order for executing tasks may be changed by the expert;
28. The employee's competency assessment system, including
a. one or more data storage devices;
b. one or more processors;
c. tools ensuring network interaction with remote computer systems; d. at least one display;
e. at least one input device;
f. one or more computer programs,
where one or more computer programs are read from one or more data storage devices and run with one or more processors, with one or more programs comprising the following instructions:
■ Obtainment of the assessment scenario comprising:
o at least two tasks, in which case
at least one task is an exercise;
at least one task is commutation with the expert.
o data at least about two assessees;
o data at least about two assessing experts;
o order of executing the above-mentioned tasks for each participant;
o order of carrying out commutations between experts and participants.
■ Execution of tasks in the sequence received during the previous step for each participant, in which case
if the task being executed is an exercise, video stream and information about the current exercise progress are transmitted from the assessee at least to one expert in the order of accomplishing commutations that was received during the previous step.
■ Processing of the assessment results.
29. The system according to claim 28, wherein the assessee data comprise at least a full name and data for authorization in the system, including at least a login and a password;
30. The system according to claim 28, wherein the assessing expert data comprise at least a full name, data about the entity, the assessment system access rights, data for authorization in the system, including at least a login and a password;
31. The system according to claim 28, wherein information about the current exercise progress comprises one of the following elements or their combination:
• text entered in the input field;
• status of control elements;
• status of the exercise being performed.
32. The system according to claim 28, wherein the assessment scenario is received from a remote data storage or a remote server;
33. The system according to claim 28, wherein the assessment scenario is received from a local data storage;
34. The system according to claim 28, wherein an exercise may be performed in the form of a test, role-playing, analytic case, e-mail emulator;
35. The system according to claim 28, wherein the order for executing tasks may be changed by the expert;
36. The system according to claim 28, wherein the order for executing tasks may be changed by the expert;
37. The system according to claim 28, wherein the order of exercises to be followed by participants is formed as follows:
a. At least the following is received:
• the list of participants comprising at least two participants;
• the list of tasks comprising at least two tasks;
• the list of experts comprising at least two experts;
b. A task executing schedule is made for each participant;
c. At least one connection is made between each participant and the expert from the expert list for all tasks in the schedule, in which case the schedule must comprise at least two tasks with references to different experts from the expert list.
38. The system according to claim 37, wherein the forming of participants' task executing schedule occurs in two phases:
a. A preliminary sequence of tasks is made, in which case the sequence for execution of tasks by participants in even positions on the list differs from the sequence for participants in odd positions on the list; b. The operator prepares an optimal schedule based on the preliminary sequence of tasks.
PCT/RU2014/000995 2013-12-27 2014-12-26 Method and system of employee competency assessment field of the invention WO2015099581A2 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
EA201400144A EA201400144A1 (en) 2013-12-27 2013-12-27 METHOD AND SYSTEM ESTIMATES OF COMPETENCE OF EMPLOYEES
EA201400144 2013-12-27

Publications (2)

Publication Number Publication Date
WO2015099581A2 true WO2015099581A2 (en) 2015-07-02
WO2015099581A3 WO2015099581A3 (en) 2015-08-20

Family

ID=53479750

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/RU2014/000995 WO2015099581A2 (en) 2013-12-27 2014-12-26 Method and system of employee competency assessment field of the invention

Country Status (2)

Country Link
EA (1) EA201400144A1 (en)
WO (1) WO2015099581A2 (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109634995A (en) * 2018-09-10 2019-04-16 阿里巴巴集团控股有限公司 Main body is assessed to the method, apparatus and server of relationship
CN113034250A (en) * 2021-03-24 2021-06-25 海南大学 Trust value-based crowd sensing incentive mechanism design method

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
TW200423696A (en) * 2003-02-04 2004-11-01 Ginganet Corp Remote interview system
US20080085501A1 (en) * 2006-10-10 2008-04-10 Philadelphia Health & Education Corporation System and methods for interactive assessment of performance and learning
WO2009009820A1 (en) * 2007-07-13 2009-01-22 Flinders University Simulating patient examination and/or assessment

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109634995A (en) * 2018-09-10 2019-04-16 阿里巴巴集团控股有限公司 Main body is assessed to the method, apparatus and server of relationship
CN109634995B (en) * 2018-09-10 2023-06-30 创新先进技术有限公司 Method, device and server for evaluating relationship of main body pairs
CN113034250A (en) * 2021-03-24 2021-06-25 海南大学 Trust value-based crowd sensing incentive mechanism design method

Also Published As

Publication number Publication date
EA201400144A1 (en) 2015-06-30
WO2015099581A3 (en) 2015-08-20

Similar Documents

Publication Publication Date Title
US20030014400A1 (en) System and method for case study instruction
Stefaniak et al. The influence of perceived constraints during needs assessment on design conjecture
US20110318723A1 (en) Studying system and method with virtual cards
Lee et al. Enhancing project-based learning through student and industry engagement in a video-augmented 3-D virtual trade fair
Inge et al. Defining customized employment as an evidence-based practice: The results of a focus group study
US20130109002A1 (en) Educational virtual space
Kuttal et al. Remote pair programming in online cs education: Investigating through a gender lens
Meade et al. Playing with curricular milestones in the educational sandbox: Q-sort results from an internal medicine educational collaborative
WO2015099581A2 (en) Method and system of employee competency assessment field of the invention
Anglum et al. Integrating computer-based curricula in the classroom: Lessons from a blended learning intervention
Marco et al. Drawer: An innovative teaching method for blended learning
Liberatore et al. AIChE Virtual Communities of Practice–Supporting Faculty during the COVID-19 Pandemic
Nurhas et al. An intergenerational competency framework: Competencies for knowledge sustainability and start‐up development in the digital age
Memiah et al. Bridging the gap in implementation science: evaluating a Capacity-Building program in data management, analysis, utilization, and dissemination in low-and middle-income countries
Visco et al. Implementation of a Virtual Job Shadowing Experience for STEM Students Participating in a Corporate-STEM Connection Program
Ipsen et al. Measures of educational effort: what is essential to clinical faculty?
Imroz Interventions: Helping Clients Improve Group Dynamics and Group Results Using Technology
Gentry Group work in political science: how to get collaboration into the classroom
Tiriticco Evaluation and implementation of accessible methods to perform remote moderated usability testing on websites
Anderson et al. Providing the choice of in-person or videoconference attendance in a clinical physiology course may harm learning outcomes for the entire cohort
Slavensky et al. A Quantitative and qualitative analysis of online student retention
Vermeire et al. Digitally inclusive youth work, a blueprint
Coughlan Technology and Communication
Lee et al. Design to Improve Social Presence among University Students in Asynchronous Online Learning Environment
US20180247551A1 (en) Systems and methods for remote collaborative realtime learning

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 14875001

Country of ref document: EP

Kind code of ref document: A2

NENP Non-entry into the national phase in:

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 14875001

Country of ref document: EP

Kind code of ref document: A2