RU2624546C1 - Automatic evaluation method of employees qualification and competence - Google Patents

Abstract

FIELD: information technology.

SUBSTANCE: method includes automatic collection and primary processing of initial data about employees, text data unstructured arrays analysis, the data conversion, the competency model creation on the basis of the obtained factors, the competencies definition available at the employee, the determination of the competence development degree according to the prespecified scale, the employee qualification definition according to the prespecified scale, the competencies applications global fields determination, the competencies applications local fields determination, the consolidated accounting documents issue, based on the analysis performed.

EFFECT: employees` qualifications and competence evaluation accuracy improvement by means of the qualifications and competence evaluation process automation of all the employees data.

4 dwg

Description

The claimed invention relates to the field of information technology, and in particular to methods of analyzing unstructured data, and can be used in the field of personnel analytics to manage competencies in an organization.

The market of personnel analytics systems in the field of personnel management is on the rise - a modern model of personnel management based on competencies has made it possible to develop various types of competency models that are used in the world's leading talent management systems. Algorithms for linguistic data analysis are currently being actively applied in the field of personnel and talent management; their use is able to solve existing problems, such as:

- assessment and prediction of employee productivity;

- identification of the best specialists;

- a group of specialists in skills.

However, currently outdated and subjective methods are often used to assess competencies. The need to improve the quality of assessment of competencies and their application was the motivation for the development of a patented method.

Known document US 2012278713 (IPC G06F 3/048, published 01.11.2012), containing a description of the method and system for assessing competencies, improving the professional level and optimizing the quality of work. This decision relates to the management of competencies in an organization and, more specifically, to the training, assessment and management of individual and collective competencies in an organization using a distributed computer system.

The method includes performing a computer test by at least one user and determining the level of competence based on at least one of the answer options obtained as a result of the computer test. In addition, the answer option includes many answer options, and the method includes calculating the overall score based on the many answer options.

The core competencies of the participant (employee) are assessed using the competency assessment tool in accordance with the competency assessment method (CA 2752404, published March 29, 2012), which includes: the formation of a test competency assessment aimed at choosing the core competency, and including many questions, and, if necessary, the requirement to perform many tasks; assignment of a threshold expert assessment of the participant; providing a test assessment of competency for the first expert in the core competency to obtain the first actual expert assessment of the participant; if the first actual expert assessment of the participant is greater than the threshold value, a validation test of the competency assessment test is performed; if the first actual expert assessment of the participant is equal to or less than the threshold value, a competency assessment test is performed for the second specialist in core competency to obtain a second actual expert assessment of the participant; and if the second actual score of the participant is equal to or less than the threshold value, the competency assessment test is not checked.

In the second analogue, competency assessment is already conducted on a more accurate basis, since a comparison tool has been introduced.

The disadvantages of these analogues can be indicated: insufficient assessment accuracy associated with the incompleteness of the initial information used to assess the competence of the employee, which significantly affects the quality of the assessment and the correctness of the decision. In particular, the analogues do not take into account sources such as the results of an analysis of a text showing the quality of documents created by an employee. In addition, the received reporting documents are limited in application for making decisions in the field of personnel management - on their basis it is impossible to build an exhaustive dossier for an employee and make a development plan for him.

The present invention is the creation of a fully automated way of assessing the qualifications and competencies of employees with the following technical result, namely, improving automation while improving the accuracy of assessing qualifications and competencies of employees.

The problem is solved due to the fact that the method for automatically assessing the qualifications and competencies of employees involves using the module for the automated collection and processing of data on employees to collect and initially process the initial data on employees, including text, numeric and categorical data types from documents, personal cards of employees and any information systems of the organization containing the above types of data about employees, analysis of unstructured arrays of text data based on An ove of linguistic data extraction techniques and machine learning methods that support formalization and knowledge extraction processes, followed by aggregation by employees, conversion of data from personal cards and data of passed professional tests and processed text and metatext data into model factors using the module assessing competencies and areas of application of employee competencies by applying statistical methods.

Then, using the module for automatic assessment of employee qualifications, competencies and their areas of application, they create competency models based on the factors obtained by predefined conditions for determining whether an employee has competencies, determine the competencies that an employee has, according to a competency model by analyzing text documents created by an employee, and information about the employee obtained as a result of converting information into factors of the model for assessing competencies and areas of application of competencies employees, determining the degree of development of competence on a predetermined scale on the basis of data obtained from documents created by the employee, the personal card of the employee and information systems of the organization containing data on the results of the employee passing professional tests, determining the qualifications of the employee on a predetermined scale based on data, obtained from documents created by the employee, the employee’s personal card and organization information systems containing data on the results tats for the employee to pass professional tests, the results of determining the degree of development of the competence of the employee, the definition of global areas of application of competencies, expressed as the names of areas of activity, by classifying texts created by employees, the definition of local areas of application of competences expressed by keywords, by clustering texts created by employees, and counting the number of documents created by an employee within the scope of competencies, drawing up consolidated data reporting documents based on the analysis and their transfer to the module of the recommended content system, the module for information retrieval of employees according to the specified criteria of qualifications and competencies, the module for automatic information retrieval of employees with similar areas of application of competencies, the module for the automatic generation of a catalog of employees based on their competencies, areas of application competencies and the content they create.

Thus, with the entire set of essential features of the invention, an increase in automation is achieved, since all steps of the method are implemented using automated modules and user intervention is completely eliminated.

At the same time, the accuracy of assessing the qualifications and competencies of employees is increased by collecting more initial data, building a competency model and implementing method steps using modules that implement a larger number of analysis and determination of competencies methods, and additionally conducting qualification assessments and assessing the scope of competencies, those. the employee is evaluated by a larger amount of input data and the output is comprehensive results, determining also the area in which the employee can successfully apply it, as well as his general level of qualification.

A dossier, for example, in analogues can only be compiled on the basis of current assessed competencies. In the claimed method, more knowledge about the employee (current assessed competencies and qualifications and areas of application of employee competences, development plan) can be introduced in the dossier, and this simplifies / accelerates decision-making in the field of competency management and human analytics, in general.

The essence of the proposed method and its practical implementation is illustrated by the following schemes and description:

In FIG. 1 presents a diagram of the proposed method, in FIG. 2 presents a block diagram of a system that implements the inventive method, FIG. 3 and 4 are illustrations of reports by examples.

A method for automatically assessing the qualifications and competencies of employees includes the following stages (Fig. 1 and 2).

Automatic collection and initial processing of source data about employees, including textual, numeric and categorical data types from documents, personal cards of employees and any information systems of the organization containing the above types of employee data, is carried out using the automated collection and processing of employee data using technologies for formalizing and extracting data from poorly structured information in the network and database (DB) of the enterprise (hereinafter - ASOD module), is intended nnogo for automated collection and analysis of data about the employee.

The ASOD module receives documents in text formats from various sources, including file storage, internal databases and e-mail. In addition, ASOD provides information about employees from the enterprise information systems and personal cards of employees.

At the output, the ASOD module transmits processed documents (including text, keywords, directions and metadata of the document), as well as aggregated information about employees.

Thus, the ASOD module provides automatic extraction of data from a file storage (personal folders, e-mail, corporate database) and information from internal databases and from existing information systems of the enterprise, as well as data transfer to paired modules.

The ASOD module provides crawling, extraction and indexing of data.

The indexing method provides indexing of employee documents for subsequent full-text search in a system that implements the claimed method and is based on the Apache Solr search engine, which is part of the Apache Lucene library.

Based on the crawling method and its sub-methods, documents created by employees are collected from e-mail, a database and network folders, respectively, and information about employees is collected from their personal cards and enterprise information systems. From documents collected by this method, using the extraction method, text and metadata are extracted.

The crawling method includes a number of sub-methods that provide analysis and collection of information from corporate mail, network folders and corporate databases, respectively:

a) Sub-method of crawling mail

Analysis and collection of documents from mail is carried out using standard libraries JEE. To connect to the IMAP server, the JavaMail API (com.sun.mail.imap) is used to connect to the server via SSL IMAP (Internet Message Access Protocol).

b) Subfolder crawling network folders

The ASOD module, having at the input a list of addresses of network folders of employees, bypasses them at certain intervals and collects all documents that have not been previously collected and, therefore, are not in the database (i.e., duplicates are checked).

c) Corporate database crawling sub-method

The extraction method extracts text from documents collected by the crawler, for example, from text documents of the formats .doc, .docx, .xls, .xlsx., .Ppt, .odt, .txt, .pdf, .rtf. The method is based on the Apache Tika freeware content analysis tool, which allows the extraction of metadata and text from many different text data formats.

Then, using the ASOD module, unstructured text data arrays are analyzed based on linguistic data extraction techniques and machine learning methods that support formalization and knowledge extraction processes, followed by aggregation by employees. Analysis takes place by constructing semantic space and classification using an automatic classifier.

Directly constructing semantic space consists of three stages:

a) Building a term document matrix

The construction of a semantic space is based on the construction of a frequency matrix, which reflects the relationship between the list of lemmas and the corpus of documents in which lemmas occur. In this matrix, rows correspond to lemmas, and columns correspond to documents. Each cell of the matrix contains the weight of the lemma in the document.

b) Singular decomposition

A singular expansion allows one to lower the dimension of the original matrix (approximate the original matrix using a matrix of lower rank).

c) Formation of the result

The result of the singular decomposition are three matrices: one has an r-dimensional vector for each lemma in the corpus, the other contains an r-dimensional vector for each document, the third, dimension r, contains singular values. The first two matrices define two different vector spaces, which also differ from the space defined by the original term-document matrix.

For each lemma and document, its own vector of semantic space is written.

On the resulting semantic space, it is possible to perform semantic search, highlighting keywords and clustering.

The classification method using an automatic classifier ensures the assignment of research areas to documents (for example, physics, chemistry, etc.), as well as the selection of keywords from documents using an automatic classifier using a hierarchical structure of scientific areas.

At the next stage, using the aggregated information conversion module (hereinafter referred to as the PAI module), data from personal cards and data on passed professional tests and processed text and metatext data are converted into factors of the model for assessing the competencies and areas of application of employee competencies by applying statistical methods.

The input data of the PAI module are data from personal cards of employees, their skills, tests passed by them, documents created by them (including text, metadata, directions and keywords).

Part of the data is converted into a text format for data exchange and transferred to the module for automatic assessment of employee qualifications, competencies and their areas of application (hereinafter referred to as the AOKSP module): general length of service; experience at the enterprise; the number of documents created during the work at the enterprise; the number of key indicators met; the level of education; position; department in which the employee works; types of employee texts (R&D report, publication, etc.), the results of all professional tests passed by the employee during his work at the enterprise; directly texts of the employee; Authors of each of the texts (for accounting co-authors); assessment of the quality of the text by other employees. Then this data is converted into a text format for data exchange and transmitted to the AOKKSPK module.

Documents created by an employee are accumulated in such a way that for each employee 3 data arrays are obtained, representing: texts of documents; keywords of documents; directions of documents.

These data arrays are converted to a textual data exchange format and transferred to the AOKKSPK module.

At the next stage, using the AOKKSPK module, competency models are created on the basis of the obtained factors by predefined conditions for determining whether an employee has competence. Then, the competencies that an employee possesses are determined according to the competency model by analyzing text documents created by the employee and information about the employee obtained as a result of converting information into factors of the model for assessing competencies and areas of application of employee competencies, and determine the degree of development of competence according to a predetermined scale on based on data obtained from documents created by the employee, the employee’s personal card and organization information systems containing data on the result x passing employee professional tests.

Assessment of competencies in the AOKKSPK module is as follows. First, the main element of competency assessment is calculated, which is a set of methods and calculations that determine the presence of a certain competence and a preliminary degree of its development. That is, in fact, the identification of competency and an approximate calculation of the degree of its development take place. Then, the modifying element of competency assessment is calculated, which changes for the better or worse the preliminary degree of development of competency obtained by calculating the main element of competency assessment. That is, in essence, the assessment of competency itself takes place.

The described method provides a competency assessment according to the competency assessment formula, which consists of two elements: the main and the modifying one.

For each competency, the main element of competency assessment is different. This element is a combination of methods and calculations that determine the presence of a certain competence and a preliminary degree of its development. The maximum and minimum degrees of development depend on the selected rating scale. The degree of development is specified by a range of values, for example, the maximum degree of development is 5, the minimum is 1.

The modifying element consists of: a modifier of the basic parameters of the employee specified in his profile (work experience, length of service at the enterprise, the fact of taking advanced training courses, etc.); a modifier of professional qualities, which expresses the success of passing specialized professional checks at the enterprise; text analysis modifier, which expresses the quality of the documents created by the employee.

For each numerical parameter used in the formula (for a basic parameter modifier, professional qualities modifier and a modifier derived from the text), the effect of this parameter on the competence of the employee relative to the department in which the employee works (or the entire organization) is calculated. To simplify and speed up calculations and normalize the distribution, all numerical parameters are normalized from 0 to 1.

In general terms, the competency assessment formula looks like:

- оценка компетенции сотрудника,

- предварительная оценка степени развития компетенции основного элемента оценки компетенции,

- модификатор базовых параметров модифицирующего элемента оценки компетенции,

- модификатор профессиональных качеств модифицирующего элемента оценки компетенции,

- модификатор анализа текста модифицирующего элемента оценки компетенции.Where:

- assessment of employee competence,

- a preliminary assessment of the degree of development of competence of the main element of competency assessment,

- a modifier of the basic parameters of the modifying element of competency assessment,

- a modifier of professional qualities of the modifying element of competency assessment,

- text analysis modifier of the modifying element of competency assessment.

- максимальное значение шкалы оценки,

- приведенное значение количества релевантных документов.Where:

- the maximum value of the rating scale,

- the given value of the number of relevant documents.

The formula for calculating the modifier of the basic parameters of the modifying element of competency assessment in general:

- приведенный числовой параметр,

- модификатор важности каждого параметра, по умолчанию равен 1,

- глобальный модификатор важности элемента в формуле. По умолчанию, все элементы формулы равны по важности, поэтому

.Where:

- reduced numerical parameter,

- modifier of the importance of each parameter, the default is 1,

- global modifier of the importance of an element in a formula. By default, all elements of the formula are equal in importance, therefore

.

- приведенный числовой параметр.Where:

- reduced numerical parameter.

- приведенный числовой параметр.Where:

- reduced numerical parameter.

The given numerical parameters are determined as follows.

множества числовых параметров

приведенный числовой параметр

равен:For each numerical parameter

sets of numerical parameters

reduced numerical parameter

is equal to:

- множество всех значении числового параметра по отделу,

- множество всех значений числового параметра по отделу, укладывающихся в рамки среднего значения плюс-минус стандартное отклонение,

- среднее арифметическое значение числового параметра по отделу,

- стандартное отклонение числового параметра по отделу,

- минимальное значение числового параметра по отделу,

- максимальное значение числового параметра по отделу.Where:

- the set of all the values of the numerical parameter in the department,

- the set of all values of the numerical parameter by department, falling within the framework of the average value plus or minus the standard deviation,

- the arithmetic average of the numerical parameter in the department,

- standard deviation of a numerical parameter by department,

- the minimum value of a numerical parameter by department,

- the maximum value of the numerical parameter in the department.

Also, using the AOKKSPK module, the employee’s qualifications are determined on a predefined scale based on data obtained from documents created by the employee, the employee’s personal card and organization information systems containing data on the results of the employee’s passing professional tests, and the results of determining the degree of development of the employee’s competence. Qualification assessment in the AOKKSPK module takes place in two ways: the average total degree of development of all the competencies an employee has is calculated and the degree of employee's compliance with the next position that he can apply for within the organization is calculated.

The average total degree of development of all competencies that an employee has (in percent) reflects the employee's qualifications regarding his current condition. This type of qualification is indicated in the employee dossier:

- степень развития компетенции из множества имеющихся у сотрудника компетенций {rate₁, rate₂, …, rate_n}.Where:

- the degree of development of competence from the set of competencies an employee has {rate ₁ , rate ₂ , ..., rate _n }.

To determine the employee’s degree of conformity to the next position that he can apply for within the organization (in percent), the known parameters that the employee must have in order to apply for a specific position and the employee’s current parameters are compared. This type of qualification is indicated in the employee development plan.

To calculate this type of qualification, two types of parameters are used: competencies, which take into account the presence of certain competencies and the degree of their development, and basic parameters, which take into account, for example, the presence of a certain level of education, the total work experience and work experience at the enterprise.

At the next stage, using the AOKKSPK module, they determine global areas of application of competencies, expressed in the form of names of lines of activity, by classifying texts created by employees. At the same time, the global scope of application of competencies for an employee is formed as a set of directions for all documents created by this employee.

Local areas of application of competencies expressed by keywords are also determined here by clustering texts created by employees and counting the number of documents created by an employee within the scope of application of competencies. The AOKKSPK module converts factors for assessing the scope of competencies and data from the user profile to the scope of competencies, expressed as a list of keywords and documents created by the user that most express the specific scope of the competency. The method for assessing the scope of application of competencies is characterized by the following steps: using the available semantic space of all documents of the organization, based on factors of assessing the scope of application of competencies, a subspace of documents of a particular employee is allocated; The document subspace is clustered using the clustering algorithm.

Based on the analysis, the consolidated reporting documents are prepared in the AOKKSPK module and transferred to the data presentation modules in a textual data exchange format.

The data presentation modules are designed to provide users with relevant information for making decisions in the field of personnel analytics and include a recommendation content system module (hereinafter referred to as the CSM module), an information search module for employees according to the specified qualifications and competencies (hereinafter referred to as the IPCC module), an automatic information search module employees with similar areas of application of competencies (hereinafter referred to as the AIPSPSK module), the module for automatically creating a catalog of employees based on and competencies, scope of competences and their created content (hereinafter - module AFKS).

The CSM module is designed to recommend to employees documents created by other employees that are the most relevant competencies, qualifications and areas of application of employee competencies. The input from the CSM module receives data from the profile of the employee (his competencies, qualifications, areas of application of competencies), content created by the employee, as well as all other documents of the organization. At the output, the RCC module provides content created by other employees that corresponds to the competencies, qualifications and areas of application of the competencies of a particular employee.

The iPSCK module is designed to search for individual employees and documents by information request, as well as to search for groups of employees and the formation of working groups from them. At the input, the iPSCK module receives an unstructured or partially structured text request in natural language, and at the output it gives out, depending on the objectives of the request, relevant documents, competent employees or groups of employees.

AIPSSK module is designed for automatic information retrieval of employees with similar areas of application of competencies and competencies. The AIPSPSK module accepts input from the employee’s profile (including the competencies of the employee, his qualifications, the scope of the competencies, as well as an individual development plan) and content created by the employee. At the output, the AIPSPK module gives profiles of employees with a similar set of skills, areas of application of competencies, and whose skills correspond to the individual development plan of a particular employee.

The AFKS module is designed to automatically combine employees with similar professional qualities into a catalog. The AFKS module takes a lot of personal parameters of employees (competencies, qualifications, areas of application of competencies and other parameters from profiles), as well as content created by employees in the form of a semantic space (for example, LSA). At the output, the AFKS module provides a catalog of employees automatically generated on the basis of competencies, qualifications and areas of application of competencies.

Examples of the implementation of the proposed method.

Previously, methods for determining the presence of competence of an employee of the following competencies and a scale for the degree of their development from 1 to 5 were set:

- Academic writing,

- Work with intellectual property

- Analytics

- Software Development

- Software Testing

Example 1

All documents created by an employee of the company Ivanov Ivan Ivanovich were automatically extracted from the enterprise database.

Documents were extracted:

Publications:

1. "The form of resolving structural imbalances"

2. "Financial institutions in a globalizing economy"

3. “Innovation and investment system of Russia as an institutional basis for economic growth”

Also, his personal data was extracted from his personal card:

1. Department - research department,

2. Position - Researcher,

3. Education - PhD,

4. Work experience at the enterprise - 3 years,

5. The total length of service is 4 years.

From the organization’s information system, data on professional tests passed by him were extracted:

He passed 2 tests according to his position and education, the results of which were (6, 8, respectively) on a ten-point scale.

Test 1: Practice Patent Search

Test 2: Theoretical Foundations of the Investment System in Russia

Documents were processed in accordance with the claimed method.

The result was reporting documents containing certain competencies and the degrees of their development:

Academic Writing (3.0)

Work with intellectual property (3.0)

Global areas of application of competencies: History, Sociology, Economics.

His qualification level was determined: 60.0%

Its main local areas of application of competencies were identified, expressed by keywords:

1. Keywords: main direction of regulation, employment, theoretical basis of relations, crisis, labor, labor market, crisis condition, market development, exit factor, state.

2. Keywords: innovative investment system, innovative investment activity, the basis of economic growth, the basis of new quality, the main trend, economic growth, formation factor, theoretical foundations.

In FIG. Figure 3 shows an illustration of the report for this employee.

Example 2

All documents created by company employee Petrov Ivan Sidorovich were automatically extracted from the enterprise database.

Documents were extracted:

Analytical documents:

1. “What will the agreement between Russia and Norway lead to?”

2. "Analysis of corruption in the field of business"

Also, his personal data was extracted from his personal card:

1. Department - analytics department,

2. Position - Analyst

3. Education - Higher

4. Work experience at the enterprise - 3 years,

5. The total length of service is 4 years.

From the organization’s information system, data on professional tests passed by him were extracted:

He passed 1 test according to his position and education, the results of which amounted to 5 on a ten-point scale.

Test 1: Practice Writing Scientific Articles

The result was reporting documents containing certain competencies and the degrees of their development:

Academic Writing (2.0)

Analytics (2.5)

Global areas of application of competencies: Sociology, Economics.

His qualification level was determined: 45.0%

Its main local areas of application of competencies were identified, expressed by keywords:

Keywords: developed foreign country, corruption, procurement, experience, methodological principle, activity trend, organization, research material, production, development.

In FIG. Figure 4 shows an illustration of the report for this employee.

Claims (1)

A method for automatically assessing the qualifications and competencies of employees involves using the module for the automated collection and processing of data on employees to collect and initially process the initial data on employees, including textual, numeric and categorical data types from documents, personal cards of employees and any information systems of the organization containing the above types of employee data, analysis of unstructured text data arrays based on linguistic data extraction techniques x and machine learning methods that support formalization and knowledge extraction processes, followed by aggregation by employees, conversion of data from personal cards and data of passed professional tests and processed text and metatext data into factors of competency and model assessment using the module for converting aggregated information applying employee competencies by applying statistical methods, then using the module for automatic assessment of qualifications, competencies and their areas employee’s applications create competency models based on the factors obtained by pre-defined conditions for determining whether an employee has competencies, determine the competences that an employee has, according to a competency model by analyzing text documents created by an employee and information about an employee obtained by converting information into factors models for assessing competencies and areas of application of competences of employees, determining the degree of development of competence according to a preliminary task on a given scale based on data obtained from documents created by an employee, an employee’s personal card and information systems of the organization containing data on the results of an employee passing professional tests, determining an employee’s qualifications according to a predefined scale based on data obtained from documents created by an employee, a personal card employee and organization information systems containing data on the results of the employee passing professional tests, determination results the degree of development of employee competencies, the definition of global areas of application of competencies, expressed in the form of names of areas of activity, by classifying texts created by employees, the definition of local areas of application of competencies, expressed by keywords, by clustering texts created by employees and counting the number of documents created by an employee in within the scope of competencies, the preparation of consolidated reporting documents on the basis of the analysis and their transfer to the module p komendatelnoy content systems, information retrieval unit employees on the specified criteria of qualifications and competences, a module of automatic information retrieval employees with similar applications of competencies, module automatic generation of a directory of employees based on their competencies, scope of competences and their created content.

Images