RU2733485C1 - System and method of processing data for integrated assessment of scientific and technological project maturity based on the use of a set of parameters - Google Patents

Abstract

FIELD: computer equipment.

SUBSTANCE: group of inventions relates to computer engineering. Group of inventions is developing a new solution relating to assessment of scientific and technological projects for a wide range of scientific and technical disciplines, development of medical products and medical products, information and communication technologies and software development, as well as industrial production to assess their maturity in terms of readiness of their results for commercialization and industrial implementation.

EFFECT: technical result consists in enabling processing of heterogeneous data relating to research and development in scientific and technological field, information and communication technologies and software development, as well as in production of medicines and medical products and in industrial production, which enables to evaluate the analyzed projects and perform ranging of the obtained results.

14 cl, 8 dwg, 8 tbl

Description

FIELD OF TECHNOLOGY

The invention relates to computer technology and, in particular, to systems and methods for evaluating scientific and technological projects used to identify the current state of such projects in terms of the readiness of the results obtained during their implementation for commercialization and practical application in real sectors of the economy according to the value of the recruited points according to assessed criteria and building a rating for several assessed projects in ascending or descending order of the points scored.

An invention is a toolkit that provides support in making managerial and expert decisions on financing a project in a wide range of disciplines, such as, for example, industrial production, applied scientific and experimental development, information and communication technologies and software development, as well as in the development of medicines and medical devices. The invention can be implemented using computer technology.

The term "project maturity" (from the English. "Project maturity))) is now largely in common use in the art; for the purposes of this application, it can be understood as "project readiness", "project completion stage".

LEVEL OF TECHNOLOGY

From the prior art, technical solutions are known that provide an assessment of the level of maturity of projects based on the study of various information inherent in these projects, as well as solutions that provide an assessment of the level of maturity of technologies developed during the implementation of scientific and technological projects in terms of the level of its technological readiness, i.e. .e. determination of the position of the technology level on the time scale describing the process of its development, and, accordingly, determining the degree of readiness of the developed technology for industrial implementation and use.

For example, in modern practice, methods based on quantitative and qualitative assessments of indicators describing the technical aspects of a technology, the integration of its components and its final integration into the system and the environment are widely used to assess the level of readiness of a technology (technological readiness). The choice of indicators (criteria) and their assessment can be carried out taking into account the definitions of international standards based on the use of rules for describing the states inherent in the attributes of technology and their correlation with the technology being developed. However, such methods do not allow for a comprehensive assessment of the maturity of scientific and technological projects for a whole set of indicators that can describe the project, when making management decisions. If we talk about the assessment of projects in general, then, in modern practice, methods are currently used to assess specific information about projects related to narrowly focused areas of production and research disciplines, i.e. practically for each of these areas, methods are used that have features that do not allow them to be extended to other areas of science and technology.

In this regard, the task associated with the development of an algorithm for the process of carrying out the mentioned comprehensive assessment of maturity, including both the current state and readiness for the whole, scientific and technological projects for a wide range of disciplines based on the use of parameters describing the balanced development of projects, is urgent. with obtaining an objective result of the assessment and ensuring its visual presentation, as well as the possibility of ranking projects, which is solved in the claimed invention.

From the prior art, a solution is known, presented in the patent for invention US 10241786 B2 (publication US 20180210728, 07.26.2018) "Evaluating Project Maturity from Data Sources" (2019), aimed at building an automatic system for assessing the level of maturity of projects in the field of software development - libraries of programs and services. Evaluation is based on the study of a set of artifacts, which may include documents, source code repositories, tasks in the monitoring and bug tracking system, as well as information from other sources about the project being executed, for example, from sources located on the Internet, such as, for example, like discussion portals, blogs, etc. Note that the invention describes an evaluation method in which it is proposed to establish for each project its own individual set of artifacts based on typical templates developed in advance and compiled using software engineering standards. The scoring system of this invention establishes that the project data is sufficient at a given point in time to validate artifacts associated with different levels of project maturity, and the invention does not provide a method for classifying maturity levels. The vector of artifacts is preliminarily projected at the input onto several analytical systems that evaluate them, which assign weights to various artifacts based on the use of methods for processing data on a variety of projects, after which the final assessment of the set of artifacts is carried out and the level of maturity of the project is determined using a set of decision rules. The mechanism for classifying artifacts and constructing decision rules for this assessment system is based on the principles of machine learning, the source of information for this process is a database, into which information on all projects is entered for the subsequent creation of decision rules. The scoring system is customizable to the requirements of end users, which suggests that the mechanisms for evaluating the same projects by different users can lead to different results.

The claimed invention, in contrast to US 10241786 B2, allows you to create systems for evaluating projects related to a significantly larger range of scientific and technical disciplines than the development of software libraries and services, in addition, the claimed invention, although it is associated with the study of scientific and technological projects for sets of information about them, however, is characterized by its set of essential features - they contain a different sequence of actions and a completely different functional purpose composition of interconnected blocks and a different set of estimated parameters that best describes the balanced development of the project in terms of the practical application of its results.

In particular, the claimed invention implements a different algorithm for processing and interpreting data on scientific and technological projects according to certain parameters proposed by the authors of the invention, which differ from the known solution.

So, for example, in the claimed invention, in addition to project data, which are formed not in the form of a vector, but in the form of a multidimensional hierarchical structure, not only the data itself is used directly, but also an assessment of their quality, obtained on the basis of processing expert conclusions about the project.

The sequence of actions implemented by the inventive system also differs from the above-described known solution. In particular, in US 10241786 B2, after the construction of the vector of artifacts, there is a stage of constructing a weight vector for each set, after evaluating which a decision is made based on decision rules about the readiness of the project under consideration for completion. In the declared decision, after collecting data on the project and examining expert opinions, the processing of this information is based on a different sequence of actions. An essential feature of the claimed system is that it allows you to measure a specific parameter - a value that characterizes the degree of maturity of the project, as well as the readiness of its results for use in the real sector of the economy, which can be classified according to the scale proposed in the claimed invention. At the same time, the improvements made make it possible to increase the accuracy of a comprehensive assessment of the level of the current state of the project or its readiness for commercialization due to the presented set of blocks with endowed functionality and connections between them.

In addition, in the claimed solution, the results of the assessment are other quantitative parameters, for example, the project maturity index, which characterizes the dynamics of the development of a scientific and technological project at short time intervals, and not necessarily coinciding with the WBS control points (from the English Work Breakdown Structure - decomposition structure works, in particular, the work schedule). The claimed invention implements the functionality of ranking projects based on the maturity index values for scientific and technological projects for an unlimited range of scientific and technical disciplines in which they are performed, incl. research and applied projects, projects in the field of software development and the creation of automated, information systems and in the field of robotics, as well as in the development of medicines and medical devices.

Currently, for the practical assessment of technology, the so-called. TRL scale (Technology Readiness Level), which includes 9 levels - from the first, when the technology passes the stage from fundamental to applied research, to the ninth, which refers to the stage of actual application of the technology in a real environment and operating conditions. Typically, the task of determining the level of readiness of a technology is solved by collecting raw data related to the technology and the subsequent application of the so-called. TRL calculators, which are computing systems built on the basis of an Excel spreadsheet processor. In modern practice, technical solutions for creating systems for assessing the readiness of a technology not only as such directly, but also from the point of view of its integration into real systems, are associated with the construction of systems that somehow optimize the data processing process, data storage methods and increase the efficiency of computations.

For example, a solution is known from the prior art as presented in the invention CN 102890753 A (publication CN 102890753 A, 01/23/2013) technology readiness level (TRL) determination method based on technology readiness attribute)) (2012). The method proposed in the invention is based on the use of definitions of various levels of TRL accepted in international standards. The definition of the TRL is done using attribute data that is specific to the TRL. The method explores attributes related to the technology itself, its integration with the system and the environment. In order to determine the compliance of the attributes with the specified criteria, questionnaires (checklists) are developed, the data from which are then subjected to statistical processing to establish parameters that can be used in assessing technological maturity. The method proposed in this invention allows changes to be made to standard (typical) assessment checklists based on the examination of expert opinions for a variety of evaluated projects.

The claimed invention allows, in contrast to CN 102890753 A, to evaluate a significantly larger set of project attributes, which are combined into groups such as

- a group of project attributes that determine the technological readiness of the project (TR - Technology Readiness);

- a group of project attributes that determine the production readiness of the project (MR, from the English Manufacturing Readiness);

- a group of project attributes that determine the engineering readiness of the project (ER, from the English. Engineering Readiness);

- a group of project attributes that determine the readiness of the organizational support of the project (OR, from the English Organizational Readiness);

- a group of project attributes that determine the benefits of the project and mitigate the risks of the project (SR, from English Superiority Readiness - to achieve benefits, it is necessary to minimize risks);

- a group of project attributes that determine the market readiness of the project, i.e. readiness of the project results for commercialization and use in the real sector of the economy (CR, from the English Commercial Readiness).

Unlike CN 102890753 A, the claimed invention allows the attributes of the aforementioned degrees of readiness to be assessed on a continuous rating scale from 0 to 1, and not only on the basis of the use of checklists assuming a binary rating system of 0 or 1.

Unlike CN 102890753 A, the claimed invention allows you to make changes not only to the assessment questionnaire templates for various projects, but also to the assessment questionnaire for a specific project, which is assessed by an expert, dynamically, without affecting the library of expert questionnaire templates (checklists).

The prior art knows the invention CN 103226743 B (publication CN 103226743 B, 13.07.2016) "Aviation equipment and technology maturity assessment trl information processing method based on" (2012), aimed at building a computer-based computing system for assessing the maturity of technologies in the field of aviation equipment development ... It describes the hierarchical structure of an information base containing information about aircraft models and the values of the TRL levels they possess. The invention proposes a system based on the division of access levels to it, which, based on the user's request, and depending on his rights, either allows editing questionnaire templates for assessment, or only directly evaluates. This invention describes a method for adding assessment criteria, modifying and removing them, which can now be set not only from the standard definitions of TRL levels and their attributes, but also from the WBS project.

In contrast to CN 103226743 B, the claimed invention expands the scope of the developed assessment method, i.e. can be used for a significantly wider range of scientific, technical and industrial disciplines, and also determine not only the level of maturity of the project, but also such levels of readiness as TR, MR, CR, SR, OR, ER. In addition, in the claimed invention, the evaluation criteria are established on the basis of personal long-term practice of expert evaluation of scientific and technological projects, experience in their implementation, as well as on the basis of studying GOST and various practices of project activities, therefore, along with procedures for modifying, removing or adding criteria assessment as such, procedures are used to add or change individual indicators of these assessment criteria, and these procedures can be carried out both at the stage of creating an expert questionnaire template for project assessment, and directly at any time during the project assessment, which distinguishes the claimed invention from CN 103226743 B. It is also the fact that the claimed invention, in addition to the possibilities of editing questionnaires and conducting an assessment, provides for feedback with project executors, i.e. formation of a set of recommendations for them to improve the efficiency of work performance.

The proposed invention allows to overcome at least part of the above disadvantages or all of these disadvantages, as well as to realize the advantages of the present invention, as described in the framework of the present invention.

SUMMARY OF THE INVENTION

The objective of the present invention is to develop a new solution regarding the assessment of scientific and technological projects for a wide range of scientific and technical disciplines, the development of drugs and medical devices, information and communication technologies and software development, as well as industrial production in order to assess their maturity in terms of the readiness of their results for commercialization and industrial implementation, ensuring, incl. increasing the objectivity and reliability of expert assessment in making management decisions by the management bodies of the relevant companies and departments within the framework of their business processes for the administration of research and development in the scientific and technological sphere.

The technical result achieved by the proposed invention consists in providing the possibility of processing heterogeneous data related to research and development in the scientific and technological sphere, information and communication technologies and software development, as well as in the production of drugs and medical devices and in industrial production, which allows evaluate the projects under study and rank the results obtained. In addition, the claimed invention makes it possible to increase the performance of the system when solving the problem (i.e., it allows processing to obtain a result (product) in less time). In addition, the measurement for each of the evaluated projects of the value characterizing the level of maturity of the project as the degree of its readiness is based on a dynamically replenished dataset, has incremental properties, and, therefore, requires a minimum number of requests, therefore, less machine time and resources for processing and getting the result, including RAM. In addition, the change in the physical state of the processing unit associated with the addition of new functionality makes it possible to run the system fewer times to obtain a result and, therefore, save energy consumed by this system. In addition, a decrease in the number of processed requests (due to the use of functionality that provides incremental processing of the information data array about the project according to a specific algorithm using target criteria) also reduces the likelihood of an error affecting obtaining a more accurate and reliable result. Thus, the claimed invention provides an objective and reliable quantitative assessment (in points) that characterizes the maturity index of the project through the use of tools that take into account a certain set of criteria significant for a given task and their gradation.

и позволяющей при оценке сделать вывод о том, что результат выполнения работы удовлетворяет ранее заданным требованиям, с пороговой величиной W_порог, где свидетельства E_U характеризуются задаваемым весовым коэффициентом W(E_U), величина которого характеризует вклад каждого свидетельства E_U для принятия утвердительного решения о получении результата, соответствующего требуемому качеству, а также относительную важность свидетельств получения требуемых результатов, причем если полученный результат соответствует установленным требованиям, то E_U=W(E_U), если же качество какого-либо из оцениваемых свидетельств не соответствует заданным требованиям, то значение E_U приравнивается величине W(E_U), умноженной на поправочный коэффициент Q(E_U), причем если информация о свидетельстве в материалах проекта отсутствует, то значение Wu умножается на поправочный коэффициент, значение которого равно нулю, а если качество представленного свидетельства удовлетворяет требованиям, то значение W_U умножается на поправочный коэффициент, значение которого равно единице; осуществляется корректировка значений координат вектора достигнутых состояний с использованием матрицы взаимосвязи ММ, элементы которой определяют взаимосвязь между состояниями заданного уровня зрелости проекта с использованием данных требований нормативных документов и стандартов, предъявляемых к процессам выполнения инновационных проектов, причем каждый элемент матрицы устанавливается на основе правила MM_I,J=l/R, если состояние S_I связано с состоянием S_J, и ноль, если нет, причем значение R равно количеству всех связей состояния S_I с остальными состояниями, в том числе самим собой, и для корректировки значения вектора состояний А он умножается справа на матрицу ММ по правилам матричного умножения, причем если в итоге каждая компонента вектора S равна 1, то текущий уровень зрелости проекта TPML в данный момент времени достигнут, а если он совпадает с прогнозным значением, установленным в контракте или соглашении на выполнение проекта, то проект выполняется в соответствии с установленными требованиями и не требует принятия мер оперативного реагирования; осуществляется вычисление значения уровня зрелости проекта, TPML_L, с использованием декомпозиции процесса выполнения проекта как системы состояний, задач, работ и свидетельств получения результатов работ в процессе выполнения проекта, где каждому уровню зрелости L проекта соответствует набор состояний и соответствующие им количественные значения L|S={S_I, I=1,2,…N_L}, причем значения S_I равны единице («1»), если состояния достигнуты, и нулю («0»), если нет, и для достижения указанных состояний исполнителями проекта должны быть решены соответствующие задачи, которым также ставятся в соответствие количественные значения, по крайней мере, для показателя S_I|Т={TJ, J=1,2,…,NS_I}, причем значения T_J равны единице, если соответствующая задача решена, и - нулю, если нет, и каждая задача разбивается на ряд работ, направленных на достижение соответствующей готовности проекта - TR, MR, ER, OR, SR, CR, которым также ставятся в соответствие количественные значения, по крайней мере, для задачи T_J|A={A_M, M=1,2,…,N_TJ}, причем значения A_M равны единице, если работа выполнена, и - нулю, если нет, причем итогом выполнения этих работ являются результаты, которым также ставятся в соответствие количественные значения, по крайней мере, для работы A_M|Е={E_U, U=1,2,…,N_AM}, а доказательства получения результатов являются описания и свидетельства в совокупности с фактами, E_U, которые представляются исполнителями проекта в составе документов о ходе выполнения проекта, где значениям E_U также ставятся в соответствие количественные величины, причем такие данные о полученных результатах являются набором оцениваемых свидетельств и значение E_U зависит от того, каково качество содержащихся в проектной документации E_U|D={D_Y, Y=1,2,…,N_EU} подтверждающих сведений об этом свидетельстве, причем: значение уровня зрелости проекта TPML_L равно уровню зрелости проекта, L, если каждое вычисленное значение достигнутого состояния уровня зрелости проекта, S_I, равно единице, или равно значению L минус единица в противном случае; осуществляется вычисление значения индекса зрелости проекта, причем: осуществляется вычисление общего количества всех состояний N уровня зрелости, как достигнутых - NN, так и не достигнутых, как сумма всех состояний N уровня зрелости; осуществляется расчет значения индекса зрелости проекта I_M проекта по формуле I_M=TPML_L - 1 + NN/N; осуществляется вычисление значений критериев K_TR, K_MR, K_ER, K_OR, K_SR, K_CR, определяющих полноту выполнения работ по проекту, описывающих в совокупности сбалансированное развитие проекта и направленных на достижение технологической готовности, TR; производственной готовности, MR; инженерной готовности, ER; готовности организационного обеспечения проекта, OR; работ по обеспечению преимуществ проекта и сглаживанию рисков проекта, SR; работ по определению рыночной готовности и коммерциализации результатов проекта, CR; на определенном уровне зрелости в момент проведения оценки уровня зрелости проекта, причем: осуществляется вычисление значений NN_TR, NN_MR, NN_ER, NN_OR, NN_SR, NN_CR, определяющих количество реально выполненных работ, рассчитанное по координатам скорректированного вектора достигнутых состояний S, для чего сначала осуществляется корректировка значений T_J, где если значение S_I равно единице, то все связанные с ним значения T_J равны единице, а если значение SI равно нулю, то все связанные с ним значения T_J равны нулю; осуществляется корректировка значений A_M, где если значение T_J равно единице, то все значения A_M, ассоциированные с ним, устанавливаются равными единице, а если значение T_J равно нулю, то все значения A_M, ассоциированные с ним, устанавливаются равными нулю; для вычисления полноты выполнения работ на определенном уровне зрелости проекта, относящихся к одной из групп параметров проекта, осуществляется вычисление количества элементов A_M, значение которых равно 1 - (NN_TR, NN_MR, NN_ER, NN_OR, NN_SR, NN_CR), и осуществляется вычисление общего количества полученных элементов A_M - (N_TR, N_MR, N_ER, N_OR, N_SR, N_CR); осуществляется вычисление значений K_TR, K_MR, K_ER, K_OR, K_SR, K_CR как результата деления NN_TR, NN_MR, NN_ER, NN_OR, NN_SR, NN_CR на N_TR, N_MR, N_ER, N_OR, N_SR, N_CR соответственно.According to one of the implementation options, a method is proposed for a comprehensive assessment of the maturity of a scientific and technological project, in which, based on the use of a set of parameters: the formation of an array of information is carried out using the data contained in the requirements of regulatory documents and standards for the project execution processes, and identifying and characterizing the levels of maturity of projects and including a basic set of data, including a basic list of states, a basic list of tasks, a basic list of works, a basic list of evidence confirming the receipt of results of work with the required quality by adding parameters that identify and characterize the levels of maturity of projects; the formation of an array of information is carried out containing the values of quantities characterizing the weights of each of the evidence for making a decision that the work has been completed in full with the required quality, and the values characterize the importance of each of the assessed evidence in comparison with other evidence; the formation of templates of expert questionnaires from these data arrays of information is carried out; a data set is formed from the data contained in the project documents; the parameters of the initial and final values of the estimated parameters and criteria of the project maturity level are set, the timeline with the control points located on it and the predicted values of the project maturity index, the forecast of their change over the project execution time, which characterize the project, from the data of the generated data set, where the forecast is the calculation of the parameters of the project on the timeline with the establishment of the fact whether the forecast is justified or not; the quality of the facts extracted from the assessed documents is assessed using the specified values of the assessment of the facts and the data of the documents of the project executors to assess the evidence confirming the receipt of the results, while the obtained assessment corresponds to the numerical value of the degree of confidence at which the evidence confirms the presence of the result in the project materials, for description of which it is provided by the project executor as part of the project documentation; the calculated vector of the achieved states of the project maturity level is calculated, where the adjusted vector of the achieved states of the maturity level is formed after correcting the result of calculating the states using the relationship matrix, and: the value of the achieved state of the project maturity level is calculated, S _I for each of the projects by multiplying the quantitative values of the indicators T _J for each task, each of which characterizes the result of its solution and is calculated by multiplying all values A _{M of the} results of work in order to determine the completeness of the solution to the problem, where the index M takes values from 1 to N _TJ , and the value of A _M is equal to one for the work performed A _M , if the value of W _{A is} greater than or equal to the value of W threshold, or for uncompleted work A _M is equal to zero, if the value of W _{A is} less than the threshold value W _threshold , for works from the list of works T _J | A = {A _M , M = 1,2, ..., N _TJ }, aimed at achieving the corresponding appropriate readiness of the project - TR, MR, ER, OR, SR, CR, where TR - technological readiness, MR production readiness, ER engineering readiness, OR - readiness of organizational support of the project, SR - works to ensure project benefits and mitigate project risks, CR - works to determine market readiness and commercialization of project results, at a certain level of maturity L of the project, associated with the indicator T _{J of the} task, and evidence from the list of evidence of the results of their implementation A _M | E = {E _U , U = 1,2, ... , N _AM } specified in the documents submitted by the project executor to establish the fact of the work performed, where the quantitative value of the work performance is established for the entire set of evidence included in it based on the comparison of the W _A value calculated by the formula

and allowing, during the assessment, to conclude that the result of the work fulfills the previously specified requirements, with a threshold value W _threshold , where the evidence E _{U is} characterized by a given weighting factor W (E _U ), the value of which characterizes the contribution of each evidence E _U for making an affirmative decision on obtaining a result corresponding to the required quality, as well as the relative importance of evidence of obtaining the required results, and if the result obtained meets the established requirements, then E _U = W (E _U ), if the quality of any of the assessed evidence does not meet the specified requirements, then the value of E _{U is} equal to the value of W (E _U ) multiplied by the correction factor Q (E _U ), and if there is no information about the certificate in the project materials, then the value of Wu is multiplied by the correction factor, the value of which is zero, and if the quality of the submitted certificate satisfies requirements, then the value W _U multiplied by a correction factor, the value of which is equal to one; the values of the coordinates of the vector of achieved states are adjusted using the MM interrelation matrix, the elements of which determine the interrelation between the states of a given level of project maturity using these requirements of regulatory documents and standards for the processes of implementing innovative projects, and each element of the matrix is set based on the MM _I rule _{, J} = l / R, if the state S _I is associated with the state S _J , and zero, if not, and the value of R is equal to the number of all connections of the state S _I with other states, including itself, and to correct the value of the state vector A it is multiplied on the right by the MM matrix according to the rules of matrix multiplication, and if in the end each component of the vector S is equal to 1, then the current maturity level of the TPML project at this point in time is reached, and if it coincides with the forecast value established in the contract or agreement for the implementation of the project, then the project is carried out in accordance with the established requirements and does not require a prompt response; the value of the project maturity level, TPML _L , is calculated using the decomposition of the project execution process as a system of states, tasks, works and evidence of the receipt of work results during the project execution, where each maturity level L of the project corresponds to a set of states and the corresponding quantitative values L | S = {S _I , I = 1,2, ... N _L }, and the S _I values are equal to one ("1") if the states are reached, and zero ("0"), if not, and to achieve the indicated states by the project executors the corresponding tasks must be solved, which are also assigned quantitative values, at least for the indicator S _I | Т = {TJ, J = 1,2, ..., NS _I }, and the T _J values are equal to one if the corresponding problem solved, and - to zero, if not, and each task is divided into a series of activities aimed at achieving the appropriate readiness of the project - TR, MR, ER, OR, SR, CR, which are also assigned quantitative values, at least for the achie T _J | A = {A _M , M = 1,2, ..., N _TJ }, and the values of A _M are equal to one if the work is done, and - zero if not, and the result of these works are the results, which are also quantitative values are matched, at least for the work A _M | E = {E _U , U = 1,2, ..., N _AM }, and the evidence of obtaining the results are descriptions and evidence in combination with facts, E _U , which are presented by the project executors as part of the project progress documents, where the E _U values are also assigned quantitative values, and such data on the results obtained are a set of assessed evidence and the E _U value depends on the quality of the E _U | D contained in the project documentation = {D _Y , Y = 1,2, ..., N _EU } of supporting information about this certificate, moreover: the value of the maturity level of the project TPML _L is equal to the level of maturity of the project, L, if each calculated value of the achieved state of the level of maturity of the project, S _I , equal to one, or pa the value of L minus one otherwise; the value of the project maturity index is calculated, and: the total number of all states of the N maturity level is calculated, both achieved - NN, and not achieved, as the sum of all states of the N maturity level; the value of the maturity index of the project I _{M of the} project is calculated according to the formula I _M = TPML _L - 1 + NN / N; the calculation of the values of the criteria K _TR , K _MR , K _ER , K _OR , K _SR , K _CR , determining the completeness of the work on the project, describing in the aggregate the balanced development of the project and aimed at achieving technological readiness, TR; production readiness, MR; engineering readiness, ER; readiness of organizational support of the project, OR; work to ensure project benefits and mitigate project risks, SR; work to determine market readiness and commercialization of project results, CR; at a certain level of maturity at the time of assessing the level of maturity of the project, and: the values of NN _TR , NN _MR , NN _ER , NN _OR , NN _SR , NN _CR are calculated, which determine the number of actually performed work, calculated by the coordinates of the adjusted vector of achieved states S, for which, first, the T _J values are adjusted, where if the S _I value is equal to one, then all associated T _J values are equal to one, and if the SI value is zero, then all the associated T _J values are zero; adjusting the A _M values, where if the T _J value is equal to one, then all the A _M values associated with it are set equal to one, and if the T _J value is zero, then all the A _M values associated with it are set equal to zero; to calculate the completeness of work at a certain level of maturity of the project related to one of the groups of project parameters, the number of elements A _M is calculated, the value of which is 1 - (NN _TR , NN _MR , NN _ER , NN _OR , NN _SR , NN _CR ) , and calculating the total number of received elements A _M - (N _TR , N _MR , N _ER , N _OR , N _SR , N _CR ); the values K _TR , K _MR , K _ER , K _OR , K _SR , K _CR are calculated as a result of dividing NN _TR , NN _MR , NN _ER , NN _OR , NN _SR , NN _CR by N _TR , N _MR , N _ER , N _OR , N _SR , N _CR, respectively.

In one of the private implementation options, the initial and final values of the estimated parameters and criteria for the level of maturity of the project are formed by the current level of maturity and from the project data contained in the bids, explanatory notes and project references.

In one of the private implementation options, an additional set of assessed attributes is formed, which are set by the governing body for a project or a group of projects to change project templates or by an expert to evaluate a project to change the questionnaire.

In one of the private implementation options, when conducting an expert assessment of the quality of the certificates provided by the contractor about the results obtained during the project, the project data is used, while the expert questionnaire is changed by changing or expanding the basic list of states of the maturity level, the basic list of tasks, the basic list of works, a basic list of certificates of receipt of the results of work, as well as a basic list of documents that must contain certificates.

In one of the private implementations, values of quantities characterizing the weights of each of the certificates are added for the changed or added to the list of work results in order to make an affirmative decision on the performance of work with the required quality.

In one of the particular variants of implementation, the assessment of the quality of the facts extracted from the assessed documents includes an assessment of the quality of the facts about the results obtained during the implementation of the project in the format of a questionnaire.

In one of the private implementation options, the results of processing the results of calculations for the project are presented in one of the well-known formats of presenting data to the user, including the results of ranking projects, indices of maturity of projects, the dynamics of changes in the index of maturity of the project, including in comparison with the predicted degree of completion characterizing the completeness of the project works associated with project parameters TR, MR, ER, OR, SR, CR.

In one of the private implementation options, a set of recommendations for project executors is formed, and if during the mentioned processing it was found that the development of the project lags behind the planned indicators, then recommendations are formed to the project executors aimed at increasing the efficiency of the work they perform, containing the calculated quantitative values of the evaluated criteria maturity of the project.

In one of the particular implementations, if the TPML _L value is not equal to the L value, then the calculation of the adjusted vector of the achieved states of the project maturity level is repeated, and if the TPML _L value is equal to the L value, then the fact that the plan coincides with the fact at a given time is established.

In one particular embodiment if the value I _M project the index of maturity is less than the forecasted value is performed index of the maturity of the project I _plan to develop management solutions and if the value of the maturity index Project I _M is greater than or equal to the forecast of the maturity index project I _plan is established fact normal development of the project.

In one of the particular implementation options, the project maturity index I _M is determined by the formula I _M = TPML _L - 1 + NN / N, where N is the dimension of the vector S, and the value of the project maturity index I _M determines the dynamics of the project at different time intervals of the project within the same level of maturity of the project, and, if its value coincides or exceeds the predicted value at a certain point in the time of the project implementation, established in the documents for the implementation of the project, then the fact is established that the project is being carried out in accordance with the established requirements and does not require operational measures response.

In one of the particular variants of implementation, if the value of any of the criteria K _TR , K _MR , K _ER , K _OR , K _SR , K _CR , which determine the completeness of the work on the project, associated with the corresponding group of project parameters and, together, describe balanced development project: works aimed at achieving technological readiness (TR), production readiness (MR), engineering readiness (ER), organizational readiness of the project (OR), work to ensure project benefits and mitigate project risks (SR), work to determine the market the readiness and commercialization of the project results (CR) is less than one, then the work associated with the corresponding parameter is considered not completed, so it is necessary to take measures of operational intervention in the course of the project, with the presentation of this fact to the user, and if the value of the criterion of any of the criteria K _TR , K _MR , K _ER , K _OR , K _SR , K _CR does not exceed a predetermined criterion value, then the project has a significant risk of non-implementation, which is the basis for making a decision to terminate the project.

In one of the private implementations, the following is carried out: the formation of an array of data of parameters that identify and characterize the maturity levels of projects, a basic list of indicators, a basic list of tasks, a basic list of works, a basic list of work results, a basic list of documents, and their input into the memory block of a computer-implemented system , connecting to the database of the monitoring system for the progress of the project via wired or wireless communication channels and importing project documents from it for the subsequent assessment of the state of the project, preparation and input of the predicted (planned) values of the estimated parameters into the computer-implemented system - the level of maturity of the project, index the maturity of the project, indicating the points in time at which the execution of the forecast should be established, expert assessment of the quality of the evidence presented by the project executor in the project documents and entering the obtained estimates into a computer-implemented system, while the resulting estimate is consistent t the numerical value of the degree of confidence determined by the expert, with which it can be argued that the evidence confirms the presence of the result in the materials of the projects, for the description of which it is presented by the project executor as part of the project documentation; processing the data array according to the corresponding criteria, followed by evaluating the processing result for each criterion with the assignment of a value from a given range of values.

According to another embodiment, a system is provided comprising at least one computing device that is configured to execute a method according to any of the preceding claims.

BRIEF DESCRIPTION OF THE GRAPHIC MATERIALS

FIG. 1 illustrates an exemplary embodiment of a block diagram of a system implementing the described invention;

FIG. 2 illustrates an exemplary embodiment of the present invention;

FIG. 3 illustrates an example of calculating the value of the maturity level of the project TPML (from the English TPM Level, Technology Project Maturity Level - the level of maturity of the project);

FIG. 4 illustrates an exemplary computation of a vector of attained maturity states;

FIG. 5 illustrates an example of calculating a project maturity index value;

FIG. 6 illustrates an exemplary calculation of a task metric T value;

FIG. 7 illustrates an exemplary version of calculating the values of K _TR , K _MR , K _ER , K _OR , K _SR , K _CR , in particular, an exemplary version of the calculation (determination) of the completeness of the work on the project is shown, which together describe the balanced development of the project: work directed to achieve technological readiness (TR), production readiness (MR), engineering readiness (ER), organizational readiness of the project (OR), work to ensure project benefits and mitigate project risks (SR), work to determine market readiness and commercialize project results (CR), at a certain maturity level at the time of the project maturity assessment;

FIG. 8 illustrates an example of data visualization for the criteria K _TR , K _MR , K _ER , K _OR , K _SR , K _CR , which determine the completeness of the work on the project at the time of assessing the level of maturity of the project for control points at the "equator" of the project and in its end.

DESCRIPTION OF OPTIONS FOR CARRYING OUT THE INVENTION

Objects and features of the present invention, methods for achieving these objects and features will become apparent by reference to exemplary embodiments. However, the present invention is not limited to the exemplary embodiments disclosed below, but may be embodied in various forms. The essence recited in the description is nothing more than specific details provided to assist a person skilled in the art in a comprehensive understanding of the invention, and the present invention is defined only within the scope of the appended claims.

The following terminology is used in the present invention.

A scientific and technological project is a complex of works, including research aimed at applying new knowledge to achieve practical goals and solving specific problems in order to create new types (types) of products and technologies, activities that are aimed at creating new materials, products, processes, devices, services, systems or methods and their further improvement, as well as plans and measures for the further implementation (industrial development) of the results.

Technology - a way of converting matter, energy, information in the process of manufacturing products, processing and processing materials, assembling finished products, quality control, management.

Technology Project Maturity (TPM) is a system for a comprehensive assessment of the current state or readiness of a scientific and technological project for a wide range of disciplines based on the use of a set of parameters that best characterize the balanced development of the project and expanding the standard capabilities of the TRA (Technology Readiness Achievement) and TRL methods through the use of a unified scale of levels of technological readiness of the project, a scientifically based set of parameters and indicators of the project, as well as special methods of their processing.

Scale of project maturity levels - a list of stages (states) and stages of an ongoing scientific and technological project from the formulated idea to the organization of serial production. The scale of the maturity levels of the project is characterized by levels from one ("1") to nine ("9").

The maturity level of the project (TPM Level, TPML) is a characteristic of the compliance of a set of project parameters that determine the balanced development of the project, and determines the degree of readiness of the results obtained in the project for commercialization, a certain stage (state) or stage of its implementation, determined by the scale of project maturity levels. The description of the TPML levels is given in Tables 1-5, which, as well as the scales indicated in them, in a particular case are one of the features of the invention and have not been noticed until now in any branch of science and technology.

The state of the maturity level of the project is a characteristic that determines the stages of development of a scientific and technological project within the same maturity level.

Interconnection matrix is a system of interconnection of stages (states) or stages of a certain level of project maturity, built on the basis of an analysis of the requirements of regulatory documents for the implementation of a project for the development of innovative products (GOST, OST, TU, etc.), and expressed in mathematical form. For example, the analysis of the results of experimental tests of the prototype of the sample cannot be performed earlier than the prototype of the sample is made.

The index of maturity of the project (I _M ) is a characteristic of the dynamics of the development of the project at short time intervals. It is represented as a fractional number from zero ("0") to nine "9") and is used when ranking projects.

The dynamics of the development of a scientific and technological project is the interval of changes in the maturity index of the project from its start to the completion of implementation.

Technological readiness (TR) of a project is a parameter that describes the degree of product readiness from a mock-up to an industrial design and a finished product; the degree of completeness of its testing - from demonstrating basic characteristics in laboratory conditions to demonstrating a full set of characteristics in real conditions as part of the final product; manufacturing method - from laboratory to industrial; etc.

Manufacturing readiness (MR) of a project is a parameter describing the degree of readiness of resources for economical production: equipment, compatibility of technological processes, certification of raw materials, readiness of specialists or production partners; selection of manufacturing technologies based on a competitive price; development of the cost model; Lean; production quality control system, etc.

Engineering readiness (ER) of the project - a parameter that describes the degree of engineering approval, product compatibility for the end use; availability of engineering support, the use of CAD / CAE / CAM systems (CAD - Computer Aided Design analogue of CAD, CAE - Computer Aided Engineering, CAM - Computer-Aided Manufacturing) for the development of components, assessment of the effects of certification and scaling; readiness of technical documentation; demonstration of manufacturing technology, etc.

The readiness of the organizational support of the project (OR) is a parameter that describes the readiness of the project team and the project executing organization: organizational and legal form, management system; business model, which is refined as the product is ready; availability of budgets required to carry out work at the next level; agreements with partners: letters, contracts, etc.

Project Benefits and Project Risk Mitigation (SR) is a parameter that describes the processes of certification and compliance with standards: industrial, environmental, safety, etc .; risks at the main production sites and sales markets and an action plan to reduce them; strategy for the protection of intellectual property and its implementation, etc.

Market readiness of the project (CR) - a parameter describing competitive advantages; pricing models; consumer sectors; readiness of the team: composition, competencies, corporate agreement, transfer of intellectual property rights; motivation system, etc.

The maturity of scientific and technological projects is assessed according to the following criteria:

- correspondence of the value of the current level of maturity of the project to the planned (forecast) value, calculated on the basis of the assessment of the results obtained, as determined by the requirements of the contract or agreement for the implementation of the project;

- project maturity index ( _IM ) and comparison of its calculated value with the planned (forecast) value;

- the criteria K _TR , K _MR , K _ER , K _OR , K _SR , K _CR , determining the completeness of the work on the project, which together describe the balanced development of the project: work aimed at achieving technological readiness (TR), production readiness (MR) , engineering readiness (ER), organizational readiness of the project (OR), work to ensure project benefits and mitigate project risks (SR), work to determine market readiness and commercialize project results (CR), at a certain maturity level at the time of the level assessment maturity of the project.

The system, an exemplary embodiment of which is shown in FIG. 1 includes a user's computing device (user terminal device) 19, which is a computer, for example, a personal computer, a server, a user's mobile computing device, in particular a mobile phone, a smartphone, a tablet computer, etc.

The user's computing device 19 includes a data processing unit (analytical data processing unit) 29, a memory unit 37, including a database (DB) of expert questionnaires and conclusions 31, a database of the results of assessing the status of projects 33, a database of project documents 35, and also includes blocks input and output, in particular, an output unit (a block for outputting the results of analytical processing) 23 and an input block 24, and the input block 24 includes a data input module about a scientific and technological project (documents, parameters identifying the project) 25 and an expert evaluation data input module the quality of facts and new attributes expanding the basic sets for a specific project (if necessary, editing the basic set) 27. In a particular case, the mentioned memory block 37 may include a database of a monitoring system for scientific and technological projects 13, or a database for a monitoring system for scientific and technological projects 13 can be connected to the user's computing device 19 via Internet using a communication module, in particular, a network adapter 21 of the user's computing device 19.

The output unit (the results of analytical processing) 23 and the input unit 24 are connected at least with the data processing unit 29 and the network adapter 21, and using the input unit 24 and the output unit 23, interaction of at least one operator 11 and / or at least one expert 15 with a system, an example of which is shown in FIG. 1. Also, using the input unit 24 and the output unit 23, data is transmitted (via wired and / or wireless communication lines) with external databases, for example, the database of the monitoring system of scientific and technological projects 13, in particular, monitoring the progress projects. Such a database of the monitoring system for scientific and technological projects 13 can contain (store) documents, for example, submitted by project executors to the management bodies during the selection of projects, as well as during their implementation.

Using the input block 24, expert 15 enters into the described system the results of an expert assessment of the quality of information (facts) extracted by expert 15 from documents submitted by the project executor and containing descriptions of the results obtained by him, and, if necessary, for a specific project, a set of attributes that are used in the assessment compliance of the project with a certain level of maturity TPML can be extended or edited by an expert 15 to evaluate the project in question in the form of an expert questionnaire; parameters identifying the project, including the quantitative value of the assessed characteristics at the beginning of the project and the predicted value of the assessed characteristics at the end of the project, as well as the predicted (planned, predicted) distribution of the assessed characteristics along the timeline of the project, indicating control points for assessing the state the project (in particular, this approach is an extension of the well-known WBS method, since it determines not only the results that should be obtained at a certain point in time, but also establishes a list of works that need to be performed at a given point in time); documents containing descriptions of the obtained results of the work on the project). Regarding the mentioned attributes: each project has a certain set of activities, outputs, etc. etc. To establish the fact that the required work has been completed, the results have been obtained, etc. etc., the expert answers the questions (one, several, many, etc.) of the expert questionnaire, and any questions can be asked in relation to any object of examination. So, for example, in order to establish that preliminary design has been carried out, an expert may be asked questions such as: Are there any documents? Do they correspond to GOST in form? Do they comply with GOST in content? For example, the expert answered all these questions in the affirmative. The GOST says that the completeness of the documents and their internal content of the TK (technical task) is established by the contractor, based on his ideas about the sufficiency and necessity of the data indicated in it. If it is necessary for the appearance (for example, creation) of any document from the category of optional ones, in order to prove the achievement of the result (certain, any, one of), the presence of such a document is required and it must be verified by an expert, and for verification it can be indicated (in particular , specified, including predetermined) the number of confirmation facts, in particular, in an arbitrary number (required for confirmation). In this exemplary case, an attribute is a set of the mentioned facts and documents, and in relation to them, verification criteria are established (set), both in quantity and in the degree of their influence to establish the fact whether the work has been completed or not and whether the required result has been obtained.

Also, for each result, a number of characteristics can be established that must be verified for it, and additional verification (for example, in accordance with the decision of an expert) of additional entities, for example, additional characteristics (including documents, etc.), is required, moreover, such entities are added for verification, for example, to a questionnaire, in particular, in accordance with the rules for creating a questionnaire. Such (in a particular case, all) information attributes (indicators) with their values of weight coefficients (for an existing system) are already in the proposed system, in particular, in block 31, but for a specific project they can be changed, so that in a particular case, flexibility and breadth of applications of the present invention are provided. Thus, in a particular case, attributes are those information objects that can be changed (in the described system), for example, level indicators, a list of tasks, a list of works, a list of results, a list of documents, a list of facts, probabilities, etc.

So, the data entered by the expert 15 or the operator 11, in particular, using at least one input device, for example, a keyboard, is transferred to the data input module about the scientific and technological project 25 and then transferred to the data processing unit 29, as well as can be transferred to the database of the monitoring system of scientific and technological projects 13.

So, the data entered by the expert 15 is transferred to the data entry module for the expert assessment of the quality of facts and new attributes that expand the basic sets for a specific project 27 and then are transferred to the data processing unit 29, and also, in a particular case, can be transferred to the database of the monitoring system scientifically -technological projects 13.

The aforementioned output unit 23 outputs data in a format that is convenient for the operator to understand (for example, in graphical, tabular and / or text formats) 11, as well as other operations to output data from the described system, including the presentation of recommendations to project executors, which are also transferred to the database of the monitoring system of scientific and technological projects 13.

The memory block 37 contains a database of expert questionnaires and conclusions 31, a database of the results of assessing the state of projects 33, a database of project documents 35, in which templates of expert questionnaires are stored, built on the basis of the use of stages (states) of the assessed maturity level TPML, a basic list of tasks that should be resolved to achieve states of maturity level, a basic list of works that must be performed to solve certain tasks, a basic list of results obtained in certain works, and containing the values of the weights set for evidence confirming the receipt of the results of work, a basic set of documents (with the requirements to them both in design and in content), in which the project implementer must provide descriptions (facts) of the results obtained to confirm the solution of problems and achieve indicators of the level of maturity of the project; information about documents (or documents) on the project, obtained as a result of processing search queries by external databases, etc.

So, in the database of expert questionnaires and conclusions 31, the mentioned templates, adapted for the project of the questionnaire, expert assessments, etc., can be stored, the results of the assessment of the status of projects can be stored in the database of the results of assessing the status of projects 33, and the mentioned documents are stored in the database of project documents 35 at least one project.

The memory unit 37 is connected to the data processing unit 29, in particular, the corresponding data obtained from the input unit 24 and / or resulting from the processing by the data processing unit 29 are stored in the said databases, and data is transferred from the corresponding data upon request from the unit data processing 29. So, the data processing unit 29 interacts with the databases of the memory unit 37 (and the database of the monitoring system for scientific and technological projects), the input unit 24 and the output unit 23 and processes information from the databases and stores it in the database, in in particular, in the database of the results of assessing the status of projects 33 the results of processing, in particular, analysis (assessment). The coordination of the actions of the components of the described system is carried out by the processor of the described system, which implements the algorithms, in particular, implements the described innovations in the form of executable commands.

Using wired and / or wireless lines, the user's computing device 19 communicates with at least one database of the monitoring system for scientific and technological projects 13, to which the user's computing device 19 transfers search queries related to a specific project, and also, after receiving the results of processing requests, saves in at least one database of the memory block 37 for further program processing, and also transfers (in particular, exports) to the above database of the monitoring system of scientific and technological projects 13 information necessary and used for correction (including in automatic mode by means of the described system, for example, by the data processing unit 29, in particular, by the correction module) of the work plans by the project executors in case of the lag of the project development dynamics from the planned one, including their further presentation to users. The aforementioned requests are transmitted (including by the project identifier) to at least one database for the documents that will be processed (in particular, considered), for example, by an expert, in particular, the list of documents and the documents themselves. The mentioned information, necessary and used for correction, can be used in the management body, for example, to create (draw up) an additional agreement to the contract to change the work schedule, or to make the necessary changes, for example, in their project management systems, in particular by performers ...

FIG. 2 shows an exemplary embodiment of the present invention.

In step 52, the preparation (for example, in automatic mode by means of the described system and / or operator 11) an array of information identifying and characterizing the levels of maturity of projects is carried out, in particular, a basic list of states (stages), a basic list of tasks, a basic list of works, a basic list facts (evidence) confirming the receipt of the results of work with the required quality. Thus, the operator, using the input device, enters the above parameters into the system that identify and characterize the maturity levels of projects. Also, an array of information is being prepared containing the values of the quantities W (E _U ) characterizing the weights (contribution) of each of the evidence for making a decision that the work has been completed in full with the required quality. These values also characterize the relative importance of each of the evaluated evidence in comparison with other evidence. So, for example, the mentioned prepared array of information (at the TPML5 level) may include the following data:

- State 1 - Based on the analysis of the model tests, a task was drawn up for an experimental sample (EO) on a scale close to real;

- Task 1.2 - Draw up a technical assignment for an experimental sample on a scale close to real;

- Work 1.2.3 - Determine the goals, objectives, functions of the experimental sample;

- Result 1.2.3.1 - The goals of creating the EA have been determined;

- Testimonials -

-1.2.3.1.1. There is a TK document;

-1.2.3.1.2. Such a document contains the section "Objectives of the experimental design";

-1.2.3.1.3. The content of the section "Purpose of creating an experimental sample" meets the requirements of GOST.

Weight (Certificate 1.2.3.1.1) = 0.25.

Weight (Certificate 1.2.3.1.2) = 0.25.

Weight (Certificate 1.2.3.1.3) = 0.85.

Using the above-mentioned base lists, templates of expert questionnaires are prepared automatically and / or by the operator using information objects and weight coefficients stored in at least one database, for example, in (corresponding) tables of the database (s). The generated information arrays are entered into the system and transferred to the memory unit 37 for storage and use (in particular, for processing), including at least by the data processing unit 29, in particular for automated calculation (TPML maturity level, maturity level states , work results, completeness of partial levels of readiness, etc.), as described within the scope of the present invention. In a particular case, a template is prepared (by means of the described system, using the means of the described system, for example, by an operator, etc.) of basic sets, which can be used by an expert, by means of the described system, and the template can be edited, for example, at least at least by one user, for example, an expert, or in automatic mode. After filling out (by an expert) the questionnaire, the described system processes at least the data contained in the questionnaire in automatic mode.

In step 56, a connection is established between the database of the monitoring system of scientific and technological projects (the database of information of the monitoring system for the progress of the project) 13 and the user's computing device 19 and the import of project documents is carried out to assess the current state of the project, in particular, the communication module, in particular , which is a network adapter 21, in particular, the documents themselves are requested, including for processing by an expert (an expert must view, check, etc. such documents), in the monitoring process (for example, by a monitoring system), etc., and import of project documents is carried out from the database of the monitoring system of scientific and technological projects 13 to the database of project documents 35.

In step 60, using the information extracted (by means of the described system and / or by the operator, including on the basis of the schedule contained in the contract) from the project documents, the initial and final values of the estimated parameters and criteria of the project maturity are set (set). a timeline with control points located on it and predicted values of the project maturity index, in particular, those specified during the initial project setup, for example, by the operator, as well as a forecast of their changes over the project execution time (analogous to WBS), in particular, before the project starts. The input into the system of such parameters that identify and characterize a specific scientific and technological project is also carried out by the operator 11 using input devices through the input unit 24. Thus, when concluding contracts for research and development (research work), R&D (development work) is carried out processing (by the operator and / or the means of the described system) the work planned in the schedule, comparing them with the maturity level indicators, tasks and works contained in it, and drawing up a plan containing the said extracted information. In a particular case, the conclusion of the contract and the drawing up of the schedule is already associated with the TPML scale (linked to the TPML scale), and in this case the process of the mentioned comparison can be excluded. With regard to the initial and final values of the estimated parameters and criteria of the level of maturity of the project, in a particular case, the processing of bids is carried out (for example, by governing bodies or organizations authorized by them, including using the means of the described system and / or the means of the described system), the assessment is carried out ( including express assessment) of the current level of maturity, processing of explanatory notes, certificates, etc. Further, the data obtained as a result of processing is used to evaluate the application in the described system. Further, for a condition (a given condition, i.e., assuming the fact) that everything that the performers want to do has been done, moreover successfully, the processing of the TK (data contained in the TK), the schedule is carried out and a re-check of that the same project in the described system. Thus, in a particular case, two values of the maturity level are determined at the input and output. Then the input data is checked (for example, by the control). If the value of the maturity level of the project at the entrance is small, then such an application cannot become the winner of the competition. In particular, the mentioned set (set) forecast is a calculation of what should be along the time scale. In particular, a (any) plan is a prediction that either comes true or not.

In step 64, the quality of the facts extracted from the assessed documents is assessed, and the values of the assessment of facts are entered into the described system, and, if necessary, in addition to the basic set of assessed necessary attributes, the assessment can be carried out by an expert 15 and / or in an automated mode by means the described system, for example, a data processing unit 29 or a module of a data processing unit, for example, an expert judgment module. The aforementioned basic set is a set that was created (compiled), for example, on the basis of processing (studying) GOSTs that determine the order and stages of work (for example, GOST SRPP) practices for implementing innovative projects in various development institutions, etc. The additional set referred to is a set that is either set by the governing body for a specific project (group of projects), which means a change (and requires a change) in the template for these projects, or by an expert to evaluate a specific project - i.e. the specific profile changes.

To evaluate the evidence confirming the receipt of the results, the documents submitted by the project executors at certain points in time to the management bodies of the project being executed are used. These documents are entered (in particular, transferred) to the system after connecting to the database of the monitoring system of scientific and technological projects 13 and importing project documents to assess the current state of the project in step 56.

Project documents obtained from the database of the monitoring system of scientific and technological projects 13 are used when carrying out an expert assessment (in automatic or semi-automatic mode, including by an expert 15) of the quality of the certificates presented by the contractor about the results obtained during the implementation of the project, while taking into account the specifics of the project in question, when preparing an expert opinion, the expert questionnaire can be modified (including by an expert 15) by changing or expanding the basic list of states of the maturity level, a basic list of tasks, a basic list of works, a basic list of evidence of work results, as well as a basic list of documents which must contain the evidence. So, added (in particular, indicated) for the changed or added to the list of work results the values of the quantities W (E _U ) characterizing the weights (contribution) of each of the evidence for making an affirmative decision on the performance of work with the required quality. The modified questionnaire is stored in the memory block 37, at least in one database, for example, in the database of expert questionnaires and conclusions 31. The mentioned modification of the expert questionnaire, in a particular case, provides flexibility in setting up the tool and expanding its application areas for evaluating projects in various fields of science and technology.

Further, in step 68, including after adding the data of the expert assessment of the quality of the presented facts about the results obtained during the execution of the project in the form of an expert questionnaire, by means of the described system, in particular, by the data processing unit 29, by means of the described system, for example, by the data processing unit 29, the criteria for evaluating the maturity of a scientific and technological project are automatically calculated, which are subjected to analytical processing and transmitted to the output unit (information output unit, analytical processing results output unit, materials presentation unit) 23.

In step 72, the results of analytical processing of the calculation results for the project (projects) are output in one of the well-known formats for presenting data (information) to users (including operator 11 and expert 15), for example, in tabular and graphical forms using the means of the described system, in particular, using the output unit (reporting unit and visualization of results) 23. Thus, the presentation is carried out to the user, for example, the operator (expert, etc.), in particular, the output is carried out, the results of ranking projects, the index (s) of the maturity of projects , the dynamics of changes in the maturity index of the project, including in comparison with the predicted, the degree of completion (completeness) of the work associated with the project parameters TR, MR, ER, OR, SR, CR.

Also, the data processing unit 29 generates a set of recommendations to the executors of projects, which are also presented to users by the output unit 23 using one of the known information presentation devices, for example, using a monitor.

If in the course of the mentioned processing it was found that the development of the project lags behind the planned indicators, then by means of the system, including the data processing unit 29, as well as optionally by the expert 15, recommendations are prepared for the project executors, aimed at increasing the efficiency of their work. These recommendations, including the calculated quantitative values of the assessed criteria for the maturity of the project, through the established connection to the database of the monitoring system of scientific and technological projects 13 are transferred (exported) to the corresponding sections of the above-mentioned database of the monitoring system of scientific and technological projects 13 and, subsequently, are presented to the executors of projects , in particular, are communicated to the project executors by the methods established in the mentioned monitoring system or by the output unit 23.

Also, the means of the described system, for example, the data processing unit 29, can be used to build a rating from several evaluated projects by the quantitative value of their maturity indices in points, as an additional function of the described system. The above-mentioned construction of a rating from several assessed projects by the quantitative value of their maturity indices in points can be carried out, for example, by creating a table of projects, in particular, projects in progress, each is assigned an index value, after which the projects are sorted by index values, for example, in descending order index, the result of which is a rating that can be transferred to the management body for analysis (and processing), and as a result of sorting, at least one project (at least one) with a minimum index and a project (at least one) with maximum index.

FIG. 3 shows an example of calculating the maturity level of a TPML project.

To calculate the value of the current maturity level of the TPML project, a model is used, built on the basis of the decomposition of the project execution process as a system of states, tasks, activities and evidence of the receipt of work results in the process of project execution, and the decomposition of the project execution process is based on the requirements of regulatory documents and standards. In this model, each maturity level L (from the English Level) of the project corresponds to a certain set of states (from the English stage, state) and the corresponding quantitative values L | S = {S _I , I = 1,2, ... N _L }, and the S _I values are equal to one ("1"), if the states are achieved, and zero ("0"), if not. To achieve these states, the project performers must solve the corresponding tasks (from the English Task), which are also assigned quantitative values, in particular for the indicator S _I | T = {T _J , J = 1,2, ..., N _SI } , and the values of T _J are equal to one ("1"), if the corresponding problem is solved, and zero ("0"), if not. Each task is divided into a number of works (from the English Act) aimed at achieving the appropriate readiness of the project - TR, MR, ER, OR, SR, CR, which are also assigned quantitative values, in particular for the task T _J | A = { A _M , M = 1,2, ..., N _TJ }, and the values of A _M are equal to one ("1"), if the work is done, and zero ("0"), if not. The result of these works are specific results, which are also assigned quantitative values, in particular for work A _M | E = {E _U , U = 1,2, ..., N _AM }. Evidence of the results - descriptions, evidence, facts - E _U (from the English Evidence), in turn, is presented by the project executors as part of the reporting documentation on the progress of the project. Quantitative values are also assigned to the E _U values. This information about the results obtained within the framework of the model is a set of evaluable evidence. The value of this characteristic E _U depends on the quality of the supporting information about this certificate contained in the project documentation (D - from the English Document) E _U | D = {D _Y , Y = 1,2,…, N _EU }.

Each of the evidence presented by the project executors cannot fully allow making an unambiguous conclusion about the achievement of a certain result, this can be done only if there is a whole set of supporting evidence. Within the framework of the proposed model, each evidence E _U has a weighting coefficient 0 <W (E _U ) ≤ 1, initially established by an expert method and having the following meaning: the value W (Eu) characterizes the contribution of each evidence E _U for making an affirmative decision on obtaining a result corresponding the required quality, and the relative importance of evidence that the required results are being obtained. If the result obtained meets the established requirements, then E _U = W (E _U ), if the quality of any of the assessed evidence does not meet the specified requirements, then the value of E _U is equal to the weight factor multiplied by the correction factor 0≤Q (E _U ) ≤ 1 (from the English Quality), E _U ⁼ W (E _U ) * Q (E _U ), in particular, if there is no information about the certificate in the project materials, then the value of W _U is multiplied by a correction factor, the value of which is zero ( "0"), and if the quality of the submitted certificate meets the requirements, then by a correction factor, the value of which is equal to one ("1"). In other cases, the expert determines the value of the correction factor based on personal experience, or on the basis of the requirements for the assessment process established by the management body. This approach to assessing the results allows you to vary the requirements for their quality for various projects or ongoing competitions for their selection according to the following rule, if the value of the value W _A = Σ (W (E _U ) * Q (E _U )) / ΣW (E _U ) , calculated from the totality of the evidence included in it will be greater than a certain threshold value 0 <W _threshold ≤1, then the result can be counted, and not - otherwise.

In the used information model of the project maturity level, the basic set of supporting evidence is determined on the basis of processing an array of information about already completed projects using the method of expert assessments, and the model allows for the expansion of this basic set at the stage of assessing a specific project by an expert.

In step 80, the data processing unit 29 receives from the memory unit 37 the adjusted vector of the achieved states of the project maturity level. The aforementioned corrected vector of the achieved states of the maturity level is generated automatically by the system means after correcting the result of calculating the states using the interrelation matrix, in particular, some values equal to one ("1") for them can be replaced by values equal to zero ("0").

In step 83, data processing unit 29 calculates the maturity level of the project using the formula

In step 86, the data processing unit 29 checks that the TPML _L value is equal to the _L value. If the TPML _L value is not equal to the L value, then in step 86 it returns to step 80. For example, if the project level is in accordance with the schedule , should have a value equal to four ("4"), but as a result of its evaluation by the proposed system it was found that this is not the case, then further verification of the project level will be carried out for a level equal to three ("3") to make sure that the project is at a maturity level of three ("3"), and so on, until a maturity level of 1 to 4 is fully confirmed.

If it is established in step 86 that the TPML _L value is equal to the L value (i.e., at a given time, the plan coincides with the fact), then in step 92, the obtained value of the maturity level is stored in the database of the results of assessing the state of projects 33 of the memory block 37. In a particular case, after obtaining the TPML _L value, the values are calculated as shown in FIG. 4 to FIG. 7.

FIG. 4 shows an example of the calculation of the vector of achieved states.

In step 110, the data processing unit 29 receives from the database of expert questionnaires and the conclusions 31 of the memory unit 37 a set of attributes characterizing the tasks performed to achieve the states of a certain (in particular, a given) level of project readiness L, in particular for the state S _I | T = {T _J , J = 1,2, ..., N _SI }, obtained from the data of the expert questionnaire template.

Further, in step 114, the data processing unit 29 calculates the value of the achieved state of the maturity level of the project according to the formula S _I = T ₁ * T ₂ * ... * T _NSI , that is, by multiplying the values of T _J (equal to zero or one), the calculation of which shown in FIG. 6.

In step 118, after calculating (calculating) the values of all S _I for all tasks of the maturity level, the data processing unit 29 corrects the values of the coordinates of the vector of the achieved states using the relationship matrix.

Thus, the obtained vector S is corrected using the MM interrelation matrix, the elements of which determine the interrelation between the states of the project maturity level, established by expert means using (based on the study) the requirements of regulatory documents and other standards that apply to the processes of implementing innovative projects. Each element of the relationship matrix is established based on the rule MM _{I, J} = 1 / R if the state S _I is associated with the state S _J , and 0 if not. The R value is equal to the number of all connections of the S _I state with the rest of the states (including itself). For example, if the state S ₁ cannot be reached earlier than the states S ₂ and S ₄ are reached, then R = 3, MM _1,1 = MM _1,2 = MM _1,4 = 1/3, and the remaining elements of the matrix MM in 1st line will be 0.

To correct the value of the state vector S, it is multiplied on the right by the MM matrix according to the rules of matrix multiplication. If, in the end, each component of vector S equals 1, then this means that the current level of maturity of the TPML project at this point in time has been reached, and if it coincides with the forecast value, which can be determined based on the information provided in the contract or agreement for the implementation of the project , then it is concluded that the project is being carried out in accordance with the established requirements and does not require prompt response measures.

In step 122, the obtained vector of the achieved states is stored in the memory block 37, in particular, in the database of the results of evaluating the state of projects 33.

FIG. 5 shows an example of calculating the value of the project maturity index.

In step 140, the data processing unit 29 receives from the database of expert questionnaires and the conclusions 31 of the memory unit 37 of the adjusted vector of the achieved states (the dimension of the vector N) at the given maturity level of the TPML project. The corrected vector is the result of automatic correction using a relationship matrix describing the relationships between states in terms of the correct sequence of problem solving to achieve the maturity level. For example, problem number ten was solved, but problem number five was not, but problem number ten could not be solved before problem number five, therefore, problem number ten becomes unsolved, which is an example of the mentioned correction. The result of processing the adjusted vector of maturity level indicators in block 140 is the number NN of maturity level states, the values of which are equal to one ("1"), i.e. the number of states achieved by solving problems with the required quality.

Further, in step 143, the data processing unit 29 calculates (calculates) the total number of all states N of the maturity level, both achieved and not achieved.

In step 146, the data processing unit 29 calculates the value of the project maturity index I _{M of the} project according to the formula I _M = TPML _L -1 + NN / N.

Further, in step 149, the data processing unit 29 checks whether the value of the maturity index of the project 1 m is greater than or equal to the planned (predicted)

values of the maturity index of the project I _plan (I _M ≥I _plan ). If the value of the maturity index of the project I _{M is} less than the planned (predicted) value of the maturity index of the project I _plan , then in step 155, the means of the described system (for example, a data processing unit) and / or an expert make management decisions and in step 158 the obtained index value is saved the maturity of the project in the database of the results of assessing the status of projects 33 memory blocks 37. If the value of the maturity index of the project I _{M is} greater than or equal to the planned (forecast) value of the maturity index of the project I _plan , then in step 152 the fact of the normal development of the project is established, i.e. the project is in progress in accordance with the requirements established in the contract or agreement for the implementation of the project and, therefore, does not require immediate response measures, and in step 158, the obtained value of the project maturity index is stored in memory block 37.

FIG. 6 shows an example of calculating the quantitative value of the indicator T _{J of the} problem, characterizing the result of its solution. If the T _J value is equal to one ("1"), then this means that it is solved, and if it is zero ("0"), then it is not.

A task can include several required tasks. If all such compulsory works are completed (completed), then the indicator of the task T _J is equal to one ("1"), if not, then zero ("0"), where "0" - the problem was not solved with the required quality.

An example of compulsory works of applied research of the maturity level "The fundamental concept of technology is formulated and the usefulness of the technology is substantiated", for which the main directions of research are formulated, the prospects of the developed technology and its demand are substantiated, and the infrastructure (belonging, composition) necessary for the implementation of the project (research, production and implementation of the results) for TPML1 is shown in Table 6. Table 6 presents the basic set of tasks and activities for the maturity state “Analysis of existing solutions on the market”.

For the data given in Table 6, the number of works, in a particular case, is 7, in particular, for TR - 1, for SR - 5, for CR - 1.

In step 171, the data processing unit 29 receives from the database of expert questionnaires and the conclusions 31 of the memory block 37 of the list of works T _J | A = {A _M , M = 1, 2, ..., N _TJ } aimed at achieving the appropriate readiness of the project - TR, MR, ER, OR, BR, CR, at a certain level of maturity L of the project, associated with the characteristic (indicator) T _{J of the} task, and a list of evidence of the results of their implementation A _M | E = {E _U , U = 1, 2 ,…, N _AM }, which must be indicated in the documents submitted by the project executor in order to conclude that the work has been done.

The facts (evidence) Ei are characterized by a weighting coefficient that is initially established by an expert and has the following meaning: the value W (E _U ) characterizes the contribution of each evidence Ei to making an affirmative decision to obtain a result corresponding to the required quality, as well as the relative importance of evidence of obtaining the required results. If the result obtained meets the established requirements, then E _U = W (E _U ), if the quality of any of the assessed evidence does not meet the specified requirements, then the value of E _{U is} equal to the value W (E _U ) multiplied by the correction factor 0≤Q (E _U ) ≤1, in particular, if there is no information about the certificate in the project materials, then the value of W _U is multiplied by the correction factor, the value of which is zero ("0"), and if the quality of the submitted certificate meets the requirements, then the correction factor, whose value is equal to one ("1"). In other cases, the expert determines the value of the correction factor based on personal experience, or on the basis of the requirements for the assessment process established by the management body.

In step 174, the data processing unit 29 calculates the quantitative value of W _A for the entire set of evidence included in it according to the formula W _A = Σ (W (E _U ) * Q (E _U )) / ΣW (E _U ), the analysis of the value of which allows the evaluation to conclude that the result of performing the work can be considered as satisfying the previously specified requirements based on the comparison of this value with the threshold value, which is carried out in step 177.

In step 177, the data processing unit 29 checks whether the value W _{A is} greater than or equal to the value W threshold or not W _A ≥W _threshold ). If the value W _A is greater than or equal to the value W of the threshold, then in step 180 the value A _M is equal to unity (A _M = 1), which means that the work _AM performed, and if the value W _A is less than the W threshold, then in step 183 A value of _M is equal to zero (A _M = 0) which means that the work is not performed _AM. Multiplying all the values of A & _M implementation of the results to determine whether the problem is solved completely.

Further, in step 186, the data processing unit 29 calculates the value of T _J according to the formula T _J = A ₁ * A ₂ ... * A _NTJ , i.e. after determining all the values of А _{М, the} value T _J = A ₁ * A ₂ * ... * A _NTJ can be calculated It is easy to see that each parameter T _J can take values 0 or 1. To calculate the value of the achieved state of maturity level S _I , the expression S is used _I = T ₁ * T ₂ *… * T _NTJ .

Thus, the result of the calculations carried out based on the results of the expert assessment of the project is the vector of achieved states S, each component of which is 1 if it is established that the state has been reached, and 0 if the state has not been achieved.

In step 189, the data processing unit 29 stores the obtained value of the quantitative value T of the task in the database of the results of assessing the state of projects 33 of the memory unit 37.

FIG. 7 shows an example of calculating the values of K _TR , K _MR , K _ER , K _OR , K _SR , K _CR .

Thus, in FIG. 7 shows an exemplary option for determining the completeness of the project work, which together describe the balanced development of the project: work aimed at achieving technological readiness (TR), production readiness (MR), engineering readiness (ER), organizational readiness of the project (OR), works to ensure project benefits and project risk mitigation (SR), work to determine market readiness and commercialize project results (CR), at a certain level of maturity at the time of assessing the level of maturity of the project.

In step 201, the data processing unit 29 receives from the database of the results of assessing the state of projects 33 of the memory unit 37 values of the number of jobs N associated with the project attribute groups TR, MR, ER, OR, BR, CR at a given level of project maturity into the data processing unit to determine the coefficients K _TR , K _MR , K _ER , K _OR , K _SR , K _CR , as well as the values of the adjusted vector of the achieved states of the project maturity level.

In step 205, the data processing unit 29 calculates the values NN _{TR, MR, ER, OR, SR, CR} , which determine the number of actually performed jobs, calculated according to the coordinates of the adjusted vector of attained states S. For this, the values of T _{J are} first corrected according to the rule: if S _I = 1, then all associated with it T _J = 1, and if S _I = 0, then all associated with it T _J = 0. Similarly adjusted value A _M for subsequent calculation criteria _{K TR, K MR, K ER} , K OR, K SR, K CR: if the value of T _J = 1, then all the values of _M associated with them are set to 1, and if the value T _J = 0, then all the values of _M associated with them are set to 0.

As described in the framework of the present invention, each work is aimed at obtaining a result that characterizes one of the groups of project parameters - TR, MR, ER, OR, SR, CR, i.e. has a type corresponding to a certain level of availability. The sum of the number of jobs with the same type gives the required value of the total number of jobs of this type, and the amount of jobs performed with this type allows you to estimate the completeness of the work.

Thus, to calculate the criteria K _TR , K _MR , K _ER , K _OR , K _SR , K _CR , in particular, determining the completeness of the work on the project - work aimed at achieving technological readiness (TR), production readiness (MR) , engineering readiness (ER), organizational readiness of the project (OR), work to ensure project benefits and mitigate project risks (SR), work to determine market readiness and commercialization of project results (CR), from the components of the adjusted vector А _{М is} carried out, into in particular, by means of at least one database, the selection (selection) of those that belong to a certain group TR, MR, ER, OR, SBR and CR.

Further, the data processing unit 29, in particular to calculate the completeness, calculates the number of elements, the value of which is 1 - NN _{TR, MR, ER, OR, SR, CR} (identical to the record in the form of NN _TR , NN _MR , NN _ER , NN _OR , NN _SR , NN _CR ), and the calculation (determination) of the total number of selected elements is carried out - N _{TR, MR, ER, OR, SR, CR} (identical to the record in the form of N _TR , N _MR , N _ER , N _OR , N _SR , N _CR ). Thus, the sum of those (values) that are equal to one is divided by the total number of jobs of this type, i.e. if the value of an element is equal to one, then the selection is carried out, if zero, then no, and then the cardinality of the obtained set is calculated, i.e., in particular, its elements are recalculated in order from one and further.

In step 211, the data processing unit 29 calculates the values of K _{TR, MR, ER, OR, SR, CR} (identical to the record in the form of K _TR , K _MR , K _ER , K _OR , K _SR , K _CR ) according to the formula K _{TR, MR, ER, OR, SR, CR} = NN _{TR, MR, ER, OR, SR, CR} / N _{TR, MR, ER, OR, SR, CR} .

In step 215, the data processing unit 29 (or, for example, by means of the corresponding database) stores the obtained (calculated) values of the coefficients K _TR , K _MR , K _ER , K _OR , K _SR , K _CR (identical to the record in the form of K _{TR, MR, ER, OR, SR, CR} ) in the database of the results of assessing the state of projects 33 of the memory block 37.

If the values of any of the criteria K _TR , K _MR , K _ER , K _OR , K _SR , K _{CR are} less than one ("1"), this means that the work associated with the corresponding group of project parameters has not been completed, which means the need to take operational measures in the course of the project, which is at least communicated to users, as described in the framework of the present invention, and if, in a particular case, the value of the criterion of any of the criteria K _TR , K _MR , K _ER , K _OR , K _SR , K _CR does not exceed a predetermined value (criterion value), in particular, set by the requirements of the governing bodies and monitoring the progress of the project based on statistical processing of information about all previously completed projects, for example 0.15 (the deviation is not less than 85%), this means that the project has a significant risk of failure, which is at least communicated to users, as described in the framework of the present invention.

When evaluating several projects according to the described method, a rating (or ranking) can be built according to the numerical value in the points of the maturity index of each of the projects in ascending or descending order of the received point values, as described in the framework of the present invention.

FIG. 8 shows an exemplary version of data visualization for the criteria K _TR , K _MR , K _ER , K _OR , K _SR , K _CR - completeness of work on the project, at the time of assessing the level of maturity of the project for control points at the "equator" of the project and in its end.

The inventive method was tested when evaluating 90 scientific and technological projects. Each project was assessed according to attributes related to the following groups of parameters that characterize its value: technological readiness (TR), industrial readiness (MR), engineering readiness (ER), organizational readiness of the project (OR), benefits and risks of the project (SR), market readiness and commercialization (CR). Evaluations were conducted at three points in the project timeline at the start of the project, midway through the project timeline, and before completion. The resulting set of quantitative assessments for each project made it possible to build a rating from the evaluated projects, which corresponded to the degree of their completion in terms of their readiness for practical use, in addition, analytical reports were prepared to develop managerial decisions in relation to projects, the indicators of which significantly differed from the values calculated after the competitive procedures for the projects under study.

Below is an example of assessing the degree of readiness according to the estimated parameters for one of the scientific and technological projects on the topic "Development and creation of effective separating devices for obtaining conditioned (with a concentration of 99.8%) heavy water at FSUE PA Mayak".

Based on the study of the tender application for this project, a plan for monitoring and assessing the level of its maturity was drawn up, and the control time points for the assessment were selected in the following sequence: start of the project "equator" (middle) of the project (3rd stage of the project schedule) end of the project (5th stage of the project schedule). Based on the project expert opinions on the project available in the monitoring system for these points of measurement of the maturity level of this project, quantitative values of W (E _U ) values characterizing the contribution of each certificate E _U were introduced into the templates of expert questionnaires for assessing the maturity level to make an affirmative decision works with the required quality. After entering the prepared data into the computer-implemented system according to the methods described earlier, the quantitative values of the criteria used to assess the maturity of the project were calculated, incl. the maturity level of the project (TPML), the index of project maturity (I _M), the completeness of the implementation of the project K: work aimed at achieving technological readiness (TR), production ready (MR), engineering preparedness (ER), availability of organizational support of the project ( OR), work to ensure project benefits and mitigate project risks (SR), work to determine market readiness and commercialize project results (CR). The calculation results, in comparison with the planned values, are presented in table 7.

Data of the planned and actual values of the project maturity criteria (table 7) for the criteria K _TR , K _MR , K _ER , K _OR , K _SR , K _CR - completeness of the work on the project, for control points at the "equator" (middle) of the project 404 and at the end of the project, 408 are graphically represented in FIG. 8.

From the graph shown in FIG. 8 it can be seen that at the "equator" of the project, 404, it can be concluded that the project is developing unbalanced, in particular, the project executors did not perform a significant amount of work to determine the market readiness and commercialization of project results (CR), as well as work on the achievement of production readiness (MR), and if at the end of the 408 project, after the prompt response of the management body of the project executor, the entire complex of works on preparation of production (MR) for the industrial implementation of the project results was completed, then the work on determining the market readiness and commercialization of the project results (CR) were never fully implemented.

Evaluating the results of the project as a whole, we can conclude that during its implementation the level of maturity did not reach the required value.

The described system can be a personal computer with an operating system installed on it, belonging to the most common types of operating systems MS Windows, Mac OS or Linux, containing a central processor, random access memory, hard disk drive (or memory block, or storage data, or DB), keyboard, monitor, communication devices for working with Internet resources (network adapter), as well as an output device (data output / display of processed data, for example, a monitor, or a monitor together with a printer). Computational tasks can be solved under the control of a central processor, which is an electronic unit, or an integrated circuit (microprocessor) that executes machine instructions (program code), depending on the machine instructions entered into the processor. In the claimed invention, machine instructions can be expressed in the form of functions prescribed for execution by the processor, namely, providing the calculation of quantitative values of the level of maturity of the project, quantitative values of the results of solving problems, degrees of confidence in the performance of work and comparison of these values with threshold values, quantitative values of the results performance of work, calculation of the vector characterizing the achievement of the maturity level states, correction of the vector characterizing the achievement of maturity level indicators using the interconnection matrix, calculation of the completeness coefficients for the works describing in aggregate the balanced development of the project aimed at achieving technological readiness (TR), production readiness ( MR), engineering readiness (ER), organizational readiness of the project (OR), work to ensure project benefits and mitigate project risks (SR), work to determine market readiness and commercial project results (CR), calculation of the project maturity index and its comparison with the predicted value at a given time, sorting (ranking) of projects by the maturity index value. The processor is configured to process information (data) obtained by the claimed method using the evaluation criteria TPML, I, K _TR , K _MR , K _ER , K _OR , K _SR , K _CR . The data processing unit is usually installed in a system unit that executes machine-executable instructions. The claimed system, in one of the variants of execution, can be built on the von Neumann architecture, which defines the well-known principle of joint storage of instructions (instructions) and data in the computer memory.

As described within the framework of the present invention, in a particular case, the described invention, including the described calculation method, uses the logic of the scientific and technological project both in accordance with the requirements of GOST and the requirements of various practices for the implementation of innovative projects, for which a model can be used, describing the order of this process.

A basic set of indicators is set for the model, namely:

- when carrying out a project at each maturity level L, it is necessary to conduct various studies that form a set of states for each level - the results of these studies L | S = {S _I , I = 1, 2,… N _L }. In a particular case, for each level, the number of these states N _{L is} different when carrying out different

number of studies. Further, one level is selected as an example (in particular, any selected level will be used for the example);

- each of the mentioned studies requires the solution of at least one problem (one, several, multiple problems). The set of such tasks T for the selected indicator of this level, for example, the I-th, is the value S _I | T = {T _J , J = 1, 2, ..., N _SI }, where N _SI is the number of such tasks just to achieve this the selected state at the selected maturity level;

- the process of solving problems is divided into works. If one of the tasks J is selected for the selected state, then its set of activities A can be represented as T _J | A = {A _M , M = 1, 2,…, N _TJ }. Each work is aimed at achieving a certain group of project indicators TR, MR, ER, OR, SR, CR, and has the corresponding type, which coincides with the group of project indicators, which is used to calculate the completeness of work for each of the readiness levels - TR, MR, ER , OR, SR, CR;

- then the work of the selected task of the selected indicator is selected. Each work includes a set of results and evidence of their receipt A _M | E = {E _U , U = 1, 2,…, N _AM }. These evidences have different contributions (weights) for making an affirmative decision about obtaining a result with the required quality, as well as the relative importance of evidence of obtaining the required results, the facts have different significance (or contribution) for the performance of the work. All certificates are contained in the design documentation - E _U | D = {D _Y , Y = 1, 2,…, N _EU }.

Table 6 above contains a fragment of the model for the first (1st) level of project maturity. The first grouping level (the first group) includes table rows for the states of the maturity level, in particular, Table 6 provides information for one of the states - State S3. The analysis of existing solutions on the market is carried out.

The second level of grouping (second group) includes the rows of the table for the tasks that need to be solved to achieve the indicators, in particular, Table 6 indicates the tasks that need to be solved to achieve the specified indicator of the maturity level.

The third level of grouping defines the set of jobs A for solving these problems.

For each work, a set of facts (evidence) can be indicated, in particular, the content of indicators, with their weights, as well as a set of requirements that must be met by evidence (in particular, probabilities).

For the assessment process:

- for the achieved / fulfilled level, if all its states are reached, i.e. all studies have been carried out, the quantitative value S of each of the states is equal to 1 or 0, which makes it possible to establish the result of reaching the maturity level using the product TPML _L = S ₁ * S ₂ *… * S _NL ;

- each of the indicators S _{I is} achieved (ie equal to 1), if all the tasks included in it are solved: S _I = T ₁ * T ₂ *… * T _NSI ;

- the work is completed if all the results for it are confirmed by the totality of evidence: T _J = A ₁ * A ₂ *… A _NTJ ;

- if in the process of evaluating the facts (obtained from the project documents), for example, by an expert and / or by means of the described system, the fact is established that they have the necessary quality, i.e. can be used to confirm a certain result, then, using the basic set of indicators, namely the values of the weight coefficients for the evidence W _(E) , the calculation (calculation) of the value W _{A is} carried out for each work on the set of its evidence: W _A = Σ (W ( E _U ) * Q (E _U )) / ΣW (E _U ), and if the quality of the facts is insufficient (for example, if the expert begins to doubt the quality of the facts), then the value of W ( E _U ) for a certain fact by multiplying this value by the correction factor 0≤Q (E _U ) ≤1, for example, W (E _U ) can be reset to zero, the value of W (E _U ) can be reduced, for example, by half, or a new value W (E _U ) can be set. For example, an expert can press the "Disagree" button to reset W (E _U ), or, for example, the button, "Not completely", and W (E _U ), is halved ("half"), or can enter (set) new (own) value of such a quantity;

- after receiving the value W _A , such a value is compared with a predetermined threshold value, and if such a value is greater than a predetermined threshold W _threshold , then the quantitative value of work performance is equal to one ("1"), and if less than a threshold, then zero ("0");

- then all the values of A are multiplied to obtain the value of T, the multiplication of which, in turn, gives the value of S;

- in a particular case, some states S cannot be reached earlier than others. Such a relationship is determined by a relationship matrix, an approximate version of which for a maturity level equal to one is shown in Table 8, taking into account the fact that to achieve a project maturity level equal to 1, it is necessary to fulfill 10 indicators from S ₁ to S ₁₀ . This matrix is built on the basis of the sequence of research in time, which is determined, in a particular case, from GOST and the practice of implementing innovative projects, as well as taking into account the opinion of experts;

- if all values of S are represented as a vector, the corrected vector S is the result of its right multiplication by the relationship matrix. In a particular case, not all values equal to one will remain unchanged (equal to one). In a particular case, some can be zeroed (in particular, their values can become equal to zero). This resulting vector is the adjusted maturity state vector. In this case, changes are tracked when the numerical value of the level state changes from one to zero;

- if the value of some state has become equal to zero, then all the values of T associated with it (which are “tied” to it) are reset, even if they were calculated by the means of the described system as a unit (“1”);

- if any of the values of T is zeroed, then all the values of A that are associated with it (which are "tied" to it) are reset to zero;

- then the number of all work results for each type of work is calculated, which are equal to one. This is the number divided by the total number of jobs of this type, which gives an idea of the completeness of the work of this type. In particular, the basic set of jobs for a maturity level equal to L = 1 is 104 jobs, and the number of jobs for type TR is 23, for MR - 4, for ER - 0, for OR - 7, for SR - 38, for CR - 32.

Any basic set of model indicators can be supplemented by the operator (administrator, etc.) of the described system, for example, by changing the template of the expert questionnaire for certain projects or by an expert, adapting a specific questionnaire for a specific project, or automatically by means of the described system as a result of data processing, as described in the framework of the present invention.

In a particular case, the described invention is an analytical tool for supporting an expert decision, and can be aimed at increasing the efficiency of financial investments in scientific and technological projects, and also allows you to find answers to the questions which of the projects are most ready for practical use in real sectors of the economy, and which have no prospects for commercialization in terms of the risks of their implementation and the possible financial costs of their implementation. The resulting set of quantitative estimates for projects ensures their ranking according to the degree of readiness for commercialization, investment attractiveness and priority in making managerial decisions.

The claimed invention was tested on scientific and technological projects carried out in various scientific and technical subject areas. At the same time, the invention can also be used in the management of scientific and technological projects and in other fields of science, technology and production, it can contribute to the innovative development of the research and development sector, ensure the competitiveness of the products and services created, as well as increase the efficiency of spending budget funds allocated for development. national scientific and technological sphere.

In conclusion, it should be noted that the information given in the description are examples, which do not limit the scope of the present invention defined by the claims. One skilled in the art realizes that there may be other embodiments of the present invention consistent with the spirit and scope of the present invention.

Claims (39)

1. A computer-implemented method of data processing for a comprehensive assessment of the maturity of a scientific and technological project, carried out using a computing device and containing the stages at which: - the formation of an array of information is carried out, compiled using the data contained in the requirements of regulatory documents and standards for the project execution processes, identifying and characterizing the maturity levels of projects and including a basic set of data, including a basic list of states, a basic list of tasks, a basic list of works, a basic list of evidence confirming the receipt of work results with the required quality by adding parameters that identify and characterize the maturity levels of projects; - the formation of an array of information is carried out containing the values of quantities characterizing the weights of each of the evidence for making a decision that the work has been completed in full with the required quality, and the values characterize the importance of each of the assessed evidence in comparison with other evidence; - the formation of templates of expert questionnaires from these data arrays of information is carried out; - a data set is formed from the data contained in the project documents; - the parameters of the initial and final values of the estimated parameters and criteria of the project maturity level are set, the timeline with the control points located on it and the predicted values of the project maturity index, the forecast of their change over the project execution time, which characterize the project, from the data of the generated data set, where forecast is the calculation of the parameters of the project on a time scale with the establishment of the fact whether the forecast is justified or not; - the quality of the facts extracted from the assessed documents is assessed using the set values of the assessment of the facts and the data of the documents of the project executors to assess the evidence confirming the receipt of the results, while the obtained assessment corresponds to the numerical value of the degree of confidence at which the evidence confirms the presence of the result in the project materials for the description of which it is presented by the project executor as part of the project documentation; - the calculated vector of the achieved states of the maturity level of the project is carried out, where the adjusted vector of the achieved states of the maturity level is formed after correcting the result of calculating the states using the interrelation matrix, moreover: - the value of the achieved state of the level of maturity of the project S _I for each of the projects is calculated by multiplying the quantitative values of the indicators T _J for each task, each of which characterizes the result of its solution and is calculated by multiplying all the values A _{M of} the work results in order to determine the completeness of the solution to the problem, where the M index takes values from 1 to N _TJ , and the value of A _M is equal to one for the performed work A _M if the value of W _{A is} greater than or equal to the value W of the threshold, or for the outstanding work A _M is equal to zero if the value of W _{A is} less than the threshold values W _threshold , for works from the list of works T _J | A = {A _M , М = 1, 2, ..., N _TJ }, aimed at achieving the appropriate readiness of the project - TR, MR, ER, OR, SR, CR, where TR - technological readiness, MR - production readiness, ER - engineering readiness, OR - organizational readiness of the project, SR - works to ensure project benefits and smoothing y project risks, CR - work to determine the market readiness and commercialization of project results, at a certain level of maturity L of the project, associated with the indicator T _{J of the} task, and evidence from the list of evidence of the results of their implementation A _M | E = {E _U , U = 1, 2, ..., N _AM } specified in the documents submitted by the project executor to establish the fact of the work performed, where the quantitative value of the work performance is established for the entire set of certificates included in it based on the comparison of the value W _A calculated by the formula W _A = Σ (W (E _U ) * Q (E _U )) / ΣW (E _U ) and allowing the assessment to conclude that the result of the work fulfills the previously specified requirements, with a threshold value W _threshold , where the evidence E _{U is} characterized by a given weight coefficient W (E _U ), the value of which characterizes the contribution of each evidence E _U for making an affirmative decision to obtain a result corresponding to the required quality, as well as e the relative importance of evidence of obtaining the required results, and if the result obtained meets the established requirements, then E _U = W (E _U ), if the quality of any of the assessed evidence does not meet the specified requirements, then the value of E _{U is} equal to the value W (E _U ) multiplied by the correction factor Q (E _U ) (0≤Q (E _U ) ≤1), and if there is no information about the certificate in the project materials, then the value of W _U is multiplied by the correction factor, the value of which is zero, and if the quality the submitted certificate satisfies the requirements, then the value of W _U is multiplied by a correction factor, the value of which is equal to one; - the values of the coordinates of the vector of achieved states are adjusted using the MM interrelation matrix, the elements of which determine the interrelation between the states of a given level of project maturity using these requirements of regulatory documents and standards for the processes of implementing innovative projects, and each element of the matrix is established based on the MM _I rule _{, J} = 1 / R, if the state S _I is associated with the state S _J , and zero, if not, and the value of R is equal to the number of all connections of the state S _I with other states, including itself, and to correct the value of the state vector A it is multiplied on the right by the MM matrix according to the rules of matrix multiplication, and if in the end each component of the vector S equals 1, then the current maturity level of the TPML project at this time has been reached, and if it coincides with the predicted value established in the contract or agreement for the implementation of the project , then the project is carried out in accordance with the the specified requirements and does not require prompt response measures; - the value of the maturity level of the project, TPML _L , is calculated using the decomposition of the project execution process as a system of states, tasks, works and evidence of the receipt of work results in the process of project execution, where each maturity level L of the project corresponds to a set of states and the corresponding quantitative values L | S = {S _I , I = 1, 2, ... N _L }, and the S _I values are equal to one ("1"), if the states are achieved, and zero ("0"), if not, and to achieve the indicated states by the project executors the corresponding problems must be solved, which are also assigned quantitative values, at least for the indicator S _I | T = {TJ, J = 1, 2, ..., NS _I }, and the T _J values are equal to one if the corresponding problem solved, and zero if not, and each task is divided into a series of activities aimed at achieving the appropriate readiness of the project - TR, MR, ER, OR, SR, CR, which are also assigned quantitative values, at least for tasks T _J | A = {A _M , M = 1, 2, ..., N _TJ }, and the values of A _M are equal to one if the work is done, and zero if not, and the result of these works is the results, which are also set in accordance with quantitative values, at least for the work A _M | E = {E _U , U = 1, 2, ..., N _AM }, and the evidence of obtaining the results are descriptions and evidence in combination with facts, E _U , which are presented executors of the project as part of documents on the progress of the project, where the values of E _{U are} also assigned quantitative values, and such data on the results obtained are a set of assessed evidence and the value of E _U depends on the quality of the E _U | D = {D _Y , Y = 1, 2, ..., N _EU } of supporting information about this certificate, moreover: - the value of the maturity level of the project TPML _L is equal to the level of maturity of the project L, if each calculated value of the achieved state of the level of maturity of the project S _I is equal to one, or equal to the value of L minus one otherwise; - the value of the project maturity index is calculated, and: - the total number of all states of the N maturity level is calculated, both achieved - NN, and not achieved, as the sum of all states of the N maturity level; - the value of the project maturity index I _{M of the} project is calculated according to the formula I _M = TPML _L -1 + NN / N; - the calculation of the values of the criteria K _TR , K _MR , K _ER , K _OR , K _SR , K _CR , determining the completeness of the work on the project, describing in the aggregate the balanced development of the project and aimed at achieving technological readiness TR; production readiness MR; ER engineering readiness; readiness of organizational support of the OR project; works to ensure project benefits and mitigate risks for the SR project; work to determine market readiness and commercialization of the results of the CR project; at a certain level of maturity at the time of the assessment of the level of maturity of the project, and: - the calculation of the values NN _TR , NN _MR , NN _ER , NN _OR , NN _SR , NN _CR , which determine the number of actually performed work on the coordinates of the corrected vector of achieved states S; - the adjustment of the T _J values is carried out, where if the S _I value is equal to one, then all the associated T _J values are equal to one, and if the SI value is zero, then all the associated T _J values are equal to zero; - the adjustment of the A _M values is carried out, where if the T _J value is equal to one, then all the A _M values associated with it are set equal to one, and if the T _J value is zero, then all the A _M values associated with it are set equal to zero ; - to calculate the completeness of the work related to one of the project parameter groups, the number of elements A _M is calculated, the value of which is 1 - (NN _TR , NN _MR , NN _ER , NN _OR , NN _SR , NN _CR ), and the total number of received elements A _M - N _TR , N _MR , N _ER , N _OR , N _SR , N _CR ; - the values of K _TR , K _MR , K _ER , K _OR , K _SR , K _CR are calculated as a result of dividing NN _TR , NN _MR , NN _ER , NN _OR , NN _SR , NN _CR by N _TR , N _MR , N _ER N _OR , N _SR , N _CR, respectively. 2. The method according to claim 1, in which the initial and final values of the estimated parameters and criteria of the project maturity level are formed from the current maturity level and from the project data contained in the bids, explanatory notes and project references. 3. The method according to claim 1, in which an additional set of assessed attributes is generated, which are set by the management body for a project or a group of projects to change project templates or by an expert to evaluate a project to change the questionnaire. 4. The method according to claim 1, in which, when carrying out an expert assessment of the quality of the certificates provided by the contractor about the results obtained during the project, the project data is used, while the expert questionnaire is changed by changing or expanding the basic list of states of the maturity level, the basic list of tasks, the basic a list of works, a basic list of certificates of obtaining the results of works, as well as a basic list of documents that must contain certificates. 5. The method according to claim 4, in which the values of quantities characterizing the weights of each of the certificates are added for the changed or added to the list of work results in order to make an affirmative decision on the performance of work with the required quality. 6. The method according to claim 1, in which the quality assessment of the facts extracted from the assessed documents includes an assessment of the quality of the facts about the results obtained during the project execution in the form of a questionnaire. 7. The method according to claim 1, in which the results of processing the results of calculations for the project are presented in one of the well-known formats of presenting data to the user, including the results of ranking projects, indices of maturity of projects, dynamics of changes in the index of maturity of the project, including in comparison with the predicted, characterizing completeness of the project the degree of completion of the work associated with the project parameters TR, MR, ER, OR, SR, CR. 8. The method according to claim 1, in which a set of recommendations for project executors is formed, and if during the said processing it was found that the development of the project lags behind the planned indicators, then recommendations are formed to the project executors aimed at increasing the efficiency of the work performed by them, containing the calculated quantitative values of the evaluated criteria for the maturity of the project. 9. The method according to claim 1, in which if the TPML _L value is not equal to the L value, then the calculation of the adjusted vector of the achieved states of the project maturity level is repeated, and if the TPML _L value is equal to the L value, then the fact of the plan coincidence with the fact at the moment is established time. 10. The method according to claim 1, in which if the value of the maturity index of the project I _{M is} less than the predicted value of the maturity index of the project I _plan , then management decisions are made, and if the value of the maturity index of the project I _{M is} greater than or equal to the predicted value of the maturity index of the project I _plan , then the fact of the normal development of the project is established. 11. The method according to claim 1, in which the project maturity index I _M is determined by the formula I _M = TPML _L -1 + NN / N, where N is the dimension of the vector S, and the value of the project maturity index 1 m determines the dynamics of the project at various time intervals for the project within one level of maturity of the project, and if its value coincides or exceeds the predicted value at a certain point in the time of the project execution, established in the documents for the implementation of the project, then the fact is established that the project is being carried out in accordance with the established requirements and does not require acceptance rapid response measures. 12. The method according to claim 1, in which if the value of any of the criteria K _TR , K _MR , K _ER , K _OR , K _SR , K _CR , determining the completeness of the work on the project associated with the corresponding group of project parameters and describing in the aggregate, balanced project development: works aimed at achieving technological readiness (TR), production readiness (MR), engineering readiness (ER), organizational readiness of the project (OR), work to ensure project benefits and mitigate project risks (SR), of work to determine the market readiness and commercialization of project results (CR) is less than one, then the work associated with the corresponding parameter is considered not completed, so it is necessary to take prompt interventions in the course of the project with the presentation of this fact to the user, and if the value of the criterion of what - either of the criteria K _TR , K _MR , K _ER , K _OR , K _SR , K _CR does not exceed a predetermined criterion value the project has a significant risk of non-implementation, which is the basis for making a decision to terminate the project. 13. The method according to any of the preceding paragraphs, which comprises: - formation of an array of data of parameters that identify and characterize the levels of maturity of projects, a basic list of indicators, a basic list of tasks, a basic list of works, a basic list of work results, a basic list of documents, and their input into the memory block of a computer-implemented system, - connecting to the database of the monitoring system for the progress of the project via wired or wireless communication channels and importing project documents from it for the subsequent assessment of the project status, - preparation and input into a computer-implemented system of predicted (planned) values of the estimated parameters - the level of maturity of the project, the index of maturity of the project, indicating the points in time at which the forecast should be set, - expert assessment of the quality of the evidence presented by the project executor in the project documents, and the input of the obtained assessments into a computer-implemented system, while the obtained assessment corresponds to the numerical value of the degree of confidence determined by the expert, with which it can be argued that the certificate confirms the presence of the result in the project materials, for the description of which it is presented by the project executor as part of the project documentation; - processing of the data array according to the relevant criteria, followed by the evaluation of the processing result for each criterion with the assignment of a value from a given range of values. 14. A data processing system for a comprehensive assessment of the maturity of a scientific and technological project, containing at least one computing device, which is configured to execute the method according to any of the preceding claims.

Images