KR20090002243A - Evaluation method for r&d projects supported by government - Google Patents

Abstract

A method of analyzing results of government supported research and development projects is provided to carry out a survey for obtaining quantitative and qualitative indexes required to evaluate results of companies and research institutes receiving funds from the government and analyze results of research and development projects based on the survey result. A method of analyzing results of government supported research and development projects includes a first step(120) for collecting data of a survey about an individual research and development project, a second step(101) for grasping relation among a fund related factor, a management environmental factor and a result evaluation factor and calculating a composite index by using the data, a third step(111) for detecting factors which largely affect technical and economical results of the project, a fourth step(102) for producing an efficiency index for the individual project, a fifth step(112) for selecting a case which shows s fruitful result as an object of bench marking, a sixth step(103) for performing analysis for detecting a factor which largely affects the efficiency index, and a seventh step(104) for providing information detecting a factor which affects the result of a following research and development project.

Description

Evaluation method for R & D Projects supported by Government}

1 is a view showing a management procedure to be performed for the performance evaluation of research and development projects,

2 is a view showing a summary of the overall analysis process of the performance evaluation system of the R & D project,

3 is a view showing a time difference between the results of research and development projects,

FIG. 4 is a diagram showing an example of a structural equation model for estimating the relationship between the fund-related factors, management environment factors, and performance evaluation factors and the composite index.

5 is a diagram showing an example of EMS software used to calculate an efficiency index for an individual R & D project.

FIG. 6 is a diagram illustrating an example of a decision tree analysis result for grasping characteristics of R & D projects showing high performance.

<Description of the symbols for the main parts of the drawings>

201: Survey 202: Structural Equation Model

203: Data Envelopment Analysis 204: Decision Tree Analysis

The present invention relates to a performance analysis method of a government-funded R & D project, and more specifically, from the quantitative and qualitative fund-related factors, management environment factors, and performance evaluation factors for corporations and university research institutes that receive funds from the government. In addition, the government can not only measure economic performance, but also improve the beneficiary's performance by identifying factors that can analyze the efficiency of funds and determine success / failure for companies and research institutes. It relates to the performance analysis method of supporting R & D projects.

The policy fund projects currently being promoted by the government are carried out in various fields including agriculture, finance, and information and communication, and the scale of each policy project is very large, reaching hundreds of billions.

However, in the case of poor performance of research activities carried out by supported companies and university research institutes, not only will there be a loss of funds, but also the opportunity costs that are not invested in other promising R & D projects. Industrial losses may occur.

Therefore, even though accurate performance evaluation must be carried out for the effective operation of the funds supported by the policy fund project, a systematic performance evaluation frame is not established.

Looking at the problems of the existing performance analysis in detail, divided into short-term (output), medium-term (outcome), long-term performance (impact) according to whether the current research and development has been completed, and how long has passed after completion Performance should be assessed, but this time gap is not fully taken into account.

In addition, the existing factors considered for the performance evaluation, using the SCI paper publication number, patent registration results, etc. to grasp the degree of the performance, grouping a variety of factors having a similar nature as a composite index (composite) There is no performance evaluation to calculate the index).

And although there is a relationship between the fund management factors, management environment factors, and performance evaluation factors that are considered for the performance evaluation, detailed analysis is not carried out, and in the performance analysis, Since the process of calculating weights is arbitrary, accurate performance analysis is difficult.

Lastly, the analysis that can be used to select a target company for a new business through the strict evaluation of past completion tasks is currently insufficient, and furthermore, the hierarchical relationship between the explanatory factors of the above performance analysis is not known.

The present invention has been made in view of the above, and has the following objects.

1. Surveys were conducted to obtain qualitative and quantitative indicators necessary for the performance evaluation of companies and research institutes that have received or received funding through policy funding projects. To be the basis for efficient R & D operation.

2. Find out the relationship among the fund-related factors, management environment factors, and performance evaluation factors surveyed from the questionnaire, and calculate a composite index grouping similar factors, which can be used as feedback information to improve performance. .

3. By calculating efficiency indices of individual R & D projects of companies and research institutes that have received or received funding through policy funding projects, they will be able to benchmark high-performance projects and achieve high performance. To provide useful information.

4. Based on survey data on R & D projects of companies and research institutes that have received or received funding through policy funding projects, identify factors influencing the success and failure of individual projects in selecting future projects. Extract success factors that should be emphasized.

5. As a data analysis technique for realizing the various analyzes, a structural equation, data envelopment analysis, and decision tree analysis are used in combination.

In order to achieve the above object, the present invention provides a method for analyzing performance of a government-supported R & D project.

A first step of collecting survey data on individual research and development projects; A second step of identifying the relationship between the fund-related factors, the management environment factors, and the performance evaluation factors and calculating the composite index using the survey data on the R & D project; A third step of identifying factors that have the greatest influence on the technical and economic performance of the task; A fourth step of dividing the composite indexes calculated in the step into input and output elements to calculate an efficiency index for each task; A fifth step of selecting an example of high performance among research and development tasks as the target of benchmarking through the efficiency index; A sixth step of carrying out an analysis for identifying factors influencing the efficiency index; And the seventh step of providing information that can determine the factors influencing the success or failure of future R & D projects through the analysis.

In a preferred embodiment, the survey in the first step may include a first step of designing a fund-related factor, a management environment factor, and a performance evaluation factor for a research and development project for which performance evaluation is to be performed; A second step of defining how to measure factors designed for each research field; And a third step of conducting a questionnaire survey on the research and development task.

In a more preferred embodiment, the factors considered for the performance evaluation in the first step are to identify financial factors and unique indicators that reflect the characteristics of each sector through discussion with existing literature, examples and practitioners. It is characterized by.

In addition, the second step is to measure not only the quantitative indicators such as the number of SCI article publications and patent registrations, but also how to measure qualitative factors such as current technology level, R & D leadership, and R & D strategic plan. It is characterized in that the step of determining whether.

In addition, the third step is characterized by performing a questionnaire for performance evaluation by categorizing into three categories, short-term, medium-term and long-term results according to the completion of the R & D project and the period after completion.

In addition, the second step is characterized by calculating the composite index by identifying the relationship between the fund-related factors, management environment factors, performance evaluation factors and grouping factors through the structural equation model.

In addition, the structural equation model is an input factor such as support amount for each project, project period, etc., environmental factors such as technology development, physical and human resources, and R & D management, and short-term direct technical performance of the companies supported. It is characterized by consisting of long-term performance that can be said to be the national ripple effect measured in terms of performance, the economic performance derived from the above technical performance, medium-term.

In addition, the third step is characterized by finding the factors that have a significant impact on the technical and economic performance and the ripple effect through the structural equation model and to provide feedback information to improve the individual performance.

In addition, in the fourth step, data envelopment analysis is performed to measure the efficiency index for the individual task.

In addition, the data envelopment analysis is characterized by being performed by dividing the performance-related factors such as short-term, medium-term and long-term performance as output elements, and the factors related to funding and management environment among the composite index.

In addition, the fifth step is characterized in that to select the task showing the efficient performance based on the efficiency index for each task calculated through the data envelopment analysis as the target of benchmarking.

In addition, the sixth step utilizes the efficiency and inefficiency of the research results of individual companies and university research institutes identified through the efficiency index, and the financial and non-financial information of the companies. Identify the factors that most closely affect the

In addition, the decision tree analysis is performed by using the efficiency index for each task calculated through data envelopment analysis as a dependent variable, and the characteristic variable (financial / non-financial) for the task as an independent variable (input variable). do.

In addition, all the research and development tasks are defined as success when the value of the efficiency index calculated through the data envelopment analysis before the decision tree analysis exceeds a certain value and failure when the value below the predetermined value is defined as failure. They are divided into successes or failures, and then the analysis is performed.

In addition, the criteria for collecting performance data for performance analysis of individual tasks are collected only for short-term performance (technical performance) for current R & D projects, and for tasks that have not been 3 years since completing the above tasks. It collects data on short-term and medium-term performance (economic performance), and collects data on short-term, medium-term and long-term performance (ripple effects) for tasks that are three years after completing the above task. It is done.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

First, call the company and university research institutes who have experiences or are currently receiving funding through the policy funding project, and then call the web and post at the convenience of the target company and university research institute. Send and retrieve questionnaires by email, fax, or e-mail.

The questionnaire includes financial information such as capital, sales, and operating profit margins, as well as non-financial information such as company history, number of employees, manager age, industrial classification, etc. It should also include information such as the number of SCI papers published and the level of technology being developed.

The analysis is performed using a DB that stores the results of responses to these questions.

The performance analysis method according to the present invention has three logical processes, and each detailed component is as follows.

First, in order to establish a performance measurement model from the collected data, it is necessary to extract various factors such as fund-related factors, management environment factors, and performance evaluation factors, and to use the structural equations to identify relationships among factors and to calculate complex indices. Stage 1; Data Envelopment Analysis (203) analyzes the efficiency of the research results of individual companies and university research institutes that received funds by dividing the composite index calculated in the first step into input and output, respectively. A second step of identifying through; Decision tree analysis to determine the factors that most closely affect the efficiency among beneficiaries by utilizing the financial and non-financial information of the companies on the efficiency / inefficiency of the research results of individual companies and university research institutes identified in the second step. (204) can be divided into a third step to identify through the (Decision Tree).

In particular, in the first stage, the data required for the performance evaluation of R & D projects is currently under R & D or completion of R & D, depending on the research field (information and communication, science and technology, agriculture, medicine, etc.) currently being developed. The survey should be completed differently depending on how many years have passed.

When the above mentioned analyzes are carried out, not only how much the individual R & D projects have achieved, but also the factors that have the greatest influence on the relationship and the performance among the factors considered for the performance evaluation of the individual projects. This can provide information on what factors should be focused on when screening for future research and development projects.

These analyzes allow for more efficient management of R & D funds.

Hereinafter, with reference to the accompanying drawings, the configuration and operation of the embodiment of the present invention will be described in detail.

1 is a sequence of the effects that can be obtained in performing the performance evaluation of the research and development of companies and institutes that have received or received funding through the policy funding business when the performance evaluation method proposed in the present invention is introduced. I explain it as it is.

The figure includes a process of collecting data 120 of individual R & D projects and presenting a strategy for efficient R & D funding by performing performance evaluation using the above data.

However, most of the studies that are currently performing performance analysis are only statistical analysis such as the number of SCI paper publications, patent registrations, and sales growth.

In order to compensate for these drawbacks, in the present invention, fund-related factors, management environment factors, and performance evaluation factors using the survey data 120 for R & D projects that have completed R & D or are currently being funded. Identify relationships and estimate composite indexes (101) and identify the factors that have the greatest impact on technical and economic performance (111).

In addition, by dividing the composite index calculated in step 101 into the input and output to calculate the efficiency index for the individual task (102).

The efficiency index 102 selects high performing cases of R & D projects as benchmarking targets to provide information for inefficient tasks to produce more efficient results (112). .

In addition, an analysis is conducted to identify factors that have a significant effect on the efficiency index (103).

Through such analysis, it is possible to provide efficient information for the development of R & D fund by providing information to identify factors influencing success or failure of future R & D projects (104).

2 shows a detailed analysis process of the performance evaluation method.

In the present invention, the analysis is performed based on the data obtained through the survey 201 of the research and development projects of companies and institutes that have received or received funding through policy funding projects.

In addition, as described above, the entire analysis process is largely conducted by identifying the relationship between the fund-related factors, the management environment factors, and the performance evaluation factors using the Structural Equation Model 202, and calculating the composite index, and the individual R & D tasks. It consists of three parts: data envelopment analysis (203) for efficiency index calculation and decision tree analysis (204) for success / failure factor analysis.

The first step of the survey 201 is a step 211 of designing the fund-related factors, management environment factors, and performance evaluation factors for the research and development project to perform the performance evaluation.

The R & D projects currently undertaken by companies and research institutes that have received or received funding through policy funding projects vary widely from information and communication, science and technology, medicine, agriculture and fisheries. Although the factors considered are common factors such as financial factors, it is necessary to identify the unique indicators reflecting the characteristics of each sector through discussions with practitioners as well as the existing literature and examples.

The second step is to define measurement indicators for fund-related factors, management environment factors, and performance appraisal factors designed for each research area (212), and to define how to measure the above factors.

There are also quantitative indicators such as the number of SCI papers published and patents registered at this stage, but how to determine how to measure qualitative factors such as the current level of technology, R & D leadership, and R & D strategic planning. Step.

The final step of the survey 201 is a step (213) to perform the survey for the research and development project composed of the questionnaire.

The survey method is to send and retrieve the questionnaire by web page, mail, fax, and e-mail at the convenience of the phone call with the R & D project manager.

When conducting a survey, some tasks are currently under research and development, and some have been completed several years.Therefore, depending on the completion of the research and development project to be surveyed and the time period after completion of the survey, the short-term and output ), And long-term performance (impact), the questionnaire for performance evaluation should be conducted.

Detailed criteria for this can be found in FIG. 3.

The survey data collected through the criteria defined in FIG. 3 include various fund-related factors, management environment factors, and performance evaluation factors, and these evaluation indicators identify the relationship between the above factors through the structural equation model 202. In addition to grouping factors, grouping factors is used to calculate the composite index.

The factors constituting the structural equation model 202 are input factors such as support amount, project period, etc. for each task, as well as environmental factors such as technology development, material / human resources, and R & D management. It consists of short-term performance, which is the direct technical performance of companies received, medium-term economic performance derived from such technical performance, and long-term performance, which is the national ripple effect measured in the macroscopic aspect.

Details of the structural equation model 202 can be found in FIG. 4.

The next step is to perform data envelope analysis (203) to measure the efficiency of individual tasks.

For the data envelopment analysis (203), to use the composite index calculated through the structural equation model (202).

Among these composite indexes, fund-related factors and management environment factors are called inputs, and performance evaluation factors such as short-term, medium-term, and long-term outcomes are classified into outputs.

The efficiency index for each task calculated through the data envelopment analysis 203 may be used as a dependent variable of the decision tree analysis 204, which is the final step in the performance evaluation analysis.

Details of the data envelopment analysis 203 can be found in FIG. 5.

The final step of FIG. 2 is a decision tree analysis in which the efficiency index for each task calculated through the data envelope analysis 203 is used as a dependent variable, and the characteristic variable (financial / non-financial) for the task is an independent variable (input variable). (204).

Prior to performing the decision tree analysis 204, the case where the value of the efficiency index calculated through the data envelopment analysis 203 exceeds 100 is defined as success, and the case where the value does not exceed 100 is defined as failure. Divide the development tasks into successes or failures and then perform the analysis.

This analysis makes it easy to identify the causal factors among many factors that influence the success and failure of a task, and thus, it is possible to extract effective success and discrimination factors that should be focused on future task selection. This makes it easier to screen for R & D projects that show a high performance.

3 is a time difference between R & D results because individual tasks have different completion periods after completion and completion of individual tasks when performing performance analysis on R & D tasks of companies and research institutes funded through policy funding projects. The short-term, mid-term, and long-term performances are classified into the following categories, and data for performance analysis is collected.

More specifically, data on short-term outcomes (technical achievements) can be collected for R & D projects that are currently under development or that have not been three years since they were completed.

In addition, data on medium-term outcomes (economic performances) can be collected for assignments within 5 years after completion of R & D.

In addition, three years after the completion of R & D, data on long-term performance (ripple effect) can be collected.

The criteria for collecting performance data for performance analysis on individual tasks considering these time differences are as follows.

Only collect data on short-term outcomes (technical outcomes) for ongoing projects; short-term outcomes (technical outcomes) as well as medium-term outcomes (economic outcomes) Collect data.

Finally, for tasks that are three years old after completing the task, collect data on short-term performance (technical performance), medium-term performance (economic performance) as well as long-term performance (ripple effects).

FIG. 4 is an example of a structure using the structural equation model 202 as a structural relationship between various factors considered for performance evaluation of R & D tasks of companies and research institutes funded through the policy funding project.

This figure shows the structural relationship between the Criteria, which should be considered for performance evaluation in MBNQA, by using the structural equation model 202. The factor on the right is the ripple effect, which is the long-term performance of the task. The factor on the left represents the economic performance, which is medium-term, and the factor on the left of the economic performance represents the technical performance.

The factors at the top left represent the company's management factors, and the factors at the bottom left represent the financing support factors, and the factors at the bottom right represent R & D management factors of the support organization in charge of the project.

Each of these factors includes several measurement variables underneath.

Figure 5 shows the results of analyzing the data envelope analysis 203 using the EMS software for the calculation of the efficiency index for the R & D individual tasks of companies and research institutes funded through the policy fund project.

Looking at the drawings in detail, the DMU (Decision Making Unit) represents an individual task to calculate the efficiency index.

For example, if there are 100 R & D projects to calculate the efficiency index, the number of DMUs is 100. Score is an efficiency index for an individual DMU.

When the score is 100%, the DMU is efficient, and other DMUs that are not colored are inefficient R & D tasks.

Benchmarks also indicate which DMUs are benchmarked so that individual DMUs can be efficient units. For example, DMU 1 means that you should refer to DMUs 4, 6, and 7 to increase efficiency.

6 is an example of the results of a decision tree analysis to identify significant variables that affect the performance analysis of R & D projects of companies and research institutes funded through policy funding projects.

Looking at the drawings in detail, the efficiency index calculated through the data envelope analysis (203) divides individual tasks into efficient tasks (1) and inefficient tasks (0), and describes general characteristics of companies and research institutes funded by explanatory variables ( H10, H6, L1, F3, M4) to consider the analysis.

For example, among the research and development projects, the value of the explanatory variable H10 is less than 5, the value of the explanatory variable H6 is less than 5, and the value of the explanatory variable F3 is less than 4 has been investigated. Among the 30 tasks, 30 tasks can be categorized. Among them, 21 companies and research institutes that performed R & D tasks efficiently. Through this classification model, it is possible to grasp the characteristics of companies and research institutes that are carrying out efficient R & D projects in the future.

While the invention has been shown and described with respect to certain preferred embodiments thereof, the invention is not limited to these embodiments, and has been claimed by those of ordinary skill in the art to which the invention pertains. It includes all the various forms of embodiments that can be carried out without departing from the spirit.

As described above, according to the performance analysis method of the government-supported research and development project according to the present invention, the performance analysis on the research and development projects of the companies and university research institutes that received funds through the policy fund project promoted by the government In this study, not only the important factors affecting this can be identified using the structural equation model, but also the performance evaluation factors measured in consideration of various aspects can be provided as a composite index to provide feedback information for future performance improvement. have.

In addition, data envelopment analysis of individual R & D projects is conducted to evaluate the efficiency of individual projects and to select the most efficient cases to be the target of benchmarks.

Lastly, by applying various input and output variables used for data envelopment analysis to decision tree analysis, success factor analysis can be provided, which can provide information to enable efficient management of policy funding projects at the government level.

Claims (10)

In the performance analysis method of the government-funded research and development project, A first step 120 of collecting survey data on individual R & D tasks; A second step (101) of using the survey data on the R & D project to determine the relationship between the fund-related factors, management environment factors, and performance evaluation factors and to calculate a composite index; A third step (111) of identifying factors that have the greatest influence on the technical and economic performance of the task; A fourth step (102) of calculating efficiency indexes for individual tasks by dividing the composite indexes calculated in the step into inputs and outputs; A fifth step (112) of selecting a case showing a high performance among the R & D tasks through the efficiency index as an object of benchmarking; A sixth step (103) of performing an analysis for identifying a factor that has a significant influence on the efficiency index; A seventh step 104 of providing information capable of determining factors influencing success or failure of future R & D projects through the analysis; Performance analysis method of the government-funded research and development project, characterized in that it comprises a. The method of claim 1, wherein the survey of the first step A first step (211) of designing fund-related factors, management environment factors, and performance evaluation factors for R & D tasks to be performed; A second step (212) defining how to measure funding factors, management environment factors, and performance evaluation factors designed for each research field; A third step (213) of conducting a questionnaire survey on the research and development task; Performance analysis method of the government-funded research and development project, characterized in that it comprises a. The method according to claim 1, In the second step, the government-funded R & D project is characterized by using the structural equation model (202) to determine the relationship between fund-related factors, management environment factors, and performance evaluation factors, and grouping the factors. Performance analysis method. The method according to claim 3, The structural equation model 202 includes fund-related factors such as support factors for individual projects, project periods, management environment factors such as technology development, physical and human resources, and R & D management, and direct support of companies supported. A performance analysis method of government-funded R & D projects with performance evaluation factors consisting of short-term performance as a technical performance, medium-term performance as an economic performance derived from the above technical performance, and long-term performance as a national ripple effect measured in the macroscopic aspect . The method according to claim 1, And performing data envelopment analysis (203) to measure the efficiency index for each individual task in the fourth step. The method according to claim 5, The data envelopment analysis 203 is characterized in that the performance-related factors, such as short-term, medium-term and long-term performance, are classified as output elements, and the factors related to funding and management environment are among the composite indexes. Performance analysis method. The method according to claim 1, The sixth step is based on the decision tree analysis (204). A method of performance analysis of a government-funded R & D project characterized by identifying the factors that most closely affect efficiency. The method according to claim 7, The decision tree analysis 204 is performed using the efficiency index for each task calculated through the data envelopment analysis 203 as a dependent variable and the characteristic variable (financial / non-financial) for the task as an independent variable (input variable). Performance analysis method of the government-funded research and development project, characterized in that. The method according to claim 8, Before performing the decision tree analysis 204, the case where the value of the efficiency index calculated through the data envelopment analysis 203 exceeds a predetermined value is defined as a success, and a case below the predetermined value is defined as a failure, A method for performance analysis of a government-funded R & D project characterized by performing an analysis after dividing all R & D tasks by success or failure. The method according to claim 1, Criteria for collecting performance data for performance analysis of individual projects collect only data on short-term performance (technical performance) for current research development projects, and short-term for projects not more than three years after completing the above tasks. It collects data on performance and medium-term performance (economic performance), and collects data on short-term, medium-term and long-term performance (ripple effects) for tasks that are three years after completing the above tasks. Performance analysis of government-funded R & D projects.

Images