KR101969312B1 - Evaluation apparatus for research project, and control method thereof - Google Patents

Abstract

The present invention relates to a research proposal evaluation device and an operation method thereof which evaluate a research proposal target to be evaluated through a method of machine learning quantitative data and may further provide feedback on an improvement method for a result of the evaluation.

Description

Evaluation plan evaluation device and its operation method {EVALUATION APPARATUS FOR RESEARCH PROJECT, AND CONTROL METHOD THEREOF}

The present invention relates to a method of evaluating the evaluation target research plan through the method of machine learning quantitative data, and also to feedback the improvement method for the evaluation result.

In order to maximize the success rate of technology commercialization, it is important to establish a structured research plan at the planning stage, which is the first stage of the life cycle process of technology commercialization.

Therefore, governments, public institutions, corporations, universities, and research institutes that support researchers' research and development are making great efforts to select structured research proposals in order to enhance the efficiency of support and promote the success of technology commercialization. to be.

In this regard, the research proposal selection process involves the process of selecting a research plan with a high evaluation score by evaluating the research proposals prepared by the researchers who will conduct the research.

Until now, a qualitative method based on peer evaluation, in which a small number of experts in the field participate as evaluators, evaluates the proposal.

In this qualitative method, the point of evaluation, namely, the composition of the evaluation index and the weighting for each evaluation item may be different depending on the type and purpose of the research, but ultimately, the success and success of R & D and technology commercialization can be obtained. Judgment is based on expected performance.

However, since the success and the expected outcome are determined in the future, it is very difficult to objectively evaluate the research proposal using the qualitative evaluation method based on peer evaluation which has been mainly used up to now.

This is because subjective judgment based on evaluator's intellectual level and experience is necessarily reflected, which may lead to incorrect evaluation results in some cases.
As a background technology of the present invention, Republic of Korea Patent Publication No. 10-2005-0011590 (January 29, 2005), Republic of Korea Patent Publication No. 10-2016-0121042 (October 19, 2016), Republic of Korea Patent Registration 10-1170290 (registered on July 25, 2012), Republic of Korea Patent Publication No. 10-1300193 (registered on August 20, 2013), Republic of Korea Patent Publication No. 10-1718964 (registered on March 16, 2017).

The present invention has been created in view of the above circumstances, and an object of the present invention is to provide a consulting method for evaluating an evaluation target research plan through a machine learning method for quantitative data and feedback on an improvement method for the evaluation result. To provide an environment.

In the study plan evaluation apparatus according to an embodiment of the present invention for achieving the above object, when the study subject research information related to the study target evaluation target is obtained, the study target study plan for the learning subject research information A setting unit configured to set a performance item that matches the R & D performance item and an attribute item that may be related to an attribute value of the performance item; Based on the results of the machine learning of the subject study subject information, a core attribute that is at least some attribute item having a higher relevance to the performance item than other attribute items among the attribute items that may be related to the attribute value of the performance item. A decision unit to determine a performance estimation rule from which the performance item is derived based on an item and the core attribute item; And a predicting unit extracting attribute values of the attribute items matching the core attribute items from the evaluation target research plan, and substituting the extracted attribute values into the performance estimation rule to predict the expected performance of the R & D performance items. Characterized in that.

Specifically, when the research plan evaluation apparatus is required to detect abnormality of at least one of the attribute items and the performance items in the evaluation target study plan, the machine-learned study subject information and the evaluation target study plan are machine-learned. Based on the result, the performance item whose difference between the attribute value and the predicted value is greater than or equal to the threshold is detected as an anomaly, or the attribute item or the performance item whose difference between the attribute value and the statistical value is greater than or equal to the threshold is an abnormal item. The apparatus may further include an abnormality detecting unit.

Specifically, the research proposal evaluation apparatus, when consulting on the evaluation target research proposal, is required to improve the expected expected performance for the R & D performance items, the research target evaluation according to the machine learning results It may further include a consulting unit that selects an attribute item that needs to be changed among the attribute items matching the core attribute items in the plan as the change target item and provides the feedback information.

In detail, the change target item may be selected based on a high priority of relevance between the key attribute items matched in the study target task information and the performance item.

Specifically, the change target item is a degree to which the relative position of the attribute value in the evaluation target study plan is out of the center of the distribution with respect to the attribute value distribution of the matching core attribute items in the study target project information. Can be selected based on a large priority.

Specifically, the setting unit may set at least some of the property items posted in the study subject information as the property items as it is, or a ratio of attribute values between different property items posted in the study subject information. May be set as the attribute item.

Specifically, the study subject study information may include existing study subject information in which similarity between attribute values for each attribute item with the evaluation target study plan is identified as a threshold value or more among the existing study subject information.

According to an embodiment of the present invention, a method of operating a research plan evaluation device according to an embodiment of the present invention may include: evaluating the subject study information when learning subject study information related to an evaluation subject study plan is obtained. A setting step of setting a performance item matching the R & D performance item of the target research plan and an attribute item that may be related to the property value of the performance item; Based on the results of the machine learning of the subject study subject information, a core attribute that is at least some attribute item having a higher relevance to the performance item than other attribute items among the attribute items that may be related to the attribute value of the performance item. A decision step of determining a performance estimation rule for deriving the performance item based on an item and the core attribute item; And extracting an attribute value of the attribute item matching the core attribute item from the evaluation target research plan, and substituting the extracted attribute value into the performance estimation rule to predict an expected outcome for the R & D performance item. It is characterized by including.

Specifically, the method, when detecting the abnormality of at least one of the attribute item and the performance item in the evaluation target research plan, prior to the determination step, the subject study information and the evaluation target study plan Based on the results of the machine learning, the performance item whose difference between the attribute value and the predicted value is greater than or equal to the threshold value is detected as an anomaly, or the attribute item or the performance item whose difference between the attribute value and the statistical value is greater than or equal to the threshold value is selected. The apparatus may further include an abnormality detecting step of detecting the abnormality item.

Specifically, the method, in connection with consulting on the target research plan, is required to improve the expected outcome for the R & D performance item, and according to the machine learning result, the core of the target research plan The method may further include a consulting step of selecting an attribute item that needs to be changed among the attribute items matching the attribute item as a change target item and providing it as feedback information.

In detail, the change target item may be selected based on a high priority of relevance between the key attribute items matched in the study target task information and the performance item.

Specifically, the change target item is a degree to which the relative position of the attribute value in the evaluation target study plan is out of the center of the distribution with respect to the attribute value distribution of the matching core attribute items in the study target project information. Can be selected based on a large priority.

In detail, the setting may include setting at least some of the property items posted in the study subject information as the property items or having attribute values between different property items posted in the study subject information. A ratio may be set as the attribute item.

Specifically, the study subject study information may include existing study subject information in which similarity between attribute values for each attribute item with the evaluation target study plan is identified as a threshold value or more among the existing study subject information.

Therefore, in the research proposal evaluation apparatus and its operation method of the present invention, through the method of machine learning quantitative data, the success of R & D and technology commercialization of the evaluation target research proposal and the expected performance expected to be obtained upon success are obtained. By proposing a plan proposal that can be objectively judged, value can be created in a way that supports the R & D support agency's rational decision-making on research plan evaluation and saves time and costs.

In addition, the research proposal evaluation apparatus and its operation method of the present invention, based on the evaluation results of the evaluation target research proposal, a method for improving the success of R & D and technology commercialization and the expected performance expected to be obtained upon success. Feedback can be advised to the researcher who created the research proposal, thus creating new value for the researcher by consulting the researcher on how to organize the research proposal.

1 is a schematic diagram illustrating a research proposal evaluation system according to an embodiment of the present invention.
Figure 2 is a block diagram for explaining the configuration of the research proposal evaluation apparatus according to an embodiment of the present invention.
Figure 3 is an exemplary view for explaining the results of the machine learning according to an embodiment of the present invention.
Figure 4 is a flow chart for explaining the process of predicting the expected performance of the R & D performance item according to an embodiment of the present invention.
Figure 5 is a flow chart for explaining the process of detecting the abnormality of the content of the evaluation target research plan according to an embodiment of the present invention.
6 is a flowchart illustrating a consulting process according to an embodiment of the present invention.

Hereinafter, with reference to the accompanying drawings will be described an embodiment of the present invention.

1 shows a research proposal evaluation system according to an embodiment of the present invention.

As shown in FIG. 1, the research plan evaluation system according to an embodiment of the present invention includes an information providing apparatus 10 for providing existing research project information, and a research plan evaluation apparatus 20 for evaluating an evaluation target research plan. It may have a configuration that includes).

The information providing device 10 refers to a device for providing information on an existing research project related to a research target to be evaluated. For example, the information providing device 10 may correspond to a national science and technology information system.

In the information providing device 10, a total of 422 kinds of research tasks, for example, basic information, performance information, participating personnel information, evaluator information, research facilities and equipment information, etc., for each previously performed research project according to the national R & D information standard notification. Research project information is divided into two categories, which can be classified into attribute and performance items according to their characteristics.

For reference, the attribute items include the nature and form of the support subject of the research development fund, the size and composition of the research development fund, the field to which the technology belongs, the composition of the nature and participation of the subject, the research period, the research objectives and the contents, and the expectations. Effects, and other items that determine the nature of research and development.In addition, the performance items include project evaluation results, manpower training results, academic achievements such as papers, patents, reports, and academic presentations, securing intellectual property rights, technology transfer, It may include items that indicate the achievements of commercialization of technology such as startups, the possibility of commercialization success, and other R & D achievements.

The research plan evaluation device 20 refers to a device for evaluating a research target to be evaluated through a method of machine learning quantitative data based on existing research project information. PC) or a server to which such a computer is connected.

When the research document evaluation device 20 is implemented in the form of a server, for example, it may be implemented in the form of a web server, a database server, a proxy server, or the like. One or more of the various software programs that can operate on a network may be installed, and may also be implemented as a computerized system. In addition, the network may be an http network, and may be a private line, an intranet, or any other network, and furthermore, the connection between each component in the system according to an embodiment of the present invention may be used by any hacker or It can be connected to a secure network so that it is not attacked by another third party.

As a result, the research plan evaluation system according to an embodiment of the present invention may evaluate the study target evaluation plan through a method of machine learning quantitative data based on the existing research project information based on the above-described configuration.

On the other hand, as a conventional method of evaluating the proposal, a qualitative method based on peer evaluation, in which a small number of experts in the field participate as an evaluator, evaluates the proposal.

In this qualitative method, the point of evaluation, namely, the composition of the evaluation index and the weighting for each evaluation item may be different depending on the type and purpose of the research, but ultimately, the success and success of R & D and technology commercialization can be obtained. Judgment is based on expected performance.

However, since the success and the expected outcome are determined in the future, it is very difficult to objectively evaluate the research proposal using the qualitative evaluation method based on peer evaluation which has been mainly used up to now.

This is because subjective judgment based on evaluator's intellectual level and experience is necessarily reflected, which may lead to incorrect evaluation results in some cases.

In addition, electronic systems that have been developed to support these qualitative evaluation methods are more than simply conducting offline meetings to store, aggregate, and manage qualitative evaluations by using managers and evaluators online. It is a level that helps to proceed quickly and easily, and it cannot be a technology that solves the fundamental limitations of qualitative evaluation.

On the other hand, methods and systems for pre-feasibility evaluation of the technology commercialization plan have been provided based on prior patent analysis data, technology trend analysis data, market trend analysis data, etc. However, it is not possible to solve the fundamental limitations of qualitative evaluation in that the evaluation indicators and the evaluation process developed artificially should be focused on qualitative evaluation by experts.

 Therefore, in an embodiment of the present invention, it is possible to evaluate a research target to be evaluated through a method of machine learning quantitative data rather than a qualitative method, and further, to propose a new method for feeding back a method for improving the evaluation result. Hereinafter, the configuration of the research plan evaluation device 20 for realizing this will be described in more detail.

Figure 2 shows the configuration of the research proposal evaluation device 20 according to an embodiment of the present invention.

As shown in FIG. 2, the research plan evaluation apparatus 20 according to an embodiment of the present invention includes a confirmation unit 21 for confirming an evaluation target research plan and an acquisition unit 22 for obtaining study target study plan information. , The setting unit 23 for setting attribute items and performance items for the study plan information to be studied, the learning unit 24 for processing machine learning for the study plan information to be studied, and the items and rules based on the machine learning results are determined. Decision unit 25 may include, and the prediction unit 26 for predicting the expected performance for the R & D performance items.

In addition, the research plan evaluation apparatus 20 according to an embodiment of the present invention, in addition to the above configuration, the abnormality detection unit 27 for detecting an abnormality item, and the consulting unit 28 for providing feedback for improving the prediction result. It may have a configuration that further includes.

The verification unit 21, the acquisition unit 22, the setting unit 23, the learning unit 24, the determination unit 25, the prediction unit 26, the abnormality detection unit 27, and the consulting unit 28. The entire configuration or at least a part of the research proposal evaluation device 20 including a may be implemented in the form of a hardware module or a software module, or may be implemented in the form of a combination of a hardware module and a software module.

Here, the software module may be understood as, for example, an instruction executed by a processor that controls an operation in the proposal evaluation apparatus 20, and the instruction may be a form mounted in a memory in the proposal evaluation apparatus 20. I can have it.

On the other hand, the research plan evaluation device 20 according to an embodiment of the present invention, in addition to the above-described configuration of the communication unit 29 which is an RF module that is responsible for the actual communication function with a remote device including the information providing device 10. The configuration may further include.

Here, the communication unit 29 may include, but is not limited to, for example, an antenna system, an RF transceiver, one or more amplifiers, tuners, one or more oscillators, a digital signal processor, a codec chipset, a memory, and the like. Any known circuit to be performed may be included.

The research plan evaluation apparatus 20 according to an embodiment of the present invention predicts the expected performance of the R & D performance item of the evaluation target research plan based on the above-described configuration, and further, improves the prediction result. Feedback may be provided, which will be described below in more detail with respect to each component in the research proposal evaluation device 20 for realizing this.

Confirmation unit 21 performs a function to confirm the evaluation target research plan.

Specifically, the verification unit 21 confirms the attribute items that are classified and posted in the evaluation target research plan before the actual evaluation of the evaluation target research plan.

Such attribute items and performance items may be inputted from a researcher or a manager who created an evaluation target proposal through, for example, data mining of the inputted evaluation target proposal or through a separately provided UI. It can be confirmed through.

For reference, in order to carry out the national R & D project, the research plan should be prepared in accordance with the standardized form for each project by the information on the attribute items according to the national R & D information standard notification. It is possible to check the posted attribute items separately from the target study plan.

In addition, when the identification of the attribute item of the evaluation target research plan is completed, the verification unit 21 determines the performance item that is to be predicted about the evaluation target research plan, that is, the R & D performance item.

As mentioned above, the results of R & D performance, project evaluation results, manpower training results, academic achievements such as papers, patents, reports, academic presentations, intellectual property rights, technology commercialization performances such as technology transfer, start-up success, Other items may indicate the research and development achievements.

The acquirer 22 performs a function of acquiring the study subject study information.

More specifically, the acquiring unit 22 acquires the study subject research information related to the evaluation target research plan from the information providing apparatus 10 when the verification of the research target to be evaluated and the determination of the R & D performance item are completed. Done.

At this time, the acquisition unit 22 is the existing research information stored in the information providing device 10, the existing research project information that the similarity between the attribute value of each attribute item with the evaluation target research plan is confirmed above the threshold value It can be acquired by research subject information.

Here, the similarity between the attribute values is greater than or equal to the threshold, for example, that the research plan, the field of study, the total research period, and the study period of the year among the existing research project information coincide with each other, and the degree of coincidence is higher than the set value. As a matter of course, the threshold of similarity with the attribute value for the selection of the study subject research information may be freely designated according to the setting of the administrator.

For reference, the higher the threshold value of similarity with the attribute value is, the smaller the number of study object information acquired will be. On the contrary, if the threshold value of similarity with the attribute value is specified, the acquired study object information is acquired. It can be expected that there will be fewer cases of.

The setting unit 23 performs a function of setting attribute items and performance items for the study subject information to be studied.

In detail, the setting unit 23, when the learning subject research information related to the evaluation target research plan is obtained, the performance item matching the research development result item of the evaluation target research plan with respect to the learning subject research information and An attribute item that may be associated with an attribute value of the performance item is set.

In this case, the setting unit 23 sets at least some of the property items posted in the study subject information as the attribute items or sets the ratio of property values between different property items posted in the study subject information. Can be set as an attribute item.

For example, according to the national R & D information standard notification, the total number of participating researchers and the number of participating researchers in A degree are set as attribute items for evaluation as it is, or compared to the total of participating researchers. The percentage of researchers participating in the degree can also be configured as an additional attribute item.

The learning unit 24 performs a function of processing the machine learning for the study subject information of the learning.

Specifically, the learning unit 24, when the setting of the attribute item and the performance item for the study subject information to study is completed, the machine learning about this process by the correlation between the attribute item and the performance item derived from such attribute item Make sure to check.

Machine learning here includes, for example, linear regression modeling, logistic regression modeling, decision tree modeling, neural network modeling, association modeling, clustering modeling, outlier detection modeling, nearest neighbor modeling, ensemble modeling, and the like.

Specifically, logistic regression modeling, decision tree modeling, neural network modeling, ensemble modeling, etc. can be applied to performance items classified into categorical types such as success or failure, and models with the highest classification accuracy can be used.

 In addition, linear regression modeling, decision tree modeling, and neural network modeling can be applied to performance items represented by continuous numbers such as the number of publications.

In addition, linear regression modeling, logistic regression modeling, association modeling, clustering modeling, outlier detection modeling, nearest neighbor modeling, etc. can be used for the purpose of detecting (exception detection) of exaggerated research plans through machine learning, which will be described below. have.

The decision unit 25 performs the function of determining the key attribute items and the performance estimation rule based on these key attribute items.

Specifically, the determination unit 25 is based on the result of the machine learning the study target project information, at least a higher degree of relevance with the performance item than other attribute items among the attribute items that may be associated with the attribute value of the performance item. Based on some attribute items, the core attribute items, and the performance estimation rule that can derive the performance items based on the core attribute items.

The predicting unit 26 performs a function of predicting the expected performance of the R & D performance item of the research target to be evaluated.

Specifically, the prediction unit 26 extracts the actual attribute values of the attribute items matching the core attribute items from the evaluation target research plan when the determination of the core attribute items and the performance estimation rule using the same is completed. In addition, the expected performance is predicted for the R & D performance items determined for the research target to be evaluated by substituting the extracted actual attribute values into the performance estimation rule.

In order to better understand the description, the specific method of the above-described process from checking the evaluation target plan to predicting the expected outcome will be described with reference to the following embodiments.

First, as a first example, the research plans named A01 to A10, which were prepared to participate in the national R & D projects in the field of biotechnology, were identified as the research targets for evaluation, and the possibility of successful commercialization of these research plans was determined as R & D performance items. The case can be illustrated.

In this case, considering the characteristics of the research target to be evaluated, 1,463 research information carried out during a specific year in the biotechnology field can be acquired as the research project information to be used as machine learning data.

Here, the research subject information to be studied is not limited to using only the task information limited to a specific field and a specific year as the machine learning data, and the existing project information for the machine learning can be limited in consideration of various characteristics of the research plan to be evaluated. Of course.

In other words, the research subject information can be obtained by limiting not only the source of information of the existing research project information, research field, year of research, but also various conditions including budget scale, research period, research institute, research development character, etc. .

And, for the acquired study subject information, according to an embodiment of the present invention, the base year, the name of the department, whether or not to continue the task, the Korean title, the total research period, the type of research subject, region, field of study, 6T classification code, Technology life cycle name, detailed task characteristics, multi-year agreement, year of study, project implementation institution, subject to practical use, R & D stage, R & D characteristics, project progress status, government investment research fund, research fund A item amount, research fund B Item Amount, Research Cost C Item Amount, Research Cost D Item Amount, Research Amount E Item Amount, Research Amount F Item Amount, Research Amount G Item Amount, Research Amount H Amount, Total Research Amount, Joint Research Firm, Joint Research University, Joint Research Bureau Whether it is a joint research foreign research institution, whether it is a joint research institute, other research institutes, the total number of participating researchers, the number of participating researchers by A sex, the number of participating researchers by B gender, the number of participating researchers by A degree, the number of participating researchers by B degree, C Number of researchers participating in degree, Participants in A degree, Participants in B degree, Participants in C degree, Participants in A major, Participants in B major, Participants in C major, Participants in C major, Participants in D major, Participation in E major The number of researchers and F-participating researchers can be set as an attribute item, and the feasibility of commercialization corresponding to the R & D performance item to be predicted can be set as a performance item.

In addition, when the setting of the attribute items and the performance items for the study subject information is completed, the decision tree modeling using four methods such as CHAID, Quest, CART, and C5.0 is used for the study subject information. Machine learning can be processed, in which case the results of classification accuracy can be obtained as in [Table 1] below.

Decision Tree division training black CHAID exact(%) 94.67 88.63 error(%) 5.33 11.37 Quest exact(%) 91.15 88.29 error(%) 8.85 11.71 CART exact(%) 95.19 93.31 error(%) 4.81 6.69 C5.0 exact(%) 95.88 93.65 error(%) 4.12 6.35

According to the above results, in case of C5.0 method, the classification accuracy of training data using 95% of the total research project information was 95.88%, and the classification accuracy of the remaining 30% of test data was 93.65%. Since the decision tree model obtained by the C5.0 method can be used as a machine learning method to predict the expected outcomes for R & D performance items,

At this time, according to the decision tree model obtained by the C5.0 method, the four key attribute items that have a great influence on the determination of commercialization success, which are performance items, are four such as total research expenses, total research period, government investment research expenses, and R & D stage. Also, the performance estimation rule composed of these core attribute items can be determined as shown in FIG.

Furthermore, when the performance estimation rule is determined, the total research expenses, total research period, government investment research expenses, and research, which are the key attribute items in the evaluation target research plan, in order to predict the success or failure of commercialization, for the research target for the evaluation target research plan By extracting the attribute value (information) of the item in the development stage and substituting the extracted attribute value into the decision tree tree model, it can be predicted by evaluating the possibility of commercialization as shown in [Table 2] below.

Research plan Total research fund Expected Study Period Government Investment Research Fund R & D stage Predictability of Commercialization Success A01 1,190 1 year 1,190 Etc success A02 650 2 years 100 Basic research success A03 534 3 years 400 Basic research success A04 5 6 months 5 Basic research failure A05 610 5 years 300 Basic research success A06 1,210 7 years 900 Basic research success A07 600 5 years 600 Applied Research success A08 18 4 years 18 Basic research failure A09 604 7 years 604 Basic research success A10 42 3 years 42 Basic research failure

According to the prediction results, if the possibility of commercialization is the most important and absolute criterion for selecting a project from the standpoint of evaluating the proposal, the research plans such as A04, A08, and A10, which are unlikely to be successful, may be excluded.

In addition, as a second embodiment, the case of predicting the number of publications among the performance items expected to be obtained by conducting a study according to the evaluation target research plan can be exemplified.

In this case, the process before the machine learning of the subject to be studied is similar to the first embodiment described above, except that the number of thesis publications is set as an outcome item that is set for the subject to be studied, and the subject research subject The difference is that machine learning is performed using linear regression modeling of information.

As such, according to the results of the machine learning using linear regression modeling, according to the order of importance, the number of participating researchers in A degree, the amount of research item H, the number of participating researchers in B degree, the number of participating researchers in B major, the total amount of research funds, the amount of research fund C items, and A The core attribute items can be derived from the number of major participating researchers, the amount of research fund B items, the amount of research fund G items, the number of participating researchers in C degree, the number of participating researchers by A gender, and the amount of research item E items.

In addition, the rule for estimating the number of articles published using key attribute items, that is, the linear regression formula, is -0.376 + 0.443 * Number of A-participating researchers + 2.3 * 10 ^-9 * Amount of research fee C items-0.063 * Number of C-participating researchers + 0.136 * B major Number of Participating Researchers + 1.9 * 10 ^-8 * Research Fund H Item Amount-1.2 * 10 ^-8 * Research Fund E Item Amount + 9.6 * 10 ^-10 * Total Research Funds-0.171 * Number of Participating Researchers by A + 0.104 * Number of Participating A Researchers + 3.7 * 10 ^-9 * Amount of research grant item G + 4.7 * 10 ^-9 * Amount of research grant item B + 0.056 * It can be derived from the number of participating researchers.

At this time, the statistical significance test results for the above linear regression equation (performance estimation rule) are derived as shown in [Table 3] and [Table 4] below, and the significance probability for all variables was less than 0.05.

R R2 Modified R2 Standard error of the estimate 0.778 0.605 0.601 3.8198

Sum of squares Degrees of freedom Mean square F Significance return 27,841 12 2320 159 0.000 Residual 18,166 1,245 15 all 46,007 1,257

In addition, when the linear regression equation corresponding to the performance estimation rule is determined, the number of researchers participating in the A degree, which are the key attribute items in the evaluation target research plan, and the research expenses in order to predict the number of research publication performance items on the evaluation target research plan Amount of H items, number of participating researchers in B degree, number of participating researchers in B major, total amount of research expenses, amount of research expenses C item, number of participating researchers in A major, amount of research expenses B item, amount of research expenses G item, number of participating researchers in C degree, participating researchers by A gender The number of research items, the amount of research fee E items, and the extracted attribute values are substituted into the above linear regression equations. .

Research
Plan A-level research participants Research Fund H Item Amount B-participated researchers Participant in B major syncopation Thesis Publication Counts Forecast A01 8 73,000,000 One 9 ... 4 A02 4 44,000,000 3 0 One A03 2 5,000,000 3 One 2 A04 0 0 0 0 0 A05 3 90,958,000 3 One 2 A06 10 127,315,000 0 2 9 A07 0 83,500,000 4 3 One A08 0 0 0 0 0 A09 0 78,782,000 0 0 One A10 0 9,762,000 0 0 0

If the number of publications is the most important and absolute criterion for selecting a project from the standpoint of evaluating the proposal, the proposal may be selected in the order of A06, A01, A03, A05, etc.

Of course, a comprehensive evaluation can be performed by predicting various R & D performance items, and by combining the prediction result derived from the research plan evaluation method provided in an embodiment of the present invention with the qualitative evaluation result of a few experts. Support rational decision making

After the detailed description referring to the embodiment from the process of confirming the evaluation target research plan to the process of predicting the expected outcome, the description of the remaining components of the research project evaluation device 20 will be continued.

The abnormality detection unit 27 detects an abnormality of the content of the research target to be evaluated.

Specifically, when the abnormality detection unit 27 is required to detect the abnormality of the contents of the evaluation target research plan, that is, at least one of the attribute item and the performance item, both the subject study information and the target study plan are machine learning. Using the result, the performance item whose difference between the attribute value and the predicted value is greater than or equal to the threshold is detected as an anomaly, or the attribute or performance item whose difference between the attribute value and the statistical value is greater than or equal to the threshold is used as an anomaly. Will be detected.

The above detection configuration can be used to find a research proposal that has an expected result that the investigator has exaggerated, or a research proposal that has an outlier in the attribute item. The specific method in the detection process will be described with reference to the following examples.

First, as a third embodiment, in connection with the second embodiment described above, a researcher presents an inflated expectation result in the research proposal for the purpose of increasing the possibility of winning an order due to excessive confidence in his or her research plan. The case can be illustrated.

In this case, the results of comparing the expected number of thesis publication expected by the researcher and the number of thesis publication predicted according to the method of evaluating the proposal of the present invention with respect to the evaluation target research plan described in Example 2 are shown below. It can be expressed as

Research
Plan Estimated publication number Researcher expectation Absolute error Error rate A01 4 5 One 20% A02 One 3 2 67% A03 2 3 One 33% A04 0 10 10 100% A05 2 5 3 60% A06 9 10 One 10% A07 One 5 4 80% A08 0 3 3 100% A09 One 2 One 50% A10 0 One One 100%

According to the above [Table 6], with reference to the error and error rate, it can be judged that the expected performance of the researchers presented in the A04, A07, and A08 proposals is more exaggerated than the other proposals.

In addition, as a fourth embodiment, a case of detecting whether an attribute item is abnormal or not, as well as an exaggeration of the expected performance for the performance item described in the third embodiment, may also be exemplified.

In other words, it is possible to identify through machine learning whether any of the attribute items corresponding to the input factors of the research proposal prepared by the researcher are probabilistically out of the range of outliers.

In this case, through the modeling method that detects the abnormality and uses it as a machine learning data including the research targets and the existing project information, it is possible to determine whether there are statistical outliers among the attribute items and the performance items of the evaluation targets. The result can be expressed as shown in [Table 7] below.

Study plan Outliers Outlier Index Cluster number Ideal item 1 Ideal item 1
Contribution Ideal item 2 Ideal item 2 contribution A01 normal 2.976 One Number of researchers participating in C major 0.268 Percentage of Participants in A Degree 0.264 A02 normal 0.62 2 Research Grant G Item Amount 0.153 Research Fund C Item Amount 0.078 A03 More than 8.757 2 Research Fund E Item Amount 0.472 C-level researchers 0.472 A04 normal 0.326 One Percentage of Participants in A Degree 0.364 R & D stage 0.33 A05 normal 0.579 2 Research Fund D Item Amount 0.092 Research Fund C Item Amount 0.075 A06 normal 0.558 2 Research Grant G Item Amount 0.072 Percentage of Participants in A Degree 0.06 A07 normal 2.794 One Research Fund E Item Amount 0.777 R & D stage 0.134 A08 normal 0.322 One Percentage of Participants in A Degree 0.368 R & D stage 0.334 A09 normal 0.39 One Percentage of Participants in A Degree 0.305 R & D stage 0.276 A10 normal 0.325 One Percentage of Participants in A Degree 0.365 R & D stage 0.331

According to the above [Table 7], when the group with the outlier index of the top 1% among the machine learning data including the research proposals and the existing project information is classified as the outlier, the A03 proposal is the outlier index. It can be confirmed that the attribute items belonging to the above range of 8.757, and the attribute items that contributed to the A03 research plan to be judged as above range are the amount of research items E and the number of participating researchers in the C degree.

Therefore, from the standpoint of evaluating the proposal, the selection of the A03 proposal may be withheld or excluded depending on the detection result of the anomaly described above.

After completing the detailed description of the embodiment of the process for detecting the abnormal items in the research plan to be evaluated above, the description of the last configuration of the research project evaluation device 20 will be continued.

The consulting unit 28 performs a function of providing feedback for improving the prediction result of the R & D performance item.

Specifically, the consulting unit 28, in relation to consulting on the research target to be evaluated, is required to improve the expected performance of the R & D performance items, and according to the machine learning results, the core attribute items in the research target to be evaluated. The attribute items that need to be changed among the attribute items matched with and are selected as change target items and provided as feedback information.

In this case, the change target item selected as the feedback information is selected based on a high priority of relevance between the key attribute items and the performance items matched in the study target information, or the key matched within the study target information. With regard to the distribution of the attribute values of the attribute item, the relative position of the actual attribute value in the evaluation plan can be selected based on a high priority that is out of the center of the distribution.

In the case of the configuration that selects the items to be changed and provides them as feedback information, based on the results of the evaluation, the research plan is designed to improve the success of R & D and technology commercialization and the expected performance expected to be obtained upon success. It can be used for research proposal consulting methods that provide feedback and advice to the authors.

In order to help the understanding of the description, the specific method in the consulting process will be described with reference to embodiments.

As a fifth embodiment, it may be assumed that the prediction result according to the second embodiment, that is, the expected performance of the article publication number item is improved.

In this case, the new study plan A09 mentioned in the second embodiment of the present invention can be evaluated, and as a result, the number of article publications can be predicted according to the linear regression model, which is a performance estimation rule.

In order to improve the expected number of publications in the research plan in this field, according to the linear regression model in the second embodiment described above, the A degree participating researcher, the research fund H item amount, the B degree participating researcher, and B Number of participating researchers, total research expenses, research expenses C item amount, A major participating researchers, research B item amount, research fund G item amount, C degree participating researchers, A number of participating researchers by gender, and research fund E items Can be selected as an attribute variable.

At this time, the number of researchers participating in A degree, the amount of research expenses H, the number of researchers participating in B degree, the number of research participants participating in B major, the total amount of research expenses, the amount of research expenses C, A major Priority may be determined in the order of the number of participating researchers, the amount of research fund B item, the amount of research fund G item, the number of participating researchers in C degree, the number of participating researchers by gender, and the amount of research item E item.

Of course, in prioritization, priorities are determined based on the degree of deviation of the relative position of the core attribute variable of the evaluation target plan from the distribution center of the values of the core attribute variables of the existing project information. It may be.

In conclusion, if the goal is to increase the number of papers published in the field of study, the recruitment of A-level participant researchers, the increase in the amount of research fund H items, the recruitment of B-level participant researchers, and the number of B-participant researchers Feedback can be given to the researcher who prepared the research plan and advised.In carrying out the national R & D project, if there is a restriction on the change of certain attribute items, such as the amount of research expenses H, due to the nature of the project, only the change of the attribute items is advised. I can do it.

As described above, according to the configuration of the research plan evaluation apparatus 20 according to an embodiment of the present invention, unlike the qualitative evaluation method of a few experts, quantitative based on the results of the machine learning on the existing research project information The results of the evaluation are provided, which allows for a more rational and objective future prediction of the proposal. In addition, the electronic system provided in the prior art document only helps to accelerate the qualitative evaluation result more quickly and easily, and even if data is used indirectly in the evaluation, it can only rely on the expert's qualitative evaluation. Considering its limitations, it provides a fundamental solution to this problem. In other words, it provides information that can make rational and objective judgments based on existing R & D information without completely qualitative expert judgment regarding research plan evaluation and consulting. In addition, the machine learning-based research proposal evaluation method can replace the existing qualitative evaluation method by itself, but it can support more effective and reasonable decision making by combining the qualitative evaluation results of the existing few experts. In addition, according to the method of providing consulting-based feedback information, it suggests a more rational and objective way of revising the research plan based on the machine learning data, through which the researcher who prepared the research plan establishes a more structured research plan, The possibility of success of technology commercialization can be improved.

After the description of the configuration of the research proposal evaluation device 20 according to an embodiment of the present invention, the following description of the operation method of the research proposal evaluation device 20 with reference to FIGS. Shall be.

First, a process of predicting R & D performance items according to an embodiment of the present invention will be described with reference to FIG. 4.

First, the confirmation unit 21 confirms the attribute items that are classified and posted in the evaluation target research plan prior to the actual evaluation of the evaluation target research plan (S10).

Such attribute items and performance items may be inputted from a researcher or a manager who created an evaluation target proposal through, for example, data mining of the inputted evaluation target proposal or through a separately provided UI. It can be confirmed through.

For reference, in order to carry out the national R & D project, the research plan should be prepared in accordance with the standardized form for each project by the information on the attribute items according to the national R & D information standard notification. It is possible to check the posted attribute items separately from the target study plan.

At this time, when the confirmation of the attribute item of the evaluation target research plan is completed, the verification unit 21 determines the performance item that is to be predicted about the evaluation target research plan, that is, the R & D performance item.

As mentioned above, the results of R & D performance, project evaluation results, manpower training results, academic achievements such as papers, patents, reports, academic presentations, intellectual property rights, technology commercialization performances such as technology transfer, start-up success, Other items may indicate the research and development achievements.

Subsequently, when the confirmation of the research target to be evaluated and the determination of the R & D performance item are completed, the acquirer 22 acquires the study target project information related to the research target to be evaluated from the information providing apparatus 10 ( S11).

At this time, the acquisition unit 22 is the existing research information stored in the information providing device 10, the existing research project information that the similarity between the attribute value of each attribute item with the evaluation target research plan is confirmed above the threshold value It can be acquired by research subject information.

Here, the similarity between the attribute values is greater than or equal to the threshold, for example, that the research plan, the field of study, the total research period, and the study period of the year among the existing research project information coincide with each other, and the degree of coincidence is higher than the set value. As a matter of course, the threshold of similarity with the attribute value for the selection of the study subject research information may be freely designated according to the setting of the administrator.

For reference, the higher the threshold value of similarity with the attribute value is, the smaller the number of study object information acquired will be. On the contrary, if the threshold value of similarity with the attribute value is specified, the acquired study object information is acquired. It can be expected that there will be fewer cases of.

Then, the setting unit 23, when the learning subject research information related to the evaluation target research plan is obtained, the performance item matching the research and development performance items of the evaluation target research proposal, An attribute item that may be associated with an attribute value of the performance item is set (S12).

In this case, the setting unit 23 sets at least some of the property items posted in the study subject information as the attribute items or sets the ratio of property values between different property items posted in the study subject information. Can be set as an attribute item.

For example, according to the national R & D information standard notification, the total number of participating researchers and the number of participating researchers in A degree are set as attribute items for evaluation as it is, or compared to the total of participating researchers. The percentage of researchers participating in the degree can also be configured as an additional attribute item.

Next, when the setting of the attribute item and the performance item for the study subject information to be studied is completed, the learning unit 24 processes the machine learning for this to establish a correlation between the attribute item and the performance item derived from the attribute item. To check (S13).

Machine learning here includes, for example, linear regression modeling, logistic regression modeling, decision tree modeling, neural network modeling, association modeling, clustering modeling, outlier detection modeling, nearest neighbor modeling, ensemble modeling, and the like.

Specifically, logistic regression modeling, decision tree modeling, neural network modeling, ensemble modeling, etc. can be applied to performance items classified into categorical types such as success or failure, and models with the highest classification accuracy can be used.

 In addition, linear regression modeling, decision tree modeling, and neural network modeling can be applied to performance items represented by continuous numbers such as the number of publications.

In addition, linear regression modeling, logistic regression modeling, association modeling, clustering modeling, outlier detection modeling, nearest neighbor modeling, etc. can be used for the purpose of detecting exaggerated research plans through machine learning.

Further, the determination unit 25 is based on the result of the machine learning the study target project information, at least a portion of the attribute items that may be associated with the attribute value of the performance item is higher than the other attribute items are more related to the performance item. Based on the core attribute item, which is an attribute item of, and the performance estimation rule that can derive the performance item based on the core property item (S14).

Thereafter, the prediction unit 26 extracts the actual attribute values of the attribute items matching the core attribute items from the evaluation target study plan when the determination of the core attribute items and the performance estimation rule using the same is completed. The expected performance is predicted with respect to the R & D performance item determined for the R & D target subject by assigning the extracted actual attribute value to the performance estimation rule (S15).

Next, with reference to Figure 5 will be described the process of detecting the abnormality of the content of the evaluation target research plan according to an embodiment of the present invention.

Prior to the description, the abnormality detection process, which will be described below, may be used to find a research proposal that has an exaggerated expected performance or a research proposal that has an abnormality in attribute items. Prior to such a detection process, referring to FIG. It is assumed that the entire process (S10 to S15) to predict the R & D performance item described above or the process (S10 to S13) to perform machine learning as necessary is already processed.

First, the abnormality detection unit 27 determines whether an abnormality detection of the contents of the research target to be evaluated is required on the premise that the entire process up to predicting R & D performance items or the process until performing machine learning has already been processed. Check (S21).

Then, when the abnormality detection unit 27 is required to detect the abnormality of the contents of the evaluation target research plan, that is, at least one of the attribute item and the performance item, both the subject study information and the target study plan are machine-learned. The results are used to detect performance items whose difference between the attribute value and the predicted value is greater than or equal to the threshold, or to detect the attribute or performance item that is greater than or equal to the threshold between the attribute value and the statistical value. (S22).

Finally, a consulting process for improving the results of predicting the R & D performance item according to an embodiment of the present invention will be described with reference to FIG. 6.

Prior to the discussion, the consulting process described below will be used to provide feedback to the researcher who prepared the research proposal and advise how to improve the likelihood of success in R & D and commercialization and the expected outcome that will be obtained upon success. Prior to such a consulting process, it is assumed that the entire process (S10 to S15) until the R & D performance item described with reference to FIG. 4 is already processed.

First, the consulting unit 28 confirms whether the improvement of the expected performance expected for the R & D performance item is required that the entire process until the R & D performance item has already been processed (S31).

Then, the consulting unit 28 needs to change among the attribute items that match the core attribute items in the research target to be evaluated according to the machine learning results when the expected improvement of the expected performance is required for the R & D performance items. Is selected as a change target item (S32).

In this case, the item to be changed is selected based on a high priority between the core attribute items matched in the study subject information and the performance items, or the attribute values of the core attribute items matched in the study subject information. For distributions, the relative position of the actual attribute values in the study plan to be evaluated may be selected on the basis of a high priority that deviates from the center of the distribution.

Thereafter, the consulting unit 28 provides the selected researcher with information on the subject of the research project as the selected feedback information, and consults a method for improving the success of R & D and technology commercialization and the expected performance expected to be obtained upon success. Can be.

As described above, according to the operation method of the research plan evaluation device 20 according to an embodiment of the present invention, unlike the qualitative evaluation method of a few experts, based on the machine learning result of the existing research project information Quantitative evaluation results are provided, which allows more reasonable and objective future prediction of the proposal. In addition, the electronic system provided in the prior art document only helps to accelerate the qualitative evaluation result more quickly and easily, and even if data is used indirectly in the evaluation, it can only rely on the expert's qualitative evaluation. Considering its limitations, it provides a fundamental solution to this problem. In other words, it provides information that can make rational and objective judgments based on existing R & D information without completely qualitative expert judgment regarding research plan evaluation and consulting. In addition, the machine learning-based research proposal evaluation method can replace the existing qualitative evaluation method by itself, but it can support more effective and reasonable decision making by combining the qualitative evaluation results of the existing few experts. In addition, according to the method of providing consulting-based feedback information, it suggests a more rational and objective way of revising the research plan based on the machine learning data, through which the researcher who prepared the research plan establishes a more structured research plan, The possibility of success of technology commercialization can be improved.

Meanwhile, the steps of the method or algorithm described in connection with the embodiments presented herein may be embodied directly in hardware or in the form of program instructions that may be executed by various computer means and recorded on a computer-readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. Program instructions recorded on the media may be those specially designed and constructed for the purposes of the present invention, or they may be of the kind well-known and available to those having skill in the computer software arts. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tape, optical media such as CD-ROMs, DVDs, and magnetic disks, such as floppy disks. Magneto-optical media, and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. The hardware device described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

Although the present invention has been described in detail with reference to preferred embodiments, the present invention is not limited to the above-described embodiments, and the technical field to which the present invention belongs without departing from the gist of the present invention as claimed in the following claims. Anyone skilled in the art will have the technical idea of the present invention to the extent that various modifications or changes are possible.

According to the apparatus for evaluating a research proposal according to the present invention and an operation method thereof, the existing technology can be evaluated by evaluating a research target to be evaluated through a method of machine learning quantitative data and also providing feedback on an improvement method for the evaluation result. It is an invention that has industrial applicability because the possibility of marketing or sales of the applied device is not only sufficient for the use of related technology but also practically obvious as it goes beyond the limit of.

10: information providing device
20: Research plan evaluation device
21: confirmation unit 22: acquisition unit
23: setting unit 24: learning unit
25: decision unit 26: prediction unit
27: abnormality detection unit 28: consulting unit
29: communication unit

Claims (15)

When the target project information related to the target project plan is obtained, the target project information related to the target project plan is related to the performance item matching the R & D performance item of the target project plan and the attribute value of the performance item. A setting unit for setting an attribute item which can be set;
Based on the result of the machine learning of the study subject research information, a core attribute that is at least some attribute item having a higher relevance to the performance item than other attribute items among attribute items that may be related to the attribute value of the performance item A decision unit to determine a performance estimation rule from which the performance item is derived based on an item and the core attribute item; And
Extracting an attribute value of an attribute item matching the core attribute item from the evaluation target research plan, and substituting the extracted attribute value into the performance estimation rule to predict an expected outcome of the R & D performance item; ,
Regarding consulting on the evaluation target research plan, when an expected improvement of expected performance is required for the R & D performance item, an attribute item matching the core attribute item in the evaluation target research plan according to the machine learning result is required. It further includes a consulting unit that selects an attribute item that needs to be changed among the items to be changed and provides it as feedback information.
In the case where abnormality detection of at least one of the attribute item and the performance item in the evaluation target study plan is required, the evaluation target study plan is based on the result of the machine study of the study subject study information and the evaluation target study plan. Further includes an anomaly detection unit that detects a performance item whose difference between the attribute value and the prediction value is greater than or equal to the threshold, or detects an attribute or performance item that has a difference between the attribute value and the statistical value by more than the threshold, as an anomaly,
The item to be changed is
The core attribute items matched within the study subject information to be studied and the performance items are selected based on a high priority.
The performance items,
It includes the results of project evaluation, manpower development performance, academic achievements including papers, patents, reports and conference presentations, and technology commercialization performances including technology transfer and start-up.
The attribute item is
Includes the nature and form of the support subject of the R & D fund, the size and composition of the R & D fund, the field to which the technology belongs, the nature and composition of the participants, the duration of the study, the objectives and contents of the subject, and the expected effects.
The performance estimation rule,
Includes a decision tree model or linear regression
The machine learning,
Regression Modeling, Logistic Regression Modeling, Decision Tree Modeling, Neural Network Modeling, Association Modeling, Clustering Modeling, Outlier Detection Modeling, Nearest Neighbor Modeling, Ensemble Modeling Research protocol evaluation device. delete delete delete The method of claim 1,
The item to be changed is
With respect to the distribution of the attribute values of the matching core attribute items in the study target project information, the relative position of the attribute values in the evaluation target study plan is selected based on a priority having a large deviation from the center of the distribution. Research proposal evaluation device, characterized in that. The method of claim 1,
The setting unit,
Setting at least some of the posted attribute items in the study subject information as the attribute items or setting the ratio of attribute values among different attribute items posted in the study subject information as the attribute items Research proposal evaluation device, characterized in that. The method of claim 1,
The study subject research information,
The research plan evaluation apparatus, characterized in that the existing research project information that the similarity between the attribute value for each attribute item with the evaluation target research plan of the existing research project information is confirmed to be above the threshold. When the target project information related to the target project plan is obtained, the target project information related to the target project plan is related to the performance item matching the R & D performance item of the target project plan and the attribute value of the performance item. Setting step of setting an attribute item;
Based on the result of the machine learning of the study subject research information, a core attribute that is at least some attribute item having a higher relevance to the performance item than other attribute items among attribute items that may be related to the attribute value of the performance item A decision step of determining a performance estimation rule for deriving the performance item based on an item and the core attribute item; And
A predicting step of extracting an attribute value of an attribute item matching the core attribute item from the evaluation target research plan, and substituting the extracted attribute value into the performance estimation rule to predict an expected outcome for the R & D performance item. ,
Regarding consulting on the evaluation target research plan, when an expected improvement of expected performance is required for the R & D performance item, an attribute item matching the core attribute item in the evaluation target research plan according to the machine learning result is required. In addition, the consulting step of providing an attribute item that needs to be changed among the item to be changed as feedback information is provided.
In the case where abnormality detection of at least one of the attribute item and the performance item in the evaluation target study plan is required, before the determination step, based on the result of machine learning the subject study information and the evaluation target study plan. Anomaly detection that detects a performance item whose difference between the attribute value and the predicted value is greater than or equal to the threshold in the evaluation target plan, or detects an attribute or performance item whose difference between the attribute value and the statistical value is greater than or equal to the threshold. More steps,
The item to be changed is
The core attribute items matched within the study subject information to be studied and the performance items are selected based on a high priority.
The performance items,
It includes the results of project evaluation, manpower development performance, academic achievements including papers, patents, reports and conference presentations, and technology commercialization performances including technology transfer and start-up.
The attribute item is
Includes the nature and form of the support subject of the R & D fund, the size and composition of the R & D fund, the field to which the technology belongs, the nature and composition of the participants, the duration of the study, the objectives and contents of the subject, and the expected effects.
The performance estimation rule,
Includes a decision tree model or linear regression
The machine learning,
Regression Modeling, Logistic Regression Modeling, Decision Tree Modeling, Neural Network Modeling, Association Modeling, Clustering Modeling, Outlier Detection Modeling, Nearest Neighbor Modeling, Ensemble Modeling Operation method of research protocol evaluation device. delete delete delete The method of claim 8,
The item to be changed is
With respect to the distribution of the attribute values of the matching core attribute items in the study target project information, the relative position of the attribute values in the evaluation target study plan is selected based on a priority having a large deviation from the center of the distribution. Method of operation of the protocol evaluation apparatus, characterized in that. The method of claim 8,
The setting step,
Setting at least some of the posted attribute items in the study subject information as the attribute items or setting the ratio of attribute values among different attribute items posted in the study subject information as the attribute items Method of operation of the protocol evaluation apparatus, characterized in that. The method of claim 8,
The study subject research information,
Method of operation of the research plan evaluation device, characterized in that the existing research project information is confirmed that the similarity between the attribute value of each attribute item with the evaluation plan of the existing research project information is above the threshold. A computer-readable medium having recorded thereon a program for executing each step of the method according to any one of claims 8 and 12-14.

Images