KR101456190B1 - Method for verifying evaluation engine of patent evaluation system - Google Patents

Abstract

The method according to one embodiment disclosed herein provides an evaluation engine verification method for a patent evaluation system. The evaluation engine verification method comprises the steps of: classifying a plurality of evaluation results performed by an expert into evaluation groups in a plurality of groups; The evaluation result of at least one group among the plurality of groups is reserved for verification, and the evaluation engine is constructed by learning based on evaluation results of the remaining groups; And performing the primary verification using the evaluation results of the at least one reserved group.

Description

[0001] METHOD FOR VERIFYING EVALUATION ENGINE OF PATENT EVALUATION SYSTEM [0002]

The present specification relates to an evaluation engine verification method for a patent evaluation system.

In recent years, there have been many efforts and achievements in intellectual property rights for technology protection in Korea compared to other developed countries. In this process, it is true that the intellectual property rights holders have a lot of intellectual property rights, and the cost and effort to maintain and maintain the intellectual property rights are one of the burden.

In addition, it is also a difficult task to identify technologies that have been so rapidly changed that they have been registered, but that are no longer required to be maintained, and which, among other things, have to be invested heavily.

Therefore, intellectual property owners are assessing their intellectual property rights by rating their own intellectual property rights, or requesting commercial or nonprofit organizations to evaluate patented technologies.

On the other hand, the results of the patent evaluation include the maintenance of the patent, the provision of the utilization strategy of the patent, the support of the research plan, the evaluation of the right, the economic and the environment, (Strategic alliance), allocation of R & D planning resources, assessment of technology-secured loans for financial institution loans, support for direct and indirect technological development projects of government, evaluation for selection of project (assignment), intangible asset value We evaluate the valuation, technical, economic, and social aspects of the valuation, evaluation of the intangible asset's technology value for the customer's technology based on clear and objective data, present value of the inventor, compensation for the inventor (for depreciation) , Technology evaluation such as intellectual property right for technology transaction (technology transfer, M & A), valuation of intellectual property rights for technology guarantee setting, investment attraction etc. Janghi wide.

However, since the time and cost of the high-level human resources required for data analysis, which is the process of reporting to the intellectual property owner after the technology evaluation of the registered patent, is significant, most of the work is manually performed, It is necessary.

1 is an exemplary diagram showing the necessity of introducing a patent evaluation system.

As can be seen from FIG. 1 (a), if there are a large number of patents, for example, hundreds or thousands of patents, it will take a considerable amount of time and money to refer the patent evaluation to a specialist.

However, as shown in FIG. 1 (b), when a large number of patents are firstly filtered by a system, only a small amount of patents, for example, dozens of patents, can be evaluated by an expert, And reduce costs.

Accordingly, one embodiment disclosed herein is directed to a system for patent evaluation. In addition, one embodiment disclosed herein is directed to verifying an evaluation engine in a system for patent evaluation.

According to an embodiment of the present invention, an evaluation engine verification method for a patent evaluation system is provided. The evaluation engine verification method comprises the steps of: classifying a plurality of evaluation results performed by an expert into evaluation groups in a plurality of groups; The evaluation result of at least one group among the plurality of groups is reserved for verification, and the evaluation engine is constructed by learning based on evaluation results of the remaining groups; And performing the primary verification using the evaluation results of the at least one reserved group.

The evaluation engine verification method for the patent evaluation system includes the steps of: calculating an evaluation result for a patent in which a value of a specific evaluation factor is equal to or higher than a certain value; And performing a secondary verification based on the rating distribution generated on the basis of the calculated evaluation result.

The secondary verification may be to verify whether a patent having a value of the specific evaluation factor of at least a certain value has a higher rating than that of the general patent.

The evaluation element may include information extracted from at least one of the bibliographic information, progress information, specification, and claims of the registered patent, or may include information extracted by processing the specification and claims of the registered patent in a natural language.

Wherein the evaluation engine includes the steps of: calculating a degree of correlation of the evaluation elements with respect to at least one of the predetermined evaluation items based on the evaluation results of the experts on the sample patent; And a step of learning an evaluation result of an expert using the evaluation element mapped to the evaluation item.

The evaluation engine verification method comprising the steps of: receiving information on a specific patent from a user device after successful verification of the evaluation engine; Receiving an evaluation request for the specific patent from the user device; And outputting the evaluation result calculated using the established evaluation engine for a specific patent identified using the information.

The evaluation result by the expert is performed for each technical field, and the evaluation engine is constructed for each technical field. In the step of outputting the evaluation result, an evaluation engine of a technical field corresponding to the technical field of the specific patent can be used have.

The evaluation items may include one or more of rightness, technicality, and usability.

The evaluation engine may be constructed on the basis of the evaluation results of the experts having a good correlation based on the calculated degree of correlation based on the results of evaluation by a plurality of experts per technical field .

Meanwhile, in order to solve the above-described problems, the solution according to the embodiment disclosed herein provides another evaluation engine verification method. The evaluation engine verification method comprising the steps of: constructing an evaluation engine by learning based on a plurality of evaluation results performed by an expert on an evaluation item for a sample patent; Calculating an evaluation result using the constructed evaluation engine for a patent in which a value of a specific evaluation element is equal to or higher than a certain value; And verifying the evaluation engine based on the generated rating distribution based on the calculated evaluation result.

Meanwhile, in order to solve the above-mentioned problems, the solution according to an embodiment disclosed herein also provides an evaluation server. Wherein the evaluation server comprises: a receiver for receiving a plurality of evaluation results performed by an expert on an evaluation item for the sample patent; The evaluation results are classified into a plurality of groups, an evaluation result of at least one of the groups is reserved for verification, an evaluation engine is constructed by learning based on the evaluation results of the remaining groups, And a control unit for performing the primary verification using the evaluation results of at least one reserved group.

According to one embodiment presented herein, a patent can be automatically evaluated by a system and the evaluation result can be presented. Further, according to the evaluation engine according to the embodiment disclosed in the present specification, the patent can be quantified and more objectively evaluated.

1 is an exemplary diagram showing the necessity of introducing a patent evaluation system.
2 is a diagram illustrating an overall configuration of a patent evaluation system according to an embodiment of the present invention.
FIG. 3 is an exemplary view showing one or more servers 100 shown in FIG. 2 in detail.
FIG. 4 is an exemplary view showing in detail the configuration of the domestic / foreign patent evaluation servers 110 and 130 in FIG.
5 is a flowchart illustrating a method of constructing an evaluation engine by learning an expert evaluation result according to an embodiment of the present invention.
FIG. 6 is a flowchart and a table showing an example of a method of verifying an evaluation engine constructed by learning an expert evaluation result.
7 is a flowchart showing another method for verifying the evaluation engine constructed by learning the expert evaluation result.
Figs. 8 and 9 are distribution charts shown in accordance with another scheme shown in Fig.
10 is a flowchart illustrating a method for providing a patent evaluation service using an evaluation engine according to an embodiment of the present invention.
11 shows the physical configurations of the evaluation servers 110 and 130 and the service servers 120 and 140, respectively, which are shown in an embodiment of the present invention.

It is noted that the technical terms used herein are used only to describe specific embodiments and are not intended to limit the invention. It is also to be understood that the technical terms used herein are to be interpreted in a sense generally understood by a person skilled in the art to which the present invention belongs, Should not be construed to mean, or be interpreted in an excessively reduced sense. Further, when a technical term used herein is an erroneous technical term that does not accurately express the spirit of the present invention, it should be understood that technical terms that can be understood by a person skilled in the art are replaced. In addition, the general terms used in the present invention should be interpreted according to a predefined or prior context, and should not be construed as being excessively reduced.

Also, the singular forms "as used herein include plural referents unless the context clearly dictates otherwise. In the present application, the term "comprising" or "comprising" or the like should not be construed as necessarily including the various elements or steps described in the specification, Or may be further comprised of additional components or steps.

Further, the suffix "module" and "part" for components used in the present specification are given or mixed in consideration of ease of specification, and do not have their own meaning or role.

Furthermore, terms including ordinals such as first, second, etc. used in this specification can be used to describe various elements, but the elements should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings, wherein like reference numerals refer to like or similar elements throughout the several views, and redundant description thereof will be omitted.

In the following description, well-known functions or constructions are not described in detail since they would obscure the invention in unnecessary detail. It is to be noted that the accompanying drawings are only for the purpose of facilitating understanding of the present invention, and should not be construed as limiting the scope of the present invention with reference to the accompanying drawings.

2 is a diagram illustrating an overall configuration of a patent evaluation system according to an embodiment of the present invention.

2, the patent evaluation system according to an embodiment of the present invention includes at least one server 100 and at least one database (hereinafter referred to as 'DB') 190 do. The one or more servers 100 may be remotely managed by the management apparatus 500 shown in FIG.

The one or more servers 100 may be connected to a wired / wireless network to provide evaluation results services and other services to the user device 600. Specifically, upon receiving a request for an evaluation service for a specific patent from a user device, the one or more servers 100 may provide an evaluation result for the specific patent.

FIG. 3 is an exemplary view showing one or more servers 100 shown in FIG. 2 in detail.

3, one or more servers 100 may include a domestic patent evaluation server 110, a domestic patent service server 120, an overseas (e.g., US) patent evaluation server 130, ) Patent service server 140 as shown in FIG. In FIG. 3, the domestic patent evaluation server 110 and the foreign (for example, US) patent evaluation server 130 are shown as being physically separated from each other, but they can be physically integrated into one server. Also, although the domestic patent service server 120 and the foreign patent service server 140 are physically separated from each other, they may be physically integrated into one server. Further, the illustrated servers 110, 120, 130, 140 may be physically integrated into one server.

3, the above-mentioned one or more databases 190 include patent information DBs 191 and 192, evaluation element (or evaluation index) DBs 193 and 194, similar patent DBs 195 and 196, , And an evaluation result DB 197, 198. Although each DB is shown separately for domestic patent evaluation and US patent evaluation, it can be merged. For example, the Korean patent information DB 191 and the foreign patent information DB 192 of a foreign country (for example, the United States of America) can be merged into one, and the Korean evaluation factor (or evaluation index) , Or the United States) evaluation index (or evaluation index) DB 194 can be merged into one. Alternatively, the DBs may all be merged into one and separated by fields.

Such DBs may be generated based on those received from the external DB provider. For this reception, the server 100 may include a data collection unit 150 for receiving a raw DB of Korean or foreign (for example, the United States) from an external DB provider. The data collecting unit 150 physically includes a network interface (NIC). In addition, the data collection unit 150 may logically be a program including an application programming interface (API). The data collector 150 may process the raw DB received from the external DB provider and store the raw DB in one or more DBs 190 connected to the server 100 in the patent information DBs 191 and 192.

The domestic / foreign patent evaluation servers 110 and 130 include specification processing units 111 and 131, natural language processing units 112 and 132, key word processing units 113 and 133, similar patent processing units 114 and 134, (Or evaluation indexes) 115 and 135, and evaluation engine sections 116 and 136. [

The specification processing units 111 and 131 extract and parse (or convert) each information from the at least one DB 190 (e.g., patent information DBs 191 and 192). For example, the specification processing units 111 and 131 may extract one or more pieces of information such as patent specifications, bibliographic information, progress information, claims, drawings, and the like, and store them in the respective fields of the evaluation element (or evaluation index) DB.

The natural language processing units 112 and 132 perform natural language processing on the extracted patent specification and text included in the claims. Here, natural language processing refers not to a special programming language for computers but to computer processing of natural language used in general paintings. For example, the natural language processing units 112 and 132 may perform sentence analysis, syntax analysis, compound word processing, and the like. In addition, the natural language processing units 112 and 132 may perform a semantic process.

The key word processing units 113 and 133 extract key words from each patent based on the result of the natural language processing. In order to extract the keyword, techniques such as VSM (Vector Space Model) and LSA (Latent Semantic Analysis) can be used. Here, the core of the patent specification is a word representing the subject of the patent specification. For example, in the case of the present patent specification, " patent evaluation " As such, it may be advantageous to extract as many keywords as possible to represent the subject matter of the individual patent specification, but it is preferable to select the appropriate number because it may increase the inaccuracies if the number is increased.

Based on the extracted key word, the similar patent processing units 114 and 134 can search for patents most similar to each patent, and store the search results in the similar patent DBs 195 and 196. In general, similar patent families are known to belong to the same subclass in the International Patent Classification (IPC), but one embodiment presented herein can search for similar patents in other subclasses beyond the same subclass . Thus, in order to improve accuracy when searching for similar patents in other subclasses, it is most important that key words are extracted correctly. In particular, as described above, if the number of key words is simply increased, an incorrect key word is extracted, thereby making it possible to make an error that a completely different patent is searched by a similar patent in another sub-class. Therefore, one embodiment of the present invention calculates a suitable number of key words based on the simulation result according to the number of key words, and searches for similar patents with the calculated number of key words.

The evaluation element (or evaluation index) processing unit 115 or 116 extracts the value of the evaluation element (or the evaluation index) from at least one of the patent specification, bibliographic information, progress information, claim range, And stores it in the evaluation element DB 194.

The evaluation factors may be different for domestic patent evaluation and foreign patent evaluation. For example, the evaluation factors for domestic patent evaluation are summarized as follows.

Evaluation Element Name Explanation One Independent claim length Number of words in independent claim 2 Claims Number of claims 3 Number of claims Number of categories (objects or methods) of independent claims 4 Independent port number Number of independent claims 5 Domestic family patent number Number of domestic family patents (family based on split application, same basic priority) 6 Overseas family patent number Family patents of overseas countries 7 Annual registration number Number of registration year 8 First, Whether there was a priority claim 9 Eligibility Date The period until the filing date 10 Number of comments submitted Number of submissions 11 Decision of Refusal The number of the charges of the judgment of refusal decision 12 Total number of citations Total number of citations 13 Joint filing Number of joint applicants 14 Number of licensees Number of licensees 15 Number of active judges Referee's billing count 16 First ref Referee's billing count 17 Early public availability Whether there was a public disclosure 18 Information Provided Number of times information is provided 19 Number of appeals Number of appeals 20 Third party review request Whether the applicant or a third party has filed a claim for review 21 Number of invalid referees The number of judgments 22 Number of judges who confirm passive scope of right The number of judgments 23 Conducted Jesus Number of embodiments 24 Number of drawings Number of drawings 25 The number of words in the description of the invention Number of words in the detailed description part of the specification 26 Number of IPC Number of IPC classification codes 27 Rights holder change Number of rights holder change 28 Litigation information If there is a lawsuit, the number of times 29 Prior Art Literature Number of prior art documents presented during the review process

On the other hand, evaluation factors for overseas patents, for example, US patent evaluation, are summarized in the following table.

Evaluation factor Explanation One Independent claim length Number of words in independent claim 2 Number of claims Number of categories (objects or methods) of independent claims 3 Independent port number Number of independent claims 4 The number of words in the description of the invention Number of words in the detailed description part of the specification 5 Claims Number of claims 6 Number of family patents in the United States Number of family patents in the United States 7 Number of Reexamination Reexamination Count 8 Interference Count Interference count 9 Reissue number Reissue number 10 Total number of citations Total number of citations 11 Number of IPC Number of IPC classification codes 12 Overseas family patent number Number of overseas country family patents 13 Annual registration number Number of registration year 14 First, Whether there was a priority claim 15 Certification of Correction Number Certification of Correction claimed 16 Rights holder change Number of rights holder change 17 Litigation information If there is a lawsuit, the number of times 18 Prior Art Literature Number of prior art documents presented during the review process

On the other hand, the above evaluation factors (evaluation indexes) are merely illustrative, and any information directly derived from the patent information can be utilized as evaluation factors (evaluation indexes). Furthermore, any information that can be indirectly derived or calculated by processing patent information can be utilized as an evaluation factor (evaluation index).

On the other hand, the evaluation engine unit (116, 136) evaluates each patent based on the evaluation factors and the evaluation mechanism stored in the evaluation element DB (193, 194) for each predefined evaluation item, and calculates the evaluation result. Further, the evaluation engine units 116 and 136 may store the calculated evaluation results in the evaluation result DB 197 and 198, respectively.

The evaluation items may be defined as rights, technicalities, and usability. Alternatively, the evaluation items may be defined as rights property and commercial property. This definition may change depending on what the primary purpose of evaluating a patent is. Accordingly, the scope of the present invention is not limited to the above-described embodiments, and can be extended to any one to which the spirit of the present invention can be applied.

The evaluation mechanism may include a weight and a learning model. The weight may be a value calculated based on a result of an expert's preliminary evaluation of several sample patents. The evaluation engine units 116 and 136 will be described later in detail with reference to FIG.

The domestic / foreign patent service servers 120 and 140 may include at least one of an evaluation report generation unit 121 and a portfolio analysis unit 122 and a portfolio analysis unit 142. The evaluation report generation units 121 and 141 generate evaluation reports based on the evaluation results stored in the evaluation results DB 197 and 198. [ The portfolio analyzing units 122 and 142 can analyze the portfolio of the patents held by the owner based on the information stored in the similar patent DBs 195 and 196. [ Also, the portfolio analyzing units 122 and 142 may analyze the status of the patent by technology field or IPC classification by the right holder. In addition, the portfolio analysis units 122 and 142 can perform various analyzes based on the similar patent DBs 195 and 196 and the evaluation result DBs 197 and 19. For example, the portfolio analysis units 122 and 142 may perform various analyzes such as analysis of patent trends, analysis of the total amount of the annual fee for each patentee, and the like.

The domestic / foreign patent service servers 120 and 140 may provide an evaluation result of the specific patent when a user requests an evaluation service for the specific patent from the user device. In addition, the evaluation report may be provided in the form of a web page, an Excel file, a PDF file, or a word file according to a user's request, or may provide the analysis result. For this service, the user authentication / authority management unit 160 may be required.

FIG. 4 is an exemplary view showing in detail the configuration of the domestic / foreign patent evaluation servers 110 and 130 in FIG.

As can be seen from FIG. 4 and the above description, the description processing units 111 and 131 receive the patent specification information from the patent information DBs 191 and 192, and parse the specification. The specification may be written in XML, for example. In order to parse the XML, the specification processing units 111 and 131 may include an XML tag processing unit.

The evaluation element processing units 115 and 135 may include a first evaluation element processing unit that processes the evaluation element based on the parsed specification, and a second evaluation element processing unit. The first evaluation element processing unit extracts an evaluation element that does not need a natural language processing result based on the parsed specification. For example, the first evaluation element processing unit calculates the values of evaluation elements that do not require natural language processing, such as the length of the independent claim, the number of claims, the number of claims, the number of independent factors, the number of domestic family patents, And stores it in the evaluation element DB 193/194.

The natural language processing units 112 and 132 perform natural language processing based on the parsed specification. The natural language processing units 112 and 132 include a morpheme analysis unit for performing an operation based on a dictionary DB and a TM analysis. Morpheme is the smallest unit of meaning and is the smallest semantic element that can no longer be analyzed. Morpheme analysis is the first step in natural language analysis. It converts input strings into morpheme strings. The TM analysis is a two-level analysis, expressed as Tm = (R, F, D), where R is a rule set, F is a finite automaton converter, and T is a triad dictionary.

When the natural language processing is completed, the second evaluation element processing unit of the evaluation element processing units 115 and 135 calculates remaining evaluation elements based on the natural language processing result. For example, the evaluation element values such as " keyword consistency with similar foreign patent families " summarized in Table 1 above are calculated and stored in the evaluation element DB 193/194.

Meanwhile, the keyword extracting unit 113/133 for extracting a keyword based on the result of the natural language processing may include a keyword candidate selecting unit, a stopword removing unit, and a key word selecting unit. The key word candidate selection unit selects candidates of key words that best represent the subject of the individual patent. The abbreviated term eliminating unit removes the insignificant words having low importance from the extracted key word candidates. The key word selection unit finally selects a proper number of key words from the remaining key word candidates from which the stopwords have been removed, and stores them in the evaluation element DB 193/194.

The accuracy according to the number of key words is summarized in the following table.

Keyword count 10 things 50 Recall frequency 22.7% 54.1% Precision 20.6% 10.9%

Referring to Table 3, the probability of being recalled (i.e., the probability that a keyword will be reused) is 22.7% when 10 keywords are used, and 54.1% when the keyword is increased to 50 keywords. However, when the number of keywords is 50, the accuracy is as low as 10.9%. On the contrary, when the number of keywords is 10, the accuracy is 20.6%. As described above, if the number of key words is simply increased, the frequency of recalling may increase but the accuracy becomes lower. Therefore, the most optimal number of key words can be calculated based on the calculated accuracy.

The similar patent extracting units 114 and 134 serve to search for similar patents on the basis of the keyword, and may include a document clustering unit, a document similarity calculating unit, and a similar patent generating unit. The document clustering unit clusters similar patents primarily based on the keyword. The document similarity calculation unit calculates the degree of similarity among the patent documents among the clustered patents. Based on the calculated similarity, the similar patent generation unit generates actually the most similar patents among the primarily clustered patent documents, and stores the results in the similar patent DBs 195 and 196.

The patent evaluation engine sections (116, 136) include a learning model section and a patent evaluation section. And the learning model unit performs learning based on the expert evaluation result DB. To do this, experts in each technology field can receive evaluation results on sample patents.

The sample patent is a set for learning, and extracts hundreds to thousands of cases from the patent information DBs 191 and 192 and generates them. The sample patent may be extracted so that the values of the evaluation factors shown in Table 1 and Table 2 are uniformly included. For example, if the value of the above-mentioned evaluation factors such as invalid judgment, active right range judgment judgment, passive rights judgment judgment, and first judgment are not 0, the patent can be applied to all registered patents (approximately several tens to millions of patents) It is quite rare. Therefore, it is preferable to extract the sample patents so that the patents having non-zero values of the respective evaluation elements are distributed at a certain ratio. When the sample patents are selected, they can be roughly divided into a plurality of sets (for example, ten sets). Learning is performed using a plurality of sets of the plurality of sets, and the learning result is verified using the remaining sets.

In order to receive the expert evaluation result, the service server 120 or 140 provides the expert with the value of the evaluation factor for each sample patent and the evaluation item described above. For example, the service server 120 or 140 provides the computer of the expert with a web page listing the evaluation items (for example, the rightness, the technicality, and the usability) described above. In addition, the service server 120 or 140 provides the computer of the expert with a web page listing the evaluation factors as listed in Table 1 or Table 2. At this time, the service server 120 or 140 may map candidates of evaluation factors related to each evaluation item. Then, the expert inputs the score for each evaluation item on the web page while viewing the value of the relevant evaluation element candidates for each evaluation item, and the service server 120, 140 receives the score , And stores it in the expert evaluation DB.

Then, the learning model unit extracts evaluation factors actually related to each evaluation item based on the expert evaluation DB. Specifically, the learning model unit analyzes the correlation between each evaluation item and each evaluation element based on the expert evaluation DB. For example, if the result of the expert evaluation result is based on DB, if the value of the evaluation factor increases, the score of the evaluation item input by the expert increases, or when the value of the evaluation factor decreases, .

For example, the results of the analysis of the evaluation factors for the evaluation item "rightness" are shown in Table 4 below.

Evaluation factor Correlation Claims 0.442313457 Independent port number 0.43624915 Number of claims 0.369720889 Independent claim length -0.331545077 Support specification -0.21671485 Number of licensees -0.149643054 The number of words in the description of the invention 0.148732909 First, -0.145481466 Rights holder change -0.114275841 Third party review request 0.114180544 Early public availability -0.093078214 Number of invalid referees 0.092559136 Number of comments submitted 0.090248983 Litigation information 0.083141351 Decision of Refusal 0.061068447 Number of IPC 0.05544511 Annual registration number 0.053290108 Filing fee 0.049645717 Domestic family patent number -0.046240513 Total number of citations 0.045334312 First ref 0.038343259 Number of judges who confirm passive scope of right 0.02519305 Information Provided 0.015832255 Number of active judges 0.014261697 Overseas family patent number -0.009559996

In the correlation value summarized in Table 4, a negative value indicates that the value of the evaluation item increases when the value of the evaluation element decreases, and a positive value indicates that the value of the evaluation item increases when the value of the evaluation element increases Respectively.

Based on the correlation values listed in Table 3, the learning model unit extracts, for example, evaluation elements having a high correlation with the evaluation item " rightness ". The correlation value is a value between -1 and +1. If the value is usually 0.2 to 0.6, the correlation is high. Therefore, the learning model unit can select, for example, the number of claims, the number of independent factors, the number of claims, and the like as evaluation factors for the evaluation item " rightness ".

In order to solve such a problem, according to one embodiment of the present invention, it is possible to additionally calculate the correlation between experts.

Field expert Correlation Right Technical Usability Electronic A B 0.64 0.39 0.83 C D 0.29 0.32 0.48 machine E F 0.60 0.23 0.55 G H 0.59 0.23 0.63 chemistry I J 0.66 0.71 0.60 K L 0.59 0.34 0.50 physics M N 0.50 0.35 0.48 O P 0.81 0.15 0.80 Bio Q R 0.64 0.66 0.19 S T 0.51 0.45 0.38

As summarized in Table 5 above, for example, the technology field can be divided into electronic, mechanical, chemical, physical, and biotechnological fields. Then, experts can be assigned to a plurality of experts in each field, and then experts in each field can be paired. For example, in the case of the electronic field, experts A, B, C, D are assigned to experts A and B, and then experts C and D are evaluated Pairing is performed on the same patent. In this case, as a result of calculating the degree of correlation between experts A and B in the electronic field, as shown in Table 5, the evaluation item "rightness" was 0.64, the evaluation item "technicality" was 0.39, and the evaluation item "usability" 0.89. If the correlations between the pair of experts are low, one may use the results of the more highly correlated pair of experts, or may assign a higher weight to one of the alternatively paired experts.

In this manner, after the learning model unit defines the evaluation factors for each evaluation item, the learning model unit performs learning based on the evaluation scores of the experts stored in the expert evaluation DB. Specifically, the learning model unit calculates a weight value based on the evaluation score of the expert stored in the expert evaluation DB, and performs the learning using the calculated weight value. At this time, the learning can be performed by technology field. As described above, since the evaluation of the sample patent by the expert is made by the technical field, similarly, the above-mentioned learning is also performed for each technology field. For example, in the case of the mechanical field and the electronic field, the longer the independent claim length is, the narrower the claim range. In the case of the chemical field, however, the length of the independent claim and the scope of the claim range may be irrelevant. Learning is done on a technology-by-technology basis. Therefore, the above-described weight values can also be separately calculated for each technical field.

The patent evaluation section of the patent evaluation engine section (116, 136) evaluates the patents according to the learned results, and stores the evaluated results in the evaluation result DB (197, 198).

5 is a flowchart illustrating a method of constructing an evaluation engine by learning an expert evaluation result according to an embodiment of the present invention.

As can be seen from FIG. 5 and the above description, the evaluation items can be defined in advance (S110). As described above, the evaluation items can be defined as rights, technicalities, and usability. Alternatively, the evaluation items may be defined as rights property and commercial property. This definition may change depending on what the primary purpose of evaluating a patent is.

Then, the service servers 120 and 140 firstly map the evaluation items and evaluation factors for the sample patent, and provide them to the expert computer (S120). The primary mapping may be to map candidates of the inferred evaluation factor associated with each item of evaluation.

Then, the evaluation result of the sample patent may be received from the expert computer (S130). The evaluation result may be a score evaluated by the expert on the evaluation item. In order to provide information to the expert computer and receive the evaluation result, the service server 120 or 140 may prepare a web page in advance.

Subsequently, based on the evaluation results of the experts on the sample patent, the degree of correlation of the evaluation factors with respect to one or more predefined evaluation items can be calculated (S140). The correlation may be a value between -1 and +1 as described above.

Then, each evaluation item and the evaluation element can be remapped based on the calculated degree of correlation (S150). Some of the evaluation elements that are primarily mapped to the respective evaluation items by the remapping may be excluded from the mapping, and other evaluation elements may be mapped to the arbitrary evaluation items.

As such, when the mapping is completed, the evaluation result mapped to the evaluation item is used to learn the evaluation result of the expert, and an evaluation engine is constructed (S160).

FIG. 6 is a flowchart and a table showing an example of a method of verifying an evaluation engine constructed by learning an expert evaluation result.

First, as can be seen with reference to Fig. 6 (a), the evaluation engine construction step S160 of Fig. 5 is shown in more detail, and then the verification step S270 is shown.

The evaluation engine building step S160 includes a step S161 of classifying the evaluation results of the sample patents by the experts into a plurality of groups, a step S161 of scheduling one group of the plurality of groups for verification, And constructing an evaluation engine (S161). When the construction of the evaluation engine through learning is completed, the evaluation servers 110 and 130 verify the evaluation engine using the result of the expert evaluation of the reserved group (S170).

As can be seen from FIG. 6 (b), for example, when the expert evaluation results are divided into 10 groups, the evaluation engine is constructed based on the results of the expert evaluation of the second to tenth groups, Th group may be used to verify the evaluation engine. Alternatively, the evaluation engine may be constructed by learning based on the results of the expert evaluation of the first and third to tenth groups, and the evaluation engine may be verified by using the second group. In this way, a single group can be verified only once. At this time, the evaluation servers 110 and 130 can correct the calculated weight according to the result of the single verification.

Alternatively, the evaluation server 110, 130 may repeat the verification. The verification can be repeated, for example, by reserving groups for verification from the first to tenth groups, and learning each group with the remaining groups. That is, the first group can be verified first, and the second group can be verified. When the verification is repeated in this manner, the evaluation servers 110 and 130 can use the average result of each verification to correct the calculated weight value.

FIG. 7 is a flowchart showing another method for verifying an evaluation engine constructed by learning the expert evaluation result, and FIGS. 8 and 9 are the distribution diagrams shown in accordance with the other method shown in FIG.

As can be seen with reference to Fig. 7, when the evaluation engine building step S160 of Fig. 5 is completed, another verification scheme 170b can be performed. The other verification scheme 170b will be described in detail as follows.

First, the evaluation servers 110 and 130 extract patents having a value of a specific evaluation factor equal to or higher than a certain value (S171b). For example, the evaluation servers 110 and 130 extract patents with three or more independent claims as evaluation elements. And. The evaluation servers 110 and 130 calculate the evaluation result of the extracted patent (S172b). The evaluation result may be for all of the evaluation items, or for some evaluation items, for example, evaluation items related to the number of independent claims.

When the evaluation results are calculated, the evaluation servers 110 and 130 classify them based on the evaluation results (S173b). The grading may be a grading model having a normal distribution model. For example, the rating may be a 9 rating system. A +, A +, B +, B, B-, C +, C, C, and C may be systems such as AAA, AA, A, BBB, BB, B, CCC, CC, And so on.

When the grading is completed, the evaluation servers 110 and 130 analyze the distributions of the patents whose values of the specific evaluation element are greater than or equal to arbitrary values in comparison with the patents which are not equal to the predetermined value, And performs verification (S1734).

For example, in the case of FIG. 8 (a), the distributions shown in comparison with patents having 10 or more independent terms and general patents are classified into the general prior art patent evaluation system and the evaluation engine shown in this patent.

That is, referring to the left side of FIG. 8 (a), in the general conventional patent evaluation system, patents whose number of independent terms is 10 or more occupy the largest number of about 40% in BB grade. On the other hand, when evaluated according to the evaluation engine presented in this patent, patents with 10 or more independent clauses are distributed 60% free of charge on the AAA rank, and general patents with less than 10 independent clauses have a normal distribution.

In addition, referring to the left side of FIG. 8 (a), in the general conventional patent evaluation system, patents that have been invalidated are distributed in a ratio of 30% in the BBB grade and 30% in the BB grade. On the other hand, when evaluated according to the evaluation engine presented in the present patent, it can be seen that patents having invalidated judgments were moved upwards in comparison with the left distributions.

Meanwhile, FIG. 9 defines values of various evaluation factors to be possessed by superior patents, and patents exceeding these values are referred to as superior patents, and then the distribution of these superior patent groups is compared with the distribution of general patent groups. As can be seen from this, when evaluated according to the evaluation engine presented in the present patent, the superior patent group is ranked higher than the general patent group.

As described above, the evaluation servers 110 and 130 analyze the distributions of the patents whose values of the specific evaluation factors are greater than or equal to arbitrary values in comparison with those of the patents which are not, and check whether the distribution is normal or not, You can verify that the evaluation engine is working properly.

10 is a flowchart illustrating a method for providing a patent evaluation service using an evaluation engine according to an embodiment of the present invention.

As can be seen from FIG. 10, the service server 120, 140 may receive information on a specific patent from the user device (S210), and may receive an evaluation request for the specific patent from the user device (S220). To this end, the service server 120, 140 may provide a web page to the user's computer.

Then, the service server 120, 140 may provide an evaluation result calculated using an evaluation engine constructed in advance for a specific patent identified using the information, and output the evaluation result to the user's computer (S230) .

At this time, the service server 120 or 140 may merely provide an evaluation result, but may generate an evaluation report and provide the evaluation report to the user's computer. The evaluation report may include an evaluation result that is calculated and a detailed description of the evaluation result. Such an evaluation report may be in the same format as the PDF, or it may be based on a web page.

The embodiments disclosed herein have been described with reference to the accompanying drawings. Here, the above-described method can be implemented by various means. For example, embodiments of the present invention may be implemented by hardware, firmware, software, or a combination thereof.

For a hardware implementation, the method according to embodiments of the present invention may be implemented in one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs) , Field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, and the like.

In the case of an implementation by firmware or software, the method according to embodiments of the present invention may be implemented in the form of a module, a procedure or a function for performing the functions or operations described above. The software code can be stored in a memory unit and driven by the processor. The memory unit may be located inside or outside the processor, and may exchange data with the processor by various well-known means.

11 shows the physical configurations of the evaluation servers 110 and 130 and the service servers 120 and 140, respectively, which are shown in an embodiment of the present invention.

As shown in FIG. 11, the evaluation servers 110 and 130 may include transmission / reception units 110a and 130a, controllers 110b and 130b, storage units 110c and 130c, and each of the service servers 120 and 140 May include transmission / reception units 120a and 140a, controllers 120b and 140b, and storage units 120c and 140c.

The storing means stores the method shown in Figs. 3 and 10 and the above-described contents. For example, the storage means 110c and 130c of the evaluation servers 110 and 130 are respectively provided with the specification processing units 111 and 131, the natural language processing units 111 and 132, the keyword processing units 113 and 133, The processing units 114 and 134, the evaluation elements (or evaluation indexes) 115 and 135, and the evaluation engine units 116 and 136 may store the respective implemented programs. The storage means 120c and 140c of the service servers 120 and 140 may store at least one of the evaluation report generating units 121 and 141 and the portfolio analyzing units 122 and 142. [

The controller controls the transmission / reception unit and the storage unit. Specifically, the controllers execute the method or the programs stored in the storage means, respectively. The controller transmits and receives signals through the transceiver.

The embodiments disclosed herein have been described with reference to the accompanying drawings. Here, terms and words used in the present specification and claims should not be construed as limited to ordinary or dictionary meanings, but should be construed as meaning and concept consistent with the technical idea disclosed in the present specification.

Therefore, the embodiments described in the present specification and the configurations shown in the drawings are only examples described in the present specification, and not all of the technical ideas disclosed in the present specification are described. Therefore, various modifications It is to be understood that equivalents and modifications are possible.

110: domestic patent evaluation server
120: Domestic patent service server
130: Overseas (e.g., US) patent evaluation server
140: Overseas (e.g., US) Patent Service Server
191, 192, Patent information DB
193, 194: evaluation element (or evaluation index) DB
195, 196: Similar patent DB
197, 198: Evaluation results DB

Claims (11)

Obtaining a plurality of sample patents divided into a plurality of groups, and obtaining evaluation results performed by an expert for each group of sample patents on an evaluation item basis;
Establishing an evaluation engine by reserving an evaluation result of at least one group among the plurality of groups for verification and learning based on evaluation results of the remaining groups;
Calculating an evaluation result for the at least one group using the constructed evaluation engine, and performing a primary verification using the expert evaluation result of the reserved at least one group;
Calculating an evaluation result for a patent in which the value of the specific evaluation element is a random value or more;
And performing a secondary verification based on the rating distribution generated on the basis of the calculated evaluation result. delete The method of claim 1,
And verifying whether a patent whose value of the specific evaluation element is higher than a certain value is higher than that of a general patent. 2. The method of claim 1,
Information extracted from any one or more of the bibliographic information, progress information, specification, and claims of the registered patent,
And the extracted information is extracted by processing natural language of the specification and claims of the registered patent. The system of claim 1, wherein the evaluation engine
Calculating a degree of correlation of the evaluation elements with respect to at least one of the predefined evaluation items based on the evaluation results of the experts on the sample patent,
Mapping each evaluation item and the evaluation element based on the calculated degree of correlation;
Through one or more of the processes of learning the evaluation results of the experts using the evaluation elements mapped to the evaluation items
Wherein the system is configured to perform the verification of the evaluation engine for the patent evaluation system. 2. The method of claim 1, wherein after successful verification of the evaluation engine,
Receiving information about a particular patent from a user device;
Receiving an evaluation request for the specific patent from the user device;
And outputting an evaluation result calculated by using the constructed evaluation engine for a specific patent identified using the information. The method according to claim 6,
The results of the evaluation by the experts are performed for each technical field,
The evaluation engine is constructed for each technical field,
Wherein the step of outputting the evaluation result uses an evaluation engine of a technical field corresponding to the technical field of the specific patent. The method according to claim 1,
Wherein the verification engine includes at least one of a validity, a validity, a technicality, and a usability. The system of claim 1, wherein the evaluation engine
Wherein the correlation is calculated based on a result of evaluation by a plurality of experts in each technical field, and is constructed based on an evaluation result of an expert having an excellent correlation based on the calculated correlation. An evaluation engine verification method for a system. delete A receiving unit that divides a plurality of sample patents into a plurality of groups and receives evaluation results performed by an expert on an evaluation item for all groups of sample patents;
Wherein the evaluation results of at least one group among the plurality of groups are reserved for verification, and the evaluation engine is constructed by learning based on the evaluation results of the remaining groups, and the at least one group A first verification is performed by using the expert evaluation result of at least one of the reserved groups, and further, based on the evaluation result calculated for a patent having a value of a specific evaluation factor equal to or higher than a certain value And performing a secondary verification based on the distributed class distribution.

Images