US20200125606A1 - Method of tuning a computer system - Google Patents
Method of tuning a computer system Download PDFInfo
- Publication number
- US20200125606A1 US20200125606A1 US16/722,279 US201916722279A US2020125606A1 US 20200125606 A1 US20200125606 A1 US 20200125606A1 US 201916722279 A US201916722279 A US 201916722279A US 2020125606 A1 US2020125606 A1 US 2020125606A1
- Authority
- US
- United States
- Prior art keywords
- computer system
- documents
- accuracy
- target
- program product
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/93—Document management systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
-
- G06N7/005—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N7/00—Computing arrangements based on specific mathematical models
- G06N7/01—Probabilistic graphical models, e.g. probabilistic networks
Definitions
- a computer system that classifies or searches documents containing texts in natural languages or graphics based on the meanings conveyed by the texts or graphics can be useful in many areas.
- the texts may include numerals, formulas or other suitable forms.
- the graphics may include drawings, photos, diagrams, charts or other suitable forms.
- neither natural languages nor graphics are completely accurate.
- one word may have multiple meanings; multiple words may have the same meaning; and the meaning of a word, phrase, sentence or graphic may be affected by the context or the tone of the author.
- the meanings conveyed by a document may depend on information extrinsic to the document itself. For example, a reader's prior experience or mood may affect perception of graphics and texts in a document.
- the inaccuracy of natural languages and graphics or the effect of extrinsic information may lead to errors of the computer system. The errors may cause significant legal or financial risks to a user of the computer system.
- Disclosed herein is a method comprising: obtaining a target of accuracy of a computer system configured to classify documents or to locate texts satisfying a criterion in documents; tuning the accuracy of the computer system by adjusting a characteristic of the computer system based on the target.
- obtaining the target is based on a measure of risks caused by errors of the computer system.
- the method further comprises determining a measure of the accuracy; wherein adjusting the characteristic comprises comparing the measure with the target.
- the characteristic is an amount of training of the computer system.
- the characteristic is an amount of output validation of the computer system.
- the computer system is configured to iteratively receive validated data and wherein the characteristic is an amount of the validated data.
- the characteristic is a bias of the computer system.
- the computer system comprises a plurality of nodes, the nodes comprising one or more processors; wherein the characteristic is an attribute of couplings among the nodes.
- the computer system is configured to classify the documents into a plurality of classes.
- the computer system is probabilistic.
- the accuracy is a function of true positives, false positives, true negatives or false negatives of the computer system, or a function of the number of true positives, false positives, true negatives or false negatives of the computer system.
- the accuracy is a function of precision, recall or F-score of the computer system with respect to a class.
- Disclosed herein is a computer program product comprising a non-transitory computer readable medium having instructions recorded thereon, the instructions when executed by a computer implementing any of methods above.
- FIG. 1A schematically shows the function of a computer system that classifies documents based on their contents.
- FIG. 1B schematically shows the function of a computer system that locates, in multiple documents, texts satisfying a criterion.
- FIG. 1C schematically shows that the computer system of FIG. 1A or FIG. 1B may include a plurality of nodes that may collectively or individually have one or more processors.
- FIG. 2A schematically shows flowcharts of adjusting and using the computer system of FIG. 1A or FIG. 1B .
- FIG. 2B schematically shows a flowchart of iteratively adjusting and using the computer system of FIG. 1A or FIG. 1B .
- FIG. 3 schematically shows a flowchart of a method according to an embodiment.
- FIG. 4A - FIG. 4E show the progress of the credibilities in an example, where the actual value of the largest score is 0.8 and the actual value of the smallest score is 0.1.
- FIG. 1A schematically shows the function of a computer system 100 A that classifies documents based on their contents.
- the computer system 100 A receives multiple documents 101 , analyzes the contents of the documents 101 , and allocates the documents 101 into two or more classes (e.g., 101 A, 101 B, . . . , 101 Z) based on the contents.
- the computer system 100 A may be used, for example, for electronic discovery, where the computer system 100 A determines whether each of the documents 101 is relevant to a particular case or a particular issue in a case.
- FIG. 1B schematically shows the function of a computer system 100 B that locates, in multiple documents, texts satisfying a criterion.
- the computer system 100 B receives multiple documents 102 and a criterion 103 , analyzes the criterion 103 and the contents of the documents 102 , and locates texts that satisfy the criterion 103 in the documents 102 .
- the criterion 103 may be having meanings related to a particular notion or having forms conforming to a particular pattern.
- the computer system 100 B may determine whether a document among the documents 102 contains any text satisfying the criterion 103 , and may determine where the text satisfying the criterion 103 is in that document.
- the computer system 100 B may be used, for example, for exercising due diligence in a transaction.
- the computer system 100 B may locate portions among a set of contractual documents that contain provisions of potential interest such as intellectual property assignments or indemnities.
- FIG. 1C schematically shows that the computer system 100 A or 100 B may include a plurality of nodes 199 .
- the nodes 199 may collectively or individually have one or more processors.
- the nodes 199 form a network by coupling with one another.
- the one or more processors determine the output of each node 199 as a function of outputs of some other nodes 199 or inputs into the computer system 100 A or 100 B.
- Couplings among the nodes 199 may have adjustable attributes such as the parameters of the function. Changing the attributes may change strengths of the couplings (i.e., the magnitudes of dependency among the outputs of the nodes 199 ) or the topology of the network.
- the computer system 100 A or 100 B may be adjusted (e.g., by changing the adjustable attributes of the couplings among the nodes 199 ) based on labeled data.
- FIG. 2A schematically shows flowcharts of adjusting and using the computer system 100 A or 100 B.
- Labeled data 201 are provided to a procedure 210 , which adjusts the computer system 100 A or 100 B based on the labeled data 201 .
- the labeled data 201 may include a set of documents and the classes they belong to.
- the labeled data 201 may be a set of documents respectively labeled as “relevant” and “irrelevant.” In the scenario of locating texts in documents, the labeled data 201 may include a set of documents and the location of texts satisfying the criterion 103 in these documents.
- Unlabeled data 211 are provided to the computer system 100 A or 100 B, which turns the unlabeled data 211 to labeled data 212 .
- FIG. 2B schematically shows a flowchart of iteratively adjusting and using the computer system 100 A or 100 B.
- Labeled data 201 are provided to the procedure 210 , which adjusts the computer system 100 A or 100 B based on the labeled data 201 .
- Unlabeled data 211 are provided to the computer system 100 A or 100 B, which turns the unlabeled data 211 to labeled data 212 .
- Validated data 213 are produced by validating the labeled data 212 in a procedure 220 , in which the labels in the labeled data 212 are either upheld or overturned. The validation in the procedure 220 may be by a manual review of the labeled data 212 or by using another computer system.
- the validated data 213 are then provided to the procedure 210 to further adjust the computer system 100 A or 1008 .
- Validation may be performed after the computer system 100 A or 1008 produces each of the labeled data 212 or after the computer system 100 A or 1008 produces a batch of the labeled data 212 .
- the computer system 100 A or 1008 may be probabilistic.
- the unlabeled data 211 include a set of documents to be classified and the computer system 100 A or 1008 gives a score to each of the documents, where the score represents the probability of that document falling within a particular class (e.g., the “relevant” class or the “irrelevant” class).
- the scores may or may not be provided to the user of the computer system 100 A or 1008 .
- the documents in the unlabeled data 211 may be ranked based on their scores.
- a threshold may be applied on the scores and documents with scores above the threshold are allocated into one class and the documents with scores below the threshold are allocated into another class.
- a bias may be applied to the threshold. Adjusting the bias allows shifting documents among different classes.
- FIG. 3 schematically shows a flowchart of a method according to an embodiment.
- a target of accuracy of the computer system 100 A or 1008 is obtained.
- the accuracy of the computer system 100 A or 1008 may be a function of true positives, false positives, true negatives or false negatives of the computer system 100 A or 1008 .
- the accuracy of the computer system 100 A or 1008 may be a function of the “precision” or “recall” of the computer system 100 A or 1008 .
- the precision is defined as the ratio of the number of true positives to the sum of the number of true positives and the number false positives, with respect to one of the classes.
- the recall is defined as the ratio of the number of true positives to the sum of the number of true positives and the number false negatives, with respect to one of the classes.
- the precision may be the fraction that is actually relevant among the documents the computer system 100 A or 100 B has classified as relevant; the recall may be the fraction the computer system 100 A or 100 B has classified as relevant among all the documents that are actually relevant.
- the accuracy of the computer system 100 A or 100 B may be a function of the F-score of the computer system 100 A or 1008 .
- the F-score is the harmonic mean of the precision and recall.
- the target of accuracy may be obtained based on a measure of risks caused by errors of the computer system 100 A or 1008 .
- the user of the computer system 100 A or 100 B may provide subjective measures of risks for a number of different measures of the accuracy and provide a confidence threshold (e.g., a threshold between acceptable risks and unacceptable risks). For example, a user may consider the precision of 30%, 70%, 90% and 99% respectively as being catastrophic, severe, acceptable and satisfactory, respectively, and require a confidence threshold of 90%.
- the qualitative answers of “catastrophic,” “severe,” “acceptable” and “satisfactory” may be respectively assigned confidence levels of 1%, 50%, 80% and 100%.
- a statistical model may be used to ascertain the target of accuracy (the precision in this example) based on the subjective measures of risks and the confidence threshold.
- the accuracy of the computer system 100 A or 100 B is tuned by adjusting a characteristic of the computer system 100 A or 100 B based on the target of accuracy.
- the characteristic is an amount of training of the computer system 100 A or 1008 .
- the amount of training may be the amount of labeled data 201 or a combination of the amount of labeled data 201 and the amount of validated data 213 .
- the characteristic is an amount of output validation of the computer system 100 A or 1008 .
- the amount of output validation may be the amount of validated data 213 or the frequency or sample rate of the validation in the procedure 220 .
- the characteristic is the bias of the computer system 100 A or 100 B.
- the method may include an optional procedure 302 , in which a measure of accuracy of the computer system 100 A or 100 B is determined and adjusting the characteristic in procedure 303 includes comparing the measure of accuracy with the target of accuracy.
- the computer system 100 A or 100 B is probabilistic.
- the N unvalidated documents (D 1 , D 2 , . . . , D n , . . . , D N ) in the labeled data 212 thus respectively have scores, where each of the scores represents the probability of the corresponding document falling within a particular class (e.g., the “relevant” class).
- the function ⁇ is a harmonic function:
- p n , m 1 ( 1 - n - 1 N - 1 ) ⁇ 1 p 1 , m + ( n - 1 N - 1 ) ⁇ 1 p N , m .
- the credibilities (C 1 , C 2 , . . . , C m , . . . , C M ) for the M hypotheses are adjusted using the Bayesian inference.
- FIG. 4A - FIG. 4E show the progress of the credibilities in an example, where the actual value of the largest score is 0.8 and the actual value of the smallest score is 0.1.
- FIG. 4A - FIG. 4E respectively show the credibilities after 0, 10, 50, 100 and 250 documents have been validated. Even after only 10 documents have been validated, as shown in FIG. 4B , the hypotheses on the value of the largest score being near 0.8 are already favored (i.e., having higher credibilities). By the time when 50 documents have been validated, as shown in FIG. 4C , the hypotheses with high values of the smallest score are disfavored (i.e., their credibilities dropping). By the time when 250 documents have been validated, as shown in FIG. 4E , only those hypotheses close to the actual value of the largest score of 0.8 and the actual value of the smallest score of 0.1 are favored.
- the method disclosed herein may be embodied in a computer program product.
- the computer program product includes a non-transitory computer readable medium having instructions recorded thereon, the instructions when executed by a computer implementing the method disclosed herein.
Abstract
Description
- A computer system that classifies or searches documents containing texts in natural languages or graphics based on the meanings conveyed by the texts or graphics can be useful in many areas. The texts may include numerals, formulas or other suitable forms. The graphics may include drawings, photos, diagrams, charts or other suitable forms. However, neither natural languages nor graphics are completely accurate. For example, one word may have multiple meanings; multiple words may have the same meaning; and the meaning of a word, phrase, sentence or graphic may be affected by the context or the tone of the author. Further, the meanings conveyed by a document may depend on information extrinsic to the document itself. For example, a reader's prior experience or mood may affect perception of graphics and texts in a document. The inaccuracy of natural languages and graphics or the effect of extrinsic information may lead to errors of the computer system. The errors may cause significant legal or financial risks to a user of the computer system.
- Disclosed herein is a method comprising: obtaining a target of accuracy of a computer system configured to classify documents or to locate texts satisfying a criterion in documents; tuning the accuracy of the computer system by adjusting a characteristic of the computer system based on the target.
- According to an embodiment, obtaining the target is based on a measure of risks caused by errors of the computer system.
- According to an embodiment, the method further comprises determining a measure of the accuracy; wherein adjusting the characteristic comprises comparing the measure with the target.
- According to an embodiment, the characteristic is an amount of training of the computer system.
- According to an embodiment, the characteristic is an amount of output validation of the computer system.
- According to an embodiment, the computer system is configured to iteratively receive validated data and wherein the characteristic is an amount of the validated data.
- According to an embodiment, the characteristic is a bias of the computer system.
- According to an embodiment, the computer system comprises a plurality of nodes, the nodes comprising one or more processors; wherein the characteristic is an attribute of couplings among the nodes.
- According to an embodiment, the computer system is configured to classify the documents into a plurality of classes.
- According to an embodiment, the computer system is probabilistic.
- According to an embodiment, the accuracy is a function of true positives, false positives, true negatives or false negatives of the computer system, or a function of the number of true positives, false positives, true negatives or false negatives of the computer system.
- According to an embodiment, the accuracy is a function of precision, recall or F-score of the computer system with respect to a class.
- Disclosed herein is a computer program product comprising a non-transitory computer readable medium having instructions recorded thereon, the instructions when executed by a computer implementing any of methods above.
-
FIG. 1A schematically shows the function of a computer system that classifies documents based on their contents. -
FIG. 1B schematically shows the function of a computer system that locates, in multiple documents, texts satisfying a criterion. -
FIG. 1C schematically shows that the computer system ofFIG. 1A orFIG. 1B may include a plurality of nodes that may collectively or individually have one or more processors. -
FIG. 2A schematically shows flowcharts of adjusting and using the computer system ofFIG. 1A orFIG. 1B . -
FIG. 2B schematically shows a flowchart of iteratively adjusting and using the computer system ofFIG. 1A orFIG. 1B . -
FIG. 3 schematically shows a flowchart of a method according to an embodiment. -
FIG. 4A -FIG. 4E show the progress of the credibilities in an example, where the actual value of the largest score is 0.8 and the actual value of the smallest score is 0.1. -
FIG. 1A schematically shows the function of acomputer system 100A that classifies documents based on their contents. Thecomputer system 100A receivesmultiple documents 101, analyzes the contents of thedocuments 101, and allocates thedocuments 101 into two or more classes (e.g., 101A, 101B, . . . , 101Z) based on the contents. Thecomputer system 100A may be used, for example, for electronic discovery, where thecomputer system 100A determines whether each of thedocuments 101 is relevant to a particular case or a particular issue in a case. -
FIG. 1B schematically shows the function of acomputer system 100B that locates, in multiple documents, texts satisfying a criterion. Thecomputer system 100B receivesmultiple documents 102 and acriterion 103, analyzes thecriterion 103 and the contents of thedocuments 102, and locates texts that satisfy thecriterion 103 in thedocuments 102. For example, thecriterion 103 may be having meanings related to a particular notion or having forms conforming to a particular pattern. Thecomputer system 100B may determine whether a document among thedocuments 102 contains any text satisfying thecriterion 103, and may determine where the text satisfying thecriterion 103 is in that document. Thecomputer system 100B may be used, for example, for exercising due diligence in a transaction. For example, thecomputer system 100B may locate portions among a set of contractual documents that contain provisions of potential interest such as intellectual property assignments or indemnities. -
FIG. 1C schematically shows that thecomputer system nodes 199. Thenodes 199 may collectively or individually have one or more processors. Thenodes 199 form a network by coupling with one another. The one or more processors determine the output of eachnode 199 as a function of outputs of someother nodes 199 or inputs into thecomputer system nodes 199 may have adjustable attributes such as the parameters of the function. Changing the attributes may change strengths of the couplings (i.e., the magnitudes of dependency among the outputs of the nodes 199) or the topology of the network. - The
computer system FIG. 2A schematically shows flowcharts of adjusting and using thecomputer system data 201 are provided to aprocedure 210, which adjusts thecomputer system data 201. In the scenario of document classification, the labeleddata 201 may include a set of documents and the classes they belong to. For example, the labeleddata 201 may be a set of documents respectively labeled as “relevant” and “irrelevant.” In the scenario of locating texts in documents, the labeleddata 201 may include a set of documents and the location of texts satisfying thecriterion 103 in these documents. After thecomputer system data 201, it can be used to label unlabeled data.Unlabeled data 211 are provided to thecomputer system unlabeled data 211 to labeleddata 212. -
FIG. 2B schematically shows a flowchart of iteratively adjusting and using thecomputer system data 201 are provided to theprocedure 210, which adjusts thecomputer system data 201.Unlabeled data 211 are provided to thecomputer system unlabeled data 211 to labeleddata 212. Validateddata 213 are produced by validating the labeleddata 212 in aprocedure 220, in which the labels in the labeleddata 212 are either upheld or overturned. The validation in theprocedure 220 may be by a manual review of the labeleddata 212 or by using another computer system. The validateddata 213 are then provided to theprocedure 210 to further adjust thecomputer system 100A or 1008. Validation may be performed after thecomputer system 100A or 1008 produces each of the labeleddata 212 or after thecomputer system 100A or 1008 produces a batch of the labeleddata 212. - The
computer system 100A or 1008 may be probabilistic. In the scenario of document classification, theunlabeled data 211 include a set of documents to be classified and thecomputer system 100A or 1008 gives a score to each of the documents, where the score represents the probability of that document falling within a particular class (e.g., the “relevant” class or the “irrelevant” class). The scores may or may not be provided to the user of thecomputer system 100A or 1008. The documents in theunlabeled data 211 may be ranked based on their scores. A threshold may be applied on the scores and documents with scores above the threshold are allocated into one class and the documents with scores below the threshold are allocated into another class. A bias may be applied to the threshold. Adjusting the bias allows shifting documents among different classes. -
FIG. 3 schematically shows a flowchart of a method according to an embodiment. Inprocedure 301, a target of accuracy of thecomputer system 100A or 1008 is obtained. The accuracy of thecomputer system 100A or 1008 may be a function of true positives, false positives, true negatives or false negatives of thecomputer system 100A or 1008. For example, the accuracy of thecomputer system 100A or 1008 may be a function of the “precision” or “recall” of thecomputer system 100A or 1008. The precision is defined as the ratio of the number of true positives to the sum of the number of true positives and the number false positives, with respect to one of the classes. The recall is defined as the ratio of the number of true positives to the sum of the number of true positives and the number false negatives, with respect to one of the classes. In the scenario of document classification, the precision may be the fraction that is actually relevant among the documents thecomputer system computer system computer system computer system 100A or 1008. The F-score is the harmonic mean of the precision and recall. The target of accuracy may be obtained based on a measure of risks caused by errors of thecomputer system 100A or 1008. The user of thecomputer system - In procedure 303, the accuracy of the
computer system computer system computer system 100A or 1008. The amount of training may be the amount of labeleddata 201 or a combination of the amount of labeleddata 201 and the amount of validateddata 213. In another example, the characteristic is an amount of output validation of thecomputer system 100A or 1008. The amount of output validation may be the amount of validateddata 213 or the frequency or sample rate of the validation in theprocedure 220. In yet another example, the characteristic is the bias of thecomputer system - The method may include an
optional procedure 302, in which a measure of accuracy of thecomputer system - An example of adjusting the characteristic that is the amount of validated
data 213 is explained below. In this example, thecomputer system data 212 thus respectively have scores, where each of the scores represents the probability of the corresponding document falling within a particular class (e.g., the “relevant” class). Under a hypothesis Hm on the values of the largest score p1,m and the smallest score pN,m among all the scores, the score pn,m for document Dn is assumed to be a function of the largest score p1,m and the smallest score pN,m: pn,m=ƒ(p1,m, pN,m). In an example, the function ƒ is a harmonic function: -
- The hypothesis Hm has a credibility Cm that is the probability of the hypothesis Hm correctly estimating the value of the largest score as p1,m and the value of the smallest score as pN,m. If no prior assumption exists about the values of the largest score and the smallest score, the credibilities of all the M hypotheses may be equal: Cm=1/M.
- After the document Dn in the labeled
data 212 is validated, the credibilities (C1, C2, . . . , Cm, . . . , CM) for the M hypotheses are adjusted using the Bayesian inference. If the document Dn is validated as being in the particular class (e.g., the “relevant” class), a factor of pn,m/Z is applied to the credibilities; if the document Dn is validated as not being in the particular class (e.g., the “relevant” class), a factor of (1−pn,m)/Z is applied to the credibilities, where Z is a normalization factor that keeps Σm Cm=1 after the adjustment. Then, for the hypothesis Hm, a cumulative probability Qm of the target of accuracy of thecomputer system computer system computer system -
FIG. 4A -FIG. 4E show the progress of the credibilities in an example, where the actual value of the largest score is 0.8 and the actual value of the smallest score is 0.1.FIG. 4A -FIG. 4E respectively show the credibilities after 0, 10, 50, 100 and 250 documents have been validated. Even after only 10 documents have been validated, as shown inFIG. 4B , the hypotheses on the value of the largest score being near 0.8 are already favored (i.e., having higher credibilities). By the time when 50 documents have been validated, as shown inFIG. 4C , the hypotheses with high values of the smallest score are disfavored (i.e., their credibilities dropping). By the time when 250 documents have been validated, as shown inFIG. 4E , only those hypotheses close to the actual value of the largest score of 0.8 and the actual value of the smallest score of 0.1 are favored. - The method disclosed herein may be embodied in a computer program product. The computer program product includes a non-transitory computer readable medium having instructions recorded thereon, the instructions when executed by a computer implementing the method disclosed herein.
- While various aspects and embodiments have been disclosed herein, other aspects and embodiments will be apparent to those skilled in the art. The various aspects and embodiments disclosed herein are for purposes of illustration and are not intended to be limiting, with the true scope and spirit being indicated by the following claims.
Claims (28)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US16/722,279 US20200125606A1 (en) | 2018-07-13 | 2019-12-20 | Method of tuning a computer system |
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201862697679P | 2018-07-13 | 2018-07-13 | |
US16/149,523 US10558713B2 (en) | 2018-07-13 | 2018-10-02 | Method of tuning a computer system |
US16/722,279 US20200125606A1 (en) | 2018-07-13 | 2019-12-20 | Method of tuning a computer system |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/149,523 Continuation US10558713B2 (en) | 2018-07-13 | 2018-10-02 | Method of tuning a computer system |
Publications (1)
Publication Number | Publication Date |
---|---|
US20200125606A1 true US20200125606A1 (en) | 2020-04-23 |
Family
ID=69139490
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/149,523 Expired - Fee Related US10558713B2 (en) | 2018-07-13 | 2018-10-02 | Method of tuning a computer system |
US16/722,279 Abandoned US20200125606A1 (en) | 2018-07-13 | 2019-12-20 | Method of tuning a computer system |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/149,523 Expired - Fee Related US10558713B2 (en) | 2018-07-13 | 2018-10-02 | Method of tuning a computer system |
Country Status (2)
Country | Link |
---|---|
US (2) | US10558713B2 (en) |
WO (1) | WO2020011733A1 (en) |
Families Citing this family (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11188517B2 (en) | 2019-08-09 | 2021-11-30 | International Business Machines Corporation | Annotation assessment and ground truth construction |
US11294884B2 (en) * | 2019-08-09 | 2022-04-05 | International Business Machines Corporation | Annotation assessment and adjudication |
Family Cites Families (33)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5867799A (en) * | 1996-04-04 | 1999-02-02 | Lang; Andrew K. | Information system and method for filtering a massive flow of information entities to meet user information classification needs |
JP2004126840A (en) * | 2002-10-01 | 2004-04-22 | Hitachi Ltd | Document retrieval method, program, and system |
US9818136B1 (en) * | 2003-02-05 | 2017-11-14 | Steven M. Hoffberg | System and method for determining contingent relevance |
US8301482B2 (en) * | 2003-08-25 | 2012-10-30 | Tom Reynolds | Determining strategies for increasing loyalty of a population to an entity |
US8725711B2 (en) * | 2006-06-09 | 2014-05-13 | Advent Software, Inc. | Systems and methods for information categorization |
JP4405500B2 (en) * | 2006-12-08 | 2010-01-27 | インターナショナル・ビジネス・マシーンズ・コーポレーション | Evaluation method and apparatus for trend analysis system |
US8655939B2 (en) * | 2007-01-05 | 2014-02-18 | Digital Doors, Inc. | Electromagnetic pulse (EMP) hardened information infrastructure with extractor, cloud dispersal, secure storage, content analysis and classification and method therefor |
US9082080B2 (en) * | 2008-03-05 | 2015-07-14 | Kofax, Inc. | Systems and methods for organizing data sets |
US10872322B2 (en) * | 2008-03-21 | 2020-12-22 | Dressbot, Inc. | System and method for collaborative shopping, business and entertainment |
US8037043B2 (en) * | 2008-09-09 | 2011-10-11 | Microsoft Corporation | Information retrieval system |
US20100161534A1 (en) * | 2008-12-18 | 2010-06-24 | Yahoo! Inc. | Predictive gaussian process classification with reduced complexity |
US8873813B2 (en) * | 2012-09-17 | 2014-10-28 | Z Advanced Computing, Inc. | Application of Z-webs and Z-factors to analytics, search engine, learning, recognition, natural language, and other utilities |
US11074495B2 (en) * | 2013-02-28 | 2021-07-27 | Z Advanced Computing, Inc. (Zac) | System and method for extremely efficient image and pattern recognition and artificial intelligence platform |
US9916538B2 (en) * | 2012-09-15 | 2018-03-13 | Z Advanced Computing, Inc. | Method and system for feature detection |
US9461876B2 (en) * | 2012-08-29 | 2016-10-04 | Loci | System and method for fuzzy concept mapping, voting ontology crowd sourcing, and technology prediction |
US9501799B2 (en) * | 2012-11-08 | 2016-11-22 | Hartford Fire Insurance Company | System and method for determination of insurance classification of entities |
US20140143188A1 (en) * | 2012-11-16 | 2014-05-22 | Genformatic, Llc | Method of machine learning, employing bayesian latent class inference: combining multiple genomic feature detection algorithms to produce an integrated genomic feature set with specificity, sensitivity and accuracy |
US9582490B2 (en) * | 2013-07-12 | 2017-02-28 | Microsoft Technolog Licensing, LLC | Active labeling for computer-human interactive learning |
US9390086B2 (en) * | 2014-09-11 | 2016-07-12 | Palantir Technologies Inc. | Classification system with methodology for efficient verification |
WO2016167796A1 (en) * | 2015-04-17 | 2016-10-20 | Hewlett Packard Enterprise Development Lp | Hierarchical classifiers |
GB201508269D0 (en) * | 2015-05-14 | 2015-06-24 | Barletta Media Ltd | A system and method for providing a search engine, and a graphical user interface therefor |
US10395059B2 (en) * | 2015-07-15 | 2019-08-27 | Privacy Analytics Inc. | System and method to reduce a risk of re-identification of text de-identification tools |
US20170286521A1 (en) * | 2016-04-02 | 2017-10-05 | Mcafee, Inc. | Content classification |
US20170329972A1 (en) * | 2016-05-10 | 2017-11-16 | Quest Software Inc. | Determining a threat severity associated with an event |
US10460256B2 (en) * | 2016-08-09 | 2019-10-29 | Microsot Technology Licensing, LLC | Interactive performance visualization of multi-class classifier |
US11205103B2 (en) * | 2016-12-09 | 2021-12-21 | The Research Foundation for the State University | Semisupervised autoencoder for sentiment analysis |
US10679008B2 (en) * | 2016-12-16 | 2020-06-09 | Microsoft Technology Licensing, Llc | Knowledge base for analysis of text |
US10255269B2 (en) * | 2016-12-30 | 2019-04-09 | Microsoft Technology Licensing, Llc | Graph long short term memory for syntactic relationship discovery |
US20180240031A1 (en) * | 2017-02-17 | 2018-08-23 | Twitter, Inc. | Active learning system |
US11755997B2 (en) * | 2017-02-22 | 2023-09-12 | Anduin Transactions, Inc. | Compact presentation of automatically summarized information according to rule-based graphically represented information |
US10235357B2 (en) * | 2017-04-04 | 2019-03-19 | Architecture Technology Corporation | Community-based reporting and analysis system and method |
US20180300315A1 (en) * | 2017-04-14 | 2018-10-18 | Novabase Business Solutions, S.A. | Systems and methods for document processing using machine learning |
US10963941B2 (en) * | 2017-09-08 | 2021-03-30 | Nec Corporation | Method and system for combining user, item and review representations for recommender systems |
-
2018
- 2018-10-02 US US16/149,523 patent/US10558713B2/en not_active Expired - Fee Related
-
2019
- 2019-07-08 WO PCT/EP2019/068305 patent/WO2020011733A1/en active Application Filing
- 2019-12-20 US US16/722,279 patent/US20200125606A1/en not_active Abandoned
Also Published As
Publication number | Publication date |
---|---|
US20200019615A1 (en) | 2020-01-16 |
WO2020011733A1 (en) | 2020-01-16 |
US10558713B2 (en) | 2020-02-11 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US20230297890A1 (en) | Customizable machine learning models | |
US9852379B2 (en) | Systems and methods for constructed response scoring using metaphor detection | |
US8719192B2 (en) | Transfer of learning for query classification | |
US9224155B2 (en) | Systems and methods for managing publication of online advertisements | |
US20210157977A1 (en) | Display system, program, and storage medium | |
US11669740B2 (en) | Graph-based labeling rule augmentation for weakly supervised training of machine-learning-based named entity recognition | |
CN114492363B (en) | Small sample fine adjustment method, system and related device | |
US20200125606A1 (en) | Method of tuning a computer system | |
EP3929800A1 (en) | Skill word evaluation method and device, electronic device, and computer readable medium | |
CN112307336B (en) | Hot spot information mining and previewing method and device, computer equipment and storage medium | |
WO2023083176A1 (en) | Sample processing method and device and computer readable storage medium | |
KR20180128694A (en) | System and method for quality evaluation of research document | |
CN111400496B (en) | Public praise emotion analysis method for user behavior analysis | |
CN110543634B (en) | Corpus data set processing method and device, electronic equipment and storage medium | |
CN113869803A (en) | Enterprise sensitive information risk assessment method, system and storage medium | |
CN110472140B (en) | Object word recommendation method and device and electronic equipment | |
Zhang et al. | Probabilistic verb selection for data-to-text generation | |
US10621279B2 (en) | Conversion quality evaluation for digitized forms | |
US11550782B2 (en) | Machine learning systems and methods for interactive concept searching using attention scoring | |
CN112685532B (en) | Test question resource analysis method and device, electronic equipment and storage medium | |
US11947571B2 (en) | Efficient tagging of content items using multi-granular embeddings | |
US20220366346A1 (en) | Method and apparatus for document evaluation | |
Carroll et al. | Modeling the annotation process for ancient corpus creation | |
US20220164598A1 (en) | Determining a denoised named entity recognition model and a denoised relation extraction model | |
Mohan et al. | A Novel Feature Clustering Algorithm for Valuation of Descriptive Type Examination |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: RESPONSIML LTD, GREAT BRITAIN Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:LAVY, MATTHEW;TURNBULL, ALISTAIR;REEL/FRAME:051342/0104 Effective date: 20180813 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: APPLICATION DISPATCHED FROM PREEXAM, NOT YET DOCKETED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
AS | Assignment |
Owner name: LAVY, MATTHEW, UNITED KINGDOM Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:RESPONSIML LTD;REEL/FRAME:058537/0375 Effective date: 20211230 |
|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION |