US20160162458A1 - Graphical systems and methods for human-in-the-loop machine intelligence - Google Patents
Graphical systems and methods for human-in-the-loop machine intelligence Download PDFInfo
- Publication number
- US20160162458A1 US20160162458A1 US14/964,522 US201514964522A US2016162458A1 US 20160162458 A1 US20160162458 A1 US 20160162458A1 US 201514964522 A US201514964522 A US 201514964522A US 2016162458 A1 US2016162458 A1 US 2016162458A1
- Authority
- US
- United States
- Prior art keywords
- label
- pane
- annotation
- interface
- agreement
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
- 238000000034 method Methods 0.000 title claims abstract description 66
- 238000012790 confirmation Methods 0.000 claims abstract description 9
- 230000004931 aggregating effect Effects 0.000 claims description 7
- 230000010354 integration Effects 0.000 claims 6
- 230000004044 response Effects 0.000 abstract description 9
- 230000008520 organization Effects 0.000 abstract 1
- 230000009471 action Effects 0.000 description 20
- 230000015654 memory Effects 0.000 description 17
- 238000004891 communication Methods 0.000 description 15
- 230000006870 function Effects 0.000 description 15
- 230000008569 process Effects 0.000 description 13
- 238000004458 analytical method Methods 0.000 description 11
- 238000003860 storage Methods 0.000 description 10
- 238000003058 natural language processing Methods 0.000 description 9
- 238000012795 verification Methods 0.000 description 8
- 230000002776 aggregation Effects 0.000 description 6
- 238000004220 aggregation Methods 0.000 description 6
- 238000013473 artificial intelligence Methods 0.000 description 6
- 238000012545 processing Methods 0.000 description 6
- 230000000007 visual effect Effects 0.000 description 6
- 238000010586 diagram Methods 0.000 description 5
- 238000012552 review Methods 0.000 description 5
- 239000008186 active pharmaceutical agent Substances 0.000 description 4
- 230000003993 interaction Effects 0.000 description 4
- 238000010801 machine learning Methods 0.000 description 4
- 230000008901 benefit Effects 0.000 description 3
- 238000005094 computer simulation Methods 0.000 description 3
- 238000002790 cross-validation Methods 0.000 description 3
- 238000001514 detection method Methods 0.000 description 3
- 230000033001 locomotion Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 239000003086 colorant Substances 0.000 description 2
- 238000010276 construction Methods 0.000 description 2
- 239000010931 gold Substances 0.000 description 2
- 229910052737 gold Inorganic materials 0.000 description 2
- 238000012423 maintenance Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 239000013589 supplement Substances 0.000 description 2
- 238000012546 transfer Methods 0.000 description 2
- 241000283086 Equidae Species 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 238000007792 addition Methods 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000005266 casting Methods 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 238000005352 clarification Methods 0.000 description 1
- 230000000052 comparative effect Effects 0.000 description 1
- 238000013329 compounding Methods 0.000 description 1
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 230000003111 delayed effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000037213 diet Effects 0.000 description 1
- 235000005911 diet Nutrition 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 230000008713 feedback mechanism Effects 0.000 description 1
- 230000007274 generation of a signal involved in cell-cell signaling Effects 0.000 description 1
- 230000008676 import Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000000977 initiatory effect Effects 0.000 description 1
- 238000002372 labelling Methods 0.000 description 1
- 230000007786 learning performance Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 239000003550 marker Substances 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000003825 pressing Methods 0.000 description 1
- 239000000047 product Substances 0.000 description 1
- 230000001902 propagating effect Effects 0.000 description 1
- 238000007670 refining Methods 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 230000008054 signal transmission Effects 0.000 description 1
- 230000000087 stabilizing effect Effects 0.000 description 1
- 239000000126 substance Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
- G06Q50/01—Social networking
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/10—Text processing
- G06F40/166—Editing, e.g. inserting or deleting
- G06F40/169—Annotation, e.g. comment data or footnotes
-
- G06F17/241—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/242—Query formulation
- G06F16/243—Natural language query formulation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2453—Query optimisation
- G06F16/24532—Query optimisation of parallel queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
- G06F16/285—Clustering or classification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
- G06F16/288—Entity relationship models
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/33—Querying
- G06F16/332—Query formulation
- G06F16/3329—Natural language query formulation or dialogue systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/35—Clustering; Classification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/36—Creation of semantic tools, e.g. ontology or thesauri
- G06F16/367—Ontology
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/93—Document management systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/951—Indexing; Web crawling techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0481—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
- G06F3/0482—Interaction with lists of selectable items, e.g. menus
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/10—Text processing
- G06F40/12—Use of codes for handling textual entities
- G06F40/137—Hierarchical processing, e.g. outlines
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/205—Parsing
- G06F40/221—Parsing markup language streams
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/30—Semantic analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/40—Processing or translation of natural language
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/40—Processing or translation of natural language
- G06F40/42—Data-driven translation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
Definitions
- 1402805.00014_IDB014 titled “METHODS AND SYSTEMS FOR IMPROVING MACHINE LEARNING PERFORMANCE,” (Attorney Docket No. 1402805.000015_IDB015), titled “METHODS AND SYSTEMS FOR MODELING COMPLEX TAXONOMIES WITH NATURAL LANGUAGE UNDERSTANDING,” (Attorney Docket No. 1402805.00016_IDB016), titled “AN INTELLIGENT SYSTEM THAT DYNAMICALLY IMPROVES ITS KNOWLEDGE AND CODE-BASE FOR NATURAL LANGUAGE UNDERSTANDING,” (Attorney Docket No.
- a natural language modeling engine displays certain information on interfaces to confirm categorization, classification, or sorting the natural language modeling engine has performed for a set of documents into a scheme comprising various labels that may be present in the documents.
- the interfaces are dynamically linked across multiple interfaces, such that interaction or verifications on one interface may adjust the information displayed on a linked interface.
- Human communications in the digital age provide a deluge of information. Compounding the sheer volume of human communications in this technological era is the multitude of formats such human communications may come in, such as public news articles, social media posts, emails, customer feedback comments on a vendor website, or information circulated within a closed environment. It is difficult and time consuming for a human to integrate or recognize trends within these various communication formats with the volume of content involved in each, and any broad appreciation for what the creators of such communications may be expressing is therefore delayed. Recognizing trends or underlying meanings across the vast number of human communication sources, and appropriately categorizing unique nomenclature or slang that may be embedded, cannot be efficiently obtained without the aid of computer modeling tools.
- Graphic interface systems and methods for enabling human verification of natural language modeling by computer analysis tools, such as a natural language modeling engine, are described.
- the verification is achieved by presenting a document with a list of potential labels or tasks describing the document on a work unit interface and aggregating the responsive inputs to the work unit interface (hereinafter, such responsive inputs are referred to as “annotations”).
- An annotation is not necessarily a confirmation of a natural language modeling engine's prediction of a label or task of a document.
- An annotation to a document or subset of a document (referred to as a “span”) generally includes information indicating how the document or span should be classified into one or more topics or categories.
- an annotation is a corrective departure from the natural language modeling engine's prediction.
- Aggregated annotations can, in some embodiments, be displayed on related interfaces and further manipulated to determine the accuracy of a natural language modeling engine in categorizing the document into a hierarchical structure of labels and/or “tasks” associated with the document.
- a can be a clarification, reflection, sentiment, or other objective surrounding a document or label, such as, merely by way of example, “positive” or “negative” or various degrees between.
- “Tasks” may also refer to genres or groupings within labels and not merely binary interpretations of a label.
- the hierarchical structure may also be thought of as a categorization, or classification, and is hereinafter referred to as an “ontology.”
- the work unit interface may, in some embodiments, highlight or otherwise visually distinguish only a portion of the document (such as by underlining or italicizing), such portion of a document is referred to as a “span.”
- a collection of documents is accessed through a natural language modeling engine.
- the natural language modeling engine can organize the documents into an ontology by analyzing, grouping, and classifying them based on the words within the documents, and the natural language modeling engine's logic processing, or use of past keywords for classification.
- the ontology is displayed on a first graphic user interface (GUI).
- GUI graphic user interface
- the first GUI can include an option for the user of the first GUI, such as a project manager analyzing the documents, to send selected documents to a series of annotators to confirm the accuracy of the label or task for the particular document within the ontology as initially determined by the natural language modeling engine.
- the document is presented as part of a work unit interface displaying relevant panes for efficiently verifying the accuracy of the natural language modeling engine's ontology.
- a work unit interface is constructed by integrating a document, a label or task, a guideline describing the label/task or distinguishing it from other labels/tasks, and a human readable prompt soliciting a response from the annotator.
- the document is displayed in a document pane of the work unit interface, and a series of eligible labels or tasks are displayed in a label pane adjacent to the document pane. According to various embodiments, there may be one or more labels or tasks presented in the label pane of the work unit interface.
- the guideline describing the label or task is displayed on the work unit interface as a reference button adjacent to its respective label or task and when activated by a user of the work unit interface (such as by “clicking,” or “pressing” or hovering a cursor over the reference button) the reference button opens a guideline pane displaying the label or task description.
- the human readable prompt is displayed in a prompt pane of the work unit interface.
- the human readable prompt is generated from an intelligent queuing module of a natural language modeling engine.
- the human readable prompt requests confirmation of a label or task of the document displayed in the document pane as predicted by the natural language modeling engine, or requests the user of the work unit interface to select the most applicable label or task for the document from among a plurality of displayed labels or tasks in the label pane, and in still other embodiments requests the user identify all labels or tasks that are related to the document.
- the human readable prompt requests confirmation of a label or task of the document displayed in the document pane as predicted by the natural language modeling engine, or requests the user of the work unit interface to select the most applicable label or task for the document from among a plurality of displayed labels or tasks in the label pane, and in still other embodiments requests the user identify all labels or tasks that are related to the document.
- the work unit interface is presented on a second GUI displayed to an expert annotator.
- the second GUI when displaying a work unit interface, may include a create pane configured to receive an additional label or task, or revised guideline for a respective label or task within the label pane of the work unit interface.
- the second GUI may update the respective work unit interface on one or more second or third GUIs also configured to display the particular work unit interface.
- Third GUIs in some embodiments, display the work unit interface to annotators that are not expert annotators, and such work unit interfaces do not include a create pane.
- updates from second GUIs may change what is displayed on work unit interfaces displayed on other second or third GUIs, and also affect which work unit interfaces are displayed to other expert annotators on a second GUI or to other annotators on a third GUI.
- updates to the work unit interface may be the replacement of a label or task with a new label or task entered into a create pane.
- updates to the work unit interface may be a new human readable prompt for selection of a label or task (for example, an expert annotator could direct a work unit interface to prompt “choose a label” rather than “confirm if the presented label is applicable”).
- updates to the work unit interface may be to replace an existing guideline with a revised guideline entered into a create pane.
- updates to the work unit interface may be supplement the given label(s) or task(s) with the label or task entered into a create pane.
- updates to the work unit interface may be to supplement a given guideline to a label or task with a revised guideline entered into a create pane.
- the third GUI operated by an annotator, or second GUI operated by an expert annotator receives an action on the work unit interface responsive to the human readable prompt.
- the received action is an annotation of a label or a task of the document as requested by the human readable prompt.
- the annotation is aggregated with all annotations to that particular work unit interface displayed received all third GUIs and second GUIs that were presented the work unit interface.
- the aggregated annotations are collected by the natural language modeling engine and displayed on the first GUI for further analysis and interaction by, for example, a project manager for the collection of documents, though one of skill in the art will envision other users or suitable roles for operating first GUI to interact with an aggregation of annotations.
- the aggregated annotations are displayed on the first GUI in an annotation agreement interface comprising a series of information panes.
- a label feedback pane of the annotation agreement interface displays a plurality of label panes for each label or task within an ontology and displays the number of annotations the respective label or task received, and options to delete or edit the label or task.
- the annotation agreement interface includes a learning curve pane, such learning curve displays a graphical representation of the number of annotations received for a particular label or task and the agreement among those annotations for the accuracy of that particular label or task.
- the annotation agreement interface includes an annotation feedback pane.
- the annotation feedback pane may display an aggregate annotation agreement score representing the overall accuracy of the ontology as determined by the annotation agreements across all labels or tasks of the ontology.
- the annotation feedback pane includes an individual annotator agreement list displaying the agreement scores of each individual annotator relative to the other annotators.
- the annotation feedback pane includes a suggestion pane for collapsing labels or tasks into one another.
- the annotation agreement interface can reduce disagreements between annotators of a particular label or task, or reduce confusion annotators may have over minute differences between labels or tasks. For example, if the work unit interface displays labels of “securities” and “stock” for a particular document, an annotator may have trouble distinguishing the two and the respective labels will have a low annotation agreement score, however, if the two labels were recalculated as a common label, the annotation agreement score may improve.
- Such collapsed label or task calculations may indicate, to a project manager or user of a first GUI, the need to refine the labels or tasks within the ontology, or the need to refine the guidelines describing the labels or tasks.
- the annotation agreement interface includes an agreement per label or task graphical representation, such as a bar chart displaying the agreement per label or task relative to the other labels or tasks in the ontology that received annotations.
- the agreement per label or task graphical representation further includes a collapsed agreement per label or task graphical representation displaying the agreement per label or task if two or more labels or tasks were collapsed into one another.
- the annotation agreement interface includes a per document agreement list displaying information such as which documents received the highest agreement or which documents received the lowest agreement.
- a user of the first GUI could potentially remove documents with lower annotation agreement from the ontology to improve the accuracy of the model much in the same way other analysis methods remove ambiguous information or “noise” from those data sets.
- the aggregated annotations can then be used to display verified analysis of the natural language process engine ontology and indicate trends and sentiments contained within the collection of documents for that ontology with a certain degree of reliance (the annotation agreement score most readily serving as a proxy for reliance of the ontology).
- an ontology comprising a collection of thousands of “tweets” from a Twitter hashtag of # Tesla could divide the tweets into labels based on common words across the tweets, such as “battery,” “autonomous,” and “Elon Musk” with tasks related to each label such as “positive” or “negative” and display the number of tweets that fall within each label and task and the number of annotations to each tweet and the annotation agreement amongst annotator to give a fast overview of the general disposition of the tweets within the ontology.
- FIG. 1 is a network diagram illustrating an example network environment suitable for performing aspects of the present disclosure, according to some embodiments.
- FIG. 2 is an illustration of a system diagram showing of a first, second, and third graphic user interface (GUI) operably coupled to a natural language modeling engine, according to some embodiments.
- GUI graphic user interface
- FIG. 3 illustrates a sample interface display of a first GUI accessing a collection of documents, according to some embodiments.
- FIG. 4 illustrates a sample interface display of a first GUI presenting a landing page for a user of a first GUI to access features and maintenance, information of a collection of documents to a user, and annotation assignments according to some embodiments.
- FIG. 5 illustrates a sample interface display of a first GUI for accessing a plurality of documents to create a collection of documents, according to some embodiments.
- FIG. 6 illustrates a sample interface display of a first GUI for reviewing automatic topic modeling incident to an ontology of a collection of documents based on keywords within the collection, according to some embodiments.
- FIG. 7A illustrates a sample interface display of a first GUI for creating an ontology of a collection of documents based on keywords entered into the interface, according to some embodiments.
- FIG. 7B illustrates a sample interface display of a first GUI for visually presenting topic relevance and relationship to other topics, according to some embodiments.
- FIG. 8A illustrates a sample work unit interface arrangement of panes for collecting annotations from an expert annotator, according to some embodiments.
- FIG. 8B illustrates a sample work unit interface arrangement of panes for collecting annotations from an annotator, according to some embodiments.
- FIG. 8C illustrates a sample work unit interface arrangement of panes for displaying guideline information of a label or task, according to some embodiments.
- FIGS. 8D-E illustrates a sample work unit interface arrangement of panes for displaying annotating spans and selecting spans, according to some embodiments.
- FIG. 9A illustrates a sample interface display of a label feedback pane within a first GUI for displaying characteristics of a label within an ontology, according to some embodiments.
- FIG. 9B illustrates a sample interface display of a label feedback pane within a first GUI for displaying a learning curve of the relationship between annotation agreement as a function of the number of annotations received, according to some embodiments.
- FIG. 10A illustrates a sample interface display of an annotation feedback pane within a first GUI for displaying annotation information relative to labels or tasks and annotators, according to some embodiments.
- FIG. 10B illustrates a sample interface display of an annotation feedback pane within a first GUI for displaying graphical representations of annotation agreements relative to collapsing labels into one another, according to some embodiments.
- FIG. 10C illustrates a sample interface display of a label feedback pane within a first GUI for displaying annotation agreements relative to specific documents within the collection of documents.
- FIG. 11 illustrates a sample interface display of a rules feedback pane within a first GUI for displaying adjustment options to modeling and interpreting certain labels or tasks, according to some embodiments.
- FIG. 12 illustrates an example method for creating and integrating an interface for collecting and aggregating annotations across GUIs, according to some example embodiments.
- FIG. 13 illustrates an example method for building an annotation agreement interface for analyzing annotations of labels or tasks within an ontology, according to some embodiments.
- FIG. 14 illustrates an example method of an interface for collapsing labels or tasks into one another and displaying comparative annotation agreements, according to some embodiments.
- FIG. 15 illustrates an example method of updating a work unit interface with revised guidelines, according to some example embodiments.
- FIG. 16 illustrates a block diagram illustrating components of a machine, according to some example embodiments, able to read instructions from a machine-readable medium and perform any one or more of the methodologies discussed herein.
- Systems, methods, and apparatuses are presented for causing the display of certain information and efficiently sharing responsive inputs to verify the accuracy of computer modeling in natural language processing.
- Marketing outlets, internal compliance departments, customer service departments, or even design and engineering industries can greatly benefit from rapid analysis of communications about products and policies.
- analysis of these documents by human review can only be performed in a timely fashion by choosing a small sample of the communication. Such small sampling introduces error as to any conclusions, by casting doubt to how representative a sample is of the whole.
- Computing modeling can analyze an entire corpus of documents to rapidly identify the general trends and commentary across all documents by using artificial intelligence to recognize keywords, syntax, and relation within documents in a much more timely fashion.
- a natural language modeling engine can build an ontology of documents demonstrative of overall sentiments and underlying meaning of trends across all documents.
- FIG. 1 illustrates a network for relating users across interfaces for annotating documents that may be classified according to an ontology created by a natural language modeling engine.
- Network 100 includes user 132 operating device 130 , user 152 operating device 150 , and network based system 105 .
- User 132 or user 152 may be one of a project manager, expert annotator, or annotator.
- Device 130 or device 150 may be a mobile device, desktop computer, tablet, or other computing device configured to operate any one of the interfaces described herein.
- Connecting user 132 or user 152 to network based system 105 through device 130 or device 150 is network 190 .
- Network 190 may be a wireless network (such as wide area network, local area network), ethernet connections or other wired connection, or other suitable network system for linking computing devices.
- Network based system 105 can include a server machine 110 configured to perform natural language modeling according to some embodiments as further described in this detailed description, and a database 115 .
- Database 115 may store a collection of documents for server machine 110 to access and create an ontology around, or may store artificial intelligence rules for server machine 110 to access and apply to sorting a collection of documents, or may store guidelines and labels or tasks that have been used in other ontologies that server machine 110 can access to build additional ontologies.
- database 115 can store the labels and tasks and guidelines used in that ontology to inform server machine 110 which labels and tasks and guideline may be relevant in a subsequent “JP Morgan Chase” ontology relating to, for example purposes only, “customer service.”
- user 132 or 152 is a customer or other third party seeking to have a collection of documents analyzed or classified into an ontology and transfers a collection of documents to network based system 105 through device 130 or 150 via network 190 .
- network based system 105 creates an ontology of a collection of documents, and displays a work unit interface on a second or third GUI to verify the accuracy of the ontology to user 132 or user 152 through device 130 or 152 .
- User 130 or 150 annotates the information in their respective interface, and network based system 105 aggregates the annotations to refine the ontology or draw further conclusions about the underlying documents and displays the results on a first GUI, which may be user 132 or user 152 depending on the embodiment.
- FIG. 2 illustrates a diagram of interface system 200 relating a first, second, and third GUI with a natural language modeling engine 210 .
- natural language modeling engine 210 is network based system 105 as described in FIG. 1 .
- natural language modeling engine 210 comprises a series of modules to include a database, which may be the same database as database 115 as described in FIG. 1 ; and an input/output (I/O) module configured to receive and transmit information throughout interface system 200 .
- natural language modeling engine 210 further comprises an API module configured to communicate through an I/O module with various devices, GUIs, and operating systems interacting with natural language modeling engine 210 .
- natural language modeling engine 210 further comprises an intelligent queuing module configured to generate human readable prompts for populating a work unit interface to elicit an annotation from expert annotators on a second GUI or annotators on a third GUI.
- natural language modeling engine 210 further comprises an annotation module for constructing a work unit interface with at least a document from a database, a human readable prompt from an intelligent queuing module, and a label or task from a database.
- an annotation module is further configured to aggregate received annotations from across a plurality of work unit interfaces and compute agreements and relationships between the aggregated annotations.
- natural language modeling engine 210 further comprises a modeling module for constructing an ontology from a collection of documents on a database.
- a modeling module is configured to modify an ontology in response to aggregated annotations received from a plurality of work unit interfaces across second GUIs 214 or third GUIs 216 , or the manipulations to an aggregated set of annotations as received from a user of a first GUI 212 .
- first GUI 212 operated by a project manager or similar user role for managing the collection and using information created by the ontology
- second GUI 214 operated by an expert annotator
- third GUI 216 operated by an annotator.
- first GUI 212 , second GUI 214 and third GUI 216 may, in some embodiments, be directly connected to one another, such that an input from second GUI 214 may be transmitted to third GUI 216 without intermediary communication with natural language modeling engine 210 .
- a project manager can, through first GUI 212 , access a collection of documents in natural language modeling engine 210 .
- the project manager can select certain topics, create topics, or review a set of keywords for suggested topics provided by natural language modeling engine 210 , and the modeling module of natural language modeling engine 210 can then build an ontology for that topic with a hierarchical structure of labels and tasks of the documents related to that topic.
- a project manager may want to determine the general disposition of the tweets as, and if, they relate to Chase banking.
- the project manager can create a topic for the tweets specifically with the name “Chase Bank” and the modeling module responds by identifying and organizing the tweets into an ontology based on that topic with sub labels and tasks further refining the disposition of the tweets.
- certain tweets will be excluded, for example those relating to baseball player Chase Utley that may otherwise have the hashtag # Chase in it.
- a “relevant” and “irrelevant” label within the ontology will distinguish the # Chase tweets relating to banking or baseball.
- the annotation module of natural language modeling engine 210 creates a work unit interface to display to expert annotators operating second GUI 214 or annotators operating third GUI 216 .
- interface system 200 may include a plurality of second GUI 214 or third GUI 216 .
- the annotation module selects certain documents from the ontology displayed on first GUI 212 to populate a work unit interface.
- the document selected is merely to confirm the accuracy of the label or task of the placement within the ontology.
- the document is selected because the modeling module cannot determine which label or task of the ontology to document should be applied to based on its own processing rules.
- second GUI 214 is distinguished from third GUI 216 by having a create option when displaying a work unit interface.
- the work unit interface is described in further detail in other parts of this disclosure.
- second GUI 214 or third GUI 216 receive annotations from a respective expert annotator or annotator, and the annotation module of natural language modeling engine 210 aggregates the annotations.
- First GUI 212 displays the aggregated annotations to a project manager to indicate the accuracy of the ontology or suggestions for categorizing the information to be more accurate.
- FIG. 3 illustrates an example embodiment for a control page 300 displayed to a project manager or other user of first GUI 212 .
- control page 300 is displayed after a project manager or similar user has accessed a system operating the natural language modeling engine, such as the one described in FIG. 2 , with a login credential identifying the user as a project manager or other similar role for managing collections.
- control page 300 displays a collections pane 310 and a sharing and permissions pane 320 .
- Collections pane 310 permits a user of first GUI 212 to create a new project for analyzing a collection of documents in a natural language modeling engine by engaging a create new collection action 312 , or access previous projects by engaging a view all collections action 314 .
- sharing and permissions pane 320 permits a user of first GUI 212 to allow other users, such as user 132 or user 152 to access any analytical information of a collection of documents managed by a project manager.
- FIG. 4 illustrates an example embodiment of dashboard 400 that is presented to a user after accessing the natural language modeling engine, such as the one described in FIG. 2 .
- dashboard 400 directs a user to updates to, and new features provided by, the natural language engine model in a features pane 410 to keep first GUI functioning with the latest operational capabilities.
- Dashboard 400 in some embodiments includes a maintenance pane 412 to identify system services or other alerts, such as the availability of the natural language modeling engine.
- dashboard 400 includes a current projects pane 420 .
- Current projects pane 420 is configured to display a variety of information about projects the user accessing dashboard 400 is managing or has permissions to view, such as model accuracy of a particular project of a collection of documents, the number of annotations outstanding for that particular project, or the topics and labels and tasks of the particular project.
- Current projects pane 420 in some embodiments, includes a view all projects action button to view more projects within current projects pane 420 , or a create new topic action button to direct the user to a topic creation series of displays described more fully below.
- dashboard 400 further includes annotation pane 430 for the user of first GUI 212 to access a series of work unit interfaces created for annotating.
- dashboard 400 is displayed on second GUI 214 or third GUI 216 as described in FIG. 2 and displays annotation pane 430 to those respective users for accessing documents for annotation by the respective user.
- Dashboard 400 recognizes an expert annotator or annotator from a login credential and directs the expert annotator or annotator to the annotations pane 430 for annotating documents through their respective work unit interfaces.
- FIG. 5 illustrates a sample display for initiating the creation of an ontology from a collection of documents.
- Collection pane 500 is displayed on first GUI 212 and comprises a client pane 510 , classification pane 520 , and collection data pane 530 .
- client pane 510 includes a document action button, a topic action button, and create task button.
- a document action button permits a user of first GUI 212 to access the documents within a collection associated with a particular client, such as a customer that has uploaded documents to the natural language modeling engine.
- a topic action button permits a user of first GUI 212 to view alternative or additional topics in a collection to allow additional ontologies for those topics as necessary or determined by the user of first GUI 212 .
- a create task button permits a user of first GUI 212 to add labels or tasks to an ontology in addition to any that may have been created by a natural language modeling engine.
- classification pane 520 is displayed to the user on collection pane 500 to display a suggested ontology based on previous ontologies and collections the user of first GUI 212 has used.
- classification pane 520 is populated with other labels and tasks associated with a particular client as identified in client pane 510 . For example, if a particular client as identified through client pane 510 by a natural language modeling engine has historically and consistently used certain labels and tasks for categorizing a collection of documents, the natural language modeling engine can build a new ontology with those historic labels and tasks for a new collection of documents and display the resulting ontology in classification pane 520 .
- collection data pane 530 displays documents that are part of a collection to be analyzed.
- collection data pane 530 includes an import data action button to initiate access to a collection of documents, such as a collection provided by a client or customer, for analysis.
- classification pane 520 displays an ontology reactive to the documents present in collection data pane 530 , such that an ontology is created or previous ontologies are updated as documents accessed through collection data pane 530 are interfaced with and annotated.
- collections pane 500 includes a discover topics action button 540 to initiate an automatic construction of an ontology through a discover topics interface more fully described in conjunction with FIG. 6 .
- FIG. 6 illustrates a collection pane 500 configured to display a discover topic action button 610 .
- discover topic action button 610 prompts a natural language modeling engine to analyze the collection of documents accessed by first GUI 212 , such as through collections data pane 530 as described in FIG. 5 , for common themes and keywords.
- a first topic suggestion 612 is displayed in collection pane 500 on first GUI 212 with second topic suggestion 614 , though many other iterations and numbers of topic suggestions are possible.
- Engaging discover topic action button 610 can prompt the natural language modeling engine to form a first topic and ontology around the word “horse” and sub labels and tasks such as “breed,” “positive,” “diet” and a second topic and ontology around the word “betting” with sub labels and tasks such as “owner” and “race dates.”
- a topic suggestion includes further displays of the keywords identified to justify the creation of the topic suggestion.
- keywords pane 615 displays at least one keyword recognized through a plurality of the documents in a collection and in some embodiments, a plurality of documents 616 containing that keyword is further displayed in collection pane 500 .
- classification pane 520 is configured to display the suggested topics 612 and 614 and keywords 615 as an ontology; that is, the suggested topics 612 and keywords 615 can be used in some embodiments as the label and tasks for a collection of documents to create an ontology displayed in classification pane 520 .
- a user of first GUI 212 can review the suggested topics, and select which set of keywords identified by the natural language modeling engine best reflect the needs of the project and thereby select a topic for building an ontology to display in classification pane 520 .
- a user of first GUI 212 can see how the keywords are used to further gauge the context of the keywords and not simply the presence of the word before choosing a topic and building an ontology.
- a user of first GUI 212 can determine those documents are not truly indicative of racing or betting and remove them from the ontology, or decide to choose another topic that may have plurality of documents 616 more inline with the desired keywords.
- keywords 615 are identified independently of topic suggestion 612 or 614 .
- an ontology is created around keywords 615 without a threshold topic to group those keywords under.
- each keyword within keywords 615 is a label or task for an ontology.
- a natural language modeling engine can create an ontology from only the keywords.
- labels such as “relevant” or “irrelevant” may be more important to distinguish which documents are applicable for a label or task, as a threshold topic selection may not have filtered these documents.
- first GUI 212 can create an ontology independently through a create classification task pane 720 displayed in collection pane 500 on a first user GUI 212 .
- first GUI 212 automatically discovers suggested topics, such as suggested topics 612 and 614 depicted in FIG. 6 , and displays a keywords suggestion pane 710 populated with keyword sets identified across those topics. Keyword suggestion pane 710 can determine keywords much in the same way as described in FIG. 6 by recognizing common words and identifying major themes across a collection of documents.
- Keywords suggestion pane 710 can allow the user to quickly assess the correlation among the keyword set, the context, presence of slang, and other subjective factors to create an appropriate topic around.
- a topic label pane 730 is further presented in collection pane 500 to receive a user's specific input for the labels and tasks to a collection of documents.
- keywords can be readily used for labels or tasks
- a user of first GUI may have specific labels in mind for an ontology and can direct a natural language modeling engine to build an ontology on those directed labels.
- non-intuitive relationships can be constructed that artificial intelligence may not yet be programmed for or experienced enough to identify on its own.
- the natural language modeling engine determines which label(s) are applicable to a document and does not know where to place a particular document into the ontology and selected the document for annotation.
- a natural language modeling engine selects documents for verification of placement, despite a high likelihood of successful categorization.
- the natural language modeling engine constructs a work unit interface to efficiently receive annotations for such verification or placement.
- keyword suggestion pane 710 is a visual presentation rather than a purely textual list of keywords.
- keyword suggestion pane 710 is accompanied by topic circle graph 740 displaying possible topics based on a number of keywords supporting a topic. For example, a topic with more keywords related to it is displayed as a larger topic circle. Additionally, in topic circle graph 740 the relation between topics is visually displayed. In some embodiments, topics with more keywords in common with another topic are displayed closer together on topic circle graph 740 .
- relevant terms window 750 displays keywords across the collection of documents.
- the frequency of a particular keyword within a topic selected by a user in topic circle graph 740 is displayed in relevant terms window 750 with a comparison for how frequently that keyword appears in other documents across the collection. For example, if the word “banking” appears four hundred times across a collection of documents, and four hundred times in a particular topic a user could readily deduce that “banking” is very relevant to the collection of documents and that an ontology for the collection of documents should include the word “banking.”
- FIG. 8A depicts an example expert annotator work unit interface 800 as presented on a second GUI 214 constructed by a natural language modeling engine.
- work unit interface 800 comprises a document pane 810 , a prompt pane 820 , label pane 830 comprising at least one label or task, reference button 835 paired to a label or task within label pane 830 , create label pane 840 and create guideline pane 845 .
- expert annotator work unit interface 800 is presented on second GUI 214 upon a user logging into a natural language modeling engine with an expert login credential.
- second GUI 214 may display an annotation pane 430 directing the user to expert annotator work unit interface 800 for annotations from the user operating second GUI 214 .
- Expert annotator work unit interface 800 is constructed by a natural language modeling engine displaying in document pane 810 a document from the ontology to be annotated, and listing at least one label or task from the ontology in label pane 830 .
- natural language modeling engine accesses a database of guidelines defining or otherwise describing the label or task displayed in label pane 830 and pairs the guideline with the respective label or task.
- reference button 835 is a link to the guideline paired with a label or task in label pane 830 . The function of a reference button is further described in conjunction with FIG. 8C .
- the intelligent queuing module of natural language modeling engine 210 such as the one described in FIG. 2 , generates a human readable prompt to elicit a label or task selection by a user of expert annotator work unit interface 800 .
- human readable prompt include, but are not limited to, “select the best label for the document” from a plurality of labels in label pane 830 , “select all labels that apply to the document” from a plurality of labels in label pane 830 , “rank the labels in order of relevance” from a plurality of labels in label pane 830 , or binary response prompts such as “does this label apply to the document?” with yes and no labels in label pane 830 .
- work unit interface 800 displays document 810 ′ in the document pane 810 and prompt 820 ′ in prompt pane 820 .
- work unit interface 800 displays document 810 ′ in the document pane 810 and prompt 820 ′′ in prompt pane 820 .
- human readable prompt 820 elicits the selection or categorization of portions of text displayed in document pane 810 .
- Examples of such prompts include, but are not limited to, “select all examples of each label within the document” with a list of labels in label pane 830 and “is the highlighted section of the document an example of this label?” with yes and no labels in label pane 830 .
- work unit interface 800 displays document 810 ′ in document pane 810 with a plurality of example regions of text visually distinguished from the rest of the document to assist the annotator.
- the example regions of text are created using the API module of natural language modeling engine 210 .
- Examples of visual representations for example regions include, but are not limited to, using unique background colors around the example region to highlight the text, and underlining the example text regions.
- the example region is distinguished with variable degrees of visual representation to reflect a natural language modeling engine's confidence in selecting an example region as a correct example of a label or task. For example, in some embodiments, a thicker underlined example region indicates stronger confidence as opposed to a thinner underlined example region, or an opaque highlighted background color as opposed to a semi-transparent background color.
- prompt pane 820 is populated with the generated human readable prompt.
- expert annotator work unit interface 800 permits the user of second GUI 214 to populate create label pane 840 with a new label.
- Expert annotators creating new labels for documents can distinguish certain nuances in documents that normal annotators or a natural language modeling engine cannot, such as legal interpretations or advanced sciences that may have distinct meanings in a particular field.
- the expert annotator can update other work unit interfaces with the created labels the expert annotator has identified, and update an ontology with more accurate categorizations.
- expert annotator work unit interface 800 includes a create guideline pane 845 permitting the user of second GUI 214 to populate create label pane 845 with a revised guideline to pair with a particular label to provide more descriptive information to help other expert annotators or other annotators interpret the applicability of a label or task rather than simply rely on the guideline provided by the natural language modeling engine.
- create guideline pane 845 receives a “gold” designation from an expert annotator to indicate a particular label or task is particularly representative or a good example otherwise of the document or prompt displayed in a work unit interface.
- Such “gold” or similar exemplary marker is displayed in a work unit interface displayed to other expert annotators or annotators as a guideline explained more fully as 890 in describing FIG. 8C .
- Such designations indicate not only that an expert annotator wants to draw attention to a particular label or task, but can also be used to train other expert annotators or annotators to what the particular label or task should be representing.
- an annotation of a document is made on expert annotator work unit interface 800 by selecting a label or task displayed in label pane 830 and the annotation is recorded by a natural language modeling engine.
- work unit interface 800 subsequent to selection of a label displayed in label pane 830 , immediately displays an additional human readable prompt in prompt pane 820 and populates at least one subsequent label in label pane 830 responsive to the earlier label selection of the first human readable prompt. For example, if the expert annotator answers “Yes” to a first human readable prompt about document relevance, prompt pane 820 may immediately display an additional human readable prompt requesting the best label for the document. By contrast, if the expert annotator answers “No” to the first prompt, an additional human readable prompt is not displayed.
- the additional human readable prompt created for prompt pane 820 matches the ontology structure displayed in classification tasks pane 520 .
- annotations of a document are made for all labels and tasks in an ontology by selecting a label or task in label pane 830 for each additional human readable prompt.
- FIG. 8B illustrates an embodiment of an annotator work unit interface 850 as displayed on a third GUI 216 .
- annotator work unit interface 800 is presented on third GUI 216 upon a user logging into a natural language modeling engine with an annotator login credential.
- third GUI 216 may display an annotation pane 430 directing the user to annotator work unit interface 850 for annotations from the annotator operating third GUI 216 .
- Annotator work unit interface 850 is constructed by displaying in document pane 860 a document from the ontology to be annotated, and listing at least one label or task from the ontology in label pane 880 .
- natural language modeling engine accesses a database of guidelines defining or otherwise describing the label or task displayed in label pane 880 and pairs the guideline with the respective label or task.
- reference button 885 is a link to the guideline paired with a label or task to present to the annotator on third GUI 216 the definition of the particular paired label or task in label pane 880 . The function of a reference button is further described in conjunction with FIG. 8C .
- the intelligent queuing module of natural language modeling engine 210 such as the one described in FIG. 2 , generates a human readable prompt to elicit a label or task selection by a user of annotator work unit interface 850 .
- human readable prompt include, but are not limited to, “select the best label for the document” from a plurality of label panes 880 , “select all labels that apply to the document” from a plurality of label panes 880 , “rank the labels in order of relevance” from a plurality of label panes 880 , or binary response prompts such as “does this label apply to the document?” with yes and no label panes 880 .
- prompt pane 820 in describing expert annotator work unit interface 800 is applicable in the various prompts possible in describing prompt pane 870 .
- prompt pane 870 of annotator work unit interface 850 is populated with the generated human readable prompt.
- human readable prompt 870 elicits the selection or categorization of portions of text displayed in document pane 860 .
- Examples of such prompts include, but are not limited to, “select all examples of each label within the document” with a list of labels in label pane 880 and “is the highlighted section of the document an example of this label?” with yes and no labels in label pane 880 .
- work unit interface 850 displays document 860 ′ in document pane 860 with a plurality of example regions of text visually distinguished from the rest of the document to assist the annotator.
- the example regions of text are created using the API module of natural language modeling engine 210 .
- Examples of visual representations for example regions include, but are not limited to, using unique background colors around the example region to highlight the text, and underlining the example text regions.
- the example region is distinguished with variable degrees of visual representation to reflect a natural language modeling engine's confidence in selecting an example region as a correct example of a label or task. For example, in some embodiments, a thicker underlined example region indicates stronger confidence as opposed to a thinner underlined example region, or an opaque highlighted background color as opposed to a semi-transparent background color.
- an annotation of a document is made on annotator work unit interface 850 by selecting a label or task displayed in at least one label pane 880 and the annotation is recorded by a natural language modeling engine.
- work unit interface 850 subsequent to selection of a label displayed in label pane 880 , immediately displays an additional human readable prompt in prompt pane 870 and populates at least one subsequent label in label pane 880 responsive to the earlier label selection of the first human readable prompt. For example, if the annotator answers “Yes” to a first human readable prompt about document relevance, prompt pane 870 may immediately display an additional human readable prompt requesting the best label for the document. By contrast, if the annotator answers “No” to the first prompt, an additional human readable prompt is not displayed.
- the additional human readable prompt created for prompt pane 870 matches the ontology structure displayed in classification tasks pane 520 .
- annotations of a document are made for all labels and tasks in an ontology by selecting a label or task in label pane 880 for each additional human readable prompt.
- FIG. 8C illustrates a reference button function on either an expert annotator work unit interface 800 or annotator work unit interface 850 and the respective reference button 835 or reference button 885 .
- guideline 890 is displayed within the work unit interface.
- the guideline 890 displayed is the guideline accessed from a database of a natural language modeling engine for a particular label; in some embodiments the guideline 890 displayed is the revised guideline as created by an expert annotator through expert annotation work unit interface 800 create guideline pane 845 on second GUI 214 , and updated on all work unit interfaces present on GUIs within an interface system 200 such as one depicted in FIG. 2 .
- guideline 890 displays both the guideline accessed from a database of natural language modeling engine as well as the revised guideline, if any, created by the expert annotator.
- the annotations received on all work unit interfaces 800 and 850 are aggregated together by ontology the documents underlying the work unit interface were drawn from.
- the aggregation occurs in an annotation module of a natural language modeling engine 210 and are shared on a first GUI 212 such as depicted in FIG. 2 .
- FIG. 8D illustrates a span annotation work unit interface interaction.
- a work unit interface displays a document in a document pane and span prompt 891 in a prompt pane.
- span prompt 891 is associated with span annotation 892 in a label pane.
- a natural language modeling engine predicts spans of a document that represent a particular label or task and displays the predicted span as a highlighted text or underlined text 893 in the document displayed in the document pane of the work unit interface.
- an expert annotator or annotator annotates the span prediction with a span annotation 892 in the label pane.
- a span prompt 891 requests confirmation of multiple spans, such as “location” and “person” and a toggle or swatch or menu function in the document pane switches between the prompted spans and displays different highlighted or underlined text 893 corresponding to the span prompted in span prompt 891 .
- a span prompt requests confirmation of a document that the highlighted or underlined text represents “people” and “locations.”
- An expert annotator or annotator selects a toggle, swatch, or menu function in the document pane for “locations” and the work unit interface displays those highlighted or underlined texts the natural language modeling engine has predicted correspond to “locations.”
- the expert annotator or annotator then annotates with a span annotation 892 in the label pane and then presses the toggle, swatch, or menu function for “people” and the work unit interface displays those highlighted or underlined texts the natural language modeling engine has predicted correspond to “people.”
- the expert annotator or annotator then annotates for “people” and the natural language modeling engine processes the span annotations.
- FIG. 8E illustrate a span selection work unit interface.
- span selection prompt 894 is displayed in a prompt pane of a work unit interface requesting an annotator highlight or underline or otherwise visually distinguish a span within a document pane.
- an expert annotator or annotator annotates an otherwise unmarked document in a document pane by highlighting or otherwise visually distinguishing a span 895 within the document.
- a natural language modeling engine predicts which span corresponds to requested label or task in span prompt 894 by visually distinguishing a span in a separate method than an expert annotator or annotator would.
- a natural language modeling engine will instruct a work unit interface to present underlined spans 896 the natural language modeling engine predicts correspond to “locations,” and an expert annotator or annotator can confirm the prediction by highlighting the underlined span or highlighting other spans to annotate the document.
- FIG. 9A illustrates an annotation agreement interface 900 display on a first GUI 212 with a label feedback pane 910 , annotation feedback pane 920 , and rules pane 930 .
- label feedback pane 910 includes a by label description of each label within an ontology with the number of annotations applied to the label and an option to delete or edit the label.
- a user of first GUI 212 can remove a label from an ontology if the annotation agreement for that label is low or the user of first GUI 212 determines it is not applicable to the ontology, or edit the label such as by reviewing a create label action as provided through a second GUI 214 and determining that label more applicable or descriptive.
- the user of first GUI 212 can determine not enough annotations have been applied to the label to draw any conclusions from and decide to wait before making any adjustments to that label.
- FIG. 9B illustrates an example of a learning curve 915 within label feedback pane 910 .
- learning curve 915 is a graphical representation of the relationship between the number of annotations received for a particular label and the agreement between the annotations for the label.
- learning curve 915 is a graphical representation of the relationship between the number of annotations received for a particular label and the accuracy of the natural language model generated for that label.
- Annotation agreement to a label is calculated by an annotations module of a natural language modeling engine, such as natural language modeling engine 210 as depicted in FIG. 2 .
- the annotation agreement is a number indicating the incidence rate of mutually agreed annotations among all annotators operating second GUIs 214 or third GUIs 216 .
- an annotation agreement For example purposes only of one way to calculate an annotation agreement, if 10 annotators all annotated a document with a label of “positive” and 10 annotators did not annotate the same documents as “positive,” then an annotation agreement of 0.50 or 50% would be reflected for the “positive” label or task of the document and learning curve 915 of those labels or tasks would depict the 0.50 or 50% agreement for 20 annotations. In the same example, if the next 20 annotators gave a “positive” annotation to the same document, the annotation agreement would update to 75% for a “positive” annotation for 40 forty annotations and learning curve 915 would graphically depict this relationship between annotation agreement as a function of the number annotations.
- One of skill in the art can appreciate other annotation agreement calculation methods.
- the accuracy of the natural language modeling engine in assigning documents to appropriate labels or tasks of an ontology is derived from cross-validation processes of the annotations used in a learning curve.
- a modeling module of a natural language modeling engine such as natural language modeling engine 210 as depicted in FIG. 2 , performs cross-validation on the annotation dataset to determine ontology accuracy.
- One of skill in the art can appreciate applicable cross-validation techniques to apply to an annotation dataset such as exhaustive or non-exhaustive methods.
- FIG. 10A illustrates an example display of an annotation feedback pane 920 with annotation agreement score 1010 , individual annotator agreement list 1030 , and suggested label collapse list 1020 .
- annotation agreement score 1010 displays an aggregate annotation agreement result as a proxy for the accuracy of an ontology.
- the annotation agreement score 1010 is determined by aggregating all annotations from second or third GUIs 214 or 216 for a particular set of documents that have been categorized into an ontology.
- Such an aggregation in some embodiments, reflect the total annotation agreement of the whole ontology and give a project manager or other user of a first GUI 212 a rapid feedback mechanism of how well the natural language modeling engine categorized a collection of documents based on how human readers agreed a particular label or task applied to the same collection of documents.
- Such feedback confirmation can inform a project manager or other user of a first GUI 212 the degree of agreement among annotators on whether the labels or tasks presented on a work unit interface accurately reflected the document displayed on the work unit interface.
- a low agreement score would indicate that the labels or tasks may have been too vague or inapplicable, and the annotators could not agree on whether, or which of, the labels presented in a label pane of applied to the document of the work unit interface and therefore, perhaps the natural language modeling engine did not have the most appropriate series of labels or tasks to categorize the ontology.
- a higher agreement score could indicate the annotators found at least one label of a label pane was an accurate match to the document based on the generated prompt, and therefore the annotators found at least one the labels readily applied and the natural language modeling engine accurately captured the label or task of at least some documents.
- annotation agreement score 1010 can be broken down into a per label agreement, and in suggested label collapse list 1020 indicate which labels or tasks introduced higher disagreement among annotator and display the annotation agreement score that would result if certain labels or tasks were collapsed—or combined—with each other.
- the suggested label collapse list 1020 displays the resulting annotation agreement score 1010 of collapsing certain labels or tasks into one of several other labels or tasks of a hypothetical ontology (e.g. collapsing “Legal” with “Securities issues” and “JPMC Financial” with “Other”).
- collapsing “Legal” with “Securities issues” and “JPMC Financial” with “Other e.g. collapsing “Legal” with “Securities issues” and “JPMC Financial” with “Other”.
- collapsing “Legal” with “Other” results in an annotation agreement score 1010 of 0.773 as compared to an annotation agreement score 1010 of 0.741 if the two were separate labels or tasks.
- this represents the largest increase annotation agreement score 1010 within the suggested label collapse list 1020 a user of first GUI 212 can readily deduce that annotators had a greater difficulty distinguishing “Legal” from “Other” than they did distinguishing any other two labels or tasks within the ontology.
- a user of first GUI 212 can make several other deductions from this information, such as whether the guideline describing “Legal” or “Other” sufficiently describes the label or task, or whether the prompt should be changed to permit more nuanced distinctions, or whether the ontology itself should not include a particular label or task.
- an individual annotator agreement list 1030 displays how well a particular annotator within the aggregation of annotations agrees with other annotators. For example, as displayed in FIG. 10A , the annotator identified as “demo 4 ” has an agreement value of 0.601, the lowest of the annotator group depicted in FIG. 10A , indicating that demo 4 has a low incidence of agreeing with the other annotators (i.e. “demo 1 ,” “demo 2 ,” and “demo 3 ”) on a label or task as prompted in a work unit interface. As depicted in FIG. 10A , annotator demo 4 also has only annotated 100 documents compared to the other annotators' 550 .
- This display of information could suggest that annotator demo 4 may need to be removed from the annotation group (for example, the subject matter confuses that annotator) to compute an annotation agreement score 1010 that reflects annotations with less annotators that may not understand the subject matter and could be giving false positives on applicable labels to a natural language modeling engine.
- This information could also be used to indicate annotator demo 4 needs to be retrained on a particular subject matter depending on the degree of disagreement. For example, if annotator demo 4 selected labels of “slightly positive” as compared to other annotators selecting “positive” then a user of GUI 212 could decide to retrain annotator demo 4 .
- GUI 212 could decide to remove annotator demo 4 from the analysis.
- a user of first GUI 212 could simply decide to wait for demo 4 to annotate more documents to see if demo 4 ′s agreement value increases; such information can further be used to determine whether or not labels should be collapsed or if a project needs further annotation before drawing conclusions on the natural language modeling engine's accuracy.
- FIG. 10B illustrates an agreement per label graphical representation 1040 , and collapsed agreement per label graphical representation 1045 .
- agreement per label graphical representation 1040 displays individual annotation agreements by each label of an ontology, that, taken all together would comprise annotation agreement score 1010 as depicted in FIG. 10A . Breaking down the annotation agreements into a per label graphical representation informs a project manager or user of first GUI 212 which labels or tasks, relative to others, had the most agreement and can inform whether the label or task is likely an appropriate reflection of documents in a work unit interface.
- the annotation agreement interface includes a collapsed agreement per label graphical representation 1045 configured to display the per label annotation agreements if two or more labels were collapsed into one another.
- collapsed agreement per label graphical representation 1045 indicates that by combining two labels with a respective agreement of 0.613 and 0.274 from agreement per label graphical representation 1040 into one another, a new agreement score of 0.59 results for that combined label in collapsed agreement per label graphical representation 1045 suggesting the annotation agreement score 1010 as illustrated in FIG. 10A will improve if these two labels are combined into one another, and enabling similar deductions by a project manager or user of first GUI 212 as described in connection with those functions of FIG. 10A .
- FIG. 10C illustrates a per document agreement list 1050 within an annotation feedback pane 920 .
- a per document agreement list 1050 displays those documents within the ontology that have the highest agreement or lowest agreement among them.
- the number of documents displayed in per document agreement list 1050 can be adjusted by a user of first GUI 212 .
- the per document agreement list 1050 displays those documents within a subset of label or task of an ontology with the highest or lowest agreement among annotators.
- a user of first GUI 212 can choose to remove certain documents with low agreement to reduce the number of potentially vague or inapplicable documents within a collection (as indicated by not having strong human agreement on the applicable labels or tasks), or review the content of the particular document and label or task of a work unit interface displaying the document to determine whether the document contains a nuance that should be included as a new label or whether a guideline should be clarified to account for such nuances.
- an annotation agreement interface can further display a rules pane 930 for adjusting any of the natural language modeling engine logic processes for certain inputs.
- the rules pane comprises a phrase pane 1122 configured to display a data field for receiving a phrase or word that, if found within a document by a natural language modeling engine, will invoke a rule.
- rules pane 930 displays a weighting adjustment pane 1124 .
- weighting adjustment pane 1124 is configured to receive from a user of first GUI 212 a manipulation to a certain phrase or word in phrase pane 1122 to emphasize or de-emphasize a certain word in placing a document in a label or task category of an ontology.
- a natural language modeling engine may categorize incidence of the word “recommend” as equivalent to “recommendation” without recognizing the context of the complete phrase or document “recommend” appears in, such as “does not recommend” which would not imply an positive recommendation.
- Weighting pane 1124 permits a user of first GUI 212 to reduce or increase the significance of certain words, thereby placing greater or less emphasis on other words in the document relative to the word or phrase in phase pane 1122 . Such weighting can focus the ontology on which labels or tasks to create, or which documents should be selected for annotation.
- the natural language modeling engine may not select it for annotation to avoid using an annotator's time to accurately place a document with a low weighting.
- rules pane 930 includes add rule pane 1126 .
- Add rule pane 1126 permits a user of first GUI 212 , or in some embodiments an expert annotator operating second GUI 214 , to create a rule for a particular phrase or word in phrase pane 1122 . For example, if the word “recommend” appears in a document, add rule pane 1126 could be manipulated by receiving in a data field of add rule pane 1126 a rule such as “if this then that” logic rules, or rules to search for additional words surrounding a word in a phrase pane 1122 .
- add rule pane 1126 could receive a rule to search for preceding words such as “no,” “does not,” “isn't,” or other similar negative implicative words such that if “recommend” is paired with such a negative implicative word, the natural language modeling engine will not categorize the document as an affirmative “recommendation.”
- a label feedback pane 910 , annotation feedback pane 920 , and rules pane 930 of an annotation agreement interface of a first GUI 212 permits rapid analysis of an ontology that has been annotated through second and third GUIs 214 and 216 .
- Annotation agreement interface 900 further provides access to a variety of tools to determine where an ontology and its attendant labels or tasks can be refined to more accurately determine the underlying meaning of a collection of documents or store information for future ontologies to learn from.
- the natural language modeling engine can store that revised guideline in a database and used that guideline for future ontologies that use the same label or task the revised guideline is describing.
- the wealth of information and deductions possible from human annotations to computer analysis tools and the insights such annotations provide can greatly improve mass classification of human communications.
- FIG. 12 illustrates an example of process 1200 for verifying the accuracy of a natural language modeling engine's creation of an ontology of a collection of documents by aggregating human annotations across a series of GUIs.
- Process 1200 starts at 1210 with accessing a series of inputs through a first GUI, such inputs being those associated with being made by a project manager or similar supervisory role to a collection of documents.
- accessing inputs at 1210 includes accessing at least one document at 1212 , such document provided by a third party.
- a third party access source in some embodiments, is a customer that provides a collection of documents to be analyzed; in some embodiments, third party access source is a database of a collection of documents such as the database 115 as depicted in FIG. 1 .
- accessing inputs 1210 further includes accessing at least one first label associated with the document at 1214 .
- the first label is accessed from an ontology created around the document as built from a natural language modeling engine.
- accessing inputs 1210 further includes accessing a plurality of first guidelines describing the first label at 1216 .
- the first guideline is sourced from a database of guidelines, such as one operated by a natural language modeling engine, that are associated with a list of labels and each first guideline is a description or definition of the label or task.
- a second label or second guideline is accessed, such as from a second GUI 214 like the one operated by an expert annotator described in FIG. 2 .
- the access of inputs at 1210 and second guidelines and second labels at 1220 provide a plurality of information components for the construction of a work unit interface to permit human annotation to selected documents.
- a work unit interface is built.
- building the work unit interface at 1230 involves assigning (which depending on embodiment can mean “populating” or “placing”) the document accessed at 1212 to a document pane of the work unit interface at 1231 .
- labels are assigned to, and populated in, a label pane of a work unit interface.
- the labels assigned to the label pane at 1232 are the first labels accessed at 1214 from an ontology of a collection of documents through a first GUI.
- the labels or tasks assigned to the label pane at 1232 are the second labels accessed at 1220 from a second GUI.
- the label pane is assigned with a plurality of labels, and in still other embodiments, the plurality of labels assigned the label pane at 1232 includes both first labels and second labels.
- a human readable prompt is generated to elicit a response from a human annotator that requests a task of the document.
- the human readable prompt is generated by an intelligent queuing module of a natural language modeling engine.
- the human readable prompt is a question requesting selection of the most applicable label or task assigned in the label pane at 1232 for the document assigned in the document pane at 1231 .
- the human readable prompts requests selection of all applicable labels or tasks assigned in the label pane of a work unit interface at 1232 .
- the generated human readable prompt is assigned to a prompt pane of the work unit interface.
- a single guideline from one of the first guideline accessed at 1216 or second guideline at 1220 is paired with a single label or task assigned to the label pane of the work unit interface at 1232 .
- several labels or tasks are assigned to the label pane at 1232
- several single guidelines are paired with a single one of the several labels or tasks at 1235 .
- a reference button is created for the single guideline at 1236 .
- a reference button permits access to the full textual description of the single label paired with the single label without requiring display of the single guideline.
- the reference button is placed adjacent to the single label paired with the single button in the label pane at 1237 .
- process 1200 continues at 1240 by displaying the built work unit interface to an annotator operating a second GUI or third GUI.
- the annotator operating the second GUI is an expert annotator.
- the annotator operating the third GUI is an annotator.
- at 1250 at least one annotation is received through the work unit interface from among the second or third GUIs.
- the annotations received at 1250 are aggregated together.
- process 1300 provides a method of creating interfaces to efficiently manage and manipulate annotated documents to verify and draw conclusions as to the accuracy of a natural language modeling engine.
- Method 1300 begins at 1310 , and in some embodiments step 1310 is a subsequent to step 1260 as described in FIG. 12 .
- an annotation agreement interface is built.
- building an annotation agreement interface at 1310 includes a series of substeps 1312 , 1314 , 1316 , and/or 1318 .
- a label feedback pane is created at 1312 .
- a label feedback pane includes a plurality of labels or tasks of an ontology from a collection of documents, and in some embodiments further includes a description of the label or task, an indicator of the number of annotations, or action buttons to edit or delete the label or task from the ontology. Editing, in some embodiments, may include removing annotations from the label or task, or applying a new guideline describing the label or task.
- a learning curve pane is created.
- the learning curve pane displays an aggregation of annotations from among GUIs in a network, such as network 200 as depicted in FIG. 2 with a plurality of second GUIs 214 or third GUIs 216 .
- the learning curve pane displays a graphical representation of the relationship between the number of annotations received and the agreement between those annotations.
- the learning curve pane displays a graphical representation of the relationship between the number of annotations received for a particular label and the accuracy of the natural language model generated for that label.
- the learning curve displayed at 1314 is a learning curve for an entire collection. In some embodiments, the learning curve displayed at 1314 is a learning curve for a particular label or task.
- a project manager or user of first GUI 212 can determine whether the annotations are beginning to smooth out and/or approach a consistent agreement regardless of additional annotations or whether the number of additional annotations introduces continued variability (which would be represented as a staggered line in a learning curve) of agreement. With such information, a project manager or user of first GUI 212 can allocate annotators efficiently, such as by ceasing to request annotations for a particular label or task or collection of documents if it is apparent that additional annotations will not appreciably affect the agreement. Or, in instances with a high degree of variability with additional annotations a project manager or user of first GUI 212 could assign additional annotators to provide more human oversight to attempt to reach a consistent agreement level.
- method 1300 continues to 1316 by creating an annotation feedback pane.
- An annotation feedback pane can be configured to display a plurality of data.
- an annotation feedback pane displays an annotation agreement score.
- An annotation agreement score displays the overall agreement between annotators of the whole collection of documents within the ontology being analyzed, and can indicate the general disposition or accuracy of the entire ontology and whether further by-label or by-task analysis or manipulation is warranted given the overall agreement score or whether annotators should retrained on the definition of particular categories.
- an annotation feedback pane displays at 1316 an individual annotator agreement list with the agreement scores and number of annotations per individual annotator within the collection.
- Such an individual annotator agreement list indicates whether certain annotators have processed all documents presented them on work unit interfaces of their respective GUIs, or whether certain annotators appear to have trouble with the subject matter is indicated by low agreement scores relative to other annotators. For example, if an overall agreement, such as one displayed in an annotation agreement score, were 0.7 with individual annotator agreement list displaying individual agreements of 0.75, 0.55, 0.7, and 0.77, a project manager could determine that the individual annotator with the 0.55 score should be dropped from the annotation agreement score.
- a suggested label collapse list is created.
- a suggested label collapse list enables a project manager or user of a first GUI to quickly identify the strengths and weaknesses of an ontology and a natural language modeling engine's ability to sort documents into a particular label or task based on the document's content.
- a suggested label collapse list is created by pairing annotated labels or tasks together and displaying how the annotation agreement score would be affected if the two labels or tasks were combined into a common label or task; in other words, how an annotation agreement score would be affected if annotators were not required to distinguish between certain labels or tasks.
- Such a feature can indicate whether the natural language modeling engine selected appropriate labels or tasks, or whether guidelines paired with labels or tasks are sufficiently describing the label or task.
- an agreement per label graphical representation is created by displaying a bar graph of agreement among annotators for one or more given labels or tasks of the ontology being annotated.
- a collapsed agreement per label graphical representation is created by determining the agreement per label or task if certain labels of tasks were collapsed into one another. Such a feature further indicates which labels or tasks within an ontology are more likely to be correctly placed on an ontology by creating an interface for side by side comparison of agreements if labels or tasks were asked to be distinguished from one another or if they were combined.
- a per document agreement list is created by identifying those documents with the highest and lowest annotation agreements.
- the number of documents displayed in a per document agreement list is determined from an input by a project manager or user of first GUI 212 .
- a per document agreement list allows a project manager or user of a first GUI 212 to determine which documents gave annotators the most difficulty in agreeing on a common label or task, as well as showing which documents have unanimous agreement and make the best exemplars of a category.
- a project manager or user of first GUI can then review certain documents to determine whether anything particular is giving trouble to annotators, or even remove the document from the ontology for its lack of a clear disposition.
- method 1300 continues at 1318 to create a rules pane for receiving a phrase in a phrase pane, a weighting adjustment in a weighting adjustment pane, or a rule in an add rule pane.
- a phrase pane is configured to identify a certain phrase within a document.
- a weighting adjustment pane is configured to reduce the relevance of a phrase in the phrase pane relative to other words or phrases in the document such that a natural language modeling engine will emphasize or deemphasize certain phrases.
- an add rule pane gives direction for a natural language modeling engine to perform on a phrase in a phrase pane.
- a rule in the add rule pane can dictate how to process that document, such as by placing it in a specific label or task of an ontology, or looking for additional words or phrases before placing the document within an ontology.
- method 1300 continues at 1320 by computing initial annotation agreements from aggregated annotations such as at 1260 as depicted in FIG. 12 .
- computing initial agreements is performed by an annotations module, such as one of natural language modeling engine 210 depicted in FIG. 2 .
- Computing initial annotation agreements includes not only determining annotation agreement scores among annotators for individual labels or tasks and overall score for an ontology, but also includes in some embodiments computing a learning curve relationship between annotation agreement and number of annotations.
- computing initial annotation agreements determines annotation agreement scores if two or more labels are collapsed into one another.
- computing initial annotation agreements computes per label annotation agreements and per label annotation agreements if two or more labels are collapsed into one another and graphical relationships of each.
- computing initial annotation agreements at 1320 computes the annotation agreement per each document classified according to an ontology, and identifies the documents with the highest and lowest annotation agreements.
- the computed metrics of 1320 are populated into an annotation agreement interface at 1330 .
- a label feedback pane is populated with each label of an ontology.
- the label feedback pane is populated with a description of the label as provided by a natural language modeling engine or project manager of first GUI 212 or expert annotator operating a second GUI 214 .
- the label feedback pane is populated with the number of annotations a label or task has received from annotators operating second GUI 214 or third GUI 216 .
- the learning curve pane is populated with the computed learning curve metric of the annotation agreement related to the number of annotations, as computed at 1320 .
- the annotation feedback pane is populated with the annotation agreement score for the ontology.
- the annotation feedback pane of the annotation agreement interface is populated with a suggested label collapse list for the resultant annotation agreement scores corresponding to how the annotation agreement score will change if two or more labels or tasks are collapsed into one another.
- the annotation feedback pane of the annotation agreement interface is populated with the per label or task annotation agreements and collapsed agreement per label or task as computed at 1320 .
- a per document agreement is populated in an annotation feedback pane.
- the panes of the annotation interface populated at 1330 are displayed to a project manager or user of first GUI 212 .
- method 1300 enables follow on actions to manipulate or refine the documents and annotators to improve the ontology, such as by enabling improved annotations on other work unit interfaces, or removing documents from the ontology.
- method 1400 is a method for interacting with information displayed to a project manager or user of first GUI 212 on an annotation agreement interface.
- method 1400 begins at 1410 by receiving a request to collapse at least one first label or first task into at least one second label or second task.
- a request to collapse in some embodiments is made, in some embodiments, through the agreement per label graphical representation of the annotation feedback pane.
- the request to collapse at 1410 is made through the suggested label collapse list.
- One of skill in the art can envision numerous ways to select which labels or tasks to collapse into one another through one of the information displays of an annotation agreement interface.
- a subsequent annotation agreement is displayed in the annotation feedback pane of the annotation agreement interface after collapsing the labels or tasks in 1410 .
- Such a subsequent annotation agreement can be used as a visual comparison of the annotation agreements before and after collapsing labels or tasks.
- FIG. 15 illustrates method 1500 for changes to a work unit interface a project manager or user of first GUI 212 can make in response to information populated on an annotation agreement interface such as by method 1300 described in FIG. 13 .
- method 1500 is directed to replacing a guideline for a label in a work unit interface.
- a project manager or user of first GUI 212 decides to replace a guideline to a label or task in a work unit interface, such as by realizing a per label annotation agreement computed at 1320 in method 1300 is lower relative to other per label annotation agreements.
- an expert annotator decides to replace a guideline to a label or task after viewing a paired guideline in a work unit interface from a second GUI and enters a new guideline in the create guideline data field presented on a second GUI such as the create guideline pane 845 as depicted in FIG. 8A .
- method 1500 begins at 1510 by accessing a revised guideline through a first GUI 212 or second GUI 214 .
- the single guideline initially paired with the label or task of the work unit interface (such as at 1235 in FIG. 12 ) is unpaired with the respective label or task.
- the revised guideline accessed at 1510 is then paired with the label or task that had the single guideline unpaired at 1520 .
- a second reference button is created for the newly paired revised guideline with the label or task, and at 1550 the second reference button is displayed on the work unit interface.
- the revised guideline is included in addition to the single guideline, such that when an annotator interacting with the work unit interface presses the reference button, both the single guideline and the revised guideline are displayed to the annotator.
- FIGS. 12-15 provide a particular process and sequence of interaction among GUIs and the manipulation of the annotations generated on an ontology. Other sequences of steps may also be performed according to alternative embodiments. For example, alternative embodiments may perform the steps outlined above in a different order. Moreover, the individual sequence illustrated in FIGS. 12-15 may include multiple sub-sequences as appropriate to the individual step or direct sequences between different nodes than as illustrated. Furthermore, additional steps may be added or removed depending on the particular applications. One of skill in the art would recognize many variations, modifications, and alternatives.
- the block diagram illustrates components of a machine 1600 , according to some example embodiments, able to read instructions 1624 from a machine-readable medium 1622 (e.g., a non-transitory machine-readable medium, a machine-readable storage medium, a computer-readable storage medium, or any suitable combination thereof) and perform any one or more of the methodologies discussed herein, in whole or in part.
- a machine-readable medium 1622 e.g., a non-transitory machine-readable medium, a machine-readable storage medium, a computer-readable storage medium, or any suitable combination thereof
- 16 shows the machine 1600 in the example form of a computer system (e.g., a computer) within which the instructions 1624 (e.g., software, a program, an application, an applet, an app, or other executable code) for causing the machine 1600 to perform any one or more of the methodologies discussed herein may be executed, in whole or in part.
- the instructions 1624 e.g., software, a program, an application, an applet, an app, or other executable code
- the machine 1600 operates as a standalone device or may be connected (e.g., networked) to other machines.
- the machine 1600 may operate in the capacity of a server machine 110 or a client machine in a server-client network environment, or as a peer machine in a distributed (e.g., peer-to-peer) network environment.
- the machine 1600 may include hardware, software, or combinations thereof, and may, as example, be a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a cellular telephone, a smartphone, a set-top box (STB), a personal digital assistant (PDA), a web appliance, a network router, a network switch, a network bridge, or any machine capable of executing the instructions 1624 , sequentially or otherwise, that specify actions to be taken by that machine.
- PC personal computer
- PDA personal digital assistant
- the machine 1600 includes a processor 1602 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a radio-frequency integrated circuit (RFIC), or any suitable combination thereof), a main memory 1604 , and a static memory 1606 , which are configured to communicate with each other via a bus 1608 .
- the processor 1602 may contain microcircuits that are configurable, temporarily or permanently, by some or all of the instructions 1624 such that the processor 1602 is configurable to perform any one or more of the methodologies described herein, in whole or in part.
- a set of one or more microcircuits of the processor 1602 may be configurable to execute one or more modules (e.g., software modules) described herein.
- the machine 1600 may further include an input and output module 1610 (e.g., a plasma display panel (PDP), a light emitting diode (LED) display, a liquid crystal display (LCD), a projector, a cathode ray tube (CRT), or any other display capable of displaying graphics or video) configured to display any one of the interfaces described herein.
- an input and output module 1610 e.g., a plasma display panel (PDP), a light emitting diode (LED) display, a liquid crystal display (LCD), a projector, a cathode ray tube (CRT), or any other display capable of displaying graphics or video
- the machine 1600 may also include an alphanumeric input device 1612 (e.g., a keyboard or keypad), a cursor control device 1614 (e.g., a mouse, a touchpad, a trackball, a joystick, a motion sensor, an eye tracking device, or other pointing instrument), a storage unit 1616 , a signal generation device 1618 (e.g., a sound card, an amplifier, a speaker, a headphone jack, or any suitable combination thereof), and a network interface device 1620 .
- an alphanumeric input device 1612 e.g., a keyboard or keypad
- a cursor control device 1614 e.g., a mouse, a touchpad, a trackball, a joystick, a motion sensor, an eye tracking device, or other pointing instrument
- a storage unit 1616 e.g., a storage unit 1616 , a signal generation device 1618 (e.g., a sound card, an amplifier, a speaker,
- the storage unit 1616 includes the machine-readable medium 1622 (e.g., a tangible and non-transitory machine-readable storage medium) on which are stored the instructions 1624 embodying any one or more of the methodologies, functions, or interfaces described herein, including, for example, any of the descriptions of FIGS. 1-15 .
- the instructions 1624 may also reside, completely or at least partially, within the main memory 1604 , within the processor 1602 (e.g., within the processor's cache memory), or both, before or during execution thereof by the machine 1600 .
- the instructions 1624 may also reside in the static memory 1606 .
- the main memory 1604 and the processor 1602 may be considered machine-readable media 1622 (e.g., tangible and non-transitory machine-readable media).
- the instructions 1624 may be transmitted or received over a network 1626 via the network interface device 1620 .
- the network interface device 1620 may communicate the instructions 1624 using any one or more transfer protocols (e.g., HTTP).
- the machine 1600 may also represent example means for performing any of the functions described herein, including the processes described in FIGS. 1-15 .
- the machine 1600 may be a portable computing device, such as a smart phone or tablet computer, and have one or more additional input components (e.g., sensors or gauges) (not shown).
- additional input components e.g., sensors or gauges
- input components include an image input component (e.g., one or more cameras), an audio input component (e.g., a microphone), a direction input component (e.g., a compass), a location input component (e.g., a GPS receiver), an orientation component (e.g., a gyroscope), a motion detection component (e.g., one or more accelerometers), an altitude detection component (e.g., an altimeter), and a gas detection component (e.g., a gas sensor).
- Inputs harvested by any one or more of these input components may be accessible and available for use by any of the modules described herein.
- the term “memory” refers to a machine-readable medium 1622 able to store data temporarily or permanently and may be taken to include, but not be limited to, random-access memory (RAM), read-only memory (ROM), buffer memory, flash memory, and cache memory. While the machine-readable medium 1622 is shown in an example embodiment to be a single medium, the term “machine-readable medium” should be taken to include a single medium or multiple media (e.g., a centralized or distributed database 115 , or associated caches and servers) able to store instructions 1624 .
- machine-readable medium shall also be taken to include any medium, or combination of multiple media, that is capable of storing the instructions 1624 for execution by the machine 1600 , such that the instructions 1624 , when executed by one or more processors of the machine 1600 (e.g., processor 1602 ), cause the machine 1600 to perform any one or more of the methodologies described herein, in whole or in part.
- a “machine-readable medium” refers to a single storage apparatus or device 130 or 150 , as well as cloud-based storage systems or storage networks that include multiple storage apparatus or devices 130 or 150 .
- machine-readable medium shall accordingly be taken to include, but not be limited to, one or more tangible (e.g., non-transitory) data repositories in the form of a solid-state memory, an optical medium, a magnetic medium, or any suitable combination thereof.
- the machine-readable medium 1622 is non-transitory in that it does not embody a propagating signal. However, labeling the tangible machine-readable medium 1622 as “non-transitory” should not be construed to mean that the medium is incapable of movement; the medium should be considered as being transportable from one physical location to another. Additionally, since the machine-readable medium 1622 is tangible, the medium may be considered to be a machine-readable device.
- Modules may constitute software modules (e.g., code stored or otherwise embodied on a machine-readable medium 1622 or in a transmission medium), hardware modules, or any suitable combination thereof.
- a “hardware module” is a tangible (e.g., non-transitory) unit capable of performing certain operations and may be configured or arranged in a certain physical manner.
- one or more computer systems e.g., a standalone computer system, a client computer system, or a server computer system
- one or more hardware modules of a computer system e.g., a processor 1602 or a group of processors 1602
- software e.g., an application or application portion
- a hardware module may be implemented mechanically, electronically, or any suitable combination thereof.
- a hardware module may include dedicated circuitry or logic that is permanently configured to perform certain operations.
- a hardware module may be a special-purpose processor, such as a field programmable gate array (FPGA) or an ASIC.
- a hardware module may also include programmable logic or circuitry that is temporarily configured by software to perform certain operations.
- a hardware module may include software encompassed within a general-purpose processor 1602 or other programmable processor 1602 . It will be appreciated that the decision to implement a hardware module mechanically, in dedicated and permanently configured circuitry, or in temporarily configured circuitry (e.g., configured by software) may be driven by cost and time considerations.
- Hardware modules can provide information to, and receive information from, other hardware modules. Accordingly, the described hardware modules may be regarded as being communicatively coupled. Where multiple hardware modules exist contemporaneously, communications may be achieved through signal transmission (e.g., over appropriate circuits and buses 1608 ) between or among two or more of the hardware modules. In embodiments in which multiple hardware modules are configured or instantiated at different times, communications between such hardware modules may be achieved, for example, through the storage and retrieval of information in memory structures to which the multiple hardware modules have access. For example, one hardware module may perform an operation and store the output of that operation in a memory device to which it is communicatively coupled. A further hardware module may then, at a later time, access the memory device to retrieve and process the stored output. Hardware modules may also initiate communications with input or output devices, and can operate on a resource (e.g., a collection of information).
- a resource e.g., a collection of information
- processors 1602 may be temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, such processors 1602 may constitute processor-implemented modules that operate to perform one or more operations or functions described herein. As used herein, “processor-implemented module” refers to a hardware module implemented using one or more processors 1602 .
- processors 1602 being an example of hardware.
- processor-implemented module refers to a hardware module in which the hardware includes one or more processors 1602 .
- processors 1602 may also operate to support performance of the relevant operations in a “cloud computing” environment or as a “software as a service” (SaaS).
- At least some of the operations may be performed by a group of computers (as examples of machines 1600 including processors 1602 ), with these operations being accessible via a network 1626 (e.g., the Internet) and via one or more appropriate interfaces (e.g., an application program interface or “API”).
- a network 1626 e.g., the Internet
- API application program interface
- the performance of certain operations may be distributed among the one or more processors 1602 , not only residing within a single machine 1600 , but deployed across a number of machines 1600 .
- the one or more processors 1602 or processor-implemented modules may be located in a single geographic location (e.g., within a home environment, an office environment, or a server farm). In other example embodiments, the one or more processors 1602 or processor-implemented modules may be distributed across a number of geographic locations.
- a natural language modeling engine 210 e.g., a on a computing device or external server such as server machine 110 depicted in FIG. 1 or as part of a system of interconnected interfaces as depicted in FIG. 2
- a natural language modeling engine 210 manipulates or transforms data represented as physical (e.g., electronic, magnetic, or optical) quantities within one or more memories (e.g., volatile memory, non-volatile memory, or any suitable combination thereof), registers, or other machine components that receive, store, transmit, or display information.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Computational Linguistics (AREA)
- Artificial Intelligence (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Mathematical Physics (AREA)
- Business, Economics & Management (AREA)
- General Business, Economics & Management (AREA)
- Human Computer Interaction (AREA)
- Computing Systems (AREA)
- Software Systems (AREA)
- Tourism & Hospitality (AREA)
- Human Resources & Organizations (AREA)
- Marketing (AREA)
- Primary Health Care (AREA)
- Strategic Management (AREA)
- Economics (AREA)
- Animal Behavior & Ethology (AREA)
- Life Sciences & Earth Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Computation (AREA)
- Medical Informatics (AREA)
- Multimedia (AREA)
- Machine Translation (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
Description
- This application claims the benefits of U.S. Provisional Application 62/089,736, filed Dec. 9, 2014, and titled, “METHODS AND SYSTEMS FOR ANNOTATING NATURAL LANGUAGE PROCESSING,” U.S. Provisional Application 62/089,742, filed Dec. 9, 2014, and titled, “METHODS AND SYSTEMS FOR IMPROVING MACHINE PERFORMANCE IN NATURAL LANGUAGE PROCESSING,” U.S. Provisional Application 62/089,745, filed Dec. 9, 2014, and titled, “METHODS AND SYSTEMS FOR IMPROVING FUNCTIONALITY IN NATURAL LANGUAGE PROCESSING,” and U.S. Provisional Application 62/089,747, filed Dec. 9, 2014, and titled, “METHODS AND SYSTEMS FOR SUPPORTING NATURAL LANGUAGE PROCESSING,” the disclosures of which are incorporated herein by reference in their entireties and for all purposes.
- This application is also related to U.S. non provisional applications (Attorney Docket No. 1402805.00006_IDB006), titled “METHODS FOR GENERATING NATURAL LANGUAGE PROCESSING MODELS,” (Attorney Docket No. 1402805.00007_IDB007), titled “ARCHITECTURES FOR NATURAL LANGUAGE PROCESSING,” (Attorney Docket No. 1402805.00012_IDB012), titled “OPTIMIZATION TECHNIQUES FOR ARTIFICIAL INTELLIGENCE,” (Attorney Docket No. 1402805.00014_IDB014), titled “METHODS AND SYSTEMS FOR IMPROVING MACHINE LEARNING PERFORMANCE,” (Attorney Docket No. 1402805.000015_IDB015), titled “METHODS AND SYSTEMS FOR MODELING COMPLEX TAXONOMIES WITH NATURAL LANGUAGE UNDERSTANDING,” (Attorney Docket No. 1402805.00016_IDB016), titled “AN INTELLIGENT SYSTEM THAT DYNAMICALLY IMPROVES ITS KNOWLEDGE AND CODE-BASE FOR NATURAL LANGUAGE UNDERSTANDING,” (Attorney Docket No. 1402805.00017_IDB017), titled “ METHODS AND SYSTEMS FOR LANGUAGE-AGNOSTIC MACHINE LEARNING IN NATURAL LANGUAGE PROCESSING USING FEATURE EXTRACTION,” (Attorney Docket No. 1402805.00018_IDB018), titled “METHODS AND SYSTEMS FOR PROVIDING UNIVERSAL PORTABILITY IN MACHINE LEARNING,” and (Attorney Docket No. 1402805.00019_IDB019), titled “TECHNIQUES FOR COMBINING HUMAN AND MACHINE LEARNING IN NATURAL LANGUAGE PROCESSING,” each of which are filed concurrently herewith, and the entire contents and substance of all of which are hereby incorporated in total by reference in their entireties and for all purposes.
- The subject matter disclosed herein generally relates to creating one or more interfaces for processing human verification of natural language model accuracies. In some embodiments, a natural language modeling engine displays certain information on interfaces to confirm categorization, classification, or sorting the natural language modeling engine has performed for a set of documents into a scheme comprising various labels that may be present in the documents. In some embodiments, the interfaces are dynamically linked across multiple interfaces, such that interaction or verifications on one interface may adjust the information displayed on a linked interface.
- Human communications in the digital age provide a deluge of information. Compounding the sheer volume of human communications in this technological era is the multitude of formats such human communications may come in, such as public news articles, social media posts, emails, customer feedback comments on a vendor website, or information circulated within a closed environment. It is difficult and time consuming for a human to integrate or recognize trends within these various communication formats with the volume of content involved in each, and any broad appreciation for what the creators of such communications may be expressing is therefore delayed. Recognizing trends or underlying meanings across the vast number of human communication sources, and appropriately categorizing unique nomenclature or slang that may be embedded, cannot be efficiently obtained without the aid of computer modeling tools.
- Artificial intelligence tools can attempt to analyze and classify these information sources through metadata or other identifiers, but cannot efficiently analyze the true meaning of the internal content without verification oversight that is time consuming and costly to implement. Efficient verification of natural language modeling of a collection of documents is demanded to ensure such computer modeling tools are accurately recognizing appropriate categories of human communications within a collection of documents.
- Graphic interface systems and methods for enabling human verification of natural language modeling by computer analysis tools, such as a natural language modeling engine, are described. In some embodiments, the verification is achieved by presenting a document with a list of potential labels or tasks describing the document on a work unit interface and aggregating the responsive inputs to the work unit interface (hereinafter, such responsive inputs are referred to as “annotations”). An annotation is not necessarily a confirmation of a natural language modeling engine's prediction of a label or task of a document. An annotation to a document or subset of a document (referred to as a “span”) generally includes information indicating how the document or span should be classified into one or more topics or categories. In some embodiments, an annotation is a corrective departure from the natural language modeling engine's prediction. Aggregated annotations can, in some embodiments, be displayed on related interfaces and further manipulated to determine the accuracy of a natural language modeling engine in categorizing the document into a hierarchical structure of labels and/or “tasks” associated with the document.
- A “task,” in some embodiments, a can be a clarification, reflection, sentiment, or other objective surrounding a document or label, such as, merely by way of example, “positive” or “negative” or various degrees between. “Tasks” may also refer to genres or groupings within labels and not merely binary interpretations of a label. In this disclosure, the hierarchical structure may also be thought of as a categorization, or classification, and is hereinafter referred to as an “ontology.”
- In annotating a document, the work unit interface may, in some embodiments, highlight or otherwise visually distinguish only a portion of the document (such as by underlining or italicizing), such portion of a document is referred to as a “span.”
- In some embodiments, a collection of documents is accessed through a natural language modeling engine. The natural language modeling engine can organize the documents into an ontology by analyzing, grouping, and classifying them based on the words within the documents, and the natural language modeling engine's logic processing, or use of past keywords for classification.
- In some embodiments, the ontology is displayed on a first graphic user interface (GUI). The first GUI can include an option for the user of the first GUI, such as a project manager analyzing the documents, to send selected documents to a series of annotators to confirm the accuracy of the label or task for the particular document within the ontology as initially determined by the natural language modeling engine.
- In some embodiments, to facilitate annotation, the document is presented as part of a work unit interface displaying relevant panes for efficiently verifying the accuracy of the natural language modeling engine's ontology. In some embodiments, a work unit interface is constructed by integrating a document, a label or task, a guideline describing the label/task or distinguishing it from other labels/tasks, and a human readable prompt soliciting a response from the annotator. In some embodiments, the document is displayed in a document pane of the work unit interface, and a series of eligible labels or tasks are displayed in a label pane adjacent to the document pane. According to various embodiments, there may be one or more labels or tasks presented in the label pane of the work unit interface. In some embodiments, the guideline describing the label or task is displayed on the work unit interface as a reference button adjacent to its respective label or task and when activated by a user of the work unit interface (such as by “clicking,” or “pressing” or hovering a cursor over the reference button) the reference button opens a guideline pane displaying the label or task description.
- In some embodiments, the human readable prompt is displayed in a prompt pane of the work unit interface. In some embodiments, the human readable prompt is generated from an intelligent queuing module of a natural language modeling engine. Depending on the embodiment, the human readable prompt requests confirmation of a label or task of the document displayed in the document pane as predicted by the natural language modeling engine, or requests the user of the work unit interface to select the most applicable label or task for the document from among a plurality of displayed labels or tasks in the label pane, and in still other embodiments requests the user identify all labels or tasks that are related to the document. One of skill in the art will appreciate many variations on the human readable prompt.
- In some embodiments, the work unit interface is presented on a second GUI displayed to an expert annotator. The second GUI, when displaying a work unit interface, may include a create pane configured to receive an additional label or task, or revised guideline for a respective label or task within the label pane of the work unit interface. When a create label/task or create guideline input is received from an expert annotator on the second GUI, the second GUI may update the respective work unit interface on one or more second or third GUIs also configured to display the particular work unit interface. Third GUIs, in some embodiments, display the work unit interface to annotators that are not expert annotators, and such work unit interfaces do not include a create pane.
- In some embodiments, updates from second GUIs may change what is displayed on work unit interfaces displayed on other second or third GUIs, and also affect which work unit interfaces are displayed to other expert annotators on a second GUI or to other annotators on a third GUI. In some embodiments, updates to the work unit interface may be the replacement of a label or task with a new label or task entered into a create pane. In some embodiments, updates to the work unit interface may be a new human readable prompt for selection of a label or task (for example, an expert annotator could direct a work unit interface to prompt “choose a label” rather than “confirm if the presented label is applicable”). In some embodiments, updates to the work unit interface may be to replace an existing guideline with a revised guideline entered into a create pane. In some embodiments, updates to the work unit interface may be supplement the given label(s) or task(s) with the label or task entered into a create pane. In some embodiments, updates to the work unit interface may be to supplement a given guideline to a label or task with a revised guideline entered into a create pane.
- In some embodiments, the third GUI operated by an annotator, or second GUI operated by an expert annotator, receives an action on the work unit interface responsive to the human readable prompt. In some embodiments, the received action is an annotation of a label or a task of the document as requested by the human readable prompt. In some embodiments, the annotation is aggregated with all annotations to that particular work unit interface displayed received all third GUIs and second GUIs that were presented the work unit interface.
- In some embodiments, the aggregated annotations are collected by the natural language modeling engine and displayed on the first GUI for further analysis and interaction by, for example, a project manager for the collection of documents, though one of skill in the art will envision other users or suitable roles for operating first GUI to interact with an aggregation of annotations.
- In some embodiments, the aggregated annotations are displayed on the first GUI in an annotation agreement interface comprising a series of information panes. In some embodiments, a label feedback pane of the annotation agreement interface displays a plurality of label panes for each label or task within an ontology and displays the number of annotations the respective label or task received, and options to delete or edit the label or task. In some embodiments, the annotation agreement interface includes a learning curve pane, such learning curve displays a graphical representation of the number of annotations received for a particular label or task and the agreement among those annotations for the accuracy of that particular label or task. In some embodiments, the annotation agreement interface includes an annotation feedback pane. The annotation feedback pane, depending on embodiment, may display an aggregate annotation agreement score representing the overall accuracy of the ontology as determined by the annotation agreements across all labels or tasks of the ontology. In some embodiments, the annotation feedback pane includes an individual annotator agreement list displaying the agreement scores of each individual annotator relative to the other annotators.
- In some embodiments, the annotation feedback pane includes a suggestion pane for collapsing labels or tasks into one another. By collapsing individual labels or tasks into groups of labels or tasks, the annotation agreement interface can reduce disagreements between annotators of a particular label or task, or reduce confusion annotators may have over minute differences between labels or tasks. For example, if the work unit interface displays labels of “securities” and “stock” for a particular document, an annotator may have trouble distinguishing the two and the respective labels will have a low annotation agreement score, however, if the two labels were recalculated as a common label, the annotation agreement score may improve. Such collapsed label or task calculations may indicate, to a project manager or user of a first GUI, the need to refine the labels or tasks within the ontology, or the need to refine the guidelines describing the labels or tasks.
- In some embodiments, the annotation agreement interface includes an agreement per label or task graphical representation, such as a bar chart displaying the agreement per label or task relative to the other labels or tasks in the ontology that received annotations. In some embodiments, the agreement per label or task graphical representation further includes a collapsed agreement per label or task graphical representation displaying the agreement per label or task if two or more labels or tasks were collapsed into one another.
- In some embodiments, the annotation agreement interface includes a per document agreement list displaying information such as which documents received the highest agreement or which documents received the lowest agreement. A user of the first GUI could potentially remove documents with lower annotation agreement from the ontology to improve the accuracy of the model much in the same way other analysis methods remove ambiguous information or “noise” from those data sets.
- The aggregated annotations, either in their original annotation form, or as manipulated through the first GUI by the various pane described above, can then be used to display verified analysis of the natural language process engine ontology and indicate trends and sentiments contained within the collection of documents for that ontology with a certain degree of reliance (the annotation agreement score most readily serving as a proxy for reliance of the ontology). For example, an ontology comprising a collection of thousands of “tweets” from a Twitter hashtag of # Tesla could divide the tweets into labels based on common words across the tweets, such as “battery,” “autonomous,” and “Elon Musk” with tasks related to each label such as “positive” or “negative” and display the number of tweets that fall within each label and task and the number of annotations to each tweet and the annotation agreement amongst annotator to give a fast overview of the general disposition of the tweets within the ontology.
- These and other embodiments of the present disclosure along with many of their advantages and features are described in more detail in conjunction with the text below and attached figures.
- Some embodiments are illustrated by way of example and not limitation in the figures of the accompanying drawings.
-
FIG. 1 is a network diagram illustrating an example network environment suitable for performing aspects of the present disclosure, according to some embodiments. -
FIG. 2 is an illustration of a system diagram showing of a first, second, and third graphic user interface (GUI) operably coupled to a natural language modeling engine, according to some embodiments. -
FIG. 3 illustrates a sample interface display of a first GUI accessing a collection of documents, according to some embodiments. -
FIG. 4 illustrates a sample interface display of a first GUI presenting a landing page for a user of a first GUI to access features and maintenance, information of a collection of documents to a user, and annotation assignments according to some embodiments. -
FIG. 5 illustrates a sample interface display of a first GUI for accessing a plurality of documents to create a collection of documents, according to some embodiments. -
FIG. 6 illustrates a sample interface display of a first GUI for reviewing automatic topic modeling incident to an ontology of a collection of documents based on keywords within the collection, according to some embodiments. -
FIG. 7A illustrates a sample interface display of a first GUI for creating an ontology of a collection of documents based on keywords entered into the interface, according to some embodiments. -
FIG. 7B illustrates a sample interface display of a first GUI for visually presenting topic relevance and relationship to other topics, according to some embodiments. -
FIG. 8A illustrates a sample work unit interface arrangement of panes for collecting annotations from an expert annotator, according to some embodiments. -
FIG. 8B illustrates a sample work unit interface arrangement of panes for collecting annotations from an annotator, according to some embodiments. -
FIG. 8C illustrates a sample work unit interface arrangement of panes for displaying guideline information of a label or task, according to some embodiments. -
FIGS. 8D-E illustrates a sample work unit interface arrangement of panes for displaying annotating spans and selecting spans, according to some embodiments. -
FIG. 9A illustrates a sample interface display of a label feedback pane within a first GUI for displaying characteristics of a label within an ontology, according to some embodiments. -
FIG. 9B illustrates a sample interface display of a label feedback pane within a first GUI for displaying a learning curve of the relationship between annotation agreement as a function of the number of annotations received, according to some embodiments. -
FIG. 10A illustrates a sample interface display of an annotation feedback pane within a first GUI for displaying annotation information relative to labels or tasks and annotators, according to some embodiments. -
FIG. 10B illustrates a sample interface display of an annotation feedback pane within a first GUI for displaying graphical representations of annotation agreements relative to collapsing labels into one another, according to some embodiments. -
FIG. 10C illustrates a sample interface display of a label feedback pane within a first GUI for displaying annotation agreements relative to specific documents within the collection of documents. -
FIG. 11 illustrates a sample interface display of a rules feedback pane within a first GUI for displaying adjustment options to modeling and interpreting certain labels or tasks, according to some embodiments. -
FIG. 12 illustrates an example method for creating and integrating an interface for collecting and aggregating annotations across GUIs, according to some example embodiments. -
FIG. 13 illustrates an example method for building an annotation agreement interface for analyzing annotations of labels or tasks within an ontology, according to some embodiments. -
FIG. 14 illustrates an example method of an interface for collapsing labels or tasks into one another and displaying comparative annotation agreements, according to some embodiments. -
FIG. 15 illustrates an example method of updating a work unit interface with revised guidelines, according to some example embodiments. -
FIG. 16 illustrates a block diagram illustrating components of a machine, according to some example embodiments, able to read instructions from a machine-readable medium and perform any one or more of the methodologies discussed herein. - The following detailed description should be read with reference to the drawings when appropriate, in which identical reference numbers refer to like elements throughout the different figures. The drawings, which are not necessarily to scale, depict selective embodiments and are not intended to limit the scope of the invention. The detailed description illustrates by way of example, not by way of limitation, the principles of the invention. This description will clearly enable one skilled in the art to make and use the invention, and describes several embodiments, adaptations, variations, alternatives and uses of the invention, including what is presently believed to be the best mode of carrying out the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” include plural referents unless the context clearly indicates otherwise.
- Examples merely demonstrate possible variations. Unless explicitly stated otherwise, components and functions are optional and may be combined or subdivided, and operations may vary in sequence or be combined or subdivided. In the following description, for purposes of explanation, numerous specific details are set forth to provide a thorough understanding of example embodiments. It will be evident to one skilled in the art, however, that the present subject matter may be practiced without these specific details.
- Systems, methods, and apparatuses are presented for causing the display of certain information and efficiently sharing responsive inputs to verify the accuracy of computer modeling in natural language processing. Marketing outlets, internal compliance departments, customer service departments, or even design and engineering industries can greatly benefit from rapid analysis of communications about products and policies. In the digital age, where communication can take many forms and the volume of information within those forms is staggering, analysis of these documents by human review can only be performed in a timely fashion by choosing a small sample of the communication. Such small sampling introduces error as to any conclusions, by casting doubt to how representative a sample is of the whole.
- Computing modeling, by contrast, can analyze an entire corpus of documents to rapidly identify the general trends and commentary across all documents by using artificial intelligence to recognize keywords, syntax, and relation within documents in a much more timely fashion. By categorizing a collection of documents into topics, with a series of descriptive labels and tasks describing each document, a natural language modeling engine can build an ontology of documents demonstrative of overall sentiments and underlying meaning of trends across all documents.
- To ensure accuracy of the ontology, and confirm the artificial intelligence is appropriately categorizing documents, a series of interfaces are described for presenting and receiving human annotations of a natural language modeling engine's classification.
-
FIG. 1 illustrates a network for relating users across interfaces for annotating documents that may be classified according to an ontology created by a natural language modeling engine.Network 100 includesuser 132operating device 130,user 152operating device 150, and network basedsystem 105.User 132 oruser 152 may be one of a project manager, expert annotator, or annotator.Device 130 ordevice 150 may be a mobile device, desktop computer, tablet, or other computing device configured to operate any one of the interfaces described herein.Connecting user 132 oruser 152 to network basedsystem 105 throughdevice 130 ordevice 150 isnetwork 190.Network 190 may be a wireless network (such as wide area network, local area network), ethernet connections or other wired connection, or other suitable network system for linking computing devices. - Network based
system 105 can include aserver machine 110 configured to perform natural language modeling according to some embodiments as further described in this detailed description, and adatabase 115.Database 115 may store a collection of documents forserver machine 110 to access and create an ontology around, or may store artificial intelligence rules forserver machine 110 to access and apply to sorting a collection of documents, or may store guidelines and labels or tasks that have been used in other ontologies thatserver machine 110 can access to build additional ontologies. For example, if a previous ontology had been built for “JP Morgan Chase” and documents related to their investment banking practice,database 115 can store the labels and tasks and guidelines used in that ontology to informserver machine 110 which labels and tasks and guideline may be relevant in a subsequent “JP Morgan Chase” ontology relating to, for example purposes only, “customer service.” - In some embodiments,
user system 105 throughdevice network 190. - In some embodiments, network based
system 105 creates an ontology of a collection of documents, and displays a work unit interface on a second or third GUI to verify the accuracy of the ontology touser 132 oruser 152 throughdevice User system 105 aggregates the annotations to refine the ontology or draw further conclusions about the underlying documents and displays the results on a first GUI, which may beuser 132 oruser 152 depending on the embodiment. -
FIG. 2 illustrates a diagram ofinterface system 200 relating a first, second, and third GUI with a naturallanguage modeling engine 210. In some embodiments, naturallanguage modeling engine 210 is network basedsystem 105 as described inFIG. 1 . In some embodiments, naturallanguage modeling engine 210 comprises a series of modules to include a database, which may be the same database asdatabase 115 as described inFIG. 1 ; and an input/output (I/O) module configured to receive and transmit information throughoutinterface system 200. In some embodiments, naturallanguage modeling engine 210 further comprises an API module configured to communicate through an I/O module with various devices, GUIs, and operating systems interacting with naturallanguage modeling engine 210. In some embodiments, naturallanguage modeling engine 210 further comprises an intelligent queuing module configured to generate human readable prompts for populating a work unit interface to elicit an annotation from expert annotators on a second GUI or annotators on a third GUI. In some embodiments, naturallanguage modeling engine 210 further comprises an annotation module for constructing a work unit interface with at least a document from a database, a human readable prompt from an intelligent queuing module, and a label or task from a database. In some embodiments, an annotation module is further configured to aggregate received annotations from across a plurality of work unit interfaces and compute agreements and relationships between the aggregated annotations. In some embodiments, naturallanguage modeling engine 210 further comprises a modeling module for constructing an ontology from a collection of documents on a database. In some embodiments, a modeling module is configured to modify an ontology in response to aggregated annotations received from a plurality of work unit interfaces acrosssecond GUIs 214 orthird GUIs 216, or the manipulations to an aggregated set of annotations as received from a user of afirst GUI 212. - In some embodiments, operably coupled to natural
language modeling engine 210, such as bynetwork 190 as described inFIG. 1 , arefirst GUI 212 operated by a project manager or similar user role for managing the collection and using information created by the ontology,second GUI 214 operated by an expert annotator, andthird GUI 216 operated by an annotator. Though depicted as individually coupling to naturallanguage modeling engine 210, each offirst GUI 212,second GUI 214 andthird GUI 216 may, in some embodiments, be directly connected to one another, such that an input fromsecond GUI 214 may be transmitted tothird GUI 216 without intermediary communication with naturallanguage modeling engine 210. - In some embodiments, a project manager can, through
first GUI 212, access a collection of documents in naturallanguage modeling engine 210. The project manager can select certain topics, create topics, or review a set of keywords for suggested topics provided by naturallanguage modeling engine 210, and the modeling module of naturallanguage modeling engine 210 can then build an ontology for that topic with a hierarchical structure of labels and tasks of the documents related to that topic. - For example, for a collection of Twitter tweets with the hashtag # Chase, a project manager may want to determine the general disposition of the tweets as, and if, they relate to Chase banking. In one embodiment, the project manager can create a topic for the tweets specifically with the name “Chase Bank” and the modeling module responds by identifying and organizing the tweets into an ontology based on that topic with sub labels and tasks further refining the disposition of the tweets. In some embodiments, certain tweets will be excluded, for example those relating to baseball player Chase Utley that may otherwise have the hashtag # Chase in it. In some embodiments, a “relevant” and “irrelevant” label within the ontology will distinguish the # Chase tweets relating to banking or baseball.
- In some embodiments, the annotation module of natural
language modeling engine 210 creates a work unit interface to display to expert annotators operatingsecond GUI 214 or annotators operatingthird GUI 216. As depicted,interface system 200 may include a plurality ofsecond GUI 214 orthird GUI 216. In some embodiments, the annotation module selects certain documents from the ontology displayed onfirst GUI 212 to populate a work unit interface. In some embodiments, the document selected is merely to confirm the accuracy of the label or task of the placement within the ontology. In some embodiments, the document is selected because the modeling module cannot determine which label or task of the ontology to document should be applied to based on its own processing rules. - In one embodiment
second GUI 214 is distinguished fromthird GUI 216 by having a create option when displaying a work unit interface. The work unit interface is described in further detail in other parts of this disclosure. For purposes of describing the role of each interface in relation to one another;second GUI 214 orthird GUI 216 receive annotations from a respective expert annotator or annotator, and the annotation module of naturallanguage modeling engine 210 aggregates the annotations.First GUI 212 displays the aggregated annotations to a project manager to indicate the accuracy of the ontology or suggestions for categorizing the information to be more accurate. - To more fully describe the capabilities and structures displayed on a
first GUI 212 as described inFIG. 2 ,FIG. 3 illustrates an example embodiment for acontrol page 300 displayed to a project manager or other user offirst GUI 212. In some embodiments,control page 300 is displayed after a project manager or similar user has accessed a system operating the natural language modeling engine, such as the one described inFIG. 2 , with a login credential identifying the user as a project manager or other similar role for managing collections. In some embodiments,control page 300 displays acollections pane 310 and a sharing andpermissions pane 320. -
Collections pane 310 permits a user offirst GUI 212 to create a new project for analyzing a collection of documents in a natural language modeling engine by engaging a createnew collection action 312, or access previous projects by engaging a view allcollections action 314. In some embodiments, sharing andpermissions pane 320 permits a user offirst GUI 212 to allow other users, such asuser 132 oruser 152 to access any analytical information of a collection of documents managed by a project manager. -
FIG. 4 illustrates an example embodiment ofdashboard 400 that is presented to a user after accessing the natural language modeling engine, such as the one described inFIG. 2 . In some embodiments,dashboard 400 directs a user to updates to, and new features provided by, the natural language engine model in afeatures pane 410 to keep first GUI functioning with the latest operational capabilities.Dashboard 400 in some embodiments includes amaintenance pane 412 to identify system services or other alerts, such as the availability of the natural language modeling engine. - In some embodiments,
dashboard 400 includes acurrent projects pane 420.Current projects pane 420 is configured to display a variety of information about projects theuser accessing dashboard 400 is managing or has permissions to view, such as model accuracy of a particular project of a collection of documents, the number of annotations outstanding for that particular project, or the topics and labels and tasks of the particular project.Current projects pane 420, in some embodiments, includes a view all projects action button to view more projects withincurrent projects pane 420, or a create new topic action button to direct the user to a topic creation series of displays described more fully below. - In some embodiments,
dashboard 400 further includesannotation pane 430 for the user offirst GUI 212 to access a series of work unit interfaces created for annotating. In some embodiments,dashboard 400 is displayed onsecond GUI 214 orthird GUI 216 as described inFIG. 2 anddisplays annotation pane 430 to those respective users for accessing documents for annotation by the respective user.Dashboard 400, in some embodiments, recognizes an expert annotator or annotator from a login credential and directs the expert annotator or annotator to theannotations pane 430 for annotating documents through their respective work unit interfaces. -
FIG. 5 illustrates a sample display for initiating the creation of an ontology from a collection of documents.Collection pane 500 is displayed onfirst GUI 212 and comprises aclient pane 510,classification pane 520, andcollection data pane 530. In some embodiments,client pane 510 includes a document action button, a topic action button, and create task button. A document action button permits a user offirst GUI 212 to access the documents within a collection associated with a particular client, such as a customer that has uploaded documents to the natural language modeling engine. A topic action button permits a user offirst GUI 212 to view alternative or additional topics in a collection to allow additional ontologies for those topics as necessary or determined by the user offirst GUI 212. A create task button permits a user offirst GUI 212 to add labels or tasks to an ontology in addition to any that may have been created by a natural language modeling engine. - In some embodiments,
classification pane 520 is displayed to the user oncollection pane 500 to display a suggested ontology based on previous ontologies and collections the user offirst GUI 212 has used. In some embodiments,classification pane 520 is populated with other labels and tasks associated with a particular client as identified inclient pane 510. For example, if a particular client as identified throughclient pane 510 by a natural language modeling engine has historically and consistently used certain labels and tasks for categorizing a collection of documents, the natural language modeling engine can build a new ontology with those historic labels and tasks for a new collection of documents and display the resulting ontology inclassification pane 520. - In some embodiments,
collection data pane 530 displays documents that are part of a collection to be analyzed. In some embodiments, and as depicted inFIG. 5 ,collection data pane 530 includes an import data action button to initiate access to a collection of documents, such as a collection provided by a client or customer, for analysis. In some embodiments,classification pane 520 displays an ontology reactive to the documents present incollection data pane 530, such that an ontology is created or previous ontologies are updated as documents accessed throughcollection data pane 530 are interfaced with and annotated. - In some embodiments,
collections pane 500 includes a discovertopics action button 540 to initiate an automatic construction of an ontology through a discover topics interface more fully described in conjunction withFIG. 6 . -
FIG. 6 illustrates acollection pane 500 configured to display a discovertopic action button 610. In some embodiments, discovertopic action button 610 prompts a natural language modeling engine to analyze the collection of documents accessed byfirst GUI 212, such as throughcollections data pane 530 as described inFIG. 5 , for common themes and keywords. In some embodiments, afirst topic suggestion 612 is displayed incollection pane 500 onfirst GUI 212 with secondtopic suggestion 614, though many other iterations and numbers of topic suggestions are possible. As an illustrative example, for a collection of documents accessed byfirst GUI 212 there may be a large number of documents with the word “horse” and a large number of documents with the word “betting.” Engaging discovertopic action button 610 can prompt the natural language modeling engine to form a first topic and ontology around the word “horse” and sub labels and tasks such as “breed,” “positive,” “diet” and a second topic and ontology around the word “betting” with sub labels and tasks such as “owner” and “race dates.” - In some embodiments, a topic suggestion includes further displays of the keywords identified to justify the creation of the topic suggestion. For example, as depicted in
FIG. 6 .,keywords pane 615 displays at least one keyword recognized through a plurality of the documents in a collection and in some embodiments, a plurality ofdocuments 616 containing that keyword is further displayed incollection pane 500. In some embodiments,classification pane 520 is configured to display the suggestedtopics keywords 615 as an ontology; that is, the suggestedtopics 612 andkeywords 615 can be used in some embodiments as the label and tasks for a collection of documents to create an ontology displayed inclassification pane 520. A user offirst GUI 212 can review the suggested topics, and select which set of keywords identified by the natural language modeling engine best reflect the needs of the project and thereby select a topic for building an ontology to display inclassification pane 520. - By presenting the documents associated with a keyword, through a plurality of
documents 616, a user offirst GUI 212 can see how the keywords are used to further gauge the context of the keywords and not simply the presence of the word before choosing a topic and building an ontology. For example, using the above “horse” and “betting” topics; if the keyword for a document is “race” or “bet,” and plurality ofdocuments 616 displays advertisements for stables with the lines, “You will race over to get a stall in our stable,” or “You can bet your family will love our horses,” a user offirst GUI 212 can determine those documents are not truly indicative of racing or betting and remove them from the ontology, or decide to choose another topic that may have plurality ofdocuments 616 more inline with the desired keywords. - In some embodiments,
keywords 615 are identified independently oftopic suggestion keywords 615 without a threshold topic to group those keywords under. In some embodiments, each keyword withinkeywords 615 is a label or task for an ontology. For example, to use the “horse” and “betting” examples from above, rather than categorizing a collection of documents that may have the words “horse” and “betting” within them into distinct topics with keywords directed to those particular words, a natural language modeling engine can create an ontology from only the keywords. In these embodiments, labels such as “relevant” or “irrelevant” may be more important to distinguish which documents are applicable for a label or task, as a threshold topic selection may not have filtered these documents. - As depicted in
FIG. 7A , in some embodiments, the user of afirst GUI 212 can create an ontology independently through a createclassification task pane 720 displayed incollection pane 500 on afirst user GUI 212. These embodiments represent additional ways to create an ontology without a specific topic selection. In some embodiments,first GUI 212 automatically discovers suggested topics, such as suggestedtopics FIG. 6 , and displays akeywords suggestion pane 710 populated with keyword sets identified across those topics.Keyword suggestion pane 710 can determine keywords much in the same way as described inFIG. 6 by recognizing common words and identifying major themes across a collection of documents. From createclassification task pane 720, the user offirst GUI 212 has more autonomy in applying those keywords to create an ontology.Keywords suggestion pane 710 can allow the user to quickly assess the correlation among the keyword set, the context, presence of slang, and other subjective factors to create an appropriate topic around. - In some embodiments, a
topic label pane 730 is further presented incollection pane 500 to receive a user's specific input for the labels and tasks to a collection of documents. For example, though keywords can be readily used for labels or tasks, a user of first GUI may have specific labels in mind for an ontology and can direct a natural language modeling engine to build an ontology on those directed labels. In those embodiments, non-intuitive relationships can be constructed that artificial intelligence may not yet be programmed for or experienced enough to identify on its own. - In some embodiments, once the ontology is created, either by the natural language modeling engine discover topics function described above or by the create classification task function dictated by the user of a first GUI as described above, certain documents are selected for annotation to confirm the accuracy of the placement of the document within a label of the ontology. In some embodiments, the natural language modeling engine cannot determine which label(s) are applicable to a document and does not know where to place a particular document into the ontology and selected the document for annotation. In some embodiments, a natural language modeling engine selects documents for verification of placement, despite a high likelihood of successful categorization. In some embodiments, the natural language modeling engine constructs a work unit interface to efficiently receive annotations for such verification or placement.
- As depicted in
FIG. 7B , in some embodiments,keyword suggestion pane 710 is a visual presentation rather than a purely textual list of keywords. In some embodiments,keyword suggestion pane 710 is accompanied bytopic circle graph 740 displaying possible topics based on a number of keywords supporting a topic. For example, a topic with more keywords related to it is displayed as a larger topic circle. Additionally, intopic circle graph 740 the relation between topics is visually displayed. In some embodiments, topics with more keywords in common with another topic are displayed closer together ontopic circle graph 740. - In some embodiments,
relevant terms window 750 displays keywords across the collection of documents. In some embodiments, the frequency of a particular keyword within a topic selected by a user intopic circle graph 740 is displayed inrelevant terms window 750 with a comparison for how frequently that keyword appears in other documents across the collection. For example, if the word “banking” appears four hundred times across a collection of documents, and four hundred times in a particular topic a user could readily deduce that “banking” is very relevant to the collection of documents and that an ontology for the collection of documents should include the word “banking.” -
FIG. 8A depicts an example expert annotatorwork unit interface 800 as presented on asecond GUI 214 constructed by a natural language modeling engine. In some embodiments,work unit interface 800 comprises adocument pane 810, aprompt pane 820,label pane 830 comprising at least one label or task,reference button 835 paired to a label or task withinlabel pane 830, createlabel pane 840 and createguideline pane 845. - In some embodiments, expert annotator
work unit interface 800 is presented onsecond GUI 214 upon a user logging into a natural language modeling engine with an expert login credential. As described inFIG. 4 andannotation pane 430,second GUI 214 may display anannotation pane 430 directing the user to expert annotatorwork unit interface 800 for annotations from the user operatingsecond GUI 214. Expert annotatorwork unit interface 800 is constructed by a natural language modeling engine displaying in document pane 810 a document from the ontology to be annotated, and listing at least one label or task from the ontology inlabel pane 830. In some embodiments, natural language modeling engine accesses a database of guidelines defining or otherwise describing the label or task displayed inlabel pane 830 and pairs the guideline with the respective label or task. In some embodiments,reference button 835 is a link to the guideline paired with a label or task inlabel pane 830. The function of a reference button is further described in conjunction withFIG. 8C . - In some embodiments, the intelligent queuing module of natural
language modeling engine 210, such as the one described inFIG. 2 , generates a human readable prompt to elicit a label or task selection by a user of expert annotatorwork unit interface 800. Examples of such human readable prompt include, but are not limited to, “select the best label for the document” from a plurality of labels inlabel pane 830, “select all labels that apply to the document” from a plurality of labels inlabel pane 830, “rank the labels in order of relevance” from a plurality of labels inlabel pane 830, or binary response prompts such as “does this label apply to the document?” with yes and no labels inlabel pane 830. One of skill in the art can imagine a multitude of applicable human readable prompts. The human readable prompts are not necessarily identical across all work unit interfaces displaying the same document. In some embodiments,work unit interface 800 displays document 810′ in thedocument pane 810 and prompt 820′ inprompt pane 820. In other embodiments,work unit interface 800 displays document 810′ in thedocument pane 810 and prompt 820″ inprompt pane 820. By using different prompts for the same document, natural language modeling engine can still collect annotations from thework unit interface 800 but have a more diverse basis for aggregation information to a label. - In some embodiments, human
readable prompt 820 elicits the selection or categorization of portions of text displayed indocument pane 810. Examples of such prompts include, but are not limited to, “select all examples of each label within the document” with a list of labels inlabel pane 830 and “is the highlighted section of the document an example of this label?” with yes and no labels inlabel pane 830. In some embodiments,work unit interface 800 displays document 810′ indocument pane 810 with a plurality of example regions of text visually distinguished from the rest of the document to assist the annotator. In some embodiments, the example regions of text are created using the API module of naturallanguage modeling engine 210. Examples of visual representations for example regions include, but are not limited to, using unique background colors around the example region to highlight the text, and underlining the example text regions. In some embodiments, the example region is distinguished with variable degrees of visual representation to reflect a natural language modeling engine's confidence in selecting an example region as a correct example of a label or task. For example, in some embodiments, a thicker underlined example region indicates stronger confidence as opposed to a thinner underlined example region, or an opaque highlighted background color as opposed to a semi-transparent background color. - In some embodiments,
prompt pane 820 is populated with the generated human readable prompt. In some embodiments, expert annotatorwork unit interface 800 permits the user ofsecond GUI 214 to populate createlabel pane 840 with a new label. Expert annotators creating new labels for documents can distinguish certain nuances in documents that normal annotators or a natural language modeling engine cannot, such as legal interpretations or advanced sciences that may have distinct meanings in a particular field. By entering a new label or task into createlabel pane 840, the expert annotator can update other work unit interfaces with the created labels the expert annotator has identified, and update an ontology with more accurate categorizations. Similarly, in some embodiments, expert annotatorwork unit interface 800 includes a createguideline pane 845 permitting the user ofsecond GUI 214 to populate createlabel pane 845 with a revised guideline to pair with a particular label to provide more descriptive information to help other expert annotators or other annotators interpret the applicability of a label or task rather than simply rely on the guideline provided by the natural language modeling engine. In some embodiments, createguideline pane 845 receives a “gold” designation from an expert annotator to indicate a particular label or task is particularly representative or a good example otherwise of the document or prompt displayed in a work unit interface. Such “gold” or similar exemplary marker is displayed in a work unit interface displayed to other expert annotators or annotators as a guideline explained more fully as 890 in describingFIG. 8C . Such designations indicate not only that an expert annotator wants to draw attention to a particular label or task, but can also be used to train other expert annotators or annotators to what the particular label or task should be representing. - In some embodiments, an annotation of a document is made on expert annotator
work unit interface 800 by selecting a label or task displayed inlabel pane 830 and the annotation is recorded by a natural language modeling engine. - In some embodiments, subsequent to selection of a label displayed in
label pane 830,work unit interface 800 immediately displays an additional human readable prompt inprompt pane 820 and populates at least one subsequent label inlabel pane 830 responsive to the earlier label selection of the first human readable prompt. For example, if the expert annotator answers “Yes” to a first human readable prompt about document relevance,prompt pane 820 may immediately display an additional human readable prompt requesting the best label for the document. By contrast, if the expert annotator answers “No” to the first prompt, an additional human readable prompt is not displayed. - In some embodiments, the additional human readable prompt created for
prompt pane 820 matches the ontology structure displayed inclassification tasks pane 520. In such embodiments, annotations of a document are made for all labels and tasks in an ontology by selecting a label or task inlabel pane 830 for each additional human readable prompt. -
FIG. 8B illustrates an embodiment of an annotatorwork unit interface 850 as displayed on athird GUI 216. In some embodiments, annotatorwork unit interface 800 is presented onthird GUI 216 upon a user logging into a natural language modeling engine with an annotator login credential. As described inFIG. 4 andannotation pane 430,third GUI 216 may display anannotation pane 430 directing the user to annotatorwork unit interface 850 for annotations from the annotator operatingthird GUI 216. Annotatorwork unit interface 850 is constructed by displaying in document pane 860 a document from the ontology to be annotated, and listing at least one label or task from the ontology inlabel pane 880. In some embodiments, natural language modeling engine accesses a database of guidelines defining or otherwise describing the label or task displayed inlabel pane 880 and pairs the guideline with the respective label or task. In some embodiments,reference button 885 is a link to the guideline paired with a label or task to present to the annotator onthird GUI 216 the definition of the particular paired label or task inlabel pane 880. The function of a reference button is further described in conjunction withFIG. 8C . - In some embodiments, the intelligent queuing module of natural
language modeling engine 210, such as the one described inFIG. 2 , generates a human readable prompt to elicit a label or task selection by a user of annotatorwork unit interface 850. Examples of such human readable prompt include, but are not limited to, “select the best label for the document” from a plurality oflabel panes 880, “select all labels that apply to the document” from a plurality oflabel panes 880, “rank the labels in order of relevance” from a plurality oflabel panes 880, or binary response prompts such as “does this label apply to the document?” with yes and nolabel panes 880. One of skill in the art can imagine a multitude of applicable human readable prompts. The description ofprompt pane 820 in describing expert annotatorwork unit interface 800 is applicable in the various prompts possible in describingprompt pane 870. In some embodiments,prompt pane 870 of annotatorwork unit interface 850 is populated with the generated human readable prompt. - In some embodiments, human
readable prompt 870 elicits the selection or categorization of portions of text displayed indocument pane 860. Examples of such prompts include, but are not limited to, “select all examples of each label within the document” with a list of labels inlabel pane 880 and “is the highlighted section of the document an example of this label?” with yes and no labels inlabel pane 880. In some embodiments,work unit interface 850 displays document 860′ indocument pane 860 with a plurality of example regions of text visually distinguished from the rest of the document to assist the annotator. In some embodiments, the example regions of text are created using the API module of naturallanguage modeling engine 210. Examples of visual representations for example regions include, but are not limited to, using unique background colors around the example region to highlight the text, and underlining the example text regions. In some embodiments, the example region is distinguished with variable degrees of visual representation to reflect a natural language modeling engine's confidence in selecting an example region as a correct example of a label or task. For example, in some embodiments, a thicker underlined example region indicates stronger confidence as opposed to a thinner underlined example region, or an opaque highlighted background color as opposed to a semi-transparent background color. - In some embodiments, an annotation of a document is made on annotator
work unit interface 850 by selecting a label or task displayed in at least onelabel pane 880 and the annotation is recorded by a natural language modeling engine. - In some embodiments, subsequent to selection of a label displayed in
label pane 880,work unit interface 850 immediately displays an additional human readable prompt inprompt pane 870 and populates at least one subsequent label inlabel pane 880 responsive to the earlier label selection of the first human readable prompt. For example, if the annotator answers “Yes” to a first human readable prompt about document relevance,prompt pane 870 may immediately display an additional human readable prompt requesting the best label for the document. By contrast, if the annotator answers “No” to the first prompt, an additional human readable prompt is not displayed. - In some embodiments, the additional human readable prompt created for
prompt pane 870 matches the ontology structure displayed inclassification tasks pane 520. In such embodiments, annotations of a document are made for all labels and tasks in an ontology by selecting a label or task inlabel pane 880 for each additional human readable prompt. -
FIG. 8C illustrates a reference button function on either an expert annotatorwork unit interface 800 or annotatorwork unit interface 850 and therespective reference button 835 orreference button 885. Upon selection by a user ofsecond GUI 214 orthird GUI 216 ofreference button 835 orreference button 885 respectively,guideline 890 is displayed within the work unit interface. In some embodiments, theguideline 890 displayed is the guideline accessed from a database of a natural language modeling engine for a particular label; in some embodiments theguideline 890 displayed is the revised guideline as created by an expert annotator through expert annotationwork unit interface 800 createguideline pane 845 onsecond GUI 214, and updated on all work unit interfaces present on GUIs within aninterface system 200 such as one depicted inFIG. 2 . In some embodiments,guideline 890 displays both the guideline accessed from a database of natural language modeling engine as well as the revised guideline, if any, created by the expert annotator. - In some embodiments, the annotations received on all work unit interfaces 800 and 850 are aggregated together by ontology the documents underlying the work unit interface were drawn from. In some embodiments, the aggregation occurs in an annotation module of a natural
language modeling engine 210 and are shared on afirst GUI 212 such as depicted inFIG. 2 . -
FIG. 8D illustrates a span annotation work unit interface interaction. In some embodiments, a work unit interface displays a document in a document pane and span prompt 891 in a prompt pane. In some embodiments, span prompt 891 is associated withspan annotation 892 in a label pane. In some embodiments, a natural language modeling engine predicts spans of a document that represent a particular label or task and displays the predicted span as a highlighted text or underlinedtext 893 in the document displayed in the document pane of the work unit interface. One of skill in the art can appreciate other ways of visually distinguishing text for span annotation in a work unit interface. In some embodiments, an expert annotator or annotator annotates the span prediction with aspan annotation 892 in the label pane. - In some embodiments, a span prompt 891 requests confirmation of multiple spans, such as “location” and “person” and a toggle or swatch or menu function in the document pane switches between the prompted spans and displays different highlighted or underlined
text 893 corresponding to the span prompted inspan prompt 891. For example, a span prompt requests confirmation of a document that the highlighted or underlined text represents “people” and “locations.” An expert annotator or annotator selects a toggle, swatch, or menu function in the document pane for “locations” and the work unit interface displays those highlighted or underlined texts the natural language modeling engine has predicted correspond to “locations.” The expert annotator or annotator then annotates with aspan annotation 892 in the label pane and then presses the toggle, swatch, or menu function for “people” and the work unit interface displays those highlighted or underlined texts the natural language modeling engine has predicted correspond to “people.” The expert annotator or annotator then annotates for “people” and the natural language modeling engine processes the span annotations. -
FIG. 8E illustrate a span selection work unit interface. In some embodiments,span selection prompt 894 is displayed in a prompt pane of a work unit interface requesting an annotator highlight or underline or otherwise visually distinguish a span within a document pane. In some embodiments, an expert annotator or annotator annotates an otherwise unmarked document in a document pane by highlighting or otherwise visually distinguishing aspan 895 within the document. In some embodiments, a natural language modeling engine predicts which span corresponds to requested label or task inspan prompt 894 by visually distinguishing a span in a separate method than an expert annotator or annotator would. For example, for aspan selection prompt 894 requesting a user select a span for “locations,” a natural language modeling engine will instruct a work unit interface to present underlinedspans 896 the natural language modeling engine predicts correspond to “locations,” and an expert annotator or annotator can confirm the prediction by highlighting the underlined span or highlighting other spans to annotate the document. -
FIG. 9A illustrates anannotation agreement interface 900 display on afirst GUI 212 with alabel feedback pane 910,annotation feedback pane 920, and rulespane 930. As further illustrated inFIG. 9A , in some embodiments,label feedback pane 910 includes a by label description of each label within an ontology with the number of annotations applied to the label and an option to delete or edit the label. Fromlabel feedback pane 910, a user offirst GUI 212 can remove a label from an ontology if the annotation agreement for that label is low or the user offirst GUI 212 determines it is not applicable to the ontology, or edit the label such as by reviewing a create label action as provided through asecond GUI 214 and determining that label more applicable or descriptive. Similarly, the user offirst GUI 212 can determine not enough annotations have been applied to the label to draw any conclusions from and decide to wait before making any adjustments to that label. -
FIG. 9B illustrates an example of alearning curve 915 withinlabel feedback pane 910. In some embodiments,learning curve 915 is a graphical representation of the relationship between the number of annotations received for a particular label and the agreement between the annotations for the label. In some embodiments,learning curve 915 is a graphical representation of the relationship between the number of annotations received for a particular label and the accuracy of the natural language model generated for that label. Annotation agreement to a label, in some embodiments, is calculated by an annotations module of a natural language modeling engine, such as naturallanguage modeling engine 210 as depicted inFIG. 2 . In some embodiments, the annotation agreement is a number indicating the incidence rate of mutually agreed annotations among all annotators operating secondGUIs 214 orthird GUIs 216. - For example purposes only of one way to calculate an annotation agreement, if 10 annotators all annotated a document with a label of “positive” and 10 annotators did not annotate the same documents as “positive,” then an annotation agreement of 0.50 or 50% would be reflected for the “positive” label or task of the document and
learning curve 915 of those labels or tasks would depict the 0.50 or 50% agreement for 20 annotations. In the same example, if the next 20 annotators gave a “positive” annotation to the same document, the annotation agreement would update to 75% for a “positive” annotation for 40 forty annotations andlearning curve 915 would graphically depict this relationship between annotation agreement as a function of the number annotations. One of skill in the art can appreciate other annotation agreement calculation methods. - In some embodiments, the accuracy of the natural language modeling engine in assigning documents to appropriate labels or tasks of an ontology is derived from cross-validation processes of the annotations used in a learning curve. In some embodiments, a modeling module of a natural language modeling engine, such as natural
language modeling engine 210 as depicted inFIG. 2 , performs cross-validation on the annotation dataset to determine ontology accuracy. One of skill in the art can appreciate applicable cross-validation techniques to apply to an annotation dataset such as exhaustive or non-exhaustive methods. -
FIG. 10A illustrates an example display of anannotation feedback pane 920 withannotation agreement score 1010, individualannotator agreement list 1030, and suggestedlabel collapse list 1020. In some embodiments,annotation agreement score 1010 displays an aggregate annotation agreement result as a proxy for the accuracy of an ontology. For example, in some embodiments, theannotation agreement score 1010 is determined by aggregating all annotations from second orthird GUIs first GUI 212 the degree of agreement among annotators on whether the labels or tasks presented on a work unit interface accurately reflected the document displayed on the work unit interface. For example, a low agreement score would indicate that the labels or tasks may have been too vague or inapplicable, and the annotators could not agree on whether, or which of, the labels presented in a label pane of applied to the document of the work unit interface and therefore, perhaps the natural language modeling engine did not have the most appropriate series of labels or tasks to categorize the ontology. By contrast, a higher agreement score could indicate the annotators found at least one label of a label pane was an accurate match to the document based on the generated prompt, and therefore the annotators found at least one the labels readily applied and the natural language modeling engine accurately captured the label or task of at least some documents. - In some embodiments,
annotation agreement score 1010 can be broken down into a per label agreement, and in suggestedlabel collapse list 1020 indicate which labels or tasks introduced higher disagreement among annotator and display the annotation agreement score that would result if certain labels or tasks were collapsed—or combined—with each other. For example, as depicted inFIG. 10A , the suggestedlabel collapse list 1020 displays the resultingannotation agreement score 1010 of collapsing certain labels or tasks into one of several other labels or tasks of a hypothetical ontology (e.g. collapsing “Legal” with “Securities issues” and “JPMC Financial” with “Other”). In this example embodiment ofFIG. 10A , collapsing “Legal” with “Other” results in anannotation agreement score 1010 of 0.773 as compared to anannotation agreement score 1010 of 0.741 if the two were separate labels or tasks. As this represents the largest increaseannotation agreement score 1010 within the suggestedlabel collapse list 1020, a user offirst GUI 212 can readily deduce that annotators had a greater difficulty distinguishing “Legal” from “Other” than they did distinguishing any other two labels or tasks within the ontology. A user offirst GUI 212 can make several other deductions from this information, such as whether the guideline describing “Legal” or “Other” sufficiently describes the label or task, or whether the prompt should be changed to permit more nuanced distinctions, or whether the ontology itself should not include a particular label or task. - In some embodiments an individual
annotator agreement list 1030 displays how well a particular annotator within the aggregation of annotations agrees with other annotators. For example, as displayed inFIG. 10A , the annotator identified as “demo4” has an agreement value of 0.601, the lowest of the annotator group depicted inFIG. 10A , indicating that demo4 has a low incidence of agreeing with the other annotators (i.e. “demo1,” “demo2,” and “demo3”) on a label or task as prompted in a work unit interface. As depicted inFIG. 10A , annotator demo4 also has only annotated 100 documents compared to the other annotators' 550. This display of information could suggest that annotator demo4 may need to be removed from the annotation group (for example, the subject matter confuses that annotator) to compute anannotation agreement score 1010 that reflects annotations with less annotators that may not understand the subject matter and could be giving false positives on applicable labels to a natural language modeling engine. This information could also be used to indicate annotator demo4 needs to be retrained on a particular subject matter depending on the degree of disagreement. For example, if annotator demo4 selected labels of “slightly positive” as compared to other annotators selecting “positive” then a user ofGUI 212 could decide to retrain annotator demo4. However, if annotator demo4 selected labels of “very negative” as compared to other annotators selecting “very positive” then a user ofGUI 212 could decide to remove annotator demo4 from the analysis. In other instances, a user offirst GUI 212 could simply decide to wait for demo4 to annotate more documents to see if demo4′s agreement value increases; such information can further be used to determine whether or not labels should be collapsed or if a project needs further annotation before drawing conclusions on the natural language modeling engine's accuracy. -
FIG. 10B illustrates an agreement per labelgraphical representation 1040, and collapsed agreement per labelgraphical representation 1045. In some embodiments, agreement per labelgraphical representation 1040 displays individual annotation agreements by each label of an ontology, that, taken all together would compriseannotation agreement score 1010 as depicted inFIG. 10A . Breaking down the annotation agreements into a per label graphical representation informs a project manager or user offirst GUI 212 which labels or tasks, relative to others, had the most agreement and can inform whether the label or task is likely an appropriate reflection of documents in a work unit interface. - In some embodiments, the annotation agreement interface includes a collapsed agreement per label
graphical representation 1045 configured to display the per label annotation agreements if two or more labels were collapsed into one another. As illustrated for example purposes inFIG. 10B , collapsed agreement per labelgraphical representation 1045 indicates that by combining two labels with a respective agreement of 0.613 and 0.274 from agreement per labelgraphical representation 1040 into one another, a new agreement score of 0.59 results for that combined label in collapsed agreement per labelgraphical representation 1045 suggesting theannotation agreement score 1010 as illustrated inFIG. 10A will improve if these two labels are combined into one another, and enabling similar deductions by a project manager or user offirst GUI 212 as described in connection with those functions ofFIG. 10A . -
FIG. 10C illustrates a perdocument agreement list 1050 within anannotation feedback pane 920. In some embodiments, a perdocument agreement list 1050 displays those documents within the ontology that have the highest agreement or lowest agreement among them. In some embodiments, the number of documents displayed in perdocument agreement list 1050 can be adjusted by a user offirst GUI 212. In some embodiments, the perdocument agreement list 1050 displays those documents within a subset of label or task of an ontology with the highest or lowest agreement among annotators. By flagging the documents with the highest or lowest agreement in perdocument agreement list 1050, a user offirst GUI 212 can choose to remove certain documents with low agreement to reduce the number of potentially vague or inapplicable documents within a collection (as indicated by not having strong human agreement on the applicable labels or tasks), or review the content of the particular document and label or task of a work unit interface displaying the document to determine whether the document contains a nuance that should be included as a new label or whether a guideline should be clarified to account for such nuances. - As depicted in
FIG. 11 , an annotation agreement interface can further display arules pane 930 for adjusting any of the natural language modeling engine logic processes for certain inputs. In some embodiments, the rules pane comprises aphrase pane 1122 configured to display a data field for receiving a phrase or word that, if found within a document by a natural language modeling engine, will invoke a rule. In some embodiments,rules pane 930 displays aweighting adjustment pane 1124. In some embodiments,weighting adjustment pane 1124 is configured to receive from a user of first GUI 212 a manipulation to a certain phrase or word inphrase pane 1122 to emphasize or de-emphasize a certain word in placing a document in a label or task category of an ontology. For example, in an ontology with the word “recommend” as a label or task classification, a natural language modeling engine may categorize incidence of the word “recommend” as equivalent to “recommendation” without recognizing the context of the complete phrase or document “recommend” appears in, such as “does not recommend” which would not imply an positive recommendation.Weighting pane 1124 permits a user offirst GUI 212 to reduce or increase the significance of certain words, thereby placing greater or less emphasis on other words in the document relative to the word or phrase inphase pane 1122. Such weighting can focus the ontology on which labels or tasks to create, or which documents should be selected for annotation. Continuing from the previous example, if a document includes the word “recommend” but that word has a low weighting and therefore the user offirst GUI 212 does not consider it important, the natural language modeling engine may not select it for annotation to avoid using an annotator's time to accurately place a document with a low weighting. - In some embodiments,
rules pane 930 includes addrule pane 1126. Addrule pane 1126 permits a user offirst GUI 212, or in some embodiments an expert annotator operatingsecond GUI 214, to create a rule for a particular phrase or word inphrase pane 1122. For example, if the word “recommend” appears in a document, addrule pane 1126 could be manipulated by receiving in a data field of add rule pane 1126 a rule such as “if this then that” logic rules, or rules to search for additional words surrounding a word in aphrase pane 1122. To continue the previous example, addrule pane 1126 could receive a rule to search for preceding words such as “no,” “does not,” “isn't,” or other similar negative implicative words such that if “recommend” is paired with such a negative implicative word, the natural language modeling engine will not categorize the document as an affirmative “recommendation.” - Taken together, a
label feedback pane 910,annotation feedback pane 920, and rulespane 930 of an annotation agreement interface of afirst GUI 212 permits rapid analysis of an ontology that has been annotated through second andthird GUIs Annotation agreement interface 900 further provides access to a variety of tools to determine where an ontology and its attendant labels or tasks can be refined to more accurately determine the underlying meaning of a collection of documents or store information for future ontologies to learn from. For example, if a revised guideline for a label results in an improvedannotation agreement score 1010 of an ontology, the natural language modeling engine can store that revised guideline in a database and used that guideline for future ontologies that use the same label or task the revised guideline is describing. The wealth of information and deductions possible from human annotations to computer analysis tools and the insights such annotations provide can greatly improve mass classification of human communications. -
FIG. 12 illustrates an example ofprocess 1200 for verifying the accuracy of a natural language modeling engine's creation of an ontology of a collection of documents by aggregating human annotations across a series of GUIs.Process 1200 starts at 1210 with accessing a series of inputs through a first GUI, such inputs being those associated with being made by a project manager or similar supervisory role to a collection of documents. - In some embodiments, accessing inputs at 1210 includes accessing at least one document at 1212, such document provided by a third party. A third party access source, in some embodiments, is a customer that provides a collection of documents to be analyzed; in some embodiments, third party access source is a database of a collection of documents such as the
database 115 as depicted inFIG. 1 . In some embodiments, accessinginputs 1210 further includes accessing at least one first label associated with the document at 1214. In some embodiments, the first label is accessed from an ontology created around the document as built from a natural language modeling engine. In some embodiments, accessinginputs 1210 further includes accessing a plurality of first guidelines describing the first label at 1216. In some embodiments, the first guideline is sourced from a database of guidelines, such as one operated by a natural language modeling engine, that are associated with a list of labels and each first guideline is a description or definition of the label or task. - In some embodiments, at 1220 a second label or second guideline is accessed, such as from a
second GUI 214 like the one operated by an expert annotator described inFIG. 2 . Collectively, the access of inputs at 1210 and second guidelines and second labels at 1220 provide a plurality of information components for the construction of a work unit interface to permit human annotation to selected documents. - At 1230, in some embodiments, a work unit interface is built. In some embodiments, building the work unit interface at 1230 involves assigning (which depending on embodiment can mean “populating” or “placing”) the document accessed at 1212 to a document pane of the work unit interface at 1231. In some embodiments, at 1232 labels are assigned to, and populated in, a label pane of a work unit interface. In some embodiments, the labels assigned to the label pane at 1232 are the first labels accessed at 1214 from an ontology of a collection of documents through a first GUI. In some embodiments, the labels or tasks assigned to the label pane at 1232 are the second labels accessed at 1220 from a second GUI. In some embodiments, the label pane is assigned with a plurality of labels, and in still other embodiments, the plurality of labels assigned the label pane at 1232 includes both first labels and second labels.
- At 1233, a human readable prompt is generated to elicit a response from a human annotator that requests a task of the document. In some embodiments, the human readable prompt is generated by an intelligent queuing module of a natural language modeling engine. In some embodiments, the human readable prompt is a question requesting selection of the most applicable label or task assigned in the label pane at 1232 for the document assigned in the document pane at 1231. In some embodiments, the human readable prompts requests selection of all applicable labels or tasks assigned in the label pane of a work unit interface at 1232. One having skill in the art can envision additional human readable prompts requesting a task of a document. At 1234, the generated human readable prompt is assigned to a prompt pane of the work unit interface.
- At 1235, a single guideline from one of the first guideline accessed at 1216 or second guideline at 1220 is paired with a single label or task assigned to the label pane of the work unit interface at 1232. In some embodiments several labels or tasks are assigned to the label pane at 1232, and several single guidelines are paired with a single one of the several labels or tasks at 1235. In some embodiments, after pairing the single guideline with a single label at 1235, a reference button is created for the single guideline at 1236. A reference button permits access to the full textual description of the single label paired with the single label without requiring display of the single guideline. In some embodiments, the reference button is placed adjacent to the single label paired with the single button in the label pane at 1237.
- In some embodiments,
process 1200 continues at 1240 by displaying the built work unit interface to an annotator operating a second GUI or third GUI. In some embodiments, the annotator operating the second GUI is an expert annotator. In some embodiments, the annotator operating the third GUI is an annotator. At 1250, at least one annotation is received through the work unit interface from among the second or third GUIs. At 1260, the annotations received at 1250 are aggregated together. - In some embodiments,
process 1300 provides a method of creating interfaces to efficiently manage and manipulate annotated documents to verify and draw conclusions as to the accuracy of a natural language modeling engine.Method 1300 begins at 1310, and in some embodiments step 1310 is a subsequent to step 1260 as described inFIG. 12 . At 1310, an annotation agreement interface is built. - In some embodiments, building an annotation agreement interface at 1310 includes a series of
substeps - In some embodiments, at 1314 a learning curve pane is created. In some embodiments, at 1314 the learning curve pane displays an aggregation of annotations from among GUIs in a network, such as
network 200 as depicted inFIG. 2 with a plurality ofsecond GUIs 214 orthird GUIs 216. In some embodiments, at 1314 the learning curve pane displays a graphical representation of the relationship between the number of annotations received and the agreement between those annotations. In some embodiments, at 1314 the learning curve pane displays a graphical representation of the relationship between the number of annotations received for a particular label and the accuracy of the natural language model generated for that label. - In some embodiments, the learning curve displayed at 1314 is a learning curve for an entire collection. In some embodiments, the learning curve displayed at 1314 is a learning curve for a particular label or task. By displaying a learning curve, a project manager or user of
first GUI 212 can determine whether the annotations are beginning to smooth out and/or approach a consistent agreement regardless of additional annotations or whether the number of additional annotations introduces continued variability (which would be represented as a staggered line in a learning curve) of agreement. With such information, a project manager or user offirst GUI 212 can allocate annotators efficiently, such as by ceasing to request annotations for a particular label or task or collection of documents if it is apparent that additional annotations will not appreciably affect the agreement. Or, in instances with a high degree of variability with additional annotations a project manager or user offirst GUI 212 could assign additional annotators to provide more human oversight to attempt to reach a consistent agreement level. - In some embodiments,
method 1300 continues to 1316 by creating an annotation feedback pane. An annotation feedback pane can be configured to display a plurality of data. In some embodiments, an annotation feedback pane displays an annotation agreement score. An annotation agreement score displays the overall agreement between annotators of the whole collection of documents within the ontology being analyzed, and can indicate the general disposition or accuracy of the entire ontology and whether further by-label or by-task analysis or manipulation is warranted given the overall agreement score or whether annotators should retrained on the definition of particular categories. - In some embodiments, an annotation feedback pane displays at 1316 an individual annotator agreement list with the agreement scores and number of annotations per individual annotator within the collection. Such an individual annotator agreement list indicates whether certain annotators have processed all documents presented them on work unit interfaces of their respective GUIs, or whether certain annotators appear to have trouble with the subject matter is indicated by low agreement scores relative to other annotators. For example, if an overall agreement, such as one displayed in an annotation agreement score, were 0.7 with individual annotator agreement list displaying individual agreements of 0.75, 0.55, 0.7, and 0.77, a project manager could determine that the individual annotator with the 0.55 score should be dropped from the annotation agreement score.
- In some embodiments, at 1316 a suggested label collapse list is created. A suggested label collapse list enables a project manager or user of a first GUI to quickly identify the strengths and weaknesses of an ontology and a natural language modeling engine's ability to sort documents into a particular label or task based on the document's content. In some embodiments a suggested label collapse list is created by pairing annotated labels or tasks together and displaying how the annotation agreement score would be affected if the two labels or tasks were combined into a common label or task; in other words, how an annotation agreement score would be affected if annotators were not required to distinguish between certain labels or tasks. Such a feature can indicate whether the natural language modeling engine selected appropriate labels or tasks, or whether guidelines paired with labels or tasks are sufficiently describing the label or task.
- In some embodiments, at 1316 an agreement per label graphical representation is created by displaying a bar graph of agreement among annotators for one or more given labels or tasks of the ontology being annotated. In some embodiments, a collapsed agreement per label graphical representation is created by determining the agreement per label or task if certain labels of tasks were collapsed into one another. Such a feature further indicates which labels or tasks within an ontology are more likely to be correctly placed on an ontology by creating an interface for side by side comparison of agreements if labels or tasks were asked to be distinguished from one another or if they were combined.
- In some embodiments, at 1316 a per document agreement list is created by identifying those documents with the highest and lowest annotation agreements. In some embodiments, the number of documents displayed in a per document agreement list is determined from an input by a project manager or user of
first GUI 212. A per document agreement list allows a project manager or user of afirst GUI 212 to determine which documents gave annotators the most difficulty in agreeing on a common label or task, as well as showing which documents have unanimous agreement and make the best exemplars of a category. A project manager or user of first GUI can then review certain documents to determine whether anything particular is giving trouble to annotators, or even remove the document from the ontology for its lack of a clear disposition. - In some embodiments,
method 1300 continues at 1318 to create a rules pane for receiving a phrase in a phrase pane, a weighting adjustment in a weighting adjustment pane, or a rule in an add rule pane. In some embodiments, a phrase pane is configured to identify a certain phrase within a document. In some embodiments, a weighting adjustment pane is configured to reduce the relevance of a phrase in the phrase pane relative to other words or phrases in the document such that a natural language modeling engine will emphasize or deemphasize certain phrases. In some embodiments, an add rule pane gives direction for a natural language modeling engine to perform on a phrase in a phrase pane. For example, if the natural language modeling engine recognizing a phrase from the phrase pane in a document, a rule in the add rule pane can dictate how to process that document, such as by placing it in a specific label or task of an ontology, or looking for additional words or phrases before placing the document within an ontology. - In some embodiments,
method 1300 continues at 1320 by computing initial annotation agreements from aggregated annotations such as at 1260 as depicted inFIG. 12 . In some embodiments, computing initial agreements is performed by an annotations module, such as one of naturallanguage modeling engine 210 depicted inFIG. 2 . Computing initial annotation agreements includes not only determining annotation agreement scores among annotators for individual labels or tasks and overall score for an ontology, but also includes in some embodiments computing a learning curve relationship between annotation agreement and number of annotations. In some embodiments at 1320, computing initial annotation agreements determines annotation agreement scores if two or more labels are collapsed into one another. In some embodiments, computing initial annotation agreements computes per label annotation agreements and per label annotation agreements if two or more labels are collapsed into one another and graphical relationships of each. In some embodiments, computing initial annotation agreements at 1320 computes the annotation agreement per each document classified according to an ontology, and identifies the documents with the highest and lowest annotation agreements. - In some embodiments, the computed metrics of 1320 are populated into an annotation agreement interface at 1330. In some embodiments, a label feedback pane is populated with each label of an ontology. In some embodiments, at 1330 the label feedback pane is populated with a description of the label as provided by a natural language modeling engine or project manager of
first GUI 212 or expert annotator operating asecond GUI 214. In some embodiments, at 1330 the label feedback pane is populated with the number of annotations a label or task has received from annotators operatingsecond GUI 214 orthird GUI 216. - In some embodiments, at 1330 the learning curve pane is populated with the computed learning curve metric of the annotation agreement related to the number of annotations, as computed at 1320. In some embodiments, at 1330 the annotation feedback pane is populated with the annotation agreement score for the ontology. In some embodiments, at 1330 the annotation feedback pane of the annotation agreement interface is populated with a suggested label collapse list for the resultant annotation agreement scores corresponding to how the annotation agreement score will change if two or more labels or tasks are collapsed into one another.
- In some embodiments, at 1330 the annotation feedback pane of the annotation agreement interface is populated with the per label or task annotation agreements and collapsed agreement per label or task as computed at 1320. In some embodiments, at 1330 a per document agreement is populated in an annotation feedback pane.
- In some embodiments, at 1340 the panes of the annotation interface populated at 1330 are displayed to a project manager or user of
first GUI 212. With a fully populated annotation agreement interface displayed to such a project manager or user offirst GUI 212,method 1300 enables follow on actions to manipulate or refine the documents and annotators to improve the ontology, such as by enabling improved annotations on other work unit interfaces, or removing documents from the ontology. - As depicted in
FIG. 14 ,method 1400 is a method for interacting with information displayed to a project manager or user offirst GUI 212 on an annotation agreement interface. In some embodiments,method 1400 begins at 1410 by receiving a request to collapse at least one first label or first task into at least one second label or second task. Such a request to collapse in some embodiments is made, in some embodiments, through the agreement per label graphical representation of the annotation feedback pane. In some embodiments, the request to collapse at 1410 is made through the suggested label collapse list. One of skill in the art can envision numerous ways to select which labels or tasks to collapse into one another through one of the information displays of an annotation agreement interface. - In some embodiments, at 1430 a subsequent annotation agreement is displayed in the annotation feedback pane of the annotation agreement interface after collapsing the labels or tasks in 1410. Such a subsequent annotation agreement can be used as a visual comparison of the annotation agreements before and after collapsing labels or tasks.
-
FIG. 15 illustratesmethod 1500 for changes to a work unit interface a project manager or user offirst GUI 212 can make in response to information populated on an annotation agreement interface such as bymethod 1300 described inFIG. 13 . In some embodiments,method 1500 is directed to replacing a guideline for a label in a work unit interface. In some embodiments, a project manager or user offirst GUI 212 decides to replace a guideline to a label or task in a work unit interface, such as by realizing a per label annotation agreement computed at 1320 inmethod 1300 is lower relative to other per label annotation agreements. In some embodiments, an expert annotator decides to replace a guideline to a label or task after viewing a paired guideline in a work unit interface from a second GUI and enters a new guideline in the create guideline data field presented on a second GUI such as the createguideline pane 845 as depicted inFIG. 8A . - In some embodiments,
method 1500 begins at 1510 by accessing a revised guideline through afirst GUI 212 orsecond GUI 214. In some embodiments, at 1520 the single guideline initially paired with the label or task of the work unit interface (such as at 1235 in FIG.12) is unpaired with the respective label or task. In some embodiments, at 1530, the revised guideline accessed at 1510 is then paired with the label or task that had the single guideline unpaired at 1520. At 1540, a second reference button is created for the newly paired revised guideline with the label or task, and at 1550 the second reference button is displayed on the work unit interface. In some embodiments, the revised guideline is included in addition to the single guideline, such that when an annotator interacting with the work unit interface presses the reference button, both the single guideline and the revised guideline are displayed to the annotator. - It should be appreciated that the specific steps illustrated in
FIGS. 12-15 provide a particular process and sequence of interaction among GUIs and the manipulation of the annotations generated on an ontology. Other sequences of steps may also be performed according to alternative embodiments. For example, alternative embodiments may perform the steps outlined above in a different order. Moreover, the individual sequence illustrated inFIGS. 12-15 may include multiple sub-sequences as appropriate to the individual step or direct sequences between different nodes than as illustrated. Furthermore, additional steps may be added or removed depending on the particular applications. One of skill in the art would recognize many variations, modifications, and alternatives. - Referring to
FIG. 16 , the block diagram illustrates components of amachine 1600, according to some example embodiments, able to readinstructions 1624 from a machine-readable medium 1622 (e.g., a non-transitory machine-readable medium, a machine-readable storage medium, a computer-readable storage medium, or any suitable combination thereof) and perform any one or more of the methodologies discussed herein, in whole or in part. Specifically,FIG. 16 shows themachine 1600 in the example form of a computer system (e.g., a computer) within which the instructions 1624 (e.g., software, a program, an application, an applet, an app, or other executable code) for causing themachine 1600 to perform any one or more of the methodologies discussed herein may be executed, in whole or in part. - In alternative embodiments, the
machine 1600 operates as a standalone device or may be connected (e.g., networked) to other machines. In a networked deployment, themachine 1600 may operate in the capacity of aserver machine 110 or a client machine in a server-client network environment, or as a peer machine in a distributed (e.g., peer-to-peer) network environment. Themachine 1600 may include hardware, software, or combinations thereof, and may, as example, be a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a cellular telephone, a smartphone, a set-top box (STB), a personal digital assistant (PDA), a web appliance, a network router, a network switch, a network bridge, or any machine capable of executing theinstructions 1624, sequentially or otherwise, that specify actions to be taken by that machine. Further, while only asingle machine 1600 is illustrated, the term “machine” shall also be taken to include any collection of machines that individually or jointly execute theinstructions 1624 to perform all or part of any one or more of the methodologies discussed herein. - The
machine 1600 includes a processor 1602 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a radio-frequency integrated circuit (RFIC), or any suitable combination thereof), amain memory 1604, and astatic memory 1606, which are configured to communicate with each other via abus 1608. Theprocessor 1602 may contain microcircuits that are configurable, temporarily or permanently, by some or all of theinstructions 1624 such that theprocessor 1602 is configurable to perform any one or more of the methodologies described herein, in whole or in part. For example, a set of one or more microcircuits of theprocessor 1602 may be configurable to execute one or more modules (e.g., software modules) described herein. - The
machine 1600 may further include an input and output module 1610 (e.g., a plasma display panel (PDP), a light emitting diode (LED) display, a liquid crystal display (LCD), a projector, a cathode ray tube (CRT), or any other display capable of displaying graphics or video) configured to display any one of the interfaces described herein. Themachine 1600 may also include an alphanumeric input device 1612 (e.g., a keyboard or keypad), a cursor control device 1614 (e.g., a mouse, a touchpad, a trackball, a joystick, a motion sensor, an eye tracking device, or other pointing instrument), astorage unit 1616, a signal generation device 1618 (e.g., a sound card, an amplifier, a speaker, a headphone jack, or any suitable combination thereof), and anetwork interface device 1620. - The
storage unit 1616 includes the machine-readable medium 1622 (e.g., a tangible and non-transitory machine-readable storage medium) on which are stored theinstructions 1624 embodying any one or more of the methodologies, functions, or interfaces described herein, including, for example, any of the descriptions ofFIGS. 1-15 . Theinstructions 1624 may also reside, completely or at least partially, within themain memory 1604, within the processor 1602 (e.g., within the processor's cache memory), or both, before or during execution thereof by themachine 1600. Theinstructions 1624 may also reside in thestatic memory 1606. - Accordingly, the
main memory 1604 and theprocessor 1602 may be considered machine-readable media 1622 (e.g., tangible and non-transitory machine-readable media). Theinstructions 1624 may be transmitted or received over anetwork 1626 via thenetwork interface device 1620. For example, thenetwork interface device 1620 may communicate theinstructions 1624 using any one or more transfer protocols (e.g., HTTP). Themachine 1600 may also represent example means for performing any of the functions described herein, including the processes described inFIGS. 1-15 . - In some example embodiments, the
machine 1600 may be a portable computing device, such as a smart phone or tablet computer, and have one or more additional input components (e.g., sensors or gauges) (not shown). Examples of such input components include an image input component (e.g., one or more cameras), an audio input component (e.g., a microphone), a direction input component (e.g., a compass), a location input component (e.g., a GPS receiver), an orientation component (e.g., a gyroscope), a motion detection component (e.g., one or more accelerometers), an altitude detection component (e.g., an altimeter), and a gas detection component (e.g., a gas sensor). Inputs harvested by any one or more of these input components may be accessible and available for use by any of the modules described herein. - As used herein, the term “memory” refers to a machine-readable medium 1622 able to store data temporarily or permanently and may be taken to include, but not be limited to, random-access memory (RAM), read-only memory (ROM), buffer memory, flash memory, and cache memory. While the machine-readable medium 1622 is shown in an example embodiment to be a single medium, the term “machine-readable medium” should be taken to include a single medium or multiple media (e.g., a centralized or distributed
database 115, or associated caches and servers) able to storeinstructions 1624. The term “machine-readable medium” shall also be taken to include any medium, or combination of multiple media, that is capable of storing theinstructions 1624 for execution by themachine 1600, such that theinstructions 1624, when executed by one or more processors of the machine 1600 (e.g., processor 1602), cause themachine 1600 to perform any one or more of the methodologies described herein, in whole or in part. Accordingly, a “machine-readable medium” refers to a single storage apparatus ordevice devices - Furthermore, the machine-readable medium 1622 is non-transitory in that it does not embody a propagating signal. However, labeling the tangible machine-readable medium 1622 as “non-transitory” should not be construed to mean that the medium is incapable of movement; the medium should be considered as being transportable from one physical location to another. Additionally, since the machine-readable medium 1622 is tangible, the medium may be considered to be a machine-readable device.
- Throughout this specification, plural instances may implement components, operations, or structures described as a single instance. Although individual operations of one or more methods are illustrated and described as separate operations, one or more of the individual operations may be performed concurrently, and nothing requires that the operations be performed in the order illustrated. Structures and functionality presented as separate components in example configurations may be implemented as a combined structure or component. Similarly, structures and functionality presented as a single component may be implemented as separate components. These and other variations, modifications, additions, and improvements fall within the scope of the subject matter herein.
- Certain embodiments are described herein as including logic or a number of components, modules, or mechanisms. Modules may constitute software modules (e.g., code stored or otherwise embodied on a machine-readable medium 1622 or in a transmission medium), hardware modules, or any suitable combination thereof. A “hardware module” is a tangible (e.g., non-transitory) unit capable of performing certain operations and may be configured or arranged in a certain physical manner. In various example embodiments, one or more computer systems (e.g., a standalone computer system, a client computer system, or a server computer system) or one or more hardware modules of a computer system (e.g., a
processor 1602 or a group of processors 1602) may be configured by software (e.g., an application or application portion) as a hardware module that operates to perform certain operations as described herein. - In some embodiments, a hardware module may be implemented mechanically, electronically, or any suitable combination thereof. For example, a hardware module may include dedicated circuitry or logic that is permanently configured to perform certain operations. For example, a hardware module may be a special-purpose processor, such as a field programmable gate array (FPGA) or an ASIC. A hardware module may also include programmable logic or circuitry that is temporarily configured by software to perform certain operations. For example, a hardware module may include software encompassed within a general-
purpose processor 1602 or otherprogrammable processor 1602. It will be appreciated that the decision to implement a hardware module mechanically, in dedicated and permanently configured circuitry, or in temporarily configured circuitry (e.g., configured by software) may be driven by cost and time considerations. - Hardware modules can provide information to, and receive information from, other hardware modules. Accordingly, the described hardware modules may be regarded as being communicatively coupled. Where multiple hardware modules exist contemporaneously, communications may be achieved through signal transmission (e.g., over appropriate circuits and buses 1608) between or among two or more of the hardware modules. In embodiments in which multiple hardware modules are configured or instantiated at different times, communications between such hardware modules may be achieved, for example, through the storage and retrieval of information in memory structures to which the multiple hardware modules have access. For example, one hardware module may perform an operation and store the output of that operation in a memory device to which it is communicatively coupled. A further hardware module may then, at a later time, access the memory device to retrieve and process the stored output. Hardware modules may also initiate communications with input or output devices, and can operate on a resource (e.g., a collection of information).
- The various operations of example methods described herein may be performed, at least partially, by one or
more processors 1602 that are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured,such processors 1602 may constitute processor-implemented modules that operate to perform one or more operations or functions described herein. As used herein, “processor-implemented module” refers to a hardware module implemented using one ormore processors 1602. - Similarly, the methods described herein may be at least partially processor-implemented, a
processor 1602 being an example of hardware. For example, at least some of the operations of a method may be performed by one ormore processors 1602 or processor-implemented modules. As used herein, “processor-implemented module” refers to a hardware module in which the hardware includes one ormore processors 1602. Moreover, the one ormore processors 1602 may also operate to support performance of the relevant operations in a “cloud computing” environment or as a “software as a service” (SaaS). For example, at least some of the operations may be performed by a group of computers (as examples ofmachines 1600 including processors 1602), with these operations being accessible via a network 1626 (e.g., the Internet) and via one or more appropriate interfaces (e.g., an application program interface or “API”). - The performance of certain operations may be distributed among the one or
more processors 1602, not only residing within asingle machine 1600, but deployed across a number ofmachines 1600. In some example embodiments, the one ormore processors 1602 or processor-implemented modules may be located in a single geographic location (e.g., within a home environment, an office environment, or a server farm). In other example embodiments, the one ormore processors 1602 or processor-implemented modules may be distributed across a number of geographic locations. - Unless specifically stated otherwise, discussions herein using words such as “processing,” “computing,” “calculating,” “determining,” “presenting,” “displaying,” or the like may refer to actions or processes of a natural language modeling engine 210 (e.g., a on a computing device or external server such as
server machine 110 depicted inFIG. 1 or as part of a system of interconnected interfaces as depicted inFIG. 2 ) that manipulates or transforms data represented as physical (e.g., electronic, magnetic, or optical) quantities within one or more memories (e.g., volatile memory, non-volatile memory, or any suitable combination thereof), registers, or other machine components that receive, store, transmit, or display information. Furthermore, unless specifically stated otherwise, the terms “a” or “an” are herein used, as is common in patent documents, to include one or more than one instance. Finally, as used herein, the conjunction “or” refers to a non-exclusive “or,” unless specifically stated otherwise. - The present disclosure is illustrative and not limiting. Further modifications will be apparent to one skilled in the art in light of this disclosure and are intended to fall within the scope of the appended claims.
Claims (20)
Priority Applications (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US14/964,522 US20160162458A1 (en) | 2014-12-09 | 2015-12-09 | Graphical systems and methods for human-in-the-loop machine intelligence |
US14/964,511 US9495345B2 (en) | 2014-12-09 | 2015-12-09 | Methods and systems for modeling complex taxonomies with natural language understanding |
US15/294,156 US20170235813A1 (en) | 2014-12-09 | 2016-10-14 | Methods and systems for modeling complex taxonomies with natural language understanding |
US17/182,178 US20210232763A1 (en) | 2014-12-09 | 2021-02-22 | Graphical systems and methods for human-in-the-loop machine intelligence |
Applications Claiming Priority (5)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201462089736P | 2014-12-09 | 2014-12-09 | |
US201462089745P | 2014-12-09 | 2014-12-09 | |
US201462089747P | 2014-12-09 | 2014-12-09 | |
US201462089742P | 2014-12-09 | 2014-12-09 | |
US14/964,522 US20160162458A1 (en) | 2014-12-09 | 2015-12-09 | Graphical systems and methods for human-in-the-loop machine intelligence |
Related Child Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/221,254 Continuation US11295071B2 (en) | 2014-12-09 | 2018-12-14 | Graphical systems and methods for human-in-the-loop machine intelligence |
Publications (1)
Publication Number | Publication Date |
---|---|
US20160162458A1 true US20160162458A1 (en) | 2016-06-09 |
Family
ID=56094482
Family Applications (20)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/964,522 Abandoned US20160162458A1 (en) | 2014-12-09 | 2015-12-09 | Graphical systems and methods for human-in-the-loop machine intelligence |
US14/964,517 Active US10127214B2 (en) | 2014-12-09 | 2015-12-09 | Methods for generating natural language processing systems |
US14/964,528 Abandoned US20160162464A1 (en) | 2014-12-09 | 2015-12-09 | Techniques for combining human and machine learning in natural language processing |
US14/964,511 Active - Reinstated US9495345B2 (en) | 2014-12-09 | 2015-12-09 | Methods and systems for modeling complex taxonomies with natural language understanding |
US14/964,510 Abandoned US20160162569A1 (en) | 2014-12-09 | 2015-12-09 | Methods and systems for improving machine learning performance |
US14/964,520 Abandoned US20160162457A1 (en) | 2014-12-09 | 2015-12-09 | Optimization techniques for artificial intelligence |
US15/294,156 Abandoned US20170235813A1 (en) | 2014-12-09 | 2016-10-14 | Methods and systems for modeling complex taxonomies with natural language understanding |
US16/125,343 Abandoned US20190243886A1 (en) | 2014-12-09 | 2018-09-07 | Methods and systems for improving machine learning performance |
US16/181,102 Abandoned US20190303428A1 (en) | 2014-12-09 | 2018-11-05 | Methods for generating natural language processing systems |
US16/185,843 Abandoned US20190311024A1 (en) | 2014-12-09 | 2018-11-09 | Techniques for combining human and machine learning in natural language processing |
US16/197,190 Abandoned US20190311025A1 (en) | 2014-12-09 | 2018-11-20 | Methods and systems for modeling complex taxonomies with natural language understanding |
US16/198,453 Abandoned US20200234002A1 (en) | 2014-12-09 | 2018-11-21 | Optimization techniques for artificial intelligence |
US16/289,481 Abandoned US20200034737A1 (en) | 2014-12-09 | 2019-02-28 | Architectures for natural language processing |
US16/749,689 Abandoned US20200184146A1 (en) | 2014-12-09 | 2020-01-22 | Techniques for combining human and machine learning in natural language processing |
US16/796,812 Abandoned US20210150130A1 (en) | 2014-12-09 | 2020-02-20 | Methods for generating natural language processing systems |
US16/811,737 Active 2036-04-16 US11599714B2 (en) | 2014-12-09 | 2020-03-06 | Methods and systems for modeling complex taxonomies with natural language understanding |
US17/119,902 Active US11288444B2 (en) | 2014-12-09 | 2020-12-11 | Optimization techniques for artificial intelligence |
US17/131,486 Abandoned US20210232761A1 (en) | 2014-12-09 | 2020-12-22 | Methods and systems for improving machine learning performance |
US17/166,493 Abandoned US20210232762A1 (en) | 2014-12-09 | 2021-02-03 | Architectures for natural language processing |
US17/182,178 Abandoned US20210232763A1 (en) | 2014-12-09 | 2021-02-22 | Graphical systems and methods for human-in-the-loop machine intelligence |
Family Applications After (19)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/964,517 Active US10127214B2 (en) | 2014-12-09 | 2015-12-09 | Methods for generating natural language processing systems |
US14/964,528 Abandoned US20160162464A1 (en) | 2014-12-09 | 2015-12-09 | Techniques for combining human and machine learning in natural language processing |
US14/964,511 Active - Reinstated US9495345B2 (en) | 2014-12-09 | 2015-12-09 | Methods and systems for modeling complex taxonomies with natural language understanding |
US14/964,510 Abandoned US20160162569A1 (en) | 2014-12-09 | 2015-12-09 | Methods and systems for improving machine learning performance |
US14/964,520 Abandoned US20160162457A1 (en) | 2014-12-09 | 2015-12-09 | Optimization techniques for artificial intelligence |
US15/294,156 Abandoned US20170235813A1 (en) | 2014-12-09 | 2016-10-14 | Methods and systems for modeling complex taxonomies with natural language understanding |
US16/125,343 Abandoned US20190243886A1 (en) | 2014-12-09 | 2018-09-07 | Methods and systems for improving machine learning performance |
US16/181,102 Abandoned US20190303428A1 (en) | 2014-12-09 | 2018-11-05 | Methods for generating natural language processing systems |
US16/185,843 Abandoned US20190311024A1 (en) | 2014-12-09 | 2018-11-09 | Techniques for combining human and machine learning in natural language processing |
US16/197,190 Abandoned US20190311025A1 (en) | 2014-12-09 | 2018-11-20 | Methods and systems for modeling complex taxonomies with natural language understanding |
US16/198,453 Abandoned US20200234002A1 (en) | 2014-12-09 | 2018-11-21 | Optimization techniques for artificial intelligence |
US16/289,481 Abandoned US20200034737A1 (en) | 2014-12-09 | 2019-02-28 | Architectures for natural language processing |
US16/749,689 Abandoned US20200184146A1 (en) | 2014-12-09 | 2020-01-22 | Techniques for combining human and machine learning in natural language processing |
US16/796,812 Abandoned US20210150130A1 (en) | 2014-12-09 | 2020-02-20 | Methods for generating natural language processing systems |
US16/811,737 Active 2036-04-16 US11599714B2 (en) | 2014-12-09 | 2020-03-06 | Methods and systems for modeling complex taxonomies with natural language understanding |
US17/119,902 Active US11288444B2 (en) | 2014-12-09 | 2020-12-11 | Optimization techniques for artificial intelligence |
US17/131,486 Abandoned US20210232761A1 (en) | 2014-12-09 | 2020-12-22 | Methods and systems for improving machine learning performance |
US17/166,493 Abandoned US20210232762A1 (en) | 2014-12-09 | 2021-02-03 | Architectures for natural language processing |
US17/182,178 Abandoned US20210232763A1 (en) | 2014-12-09 | 2021-02-22 | Graphical systems and methods for human-in-the-loop machine intelligence |
Country Status (1)
Country | Link |
---|---|
US (20) | US20160162458A1 (en) |
Cited By (29)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20170293859A1 (en) * | 2016-04-11 | 2017-10-12 | Yandex Europe Ag | Method for training a ranker module using a training set having noisy labels |
US20180173698A1 (en) * | 2016-12-16 | 2018-06-21 | Microsoft Technology Licensing, Llc | Knowledge Base for Analysis of Text |
US20180349454A1 (en) * | 2017-06-02 | 2018-12-06 | Thinkspan, LLC | Universal data scaffold based data management platform |
US10382300B2 (en) | 2015-10-06 | 2019-08-13 | Evolv Technologies, Inc. | Platform for gathering real-time analysis |
US10445379B2 (en) * | 2016-06-20 | 2019-10-15 | Yandex Europe Ag | Method of generating a training object for training a machine learning algorithm |
US10650086B1 (en) * | 2016-09-27 | 2020-05-12 | Palantir Technologies Inc. | Systems, methods, and framework for associating supporting data in word processing |
US20200152302A1 (en) * | 2017-10-20 | 2020-05-14 | Google Llc | Capturing Detailed Structure From Patient-Doctor Conversations for Use in Clinical Documentation |
US20200175416A1 (en) * | 2018-11-30 | 2020-06-04 | Jpmorgan Chase Bank, N.A. | Methods for sharing machine learning based web service models |
US20200250216A1 (en) * | 2019-02-04 | 2020-08-06 | International Business Machines Corporation | Generating a domain-specific phrasal dictionary |
WO2020185900A1 (en) * | 2019-03-11 | 2020-09-17 | Roam Analytics, Inc. | Methods, apparatus and systems for annotation of text documents |
US10783328B2 (en) * | 2018-06-04 | 2020-09-22 | International Business Machines Corporation | Semi-automatic process for creating a natural language processing resource |
US10817655B2 (en) | 2015-12-11 | 2020-10-27 | Palantir Technologies Inc. | Systems and methods for annotating and linking electronic documents |
US20210065233A1 (en) * | 2019-03-12 | 2021-03-04 | Sherloq, Inc. | Interface method and system for enabling an advertisement sponsor to input data concerning leads generated in response to advertisements |
US20210226985A1 (en) * | 2018-11-02 | 2021-07-22 | KnowBe4, Inc. | System and methods of cybersecurity attack simulation for incident response training and awareness |
US20210233008A1 (en) * | 2020-01-28 | 2021-07-29 | Schlumberger Technology Corporation | Oilfield data file classification and information processing systems |
US20210326718A1 (en) * | 2020-04-16 | 2021-10-21 | Microsoft Technology Licensing, Llc | Machine learning techniques to shape downstream content traffic through hashtag suggestion during content creation |
US11188517B2 (en) | 2019-08-09 | 2021-11-30 | International Business Machines Corporation | Annotation assessment and ground truth construction |
US20210390250A1 (en) * | 2020-06-15 | 2021-12-16 | Canon Kabushiki Kaisha | Information processing apparatus |
US11294884B2 (en) | 2019-08-09 | 2022-04-05 | International Business Machines Corporation | Annotation assessment and adjudication |
US20220276881A1 (en) * | 2018-07-23 | 2022-09-01 | Google Llc | Intelligent home screen of cloud-based content management platform |
US11442944B2 (en) | 2019-10-18 | 2022-09-13 | Thinkspan, LLC | Algorithmic suggestions based on a universal data scaffold |
US11443102B1 (en) * | 2021-08-13 | 2022-09-13 | Pricewaterhousecoopers Llp | Methods and systems for artificial intelligence-assisted document annotation |
US11468100B2 (en) | 2019-10-18 | 2022-10-11 | Thinkspan, LLC | Scalable scaffolding and bundled data |
US11507629B2 (en) | 2016-10-28 | 2022-11-22 | Parexel International, Llc | Dataset networking and database modeling |
US20230134796A1 (en) * | 2021-10-29 | 2023-05-04 | Glipped, Inc. | Named entity recognition system for sentiment labeling |
US11645464B2 (en) | 2021-03-18 | 2023-05-09 | International Business Machines Corporation | Transforming a lexicon that describes an information asset |
US11645462B2 (en) | 2021-08-13 | 2023-05-09 | Pricewaterhousecoopers Llp | Continuous machine learning method and system for information extraction |
US11657044B2 (en) | 2016-10-28 | 2023-05-23 | Parexel International, Llc | Semantic parsing engine |
US11880746B1 (en) * | 2017-04-26 | 2024-01-23 | Hrb Innovations, Inc. | Interface for artificial intelligence training |
Families Citing this family (283)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8677377B2 (en) | 2005-09-08 | 2014-03-18 | Apple Inc. | Method and apparatus for building an intelligent automated assistant |
US9318108B2 (en) | 2010-01-18 | 2016-04-19 | Apple Inc. | Intelligent automated assistant |
US8977255B2 (en) | 2007-04-03 | 2015-03-10 | Apple Inc. | Method and system for operating a multi-function portable electronic device using voice-activation |
US10002189B2 (en) | 2007-12-20 | 2018-06-19 | Apple Inc. | Method and apparatus for searching using an active ontology |
US9330720B2 (en) | 2008-01-03 | 2016-05-03 | Apple Inc. | Methods and apparatus for altering audio output signals |
US8996376B2 (en) | 2008-04-05 | 2015-03-31 | Apple Inc. | Intelligent text-to-speech conversion |
US20100030549A1 (en) | 2008-07-31 | 2010-02-04 | Lee Michael M | Mobile device having human language translation capability with positional feedback |
US8676904B2 (en) | 2008-10-02 | 2014-03-18 | Apple Inc. | Electronic devices with voice command and contextual data processing capabilities |
US20120309363A1 (en) | 2011-06-03 | 2012-12-06 | Apple Inc. | Triggering notifications associated with tasks items that represent tasks to perform |
US10241752B2 (en) | 2011-09-30 | 2019-03-26 | Apple Inc. | Interface for a virtual digital assistant |
US10276170B2 (en) | 2010-01-18 | 2019-04-30 | Apple Inc. | Intelligent automated assistant |
US8682667B2 (en) | 2010-02-25 | 2014-03-25 | Apple Inc. | User profiling for selecting user specific voice input processing information |
US9262612B2 (en) | 2011-03-21 | 2016-02-16 | Apple Inc. | Device access using voice authentication |
US10057736B2 (en) | 2011-06-03 | 2018-08-21 | Apple Inc. | Active transport based notifications |
US10134385B2 (en) | 2012-03-02 | 2018-11-20 | Apple Inc. | Systems and methods for name pronunciation |
US10417037B2 (en) | 2012-05-15 | 2019-09-17 | Apple Inc. | Systems and methods for integrating third party services with a digital assistant |
US9721563B2 (en) | 2012-06-08 | 2017-08-01 | Apple Inc. | Name recognition system |
US10430506B2 (en) | 2012-12-10 | 2019-10-01 | International Business Machines Corporation | Utilizing classification and text analytics for annotating documents to allow quick scanning |
CN113470640B (en) | 2013-02-07 | 2022-04-26 | 苹果公司 | Voice trigger of digital assistant |
US10652394B2 (en) | 2013-03-14 | 2020-05-12 | Apple Inc. | System and method for processing voicemail |
US10748529B1 (en) | 2013-03-15 | 2020-08-18 | Apple Inc. | Voice activated device for use with a voice-based digital assistant |
WO2014197334A2 (en) | 2013-06-07 | 2014-12-11 | Apple Inc. | System and method for user-specified pronunciation of words for speech synthesis and recognition |
WO2014197335A1 (en) | 2013-06-08 | 2014-12-11 | Apple Inc. | Interpreting and acting upon commands that involve sharing information with remote devices |
WO2014200728A1 (en) | 2013-06-09 | 2014-12-18 | Apple Inc. | Device, method, and graphical user interface for enabling conversation persistence across two or more instances of a digital assistant |
US10176167B2 (en) | 2013-06-09 | 2019-01-08 | Apple Inc. | System and method for inferring user intent from speech inputs |
WO2015020942A1 (en) | 2013-08-06 | 2015-02-12 | Apple Inc. | Auto-activating smart responses based on activities from remote devices |
US10296160B2 (en) | 2013-12-06 | 2019-05-21 | Apple Inc. | Method for extracting salient dialog usage from live data |
US10170123B2 (en) | 2014-05-30 | 2019-01-01 | Apple Inc. | Intelligent assistant for home automation |
US9430463B2 (en) | 2014-05-30 | 2016-08-30 | Apple Inc. | Exemplar-based natural language processing |
US9715875B2 (en) | 2014-05-30 | 2017-07-25 | Apple Inc. | Reducing the need for manual start/end-pointing and trigger phrases |
US9633004B2 (en) | 2014-05-30 | 2017-04-25 | Apple Inc. | Better resolution when referencing to concepts |
EP3480811A1 (en) | 2014-05-30 | 2019-05-08 | Apple Inc. | Multi-command single utterance input method |
US9338493B2 (en) | 2014-06-30 | 2016-05-10 | Apple Inc. | Intelligent automated assistant for TV user interactions |
US9818400B2 (en) | 2014-09-11 | 2017-11-14 | Apple Inc. | Method and apparatus for discovering trending terms in speech requests |
US10127911B2 (en) | 2014-09-30 | 2018-11-13 | Apple Inc. | Speaker identification and unsupervised speaker adaptation techniques |
US10074360B2 (en) | 2014-09-30 | 2018-09-11 | Apple Inc. | Providing an indication of the suitability of speech recognition |
US9668121B2 (en) | 2014-09-30 | 2017-05-30 | Apple Inc. | Social reminders |
US10169826B1 (en) | 2014-10-31 | 2019-01-01 | Intuit Inc. | System and method for generating explanations for tax calculations |
CN113368656B (en) | 2014-11-20 | 2024-03-19 | 代表亚利桑那大学的亚利桑那校董事会 | System and method for generating liquid water from air |
US10387970B1 (en) | 2014-11-25 | 2019-08-20 | Intuit Inc. | Systems and methods for analyzing and generating explanations for changes in tax return results |
US20160162458A1 (en) * | 2014-12-09 | 2016-06-09 | Idibon, Inc. | Graphical systems and methods for human-in-the-loop machine intelligence |
US10152299B2 (en) | 2015-03-06 | 2018-12-11 | Apple Inc. | Reducing response latency of intelligent automated assistants |
US9721566B2 (en) | 2015-03-08 | 2017-08-01 | Apple Inc. | Competing devices responding to voice triggers |
US9886953B2 (en) | 2015-03-08 | 2018-02-06 | Apple Inc. | Virtual assistant activation |
US10567477B2 (en) | 2015-03-08 | 2020-02-18 | Apple Inc. | Virtual assistant continuity |
US10872384B1 (en) | 2015-03-30 | 2020-12-22 | Intuit Inc. | System and method for generating explanations for year-over-year tax changes |
US10460227B2 (en) | 2015-05-15 | 2019-10-29 | Apple Inc. | Virtual assistant in a communication session |
US10200824B2 (en) | 2015-05-27 | 2019-02-05 | Apple Inc. | Systems and methods for proactively identifying and surfacing relevant content on a touch-sensitive device |
US10083688B2 (en) | 2015-05-27 | 2018-09-25 | Apple Inc. | Device voice control for selecting a displayed affordance |
US10318591B2 (en) | 2015-06-02 | 2019-06-11 | International Business Machines Corporation | Ingesting documents using multiple ingestion pipelines |
US9578173B2 (en) | 2015-06-05 | 2017-02-21 | Apple Inc. | Virtual assistant aided communication with 3rd party service in a communication session |
US11025565B2 (en) | 2015-06-07 | 2021-06-01 | Apple Inc. | Personalized prediction of responses for instant messaging |
US10997134B2 (en) * | 2015-06-18 | 2021-05-04 | Aware, Inc. | Automatic entity resolution with rules detection and generation system |
US20160378747A1 (en) | 2015-06-29 | 2016-12-29 | Apple Inc. | Virtual assistant for media playback |
US10607298B1 (en) | 2015-07-30 | 2020-03-31 | Intuit Inc. | System and method for indicating sections of electronic tax forms for which narrative explanations can be presented |
US10140983B2 (en) * | 2015-08-28 | 2018-11-27 | International Business Machines Corporation | Building of n-gram language model for automatic speech recognition (ASR) |
US10740384B2 (en) | 2015-09-08 | 2020-08-11 | Apple Inc. | Intelligent automated assistant for media search and playback |
US10331312B2 (en) | 2015-09-08 | 2019-06-25 | Apple Inc. | Intelligent automated assistant in a media environment |
US10747498B2 (en) | 2015-09-08 | 2020-08-18 | Apple Inc. | Zero latency digital assistant |
US10671428B2 (en) | 2015-09-08 | 2020-06-02 | Apple Inc. | Distributed personal assistant |
US20180241848A1 (en) * | 2015-09-11 | 2018-08-23 | Hewlett Packard Enterprise Development Lp | Human-readable cloud structures |
US10691473B2 (en) | 2015-11-06 | 2020-06-23 | Apple Inc. | Intelligent automated assistant in a messaging environment |
US10956666B2 (en) | 2015-11-09 | 2021-03-23 | Apple Inc. | Unconventional virtual assistant interactions |
US10102201B2 (en) * | 2015-11-30 | 2018-10-16 | Soundhound, Inc. | Natural language module store |
US10049668B2 (en) | 2015-12-02 | 2018-08-14 | Apple Inc. | Applying neural network language models to weighted finite state transducers for automatic speech recognition |
US10346546B2 (en) * | 2015-12-23 | 2019-07-09 | Oath Inc. | Method and system for automatic formality transformation |
US10740573B2 (en) | 2015-12-23 | 2020-08-11 | Oath Inc. | Method and system for automatic formality classification |
US10223066B2 (en) | 2015-12-23 | 2019-03-05 | Apple Inc. | Proactive assistance based on dialog communication between devices |
US10572524B2 (en) * | 2016-02-29 | 2020-02-25 | Microsoft Technology Licensing, Llc | Content categorization |
US9747143B1 (en) * | 2016-03-30 | 2017-08-29 | International Business Machines Corporation | Multi platform based event processing |
TWI718284B (en) | 2016-04-07 | 2021-02-11 | 美商零質量純水股份有限公司 | Solar thermal unit |
RU2628431C1 (en) * | 2016-04-12 | 2017-08-16 | Общество с ограниченной ответственностью "Аби Продакшн" | Selection of text classifier parameter based on semantic characteristics |
GB201608576D0 (en) * | 2016-05-16 | 2016-06-29 | Pro Intex It Ltd | Functional behaviour test system and method |
TWI817925B (en) | 2016-05-20 | 2023-10-11 | 美商資源環球公司 | Systems and methods for water extraction control and water generation network |
US11227589B2 (en) | 2016-06-06 | 2022-01-18 | Apple Inc. | Intelligent list reading |
US10049663B2 (en) | 2016-06-08 | 2018-08-14 | Apple, Inc. | Intelligent automated assistant for media exploration |
US10586535B2 (en) | 2016-06-10 | 2020-03-10 | Apple Inc. | Intelligent digital assistant in a multi-tasking environment |
DK179049B1 (en) | 2016-06-11 | 2017-09-18 | Apple Inc | Data driven natural language event detection and classification |
DK179415B1 (en) | 2016-06-11 | 2018-06-14 | Apple Inc | Intelligent device arbitration and control |
DK201670540A1 (en) | 2016-06-11 | 2018-01-08 | Apple Inc | Application integration with a digital assistant |
US10635704B2 (en) * | 2016-07-15 | 2020-04-28 | Cisco Technology, Inc. | Automatic ontology generation for internet of things applications |
US10769592B1 (en) | 2016-07-27 | 2020-09-08 | Intuit Inc. | Methods, systems and computer program products for generating explanations for a benefit qualification change |
US11055794B1 (en) | 2016-07-27 | 2021-07-06 | Intuit Inc. | Methods, systems and computer program products for estimating likelihood of qualifying for benefit |
US10872315B1 (en) | 2016-07-27 | 2020-12-22 | Intuit Inc. | Methods, systems and computer program products for prioritization of benefit qualification questions |
US10762472B1 (en) | 2016-07-27 | 2020-09-01 | Intuit Inc. | Methods, systems and computer program products for generating notifications of benefit qualification change |
US10579941B2 (en) * | 2016-09-01 | 2020-03-03 | Facebook, Inc. | Systems and methods for recommending pages |
US10474753B2 (en) | 2016-09-07 | 2019-11-12 | Apple Inc. | Language identification using recurrent neural networks |
US10043516B2 (en) | 2016-09-23 | 2018-08-07 | Apple Inc. | Intelligent automated assistant |
US12020334B2 (en) * | 2016-10-26 | 2024-06-25 | Intuit Inc. | Methods, systems and computer program products for generating and presenting explanations for tax questions |
US10963789B2 (en) | 2016-11-28 | 2021-03-30 | Conduent Business Services, Llc | Long-term memory networks for knowledge extraction from text and publications |
WO2018101694A1 (en) | 2016-11-29 | 2018-06-07 | Samsung Electronics Co., Ltd. | Electronic apparatus and method for summarizing content |
US11281993B2 (en) | 2016-12-05 | 2022-03-22 | Apple Inc. | Model and ensemble compression for metric learning |
US11373752B2 (en) * | 2016-12-22 | 2022-06-28 | Palantir Technologies Inc. | Detection of misuse of a benefit system |
US10593346B2 (en) | 2016-12-22 | 2020-03-17 | Apple Inc. | Rank-reduced token representation for automatic speech recognition |
US11204787B2 (en) | 2017-01-09 | 2021-12-21 | Apple Inc. | Application integration with a digital assistant |
US11954098B1 (en) * | 2017-02-03 | 2024-04-09 | Thomson Reuters Enterprise Centre Gmbh | Natural language processing system and method for documents |
US10298757B2 (en) * | 2017-02-23 | 2019-05-21 | Accenture Global Solutions Limited | Integrated service centre support |
US11514058B2 (en) | 2017-03-17 | 2022-11-29 | Microsoft Technology Licensing, Llc | Context rules for a graph |
US20180268004A1 (en) * | 2017-03-17 | 2018-09-20 | Microsoft Technology Licensing, Llc | Rule hierarchies for graph adaptation |
US11270686B2 (en) | 2017-03-28 | 2022-03-08 | International Business Machines Corporation | Deep language and acoustic modeling convergence and cross training |
US10679009B2 (en) * | 2017-04-21 | 2020-06-09 | Tata Consultancy Services Limited | System and method for belief based human-bot conversation |
US10963501B1 (en) * | 2017-04-29 | 2021-03-30 | Veritas Technologies Llc | Systems and methods for generating a topic tree for digital information |
DK201770383A1 (en) | 2017-05-09 | 2018-12-14 | Apple Inc. | User interface for correcting recognition errors |
US10417266B2 (en) | 2017-05-09 | 2019-09-17 | Apple Inc. | Context-aware ranking of intelligent response suggestions |
DK180048B1 (en) | 2017-05-11 | 2020-02-04 | Apple Inc. | MAINTAINING THE DATA PROTECTION OF PERSONAL INFORMATION |
US10395654B2 (en) | 2017-05-11 | 2019-08-27 | Apple Inc. | Text normalization based on a data-driven learning network |
US10726832B2 (en) | 2017-05-11 | 2020-07-28 | Apple Inc. | Maintaining privacy of personal information |
DK201770439A1 (en) | 2017-05-11 | 2018-12-13 | Apple Inc. | Offline personal assistant |
US11301477B2 (en) | 2017-05-12 | 2022-04-12 | Apple Inc. | Feedback analysis of a digital assistant |
DK179496B1 (en) | 2017-05-12 | 2019-01-15 | Apple Inc. | USER-SPECIFIC Acoustic Models |
DK201770428A1 (en) | 2017-05-12 | 2019-02-18 | Apple Inc. | Low-latency intelligent automated assistant |
DK179745B1 (en) | 2017-05-12 | 2019-05-01 | Apple Inc. | SYNCHRONIZATION AND TASK DELEGATION OF A DIGITAL ASSISTANT |
DK201770411A1 (en) | 2017-05-15 | 2018-12-20 | Apple Inc. | Multi-modal interfaces |
DK201770431A1 (en) | 2017-05-15 | 2018-12-20 | Apple Inc. | Optimizing dialogue policy decisions for digital assistants using implicit feedback |
DK201770432A1 (en) | 2017-05-15 | 2018-12-21 | Apple Inc. | Hierarchical belief states for digital assistants |
US20180336275A1 (en) | 2017-05-16 | 2018-11-22 | Apple Inc. | Intelligent automated assistant for media exploration |
US10403278B2 (en) | 2017-05-16 | 2019-09-03 | Apple Inc. | Methods and systems for phonetic matching in digital assistant services |
US10311144B2 (en) | 2017-05-16 | 2019-06-04 | Apple Inc. | Emoji word sense disambiguation |
DK179560B1 (en) | 2017-05-16 | 2019-02-18 | Apple Inc. | Far-field extension for digital assistant services |
US20180336892A1 (en) | 2017-05-16 | 2018-11-22 | Apple Inc. | Detecting a trigger of a digital assistant |
US11340925B2 (en) | 2017-05-18 | 2022-05-24 | Peloton Interactive Inc. | Action recipes for a crowdsourced digital assistant system |
US11520610B2 (en) | 2017-05-18 | 2022-12-06 | Peloton Interactive Inc. | Crowdsourced on-boarding of digital assistant operations |
US10466963B2 (en) | 2017-05-18 | 2019-11-05 | Aiqudo, Inc. | Connecting multiple mobile devices to a smart home assistant account |
US11043206B2 (en) | 2017-05-18 | 2021-06-22 | Aiqudo, Inc. | Systems and methods for crowdsourced actions and commands |
US11056105B2 (en) | 2017-05-18 | 2021-07-06 | Aiqudo, Inc | Talk back from actions in applications |
US10657328B2 (en) | 2017-06-02 | 2020-05-19 | Apple Inc. | Multi-task recurrent neural network architecture for efficient morphology handling in neural language modeling |
MX2020000464A (en) | 2017-07-14 | 2021-01-08 | Zero Mass Water Inc | Systems for controlled treatment of water with ozone and related methods therefor. |
CN109272003A (en) * | 2017-07-17 | 2019-01-25 | 华东师范大学 | A kind of method and apparatus for eliminating unknown error in deep learning model |
US11139048B2 (en) | 2017-07-18 | 2021-10-05 | Analytics For Life Inc. | Discovering novel features to use in machine learning techniques, such as machine learning techniques for diagnosing medical conditions |
US11062792B2 (en) | 2017-07-18 | 2021-07-13 | Analytics For Life Inc. | Discovering genomes to use in machine learning techniques |
CN110019770A (en) * | 2017-07-24 | 2019-07-16 | 华为技术有限公司 | The method and apparatus of train classification models |
US11531998B2 (en) * | 2017-08-30 | 2022-12-20 | Qualtrics, Llc | Providing a conversational digital survey by generating digital survey questions based on digital survey responses |
MX2020002481A (en) | 2017-09-05 | 2021-02-15 | Zero Mass Water Inc | Systems and methods to produce liquid water extracted from air. |
WO2019050866A1 (en) | 2017-09-05 | 2019-03-14 | Zero Mass Water, Inc. | Systems and methods for managing production and distribution of liquid water extracted from air |
US10565444B2 (en) | 2017-09-07 | 2020-02-18 | International Business Machines Corporation | Using visual features to identify document sections |
US10445429B2 (en) | 2017-09-21 | 2019-10-15 | Apple Inc. | Natural language understanding using vocabularies with compressed serialized tries |
US10755051B2 (en) | 2017-09-29 | 2020-08-25 | Apple Inc. | Rule-based natural language processing |
US20190102697A1 (en) * | 2017-10-02 | 2019-04-04 | International Business Machines Corporation | Creating machine learning models from structured intelligence databases |
US11555421B2 (en) | 2017-10-06 | 2023-01-17 | Source Global, PBC | Systems for generating water with waste heat and related methods therefor |
CN107832305A (en) * | 2017-11-28 | 2018-03-23 | 百度在线网络技术(北京)有限公司 | Method and apparatus for generating information |
US10636424B2 (en) | 2017-11-30 | 2020-04-28 | Apple Inc. | Multi-turn canned dialog |
SG11202005334RA (en) | 2017-12-06 | 2020-07-29 | Zero Mass Water Inc | Systems for constructing hierarchical training data sets for use with machine-learning and related methods therefor |
US20190179883A1 (en) * | 2017-12-08 | 2019-06-13 | International Business Machines Corporation | Evaluating textual annotation model performance |
US10979457B2 (en) | 2017-12-20 | 2021-04-13 | Check Point Public Cloud Security Ltd | Cloud security assessment system using near-natural language compliance rules |
US10803108B2 (en) * | 2017-12-20 | 2020-10-13 | International Business Machines Corporation | Facilitation of domain and client-specific application program interface recommendations |
US10942955B2 (en) * | 2017-12-21 | 2021-03-09 | Shanghai Xiaoi Robot Technology Co., Ltd. | Questioning and answering method, method for generating questioning and answering system, and method for modifying questioning and answering system |
US10942954B2 (en) * | 2017-12-22 | 2021-03-09 | International Business Machines Corporation | Dataset adaptation for high-performance in specific natural language processing tasks |
US20190197433A1 (en) * | 2017-12-22 | 2019-06-27 | Wipro Limited | Methods for adaptive information extraction through adaptive learning of human annotators and devices thereof |
US10599783B2 (en) * | 2017-12-26 | 2020-03-24 | International Business Machines Corporation | Automatically suggesting a temporal opportunity for and assisting a writer in writing one or more sequel articles via artificial intelligence |
US11768852B2 (en) * | 2017-12-27 | 2023-09-26 | Marlabs Incorporated | System and method for data analysis and presentation of data |
US10963495B2 (en) | 2017-12-29 | 2021-03-30 | Aiqudo, Inc. | Automated discourse phrase discovery for generating an improved language model of a digital assistant |
US10929613B2 (en) | 2017-12-29 | 2021-02-23 | Aiqudo, Inc. | Automated document cluster merging for topic-based digital assistant interpretation |
US10963499B2 (en) * | 2017-12-29 | 2021-03-30 | Aiqudo, Inc. | Generating command-specific language model discourses for digital assistant interpretation |
US10733982B2 (en) | 2018-01-08 | 2020-08-04 | Apple Inc. | Multi-directional dialog |
US10733375B2 (en) | 2018-01-31 | 2020-08-04 | Apple Inc. | Knowledge-based framework for improving natural language understanding |
MX2020008596A (en) | 2018-02-18 | 2020-12-11 | Zero Mass Water Inc | Systems for generating water for a container farm and related methods therefor. |
WO2019164981A1 (en) | 2018-02-20 | 2019-08-29 | Pearson Education, Inc. | Systems and methods for automated evaluation model customization |
US11449762B2 (en) | 2018-02-20 | 2022-09-20 | Pearson Education, Inc. | Real time development of auto scoring essay models for custom created prompts |
US10789959B2 (en) | 2018-03-02 | 2020-09-29 | Apple Inc. | Training speaker recognition models for digital assistants |
US10579737B2 (en) * | 2018-03-06 | 2020-03-03 | Adobe Inc. | Natural language image editing annotation framework |
US10963972B1 (en) * | 2018-03-06 | 2021-03-30 | Wells Fargo Bank, N.A. | Adaptive life advisor system |
US10685655B2 (en) | 2018-03-07 | 2020-06-16 | International Business Machines Corporation | Leveraging natural language processing |
US10592604B2 (en) | 2018-03-12 | 2020-03-17 | Apple Inc. | Inverse text normalization for automatic speech recognition |
US11474836B2 (en) | 2018-03-13 | 2022-10-18 | Microsoft Technology Licensing, Llc | Natural language to API conversion |
US10747560B2 (en) * | 2018-03-20 | 2020-08-18 | Microsoft Technology Licensing, Llc | Computerized task guidance across devices and applications |
RU2691855C1 (en) * | 2018-03-23 | 2019-06-18 | Общество с ограниченной ответственностью "Аби Продакшн" | Training classifiers used to extract information from natural language texts |
US10818288B2 (en) | 2018-03-26 | 2020-10-27 | Apple Inc. | Natural assistant interaction |
US10909331B2 (en) | 2018-03-30 | 2021-02-02 | Apple Inc. | Implicit identification of translation payload with neural machine translation |
US20190332789A1 (en) * | 2018-04-27 | 2019-10-31 | Microsoft Technology Licensing, Llc | Hierarchical access rights and role based access |
US10928918B2 (en) | 2018-05-07 | 2021-02-23 | Apple Inc. | Raise to speak |
US11145294B2 (en) | 2018-05-07 | 2021-10-12 | Apple Inc. | Intelligent automated assistant for delivering content from user experiences |
WO2019217355A1 (en) * | 2018-05-08 | 2019-11-14 | Mmodal Ip Llc | Hybrid batch and live natural language processing |
US11607644B2 (en) | 2018-05-11 | 2023-03-21 | Source Global, PBC | Systems for generating water using exogenously generated heat, exogenously generated electricity, and exhaust process fluids and related methods therefor |
US10951482B2 (en) | 2018-05-16 | 2021-03-16 | Microsoft Technology Licensing, Llc | Device identification on a building automation control network |
US11281995B2 (en) * | 2018-05-21 | 2022-03-22 | International Business Machines Corporation | Finding optimal surface for hierarchical classification task on an ontology |
US10984780B2 (en) | 2018-05-21 | 2021-04-20 | Apple Inc. | Global semantic word embeddings using bi-directional recurrent neural networks |
US11456915B2 (en) | 2018-05-21 | 2022-09-27 | Microsoft Technology Licensing, Llc | Device model templates |
US10892996B2 (en) | 2018-06-01 | 2021-01-12 | Apple Inc. | Variable latency device coordination |
DK201870355A1 (en) | 2018-06-01 | 2019-12-16 | Apple Inc. | Virtual assistant operation in multi-device environments |
DK179822B1 (en) | 2018-06-01 | 2019-07-12 | Apple Inc. | Voice interaction at a primary device to access call functionality of a companion device |
US11386266B2 (en) | 2018-06-01 | 2022-07-12 | Apple Inc. | Text correction |
DK180639B1 (en) | 2018-06-01 | 2021-11-04 | Apple Inc | DISABILITY OF ATTENTION-ATTENTIVE VIRTUAL ASSISTANT |
US10944859B2 (en) | 2018-06-03 | 2021-03-09 | Apple Inc. | Accelerated task performance |
US10629205B2 (en) * | 2018-06-12 | 2020-04-21 | International Business Machines Corporation | Identifying an accurate transcription from probabilistic inputs |
US11232255B2 (en) * | 2018-06-13 | 2022-01-25 | Adobe Inc. | Generating digital annotations for evaluating and training automatic electronic document annotation models |
CN108733359B (en) * | 2018-06-14 | 2020-12-25 | 北京航空航天大学 | Automatic generation method of software program |
US20190385711A1 (en) * | 2018-06-19 | 2019-12-19 | Ellipsis Health, Inc. | Systems and methods for mental health assessment |
US10664472B2 (en) * | 2018-06-27 | 2020-05-26 | Bitdefender IPR Management Ltd. | Systems and methods for translating natural language sentences into database queries |
US11456082B2 (en) * | 2018-07-03 | 2022-09-27 | International Business Machines Corporation | Patient engagement communicative strategy recommendation |
US10839618B2 (en) | 2018-07-12 | 2020-11-17 | Honda Motor Co., Ltd. | Applied artificial intelligence for natural language processing automotive reporting system |
US11399030B2 (en) * | 2018-07-23 | 2022-07-26 | Kyndryl, Inc. | Ontology based control of access to resources in a computing system |
US11636333B2 (en) * | 2018-07-26 | 2023-04-25 | Tesla, Inc. | Optimizing neural network structures for embedded systems |
US11144581B2 (en) * | 2018-07-26 | 2021-10-12 | International Business Machines Corporation | Verifying and correcting training data for text classification |
US11392794B2 (en) * | 2018-09-10 | 2022-07-19 | Ca, Inc. | Amplification of initial training data |
US11183195B2 (en) * | 2018-09-27 | 2021-11-23 | Snackable Inc. | Audio content processing systems and methods |
US11010561B2 (en) | 2018-09-27 | 2021-05-18 | Apple Inc. | Sentiment prediction from textual data |
US10839159B2 (en) | 2018-09-28 | 2020-11-17 | Apple Inc. | Named entity normalization in a spoken dialog system |
US11170166B2 (en) | 2018-09-28 | 2021-11-09 | Apple Inc. | Neural typographical error modeling via generative adversarial networks |
US11462215B2 (en) | 2018-09-28 | 2022-10-04 | Apple Inc. | Multi-modal inputs for voice commands |
CN110969330A (en) * | 2018-09-30 | 2020-04-07 | 曹西军 | Enterprise competitive barrier assessment method and system |
WO2020082038A1 (en) | 2018-10-19 | 2020-04-23 | Zero Mass Water, Inc. | Systems and methods for generating liquid water using highly efficient techniques that optimize production |
US20200124566A1 (en) | 2018-10-22 | 2020-04-23 | Zero Mass Water, Inc. | Systems and methods for detecting and measuring oxidizing compounds in test fluids |
US10943185B1 (en) | 2018-10-23 | 2021-03-09 | Bank Of America Corporation | Supervised machine-learning training platform with isolated update testing and bi-directional update propogation |
US11475898B2 (en) | 2018-10-26 | 2022-10-18 | Apple Inc. | Low-latency multi-speaker speech recognition |
US10341491B1 (en) * | 2018-11-26 | 2019-07-02 | Capital One Services, Llc | Identifying unreported issues through customer service interactions and website analytics |
US10956671B2 (en) | 2018-11-30 | 2021-03-23 | International Business Machines Corporation | Supervised machine learning models of documents |
US11216892B1 (en) * | 2018-12-06 | 2022-01-04 | Meta Platforms, Inc. | Classifying and upgrading a content item to a life event item |
US11093708B2 (en) * | 2018-12-13 | 2021-08-17 | Software Ag | Adaptive human to machine interaction using machine learning |
EP3906508B1 (en) * | 2018-12-31 | 2024-03-13 | Intel Corporation | Securing systems employing artificial intelligence |
US11638059B2 (en) | 2019-01-04 | 2023-04-25 | Apple Inc. | Content playback on multiple devices |
US11676043B2 (en) | 2019-03-04 | 2023-06-13 | International Business Machines Corporation | Optimizing hierarchical classification with adaptive node collapses |
US11157702B2 (en) * | 2019-03-06 | 2021-10-26 | International Business Machines Corporation | Utilizing varying coordinates related to a target event to provide contextual outputs |
US11645110B2 (en) * | 2019-03-13 | 2023-05-09 | International Business Machines Corporation | Intelligent generation and organization of user manuals |
US11348573B2 (en) | 2019-03-18 | 2022-05-31 | Apple Inc. | Multimodality in digital assistant systems |
US11715030B2 (en) | 2019-03-29 | 2023-08-01 | Red Hat, Inc. | Automatic object optimization to accelerate machine learning training |
US20220188517A1 (en) | 2019-04-03 | 2022-06-16 | Singularity Systems Inc. | Hierarchical machine learning architecture including master engine supported by distributed light-weight real-time edge engines |
US20220180066A1 (en) * | 2019-04-04 | 2022-06-09 | Singularity Systems Inc. | Machine learning processing pipeline optimization |
US10614345B1 (en) | 2019-04-12 | 2020-04-07 | Ernst & Young U.S. Llp | Machine learning based extraction of partition objects from electronic documents |
MX2021012655A (en) | 2019-04-22 | 2021-11-12 | Source Global Pbc | Water vapor adsorption air drying system and method for generating liquid water from air. |
US11392757B2 (en) | 2019-04-26 | 2022-07-19 | Figure Eight Technologies, Inc. | Management of annotation jobs |
US11475884B2 (en) | 2019-05-06 | 2022-10-18 | Apple Inc. | Reducing digital assistant latency when a language is incorrectly determined |
DK201970509A1 (en) | 2019-05-06 | 2021-01-15 | Apple Inc | Spoken notifications |
US11423908B2 (en) | 2019-05-06 | 2022-08-23 | Apple Inc. | Interpreting spoken requests |
US11307752B2 (en) | 2019-05-06 | 2022-04-19 | Apple Inc. | User configurable task triggers |
US11748613B2 (en) * | 2019-05-10 | 2023-09-05 | Baidu Usa Llc | Systems and methods for large scale semantic indexing with deep level-wise extreme multi-label learning |
US11140099B2 (en) | 2019-05-21 | 2021-10-05 | Apple Inc. | Providing message response suggestions |
US11277453B2 (en) * | 2019-05-24 | 2022-03-15 | International Business Machines Corporation | Media communication management |
US11496600B2 (en) | 2019-05-31 | 2022-11-08 | Apple Inc. | Remote execution of machine-learned models |
US11289073B2 (en) | 2019-05-31 | 2022-03-29 | Apple Inc. | Device text to speech |
DK180129B1 (en) | 2019-05-31 | 2020-06-02 | Apple Inc. | User activity shortcut suggestions |
DK201970511A1 (en) | 2019-05-31 | 2021-02-15 | Apple Inc | Voice identification in digital assistant systems |
US11360641B2 (en) | 2019-06-01 | 2022-06-14 | Apple Inc. | Increasing the relevance of new available information |
US11468890B2 (en) | 2019-06-01 | 2022-10-11 | Apple Inc. | Methods and user interfaces for voice-based control of electronic devices |
US11113518B2 (en) | 2019-06-28 | 2021-09-07 | Eygs Llp | Apparatus and methods for extracting data from lineless tables using Delaunay triangulation and excess edge removal |
US11531703B2 (en) | 2019-06-28 | 2022-12-20 | Capital One Services, Llc | Determining data categorizations based on an ontology and a machine-learning model |
US10489454B1 (en) * | 2019-06-28 | 2019-11-26 | Capital One Services, Llc | Indexing a dataset based on dataset tags and an ontology |
US11354504B2 (en) * | 2019-07-10 | 2022-06-07 | International Business Machines Corporation | Multi-lingual action identification |
US20210026897A1 (en) * | 2019-07-23 | 2021-01-28 | Microsoft Technology Licensing, Llc | Topical clustering and notifications for driving resource collaboration |
US11521019B2 (en) | 2019-08-06 | 2022-12-06 | Bank Of America Corporation | Systems and methods for incremental learning and autonomous model reconfiguration in regulated AI systems |
US11915465B2 (en) | 2019-08-21 | 2024-02-27 | Eygs Llp | Apparatus and methods for converting lineless tables into lined tables using generative adversarial networks |
US12026630B2 (en) | 2019-09-23 | 2024-07-02 | Adaptai Ltd. | Artificial intelligence methods and systems for improving user adaptability using textual communication |
WO2021056255A1 (en) | 2019-09-25 | 2021-04-01 | Apple Inc. | Text detection using global geometry estimators |
US10810709B1 (en) | 2019-11-21 | 2020-10-20 | Eygs Llp | Systems and methods for improving the quality of text documents using artificial intelligence |
US10922476B1 (en) * | 2019-12-13 | 2021-02-16 | Microsoft Technology Licensing, Llc | Resource-efficient generation of visual layout information associated with network-accessible documents |
TWI798513B (en) * | 2019-12-20 | 2023-04-11 | 國立清華大學 | Training method of natural language corpus for the decision making model of machine learning |
CN111104514B (en) * | 2019-12-23 | 2023-04-25 | 北京百度网讯科技有限公司 | Training method and device for document tag model |
US11775862B2 (en) * | 2020-01-14 | 2023-10-03 | Microsoft Technology Licensing, Llc | Tracking provenance in data science scripts |
US11625934B2 (en) | 2020-02-04 | 2023-04-11 | Eygs Llp | Machine learning based end-to-end extraction of tables from electronic documents |
US11354513B2 (en) | 2020-02-06 | 2022-06-07 | Adobe Inc. | Automated identification of concept labels for a text fragment |
US11416684B2 (en) * | 2020-02-06 | 2022-08-16 | Adobe Inc. | Automated identification of concept labels for a set of documents |
US11176329B2 (en) | 2020-02-18 | 2021-11-16 | Bank Of America Corporation | Source code compiler using natural language input |
US11250128B2 (en) | 2020-02-18 | 2022-02-15 | Bank Of America Corporation | System and method for detecting source code anomalies |
US11568153B2 (en) | 2020-03-05 | 2023-01-31 | Bank Of America Corporation | Narrative evaluator |
US11934441B2 (en) | 2020-04-29 | 2024-03-19 | International Business Machines Corporation | Generative ontology learning and natural language processing with predictive language models |
US11183193B1 (en) | 2020-05-11 | 2021-11-23 | Apple Inc. | Digital assistant hardware abstraction |
US11061543B1 (en) | 2020-05-11 | 2021-07-13 | Apple Inc. | Providing relevant data items based on context |
US11341339B1 (en) * | 2020-05-14 | 2022-05-24 | Amazon Technologies, Inc. | Confidence calibration for natural-language understanding models that provides optimal interpretability |
US11443082B2 (en) * | 2020-05-27 | 2022-09-13 | Accenture Global Solutions Limited | Utilizing deep learning and natural language processing to convert a technical architecture diagram into an interactive technical architecture diagram |
US11704580B2 (en) | 2020-05-31 | 2023-07-18 | International Business Machines Corporation | Automated combination of predictions made by different prediction systems |
US11556788B2 (en) * | 2020-06-15 | 2023-01-17 | International Business Machines Corporation | Text-based response environment action selection |
US11393456B1 (en) * | 2020-06-26 | 2022-07-19 | Amazon Technologies, Inc. | Spoken language understanding system |
CN111522891A (en) * | 2020-07-03 | 2020-08-11 | 支付宝(杭州)信息技术有限公司 | Method and device for determining relationship closeness between two entities and electronic equipment |
US11490204B2 (en) | 2020-07-20 | 2022-11-01 | Apple Inc. | Multi-device audio adjustment coordination |
US11438683B2 (en) | 2020-07-21 | 2022-09-06 | Apple Inc. | User identification using headphones |
US11829720B2 (en) | 2020-09-01 | 2023-11-28 | Apple Inc. | Analysis and validation of language models |
US20220199078A1 (en) * | 2020-12-22 | 2022-06-23 | Samsung Electronics Co., Ltd. | Electronic apparatus, system comprising electronic apparatus and server and controlling method thereof |
WO2022155539A1 (en) * | 2021-01-14 | 2022-07-21 | Virtuosource Llc | A rules-based decision support system for assessment of digital content involving natural language |
US11814820B2 (en) | 2021-01-19 | 2023-11-14 | Source Global, PBC | Systems and methods for generating water from air |
US11741313B2 (en) * | 2021-02-08 | 2023-08-29 | International Business Machines Corporation | Question based chatbot generator from web API specifications |
CN115237856A (en) * | 2021-04-23 | 2022-10-25 | 伊姆西Ip控股有限责任公司 | Method, device and computer program product for marking files |
US20230008868A1 (en) * | 2021-07-08 | 2023-01-12 | Nippon Telegraph And Telephone Corporation | User authentication device, user authentication method, and user authentication computer program |
US20230091581A1 (en) * | 2021-09-21 | 2023-03-23 | Bank Of America Corporation | Personal Data Discovery |
US20230088315A1 (en) * | 2021-09-22 | 2023-03-23 | Motorola Solutions, Inc. | System and method to support human-machine interactions for public safety annotations |
US11516157B1 (en) * | 2021-09-30 | 2022-11-29 | Dell Products, L.P. | Machine learning facilitated drafting of response to communication |
US11977843B2 (en) * | 2022-01-24 | 2024-05-07 | S&P Global Inc. | Dynamic ontology for intelligent data discovery |
US20230409838A1 (en) * | 2022-05-31 | 2023-12-21 | International Business Machines Corporation | Explaining natural-language-to-flow pipelines |
US11546323B1 (en) | 2022-08-17 | 2023-01-03 | strongDM, Inc. | Credential management for distributed services |
US11736531B1 (en) | 2022-08-31 | 2023-08-22 | strongDM, Inc. | Managing and monitoring endpoint activity in secured networks |
US11765159B1 (en) | 2022-09-28 | 2023-09-19 | strongDM, Inc. | Connection revocation in overlay networks |
CN115294773B (en) * | 2022-09-29 | 2023-02-14 | 深圳市城市交通规划设计研究中心股份有限公司 | Bus lane optimal configuration method, electronic device and storage medium |
CN115904482B (en) * | 2022-11-30 | 2023-09-26 | 杭州巨灵兽智能科技有限公司 | Interface document generation method, device, equipment and storage medium |
US11916885B1 (en) | 2023-01-09 | 2024-02-27 | strongDM, Inc. | Tunnelling with support for dynamic naming resolution |
US11765207B1 (en) * | 2023-03-17 | 2023-09-19 | strongDM, Inc. | Declaring network policies using natural language |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20030212544A1 (en) * | 2002-05-10 | 2003-11-13 | Alejandro Acero | System for automatically annotating training data for a natural language understanding system |
US20050010605A1 (en) * | 2002-12-23 | 2005-01-13 | West Publishing Company | Information retrieval systems with database-selection aids |
US7664644B1 (en) * | 2006-06-09 | 2010-02-16 | At&T Intellectual Property Ii, L.P. | Multitask learning for spoken language understanding |
US20150356260A1 (en) * | 2014-06-04 | 2015-12-10 | Nuance Communications, Inc. | Nlu training with user corrections to engine annotations |
Family Cites Families (63)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5970490A (en) * | 1996-11-05 | 1999-10-19 | Xerox Corporation | Integration platform for heterogeneous databases |
WO2001059625A1 (en) * | 2000-02-10 | 2001-08-16 | Involve Technology, Llc | System for creating and maintaining a database of information utilizing user opinions |
US7730113B1 (en) * | 2000-03-07 | 2010-06-01 | Applied Discovery, Inc. | Network-based system and method for accessing and processing emails and other electronic legal documents that may include duplicate information |
WO2002005137A2 (en) * | 2000-07-07 | 2002-01-17 | Criticalpoint Software Corporation | Methods and system for generating and searching ontology databases |
US7376635B1 (en) * | 2000-07-21 | 2008-05-20 | Ford Global Technologies, Llc | Theme-based system and method for classifying documents |
US6513059B1 (en) * | 2000-08-24 | 2003-01-28 | Cambira Corporation | Adaptive collaborative intelligent network system |
US20020152202A1 (en) * | 2000-08-30 | 2002-10-17 | Perro David J. | Method and system for retrieving information using natural language queries |
US7146399B2 (en) * | 2001-05-25 | 2006-12-05 | 2006 Trident Company | Run-time architecture for enterprise integration with transformation generation |
EP1473639A1 (en) * | 2002-02-04 | 2004-11-03 | Celestar Lexico-Sciences, Inc. | Document knowledge management apparatus and method |
US20040254950A1 (en) * | 2003-06-13 | 2004-12-16 | Musgrove Timothy A. | Catalog taxonomy for storing product information and system and method using same |
BRPI0412778A (en) * | 2003-07-22 | 2006-09-26 | Kinor Technologies Inc | access to information using ontology |
US20050278362A1 (en) * | 2003-08-12 | 2005-12-15 | Maren Alianna J | Knowledge discovery system |
GB0320205D0 (en) * | 2003-08-28 | 2003-10-01 | British Telecomm | Method and apparatus for storing and retrieving data |
US20050060140A1 (en) * | 2003-09-15 | 2005-03-17 | Maddox Paul Christopher | Using semantic feature structures for document comparisons |
US7533090B2 (en) * | 2004-03-30 | 2009-05-12 | Google Inc. | System and method for rating electronic documents |
US7809548B2 (en) * | 2004-06-14 | 2010-10-05 | University Of North Texas | Graph-based ranking algorithms for text processing |
US20070106499A1 (en) * | 2005-08-09 | 2007-05-10 | Kathleen Dahlgren | Natural language search system |
US7664746B2 (en) * | 2005-11-15 | 2010-02-16 | Microsoft Corporation | Personalized search and headlines |
US20070150802A1 (en) * | 2005-12-12 | 2007-06-28 | Canon Information Systems Research Australia Pty. Ltd. | Document annotation and interface |
US7835911B2 (en) * | 2005-12-30 | 2010-11-16 | Nuance Communications, Inc. | Method and system for automatically building natural language understanding models |
EP2159717A3 (en) * | 2006-03-30 | 2010-03-17 | Sony France S.A. | Hybrid audio-visual categorization system and method |
US8131756B2 (en) | 2006-06-21 | 2012-03-06 | Carus Alwin B | Apparatus, system and method for developing tools to process natural language text |
AR062635A1 (en) * | 2006-09-01 | 2008-11-19 | Thomson Global Resources | SYSTEM, METHODS, SOFTWARE AND INTERFASES TO FORMAT APPOINTMENTS OF LEGISLATION |
JP4311432B2 (en) * | 2006-09-29 | 2009-08-12 | ブラザー工業株式会社 | Information processing apparatus and program |
US7774198B2 (en) * | 2006-10-06 | 2010-08-10 | Xerox Corporation | Navigation system for text |
WO2008086189A2 (en) * | 2007-01-04 | 2008-07-17 | Wide Angle Llc | Relevancy rating of tags |
US8738606B2 (en) * | 2007-03-30 | 2014-05-27 | Microsoft Corporation | Query generation using environment configuration |
US8214743B2 (en) * | 2007-08-07 | 2012-07-03 | International Business Machines Corporation | Data management techniques |
WO2009050521A2 (en) * | 2007-10-17 | 2009-04-23 | Iti Scotland Limited | Computer-implemented methods displaying, in a first part, a document and in a second part, a selected index of entities identified in the document |
JP4518165B2 (en) * | 2008-03-11 | 2010-08-04 | 富士ゼロックス株式会社 | Related document presentation system and program |
US20100153318A1 (en) * | 2008-11-19 | 2010-06-17 | Massachusetts Institute Of Technology | Methods and systems for automatically summarizing semantic properties from documents with freeform textual annotations |
US8165974B2 (en) * | 2009-06-08 | 2012-04-24 | Xerox Corporation | System and method for assisted document review |
US8515957B2 (en) * | 2009-07-28 | 2013-08-20 | Fti Consulting, Inc. | System and method for displaying relationships between electronically stored information to provide classification suggestions via injection |
US20110029517A1 (en) * | 2009-07-31 | 2011-02-03 | Shihao Ji | Global and topical ranking of search results using user clicks |
US8682649B2 (en) * | 2009-11-12 | 2014-03-25 | Apple Inc. | Sentiment prediction from textual data |
US20120064501A1 (en) * | 2010-04-08 | 2012-03-15 | Sukkarieh Jana Z | Systems and Methods for Evaluation of Automatic Content Scoring Technologies |
US8498950B2 (en) * | 2010-10-15 | 2013-07-30 | Yahoo! Inc. | System for training classifiers in multiple categories through active learning |
US8725739B2 (en) * | 2010-11-01 | 2014-05-13 | Evri, Inc. | Category-based content recommendation |
US20120271774A1 (en) * | 2011-04-21 | 2012-10-25 | Hirevue, Inc. | Interview frameworks |
US20120303559A1 (en) * | 2011-05-27 | 2012-11-29 | Ctc Tech Corp. | Creation, use and training of computer-based discovery avatars |
JP5866728B2 (en) * | 2011-10-14 | 2016-02-17 | サイバーアイ・エンタテインメント株式会社 | Knowledge information processing server system with image recognition system |
US11410072B2 (en) * | 2011-10-21 | 2022-08-09 | Educational Testing Service | Computer-implemented systems and methods for detection of sentiment in writing |
WO2013088287A1 (en) * | 2011-12-12 | 2013-06-20 | International Business Machines Corporation | Generation of natural language processing model for information domain |
US9652452B2 (en) * | 2012-01-06 | 2017-05-16 | Yactraq Online Inc. | Method and system for constructing a language model |
US9170715B1 (en) * | 2012-02-03 | 2015-10-27 | CounterPointe Digital Technologies LLC | System and method for mass visualization of real estate properties |
US8832162B2 (en) * | 2012-03-25 | 2014-09-09 | Think Computer Corporation | Method and system for storing, categorizing and distributing information concerning relationships between data |
US8943135B2 (en) * | 2012-07-24 | 2015-01-27 | Fard Johnmar | System and method for measuring the positive or negative impact of digital and social media content on intent and behavior |
US20140188665A1 (en) | 2013-01-02 | 2014-07-03 | CrowdChunk LLC | CrowdChunk System, Method, and Computer Program Product for Searching Summaries of Online Reviews of Products |
GB2509539A (en) * | 2013-01-08 | 2014-07-09 | Ibm | Production rule engine |
US20140257990A1 (en) * | 2013-03-06 | 2014-09-11 | TipTap, Inc. | Method and system for determining correlations between personality traits of a group of consumers and a brand/product |
US9122681B2 (en) * | 2013-03-15 | 2015-09-01 | Gordon Villy Cormack | Systems and methods for classifying electronic information using advanced active learning techniques |
US9189540B2 (en) * | 2013-04-05 | 2015-11-17 | Hewlett-Packard Development Company, L.P. | Mobile web-based platform for providing a contextual alignment view of a corpus of documents |
US9471883B2 (en) | 2013-05-09 | 2016-10-18 | Moodwire, Inc. | Hybrid human machine learning system and method |
US9348815B1 (en) * | 2013-06-28 | 2016-05-24 | Digital Reasoning Systems, Inc. | Systems and methods for construction, maintenance, and improvement of knowledge representations |
US9772994B2 (en) | 2013-07-25 | 2017-09-26 | Intel Corporation | Self-learning statistical natural language processing for automatic production of virtual personal assistants |
US9990422B2 (en) * | 2013-10-15 | 2018-06-05 | Adobe Systems Incorporated | Contextual analysis engine |
US9547640B2 (en) * | 2013-10-16 | 2017-01-17 | International Business Machines Corporation | Ontology-driven annotation confidence levels for natural language processing |
US20160019282A1 (en) * | 2014-07-16 | 2016-01-21 | Axiom Global Inc. | Discovery management method and system |
US9886247B2 (en) * | 2014-10-30 | 2018-02-06 | International Business Machines Corporation | Using an application programming interface (API) data structure in recommending an API composite |
US9860308B2 (en) * | 2014-11-25 | 2018-01-02 | International Business Machines Corporation | Collaborative creation of annotation training data |
US9852132B2 (en) * | 2014-11-25 | 2017-12-26 | Chegg, Inc. | Building a topical learning model in a content management system |
US11295071B2 (en) * | 2014-12-09 | 2022-04-05 | 100.Co, Llc | Graphical systems and methods for human-in-the-loop machine intelligence |
US20160162458A1 (en) * | 2014-12-09 | 2016-06-09 | Idibon, Inc. | Graphical systems and methods for human-in-the-loop machine intelligence |
-
2015
- 2015-12-09 US US14/964,522 patent/US20160162458A1/en not_active Abandoned
- 2015-12-09 US US14/964,517 patent/US10127214B2/en active Active
- 2015-12-09 US US14/964,528 patent/US20160162464A1/en not_active Abandoned
- 2015-12-09 US US14/964,511 patent/US9495345B2/en active Active - Reinstated
- 2015-12-09 US US14/964,510 patent/US20160162569A1/en not_active Abandoned
- 2015-12-09 US US14/964,520 patent/US20160162457A1/en not_active Abandoned
-
2016
- 2016-10-14 US US15/294,156 patent/US20170235813A1/en not_active Abandoned
-
2018
- 2018-09-07 US US16/125,343 patent/US20190243886A1/en not_active Abandoned
- 2018-11-05 US US16/181,102 patent/US20190303428A1/en not_active Abandoned
- 2018-11-09 US US16/185,843 patent/US20190311024A1/en not_active Abandoned
- 2018-11-20 US US16/197,190 patent/US20190311025A1/en not_active Abandoned
- 2018-11-21 US US16/198,453 patent/US20200234002A1/en not_active Abandoned
-
2019
- 2019-02-28 US US16/289,481 patent/US20200034737A1/en not_active Abandoned
-
2020
- 2020-01-22 US US16/749,689 patent/US20200184146A1/en not_active Abandoned
- 2020-02-20 US US16/796,812 patent/US20210150130A1/en not_active Abandoned
- 2020-03-06 US US16/811,737 patent/US11599714B2/en active Active
- 2020-12-11 US US17/119,902 patent/US11288444B2/en active Active
- 2020-12-22 US US17/131,486 patent/US20210232761A1/en not_active Abandoned
-
2021
- 2021-02-03 US US17/166,493 patent/US20210232762A1/en not_active Abandoned
- 2021-02-22 US US17/182,178 patent/US20210232763A1/en not_active Abandoned
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20030212544A1 (en) * | 2002-05-10 | 2003-11-13 | Alejandro Acero | System for automatically annotating training data for a natural language understanding system |
US20050010605A1 (en) * | 2002-12-23 | 2005-01-13 | West Publishing Company | Information retrieval systems with database-selection aids |
US7664644B1 (en) * | 2006-06-09 | 2010-02-16 | At&T Intellectual Property Ii, L.P. | Multitask learning for spoken language understanding |
US20150356260A1 (en) * | 2014-06-04 | 2015-12-10 | Nuance Communications, Inc. | Nlu training with user corrections to engine annotations |
Cited By (45)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10382300B2 (en) | 2015-10-06 | 2019-08-13 | Evolv Technologies, Inc. | Platform for gathering real-time analysis |
US10817655B2 (en) | 2015-12-11 | 2020-10-27 | Palantir Technologies Inc. | Systems and methods for annotating and linking electronic documents |
US20170293859A1 (en) * | 2016-04-11 | 2017-10-12 | Yandex Europe Ag | Method for training a ranker module using a training set having noisy labels |
US10445379B2 (en) * | 2016-06-20 | 2019-10-15 | Yandex Europe Ag | Method of generating a training object for training a machine learning algorithm |
US10650086B1 (en) * | 2016-09-27 | 2020-05-12 | Palantir Technologies Inc. | Systems, methods, and framework for associating supporting data in word processing |
US11657044B2 (en) | 2016-10-28 | 2023-05-23 | Parexel International, Llc | Semantic parsing engine |
US11507629B2 (en) | 2016-10-28 | 2022-11-22 | Parexel International, Llc | Dataset networking and database modeling |
US10679008B2 (en) * | 2016-12-16 | 2020-06-09 | Microsoft Technology Licensing, Llc | Knowledge base for analysis of text |
US20180173698A1 (en) * | 2016-12-16 | 2018-06-21 | Microsoft Technology Licensing, Llc | Knowledge Base for Analysis of Text |
US11880746B1 (en) * | 2017-04-26 | 2024-01-23 | Hrb Innovations, Inc. | Interface for artificial intelligence training |
US11100195B2 (en) | 2017-06-02 | 2021-08-24 | Thinkspan, LLC | Zero-knowledge offers in universal data scaffold based data management platform |
US20180349454A1 (en) * | 2017-06-02 | 2018-12-06 | Thinkspan, LLC | Universal data scaffold based data management platform |
US10942988B2 (en) | 2017-06-02 | 2021-03-09 | Thinkspan, LLC | Zero-knowledge encryption in universal data scaffold based data management platform |
US10949502B2 (en) * | 2017-06-02 | 2021-03-16 | Thinkspan, LLC | Universal data scaffold based data management platform |
US20230055094A1 (en) * | 2017-10-20 | 2023-02-23 | Google Llc | Capturing Detailed Structure from Patient-Doctor Conversations for Use in Clinical Documentation |
US20200152302A1 (en) * | 2017-10-20 | 2020-05-14 | Google Llc | Capturing Detailed Structure From Patient-Doctor Conversations for Use in Clinical Documentation |
JP2020528590A (en) * | 2017-10-20 | 2020-09-24 | グーグル エルエルシー | Incorporation of detailed structures from patient-doctor conversations used in clinical documentation |
US11521722B2 (en) * | 2017-10-20 | 2022-12-06 | Google Llc | Capturing detailed structure from patient-doctor conversations for use in clinical documentation |
JP7174717B2 (en) | 2017-10-20 | 2022-11-17 | グーグル エルエルシー | Capture detailed structure from patient-physician conversations used in clinical documentation |
US10783328B2 (en) * | 2018-06-04 | 2020-09-22 | International Business Machines Corporation | Semi-automatic process for creating a natural language processing resource |
US20220276881A1 (en) * | 2018-07-23 | 2022-09-01 | Google Llc | Intelligent home screen of cloud-based content management platform |
US11922188B2 (en) * | 2018-07-23 | 2024-03-05 | Google Llc | Intelligent home screen of cloud-based content management platform |
US11729203B2 (en) * | 2018-11-02 | 2023-08-15 | KnowBe4, Inc. | System and methods of cybersecurity attack simulation for incident response training and awareness |
US20210226985A1 (en) * | 2018-11-02 | 2021-07-22 | KnowBe4, Inc. | System and methods of cybersecurity attack simulation for incident response training and awareness |
US12008439B2 (en) * | 2018-11-30 | 2024-06-11 | Jpmorgan Chase Bank, N.A. | Methods for sharing machine learning based web service models |
US20200175416A1 (en) * | 2018-11-30 | 2020-06-04 | Jpmorgan Chase Bank, N.A. | Methods for sharing machine learning based web service models |
US20200250216A1 (en) * | 2019-02-04 | 2020-08-06 | International Business Machines Corporation | Generating a domain-specific phrasal dictionary |
US11328007B2 (en) * | 2019-02-04 | 2022-05-10 | International Business Machines Corporation | Generating a domain-specific phrasal dictionary |
US12014135B2 (en) * | 2019-03-11 | 2024-06-18 | Parexel International, Llc | Methods, apparatus and systems for annotation of text documents |
US20220253594A1 (en) * | 2019-03-11 | 2022-08-11 | Parexel International, Llc | Methods, apparatus and systems for annotation of text documents |
WO2020185900A1 (en) * | 2019-03-11 | 2020-09-17 | Roam Analytics, Inc. | Methods, apparatus and systems for annotation of text documents |
US11263391B2 (en) * | 2019-03-11 | 2022-03-01 | Parexel International, Llc | Methods, apparatus and systems for annotation of text documents |
US20210065233A1 (en) * | 2019-03-12 | 2021-03-04 | Sherloq, Inc. | Interface method and system for enabling an advertisement sponsor to input data concerning leads generated in response to advertisements |
US11294884B2 (en) | 2019-08-09 | 2022-04-05 | International Business Machines Corporation | Annotation assessment and adjudication |
US11188517B2 (en) | 2019-08-09 | 2021-11-30 | International Business Machines Corporation | Annotation assessment and ground truth construction |
US11442944B2 (en) | 2019-10-18 | 2022-09-13 | Thinkspan, LLC | Algorithmic suggestions based on a universal data scaffold |
US11468100B2 (en) | 2019-10-18 | 2022-10-11 | Thinkspan, LLC | Scalable scaffolding and bundled data |
US20210233008A1 (en) * | 2020-01-28 | 2021-07-29 | Schlumberger Technology Corporation | Oilfield data file classification and information processing systems |
US20210326718A1 (en) * | 2020-04-16 | 2021-10-21 | Microsoft Technology Licensing, Llc | Machine learning techniques to shape downstream content traffic through hashtag suggestion during content creation |
US20210390250A1 (en) * | 2020-06-15 | 2021-12-16 | Canon Kabushiki Kaisha | Information processing apparatus |
US11645464B2 (en) | 2021-03-18 | 2023-05-09 | International Business Machines Corporation | Transforming a lexicon that describes an information asset |
US11645462B2 (en) | 2021-08-13 | 2023-05-09 | Pricewaterhousecoopers Llp | Continuous machine learning method and system for information extraction |
US11907650B2 (en) | 2021-08-13 | 2024-02-20 | PwC Product Sales LLC | Methods and systems for artificial intelligence- assisted document annotation |
US11443102B1 (en) * | 2021-08-13 | 2022-09-13 | Pricewaterhousecoopers Llp | Methods and systems for artificial intelligence-assisted document annotation |
US20230134796A1 (en) * | 2021-10-29 | 2023-05-04 | Glipped, Inc. | Named entity recognition system for sentiment labeling |
Also Published As
Publication number | Publication date |
---|---|
US20160162476A1 (en) | 2016-06-09 |
US20200234002A1 (en) | 2020-07-23 |
US20210232761A1 (en) | 2021-07-29 |
US20170235813A1 (en) | 2017-08-17 |
US20160162464A1 (en) | 2016-06-09 |
US11288444B2 (en) | 2022-03-29 |
US20190311024A1 (en) | 2019-10-10 |
US20210165955A1 (en) | 2021-06-03 |
US20160162569A1 (en) | 2016-06-09 |
US20210232762A1 (en) | 2021-07-29 |
US20190311025A1 (en) | 2019-10-10 |
US20200184146A1 (en) | 2020-06-11 |
US20190243886A1 (en) | 2019-08-08 |
US11599714B2 (en) | 2023-03-07 |
US10127214B2 (en) | 2018-11-13 |
US20190303428A1 (en) | 2019-10-03 |
US20210150130A1 (en) | 2021-05-20 |
US20200034737A1 (en) | 2020-01-30 |
US9495345B2 (en) | 2016-11-15 |
US20160162456A1 (en) | 2016-06-09 |
US20160162457A1 (en) | 2016-06-09 |
US20210232760A1 (en) | 2021-07-29 |
US20210232763A1 (en) | 2021-07-29 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US20210232763A1 (en) | Graphical systems and methods for human-in-the-loop machine intelligence | |
US11295071B2 (en) | Graphical systems and methods for human-in-the-loop machine intelligence | |
JP6714024B2 (en) | Automatic generation of N-grams and conceptual relationships from language input data | |
US11164026B2 (en) | Graphical user interface generation based on image analysis | |
US9621601B2 (en) | User collaboration for answer generation in question and answer system | |
US11521713B2 (en) | System and method for generating clinical trial protocol design document with selection of patient and investigator | |
WO2019236360A1 (en) | Taxonomy enrichment using ensemble classifiers | |
CN112417090B (en) | Using uncommitted user input data to improve task performance | |
US11699034B2 (en) | Hybrid artificial intelligence system for semi-automatic patent infringement analysis | |
KR102285142B1 (en) | Apparatus and method for recommending learning data for chatbots | |
US9418058B2 (en) | Processing method for social media issue and server device supporting the same | |
Wijayanti et al. | Ensemble approach for sentiment polarity analysis in user-generated Indonesian text | |
Phan et al. | A sentiment analysis method of objects by integrating sentiments from tweets | |
US10614100B2 (en) | Semantic merge of arguments | |
US20230359932A1 (en) | Classification process systems and methods | |
CN113870998A (en) | Interrogation method, device, electronic equipment and storage medium | |
Hoang et al. | Recommendation of expert group to question and answer sites based on user behaviors and diversity | |
Filippas et al. | Strength in numbers: using Big Data to simplify sentiment classification | |
US11868380B1 (en) | Systems and methods for large-scale content exploration | |
US20230325422A1 (en) | Hybrid artificial intelligence system for semi-automatic patent claims analysis | |
Basu et al. | An End-to-End Topic-Based Sentiment Analysis Framework from Twitter Using Feature Set Cumulation | |
Vichare | PROBABILISTIC ENSEMBLE MACHINE LEARNING APPROACHES FOR UNSTRUCTURED TEXTUAL DATA CLASSIFICATION | |
WO2023126815A1 (en) | System and method for generating a scientific report by extracting relevant content from search results | |
Nguyen | Top-K Item Recommendations Using Social Media Networks-Using Twitter Profiles as a Source for Recommending Movies | |
Weggersen et al. | Knowledge Base Acceleration Using Features Inspired by Collaborative Filtering |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: IDIBON, INC., CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:MUNRO, ROBERT J.;WALKER, CHRISTOPHER;LUGER, SARAH K.;AND OTHERS;SIGNING DATES FROM 20160119 TO 20160226;REEL/FRAME:038624/0202 |
|
AS | Assignment |
Owner name: IDIBON (ASSIGNMENT FOR THE BENEFIT OF CREDITORS), Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:IDIBON, INC.;REEL/FRAME:047110/0178 Effective date: 20160519 Owner name: HEALY, TREVOR, CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:IDIBON (ASSIGNMENT FOR THE BENEFIT OF CREDITORS), LLC;REEL/FRAME:047110/0449 Effective date: 20161010 Owner name: AIPARC HOLDINGS PTE. LTD., SINGAPORE Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:HEALY, TREVOR;REEL/FRAME:047110/0510 Effective date: 20181006 |
|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION |
|
AS | Assignment |
Owner name: AI IP INVESTMENTS LTD, VIRGIN ISLANDS, BRITISH Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:AIPARC HOLDINGS PTE. LTD.;REEL/FRAME:055377/0995 Effective date: 20210114 |
|
AS | Assignment |
Owner name: AIPARC HOLDINGS PTE. LTD., SINGAPORE Free format text: CORRECTIVE ASSIGNMENT TO CORRECT THE COVENANT INFORMATION TO BE UPDATED FROM AIRPARC HOLDING PTE. LTD. AND REPLACED WITH TREVOR HEALY (SEE MARKED ASSIGNMENT) PREVIOUSLY RECORDED ON REEL 047110 FRAME 0510. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT;ASSIGNOR:HEALY, TREVOR;REEL/FRAME:055404/0561 Effective date: 20181006 |
|
AS | Assignment |
Owner name: 100.CO, LLC, FLORIDA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:AI IP INVESTMENTS LTD.;REEL/FRAME:055929/0975 Effective date: 20210414 |
|
AS | Assignment |
Owner name: 100.CO, LLC, FLORIDA Free format text: CORRECTIVE ASSIGNMENT TO CORRECT THE ASSIGNEE CITY PREVIOUSLY RECORDED AT REEL: 055929 FRAME: 0975. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT;ASSIGNOR:AI IP INVESTMENTS LTD.;REEL/FRAME:056151/0150 Effective date: 20210414 |
|
AS | Assignment |
Owner name: 100.CO TECHNOLOGIES, INC., FLORIDA Free format text: NUNC PRO TUNC ASSIGNMENT;ASSIGNOR:100.CO, LLC;REEL/FRAME:062131/0714 Effective date: 20221214 |
|
AS | Assignment |
Owner name: DAASH INTELLIGENCE, INC., FLORIDA Free format text: CHANGE OF NAME;ASSIGNOR:100.CO TECHNOLOGIES, INC.;REEL/FRAME:064347/0117 Effective date: 20230118 |
|
AS | Assignment |
Owner name: 100.CO GLOBAL HOLDINGS, LLC, FLORIDA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:DAASH INTELLIGENCE, INC.;REEL/FRAME:064420/0108 Effective date: 20230713 |
|
AS | Assignment |
Owner name: AI IP INVESTMENTS LTD., UNITED KINGDOM Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:100.CO GLOBAL HOLDINGS, LLC;REEL/FRAME:066636/0583 Effective date: 20231220 |