US20180260382A1 - Domain-specific method for distinguishing type-denoting domain terms from entity-denoting domain terms - Google Patents

Domain-specific method for distinguishing type-denoting domain terms from entity-denoting domain terms Download PDF

Info

Publication number
US20180260382A1
US20180260382A1 US15/454,778 US201715454778A US2018260382A1 US 20180260382 A1 US20180260382 A1 US 20180260382A1 US 201715454778 A US201715454778 A US 201715454778A US 2018260382 A1 US2018260382 A1 US 2018260382A1
Authority
US
United States
Prior art keywords
answer
domain
terms
distinguishing features
features
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
US15/454,778
Other versions
US10073831B1 (en
Inventor
Charles E. Beller
Paul J. Chase, JR.
Richard L. Darden
Michael Drzewucki
Edward G. Katz
Christopher Phipps
James E. Ramirez
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
International Business Machines Corp
Original Assignee
International Business Machines Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by International Business Machines Corp filed Critical International Business Machines Corp
Priority to US15/454,778 priority Critical patent/US10073831B1/en
Assigned to INTERNATIONAL BUSINESS MACHINES CORPORATION reassignment INTERNATIONAL BUSINESS MACHINES CORPORATION ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: PHIPPS, CHRISTOPHER, CHASE, PAUL J., JR., DRZEWUCKI, MICHAEL, KATZ, EDWARD G., RAMIREZ, JAMES E., BELLER, CHARLES E., DARDEN, RICHARD L.
Priority to US15/618,910 priority patent/US10073833B1/en
Application granted granted Critical
Publication of US10073831B1 publication Critical patent/US10073831B1/en
Publication of US20180260382A1 publication Critical patent/US20180260382A1/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G06F17/277
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/30Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
    • G06F16/33Querying
    • G06F16/332Query formulation
    • G06F16/3329Natural language query formulation or dialogue systems
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/30Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
    • G06F16/33Querying
    • G06F16/3331Query processing
    • G06F16/334Query execution
    • G06F16/3344Query execution using natural language analysis
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/30Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
    • G06F16/35Clustering; Classification
    • G06F17/21
    • G06F17/2715
    • G06F17/30654
    • G06F17/30684
    • G06F17/30705
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00Handling natural language data
    • G06F40/20Natural language analysis
    • G06F40/205Parsing
    • G06F40/211Syntactic parsing, e.g. based on context-free grammar [CFG] or unification grammars
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00Handling natural language data
    • G06F40/20Natural language analysis
    • G06F40/279Recognition of textual entities
    • G06F40/284Lexical analysis, e.g. tokenisation or collocates
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00Handling natural language data
    • G06F40/30Semantic analysis
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00Handling natural language data
    • G06F40/30Semantic analysis
    • G06F40/35Discourse or dialogue representation

Definitions

  • the present invention generally relates to natural language processing, and more particularly to a method of analyzing text to categorize large sets of domain-specific terms.
  • Natural language processing is a field of computer science, artificial intelligence, and linguistics concerned with the interactions between computers and human (natural) languages.
  • Many challenges in NLP involve natural language understanding, that is, enabling computers to derive meaning from human or natural language input, and others involve natural language generation allowing computers to respond in a manner familiar to a user.
  • a non-technical person may input a natural language question to a computer system, and the system intelligence can provide a natural language answer which the user can hopefully understand.
  • Examples of an advanced computer systems that use natural language processing include virtual assistants, Internet search engines, and deep question answering systems such as the WatsonTM cognitive technology marketed by International Business Machines Corp.
  • Text analysis is known in the art pertaining to NLP and typically uses a text annotator program to search text documents (corpora) and analyze them relative to a defined set of tags.
  • Text annotators and corpora can be domain-specific, that is, intended for use in a particular context of interest such as medicine, business processes, sports, etc.
  • the text annotator can generate linguistic annotations within the document to tag concepts and entities that might be buried in the text.
  • a cognitive system can then use a set of linguistic, statistical and machine-learning techniques to analyze the annotated text, and extract key information such as person, location, organization, and particular objects (e.g., vehicles), or identify positive and negative sentiment.
  • Front-end NLP can include identification of a lexical answer type and a focus among others.
  • a lexical answer type is a term in a question that indicates what type of entity is being asked for, i.e., the primary concept that is being discussed. Focus is essentially the subject of the text or, in the case of a question, the answer to the question or a reference to the answer (an entity).
  • a LAT in a question might be a person type, with the answer being a specific person.
  • the present invention in at least one embodiment is generally directed to a method of distinguishing at least two classes of domain-specific terms that are crucial to the domain-specific natural language processing involved in deep question answering—a set T of domain-specific terms that refer to domain entity types and a set E of domain-specific terms that refer to domain entities.
  • This is accomplished by making use of a training set P of domain terms known to refer to domain entity types and a set E′ of domain terms known to refer to domain entities to identify distinguishing features from one or more corpora specific to a particular domain wherein the distinguishing features distinguish the linguistic objects in P from the linguistic objects in E′, and using these features to classify the terms from a list specific to the particular domain.
  • an automatic machine-learning classifier can be trained using the distinguishing features, and the classifier can then be used to classify the terms from the domain specific terminology list.
  • the distinguishing features can include one or more syntactic features or one or more lexical features.
  • the training sets (the T′ and E′ sets) can be extracted from the question-and-answer pairs automatically via text analysis if manually curated lists are not available.
  • the classified terms can be included in a domain-specific lexicon which facilitates a deep question answering system to yield an answer to a question.
  • FIG. 1 is a block diagram of a computer system programmed to carry out natural language processing, including domain-specific term classification, in accordance with one implementation of the present invention
  • FIG. 2 is a table of domain-specific question-and-answer pairs from which linguistic objects are extracted in accordance with one implementation of the present invention
  • FIGS. 3A and 3B are tables of lexical answer types (T) and answer entities (E) extracted from the question-and-answer pairs of FIG. 2 in accordance with one implementation of the present invention
  • FIG. 4 is a block diagram of a classifying system constructed in accordance with one implementation of the present invention wherein linguistic objects extracted from the question-and-answer pairs are used to identify distinguishing features from a domain-specific corpus, and those distinguishing features are then used to train a natural language classifier;
  • FIG. 5 is a block diagram showing the use of the natural language classifier to categorize large sets of terms which can then be used to support a deep question answering system in accordance with one implementation of the present invention.
  • FIG. 6 is a chart illustrating the logical flow of a classification procedure in accordance with one implementation of the present invention.
  • Deep question answering systems make a distinction between terms that refer to types of entities and terms that refer to entities. These two classes of terms play important roles in the processing mechanisms built into deep question answering systems, and provisioning a deep question understanding system with adequate domain-specific lexical resources that articulate this distinction for a specific domain is one of the crucial ways in which domain adaption of such systems proceeds.
  • Terms that refer to types are often good candidates for the lexical answer type (LAT) of a question, while terms that refer to entities are often good candidates for the answer itself.
  • LAT lexical answer type
  • This distinction can be crucial to answer generation, answer scoring, answer filtering and other components of deep question answering. While a given term might both refer to a type of entity and refer to an entity, for a particular domain, terms that make good answer types tend to make bad answers and terms that make good answers are generally bad types.
  • Protein is the cornerstone of my bodybuilding nutrition plan in that it determines how many meals I eat each day.
  • the present invention achieves these objectives by leveraging existing artifacts involved in the domain adaptation task to automatically classify domain terms into those that refer to entities and those that refer to entity types.
  • this would involve extracting training sets of linguistic objects from domain-specific question-and-answer pairs, identifying features from a domain-specific corpus which can be used to distinguish these sets of linguistic objects, and using these features to classify domain terms in a large list of terms as being one of the particular linguistic objects, e.g., either a “likely LAT” or a “likely entity”.
  • Computer system 10 is a symmetric multiprocessor (SMP) system having a plurality of processors 12 a, 12 b connected to a system bus 14 .
  • System bus 14 is further connected to and communicates with a combined memory controller/host bridge (MC/HB) 16 which provides an interface to system memory 18 .
  • System memory 18 may be a local memory device or alternatively may include a plurality of distributed memory devices, preferably dynamic random-access memory (DRAM).
  • DRAM dynamic random-access memory
  • System memory 18 has loaded therein various NLP tools, including term classifier tools as taught herein.
  • MC/HB 16 also has an interface to peripheral component interconnect (PCI) Express links 20 a, 20 b, 20 c.
  • PCIe peripheral component interconnect
  • Each PCI Express (PCIe) link 20 a, 20 b is connected to a respective PCIe adaptor 22 a, 22 b, and each PCIe adaptor 22 a, 22 b is connected to a respective input/output (I/O) device 24 a, 24 b.
  • MC/HB 16 may additionally have an interface to an I/O bus 26 which is connected to a switch (I/O fabric) 28 .
  • Switch 28 provides a fan-out for the I/O bus to a plurality of PCI links 20 d, 20 e, 20 f These PCI links are connected to more PCIe adaptors 22 c, 22 d, 22 e which in turn support more I/O devices 24 c, 24 d, 24 e.
  • the I/O devices may include, without limitation, a keyboard, a graphical pointing device (mouse), a microphone, a display device, speakers, a permanent storage device (hard disk drive) or an array of such storage devices, an optical disk drive which receives an optical disk 25 (one example of a computer readable storage medium) such as a CD or DVD, and a network card.
  • Each PCIe adaptor provides an interface between the PCI link and the respective I/O device.
  • MC/HB 16 provides a low latency path through which processors 12 a, 12 b may access PCI devices mapped anywhere within bus memory or I/O address spaces.
  • MC/HB 16 further provides a high bandwidth path to allow the PCI devices to access memory 18 .
  • Switch 28 may provide peer-to-peer communications between different endpoints and this data traffic does not need to be forwarded to MC/HB 16 if it does not involve cache-coherent memory transfers. Switch 28 is shown as a separate logical component but it could be integrated into MC/HB 16 .
  • PCI link 20 c connects MC/HB 16 to a service processor interface 30 to allow communications between I/O device 24 a and a service processor 32 .
  • Service processor 32 is connected to processors 12 a, 12 b via a JTAG interface 34 , and uses an attention line 36 which interrupts the operation of processors 12 a, 12 b.
  • Service processor 32 may have its own local memory 38 , and is connected to read-only memory (ROM) 40 which stores various program instructions for system startup. Service processor 32 may also have access to a hardware operator panel 42 to provide system status and diagnostic information.
  • ROM read-only memory
  • computer system 10 may include modifications of these hardware components or their interconnections, or additional components, so the depicted example should not be construed as implying any architectural limitations with respect to the present invention.
  • the invention may further be implemented in an equivalent cloud computing network.
  • service processor 32 uses JTAG interface 34 to interrogate the system (host) processors 12 a, 12 b and MC/HB 16 . After completing the interrogation, service processor 32 acquires an inventory and topology for computer system 10 . Service processor 32 then executes various tests such as built-in-self-tests (BISTs), basic assurance tests (BATs), and memory tests on the components of computer system 10 . Any error information for failures detected during the testing is reported by service processor 32 to operator panel 42 . If a valid configuration of system resources is still possible after taking out any components found to be faulty during the testing then computer system 10 is allowed to proceed.
  • BISTs built-in-self-tests
  • BATs basic assurance tests
  • memory tests any error information for failures detected during the testing is reported by service processor 32 to operator panel 42 . If a valid configuration of system resources is still possible after taking out any components found to be faulty during the testing then computer system 10 is allowed to proceed.
  • Executable code is loaded into memory 18 and service processor 32 releases host processors 12 a, 12 b for execution of the program code, e.g., an operating system (OS) which is used to launch applications and in particular the NLP application of the present invention, results of which may be stored in a hard disk drive of the system (an I/O device 24 ).
  • OS operating system
  • service processor 32 may enter a mode of monitoring and reporting any operating parameters or errors, such as the cooling fan speed and operation, thermal sensors, power supply regulators, and recoverable and non-recoverable errors reported by any of processors 12 a , 12 b, memory 18 , and MC/HB 16 .
  • Service processor 32 may take further action based on the type of errors or defined thresholds.
  • the present invention may be a system, a method, and/or a computer program product.
  • the computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present invention.
  • the computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device.
  • the computer readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing.
  • a non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing.
  • RAM random access memory
  • ROM read-only memory
  • EPROM or flash memory erasable programmable read-only memory
  • SRAM static random access memory
  • CD-ROM compact disc read-only memory
  • DVD digital versatile disk
  • memory stick a floppy disk
  • a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon
  • a computer readable storage medium is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.
  • Computer readable program instructions described herein can be downloaded to respective computing/processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and/or a wireless network.
  • the network may comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers.
  • a network adapter card or network interface in each computing/processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing/processing device.
  • Computer readable program instructions for carrying out operations of the present invention may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like, and conventional procedural programming languages, such as the “C” programming language or similar programming languages.
  • the computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server.
  • the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider).
  • electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) may execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present invention.
  • These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.
  • These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, a programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including instructions which implement aspects of the function/act specified in the flowchart and/or block diagram block or blocks.
  • the computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions/acts specified in the flowchart and/or block diagram block or blocks.
  • each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s).
  • the functions noted in the block may occur out of the order noted in the figures.
  • two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved.
  • Computer system 10 carries out program instructions for natural language processing that uses novel analysis techniques to manage the classification of large lists of domain-specific terms. Accordingly, a program embodying the invention may include conventional aspects of various NLP tools, and these details will become apparent to those skilled in the art upon reference to this disclosure.
  • the system is tuned to specific application domains by engaging in a process known as domain adaptation.
  • This task is usually performed by an experienced NLP analyst working in concert with an expert in the particular domain of interest.
  • domain experts are called upon to submit long lists of domain terms for ingestion into the system.
  • the NLP analyst assesses the list to create domain-specific dictionaries, on the basis of general knowledge about the role of domain dictionaries in the system and additional knowledge of the domain, distinguishing terms that may be answers from terms that refer to types of answers.
  • This task is difficult and time consuming, and adequately tuning the domain dictionaries is a significant problem that calls out for an systematic solution.
  • the present invention addresses this problem.
  • Training data for this classifier is derived from the QA pairs, with the identified lexical answer types from the questions serving as T-class ground truth and the identified answer entities to the questions serving as E-class ground truth.
  • domain-specific training data can be applied to domain-specific corpora to derive a domain-specific classifier that can distinguish domain terms into domain T terms (those terms that are used in that domain typically as types of answers) and domain E terms (those terms that are used in that domain typically as answers to questions).
  • the QA pairs can be curated by any means, including manual, or using collections of previously derived QA pairs. There are preferably hundreds of QA pairs in set 50 .
  • the QA pairs may include a previous identification of LAT terms and answer entities, or they can be examined by computer system 10 using conventional text analysis to automatically identify these and other types of linguistic objects. For example, named entity recognition is known in the art and uses linguistic grammar-based techniques as well as statistical models, i.e. machine learning, to annotate sentences (including questions).
  • the QA pairs can be stored on computer system 10 or remotely.
  • Terms can be extracted from the multiple QA pairs by computer system 10 and assigned into one of at least two sets (T and E) as further seen in the tables 60 , 62 of FIGS. 3A and 3B .
  • FIG. 3A shows the set T of LATs extracted from the QA pairs
  • FIG. 3B shows the set E of entities extracted from the QA pairs.
  • the first QA pair in table 50 are “What country has the most people?” and “China has the world's largest population.” From these sentences, the term “country” has been identified as a LAT and added to table 60 , while the word “China” has been identified as an entity and added to table 62 .
  • FIG. 4 shows how the T and E tables 60 , 62 can be used in one implementation of the present invention to identify features of the domain of interest which can in turn be used to distinguish terms as different linguistic objects.
  • a feature identification module 72 running on computer system 10 takes the terms from the T and E tables 60 , 62 and searches for those terms with a domain-specific corpus or corpora 74 . Computer system 10 can then examine the usage of the particular terms as found within corpora 74 to identify features 76 which appear to be common to one class or another (LAT or entity). Any feature having statistical significance can be used, particularly syntactic features and lexical features.
  • a syntactic feature might be ‘appears as the subject of a sentence’ (e.g., “Protein is good for you’) or ‘appears as the possessor phrase’ (e.g., “Lincoln's wife was strange.”).
  • Syntactic-lexical binary features can also be used, e.g., the term occurring before the phrase “such as” or occurring after the phrase “kinds of”, ngrams (a contiguous sequence of items from a given snippet of text), or combinations of any of the foregoing.
  • FIG. 5 illustrates how the type-entity classifier 78 thus constructed can be further used to generate a domain-specific lexicon or dictionary 82 in accordance with one embodiment of the present invention.
  • Classifier 78 running on computer system 10 , receives a list of terms 84 pertaining to the domain of interest, and uses the distinguishing features (also domain-specific) to classify each term in list 84 as either a “likely LAT” or a “likely entity”.
  • the classifier can be based on features reflecting common syntactic contexts of a term as it appears in the corpus (where syntactic context might be distinguished by the sequence of words before and after the term, and the frequency of the context might be a count of the number of times the same words appear before and after words in a designated class).
  • syntactic context might be distinguished by the sequence of words before and after the term, and the frequency of the context might be a count of the number of times the same words appear before and after words in a designated class.
  • a target term from the domain terms list might be classified by determining if its distribution within a domain-specific corpus (such as in corpora 74 ) is more like the T-class terms or the E-class terms (in the simplest case by counting how many of the T-class frequent contexts it appears in and how many of the E-class frequent contexts it appears in). Other potential corpus-specific classification methods could be used.
  • the resulting lexicon 82 includes an appropriate tag for each term indicating its determined class, and can then be used by a deep question answering system 86 to facilitate the provision of a natural language answer to a natural language question. Deep question answering system 86 can also be running on computer system 10 .
  • One example of a way in which these tags can facilitate the deep question answering system is in answer scoring.
  • deep question-answering systems such as WatsonTM systems
  • one component of the process involves determining whether a term identified as a possible answer to the question is of the right type. So in the case of “Which substance was used by Stanford to . . . ?”, much of the processing involves identifying candidate answers (such as “Gas6”); if we know that in the given domain there is a type “protein”—which is a substance—and that “Gas6” is an entity of this type, then that answer would be highly scored and returned as a good result.
  • Process 90 begins by extracting sets of linguistic objects from question-and-answer pairs ( 92 ). There must be at least two kinds of linguistic objects extracted, such as lexical answer type and answer entity. Features from a domain-specific corpus are identified which distinguish the kinds of linguistic objects so extracted ( 94 ). These distinguishing features can be based on various statistical measures of different usages of the objects, particularly syntactic or lexical contexts. Terms in large lists can then be automatically classified, e.g., as either LAT or answer based on the distinguishing features ( 96 ). In the illustrative embodiment, this step is carried out with a classifier trained with the distinguishing features.
  • the present invention thereby provides an efficient and effective method of categorizing very large sets of terms associated with a particular domain. This approach not only saves countless hours of manual classification, but further provides a more robust lexicon which can help a deep question answering system provide superior results.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computational Linguistics (AREA)
  • Artificial Intelligence (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Mathematical Physics (AREA)
  • Human Computer Interaction (AREA)
  • Machine Translation (AREA)

Abstract

Large lists of domain-specific terms are classified as a particular kind of linguistic object, e.g., lexical answer type T versus canonical answer E, based on features from a domain-specific corpus which have been found to distinguish between the linguistic objects. The distinguishing features can be identified in the corpus based on sets of the linguistic objects derived from question-and-answer pairs. A classifier can be trained using the distinguishing features, and the classification carried out using that classifier. The distinguishing features can include one or more syntactic features or one or more lexical features. The linguistic objects (the T and E training sets) can be extracted from the question-and-answer pairs automatically via text analysis if manually curated lists are not available. The classified terms can be included in a domain-specific lexicon which facilitates a deep question answering system to yield an answer to a question.

Description

    STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT
  • This invention was made with United States Government support under agreement no. 2013-12101100008. THE GOVERNMENT HAS CERTAIN RIGHTS IN THIS INVENTION.
  • BACKGROUND OF THE INVENTION Field of the Invention
  • The present invention generally relates to natural language processing, and more particularly to a method of analyzing text to categorize large sets of domain-specific terms.
  • Description of the Related Art
  • As interactions between humans and computer systems become more complex, it becomes increasingly important to provide a more intuitive interface for a user to issue commands and queries to a computer system. As part of this effort, many systems employ some form of natural language processing. Natural language processing (NLP) is a field of computer science, artificial intelligence, and linguistics concerned with the interactions between computers and human (natural) languages. Many challenges in NLP involve natural language understanding, that is, enabling computers to derive meaning from human or natural language input, and others involve natural language generation allowing computers to respond in a manner familiar to a user. For example, a non-technical person may input a natural language question to a computer system, and the system intelligence can provide a natural language answer which the user can hopefully understand. Examples of an advanced computer systems that use natural language processing include virtual assistants, Internet search engines, and deep question answering systems such as the Watson™ cognitive technology marketed by International Business Machines Corp.
  • Text analysis is known in the art pertaining to NLP and typically uses a text annotator program to search text documents (corpora) and analyze them relative to a defined set of tags. Text annotators and corpora can be domain-specific, that is, intended for use in a particular context of interest such as medicine, business processes, sports, etc. The text annotator can generate linguistic annotations within the document to tag concepts and entities that might be buried in the text. A cognitive system can then use a set of linguistic, statistical and machine-learning techniques to analyze the annotated text, and extract key information such as person, location, organization, and particular objects (e.g., vehicles), or identify positive and negative sentiment. Front-end NLP can include identification of a lexical answer type and a focus among others. A lexical answer type (LAT) is a term in a question that indicates what type of entity is being asked for, i.e., the primary concept that is being discussed. Focus is essentially the subject of the text or, in the case of a question, the answer to the question or a reference to the answer (an entity). For example, a LAT in a question might be a person type, with the answer being a specific person.
  • SUMMARY OF THE INVENTION
  • The present invention in at least one embodiment is generally directed to a method of distinguishing at least two classes of domain-specific terms that are crucial to the domain-specific natural language processing involved in deep question answering—a set T of domain-specific terms that refer to domain entity types and a set E of domain-specific terms that refer to domain entities. This is accomplished by making use of a training set P of domain terms known to refer to domain entity types and a set E′ of domain terms known to refer to domain entities to identify distinguishing features from one or more corpora specific to a particular domain wherein the distinguishing features distinguish the linguistic objects in P from the linguistic objects in E′, and using these features to classify the terms from a list specific to the particular domain. In the illustrative implementations an automatic machine-learning classifier can be trained using the distinguishing features, and the classifier can then be used to classify the terms from the domain specific terminology list. The distinguishing features can include one or more syntactic features or one or more lexical features. The training sets (the T′ and E′ sets) can be extracted from the question-and-answer pairs automatically via text analysis if manually curated lists are not available. The classified terms can be included in a domain-specific lexicon which facilitates a deep question answering system to yield an answer to a question.
  • The above as well as additional objectives, features, and advantages in the various embodiments of the present invention will become apparent in the following detailed written description.
  • BRIEF DESCRIPTION OF THE DRAWINGS
  • The present invention may be better understood, and its numerous objects, features, and advantages of its various embodiments made apparent to those skilled in the art by referencing the accompanying drawings.
  • FIG. 1 is a block diagram of a computer system programmed to carry out natural language processing, including domain-specific term classification, in accordance with one implementation of the present invention;
  • FIG. 2 is a table of domain-specific question-and-answer pairs from which linguistic objects are extracted in accordance with one implementation of the present invention;
  • FIGS. 3A and 3B are tables of lexical answer types (T) and answer entities (E) extracted from the question-and-answer pairs of FIG. 2 in accordance with one implementation of the present invention;
  • FIG. 4 is a block diagram of a classifying system constructed in accordance with one implementation of the present invention wherein linguistic objects extracted from the question-and-answer pairs are used to identify distinguishing features from a domain-specific corpus, and those distinguishing features are then used to train a natural language classifier;
  • FIG. 5 is a block diagram showing the use of the natural language classifier to categorize large sets of terms which can then be used to support a deep question answering system in accordance with one implementation of the present invention; and
  • FIG. 6 is a chart illustrating the logical flow of a classification procedure in accordance with one implementation of the present invention.
  • The use of the same reference symbols in different drawings indicates similar or identical items.
  • DESCRIPTION OF THE PREFERRED EMBODIMENT(S)
  • Deep question answering systems make a distinction between terms that refer to types of entities and terms that refer to entities. These two classes of terms play important roles in the processing mechanisms built into deep question answering systems, and provisioning a deep question understanding system with adequate domain-specific lexical resources that articulate this distinction for a specific domain is one of the crucial ways in which domain adaption of such systems proceeds. Terms that refer to types are often good candidates for the lexical answer type (LAT) of a question, while terms that refer to entities are often good candidates for the answer itself. This distinction can be crucial to answer generation, answer scoring, answer filtering and other components of deep question answering. While a given term might both refer to a type of entity and refer to an entity, for a particular domain, terms that make good answer types tend to make bad answers and terms that make good answers are generally bad types.
  • In adapting a deep question answering system to a given domain, subject matter experts often provide lists of words and multi-word terms that are relevant to their domain, and these terms must be sorted into terms referring to types and terms referring to entities for them to be used appropriately in the deep question answering lexicon. Experience has shown that subject matter experts have difficulty making this distinction, and that it is a time consuming task for domain adaptation language technology experts. This distinction is often highly domain-specific. For example, the word “protein” may have a different role to play for question answering in the cancer research domain than in the body building domain. These different roles can be seen by comparing some question-and-answer (QA) sets for such domains. Here are two sample body building domain QAs where “protein” is an answer.
  • Question: What can I add to my diet to build muscle?
  • Answer: Protein is the cornerstone of my bodybuilding nutrition plan in that it determines how many meals I eat each day.
  • Question: What is seafood is an excellent source of?
  • Answer: Seafood is an excellent source of protein and it's usually low in fat.
  • Here are two sample cancer research domain QAs where “protein” is a LAT.
  • Question: What kinds of proteins act as immune system targets?
  • Answer: Researchers have spotted rare ‘flag’ proteins that act as immune system targets and are displayed on the surface of all of a patient's tumor cells, wherever they might be in the body.
  • Question: What two proteins did a Stanford team use to stop metastasis, without side effects?
  • Answer: The Stanford team seeks to stop metastasis, without side effects, by preventing two proteins—Ax1 and Gas6—from interacting to initiate the spread of cancer.
  • In customizing a deep question answering system with thousands of terms to be added, determining what role a term will play is thus a critical task, which feeds not only the deep question answering system itself, but also provides useful feedback to the domain adaptation team as to potential gaps in the taxonomy that should be filled, for example, type names that have only a few answer-level (entity) names associated with them. However, customer-provided lists of domain-specific terms are very time-consuming to sort into categories that are required for NLP systems. It would, therefore, be desirable to devise a method of automatically categorizing large sets of domain-specific terms. It would be further advantageous if the method could leverage other resources already available as part of front-end NLP.
  • The present invention achieves these objectives by leveraging existing artifacts involved in the domain adaptation task to automatically classify domain terms into those that refer to entities and those that refer to entity types. In exemplary implementations, this would involve extracting training sets of linguistic objects from domain-specific question-and-answer pairs, identifying features from a domain-specific corpus which can be used to distinguish these sets of linguistic objects, and using these features to classify domain terms in a large list of terms as being one of the particular linguistic objects, e.g., either a “likely LAT” or a “likely entity”.
  • With reference now to the figures, and in particular with reference to FIG. 1, there is depicted one embodiment 10 of a computer system in which the present invention may be implemented to carry out natural language processing including domain-specific term classification. Computer system 10 is a symmetric multiprocessor (SMP) system having a plurality of processors 12 a, 12 b connected to a system bus 14. System bus 14 is further connected to and communicates with a combined memory controller/host bridge (MC/HB) 16 which provides an interface to system memory 18. System memory 18 may be a local memory device or alternatively may include a plurality of distributed memory devices, preferably dynamic random-access memory (DRAM). There may be additional structures in the memory hierarchy which are not depicted, such as on-board (L1) and second-level (L2) or third-level (L3) caches. System memory 18 has loaded therein various NLP tools, including term classifier tools as taught herein.
  • MC/HB 16 also has an interface to peripheral component interconnect (PCI) Express links 20 a, 20 b, 20 c. Each PCI Express (PCIe) link 20 a, 20 b is connected to a respective PCIe adaptor 22 a, 22 b, and each PCIe adaptor 22 a, 22 b is connected to a respective input/output (I/O) device 24 a, 24 b. MC/HB 16 may additionally have an interface to an I/O bus 26 which is connected to a switch (I/O fabric) 28. Switch 28 provides a fan-out for the I/O bus to a plurality of PCI links 20 d, 20 e, 20 f These PCI links are connected to more PCIe adaptors 22 c, 22 d, 22 e which in turn support more I/O devices 24 c, 24 d, 24 e. The I/O devices may include, without limitation, a keyboard, a graphical pointing device (mouse), a microphone, a display device, speakers, a permanent storage device (hard disk drive) or an array of such storage devices, an optical disk drive which receives an optical disk 25 (one example of a computer readable storage medium) such as a CD or DVD, and a network card. Each PCIe adaptor provides an interface between the PCI link and the respective I/O device. MC/HB 16 provides a low latency path through which processors 12 a, 12 b may access PCI devices mapped anywhere within bus memory or I/O address spaces. MC/HB 16 further provides a high bandwidth path to allow the PCI devices to access memory 18. Switch 28 may provide peer-to-peer communications between different endpoints and this data traffic does not need to be forwarded to MC/HB 16 if it does not involve cache-coherent memory transfers. Switch 28 is shown as a separate logical component but it could be integrated into MC/HB 16.
  • In this embodiment, PCI link 20 c connects MC/HB 16 to a service processor interface 30 to allow communications between I/O device 24 a and a service processor 32. Service processor 32 is connected to processors 12 a, 12 b via a JTAG interface 34, and uses an attention line 36 which interrupts the operation of processors 12 a, 12 b. Service processor 32 may have its own local memory 38, and is connected to read-only memory (ROM) 40 which stores various program instructions for system startup. Service processor 32 may also have access to a hardware operator panel 42 to provide system status and diagnostic information.
  • In alternative embodiments computer system 10 may include modifications of these hardware components or their interconnections, or additional components, so the depicted example should not be construed as implying any architectural limitations with respect to the present invention. The invention may further be implemented in an equivalent cloud computing network.
  • When computer system 10 is initially powered up, service processor 32 uses JTAG interface 34 to interrogate the system (host) processors 12 a, 12 b and MC/HB 16. After completing the interrogation, service processor 32 acquires an inventory and topology for computer system 10. Service processor 32 then executes various tests such as built-in-self-tests (BISTs), basic assurance tests (BATs), and memory tests on the components of computer system 10. Any error information for failures detected during the testing is reported by service processor 32 to operator panel 42. If a valid configuration of system resources is still possible after taking out any components found to be faulty during the testing then computer system 10 is allowed to proceed. Executable code is loaded into memory 18 and service processor 32 releases host processors 12 a, 12 b for execution of the program code, e.g., an operating system (OS) which is used to launch applications and in particular the NLP application of the present invention, results of which may be stored in a hard disk drive of the system (an I/O device 24). While host processors 12 a, 12 b are executing program code, service processor 32 may enter a mode of monitoring and reporting any operating parameters or errors, such as the cooling fan speed and operation, thermal sensors, power supply regulators, and recoverable and non-recoverable errors reported by any of processors 12 a, 12 b, memory 18, and MC/HB 16. Service processor 32 may take further action based on the type of errors or defined thresholds.
  • The present invention may be a system, a method, and/or a computer program product. The computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present invention.
  • The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.
  • Computer readable program instructions described herein can be downloaded to respective computing/processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and/or a wireless network. The network may comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers. A network adapter card or network interface in each computing/processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing/processing device.
  • Computer readable program instructions for carrying out operations of the present invention may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like, and conventional procedural programming languages, such as the “C” programming language or similar programming languages. The computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) may execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present invention.
  • Aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer readable program instructions.
  • These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, a programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including instructions which implement aspects of the function/act specified in the flowchart and/or block diagram block or blocks.
  • The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions/acts specified in the flowchart and/or block diagram block or blocks.
  • The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts or carry out combinations of special purpose hardware and computer instructions.
  • Computer system 10 carries out program instructions for natural language processing that uses novel analysis techniques to manage the classification of large lists of domain-specific terms. Accordingly, a program embodying the invention may include conventional aspects of various NLP tools, and these details will become apparent to those skilled in the art upon reference to this disclosure.
  • In many deep question answering systems, the system is tuned to specific application domains by engaging in a process known as domain adaptation. This task is usually performed by an experienced NLP analyst working in concert with an expert in the particular domain of interest. In a typical domain adaptation exercise, domain experts are called upon to submit long lists of domain terms for ingestion into the system. The NLP analyst then assesses the list to create domain-specific dictionaries, on the basis of general knowledge about the role of domain dictionaries in the system and additional knowledge of the domain, distinguishing terms that may be answers from terms that refer to types of answers. This task is difficult and time consuming, and adequately tuning the domain dictionaries is a significant problem that calls out for an systematic solution. The present invention addresses this problem.
  • In addition to generating terminology lists , domain experts often create reasonably large sets of question-and-answer pairs that reflect the kinds of domain-specific questions that users of the deep question answering system might be expected to put to their system as well as identifying domain-specific document sets (corpora) that will contain answers to these questions. The current invention can leverage these QA pairs and these corpora to classify the domain terms as summarized above. This classification method is specific to a given domain and corpus. The general idea is to develop a text classifier to distinguish elements in the domain terms list into at least two linguistic classes, particularly a type (T) class and an entity (E) class. Training data for this classifier is derived from the QA pairs, with the identified lexical answer types from the questions serving as T-class ground truth and the identified answer entities to the questions serving as E-class ground truth. In this manner, domain-specific training data can be applied to domain-specific corpora to derive a domain-specific classifier that can distinguish domain terms into domain T terms (those terms that are used in that domain typically as types of answers) and domain E terms (those terms that are used in that domain typically as answers to questions).
  • Referring now to FIG. 2, there is depicted an exemplary set 50 of domain-specific question-and-answer pairs. For this example, the domain is world geography. The QA pairs can be curated by any means, including manual, or using collections of previously derived QA pairs. There are preferably hundreds of QA pairs in set 50. The QA pairs may include a previous identification of LAT terms and answer entities, or they can be examined by computer system 10 using conventional text analysis to automatically identify these and other types of linguistic objects. For example, named entity recognition is known in the art and uses linguistic grammar-based techniques as well as statistical models, i.e. machine learning, to annotate sentences (including questions). The QA pairs can be stored on computer system 10 or remotely.
  • Terms can be extracted from the multiple QA pairs by computer system 10 and assigned into one of at least two sets (T and E) as further seen in the tables 60, 62 of FIGS. 3A and 3B. FIG. 3A shows the set T of LATs extracted from the QA pairs, and FIG. 3B shows the set E of entities extracted from the QA pairs. For example, the first QA pair in table 50 are “What country has the most people?” and “China has the world's largest population.” From these sentences, the term “country” has been identified as a LAT and added to table 60, while the word “China” has been identified as an entity and added to table 62. Other extracted LATs include “mountain”, “rainforest”, “ocean”, “lake”, and “river”, and other extracted entities include “Mount Everest”, “Amazon River Basin”, “Marianas Trench”, “caldera”, and “Amazon”. As with table 50, there can be hundreds or even thousands of entries in tables 60, 62. The T and E sets can also be stored on computer system 10 or remotely.
  • FIG. 4 shows how the T and E tables 60, 62 can be used in one implementation of the present invention to identify features of the domain of interest which can in turn be used to distinguish terms as different linguistic objects. A feature identification module 72 running on computer system 10 takes the terms from the T and E tables 60, 62 and searches for those terms with a domain-specific corpus or corpora 74. Computer system 10 can then examine the usage of the particular terms as found within corpora 74 to identify features 76 which appear to be common to one class or another (LAT or entity). Any feature having statistical significance can be used, particularly syntactic features and lexical features. For example, a syntactic feature might be ‘appears as the subject of a sentence’ (e.g., “Protein is good for you’) or ‘appears as the possessor phrase’ (e.g., “Lincoln's wife was strange.”). Syntactic-lexical binary features can also be used, e.g., the term occurring before the phrase “such as” or occurring after the phrase “kinds of”, ngrams (a contiguous sequence of items from a given snippet of text), or combinations of any of the foregoing. These distinguishing features can be used to build a type-entity classifier 78 which is trained on the two sets T and E. Classifier 78 can also be stored on computer system 10 or remotely.
  • FIG. 5 illustrates how the type-entity classifier 78 thus constructed can be further used to generate a domain-specific lexicon or dictionary 82 in accordance with one embodiment of the present invention. Classifier 78, running on computer system 10, receives a list of terms 84 pertaining to the domain of interest, and uses the distinguishing features (also domain-specific) to classify each term in list 84 as either a “likely LAT” or a “likely entity”. In one embodiment, for example, the classifier can be based on features reflecting common syntactic contexts of a term as it appears in the corpus (where syntactic context might be distinguished by the sequence of words before and after the term, and the frequency of the context might be a count of the number of times the same words appear before and after words in a designated class). Using these kind of features, the N most frequent contexts in which terms on the T-class ground truth list appear would be extracted from the corpus along with the N most frequent contexts in which terms in E-class ground truth list appear. A target term from the domain terms list might be classified by determining if its distribution within a domain-specific corpus (such as in corpora 74) is more like the T-class terms or the E-class terms (in the simplest case by counting how many of the T-class frequent contexts it appears in and how many of the E-class frequent contexts it appears in). Other potential corpus-specific classification methods could be used. The resulting lexicon 82 includes an appropriate tag for each term indicating its determined class, and can then be used by a deep question answering system 86 to facilitate the provision of a natural language answer to a natural language question. Deep question answering system 86 can also be running on computer system 10.
  • One example of a way in which these tags can facilitate the deep question answering system is in answer scoring. In many deep question-answering systems—such as Watson™ systems—one component of the process involves determining whether a term identified as a possible answer to the question is of the right type. So in the case of “Which substance was used by Stanford to . . . ?”, much of the processing involves identifying candidate answers (such as “Gas6”); if we know that in the given domain there is a type “protein”—which is a substance—and that “Gas6” is an entity of this type, then that answer would be highly scored and returned as a good result.
  • The present invention may be further understood with reference to the chart of FIG. 6 which illustrates the logical flow for a classification process 90 in accordance with one implementation of the present invention, which may be carried out on computer system 10. Process 90 begins by extracting sets of linguistic objects from question-and-answer pairs (92). There must be at least two kinds of linguistic objects extracted, such as lexical answer type and answer entity. Features from a domain-specific corpus are identified which distinguish the kinds of linguistic objects so extracted (94). These distinguishing features can be based on various statistical measures of different usages of the objects, particularly syntactic or lexical contexts. Terms in large lists can then be automatically classified, e.g., as either LAT or answer based on the distinguishing features (96). In the illustrative embodiment, this step is carried out with a classifier trained with the distinguishing features.
  • The present invention thereby provides an efficient and effective method of categorizing very large sets of terms associated with a particular domain. This approach not only saves countless hours of manual classification, but further provides a more robust lexicon which can help a deep question answering system provide superior results.
  • Although the invention has been described with reference to specific embodiments, this description is not meant to be construed in a limiting sense. Various modifications of the disclosed embodiments, as well as alternative embodiments of the invention, will become apparent to persons skilled in the art upon reference to the description of the invention. It is therefore contemplated that such modifications can be made without departing from the spirit or scope of the present invention as defined in the appended claims.

Claims (14)

1.-7. (canceled)
8. A computer system comprising:
one or more processors which process program instructions;
a memory device connected to said one or more processors; and
program instructions residing in said memory device for distinguishing domain-specific terms from a list specific to a particular domain by extracting linguistic objects from a set of question-and-answer pairs wherein the linguistic objects include lexical answer types and answer entities, grouping the lexical answer types into a first set and grouping the answer entities into a second set, identifying distinguishing features of one or more corpora specific to a particular domain wherein the distinguishing features distinguish the lexical answer types in the first set from the answer entities in the second set, and classifying the domain-specific terms as either lexical answer type or answer entity based on the distinguishing features.
9. (canceled)
10. The computer system of claim 8 wherein said program instructions further train a natural language classifier using the distinguishing features, and the classifying uses the natural language classifier.
11. The computer system of claim 8 wherein the distinguishing features include one or more syntactic features.
12. The computer system of claim 8 wherein the distinguishing features include one or more lexical features.
13. The computer system of claim 8 wherein the extracting uses text analysis to automatically extract the linguistic objects.
14. The computer system of claim 8 wherein said program instructions further apply a lexicon of classified terms to a deep question answering system to yield an answer to a question.
15. A computer program product comprising:
a computer readable storage medium; and
program instructions residing in said storage medium for distinguishing domain-specific terms from a list specific to a particular domain by extracting linguistic objects from a set of question-and-answer pairs wherein the linguistic objects include lexical answer types and answer entities, grouping the lexical answer types into a first set and grouping the answer entities into a second set, identifying distinguishing features of one or more corpora specific to a particular domain wherein the distinguishing features distinguish the lexical answer types in the first set from the answer entities in the second set, and classifying the domain-specific terms as either lexical answer type or answer entity based on the distinguishing features.
16. (canceled)
17. The computer program product of claim 15 wherein said program instructions further train a natural language classifier using the distinguishing features, and the classifying uses the natural language classifier.
18. The computer program product of claim 15 wherein the distinguishing features include one or more syntactic features.
19. The computer program product of claim 15 wherein the distinguishing features include one or more lexical features.
20. The computer program product of claim 15 wherein the extracting uses text analysis to automatically extract the linguistic objects.
US15/454,778 2017-03-09 2017-03-09 Domain-specific method for distinguishing type-denoting domain terms from entity-denoting domain terms Active US10073831B1 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
US15/454,778 US10073831B1 (en) 2017-03-09 2017-03-09 Domain-specific method for distinguishing type-denoting domain terms from entity-denoting domain terms
US15/618,910 US10073833B1 (en) 2017-03-09 2017-06-09 Domain-specific method for distinguishing type-denoting domain terms from entity-denoting domain terms

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
US15/454,778 US10073831B1 (en) 2017-03-09 2017-03-09 Domain-specific method for distinguishing type-denoting domain terms from entity-denoting domain terms

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US15/618,910 Continuation US10073833B1 (en) 2017-03-09 2017-06-09 Domain-specific method for distinguishing type-denoting domain terms from entity-denoting domain terms

Publications (2)

Publication Number Publication Date
US10073831B1 US10073831B1 (en) 2018-09-11
US20180260382A1 true US20180260382A1 (en) 2018-09-13

Family

ID=63406451

Family Applications (2)

Application Number Title Priority Date Filing Date
US15/454,778 Active US10073831B1 (en) 2017-03-09 2017-03-09 Domain-specific method for distinguishing type-denoting domain terms from entity-denoting domain terms
US15/618,910 Active US10073833B1 (en) 2017-03-09 2017-06-09 Domain-specific method for distinguishing type-denoting domain terms from entity-denoting domain terms

Family Applications After (1)

Application Number Title Priority Date Filing Date
US15/618,910 Active US10073833B1 (en) 2017-03-09 2017-06-09 Domain-specific method for distinguishing type-denoting domain terms from entity-denoting domain terms

Country Status (1)

Country Link
US (2) US10073831B1 (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110210021A (en) * 2019-05-22 2019-09-06 北京百度网讯科技有限公司 Read understanding method and device
KR20210105830A (en) * 2020-02-19 2021-08-27 베이징 바이두 넷컴 사이언스 앤 테크놀로지 코., 엘티디. Entity word recognition method and device

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10902342B2 (en) 2016-09-16 2021-01-26 International Business Machines Corporation System and method for scoring the geographic relevance of answers in a deep question answering system based on geographic context of an input question
US10552461B2 (en) * 2016-09-16 2020-02-04 International Business Machines Corporation System and method for scoring the geographic relevance of answers in a deep question answering system based on geographic context of a candidate answer
CN112115241B (en) * 2019-06-21 2023-09-05 百度在线网络技术(北京)有限公司 Question answering method, device and equipment
CN110796338A (en) * 2019-09-24 2020-02-14 北京谦仁科技有限公司 Online teaching monitoring method and device, server and storage medium
US11397857B2 (en) 2020-01-15 2022-07-26 International Business Machines Corporation Methods and systems for managing chatbots with respect to rare entities

Family Cites Families (38)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6278996B1 (en) * 1997-03-31 2001-08-21 Brightware, Inc. System and method for message process and response
US5896321A (en) * 1997-11-14 1999-04-20 Microsoft Corporation Text completion system for a miniature computer
US6317707B1 (en) * 1998-12-07 2001-11-13 At&T Corp. Automatic clustering of tokens from a corpus for grammar acquisition
US20020032564A1 (en) * 2000-04-19 2002-03-14 Farzad Ehsani Phrase-based dialogue modeling with particular application to creating a recognition grammar for a voice-controlled user interface
US20020010574A1 (en) * 2000-04-20 2002-01-24 Valery Tsourikov Natural language processing and query driven information retrieval
US20030004706A1 (en) * 2001-06-27 2003-01-02 Yale Thomas W. Natural language processing system and method for knowledge management
JP3768205B2 (en) * 2003-05-30 2006-04-19 沖電気工業株式会社 Morphological analyzer, morphological analysis method, and morphological analysis program
US7644053B2 (en) * 2004-03-03 2010-01-05 The Boeing Company System, method, and computer program product for combination of cognitive causal models with reasoning and text processing for knowledge driven decision support
US8738359B2 (en) * 2006-10-18 2014-05-27 Honda Motor Co., Ltd. Scalable knowledge extraction
FI20060995A0 (en) * 2006-11-13 2006-11-13 Tiksis Technologies Oy Treatment of natural language
US8285697B1 (en) * 2007-01-23 2012-10-09 Google Inc. Feedback enhanced attribute extraction
US8732114B2 (en) * 2008-04-14 2014-05-20 Nuance Communications, Inc. Knowledge re-use for call routing
US8275803B2 (en) * 2008-05-14 2012-09-25 International Business Machines Corporation System and method for providing answers to questions
US20100185943A1 (en) * 2009-01-21 2010-07-22 Nec Laboratories America, Inc. Comparative document summarization with discriminative sentence selection
US9235563B2 (en) * 2009-07-02 2016-01-12 Battelle Memorial Institute Systems and processes for identifying features and determining feature associations in groups of documents
US9292493B2 (en) * 2010-01-07 2016-03-22 The Trustees Of The Stevens Institute Of Technology Systems and methods for automatically detecting deception in human communications expressed in digital form
WO2012040356A1 (en) 2010-09-24 2012-03-29 International Business Machines Corporation Providing question and answers with deferred type evaluation using text with limited structure
EP2616974A4 (en) 2010-09-24 2016-03-02 Ibm Lexical answer type confidence estimation and application
US8515736B1 (en) * 2010-09-30 2013-08-20 Nuance Communications, Inc. Training call routing applications by reusing semantically-labeled data collected for prior applications
US9015031B2 (en) * 2011-08-04 2015-04-21 International Business Machines Corporation Predicting lexical answer types in open domain question and answering (QA) systems
US8666982B2 (en) * 2011-10-06 2014-03-04 GM Global Technology Operations LLC Method and system to augment vehicle domain ontologies for vehicle diagnosis
JP2015505082A (en) * 2011-12-12 2015-02-16 インターナショナル・ビジネス・マシーンズ・コーポレーションInternational Business Machines Corporation Generation of natural language processing model for information domain
US20140006012A1 (en) * 2012-07-02 2014-01-02 Microsoft Corporation Learning-Based Processing of Natural Language Questions
US9720903B2 (en) * 2012-07-10 2017-08-01 Robert D. New Method for parsing natural language text with simple links
TWI465950B (en) * 2012-08-21 2014-12-21 Ind Tech Res Inst Method and system for discovering suspicious account groups
US9535898B2 (en) * 2013-02-06 2017-01-03 International Business Machines Corporation Natural language question expansion and extraction
US9665826B2 (en) * 2013-06-28 2017-05-30 Microsoft Technology Licensing, Llc Automated problem inference from bug repositories
US9146918B2 (en) * 2013-09-13 2015-09-29 International Business Machines Corporation Compressing data for natural language processing
US9558448B2 (en) * 2014-02-20 2017-01-31 International Business Machines Corporation Dynamic interfacing in a deep question answering system
US10380253B2 (en) * 2014-03-04 2019-08-13 International Business Machines Corporation Natural language processing with dynamic pipelines
US9892208B2 (en) 2014-04-02 2018-02-13 Microsoft Technology Licensing, Llc Entity and attribute resolution in conversational applications
US9569503B2 (en) 2014-05-23 2017-02-14 International Business Machines Corporation Type evaluation in a question-answering system
US9633309B2 (en) * 2014-06-19 2017-04-25 International Business Machines Corporation Displaying quality of question being asked a question answering system
US9754207B2 (en) * 2014-07-28 2017-09-05 International Business Machines Corporation Corpus quality analysis
US9720962B2 (en) * 2014-08-19 2017-08-01 International Business Machines Corporation Answering superlative questions with a question and answer system
US9875296B2 (en) * 2015-03-25 2018-01-23 Google Llc Information extraction from question and answer websites
US10545956B2 (en) * 2015-06-05 2020-01-28 Insight Engines, Inc. Natural language search with semantic mapping and classification
US9727554B2 (en) * 2015-11-24 2017-08-08 International Business Machines Corporation Knowledge-based editor with natural language interface

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110210021A (en) * 2019-05-22 2019-09-06 北京百度网讯科技有限公司 Read understanding method and device
KR20210105830A (en) * 2020-02-19 2021-08-27 베이징 바이두 넷컴 사이언스 앤 테크놀로지 코., 엘티디. Entity word recognition method and device
JP2021131858A (en) * 2020-02-19 2021-09-09 ベイジン バイドゥ ネットコム サイエンス アンド テクノロジー カンパニー リミテッド Entity word recognition method and apparatus
US11361002B2 (en) 2020-02-19 2022-06-14 Beijing Baidu Netcom Science Technology Co., Ltd. Method and apparatus for recognizing entity word, and storage medium
JP7096919B2 (en) 2020-02-19 2022-07-06 ベイジン バイドゥ ネットコム サイエンス テクノロジー カンパニー リミテッド Entity word recognition method and device
KR102431568B1 (en) 2020-02-19 2022-08-11 베이징 바이두 넷컴 사이언스 앤 테크놀로지 코., 엘티디. Entity word recognition method and device

Also Published As

Publication number Publication date
US10073833B1 (en) 2018-09-11
US20180260383A1 (en) 2018-09-13
US10073831B1 (en) 2018-09-11

Similar Documents

Publication Publication Date Title
US10073833B1 (en) Domain-specific method for distinguishing type-denoting domain terms from entity-denoting domain terms
US11568307B2 (en) Data augmentation for text-based AI applications
US10169706B2 (en) Corpus quality analysis
US9785887B2 (en) Extraction of semantic relations using distributional relation detection
US10394950B2 (en) Generation of a grammatically diverse test set for deep question answering systems
US10133724B2 (en) Syntactic classification of natural language sentences with respect to a targeted element
US10339168B2 (en) System and method for generating full questions from natural language queries
US20160170972A1 (en) Generating natural language text sentences as test cases for nlp annotators with combinatorial test design
US9760627B1 (en) Private-public context analysis for natural language content disambiguation
US10642874B2 (en) Using paraphrase metrics for answering questions
US10956463B2 (en) System and method for generating improved search queries from natural language questions
US20160170989A1 (en) Identification and Evaluation of Lexical Answer Type Conditions in a Question to Generate Correct Answers
US10339167B2 (en) System and method for generating full questions from natural language queries
US10482180B2 (en) Generating ground truth for questions based on data found in structured resources
US20160070693A1 (en) Optimizing Parsing Outcomes of Documents
US10586161B2 (en) Cognitive visual debugger that conducts error analysis for a question answering system
US11544312B2 (en) Descriptor uniqueness for entity clustering
CN110442877B (en) Using robotic planning as a parallel language corpus
US11422798B2 (en) Context-based word embedding for programming artifacts
US10824659B2 (en) Predicting the temporal stability of answers in a deep question answering system
US11461672B2 (en) Plug-and-ingest framework for question answering systems
US11397854B2 (en) Generation of domain thesaurus

Legal Events

Date Code Title Description
AS Assignment

Owner name: INTERNATIONAL BUSINESS MACHINES CORPORATION, NEW Y

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:BELLER, CHARLES E.;CHASE, PAUL J., JR.;DARDEN, RICHARD L.;AND OTHERS;SIGNING DATES FROM 20170206 TO 20170308;REEL/FRAME:041532/0088

STCF Information on status: patent grant

Free format text: PATENTED CASE

MAFP Maintenance fee payment

Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1551); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

Year of fee payment: 4