JP2001511565A - Text input processing system using natural language processing techniques - Google Patents

Abstract

(57) Summary The system (1480) filters documents in a set of documents retrieved from a document store in response to a query. The system (1480) obtains (1830) a first set of logical forms based on the query and a selected one of the documents in the document set. The system (1480) obtains a second set of logical forms based on the query and another one of the documents in the document set. Next, the system (1480) uses natural language processing techniques to modify the first logical form (1832, 1834) to obtain a modified logical form set. The system (1480) filters (1836) the documents in the document set based on a predetermined relationship between the modified set of logical forms and the second set of logical forms.

Description

DETAILED DESCRIPTION OF THE INVENTION

BACKGROUND OF THE INVENTION The present invention deals with the processing of text input. More specifically, the present invention relates to the use of natural language processing techniques to determine the similarity of text inputs. The invention is useful in a wide variety of applications, such as information retrieval, machine translation, natural language understanding, document similarity / clustering, and the like. However, for purposes of illustration only, the invention will be described primarily in the context of information retrieval.

Generally, information search refers to a process in which a user finds and searches for information related to the user from a large amount of stored information. When performing an information search, it is important that the user search for all necessary information (ie, it is important that it be complete).
In addition, it is important to restrict the irrelevant information searched at the same time to the user (that is, it is important to be selective). These dimensions (dimens
ion) is often expressed in terms of recovery (completeness) and accuracy (selectivity). For many information retrieval systems, it is important to perform well across both recovery and accuracy dimensions.

Some current search systems can query and search for enormous amounts of information. For example, some information retrieval systems are configured to search for information on the Internet, digital video discs, and other common computer databases. Typically, information retrieval systems
For example, it is embodied as an Internet search engine and a library catalog search engine. In addition, some information retrieval mechanisms are provided within the operating system of conventional desktop computers. For example, some operating systems have tools that allow a user to search all files on a given database or computer system based on certain terms entered by the user.

There are many known information retrieval techniques. User-entered queries in such techniques are typically presented as explicit user-generated queries or implicit queries, such as when a user requests documents similar to an existing collection of documents. A typical information retrieval system retrieves documents in mass storage at either the single word or term level. Each document is assigned a relevance (similarity) score, and the information retrieval system presents a subset of the searched documents to the user. Typically, this subset has a relevance score above a given threshold.

[0005] What makes conventional statistical search engines less accurate is that words are independent variables, that is, words in every passage of text occur independently of each other. Due to that. Independence in this context means the conditional probability of every word appearing in a document, with the presence of other words in it always being zero. That is, the document simply contains an unstructured set of words, or simply comprises a "bag of words". As will be readily appreciated, this assumption is a major mistake for any language.
English, like other languages, has a rich and complex grammar and vocabulary-semantic structure, and the meaning of words often varies widely based on the particular linguistic context in which they are used. The context, however, determines in each case the given meaning of the word and what word (s) will follow. Thus, the words that appear in a passage of text are not simply independent of each other, but rather are highly interdependent. Keyword-based search engines completely ignore this fine-grained linguistic structure. For example, consider an example query expressed in natural language, "How many hearts does an octopus have?" In the case of a statistical search engine that processes for the content words "heart" and "octopus" or their morphological roots, the stored document contains "artichoke hearts"
, squid, onions and octopus ”(artichoke heart, squid, onion,
And octopus), and thus those containing recipes that have the contained word in the material, are likely to be returned or directed to the user. The engine has obtained a match between the two contained words "octopus" and "heart", so this document has a high degree of match based on statistical measures including, for example, proximity and logical operators. Will be determined. In fact, this document is actually completely irrelevant to the query.

[0006] Various techniques are taught in the art for extracting syntactic phrase elements as the head-modifier pairs of an unlabelled relation. These elements are then indexed as terms (usually without internal structure) in a conventional statistical vector-space model.

An example of such a technique is described in JLFagan (JL Fagan), “Experiments
in Automatic Phrase Indexing for Document Retrieval: A Comparison of Syn
Tactics and Non-Syntactic Methods "(Experiment in Automatic Phrase Indexing for Document Retrieval: Comparison of Syntactic and Non-Syntactic Methods) (Cornell University, PhD Thesis, 1988, i-261). Specifically, this method analyzes natural language processing of an English sentence and extracts syntactic phrase components.Here, these phrase components are treated as terms, and a statistical vector- Index in the index using a spatial model. During the search, the user enters a natural language query, under which natural language processing is performed for analysis and stored in the index. Extract syntactic phrase components that are similar to an element, then try to match the syntactic phrase components from the query with the elements stored in the index. , Comparing this purely syntactic approach to a statistical approach that uses stochastic methods to identify elements in syntactic phrases. It concludes that no improvement can be gained and that the slight accuracy improvements that natural language processing sometimes produces do not justify the large processing costs associated with natural language processing.

Another approach based on such a syntax is described in T. Strzalkowski (T. Strzalkowski)
) “Natural Language Information Retrieval: TIPSTER-2 Final Report” ( Proceedings of Advances in Text Processing: TipsterProgram Phase 2 , DARPA, May 6-8, 1996, Tysons
Corner, Virginia, pp. 143-148 (hereinafter referred to as "DARPA dissertation");
“Natural Language Information” by Strzalkowski
Retrieval ”(natural language information retrieval) ( Information Processing and Management , V
ol. 31, No. 3, 1995, pp. 397-417), in the context of using natural language processing to select appropriate terms for inclusion in search queries.
Although this approach provides a theoretical promise, in pages 147-8 of the DARPA paper, the authors have found that due to the elaborate processing required to implement the underlying natural language techniques, It has now been concluded that it is impractical.

[...] NLP (Natural Language Processing) techniques that meet our requirements (and are at least considered to be close to these requirements) are not yet quite sophisticated in their ability to handle natural language text. It is important to remember, especially the advanced processing involved in conceptual structuring, logical forms, etc., is computationally inaccessible.
It can also be assumed that these higher techniques prove to be even more effective as they address the problem of limited representation. However, experimental evidence is sparse, and is necessarily limited to small tests. Yet another probability-based approach to this Lord is B. Katz's "Annotating
the World Wide Web using Natural Language ”
Wide Web Annotation) ( Conference Proceedings of RIAO97, Computer-assisted Information Searching in Internet, McGill University, Quebec, Canada , June 25-27, 1997 , Vol. Announcement"
). By creating the subject-verb-object expression while preserving the Internet structure, as described in Katz's announcement, it is possible to cope with fine syntactical alterations during the search.

[0010] These syntactic approaches have not provided much improvement. In other words, because there was no prospect of improving the natural language processing systems available at the time,
We moved from trying to directly improve the accuracy and recovery of initial query results to improving the user interface. That is, specifically, a method of refining a query based on an interactive process with the user, such as a response to the search result by the user “selecting the closest one”, and an appropriate cluster. An improvement of the user interface by a method of visualizing query results, including displaying the results in a query.

While these improvements, while useful in themselves, the accuracy improvements achievable by these improvements are still low enough to be disappointing, they completely eliminate the frustration of users specific to keyword searches. Is clearly insufficient. That is, the user is required to manually sift a relatively large set of documents with only sparse associated responses. SUMMARY OF THE INVENTION According to one exemplary embodiment, the present invention provides a method and apparatus for determining similarity between textual inputs. For a first text input, a first set of logical forms is obtained, and for a second text input, a second set of logical forms is obtained. The first and second sets of logical forms are compared and a similarity between the first and second text inputs is determined based on the comparison.

Broadly, this process involves the generation, comparison, and optional weighting of a matching logical form associated with the first and second text inputs, respectively. A logical form is a directed graph in which words representing text of any size are linked by a labeled relation. That is, the logical forms are structural relationships (ie, syntactic and semantic relationships) between important words in the input string,
In particular, the purpose and / or the relation between adjuncts are described. This depiction can take various specific forms, such as a logical form graph or any subgraph thereof, the latter comprising, for example, a list of logical form triples, each of which is illustratively , "Word_relation_word", any of which can be used with the present invention.

According to one aspect of the present invention, natural language processing is performed on each text input with respect to morphology, syntax and meaning as an example, and finally an appropriate logical form is generated for each sentence in each text input. Next, the set of logical forms for the first text input is compared to the set of logical forms associated with the second text input to determine a match between the logical forms.

Similarity, as used herein, means that two text inputs obtain a measure of how close they are in terms of semantic and syntactic structure and / or lexical meaning, or both. .

According to one exemplary application, the information retrieval system is based in part on natural language processing. Using the semantic information, more information about one or both of the search target document and the query is fetched to achieve higher performance or higher accuracy. Generally, such systems attempt to match the semantic content of a first text input (such as a query) with that of a second text input (such as a document to be searched) using natural language processing techniques. Such a system represents a significant advance in the art, especially with respect to obtaining improved accuracy in the information retrieval process.

Specifically, the input query is converted into one or more logical forms, and the documents retrieved by the search engine are also converted into the logical form. Compare the logical form for the query with the logical form for the document. If the logical form of the document exactly matches the logical form corresponding to the query, the document is ranked and presented to the user.

According to another aspect of the invention, the stringency associated with the matching process described above is mitigated by using paraphrased logic forms. For example, in information retrieval applications,
It may be necessary to reduce the rigor of the filtration process and prevent the destruction of related documents. For example, at times, documents in a collection that are correctly included in a query (or keyword search) may be incorrectly discarded. This can occur when the keywords from the query are present in the document but not in the exact syntactic / semantic relationship required by the logical form generated for the query. A document that is erroneously discarded in this way can be illustrated by the following example. Although this example discusses the logical form triad, it should be noted that other subgraphs of the logical form can be used as well. Assume the query is as follows: (How do spiders eat their prey?) The logical form triple that occurs for this query is eat; Dsub; spider eat; Dobj; victim.

Related documents may include the text "Many spiders consume their victims ..." (many spiders eat their prey ...). The logical form generated for this sentence is as follows. consume; Dsub; spider consume; Dobj; victim Since the logical form triple corresponding to this document does not exactly match any of the logical form triples corresponding to the query, If so, destroy this document.

In addition, it may be necessary to discard unrelated documents that would otherwise be presented to the user unless discarded. For example, a class of logical forms may frequently appear in documents in the large data storage device being searched. Such logical forms may be common in the query, regardless of the subject of the query. For example,
Suppose the query is Tell me about dogs.

One logical form triple generated for this query is tell; Dobj; me.

This can of course also appear in many documents that are not related to dogs. Therefore, such an unrelated document is presented to the user. That is, according to one aspect of the invention, one or both sets of logical forms (for one or both text inputs) are modified by paraphrasing the set of logical forms or suppressing certain logical forms. One or both sets of modified logical forms are used in the matching process.

In an exemplary information retrieval system, the system filters documents in a retrieved document set from a document storage device in response to a query. The system obtains a first set of logical forms based on the query and one selected from the documents in the set of documents. The system obtains a second set of logical forms based on the query and another one of the documents in the document set. Next, the system modifies the first logical form using natural language processing techniques to obtain a modified logical form set. The system filters documents in the document set based on a predetermined relationship between the modified set of logical forms and the second set of logical forms.

According to one aspect of the present invention, a first paraphrase logical form set indicating a paraphrase of the first logical form set is obtained using a natural language processing technique. According to another aspect of the present invention, a natural language processing technique suppresses a first predetermined class of logical forms to obtain a first set of suppressed logical forms. Next, filtering is performed based on the paraphrase logical form set and / or the suppression logical form set.

In one embodiment, a query is received and a query logical form is calculated based on the query. Execute a query and search for documents based on this query. The logical form is
For each document searched or calculated, a search is made from the data storage device. Suppress frequent query logical forms and calculate paraphrase logical forms based on the query logical forms. The paraphrase query logical form is checked against the document logical form. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Overview The present invention utilizes natural language processing techniques to create a set of logical forms corresponding to a first text input and a second text input. The present invention determines the similarity between the first and second text inputs based on the comparison of the set of logical forms. According to another aspect of the invention, one or both of the logical form sets are modified, such as by paraphrasing or suppressing certain logical forms. Although the present invention is envisioned for use in a wide variety of applications, it is described herein primarily in terms of information retrieval for illustrative purposes only.

In an information retrieval embodiment, the invention creates a set of logical forms corresponding to the input query and a set of logical forms corresponding to the set of documents returned in response to the input query. The present invention also utilizes natural language processing techniques to change the logical form corresponding to either the query or the document set, or both. In one embodiment, the modified logical form is expanded to include a paraphrase. In another embodiment, the modified logical form was processed and found not to be useful in discriminating between various documents,
Suppress the logical form of a given class. By altering the logical form in this manner, the present invention has reduced the severity associated with matching techniques, and thus has improved both accuracy and recovery in the information retrieval process.

It should be noted that this discussion proceeds, in part, with reference to the logical form triples that have the form, syntactic or semantic relationship indicated by the word, and other words. is there. However, the present invention is directed to other subgraphs.
It is also assumed that the logical form of ()) can be used as well, and all of them will be collectively referred to as a logical form here.

After reviewing the following discussion, many applications and almost any information retrieval system will readily utilize the teachings of the present invention to determine whether the search engine used therein is a conventional statistical engine or not. Regardless, those skilled in the art will clearly recognize that increased accuracy of the engine is possible. In addition, our invention is directed to any form of mass data storage, such as magnetic, optical (e.g., CD-ROM), or any other medium in which more textual information is present, whether stored on other media. It can be used to improve accuracy when retrieving text information from a database, regardless of a particular language, for example, English, Spanish, German, etc.

With this in mind, FIG. 1 shows a top-level block diagram of an information retrieval system 5 that utilizes our invention. The system 5 is formed by a conventional search engine 20, for example, a statistical search engine 20 based on keywords, followed by a processor 30. Processor 30 utilizes the natural language processing techniques of our invention to filter and re-rank the documents generated by engine 20, as described below, and to respond to user-supplied queries at other times. A search document set having higher relevance than obtained is ordered and generated.

Specifically, in operation, a user supplies a search query to the system 5. Queries are in the form of full text (commonly referred to as “literals”), and take full advantage of their semantic content through natural language processing, thereby allowing them to be obtained only by the engine 20. The accuracy should be improved. The system 5 applies this query to both the engine 20 and the processor 30. In response to the query, the engine 20 searches the entire data set 10 of stored documents and generates a search document set therefrom. Next, this document set (also referred to herein as the “output document set”) is applied as an input to the processor 30 as symbolized by line 25. Within the processor 30, each of the documents in the collection is exemplarily subjected to natural language processing in morphological, semantic and logical forms, as discussed in detail below, and for each sentence in the document. Generate a logical form. Each of these logical forms for a sentence is associated with, for example, a semantic relationship between words in a linguistic phrase within the sentence, ie, an argument and an adjunct.
t) Encode the structure. Processor 30 also analyzes the query and generates a corresponding set of logical forms. Next, processor 30 compares the morphological set for the query to the logical morphological set associated with each of the documents in the ensemble to ascertain any matches between the logical morphology in the query ensemble and the logical morphology for each document. . Documents that do not match will be excluded from further consideration. Each residual document containing at least one logical form that matches the query logical form is kept and a score is determined empirically by the processor 30. As discussed below, predetermined weights are assigned to each of the different relationship types, ie, deep subjects, deep objects, functional words, etc., that can occur within the logical form triad. The total weight (i.e., score) of each such document is, for example, the sum of the weights of all one matched triple.
That is, the same matching triples are ignored. Finally, the processor 30 arranges the held documents in a rank order based on the score and presents them to the user. Typically, starting with the document with the highest score, it is grouped by a predetermined number, for example, 5 or 10.

Since the system 5 is so versatile and can be adapted to a wide variety of different applications, the use of our invention in one exemplary situation to simplify the following discussion Will be discussed. The situation is an information retrieval system that employs a conventional keyword-based statistical Internet search engine and searches for stored records of English documents indexed in datasets from the World Wide Web. . In general, each such record contains predefined information for the corresponding document, as specified below. In other search engines, a record may include the entire document itself. The following discussion
The present invention is intended for use in conjunction with a conventional Internet search engine that contains certain information about a corresponding document and retrieves records containing web addresses where the document can be found, but is not intended to be comprehensive. In short, the final item that the engine searches is actually a document. However, in general, actually accessing a document from the Web uses an intermediate process using the address. After reviewing the following description, those skilled in the art will readily recognize how easily the present invention can be adapted for use in any other information retrieval application.

FIG. 2 is a high-level block diagram of a specific embodiment of our invention for use in connection with an Internet search engine. Our invention will be discussed primarily in the context of this particular embodiment. As shown, the system 200 is connected via a network connection 205 to a network 210 (here, the Internet, but any other such network, such as, for example, an intranet) can be used instead. And a client personal computer (PC) connected to the server 220 through the network connection unit 215
, Including a computer system 300. The server typically includes a computer 222. The computer 222 operates (hosts) an Internet search engine represented by, for example, the ALTA VISTA search engine (ALTA V
ISTA is a registered trademark of Digital Equipment Corporation of Maynard, Mass., Typically data of document records indexed by search engines and accessible on the Internet through the World Wide Web. Mass data storage device 22 as a set
7 is connected. Each such record typically contains (a) a web address (commonly referred to as a uniform resource locator--URL) at which the corresponding document can be accessed by a web browser; appear,
A default content word, and in some engines, a default content word with the relative address of each such word relative to other content words in the document; (c) often only a few lines of the document. Contains a brief summary or the first few lines of the document, and possibly (d) a description of the document, given in its Hypertext Markup Language (HTML) description field.

A user facing computer system 300 can, for example, obtain an associated web browser running on this system (available from Microsoft Corporation) to include the teachings of our invention. (Such as based on the "Internet Explorer" version 4.0 browser), appropriately modified to the server 220 and, if specified, the search engine 222 running there.
Establish an Internet connection to Thereafter, the user, here the line 201
Enter the query into the browser as if symbolized by. The browser, on the other hand, sends queries to the search engine 225 through the system and through an Internet connection to the server 220. Then, the search engine returns the data set 22
7. The query is processed for the document records stored in 7 and a set of records searched by the engine for documents determined to be relevant to the query is generated. Engine 2
25 actually indexes the document to form a document record,
27, and the actual analysis that the engine performs to select any of such stored document records is both irrelevant to the present invention and will not be further described in any of these aspects. Will not be discussed. Suffice it to say that in response to the query, engine 225 returns the set of search document records to web browser 420 via the Internet connection. Browser 42
0 analyzes the query and generates its corresponding logical form triplet while engine 225 is searching for the document, simultaneously and / or subsequently. Once the search engine has completed its search, retrieved the set of document records and served the set to the browser, the web server associated with the corresponding document itself (ie, forming the output document set) itself (A dataset associated therewith and collectively forming a "repository" of stored documents. Such a repository may be a single entity, such as in a self-contained CD-ROM based data retrieval application). Can be a data set). Meanwhile, the browser then analyzes each of the accessed documents (i.e., in the output document set) and forms a corresponding set of logical form triples for each such document. Thereafter, as discussed in detail below, the browser 420 determines a score for each document having such a match based on the logical form triples that match between the query and the retrieved document, and symbolizes it at line 203. As such, these documents are presented to the user. Documents are typically ranked in descending score order as a predefined small group of documents having the highest ranking. Then, if the user selects through the browser, the next such group follows, and so on until the user has inspected a sufficient number of such presented documents. FIG. 2 illustrates, by way of example, our invention as obtaining document records and documents from a remote server utilizing a network connection, but our invention is not so limited. As discussed in detail below in connection with FIG. 9A, both the search application and our invention run on a common computer, i.e., a local PC, and the attached data set is also stored on, for example, a CD-ROM or other suitable medium. And if accessible there, such a network-like connection is not required.

FIG. 3 and the related discussion describe a suitable computing environment in which the invention can be implemented.
It is intended to provide a brief, general description. The description of the present invention will be made at least in part in the general context of computer-executable instructions, such as program modules, being executed by a personal computer, but this is not required. Generally, program modules include routine programs, objects, components, data structures, etc., that perform particular tasks or
Or implement a specific abstract data type. In addition, handheld devices,
Multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframes
One skilled in the art will recognize that the invention may be practiced with other computer system configurations, including computers and the like. Also, the present invention
The present invention can also be implemented in a distributed computing environment in which remote processing devices linked through a communication network execute tasks. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

Referring to FIG. 3, an exemplary system embodying the present invention includes an arithmetic unit 321 (1
A general purpose computing device in the form of a conventional personal computer 320, a system memory 322, and various system components, including the system memory, to a computing unit 321 (which may include one or more processors). -Includes bus 323. System bus 323 may be any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. The system memory includes a read only memory (ROM) 324 and a random access memory (ROM).
A memory (RAM) 325 is included. A basic input / output 326 (BIOS) containing basic routines to help transfer information between elements within personal computer 320, such as during start-up, is stored in ROM 324. Further, the personal computer 320 includes a hard disk drive 327 for reading from and writing to a hard disk (not shown), and a removable magnetic disk 32.
Also included is a magnetic disk drive 328 for reading or writing to and from an optical disk drive 330 for reading or writing to a removable optical disk 331 such as a CD ROM or other optical media. The hard disk drive 327, magnetic disk drive 328, and optical disk drive 330 are connected to the system bus by a hard disk drive interface 332, a magnetic disk drive interface 333, and an optical drive interface 334, respectively. 323 is connected. These drives and associated computer-readable media allow nonvolatile storage of computer-readable instructions, data structures, program modules, and other data for personal computer 320.
In the example environment described here, a hard disk, a removable magnetic disk 3
29 and a removable optical disk 331, a magnetic cassette, a flash memory card, a digital video disk, a Bernoulli cartridge, a random access memory (RAM), and a read only memory (RO).
One skilled in the art will recognize that other types of computer readable media capable of storing computer accessible data, such as M), can be used in the exemplary operating environment.

A number of program modules can be stored on the hard disk, magnetic disk 329, optical disk 311, ROM 324 or RAM 325, including an operating system 335, one or more application programs 336, and other programs. -Contains module 337 and program data 338. The user can operate the keyboard 340 and the pointing device 342.
Commands and information can be input to the personal computer 320 through an input device such as. Other input devices (not shown) include a microphone, joystick, game pad, satellite parabolic reflector (satellite)
dish), a scanner and the like. These and other input devices often connect to the computing device 321 via the serial port interface 346. The serial port interface 346 is
It is coupled to a bus, but can also be connected to other interfaces, such as a parallel port, a game port or a universal serial bus (USB). The monitor 347 or other type of display device may also
It is connected to the system bus 323 via an interface such as an adapter 348. In addition to the monitor 347, personal computers typically include
Other peripheral output devices (not shown), such as speakers and printers, may also be included.

Personal computer 320 can also operate in a networked environment using logical connections to one or more remote computers, such as remote computer 349. Remote computer 349 may be any other personal computer, server, router, network PC, peer device, or other node, typically personal computer 32.
0 may include many or all of the elements described above with respect to FIG.
Shows only the memory storage device 350. The logical connections shown in FIG. 1 include a local area network (LAN) 351 and a wide area network (LAN).
WAN) 352. Such networking environments are common to offices, enterprise-wide computer network intranets, and the Internet.

When used in a LAN network processing environment, the personal computer 320 connects to a local area network 351 via a network interface or adapter 353. When used in a WAN networking environment, the personal computer 320 typically has a modem 354
Or other means for establishing communication over a wide area network 352 such as the Internet. The modem 354 may be internal or external, but is connected to the system bus 323 via a serial port interface 346. In a networked environment, program modules depicted relative to the personal computer 320, or portions thereof, may be stored in the remote memory storage device. It will be appreciated that the network connections shown are exemplary and other means of establishing a communications link between the computers may be used.

FIG. 4 shows a top-level block diagram of an application program 400 executed inside the computer 300 shown in FIG. These programs, to the extent relevant to the present invention, comprise a search process 600 (discussed in detail below in connection with FIGS. 6A and 6B) to implement our invention, as shown in FIG. Was
Web browser 420 is included. Web browser and, for example, ALTA VI
Assuming that an Internet connection has been established with a statistical search engine selected by the user, such as a STA search engine, the user may then symbolize at line 422 shown in FIG. Process 600 contains full text (
"Literals") provide search queries. This process forwards the query through the web browser to the search engine, as symbolized at line 426. In addition, although not specifically shown, the process 600 also internally analyzes the query to generate a corresponding logical form triple and then stores it locally within the computer 300. In response to the query, the search engine returns to line 43
The statistically retrieved document record set is symbolized by
Supply 0. Each of these records, as noted above, can access the document, a web address, i.e., a URL, and the remote web server where the document is located can be transmitted via the Internet to the computer containing the document. Include the appropriate command (s) needed to fully download the file. Once process 600 has received all records,
The process then sends the appropriate commands via the web browser 420 and as symbolized at line 436 to access and download all the documents specified by the record (ie, form the output document set). Do). These documents are then accessed sequentially from the corresponding web server and downloaded to web browser 420, specifically process 600, as symbolized at line 442. Once these documents have been downloaded, the process 600 analyzes each such document and generates its logical form triad and stores it locally.
Thereafter, by comparing the logical form triples for the query to the logical form triples for each document, the process 600 determines a score for each document containing at least one matching logical form triple and based on those scores. Web documents to present these individual documents to users in ranked order by ranking document scores by group in descending order, as symbolized by line 446, and finally ranking these individual documents. Instructs the browser 400. The browser 400 is connected to the display 3
800 (see FIG. 3) generates the appropriate select button on the screen through which the user selects by appropriately "clicking" on it using his (her) mouse, as desired. Each successive document group can be displayed.

At this point, in order to determine the semantic information and to maximize the usefulness of the logical form in storing and encoding, deviating from discussing the process of implementing our invention, the logic used in the present invention The forms and logical forms triples will be illustrated and described to the extent relevant, and will provide a brief summary of how to generate them.

In a broad sense, a logical form is a directed graph in which words representing text of any arbitrary size are linked in a classification relationship. A logical form describes a semantic relationship between important words in a phrase and may include its nypernym and / or synonyms. As discussed and shown in FIGS. 5A-5D, a logical form may utilize any one of a number of different forms, for example, a logical form graph or any subgraph, such as a list of logical form triples. Can be. Each of the triples is in the form of “word-relation-word” as an example. Our invention, when specified and embodied, generates and compares logical form triples, but our invention allows for the depiction of semantic relationships between words, as noted above. Any other format is readily available. As used herein, all of them are included in the term logical form.

The logical form triads and their structure can be best understood through a series of increasingly complex text examples, so first consider FIG. 5A. This figure shows an example input string 510, specifically, “The octopus has three hear.
For the sentence "ts." (octopus has three hearts), a logical form graph 515 and a logical form triple 525 are shown.

In general, in one exemplary embodiment, to generate a logical form triple in an exemplary input string, eg, input string 510, the string is first parsed and broken down into its constituent words. Then, for each such word, using a predetermined record in the stored vocabulary (not to be confused with a document record used by a search engine), these grammatical rules are used to construct these words. The records corresponding to the words themselves are incorporated into a larger structure or analysis, and then again joined in sequence by predefined grammar rules to form a larger structure, such as a syntactic parse tree. Next, a logical form graph is constructed from the parse tree. Whether individual rules are applicable to individual constituent sets is governed in part by the presence or absence of certain corresponding attributes and their values in word records. Next, the logical form graph is converted into a series of logical form triples. By way of example, our invention uses a vocabulary that has about 165,000 head word entries. This vocabulary includes, for example, pronouns, conjunctions, verbs, nouns, functional words,
And various classes of words, such as quantifiers, that define the syntactic and semantic properties that are unique to the words in the input string so that a parse tree of the input string can be constructed. Obviously, the logical form (and, for that matter, some other representation, such as a logical form triad or a logical form graph within a logical form capable of describing semantic relationships) can be pre-computed. On the other hand, the corresponding document is indexed and recorded in, for example, a record for the document, and once the document is searched, it can be subsequently accessed and used without being calculated later. Using such pre-computation and storage, as will be done in another embodiment of our invention discussed in detail below in connection with FIGS. 10-13B, to process any document retrieved according to our invention Has the advantage of dramatically reducing the amount of natural language processing required and therefore the execution time associated therewith.

That is, in an exemplary embodiment, for an input string such as a sentence 510 shown in FIG. 5A, a morphology is first determined by using a record defined in the vocabulary for each constituent word. And a so-called "stem" (or "base") form is generated for it. Stem forms are used in turn to normalize different word forms, eg, verb tenses and noun singular variations, to a common morphological shape for use by the parser. Once the stem shape has been generated, the parser parses the input string using the grammar rules and the attributes in the constituent word records, and generates a syntactic parse tree for it. This tree shows the structure of the input string, i.e., each word or phrase in the input string, for example, the noun phrase "The octopus", its corresponding grammatical function category, for example, NP for the noun phrase, and Draw a link (several links) to each syntactically related word or phrase. For the example sentence 510, its associated syntactic parse tree is as follows:

[0044]

[Table 1]

The starting node, located in the upper left corner of the tree, defines the type of the input string to be parsed. The sentence type is “DECL” in the declarative sentence (here) and “IMP” in the command sentence.
R "and the query include" QUES. "Displayed vertically to the right and below the start node is a first level analysis, which has a head node denoted by an asterisk. This is typically a main verb (here, the word “has”), a premodifier (here, the noun phrase “The octopus”), followed by a postmodifier (noun, The phrase "three hearts") follows.Each leaf of the tree contains a lexical term or punctuation, where as labels "NP" specifies a noun phrase and "CHAR" indicates punctuation.

Next, the syntactic parse tree is further processed using a different set of rules to produce a logical form graph, such as graph 515 for input string 510. The process of generating a logical morphological graph involves extracting the underlying structure from a syntactic analysis of the input string. The logical form graph includes semantic relationships between them and words defined as having the functional nature of the relationships. The “deep” cases and functional roles used to categorize different semantic relationships include:

[0047]

TABLE 2 Dsub-deep subject Dind-deep indirect object Dobj-deep object Dnom-deep predicative subject Dcmp-deep object complement Table 2 To identify all semantic relationships in the input string , Test each node in the parse tree of the string. In addition to the above relationships, other semantic roles are used, for example:

[0048]

TABLE 3 PRED-predicate PTCL-invariant in two-part verb Ops-functional word, eg, adjective Nadj-adjective that modifies a noun Dadj-predicative adjective PROPS-other unspecified clauses Designated Modifiers MODS-Other unspecified modifiers that are not clauses Table 3 Additional semantic labels are defined similarly. For example,

[0049]

In any case, the result of such an analysis on the input string 510 is a logical form graph 515. Semantic relationships within the input string (eg, "Oc
The words that indicate “topus” and “Have” are connected to each other and specify the relationship between them as a link attribute (eg, Dsub). This graph is represented by a graph 515 for the input string 510. Captures the argument and the structure of adjuncts for each input string.In particular, logical morphological analysis provides features and structures that describe functional words, such as prepositions and articles, in the graph. In one embodiment, logical form analysis also resolves anaphora, ie, defines the correct antecedent relationship between, for example, a pronoun and a correlated noun phrase. Detect and delineate the proper functional relationship to omissions. In that case, the corresponding logical form triad is simply read from the logical form graph and stored as a set, each triad containing two node words and their In the example input string 510, the logical form triad 525 was derived from the processing graph 515. Here, the logical form triad 525 is represented by three individual triads. , And jointly conveys semantic information unique to the input string 510.

Similarly, as shown in FIGS. 5B-5D, in the input strings 530, 550 and 570, the sentence “The octopus has three hearts and
two lungs. "(The octopus has three hearts and two lungs.)," The octopus has
Logical forms for "three hearts and it can swim." (the octopus has three hearts and can swim) and "I like shark fin soup bowls." Graphs 535, 555 and 575 and logical form triples 540, 560 and 580 are obtained, respectively.

Apart from the conventional method, there are three logical form structures required for additional natural language processing in order to correctly generate all the logical form triples. , Including a traditional "graph walk". "
In the example of the sentence "The octopus has three hearts and two lungs", that is, in the case of coordination, as in the input string 530, for each word, its semantic relationship, and the value of the component to be adjusted, a logical Create a morphology triad: According to the "special" graph walk, in Figure 540 we see that there are two logical morphology triples "have-Dobj-heart" and "have-Dobj-lung". When only the conventional graph walk is used, one logical form triad "have-Dobj-
and ”. Similarly,“ The octopus has three hearts and it can s
wim ", as in the input string 550,
For elements with (erent) (Refs), a logical form triple is created for each of the words, their semantic relationships, and the value of the Ref attribute, in addition to the conventional triplet where the graph walk occurs. According to this special graph walk, in the triple 560, in addition to the conventional logical form triple "swim-Dsub-it", there is a logical form triple "swim-Dsub-octopus". Finally, "I like shark fin
In the example of the sentence "sou bowls", ie, for components having a noun modifier, as in the input string 570, to represent the possible internal structure of the noun complex:
Create additional logical form triples. The traditional graph walk creates logical form triads "bowl-Mods-shark", "bowl-Mods-fin" and "bowl-Mods-soup",
Reflected a possible internal structure [[shark] [fin] [sup] bowl]. In the special graph walk, the following possible internal structures [[shark fin]
[Soop] bowl] and [[shark] [fin soup] bow
l], and [[shark [fin] soop] bowl], respectively, with the following additional logical form triples “fin-Mods-shark”, “soup-Mods-fin”, and “soup-Mods”: -shark ”is created.

Since the specific details of morphological, syntactic, and logical formalism are not relevant to the present invention, all further details regarding them are omitted. However, for further details in this regard, filed Jun. 28, 1996 and filed with serial no.
“Method and System for Computing Semantic L granted No. 674,610
ogical Forms from Syntax Trees ”and“ Serial Number 08/8, filed March 7, 1997, specifically filed on March 7, 1997.
86,814, "Information Retrieval Utilizaing Sematnic Re
Reference is made to the reader with co-pending U.S. patents entitled "presentation of Text" (information retrieval using semantic representation of text), both of which are assigned to the assignee of the present application. It shall be included in the present application.

With this summary of the logical form and its structure in mind, we will return to our discussion of the process of practicing the present invention. The flowcharts of our invention utilized in the search process 600, as used in the specific embodiment of our invention shown in FIGS. 2, 3 and 4, are collectively shown in FIGS. 6A and 6B.
Shown in The correct alignment of the drawings with respect to these figures is shown in FIG. With the exception of the operation shown in dashed block 225, the remaining operations shown in these figures are performed within a computer system, for example, client PC 300 (see FIGS. 2 and 3), and specifically within web browser 420. I do. For ease of understanding, the reader may simultaneously refer to FIGS. 2, 3, 6A and 6B throughout the following discussion.

Upon starting the process 600, execution first proceeds to block 605. Executing this block causes the user to access the full text (
(Literal) prompts for a query. The query can be a single question (for example, "Is there an air-conditioned hotel in Bali?) Or a single sentence (for example, contact information about the entire fireworks festival in Seattle during July. Please), or part of a sentence (eg, "clothing in Ecuador"). Once this query is obtained, execution proceeds via path 607 to block 610
, And through the path 643 to the path 645. Block 6
Execution of 45 calls the NLP routine 700 to analyze the query and build its corresponding logical form triplet set and store it locally. Upon execution of block 610, a full-text query is transmitted from a web browser 420 via an Internet connection, such as an engine 225 located on a server 220, as symbolized by a dashed line 615, to a remote search engine. Send to At this point, block 625 is executed by the search engine to retrieve the document record set in response to the query. Once this set is formed, it is returned from the remote server to the computer system 300, and specifically to the web browser 420 running thereon, as symbolized by the dashed line 630.
Thereafter, block 635 is executed to receive the record set, extract a URL from the record for each record, access a web site at the URL,
Download the relevant file containing the document corresponding to the record from there.
Once all the documents have been downloaded, block 640 is executed. For each such document, the block first extracts all text from the document. The text includes any text located within the HTML tags associated with the document. Thereafter, in order to simplify the natural language processing that operates on only one document at a time, the text of each document is decomposed into a text file by a conventional document breaker. In this case, each sentence (or question) occupies a separate line on the file. Thereafter, block 640 repeatedly calls the NLP routine 700 (discussed in detail below in connection with FIG. 7) for each line of text in the document, analyzes each of these documents, and analyzes each line of text in the document. A corresponding logical form triad set is constructed for each and stored locally. Although the operations at block 645 have been discussed as essentially performing in parallel with the operations at blocks 610, 635 and 640, based on considerations of actual implementation, the operations at the formation block may be performed at blocks 610, 635 and It is also possible to perform continuously either before or after the operation in 640. Or,
As in the case of another embodiment of our invention discussed below in connection with FIGS. 10-13B, the logical form triad for each document is pre-computed and the document is searched for later access and use. It is also possible to store in. In that case, these triads can simply be accessed without calculating during document retrieval. In this case, the triad may in some way characterize the stored document:
Alternatively, for example, it is also possible to store the document as a separate entry in either a record of the document or a data set including the document.

In any case, returning to the process 600 shown in FIGS. 6A and 6B, once the logical form triad set has been constructed and completely stored for the query and for each of the search documents in the output document set. Thereafter, block 650 is executed. This block compares each of the logical form triples in the query with each of the logical form triples for each of the search documents, and matches any of the triples in the query with any of the triples in any of the documents. For a match between An example form of a match is for a node word:
Perfect match between the two triads and the relationship in these triads (iden
tical match). That is, an exemplary pair of logical forms, wordl
In a-relation1-word2a and word1b-relation2-word2b, a match is found only if the node words word1a and word1b are the same, node words word2a and word2b are the same, and relation1 and relation2 are the same. Occurs. If all three elements of one triplet do not completely match the corresponding elements of another triplet, then the two triplets will not match. Once block 650 is completed, block 655 is executed to retrieve a search document that does not show a matching triple, ie,
Discard all search documents that do not have a triple that matches any of the triples in the query.
Thereafter, block 660 is executed. Block 660 indicates that for all remaining documents, the associated triad associated type (existing for each of these documents)
(Several types) and their weights. That is, for each of the different relation types that can occur in the logical form triad, as shown in the table 800 of FIG.
Assign corresponding weights. For example, as shown, the example relationships D obj, D s
Predetermined fixed numerical weights 100, 75, 10 and 10 may be assigned to ub, Ops and Nadj, respectively. The weight reflects the relative importance attributed to the relationship in indicating an exact semantic match between the query and the document. The actual numerical values of these weights are usually defined empirically. As will be described in more detail below in connection with FIG. 8B, the score for each of the remaining documents is a default function of the weight for each one matching triple (ignoring all overlapping matching triples). As an example, here, a numerical sum is used. Once the documents have been weighted in this way, block 665 is executed to sort the documents in rank order in descending score order. Finally, block 670 is executed to display the documents, typically in rank order, for a typical set of five or ten small documents that has the highest score. Thereafter, the user ranks the document into the computer system (client PC) 300 by, for example, appropriately "clicking" his (her) mouse on the corresponding button displayed by the web browser 420. , And so on until all the ranked documents have been fully tested by the user in succession. At the end of the test, process 60
0 is complete.

FIG. 7 shows a flowchart of the NLD routine 700. This routine, given a line of input text, whether it is a query, a sentence in a document, or a fragment of text, constructs a corresponding logical form triple.

That is, upon entering the routine 700, block 710 is first executed to process a line of input text and generate a logical form graph, such as the exemplary graph 515 shown in FIG. 5A. This processing includes, by way of example, morphological and semantic processing,
Generate a parse tree and then compute a logical form graph from this. afterwards,
As shown in FIG. 7, block 720 is executed to extract (read out) the corresponding set of logical form triples from the graph. Once this has been done, block 730 is executed to generate each such logical form triple as a separate and differently formatted text string. Finally, block 740 is executed to store the line of input text and the set of logical form triples for that line as a series of formatted text strings in a dataset (or database). Once this set has been completely stored, execution exits block 700. Alternatively, instead of a logical form triad, a different representation, eg, a logical form graph associated with a logical form, is used with our invention. If so, it is easy to modify blocks 720 and 730 to generate that particular form as a formatted string, and block 740 stores that form in a data set instead of a logical form triad I do.

To fully understand how our invention compares and weighs matching logical form triples, and further ranks corresponding documents, consider FIG. 8B. This figure schematically illustrates the logical form triad comparison, document scoring, ranking, and selection process in accordance with the teachings of our invention. 6A and 6B for the example query and the three example search document collections.
Are performed in blocks 650, 660, 665 and 670 shown in FIG. For illustrative purposes, a user supplies a full text query 810 to our search system, and the query is "How many hearts does an octopus have?" Assume. Also assume that in response to this query, three documents 820 were eventually retrieved by the statistical search engine. Of these documents, the first (named Document 1) is the artichoke core (ar
It is a recipe containing tichoke heart) and octopus. The second document (named Document 2) is a dissertation on octopus. The third document (named Document 3) is a dissertation on deer. Transform these three documents and queries into their constituent logical forms. The process for this is generally represented by "NLP" (natural language processing). The queries and the resulting logical form triples for Document 1, Document 2 and Document 3 are provided in blocks 830, 840, 850 and 860, respectively.

Once these triads have been defined in this way, then the dashed lines 845, 855,
The logical form triples for the query are sequentially compared to the logical form triples for Document 1, Document 2 and Document 3, respectively, as symbolized at and 865, and any document is identified as any logical form in the query. Check to see if any triads that match the morphological triad are included. Documents that do not contain such a matching triple, as in Document 1, are discarded, and are therefore not considered further. On the other hand, document 2
And Document 3 contains a matching triple. That is, the document 2 includes three such triples, for example, “HAVE-Dsub-OCTOPUS” and “HAVE-Dsub-H” related to one sentence.
EART ”and“ HAVE-Dsub-OCTOPUS ”by way of example, which are related to another sentence (these sentences are not specifically shown) .Of these triples, two are identical. That is, “HAVE-Dsub-OCTOPUS” is the same, For example, the score for this document is the numerical sum of the weights of all the matching triplets in the document. Also, ignore any triples that overlap.
An example of ranking the relative weights of the different types of relationships that can occur in a triad in descending order from their maximum weight to their minimum weight first shows a verb-object combination (
Doc), verb-subject combinations (Dsub), prepositions and functional words (eg, Ops), and finally modifiers (eg, Nadj). Such a weighting scheme is shown in the exemplary triple weighting table 800 shown in FIG. 8A. To simplify this figure, the table 800 does not include all of the different relationships that may occur within the logical form triad, but only those that are relevant to the triad shown in FIG. 8B. In this metric, the individual triads that contribute to that score in each document are indicated by a check ("re") mark. Of course, other than the previously selected one, another metric may be determined in advance for scoring the document and used. For example, multiply instead of add weights to increase document selectivity (discrimination), include multiple matches of the same type, and / or
Alternatively, a different method is defined in advance, such as excluding the weights of the triples other than those noted above, and the weights are added. In addition, for any document, the score may be calculated in some way by the node words in the triad itself in the document, or the frequency or semantic content of these node words in the document, the specific node words in the document. It is also possible to take into account the frequency or semantic content of the document, or the frequency of the particular logical form (or paraphrase) and / or the particular logical form triple in the document, and the length of the document .

Accordingly, the exemplary score determination metric described above, and table 800 of FIG. 8A
Assuming the weights described in, the score for document 2 is 175. this is,
It is associated with the first sentence in the document, formed by combining the weights for the first two triples shown in block 850, ie, 100 and 75. The third triple in this document is relevant to the second sentence, is posted in this block, and is ignored because it matches one of the other triples already in the document. Similarly, the score for document 3 is 100. In this particular document, as posted at block 860, the weight for the only matching triple is formed, here 100. Based on these scores, Document 2 is ranked higher than Document 3 and these documents are presented to the user in this order. Although not occurring here, if any two documents have the same score, they are ranked in the same order provided by a conventional statistical search engine and presented to the user in that order.

Obviously, the various parts of the process used to implement our invention may be located on a single computer or distributed among different computers that collectively form an information retrieval system. Those skilled in the art will readily recognize that this is also possible. In this regard, FIGS. 9A-9C each show three different embodiments of an information retrieval system that incorporates the teachings of the present invention.

One such alternative embodiment is shown in FIG. 9A. Here, all processing is PC
Located within a single local computer 910. In this case, the computer 910 operates a search engine through which the input documents are indexed and in response to a user-supplied full-text query, a data set (CD-ROM or other storage medium). (Such as those locally located there or accessible to the computer) to generate a set of search documents that ultimately forms the set of output documents. This computer is also responsible for the processing of our invention, analyzing both the query and each such document to generate its corresponding logical form triad set, then comparing the triad sets and discussing it earlier. The documents are selected, scored and ranked as before, and the results are finally presented to, for example, a local user who is or has access to it.

Another alternative embodiment is shown in FIG. 9B. This includes the specific contents shown in FIG. 2, and forms a search system with client PCs connected to a remote server via a network. Here, the client PC 920 is connected to the remote computer (server) 930 via the network connection 925. A user located at client PC 920 enters a full-text query, while the PC sends the full-text query over a network connection to a remote server. In addition, the client PC analyzes the query and generates a corresponding logical form triplet set. The server may, for example, operate a conventional statistical search engine, thus undertaking a statistical search in response to a query and generating a set of document records. The server then returns the record set, and eventually returns each document in the output document set to the client PC, either at the client's command or autonomously based on the capabilities of the search engine or associated software. I do. next,
The client PC analyzes each of the corresponding documents in the output document set and receives it to generate a logical form triad set for it. The client PC then compares the triad set appropriately, selects, scores, and ranks the documents as discussed above, and finally presents the results to the local user. Complete the process.

Another embodiment is shown in FIG. 9C. This embodiment employs the same physical hardware and network connections as in FIG.
20 accepts a full text query from a local user and further passes the query over a network connection 925 to a remote computer (server) 930.
Send to Instead of just running a traditional search engine, this server
We also perform natural language processing according to our invention. In this case, the server, not the client PC, properly analyzes the query and generates a corresponding set of logical form triples for it. The server also downloads each search document in the output document set, if necessary, and then analyzes each such document to generate a corresponding logical form triplet set. The server then appropriately compares the triad set for the query and the document, and performs document selection, scoring and ranking as discussed above. Once this ranking has been performed, the server 930 then sorts the remaining search documents in order of rank via the network connection 925 to the client PC 9.
20 and display it there. The server may transmit these documents in groups as directed by the user as specified above, or may transmit all of them in sequence and select between them for each group to display on the client PC. It is possible.

In addition, the remote computer (server) 930 can use the conventional search,
The processing need not be performed by a single computer that performs all of the natural language and related processing, and may be a distributed processing system as illustrated in FIG. 9D. In this case, the processing undertaken by this server is distributed among the individual servers therein. Here, the server 930 is formed by a front-end processor 940, and a message is transmitted via a connection 950 to a series of servers 960 (server 1, server 2).
,. . . , Server n). Each of these servers implements certain parts of our inventive process. In this regard, the server 1 can be used to index the input document into a data set on a mass data storage device and to make it searchable later. The server 2 implements a search engine, such as a conventional statistical engine, and responds to user-supplied and submitted queries by the front-end processor 940 from the mass data storage device for document records. Sets can be searched. These records are retrieved from the server 2 via the front end processor 940 for later processing, such as downloading each corresponding document in the output document set from the corresponding web site or database. , For example, to the server n. Also, the front end
Processor 940 sends the query to server n. Server n then properly analyzes the query and each document, generates a corresponding set of logical forms of triads, then compares the triad sets appropriately, selecting documents, scoring, and, as discussed above, The ranking is performed, and the ranked documents are processed by the front-end processor 94.
0 and is returned to the client PC 920 and displayed here in the order of rank. Of course, the various operations used in our inventive process may be any one of many alternatives.
Or static or dynamic, runtime and / or
Alternatively, distribution among the servers 960 is also possible according to other conditions occurring there. Further, server 930 is illustratively implemented in a known sysplex configuration, and has a shared direct access storage device (or other similar distributed multiprocessing environment) accessible therein. It is also possible to have a DASD (direct access storage device), for example, with a conventional search engine and a database for vocabulary both used for natural language processing and stored thereon.

So far, the invention has been described as downloading a document in response to each retrieved document record and then analyzing the document locally, for example by a client PC, to generate its corresponding logical form triple. Instead, these triads, while indexing documents by search engines,
It can also occur. In this regard, while the search engine locates each new document and indexes it, for example, by using a web crawler, the engine downloads the complete file for the document and then immediately or later, The batch process allows the document to be pre-processed by analyzing the document and generating its logical form triad. To complete the preliminary processing, the search engine then stores these triples in its database as part of the indexed record for the document. Subsequently, whenever the document record is retrieved, such as in response to a search query, the triple for that document record is returned to the client PC as part of the document record for purposes such as comparison. Preprocessing documents in the search engine has the advantage that most of the processing time on the client PC can be reduced, thereby improving client throughput.

Further, while our invention has been discussed in the specific context of use with an Internet-based search engine, our invention is directed to (a) dedicated network facilities, whether or not based on the Internet; Or any other network-accessible search engine accessible by, or (b) CD-ROM based data retrieval applications represented by encyclopedias, almanacs or other self-contained single datasets; A personalized search engine (localized search engine) that works with datasets stored on it
enjine), and / or (c) are equally applicable for use with any combination thereof. The invention can equally be used in any other suitable application.

With the above in mind, FIGS. 10A and 10B collectively illustrate yet another embodiment of the present invention. This involves generating a logical form triple by pre-processing the document, and converting the resulting triple, document record and the document itself into a self-contained single dataset, on one or more CD-ROMs or other transportable large data sets. Collectively stored on a common storage medium such as a storage medium (represented by removable hard disk, tape, or magneto-optical or high-capacity magnetic or electronic storage) and easily distributed to end users It is what it was. The correct illustration of the drawing paper for these figures is shown in FIG. By collectively arranging the search application itself and the attached data set to be searched on a common medium,
A stand-alone data retrieval application is provided, thus eliminating the need to network to a remote server to retrieve documents.

As shown, this embodiment consists essentially of three components. Document indexing component 1005 ₁ replication component 1005 _2, and a user component 1005 _3. Component 1005 ₁ collects documents, data sets, to index the dataset 1030 as illustrated. Dataset 1030, on the other hand, forms a document repository for self-contained document search applications, such as, for example, encyclopedias, yearbooks, special libraries (such as judgment reports), periodicals collection, and the like. Due to the dramatic cost reduction of duplicating CD-ROMs and other forms of media with large storage capacities, this embodiment provides cost-effective mass distribution with accurate collection and search capabilities for a wide range of user communities. It is particularly attractive for document collection.

In any event, documents to be indexed in the incoming data set are collected from any number of a wide variety of sources and provided to computer 1010 sequentially. The computer implements a document indexing engine with appropriate software stored in memory 1015. The engine establishes a record within dataset 1030 for each such document, stores the information in the record for that document, and stores the entry in the dataset appropriately, including a copy of the document itself. I do. The engine 1015 executes the triple generation process 1100. This process, which is described in detail below in connection with FIG. 11, is performed separately for each document to be indexed. In essence, the process analyzes a textual phrase in a document, essentially as discussed above with respect to block 640 shown in FIGS. 6A and 6B, and by doing so, A morphological triad set is constructed for the document and stored in dataset 1030. All other processes, including the generation of the appropriate records, performed by the indexing engine 100 for indexing documents, shown in FIGS. 10A and 10B, are not relevant to the present invention and will not be described in detail. I will not respond. Once the triad set has been generated by the process 1100, the engine 10
Suffice it to say that 15 stores this collection along with a copy of the document itself, and the document record created for it, on dataset 1030. Thus, at the end of every indexing operation, dataset 1030 not only stores a complete copy of every indexed document and records therein, but also a logical form triad set for that document. I do.

[0071] Once was properly index the all of the desired document, by then copy component 1005 _2, to create a duplicate of the data set 1 030 itself be regarded as the "master data set". The component 1005 ₂ internal, conventional media replication system 1040, a copy of the contents of the master data set supplied through line 1035, search software, including the search process and the user installation program is supplied through a line 1043 It is repeatedly written onto a common storage medium, such as one or more CD-ROMs, with appropriate file copies, collectively forming a single document search application. The system 1040, each replication 1050 _1, 1050 _2,. . . Generate a series of 1050 media replicas 1050 with 1050 _n . As shown specifically replicating 1050 ₁ replication of all hands are the same, including document search copy of the application files to be supplied through the line 1043, and a copy of the data set 1030 is supplied through a line 1035. Depending on the size and organization of the dataset,
Each copy may span one or more separate media, for example, separate CD-ROMs. Subsequently, the copies are distributed throughout the user community, typically by sale of usage rights, as symbolized by the dashed line 1055. User component 100
As shown in _3, once the user, for example, if User _j have obtained a copy, such as a CD-ROM _j (also shown as a CD-R OM1060), the user, the document retrieval application containing our invention , Computer system 1070 (even though not of the same architecture, the client PC 30 shown in FIG.
0, etc.), it is possible to execute a data set stored in the CD-ROM _j and retrieve a desired document therefrom. That is, after the user obtains the CD-ROM _j , the user inserts the CD-ROM into the PC 1070, proceeds to the execution of the installation program stored on the CD-ROM, and executes the document search application file. The document retrieval application 1085 is fixed on the PC by making a copy and installing it in a predetermined directory in the PC's memory 1075, typically on a hard disk. The application includes a search engine 1090 and a search process 1200.
including. Once the installation is complete and the application 1085 is invoked, the user can then search through the dataset on CD-ROM _j by providing the application with the appropriate full-text query. In response to the query, the search engine searches the dataset for a set of documents that includes a record for the document and a logical form triple stored for each such document. The query also feeds the search process 1200. This process is similar to the search process 6 discussed above in connection with FIGS. 6A and 6B.
Very similar to that of 00, it analyzes the query and builds a logical form triple for it. The process 1200 shown in FIGS. 10A and 10B then compares the logical form triple for each retrieved document in the collection, specifically the record, to the triple for the query. Based on the occurrence of triples among them and their weights, the process 1200 then determines a score for each of the documents with at least one matched triplet in the manner detailed above, The documents are ranked in descending order of score, and ultimately the user is visually presented with a small set of document records, typically 5 to 20 or less, having the highest ranking. The user can review these records and then instruct the document retrieval application to retrieve and display an entire copy of any relevant document that may be of interest. Once the user has reviewed the first set of document records for the first set of searched documents, the user then requests the next set of document records with the next highest ranking and reviews all the document records thus searched. You can continue until you are done. Although the application 1085 initially returns a ranked document record in response to the query, the application could alternatively return a ranked copy of the document itself in response to the query.

FIG. 11 shows the document indexing engine 10 shown in FIGS. 10A and 10B.
15 shows a triple generation process 1100 that is performed. As discussed above, the process 1100 analyzes this textual phrase in the document, thereby constructing a corresponding set of logical form triples for the document, and storing this in the data set 1030. Preprocess documents to be indexed. That is, process 1
Upon entering 100, block 1110 is executed. This block first extracts all text from the document, including any text located within the HTML tags associated with the document. Thereafter, in order to simplify the natural language processing that operates on only one sentence at a time, the text of each document is decomposed into a text file by a conventional sentence decomposing unit. Here, each sentence (or question) indicates a separate line in the file. Thereafter, block 1110 calls a separate NLP routine 1300 (discussed in detail below in connection with FIG. 13A) for each line of text in the document, analyzes the document, and interprets the logical form corresponding to that line. Build an association and store locally in database 1030. Once these operations have been completed, execution exits block 1110 and process 1100.

A flowchart of our search process 1200 for use in the specific embodiment of our invention shown in FIGS. 10A and 10B is collectively shown in FIGS. 12A and 12B. The correct alignment of the drawing paper with respect to these figures is shown in FIG. In contrast to the search process 600 (shown in FIGS. 6A and 6B and discussed in detail above), the operations shown in FIGS. 12A and 12B are all common computer systems, here PC 1070 (FIGS. 10A and 10B). (See 10B)
Run on For ease of understanding, readers will read throughout the following discussion
10A and 10B may be simultaneously referred to.

Upon entering the process 1200, execution first proceeds to block 1205. Executing this block prompts the user to enter a full text query. Once this query is obtained, execution proceeds by splitting through path 1207 to block 1210 and through path 1243 to path 1245. Block 124
Execution of 5 calls the NLP routine 1350 to analyze the query and build a corresponding logical form triple and store it locally in memory 1075.
Executing block 1210 causes a full
Send the text query to the search engine 1090. At this point,
The search engine executes block 1220 to retrieve both the set of document records that responded to the query and the associated logical form triple associated with each such record. Once this set and its associated logical form triad have been retrieved, dashed line 1230
Are returned to the process 1200 and specifically to the block 1240 therein, as symbolized by. Block 1240 simply describes search engine 1
It only receives this information from 090 and stores it in memory 1075 for later use. The operations in block 1245 are described in blocks 1210, 1090
And that operations in block 1245 have been discussed as performing essentially in parallel with the operations in block 1245, the operations in block 1245 are based on considerations of actual implementation.
It is also possible to perform continuously either before or after the operation in 10, 1090 or 1220.

Once the logical form triad set for each of the queried and retrieved document records has been stored in memory 1075, block 1250 is executed. This block compares each of the logical form triples in the query with each of the logical form triples for each retrieved document record, as described in detail above, and , Find a match between any of the triples in any of the corresponding documents. Once block 1250 is completed, block 1255 is executed to discard all search records for documents that do not have a matching triple, ie, do not have a triple that matches any of the triples in the query. . Thereafter, block 1260 is executed. Block 1260 causes all remaining document records to be scored as defined above, and based on the matched triad relation type (s) and the weights that are present for each of the corresponding documents. assign. Once the document records have been weighted in this manner, block 1265 is executed to rank the records in descending score order. Finally, block 1270 is executed to display the records, typically in rank order, for typically five or ten small predefined documents that have the highest scores. The user then causes the system to display the next set of ranked document records, eg, by "clicking" his (her) mouse on the corresponding button displayed by computer system 1070, as appropriate. And so on until the user has thoroughly tested (and has accessed and tested any document of interest therein) all of the ranked document records in succession. At the end of the test, the process 1200 has completed execution and exits.

FIG. 13A shows the NLP executed within the triple generation process 1100 shown in FIG.
13 shows a flowchart of a routine 1300. As mentioned above, the NLP routine 1300 analyzes the incoming and indexed document, specifically a single line of text for it, and constructs the corresponding logical form triad set for that document. Then, it is stored locally in the data set 1030 shown in FIGS. 10A and 10B. Routine 1300 is an NLP routine 700 shown in FIG. 7 and discussed in detail above.
Works essentially the same as

That is, upon entering the routine 1300, block 1310 is first executed to process the lines of input text to generate a logical form graph, such as the exemplary graph 515 shown in FIG. 5A. Thereafter, as shown in FIG. 13A, block 1320 is executed to extract (read) the corresponding set of logical form triples from the graph. Once this has been done, block 1330 is executed to generate each such logical form triple as a separate and differently formatted text string. Finally, block 1340 is executed to store in the data set 1030 the logical form triplet for the line, as a line of input text, and a series of formatted text strings. Once this set has been completely stored, execution exits block 1300. Alternatively, instead of a logical form triad, a different representation, such as a logical form graph or subgraph associated with a logical form, is used with our invention. If so, it is easy to modify blocks 1320 and 1330 so that the particular form occurs as a formatted string, and block 1340 replaces the form with a data set instead of a logical form triple. Store.

FIG. 13B shows a flowchart of an NLP routine 1350 that executes within search process 1200. As described above, the NLP routine 1350 determines whether the user Us
er _j analyzes the query that it supplies to the document retrieval application 1085 (shown in FIGS. 10A and 10B), constructs a corresponding set of logical form triples, and stores it locally in memory 1075. The only operational difference between routine 1350 and routine 1300, discussed in detail above in connection with FIG. 13A, is in the location where the corresponding triple is stored. That is, the NLP routine 1300 stores the data in the data set 1030 by executing the block 1340, and the NLP routine 1350 stores the data in the memory 1075 by executing the block 1390. The operations performed by the other blocks of routine 1350, ie, blocks 1360,1
370 and 1380 are blocks 1310, 132 in routine 1300
Since they are substantially the same as 0 and 1330, respectively, the detailed description of the former block is omitted.

In order to experimentally test the performance of the search process of our invention, schematically described above in connection with FIG. 1, the ALTA VISTA search engine was used as the search engine in our search system. . The engine is publicly accessible on the Internet, boasts of indexing 31,000,000 web pages, and is widely used (currently about 28,000,000 daily).
Hits), a traditional statistical search engine. directory·
MICROSOFT OFFI using various natural language processing components, including files
The search process 600 of our invention was implemented on a standard Pentium 90 MHz PC built into a grammar checker that forms part of the CE97 program suite ("OFFICE" and "OFFICE97" M in Redmond Washington
icrosoft Corporation (Microsoft Corporation). An on-line pipeline processing model was used. That is, documents were collected and processed online in a pipeline while the user was waiting for the next result. With this particular PC,
It took about 1/3 to 1/2 second to generate a logical form triple for each sentence.

A volunteer was requested to generate a full text query for submission to a search engine. A total of 121 extensive queries were generated.
The following are the representatives. ”Why was the Celtic civilization so eas
"Why do antibiotics work on colds but not on viruse?" (Why was the Celtic civilization so easily conquered by the Romans?)
s? ”(Why do antibiotics work for colds but not viruses?),“ Wh
o is the governor of Washington? ”,“ Where do
es the Nile cross the equator? "Where did the Nile cross the equator?" And "When did they start vaccinating for small pox?" , And when the document was obtained, the top 30 documents returned in response to each query were obtained. In a situation where less than 30 documents returned for some of the queries, it returned. All documents were used, accumulating for all 121 queries, yielding 3361 documents (ie, "raw" documents).

Each of the 3361 documents and 121 queries was analyzed by our inventive process to generate the corresponding logical form triad set. Compare the sets appropriately,
The resulting documents were selected, scored, and ranked as discussed above.

These documents were manually and separately evaluated for relevance to the corresponding query to retrieve all 3361 documents. To assess relevance, we used an evaluator who was unaware of our specific experimental goals, and assigned each of these 3361 documents to
The relevance to the corresponding query was manually and subjectively ranked as "optimal", "related" or "unrelated". The best document contained an explicit answer to the corresponding query. Relevant documents did not include an explicit answer to the query, but were related to it. Unrelated documents were not considered useful responses to the query. For example, some documents were not relevant to the query in a language other than English, or could not be retrieved from the corresponding URL provided by the ALTA VISTA engine (i.e., the "Cobb Web" link). To increase the accuracy of the evaluation, a second evaluation staff inspected a subset of these 3361 documents. That is, a document (431 of 3361 documents) that had at least one logical form triple corresponding to the logical form triple in the corresponding query, and a logical form triple previously ranked as relevant or optimal. This is a document having no association (102 out of 3361 documents). Any discrepancies in these rankings for the document were reviewed by a third evaluator, acting as a "tie breaker".

As a result of this experiment, across all the documents involved, our search system compared the raw documents returned by the ALTA VISTA search engine to the overall (ie, all of the selected documents) In accuracy, from about 16% to about 47% to about 200%
% Improvement, with the top five documents from about 26% to about 51% to about 1%.
A 00% improvement was obtained. In addition, with the use of our inventive system, the first document that returned as optimal gained about 17% to about 35% to about 113% accuracy improvement over that for the raw document.

Although our invention has been specifically described in the context of use with a statistical search engine, our invention is not so limited. In that regard, in information retrieval applications, our invention can process search documents obtained by virtually any form of search engine, improving the accuracy of that engine.

Instead of using fixed weights for different attributes in the logical form triple, these weights can be dynamically changed, and in fact, they can be adaptive. To achieve this, a learning mechanism such as, for example, a Bayesian or a neural network is incorporated into our inventive process, and the numerical weight for each different logical form triple is changed to an optimal value based on learning experience. It is also possible to make it.

Although our inventive process required a logical form triad to accurately match, as discussed above in one exemplary embodiment, a sufficiently similar semantic content For the purpose of distinguishing between, it is also possible to relax the criteria for determining a match and include paraphrases as matches. Paraphrase may be lexical or structural, or may involve the generation of abstract logical forms, as described below. An example of a lexical paraphrase would be either a broader term or a synonym. Structural paraphrases are illustrated by the use of either noun appositives or relative clauses. For example, the structure of a noun equivalent, such as "the president, Bill Clinton", is "Bill Clinton,
It is natural to consider it as a matching relative clause structure, such as "who is president". At a semantic level, how two words are semantically similar to each other About fine judgment (fine_grained judg
ment), the query "Where is coffee grown?" and "Coffee is frequently farmed in tropical mo"
You can see a match between texts in the corpus, such as "untainous regions." (Coffee is often grown in tropical mountainous regions.) In addition, whether or not a match exists The procedure for making the determination can also vary depending on the type of query being queried, for example, if a query asks where something is, the procedure is considered to match the query To this end, it should be asserted that the "location" attribute exists in any triad associated with the text being examined. Therefore, the "match" of the logical form triad is defined to include not only a perfect match but also a result obtained from a relaxed match condition, a match condition by judgment, and a changed match condition.

Further, our invention improves overall accuracy, even when easily combined with other processing techniques centered on searching for non-textual information, for example, graphics, tables, videos or others. Can be. Generally speaking, non-text content in a document is often accompanied by linguistic (text) descriptions within the document, such as, for example, figure legends or short descriptions. Thus, the use of our inventive process, that is, its natural language components, can be used to analyze and process linguistic descriptions that often accompany non-text content. First, documents are retrieved using our natural language processing techniques to locate a document set having linguistic content semantically relevant to the query, and then process this document set with respect to its non-text content, The document (s) with the text and non-text content to be copied can be located. Alternatively, first perform a document search on the non-text content, retrieve the document set, and then process the document set with respect to its linguistic content according to our inventive techniques;
It is also possible to locate the relevant document (s).

FIG. 14 is a simplified functional diagram of an information retrieval system 1480 according to one aspect of the present invention. The system 1480 includes a search engine 1482, a search engine 14
84, and statistical data storage 1486. It should be noted that the entire system 1480, or a portion of the system 1480, can be implemented in the environment shown in FIG. For example, search engine 1482 and search engine 1484
Simply implemented as computer readable instructions stored in memory 322,
It can be executed by the CPU 321 to execute a desired function. Or,
Search engine 1482 and search engine 1484 may also be provided on any type of computer readable media, such as described with respect to FIG. In addition, search engine 1482 and search engine 1484 may be provided in a distributed processing environment and run on separate processors. In addition, statistical data storage 1486 may be stored in the memory components discussed with respect to FIG. 3, may be stored on memory located within wide area network 352, and For example, it can be stored in the memory 350 accessible through the local area network 351. In another exemplary embodiment, storage 1486 may be located in a portion of memory 322 and accessed by an operating system within computer 320.

In any case, text input (ie, a query) may be input to search engine 1482 through any suitable input mechanism, such as keyboard 340, mouse 342, or the like.
To supply. Search engine 1482 performs a number of functions based on the query. In a preferred embodiment, the search engine 1482 uses the text input to
Formulate a Boolean query and provide the Boolean query to search engine 1484.

The search engine 1484 is, in one exemplary embodiment, a Diary from MA, Maynard.
A search engine offered by gital Equipment Corporation under the commercial designation Alta Vista. Alta Vista search engine is the traditional Internet search engine. In such an embodiment, search engine 1482 connects to search engine 1484 via a suitable Internet connection. Of course, other search engines can be used as well.

In the exemplary embodiment, search engine 1484 is a statistical search engine that has access to statistical data rules 1484. Such search engines typically incorporate statistical processing into the search methodology used to search data store 1486.

Data storage 1486 typically contains a data set of document records indexed by search engine 1484. Each such record is, for example, a web address at which the corresponding document can be accessed by a web browser, possibly a short summary of the document and predefined content words that appear in the document, and a hypertext markup language (HTML). Contains the description of the document when given in the description field. In addition, statistical data storage 1486 may also include data indicating the logical form calculated for the document that is indexed therein. In one exemplary embodiment, the logical form associated with the index heading corresponds to the language originally used for the indexed document. In another exemplary embodiment, as will be described in more detail below, the logical forms include paraphrased logical forms and are modified to suppress high frequency logical forms.

Statistical search engine 1484 typically includes a statistical data store 148.
A numerical scale is calculated for each document record retrieved from 6. This numerical measure is based on the query provided to search engine 1484. Such numerical measures may include, for example, term frequency * inverse document frequency (tf * idf).

In any case, search engine 1484 causes search engine 1482 to
Either the identified document record or the document itself is ranked back in the order of the statistical measure calculated for each document record. In one exemplary embodiment, search engine 1482 performs additional natural language processing on the returned document or record to narrow down the ranking of the document or record. Next, the document or record is presented to the user as an output document set according to the narrowed ranking.

FIG. 15 is a more detailed functional block diagram of search engine 1484, illustrating how statistical data storage 1486 is created in accordance with an exemplary embodiment of the present invention. FIG. 15 shows a document 1588 stored on any suitable storage device. Such storage may include computers in a distributed computing environment, storage accessed by an operating system in computer 320, computers accessible through a wide area network (such as the Internet), library databases, or documents. It can be any other suitable location stored. Document 1588 is typically accessible by search engine 1484 through a web crawler component, referred to herein as document indexer 1590. Document indexer 1590 accesses documents 1588 and indexes them in a known manner to generate a record associated with each of the accessed documents.

The search engine 1484 also includes a logical form generator 1592 and a logical form changer 1594. The logical form generator 1592 also accesses the document and creates a logical form corresponding to each accessed document.

The logical form generator 1592 generates a logical form based on the input text.
Briefly, semantic analysis generates a logical form graph that describes the meaning of the text input. The logical form graph includes nodes and links, and the links are labeled to indicate the relationship between the node pairs. A logical form graph represents, for example, a more abstract level of analysis than a parse tree. Because this analysis normalizes many syntactic or morphological variations.

The logical form changing unit 1594 receives the logical form generated by the logical form generating unit 1592, and changes the logical form. For example, the changing unit 1594 creates a paraphrased logical form set based on the original logical form, and creates a predetermined class of logical forms (high-frequency logical forms) that do not help distinguish between various documents. Like form).

The records created by the document indexer 1590, together with the changed logical form set,
By way of example, it may be provided to a statistical data store 1486 and stored for later access by the search engine 1484 in response to queries provided through the search engine 1482. The logical form changing unit 1494 will be described in more detail below.

FIG. 16 is a more detailed block diagram of the search engine 1482. In the exemplary embodiment, search engine 1482 includes input logical form generator 1696, logical form changer 1698, Boolean query generator 1600, and filter 1602.
including. On the other hand, the filter 1602 includes a logical form comparison unit 1604 and a document rank generation unit 1606.

The query input by the user is supplied to the Boolean query generator 1600. The Boolean query generator 1600 generates a Boolean query based on a user input query as in a conventional information retrieval system. Search for Boolean queries
Engine 1484, which performs a query against statistical data storage 1486. In response, the statistical data storage 1
486 stores the document record (including the changed logical form set) in the search engine 1
484, and then search engine 1484 converts them to search engine 148.
2 to the filter 1602.

The query is also supplied to the input logical form generator 1596. The generator 1596 generates one or more logical forms and their relationship to one another based on the original words in the query. The generation of the logical form is performed by the logical form generator 1592 in FIG.
Is performed in the same manner as described above.

The original logical form is supplied to a logical form changing unit 1698, and these logical forms are changed to include, for example, a paraphrase logical form set to suppress a high-frequency logical form. Further, the set of changed logical forms is compared with the logical form comparing unit 1604 in the filter 1602.
To supply.

The logical form comparing unit 1604 compares the modified logical form set based on the query with the modified logical form set based on the document retrieved from the data storage device 1486. If any of the set of modified logical forms based on the query matches the one based on the document, the logical form comparison unit 1604 assigns a weight to a specific document including the matched logical form. This weight is based on the number and type of matches associated with each document.
Any documents that do not contain any matches can be discarded and not presented to the user, or presented to the user with an indication that the document is unlikely to be relevant to the query.

The document record including the match is supplied to the document rank generating unit 1606 together with the weight assigned by the logical form comparing unit 1604. The document rank generating unit 1606 ranks the documents based on the weights assigned by the logical form comparing unit 1604,
The ranking output is presented to the user as an output document set.

FIG. 17 is a flowchart showing the operation of the system shown in FIG. 16 in more detail. First, an input query is performed against the statistical data store 1486 to filter 1602 the document records and the change logic associated with those document records.
To supply. This is indicated by blocks 1708 and 1710. The generator 1696 then generates a logical form based on the original content of the query. This is indicated by block 1712. Next, the logical form based on the query is converted to a logical form changing unit 169.
8 to change. This is indicated by block 1714.

The filter 1602 then selects the first of the document records supplied by the search engine 1484 in response to the query. This is indicated by block 1716. Logical form comparator 1604 determines whether any of the changed query logical forms corresponds to the changed document logical form. If not, the document is assigned a zero score, and the filter 1602 determines whether there are any additional documents that need to be compared. This is indicated by blocks 1718, 1720 and 1722.

However, if any of the changed query logical forms matches any of the changed document logical forms, the logical form comparison unit 1604 assigns a weight to the document to be analyzed. This is indicated by block 1724. Again, the filter 1602 returns to block 17
As shown at 22, a determination is made as to whether there are any additional documents that need to be compared.

If there are no more documents that need to be compared, the document rank generator 1606
The documents are ranked according to the weights assigned by the logical form generation unit 1604. The ranked output is then presented to the user. This is indicated by blocks 1726 and 1728.

FIG. 18 is a flowchart showing the operation of logical form changing section 1594 shown in FIG. 15 and logical form changing section 1698 shown in FIG. It will be appreciated that the present invention contemplates the use of modification logic, either discussed on the query side or the data side, or both, as discussed in more detail below. For the purposes of this discussion, a logical form changer is shown on both the query and data sides.

In any case, the logical form change unit first receives the original logical form generated based on either the query or the document to be analyzed. Block 183
Indicated by 0. Next, the logical form changing unit generates a paraphrase of the original logical form. This paraphrase can be formed in any of a number of ways. Some of them are described below. The occurrence of the paraphrase logic form is indicated by block 1832.

Next, the logical form changing unit suppresses a logical form of a predetermined class (a variety of logical forms is also possible). The number is discussed below. This suppression is indicated by block 1834. In other words, after the suppression, the filter 10
2 to filter the document based on the logical form remaining after suppression. This is indicated by block 1836. (Generation of Modified Logical Form) FIG. 19 is a flowchart showing the generation of paraphrase logical form and suppression of the logical form better. (Semantic or lexical paraphrase) One of the logical form change units receives the original logical form. Next, the logical form change unit first forms a lexical paraphrase logical form by performing a semantic expansion of the words in the original logical form. This is indicated by block 1938. A lexical paraphrase logical form is then generated based on the semantically expanded words and using the original structural connection in the original logical form. This is indicated by block 1940.

In one exemplary embodiment, to perform semantic expansion, each contained word in the original logical form is tested, and a synonym, broader term, narrower term, or semantic expansion is performed on the original contained word. Expand the word to include other words that have a relationship. For example, in one embodiment, the logical form modifiers 94 and 98 are given access to a reference corpse, such as a thesaurus, a dictionary, or a computational lexicon, such as a WordNet or MindNet vocabulary. It can identify synonyms, broader terms, narrower terms, or other semantic relationships and identify possible lexical paraphrases between the query and the document.

Thus, for example, if the input query is How do spiders eat their victims ?, the original logical form generated based on this query is It is as follows. eat; Dsub; spider eat; Dobj; victim The lexical or semantic expansion of the word “eat” (consume) (
Consumption) is obtained. The lexical or semantic expansion of the word "spider" also yields "arachnid" and "wolf spider". On the other hand, these expansions lead to additional paraphrased logic forms for eat; Dsub; spider, as follows. consume; Dsub; spider eat; Dsub; arachnid consume; Dsub; arachnid eat; Dsub; wolf_spider consume; Dsub; wolf_spider
Is obtained. Therefore, the paraphrase logical form based on the logical form eat; Dobj; victim is as follows. consume; Dobj; victim eat; Dobj; prey This technique tends to retain related documents that are returned based on the query. Thus, this technique increases the recovery within this document collection without compromising accuracy. (Structural paraphrase) After expanding the original logical forms with respect to vocabulary, they are structurally expanded to obtain additional paraphrase logical forms. Using the more rigorous technique described in the reference mentioned above and included in the present application, the relevant documents returned by the search engine may be modified such that the contained words in the query appear in a single word in the document. Even in the text, it may be discarded. This typically occurs when there is a syntactic or semantic paraphrase between the query and the text of the document, but the logical form based on the query and the logical form based on the document exactly match If not.

In order to properly retain documents that meet these criteria, the logical form modifier implements structural paraphrase rules to generate additional logical forms based on the original logical form. This additional logical form captures the usual syntactic / semantic paraphrases and normalizes the difference between how a user expresses a query and how related documents express similar concepts. It is intended to be To do this,
The logical form change unit augments the basic logical form generated based on the original input text.

For example, if the original query is How many moons does Jupiter have? (How many moons does Jupiter have?) The original logical form triple based on this query is have; Dsub; Jupiter have; Dobj; moon moon; Ops; many.

Here, Ops is a function word relation. By implementing the structural paraphrasing rules according to one aspect of the present invention, the logical form changer generates the following additional logical forms. Moon; PossBy; Jupiter The containing word is the same as the original logical form, but the structural connection is different, but it can be seen that it is a related structural connection. This allows matching against indexed documents containing the same logical form.

In other examples of structural paraphrase rules, further complications are possible. For example, suppose the input query is Find me information on the crystallization of viruses. This creates a computational logic triad as follows: crystallization; of; virus A few pieces of information to consider may be needed to match a query against relevant literature, including text, describing how "viruses crystallize" There is. Such information includes:

[0119] 1. A regular paraphrase exists between Dsub / verb and some English nounization. 2. The noun "crystallization" appears in the default dictionary in the verb base "crystalli
ze ".

[0120] 3. "Virus" is categorized as animate in the dictionary. Together, these pieces of information allow additional structural paraphrase logic forms to be taken as hypotheses for the query and generated for matching. crystallize; Dsub; virus The "animacy" of "virus" is used to predict whether this paraphrase should be expressed as a subject or object relation. Cross-linguistically (cross_lin
Living things are more likely to be the subject (verb) of the verb than inanimate objects. Thus, if the query asks about "crystallization of sugar", it will generate an additional paraphrase logic form crystallize; Dobj; sugar.

Various logical form paraphrase rules have been implemented to normalize a number of syntactic paraphrases. These include the following: 1. Possession syntax, 2. Nounization / verb object and subject, noun complex / verb object ("pr
gram computers ”and“ computer programs ”).

[0122] 3. Noun modifiers (such as "King of Spain" and "Spanish King"). 4. Reciprocal constructs (such as "John kissed Mary" and "Mary kissed John"). 5. Adjective / predicative adjectives (such as "That woman is tall" and "That tall woman").

6. Light verb constructions / verbs ("The president made a decision" and "The P
Appendix A contains code that illustrates examples of embodiments of the rules described above. In each case, these rules are more relevant while still severely limiting the matching process. Performing structural extensions or structural paraphrasing of the original structural relationship is indicated by block 1942 in Figure 19. The paraphrasing rules discussed above, and other such rules, are empirical. Or any other suitable means.

The structural paraphrase can be performed on both the indexing side and the query side of the information search system. However, if the structural paraphrasing is implemented on the indexing side, the size of the index may become large, which is not desirable. Thus, in one exemplary embodiment, the structural paraphrase is performed only on the query side of the information retrieval system.

It should also be noted that structural paraphrases can be performed either before or after the semantic paraphrases shown in blocks 138 and 140. In addition, structural paraphrases can be made based on additional logical forms that have occurred during semantic expansion. This is indicated by blocks 1944 and 1946. Meta-structure paraphrases Additional paraphrase logical form sets that can be generated by the logical form changer include the generation of abstract logical forms. For example, even when prompting a user to enter a natural language query into a search engine, many users still cannot provide a well-formed query with a syntactic / semantic relationship of interest using a large number of contained words. . Rather, many queries fall into a category referred to herein as "keyword queries." Such keyword queries include true keyword queries such as "dog", "gardening" (horticulture), "The Renaissance", "Buffalo Bill". Also, the keyword query may be in the form of keywords in a crest-type “frame” sentence, such as “Tell me about dogs”, “I want information”
on gardening ”(for information on gardening) and“ What do you have on
dinosaurs? ”(what do you think about dinosaurus)? It does not give a useful linguistic context.
Includes matching techniques to address these queries.

First, the query is identified as a keyword query based on its structure, as indicated by block 1948 in FIG. The query consists of only one contained word (or sequence of contained words treated as a compound contained word, also known as a multi-word expression), or one or more that occur within an explicitly identified common query structure , The query is identified as a keyword query.
An example of a multi-word expression is "Buffalo Bill". This is treated as a single word with an internal structure.

The following rules give an example describing the structure used to identify a keyword query of the form “Who was Buffalo Bill”. If the verb is "be", the Dnom (deep nominative) is "who", or the Dsub is not syntactically changed, except for the immediately preceding determinant or prepositional phrase, Dsub for matching purposes Is treated as a keyword. Once a query has been identified as a keyword query, for matching purposes,
Generates various abstract logical forms. The query is "Who was Buffalo Bill?" In the above example, the following abstract logical form occurs. Buffalo_Bill Dsub_of_be; Dsub; Buffalo_Bill Dsub_of_verb; Dsub; Buffalo_Bill These abstract logical forms do not directly correspond to any included in the original logical form generated based on the query. However, they are potentially consistent with the corresponding logical form created in the document record when the document is indexed and stored in the statistical data store 1486. For example, "Buffalo Bill
When processing a document titled "", the logical form changing unit 1594 shown in FIG. 15 generates the following abstract logical form and adds it to the index in the structural data storage device 1596. heading_or_title; Dsub; Buffalo_Bill Also, during indexing of documents, any logical form, including the verbs “be” and Dsub, creates special logical forms such as: Dsub_of_be; Dsub; WORD (eg, Dsub_of_be; Dsub; Buffalo_Bill) In addition, if the logical form includes a verb other than Dsub and "be", create an additional abstract logical form as follows: Dsub_of_verb; Dsub; WORD (eg, Dsub_of_verb; Dsub: Buffalo_Bill) The abstract logical form created for the keyword query at the time of indexing and at the time of the query allows the information retrieval system to use the linguistic structure of the data side. To identify documents that are most likely to be primarily related to the keywords contained in the keyword query (eg, the abstract logical form on the data side can be matched to the keyword query). Represents a possible document meta-structure).

In addition, even if a document does not have a title including a keyword, a sentence in the document can be analyzed to determine the meta structure of the document. For example, the subject of a sentence, in particular, the subject of a sentence whose main verb is “be” is the subject or topic of the sentence.
Often it is. By preferentially matching keyword queries to documents containing text about the keywords, even for keyword queries,
Accuracy can be improved. For example, the query is "Buffalo Bill" and the first document is Buffalo Bill was a showman, usually acting as the part of himself in o
The second document, One of the most active performers, contains the text, "ne of Buntline's melodrams." in American cinema, Keitel Demonstrat
ed his versatile talents in the 1970's in drama, Alice Doesn't Live He
re Anymore (1974); an artfull western Buffalo Bill and the Indians, or S
itting Bulls history lesson (1976); and a black comedy, Mother, Jugs, and
Speed (1976). One of the most active actors in American cinema, Keitel is a 1970s drama, Alice no longer lives here (1974), an artistic western,
Buffalo Bill and Indian or Sitting Bulls History Lesson (
1976), and black comedy, mother, prison, and stimulants (197
In 6), he demonstrated his various talents. , The abstract logical form that occurs at the time of indexing for that document, and the abstract logical form that occurs for the keyword query at the time of the query, The collation is preferentially performed not on the document but on the first document.
This is because the first document contains a keyword query as the subject of a sentence,
This is because the second document is not.

An additional example of an abstract logical form is one that is created based on a defining sentence. An example of the definition sentence is as follows. Lava, molten rock which flows from volcanoes This type of defining text can be identified by examining cues, including linguistic and formal structures. it can. Most often, such sentences are parsed as a noun phrase containing a single noun or multi-word expression, followed by a comma, followed by a peer noun phrase. This generates an abstract logical form of the form article_title_or_heading; Dsub; lava.

This indicates the meta-structure (or overall content) of the document and can be used to match against keyword queries that require such text. Obtaining an abstract logical form for the meta-structure of the document and obtaining an abstract logical form for the keyword query is performed by blocks 1950 and 195 of FIG.
Indicated by 2. (Suppression of Certain Logical Forms) The logical form change units 1594 and 1698 according to another aspect of the present invention also suppress a certain class of logical forms. For example, some logical forms do not become good discriminators of the relevant document, but produce false positive matches. Typically, such logical forms correspond to high frequency logical forms such as "be;Locn;there". The logical form of this class is "stopw" found in Boolean search systems.
ord "can be thought of as syntactic / semantic synonyms of" ord. "Additional examples of logical forms of this class include: Certain verbs / invariants: come; ; to (he) (I came to a decision, J
ohn came to a stop. (I came to a decision, John came to the stop) High frequency verb: be; Dsub: John (John is tired, John is the largest elephant
in the world) (John is tired; John is the largest elephant in the world) Synonyms: eat (eat); Dsub; he (he) (he ate at home) (he ate at home) Logical forms: tell; say Dobj; me (tell me about dogs) These and other such logical forms are identified empirically or by other appropriate means But typically correspond to logical forms that lead to incorrect matches. According to one aspect of the invention, the logical forms of this class are:
Identify and suppress in either the query or the document record, or both.
This is indicated by blocks 1954 and 1956 in FIG.

In addition, some of these logical forms can be suppressed during the generation of the logical form based on the query. For example, a logical form of the form “give (give); Dobj; information (information)”, if not suppressed during document indexing, would produce “what databases give information on cancer?” May be useful in matching against queries such as In that case, the user requests the identity of a particular database, and the query is very specific. On the other hand, logical forms of the form "give (given); Dobj; information (information)" are suppressed during processing of a query of the form "give me information on X". Identify this query as a keyword query and suppress the identified logical forms.

After obtaining all of the logical and modified logical forms based on lexical and semantic paraphrases, structural paraphrases, generation of abstract logical forms, and suppression of logical forms,
The modified logical form set is supplied to the filter 1602 for further processing. Figure 1
Nine blocks 1958. As discussed above, the filter 1602 looks for a match between the query-based logical changes and those based on the document. Conclusion Thus, it can be seen that the present invention provides a system for determining the similarity between two or more text inputs. In addition, one aspect of the present invention is well suited for significantly improving accuracy in information retrieval applications by identifying documents that are more relevant than previous techniques in the set of documents returned by the search engine. Also, the present invention
Increase recovery by reducing the number of relevant documents discarded during the filtration process.

One aspect of the present invention, by way of example, is to create and compare logical forms based on two textual inputs and expand the original word lexically or semantically, thereby connecting the original structural connection. Paraphrased logical forms are created by structurally expanding and / or by creating abstract logical forms that indicate the metastructure of either or both of the text inputs (eg, documents or queries, or both). The present invention also suppresses certain logical forms, by way of example. Of course, paraphrase and suppression need not be the same for both sets of logical forms, but can be different for each.

It should also be noted that currently hashing techniques are employed to chop the indexes contained in the statistical data store 86 to smaller sizes. Of course,
Any suitable hashing technique can be used. The present invention can be used equally with a hash representation of an index or with a complete representation of an index.

While the present invention has been described with reference to preferred embodiments, workers skilled in the art will recognize that changes may be made in form and detail without departing from the spirit and scope of the invention.

[0136]

[Table 5]

[0137]

[0138]

[0139]

[0140]

[Brief description of the drawings]

FIG. 1 shows a top-level block diagram of our information retrieval system 5 according to the present invention.

FIG. 2 illustrates a high-level embodiment of an information retrieval system 200 of the type shown in FIG. 1 utilizing our teachings of the present invention.

3 shows a block diagram of a computer system 300, specifically a client personal computer, housed inside the system 200 shown in FIG.

4 shows a top-level block diagram of an application program 400 executed inside the computer 300 shown in FIG.

FIG. 5A illustrates different corresponding examples of the changing complexity of English sentences and their corresponding logical form factors.

FIG. 5B illustrates different corresponding examples of the changing complexity of English sentences and their corresponding logical form factors.

5A-5C illustrate different corresponding examples of the changing complexity of English sentences and their corresponding logical form factors.

5A-5D illustrate different corresponding examples of the varying complexity of English sentences and their corresponding logical form factors.

FIG. 6 shows the correct alignment of the drawing sheets of FIGS. 6A and 6B.

FIG. 6A shows a flowchart of a search process 600 of our invention.

FIG. 6B shows a flowchart of a search process 600 of our invention.

FIG. 7 shows a flowchart of an NLP routine 700 that executes within process 600.

FIG. 8A illustrates an exemplary collation logic form triple weight table 800. FIG.

8B shows blocks 650, 660, 665 and 670, all shown in FIGS. 6A and 6B.
Graphically illustrates the logical form triad comparison and document scoring, ranking and selection process in accordance with the teachings of our invention for an exemplary query and three exemplary statistically retrieved document sets performed at Show.

FIG. 9A illustrates three different embodiments of an information retrieval system incorporating our teachings of the present invention.

9A and 9B illustrate three different embodiments of an information retrieval system incorporating the teachings of the present invention.

FIG. 9C illustrates three different embodiments of an information retrieval system that incorporates the teachings of the present invention.

FIG. 9D illustrates an alternative embodiment of the remote computer (server) 930 shown in FIG. 9C for use in implementing another different embodiment of our invention.

FIG. 10 illustrates the precise registration of the drawing sheets of FIGS. 10A and 10B.

FIG. 10A shows yet another embodiment of our invention where the logical form triple for each document is pre-computed, stored with the document record, and accessed during a later document retrieval process.

FIG. 10B shows yet another embodiment of our invention where the logical form triples for each document are pre-computed, stored with their document records, and accessed during a later document retrieval process.

FIG. 11 illustrates a triple generation process 1100 performed by the document indexing engine 1015 shown in FIGS. 10A and 10B.

FIG. 12 shows the accurate registration of the drawing sheets of FIGS. 12A and 12B.

FIG. 12A shows a flowchart of the search process 1200 of our invention running in the computer system 300 shown in FIGS. 10A and 10B.

FIG. 12B shows a flowchart of the search process 1200 of our invention running in the computer system 300 shown in FIGS. 10A and 10B.

FIG. 13A shows a flowchart of an NLP routine 1300 executed within the triple generation process 1100.

FIG. 13B shows a flowchart of an NLP routine 1350 executed within the search process 1200.

FIG. 14 is a functional block diagram showing an embodiment of the present invention.

FIG. 15 is a functional block diagram illustrating indexing of documents according to one aspect of the present invention.

FIG. 16 is a more detailed block diagram of a search engine according to one aspect of the present invention.

FIG. 17 is a flowchart showing processing of the system shown in FIG. 16;

FIG. 18 is a flow diagram illustrating a logical form change of a natural language processor according to one aspect of the present invention.

FIG. 19 is a more detailed block diagram illustrating a logical form modification of a natural language processor according to one aspect of the present invention.

[Procedure amendment]

[Submission date] March 10, 2000 (2000.3.10)

[Procedure amendment 1]

[Document name to be amended] Drawing

[Correction target item name] All figures

[Correction method] Change

[Correction contents]

[Procedure amendment]

[Submission date] October 27, 2000 (2000.10.27)

[Procedure amendment 1]

[Document name to be amended] Drawing

[Correction target item name] All figures

[Correction method] Change

[Correction contents]

FIG.

FIG. 2

FIG. 3

FIG. 4

FIG. 5

FIG. 6

FIG. 7

FIG. 8

FIG. 9

FIG. 10

FIG. 11

FIG.

FIG. 13

FIG. 14

FIG.

FIG. 16

FIG.

FIG.

FIG.

──────────────────────────────────────────────────続 き Continuation of front page (81) Designated country EP (AT, BE, CH, CY, DE, DK, ES, FI, FR, GB, GR, IE, IT, LU, MC, NL, PT, SE ), CN, JP (72) Inventor Dolan, William B. 98052, Washington, United States of America 98052, Redmond, One Hundred Fifty Sard Court North East 7412 (72) Inventor Vanderwende, Lucy H. Washington, United States of America Northeast Surteeth Street, Bellevue, State 98008, 16415 (72) Inventor Braden-Harder, Lisa Virginia, USA 20194, Reston, Creek Bend Drive 12003 F-term (reference) 5 B075 ND03 NR12 PQ74 PR04 PR06 QM05 QP03 5B091 AA11 AA15 AB17 BA03

Claims (44)

[Claims] 1. A method for determining a similarity between a first and a second text input, comprising: obtaining a first set of logical forms based on the first text input; and based on the second text input. Obtaining a second set of logical forms, comparing the first and second set of logical forms, and determining a similarity between the first and second text inputs based on the comparing step. A method comprising the steps of: 2. The method of claim 1, wherein the comparing step comprises: determining whether any logical form in the first set matches any logical form in the second set. A method characterized by comprising: 3. The method of claim 2, wherein determining similarity comprises: determining a degree of similarity between the first and second text inputs based on a match between the first and second sets of logical forms. Assigning a score that reflects the score. 4. The method of claim 1, further comprising: obtaining a first paraphrase set based on the first set of logical forms. 5. The method of claim 4, wherein the comparing comprises: comparing the first set of paraphrases to the second set of logical forms; and any one of the first set of paraphrases. The paraphrased logical form of the second
Determining whether there is a match with any of the logical forms in the set of logical forms. 6. The method of claim 5, further comprising the step of: obtaining a second paraphrase set based on the second set of logical forms. 7. The method of claim 6, wherein the comparing step further comprises: comparing the first paraphrase logical form set with the second paraphrase logical form set; Is the second logical form
Making a determination as to whether any of the paraphrase logical forms in the set of paraphrase logical forms match. 8. The method of claim 1, wherein said first text entry comprises an information retrieval query and said second text entry comprises at least one document retrieved based on said query. how to. 9. The method of claim 1, wherein said second text entry comprises an information retrieval query and said first text entry comprises at least one document retrieved based on said query. Method. 10. The method of claim 5, wherein obtaining the first set of logical forms comprises: obtaining a prototype word and a prototype structural relationship between the prototype words based on the first text input. A method characterized by comprising: 11. The method of claim 10, wherein the prototype structural relationship comprises a prototype structural relationship between the prototype words, and wherein obtaining a first paraphrase logical form set comprises: Obtaining additional logical forms, including expanded words, connected in said archetypal structural relationship. 12. The method of claim 11, wherein the prototype words include a first prototype word and a second prototype word connected by the prototype structure relationship.
Obtaining an additional logical form includes expanding the first prototype word with respect to vocabulary and including a first related word that is semantically related to the first prototype word; expanding the second prototype word with respect to vocabulary Including a second related word semantically related to the second prototype word; connecting different ones of the first and second related words to each other according to the prototype structural relationship to obtain the additional logical form. A method comprising at least one of: 13. The method of claim 12, wherein expanding the first prototype word with respect to vocabulary or expanding the second prototype word with respect to vocabulary comprises: synonyms for the first and second prototype words. The step of obtaining 14. The method of claim 12, wherein expanding the first prototype word with respect to vocabulary or expanding the second prototype word with respect to vocabulary comprises: a broader term for the first and second prototype words. The step of obtaining 15. The method of claim 12, wherein expanding the first prototype word with respect to vocabulary, or expanding the second prototype word with respect to vocabulary, comprises subwords for the first and second prototype words. The step of obtaining 16. The method of claim 10, wherein obtaining a first paraphrase logical form set comprises: obtaining an expanded structural relationship related to the prototype structural relationship; and connecting the prototype word to the expanded structural relationship. Obtaining said paraphrased logic form. 17. The method of claim 16, wherein obtaining a first set of logical forms further comprises: obtaining an expanded word semantically related to the prototype word; Connecting. A method comprising: 18. The method of claim 17, wherein obtaining the first set of paraphrased logical forms further comprises connecting the expanded word with the expanded logical relationship. 19. The method of claim 10, wherein the first set of logical forms includes at least one contained word, and obtaining a first set of paraphrased logical forms comprises: a first abstraction based on the contained words. Obtaining a set of logical forms. 20. The method of claim 19, wherein the first text input comprises a document search query and obtaining a first set of abstract logical forms comprises: generating the first set of abstract logical forms. Identifying the query as a keyword query prior to causing the content word to be modified by other content words based on the structure of the query. 21. The method of claim 10, wherein the second text input comprises a document, and further comprising: obtaining a second paraphrase logical form set based on the second logical form set. how to. 22. The method of claim 21, wherein obtaining the second set of logical forms comprises obtaining an abstract set of logical forms indicating a meta-structure of the document. 23. The method of claim 22, wherein the metastructure of the document comprises:
Indicating a general subject of the document. 24. The method of claim 23, wherein the step of obtaining the set of abstract logical forms indicating a meta-structure of the document comprises: obtaining the set of abstract logical forms based on format information corresponding to the document. A method comprising the steps of: 25. The method of claim 23, wherein the step of obtaining the set of abstract logical forms indicating a meta-structure of the document comprises: obtaining the set of abstract logical forms based on a title of a sentence in the document. A method comprising the steps of: 26. The method of claim 23, wherein the step of obtaining the set of abstract logical forms indicating a meta-structure of the document comprises: obtaining the set of abstract logical forms based on the subject of a sentence in the document. A method comprising the steps of: 27. The method of claim 21, further comprising the step of suppressing other logical forms other than the first and second paraphrase logical form sets based on the contained words. And how. 28. A method for filtering a sentence in a document set retrieved from a document storage device in response to a query, the method comprising: filtering a first logic based on the query and one selected from documents in the document set. Obtaining a set of forms; obtaining a second set of logical forms based on the query and another of the documents in the set of documents; a first paraphrase showing at least a paraphrase of the first set of logical forms Obtaining a set of logical forms; and filtering the documents in the set of documents based on a predetermined relationship between the first set of paraphrased forms and the second set of logical forms. And how. 29. The method of claim 28, wherein the step of filtering includes the step of providing an output indicating a ranking order of the documents in the document set based on the predetermined relationship. how to. 30. A method for filtering sentences in a document set retrieved from document storage in response to a query, comprising: obtaining a first set of logical forms based on the query and one selected from the document set. Obtaining a second set of logical forms based on the query and another one of the set of documents; suppressing at least a logical form of a first predetermined class in the first set of logical forms; 1) obtaining a set of suppressed logical forms; and filtering the documents in the document set based on a predetermined relationship between the first set of suppressed logical forms and the second set of logical forms. A method characterized by the following. 31. The method of claim 30, wherein the step of suppressing comprises the step of suppressing logic forms having a predetermined structure. 32. The method of claim 30, wherein the step of suppressing comprises the step of suppressing logic forms that occur at a frequency above a threshold frequency level. 33. The method of claim 30, further comprising the step of suppressing a second predetermined class of logical forms in the second set of logical forms, wherein the second predetermined class includes the first predetermined class. A method characterized by being different from a predetermined class. 34. The method according to claim 30, wherein a suppressing step is performed before the step of obtaining the first set of logical forms. 35. The method of claim 30, wherein the step of suppressing is performed substantially simultaneously with the step of obtaining the first set of logical forms. 36. The method of claim 30, wherein the step of obtaining the first set of logical forms is followed by the step of suppressing. 37. A computer readable medium comprising computer readable data stored thereon, the computer readable data comprising index data indicative of the content of a document in a document collection; A set of abstract logical forms indicating a meta-structure of each of the documents in the set. 38. The computer readable medium of claim 37, wherein the meta structure of each document indicates an overall subject of the document. 39. The computer readable medium of claim 38, wherein the set of abstract logical forms is based on format information corresponding to each document. 40. The computer-readable medium of claim 38, wherein the set of abstract logical forms is based on the subject of a sentence in each document. 41. The computer readable medium of claim 38, wherein the set of abstract logical forms is based on the subject of a sentence in each document. 42. A computer readable medium, including computer readable data stored thereon, wherein the computer readable data, when executed by a computer, comprises the following steps: the query and the document Obtaining a first set of logical forms based on one selected from the set of documents; obtaining a second set of logical forms based on the query and another one of the documents in the set of documents; Modifying at least the first set of logical forms using language processing to obtain a first set of modified logical forms; based on a predetermined relationship between the first set of modified logical forms and the second set of logical forms Filtering the documents in the document set. Computer readable medium, characterized in that for filtering a document that the computer. 43. A method for determining similarity between first and second text inputs, comprising: obtaining a first set of logical forms based on the first text input; and based on the second text input. Obtaining a second set of logical forms; suppressing at least a first predetermined class of logical forms in the first set of logical forms to obtain a first set of suppressed logical forms; Determining a similarity between the first and second textual inputs by comparing to a second set of logical forms. 44. The method of claim 6, wherein the step of obtaining the first paraphrase logical form set is performed using a first paraphrase technique, and the step of obtaining the second paraphrase logical form set comprises: Performing using a second paraphrase technique different from the paraphrase technique.