JP2001312504A - Method and system to extract knowledge - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a data warehouse that can include an indexed database and also can store data on information stored in an external database. SOLUTION: The information extracting system that processes querys for extracting information from the database is provided with a mechanism that finds characteristics and fragments of the characteristics within the indexed database, an evaluation mechanism that evaluates repeatedly sub-querys using the characteristics and the fragment of the characteristics being found after recognizing the multi-level sub-queries being included in the query, and a mechanism that gathers the results of the repeated evaluations for the query and the sub-query succeeding the calculation of the total result of the query and also stores the results in a memory.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION This application is a preliminary application, Kenneth, filed together and assigned to the same assignee.
P. US Patent Application Serial No. 60 / 094,350, July 28, 1998, entitled "Knowledge Extraction System and Method" by Bacrawski;
P. It is related to two applications, U.S. Patent Application Serial No. 60 / 094,110, July 24, 1998, entitled "Distributed Object Retrieval System and Method" by Bacrawski. The disclosure of the above application is incorporated herein by reference. This application is also related to patent application Kenneth P.K., filed together and assigned to the same assignee. Bac
Lawski also relates to US Patent Application No. xxx, xxx entitled "Distributed Computer Database System and Method for Performing Object Search", the disclosure of which is incorporated herein by reference.

[0002] The present invention relates to computer database systems, and more particularly, to distributed computer database systems.

[0003]

2. Description of the Related Art Enterprises collect large amounts of data on customers, products, operations, and business activities as routine work.
The insights hidden in this data can contribute to marketing, operating costs and strategic decisions. For example, if there is a strong correlation between a customer who buys one product and a customer who buys another product, a customer who buys only one of the products is a prospect who is likely to buy the other product is there.

[0004] Data analysis processing is mainly performed by using a statistical method for extracting data correlation and other patterns. This type of processing is called data mining (d
data mining, knowledge discovery (known)
Ge discovery) and knowledge extraction (kno)
wredge extraction). Searching for a specific pattern or a specific type of pattern from a large amount of data is performed by querying a pattern (pat
will be referred to as a turn query.

[0005] Large companies usually have a large number of databases, many of which are transactional databases. The requirements of these databases often conflict with the requirements of data mining. Transactional database (transactional data)
base) is updated with a small transaction operating in real time. On the other hand, data mining uses a large pattern query that does not need to be performed in real time. In order to resolve this conflict, the data warehouse (data wa)
It is now common to download data from various sources into a centralized resource called the house.

[0006] Downloading and centralizing data from various, and sometimes disparate, sources requires multiple tasks. Extract data from sources, convert it to a common integrated data model, eliminate or correct erroneous or incorrect data and clean it up, all data is stored to form yet another database Need to be integrated into a central warehouse. In addition, it is necessary to confirm that all instances of all business entities, such as customers, products, or employees, have been correctly identified. This is referred to as reference uniformity.
It is known as the problem of al integrity. All of these tasks are difficult to perform while ensuring uniformity of reference, especially when trying to download data from a database that identifies business entities in a slightly different way. Current technology downloads data to data warehouses as an activity independent of data mining. In contrast to data mining, which has a large body of research literature and many commercial products, data warehousing lacks a strong theoretical foundation and few good commercial products.

Because the data warehouse integrates a wide variety of data sources, it specifies an integrated data model for the data warehouse and a data mapping that extracts, transforms, and cleans data from each data source. It is necessary to. In this technical field, rich (information-rich) data such as object-oriented data models
It is known that models are more suitable for defining such integrated data models and defining data mappings than more restrictive data models, for example, data models such as relational models. It has become. However, most data warehouses use a flat record structure, such as a relational model. Because relational databases have very limited data structures, synthesizing more complex data structures is difficult and error prone. Some of the types of data that are not well suited for storage in relational databases include character data, especially hypertext documents, images, sounds, multimedia objects, and attributes with multiple values. Relational databases have a very large number of potential attributes, and are not well-suited for representing records where only some are used in any given record.

[0008] An object database is generally composed of a collection of data objects or information objects. Each information object is uniquely identified by an object identifier (OID). Each information object can have features, and some features can have associated values. An information object can contain or reference another information object.

To support information retrieval in a database including a warehouse database, an index (i
A special search structure called ndx) is used. Large databases require a corresponding large index structure to maintain pointers to stored data. Such an index structure can be larger than the database itself. The state of the art requires a separate index for each attribute or feature. Although this technique can be extended to index a small number of attributes or features in a single index structure, this technique does not work well when there are hundreds or thousands of attributes. Further, there is considerable overhead associated with maintaining the index structure. Since this limits the number of attributes or features that can be indexed, those that are supported must be carefully selected. Transactional database (transa
In the contextual database, the workload is usually well understood, so that the index can be selected to optimize database performance. Choosing which attributes to index is much more difficult because data warehouses generally do not have a well-defined workload.

Further information on the above concepts can be obtained by reference to the following publications:

[00011] L. Aiello, J .; D
oyle, and S.M. Shapiro, edi
tors. Proc. Fifth Intern.
Conf. on Principles of K
nowrepresentation a
nd Reasoning. Morgan Kauf
man Publishers, San Mate
o, CA, 1996. 2. K. Bacrawski. Distribu
ted computer database sys
tem and method, Decmber
1997. United States Pattern
t No. 5,694,593. Assigned
to Northeastern Universi
ty, Boston, MA. (Distributed computer /
Database system and method, December 1997
Monthly, U.S. Pat. No. 5,694,593, transferred to Northeastern University, Boston, Mass., USA). A. Del Bimbo, editor.
The Nth International Co
nreference on Image Analysis
s and Processing, volume
1311. Springer, September
1997. 4. N. Fridman Noy. Knowle
dge Representation for In
tellent InformationRetr
ieval in Experimental Sci
encodes. PhD thesis, College
of Computer Science, Nor
theastern University, Bos
ton, MA, 1997. 5. M. Hurwicks. Take your
data to the cleaners. Byt
e Magazine, January 1997. 6. Y. Ohta. Knowledge-Bas
ed Interpretation of Outd
or Natural Color Scenes.
Pitman, Boston, MA, 198
5. 7. A. Tversky. Features o
f similarity. Psychological
al review, 84 (4): 327-352,
July 1977. 8. S. Weiss and N.W. Indurk
hya. Predictive Data Mini
ng: A Practical Guide. Mo
rgan Kaufmann Publishers,
Inc. , San Francisco, CA,
1998. 9. J. -L. Weldon and A. J
och. Datawarehouse buildi
ng blocks. Byte Magazine,
January 1997.

[0012]

The disclosures of the publications referred to in the Background of the Invention are hereby incorporated by reference. It would be desirable to provide an improved version of a system for data warehousing and data mining that overcomes the performance and other problems and limitations of current systems.

[0013]

SUMMARY OF THE INVENTION The present invention improves the data warehousing infrastructure and support by integrating the two activities of data warehousing and data mining. The term "knowledge extraction" is used herein to refer to the integration of data warehousing and data mining activities.

[0014] The present invention relates to an information retrieval apparatus and method for processing a query from a user including a query (query) for retrieving information from a data warehouse. The apparatus may include a plurality of features (f) in the index database.
mechanisms for finding (or locating) feature fragments and sub-queries at multiple levels included in the query.
query) and an evaluation mechanism for iteratively (or recursively) evaluating a subquery using each of the found features and feature fragments, and an evaluation ( A mechanism for collecting and storing a plurality of results of an iterative (recursive) evaluation of subsequent queries and subqueries after the calculated results are obtained.

[0015] As used herein, "evaluation" is the process by which a response to a query is generated. Information that matches the criteria described in the query, an information location specifier (the location of the information storage location). Specifier) or data about the information. Iterative evaluation is a type of query evaluation in which a new query, called a subquery, is generated from the query and evaluated. The subquery generated in this way is a query tree (query tree).
e), the first query is considered the base node, and each subquery is a level defined by its relevance to the predecessor at the time it was generated, the corresponding level In the tree. All of the subqueries, the ancestor and child queries, are evaluated iteratively and the results are collected,
Stored and provided to the user in response to a query.

The present invention eliminates the need in a conventional retrieval system to provide a new, independent, centralized replica of data in various external databases in a data warehouse. . The present invention avoids such data replication problems in conventional systems, such as data becoming stale or errors prone to occur during replication for warehousing. Instead, the data warehouse can contain an index database, which provides data on information stored in external databases, such as information location specifiers on data inside these databases, information and statistics on relations, etc. Store the entry to be made. The present invention can also provide a robust and versatile index system. The index of the present invention supports the indexing of sporadic records that have a large number of potential attributes, but only some of which are used in a particular record. The invention also supports the indexing of a very large number of attributes, for example in a substantially uniform data structure, which makes it very easy to determine the workload characteristics required to achieve high performance. .

More specifically, according to one aspect of the present invention, a distributed computer database system is interconnected by a network to provide a single data database.
A system adapted to form a warehouse and data mining engine, comprising one or more front-end computers and one or more computer nodes, the system comprising an image, a sound stream, Index objects, including video streams, plain text and structured text.
Objects from external databases are downloaded from the network by nodes called warehouse nodes. The warehouse node extracts some features from the object (the features extracted from the object are called object features), fragments the extracted features into a number of feature fragments, and hashes these feature fragments. ). Each of the hashed feature fragments is sent to one node on the network, called the index node. Each node on the network that receives the hashed feature fragment performs a search in the corresponding partition of the index database using the hashed feature fragment of the object. Local database search results are collected by warehouse nodes. The warehouse node uses these results to determine if the object has already been indexed in the data warehouse. The warehouse node extracts features from the object, fragments the features, and hashes these feature fragments. Each of the hashed feature fragments is sent to one node on the network. Each node on the network receiving the hashed feature fragment stores the feature in the corresponding partition of the index database using the hashed feature fragment of the object.

The query can be, for example, a pattern query (a query for a pattern). Pattern queries are searches for patterns in the data. The pattern query from the user is sent to one of the front-end computers.
To the index node called the home node of the data mining engine.
Forward this pattern query to one of the nodes. The home node decomposes the pattern query into one or more sub-queries, each sub-query stored in memory and computer-executable implementing object features (features about the object) and methods, such as computations. Including programs. This calculation may involve another subquery. The home node fragments each of the subquery features into one or more subquery feature fragments and hashes the feature fragments. Each of the subquery feature fragments is sent to one node on the network according to the hashed feature fragments. Each node on the network that receives the subquery performs a search on the corresponding partition portion of the index database using the hashed feature fragments of the subquery, and the accessed data is used by the subquery calculation. If the calculation of the subquery includes another subquery (this can be zero or one)
(Which may include one or more subqueries), the other subquery is iteratively evaluated, and the data obtained from this iterative evaluation is used in the subquery calculation. The search results of the local index database and the results of all iterative evaluations are collected by the home node. The result of the pattern query is determined by the home node and returned to the user.

In another aspect of the invention, a distributed computer database system is a system interconnected by a network and configured to function as a knowledge extraction engine, wherein the system comprises one or more computers. Including a front-end computer and one or more computer nodes, the system supports both data warehousing and data mining activities.

Consider first the data warehouse activity (the activity of storing data in the warehouse). Downloading objects from another database to the warehouse is performed by the warehouse node. For objects downloaded from another database, the warehouse node first determines whether the object has already been represented in the data warehouse by downloading from yet another database. If so, the warehouse node extracts one or more features of the object, fragments each of the object features into a plurality of feature fragments, and hashes each of these feature fragments. A portion of each of the hashed feature fragments is used as an addressing index (adres) when the warehouse node sends the hashed object feature to an index node on the network.
Used by the warehouse node as a sing index. Each index on the network that receives the hashed object feature fragment
The node performs a search against the corresponding index database using the hashed object feature fragments. The node that finds the data corresponding to the hashed object feature is the OID of the warehouse object that holds this feature fragment.
(Object identifier) is returned. Such OIDs are collected by the warehouse node and a similarity function is calculated. This similarity function is used to determine if the object is already stored in the data warehouse. If it is determined that the object is represented in the data warehouse, the OID of the warehouse object is used for the downloaded object. If not, a unique (uniquely specified) OID is selected for the object. The warehouse node extracts the features of the object, fragments the features, and hashes these feature fragments. A portion of each of the hashed feature fragments is used as an addressing index when the warehouse node sends the hashed object feature fragments to an index node on the network where the features are stored in the data warehouse. Used by house nodes.

Next, consider the data mining activity.
A user who wants to evaluate a query, such as searching for a pattern in the data, sends the query to one of the front-end computers, which then forwards the query to one of the index nodes of the network. The node that receives this query is called the data warehouse home node and breaks it down into one or more subqueries. One subquery includes features and a computer-executable program that implements methods such as calculations, which can include additional subqueries. The home node stores them, fragments the features of each subquery into one or more subquery feature fragments, and hashes each of the subquery feature fragments. A portion of each of the hashed feature fragments is used by the home node as an addressing index when the home node sends a subquery to a node on the network. Each index node on the network receiving the subquery performs a search against the corresponding index database using the hashed subquery features. The node that finds the data corresponding to the hashed subquery feature fragment performs the computation specified in the subquery. If this calculation contains no other subqueries, the result of this calculation is
Returned to node. If the calculation involves another subquery, the node assumes the role of the home node for the subquery included in the calculation. More specifically, the node hashes the feature fragments of the included subquery and sends the subquery to other nodes. This process continues iteratively until the calculations are completed and the final result is returned to the original home node. Upon receiving the calculation result, the home node returns the original pattern
Execute the rest of the data set specified in the query and send this information to the front-end node. The front end node formats the response to the user and sends the formatted response to the user.

[0022]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Referring to FIG. 1 for a general schematic diagram, a distributed computer database
One embodiment 100 of the system includes a user computer 102 communicating with a front-end computer 104, for example, via a network 106. Alternatively, front-end computer 104 may be a user computer. The front-end computer 104 also communicates with one data warehouse and data mining engine, which is
It includes one or more computer nodes 106, 108 interconnected by an area network 110. Individual computer nodes 106, 108 may include local disk 112 or, alternatively or additionally, obtain data from a network disk server (not shown).

There are several types of data warehouse computer nodes 106, 108, including index nodes 106 and warehouse nodes 108. The data warehouse nodes 106, 108 need not represent independent computers. In one embodiment, the data warehouse is an index warehouse.
It is implemented as a single computer that performs all the roles of node 106 and warehouse node 108. In another embodiment, the data warehouse is implemented as a separate computer for each of index node 106 and warehouse node 108. One skilled in the art will recognize that many variations are possible that fall within the scope and spirit of the invention.

Consider first an exemplary method 200 for downloading an object, and referring to FIG. 2, in one embodiment, the object is stored in an external database 2 by one or more warehouse nodes 108.
01 (step 201). For example, from a previous download from another database,
If the object is already represented in the data warehouse, the warehouse node 108
Extracting a number of features from an object for the purpose of identifying the object, as specified in the warehouse's integrated data model. For example, a person may be identified by an employer "ID", account number, name, address, telephone number, email address, etc., or some combination thereof.

Various feature extraction techniques can be used. With relational attribute values, such as transaction dates, possible values can be divided into a set of non-overlapping ranges in a row. Dividing the field values in this manner is called discretionization. The actual value may also be included in the index entry.

By parsing (decomposing) the structured document to create a data structure, and then dividing this data structure into (possibly overlapping) substructures called fragments, the features are converted from the structured document. Is extracted.
Fragment related to subquery (fragmen
This is called a probe because t) is used to find a matching fragment in the database.

The features extracted from the unstructured document are organized into a data structure containing a set of interrelated substructures,
As is the case with structured documents, component s (possibly duplicates)
substructures), and these component substructures are unstructured document fragments.

A wide variety of feature extraction algorithms have been developed for media such as sound, images, video streams, etc., such as edge detection algorithms, segmentation algorithms and object classification algorithms for images. Fourier transforms and wavelet transforms as well as many filtering algorithms are also used to extract features from images and sounds. Features can also be added to objects by manual or semi-automatic means. Such added features may be annotations or metadata (meta-data
a). Depending on whether the annotation is a relational database record, a structured document, or an unstructured document, features are extracted from the annotation using one of the techniques described above. If a feature has a value associated with it, these values are partitioned. You can also specify the relevance between features. For example, one feature may be included in or adjacent to another feature. The integrated data model specifies the feature extraction algorithm as well as the structure of the feature.

The warehouse node 108 encodes each feature fragment of the object using a predetermined hash function. The data in the system is previously stored locally at various index nodes using this hash function to generate an index to the data in the local database. In other words, the same hash function is used to generate an index for data storage (for storing data), and by generating a hashed probe for the object, the data is stored when the data is stored.
Ensures that data is evenly distributed across the index nodes 106 of the warehouse.

In one embodiment, the hash value obtained by using the hash function has a first part, which is the index of the destination to which the data is to be stored (the destination). Used to identify the index node to which the feature fragment will be sent as a node or probe. The hash value also has a second part, which is the local index value, which is used when storing data or when
It is used to determine a location (storage location) in memory when data is to be retrieved from a node. That is, the hashed object feature fragment is determined by the first part of the hash value,
Specific index node 1 in the data warehouse
06 are distributed as probes (step 202).

The index node 106, such that the hashed feature fragment used when the data was initially stored in the index node 106 and the index node's probe match, will Respond to the retrieval message by sending the OID that matches the hashed feature fragment to warehouse node 108 (step 203). That is, any match between the hashed probe and the local hash table of the hashed feature fragments is the warehouse node 108 that first hashed the object feature fragment.
Returned to or collected there.

The warehouse node 108 then
It is determined whether one of the IDs represents the same object as the object to be stored in the warehouse. This determination is made by the warehouse node comparing the degree of similarity between the object to be stored in the warehouse and the object whose OID has been returned. In one embodiment, the measure of similarity is determined by features common to the objects and features of the object that is to be stored in the warehouse, rather than features of the object for which the OID was returned.

The measure of this similarity is the Tv referenced above.
ersky feature contrast model (Featur
e Contrast Model). The first term adds a positive number to the similarity value, and the second term adds a negative number. Further, the second term is multiplied by a predetermined constant so that the second set of features is
The effect on the similarity is less than the characteristics of the set.

If it is determined that the object is represented in the data warehouse, the OID is already available for that object. If not, a unique OID is selected for the object.

The warehouse node 108 then extracts all of the object's features according to the data warehouse's integrated data model. The feature extraction technology has been described above. The warehouse node 108 fragments (divides) each feature into feature fragments and encodes each feature fragment of the object using a predetermined hash function as described above. In one embodiment, the hash value obtained using the hash function will be transmitted and stored (step 20).
4) A first part used to identify the index node to which the data is to be stored (storage destination), and a local index used to determine where to store the data (step 205) in the index node. A second part that is a value.

Next, an exemplary method 30 for processing a query 30
0, and with reference to FIG.
A query is transmitted from the computer 102 (step 3
01) In one embodiment, the front-end computer 104 receives the query. front end·
Computer 104 is responsible for establishing a connection with user computer 102 so that the user can send objects and receive responses in an appropriate format.
Front-end computer 104 is also responsible for some authentication and management functions. In one embodiment, front-end computer 104 is a World Wide Web server that communicates with user computer 102 using the HTTP protocol.

After confirming that the query is acceptable, the front-end computer 104 sends the data
Perform any reformatting needed to adapt the query to the requirements of the warehouse.
The front-end computer 104 sends a query to one of the data warehouse index nodes 106 (step 302), which is defined as the data warehouse home node 107 for the query. .

The home node 107 sends a plurality of queries (1
(Or more). Each subquery has one characteristic and specifies a computer-executable method, such as a calculation. This calculation determines what action the subquery will take. The most common calculations are statistical functions that aggregate information stored in a data warehouse. This calculation can include similarity measures, such as the lowest strength required to accept a match, and statistical calculations, such as the mean or standard deviation. This calculation can include another subquery.

For each subquery, home node 1
07 fragments the subquery features into subquery feature fragments and encodes the feature fragments using a predetermined hash function as described above.
The hashed feature fragments and subqueries are sent by the home node to the index node using the hashed feature fragments described above (step 303).

If the index feature fragment used when the data is first stored in the index node matches the hashed feature fragment, the index node 106 determines that the data matches the hashed feature fragment. Retrieves the data in the local hash table of index entries and responds to the subquery by performing the calculations specified in the subquery. If the calculation involves any other subqueries, the index node is called a component subquery and serves as the home node for the new query processed as described above (step 304). For example, a subquery may be used to find sales of other products that correlate to each customer who purchased the equipment last month. Regardless of whether this calculation includes another subquery or not, the index node returns the result of the calculation to the home node 107 of the subquery it received (step 305).

Upon receiving the results of all the subqueries of the original query, the home node 107 performs some sort of data aggregation, for example, calculating the average or standard deviation specified by the original query, and sends the obtained information to the user. return.
In one embodiment, the returned information is sent to front-end computer 104 (step 30).
6) The computer formats the response appropriately and then sends the response to the user (step 30).
7). In another embodiment, the information to be returned may include user computer 102 via network 105 without, for example, front-end computer 104.
Sent directly to.

Consider now the message format used in the preferred embodiment, and referring to FIG.
An exemplary format for a warehouse message is 4
Contains two fields. These include a header 402, an object identifier (QID) 403, a hashed object fragment (HOF) 404, and a value 4
05. Header field 402 specifies that this message is a warehouse message, and also specifies the destination index node. The destination index node is determined on the first part of the hashed object fragment. OID field 403
Contains an object type specifier (object type specifier) and an object identifier. HOF field 404 includes a fragment type specifier (fragment type specifier) and a second portion of the hashed object fragment created by the hash module (see FIG. 5). Value field 405 contains the value of the option associated with the fragment. The fragment format specifier determines whether the warehouse message includes a value field 405, and if the warehouse message includes a value field, the fragment format specifier determines the size of the value field.

Referring to FIG. 4b, an exemplary format of a warehouse response message includes two parts.
These are identifiers and values. The identifier part has four fields. Header 406, OID1 407,
OID2 408 and weight 409. header·
Field 406 specifies that this message is a warehouse response message and the destination warehouse
Also specify the node. The destination warehouse node is the warehouse node to which the corresponding warehouse message was received. The two OID fields 407, 408 contain an object type specifier and an object identifier. First OID field 407
Is the same as the OID field 403 of the corresponding warehouse message. Second OID field 408
Identifies the object that has been indexed so far. Weight field 409 contains an optional weight associated with the object identified by OID 1407.
The object type specifier of OID1 determines whether the warehouse response message includes a weight field,
If the warehouse response message includes a weight field, the OID1 object type specifier determines the size of the field. The value portion of the warehouse response message contains a plurality of fields 4 to contain data related to the object identified by OID2 408.
10 inclusive. The structure and size of the value part are determined by the OID2 object format specifier.

Referring to FIG. 4c, an exemplary format of the insert message has four fields. These are a header 411, an OID 412, a HOF 413, and a value 414. Header field 414 specifies that the message is an insert message, and also specifies the destination index node. The destination index node is determined on the first part of the hashed object fragment. OID field 412
Contains an object type specifier and an object identifier. The HOF field 413 contains the fragment type specifier and the second part of the hashed object fragment created by the hash module (see FIG. 5). Value field 414 contains the value of the option associated with the fragment. The fragment format specifier determines whether the insert message includes a value field 414, and if the insert message includes a value field, the fragment format specifier determines the size of the value field.

Referring to FIG. 4d, an exemplary format of a subquery message has two parts. These are the identifier and the subquery. The identifier part has four fields. A header 415, a subquery identifier (QSID) 416, a hashed query fragment (HQF) 417, and a value 418. Header field 415 indicates that this message is a subquery
Specifies that this is a message, and also specifies the destination index node. The destination index node is determined on the first part of the hashed query fragment. QSID field 416 contains a query type specifier and a subquery identifier. HQF field 417
Contains a fragment type specifier and a second part of the hashed subquery fragment created by the hash module (see FIG. 5). Value field 418 contains the value of the option associated with the fragment. The fragment format specifier determines whether the subquery message includes a value field 418, and if the subquery message includes a value field, the fragment format specifier determines the size of the value field.
The subquery portion of the subquery message includes a plurality of subqueries. Subqueries without subqueries
The message is called a simple subquery message.

Referring to FIG. 4e, an exemplary embodiment of a subquery response message has two parts, an identifier and a value. The identifier part consists of two fields, header 42
0 and QSID421. Header field 42
0 specifies that this message is a subquery response message, and also specifies the destination index node. The destination index node is the corresponding subquery
It is the same as the receiving index node when the message is received. QSID field 421 contains a query type specifier and a subquery identifier. The value portion of the subquery response message has a number of fields 422 for containing the result data of the subquery. The structure of the value part is specified by the query format specifier.

Each node of the distributed computer database system includes a communication module responsible for sending and receiving messages between the nodes, as described below and shown in FIGS. To send a message, (1) queue the message (qu) before sending it over the communication medium.
e), (2) the actual transmission via the communication medium, (3) the task of processing the message when the message is received by a module determined by the message type (message type). Is necessary. The message format determines the command issued to the receiving module. Commands determine the means by which messages will be processed by the module. The destination node to which the message is sent is specified in the header field of each message. When a message is received from another node, the type of the message determines which module processes the message. The message format is specified in the header field of each message. The home node's communication module is also responsible for communicating with multiple front-end nodes. One front-end node sends a query to the home node, which sends results, such as graphs and formatted tables, to the front-end node.

Considering the exemplary embodiment of the node described above, and referring to FIG. 5, the warehousing node (warehouse node) 500 scans an external database into the warehouse by the knowledge extraction engine. A downloader (download unit) 502 that downloads objects for housing (warehouse conversion) and indexing can be provided. Each warehouse node 500 may have a different type of downloader 502. For example, one type of downloader is a standard SQL protocol, such as ODBC or relational.
Data can be downloaded from a relational database using a proprietary protocol or the like defined by the database provider. Downloading in this case is performed using one or more SQL queries (query according to the SQL protocol). In another example, the downloader may negotiate to obtain content from the article syndicators via the Internet to exchange information content (I).
CE: Information and Content
t Exchange) may be a subscriber. This is a suitable mechanism for acquiring time-sensitive contents such as newspaper article distribution. The downloader 502 transmits the object to the feature extraction unit 504.

The feature extraction unit 504 extracts features from objects. If the object is a single record in a relational database, feature extraction includes steps such as selecting the fields to be indexed, reformatting the fields, eliminating or correcting data determined to be incorrect. Image feature extraction is performed by edge detection, image object identification, and determination of relevance between image objects. In another embodiment,
Feature extraction for an image is performed by calculating a Fourier transform or a wavelet transform. Each of the Fourier transforms or wavelet transforms constitutes one extracted feature. Features are indexed using multiple insertion messages.

The feature extractor 504 maps each object identifier of the external database to the object identifier of the knowledge extraction engine. Each external database may have its own mechanism for assigning object identifiers, and features for the same object may be stored in each external database with a different object identifier. For example, one external database may use social security numbers. Another external database may use employee identifiers. Mapping from external object identifiers is achieved using multiple warehouse messages.

The fragmentor 506 calculates a fragment included in each feature. Each fragment contains a finite set of related components in the feature. In one embodiment, the feature fragment includes each attribute and each association in the data structure defining the feature. For objects in the form of relational database records, features are selected by the feature extractor 504,
The reformatted and corrected attributes. This fragment is forwarded to the hash module.

The hash module 508 calculates a hash function for the fragment. In one embodiment, the hash function is an IETF (Internet Ent
published by the Networking Working Group of the G. Gingering Task Force in October 1990, on the Internet or at the Massachusetts Institute of Technology Computer Science Laboratory (MIT Laboratory for C).
Omputer Science, Cambridge
e, MA, USA). RFC (Request For Comment) available from Rivest
t) MD4 message digest algorithm described in the 1186 specification. The hash module 508 sends either the warehouse message or the insert message to the communication module 510, depending on whether the purpose of the fragment is to implement a mapping of the object identifier or to index the features of the object, respectively. Forward.

Similarity comparator (similarity calculator) 51
2 receives the warehouse response message from communication module 510 and creates an insert message, which is forwarded to communication module 510. The similarity comparator 512 collects all responses from the warehouse for the object to which the identifier is mapped.
For each object in the response, the similarity comparator 512 determines the relevancy of each object identifier returned in the search. This determination of relevancy is made by comparing the degree of similarity between the object to which the identifier is mapped by the warehouse node and the object whose OID is returned. In one embodiment, the measure of similarity between the query and the object is a cosine measure, and the expression COS
(V, w). Here, the vector v represents a query, and the vector w represents an object. These vectors lie in space such that each fragment represents one dimension of space. If a matching OID is found, the OID is used as a mapped object identifier and the OID is
4 is transferred. If no matching OID is found, a new object identifier is selected and transferred to the feature extraction unit 504.

Referring now to FIG.
Node 600 is a fragment table module 6
02, and this fragment table module sends warehouse messages, insert messages, simple subquery messages to the communication module 6
04. For warehouse messages,
The fragment table module 602 retrieves one value (one entry) in the local hash table 603 using the hash value of the HOF field. This type specifier in the HOF field and this entry in the local hash table are forwarded to the fragment comparator 606, a component that measures similarity between fragments. For simple subquery messages, the fragment table module 602 retrieves an entry in the local hash table 603 using the hash value in the hashed query fragment field. Local hash
Entries in table 603 are returned to query processor 608 using a subquery response message. In the case of an insert message, the fragment table module 602 adds the OID and value fields of the insert message to entries in the local hash table
Change entry 3

The fragment comparator 606 receives an entry from the fragment table module 602. The comparison function is determined by the HOF type specifier transferred from the fragment table module 602. Fragment table using comparison function
OI in entry transferred from module 602
Determine the relevance of the D and value fields. 1
In one embodiment, the comparison function determines a similarity weight,
The OID with the highest similarity weight is considered relevant. Relevant OIDs and their similarity weights are transferred to communication module 604 using a warehouse response message.

The query parser (query decomposing unit) 612 stores the query calculation tree (qu) stored in the memory 613.
The query is decomposed (analyzed) into an erroneous computation tree, and the computation tree is a data structure specified in terms of a plurality of nodes and their mutual relations. The nodes of the query computation tree are either internal nodes or leaf nodes. Internal node is 1
A node that has one or more child nodes. Internal nodes specify how to combine the results of the child nodes. For example, the results may be added or averaged, or used to calculate a standard deviation value. Leaf nodes are nodes that do not always have child nodes beyond them. Leaf nodes are either constant values or simple subquery nodes. The subquery node is composed of multiple component subqueries (compon
ent sub-query).
Each component subquery is also specified using a corresponding query computation tree. The query computation tree is transferred to the query processor 608.

The query processor 608 is responsible for managing query processing. Upon receiving the query computation tree from the query parser, the processor determines the query identifier (QI
D) is assigned to the query, and a subquery identifier (SQID) is assigned to each of the leaf nodes specifying the subquery. A subquery without a component subquery is called a simple subquery. The subquery is processed by sending a subquery message using the communication module 604 to the designated index node. The query processor 608 at the designated destination index node processes the subquery message by transferring the simple subquery message to the fragment table module 602, which responds with a subquery response message.
Query processor 608 then uses the subquery
Send this subquery response message to the index node that sent the message. As a result, the query
Processor 608 sends and receives both subquery messages and subquery response messages. When the subquery response message is received, the processing specified in the query calculation tree is performed. If the subquery has a component subquery, this subquery requires the processing of an additional subquery (another subquery). The entire query (all of these subqueries,
Once these queries (including these subqueries called "nested subqueries") are computed, the results are formatted and sent to the front end when this query was received. For example, the result may be provided as a graph or table. Thus, since each of the queries or each of the nested subqueries is associated with one level within the tree, query processor 608 is responsible for processing all levels of queries within this tree.

FIG. 7 illustrates a conventional system architecture using an exemplary computer system 800. Each of the user computer, front-end computer, and computer nodes, including index nodes and warehouse nodes,
It can be implemented as an instance of computer system 800. The exemplary computer system of FIG. 7 is discussed for illustrative purposes only, and should not be considered a limitation of the present invention. In the description that follows, reference may be made to terms commonly used when describing a particular computer system, but the concept described herein is illustrated in FIG.
This applies equally to other computer systems, including systems having a different architecture than the system shown in FIG.

Computer system 800 includes a central processing unit (CPU) 805, which may include a conventional microprocessor, a random access memory (RAM) 810 for temporary storage of information, and a permanent storage of information. -Only memory (ROM) 81 for
5 is included. The memory controller 820 controls the system RA
It is provided to control M810. A bus controller 825 is provided for controlling the bus 830,
Interrupt controller 835 is used to receive and process various interrupt signals from other system components.

The mass storage is a diskette 842, CD-
Provided by the ROM 847 or the hard disk 852. Data and software can be exchanged with the client computer 800 via a removable (removable) medium, such as a diskette 842 or a CD-ROM 847. Diskette 842 is the controller 8
40 can be inserted into the diskette drive 841 connected to the bus 830. Similarly, CD-ROM
847 can be inserted into a CD-ROM drive 846 connected to the bus 830 by a controller 845. Finally, the hard disk 852 is a part of the fixed disk drive 851, and is connected to the bus 830 by the controller 850.

[0061] User input to computer system 800 can be provided by a number of devices. For example, a keyboard 856 and a mouse 857 are connected to the bus 830 by a keyboard and mouse controller 855. The audio transducer 896 functions as both a microphone and a speaker, and is connected to the bus 830 by the audio controller 897. It should be apparent to those skilled in the art that other input devices, such as pens and / or tablets, voice input microphones, etc., can be connected to client computer 800 via bus 803 and a suitable controller. DMA controller 860 is provided to perform direct memory access to system RAM 810. The visual display is generated by the video controller 865,
This controls the video display 870. Computer system 800 also includes a network adapter 890 that allows client computer 800 to be interconnected to network 895 via bus 891. Network 895 may be a local area network (LAN), a wide area network (WAN), or the Internet, using general-purpose communication lines that interconnect a number of network devices.

Computer system 800 is generally controlled and coordinated by operating system software. Among the computer system control functions, the operating system controls the allocation of system resources and performs tasks such as process scheduling, memory management, networking and I / O services.

The software implementation of the components of the embodiments described above can be implemented on a computer readable medium, such as diskette 842, CD-ROM 84 in FIG.
7, the ROM 815 is fixed on a tangible medium such as a fixed disk 852, or a network 89
5 may include computer instructions and routines that can be communicated via a modem or other interface device, such as a communication adapter 890. The medium 891 may be a tangible medium including, but not limited to, an optical communication line or a hard-wired communication line, or may be implemented with any wireless or other communication technology including, but not limited to, microwaves, infrared, etc. . This can also be the Internet. When transmitted, the software component may take the form of a digital signal embedded in a carrier. The series of instructions to the computer implements all or part of the functions described herein above in connection with the present invention. Those skilled in the art will appreciate that the instructions for such a computer may be written in many programming languages used in many computer architectures or operating systems. Further, such instructions may be stored using any current or future memory technology, including but not limited to semiconductor memory devices, magnetic memory devices, optical memory devices, or other memory devices, optical communication, infrared communication, microwave communication or The transmission may be made using any current or future communication technology, including but not limited to other transmission technologies. Such computer program products may be distributed, for example, as removable media with attached printed or electronic documents, such as shrink-wrapped software, installed on computer systems on, for example, a system ROM or fixed disk, or may be installed on the Internet, It can be distributed from a server on a network such as the web or an electronic bulletin board.

[0064]

Having described representative embodiments of the present invention, various changes and modifications can be made to achieve some of the advantages of the present invention without departing from the spirit and scope of the invention. Obtaining will be apparent to those skilled in the art. It will be apparent to those skilled in the art that other components that perform the same function may be replaced as appropriate. Further, the method of the present invention may be implemented either entirely as a software implementation using appropriate processor instructions, or as a hybrid implementation that achieves the same result using a combination of hardware and software logic. Can be. Furthermore, aspects such as the size of the memory, the particular organization of the logic and / or instructions used to implement a particular function, and other changes to the inventive concept are intended to be covered by the appended claims. Intended. Accordingly, the invention is to be construed as limited only as indicated by the appended claims.

[Brief description of the drawings]

FIG. 1 is a block diagram of one embodiment of a distributed computer database system according to the present invention.

FIG. 2 is a block diagram of the distributed computer database system of FIG. 1 in the form of a flowchart, which illustrates data warehouses from another source according to one embodiment of the present invention. It shows how to download information to.

FIG. 3 is a block diagram of the distributed computer database system of FIG. 1 in the form of a flowchart, which illustrates a method of responding to a query according to an embodiment of the present invention.

FIGS. 4a to 4e respectively illustrate the format of a warehouse message, a warehouse response message, an insert message, a subquery message, a subquery response message used in connection with the embodiment of FIGS. It is a block diagram shown.

FIG. 5 is a representative block diagram of the home node of FIGS. 1, 2 and 3 according to one embodiment of the present invention.

FIG. 6 is a representative block diagram of the index nodes of FIGS. 1, 2 and 3 according to one embodiment of the present invention.

FIG. 7 is a block diagram of a computer system according to an exemplary embodiment of each of a user computer, an index node, and a warehouse node.

Claims (17)

[Claims] 1. In a distributed computer database system in which a plurality of index nodes and a plurality of warehouse nodes are connected by a network, an object or a location of an object is warehoused in a manner conducive to knowledge extraction using a query. Wherein the method comprises: A) a warehouse node extracting a plurality of features of a first group from an object downloaded from another database; and B) the extracted object. Fragmenting each of the features into a plurality of object feature fragments; C) the warehouse node hashing each of the object feature fragments of the first group of the plurality of object features; D) wherein each of the hashed object feature fragments has a first portion and a second portion; and D) the warehouse node comprises a plurality of feature fragments of the first group. Transmitting each of the hashed object feature fragments to a corresponding one of the plurality of index nodes indicated in the first portion of each of the hashed object features; and E) the index node comprises: , The index
Using the second portion of the corresponding hashed object feature fragment to access data according to a local hash table located on the node; and F) the corresponding hashed object feature fragment. G) returning to the warehouse node a plurality of object identifiers corresponding to the accessed data; and G) the warehouse node from the plurality of object identifiers Determining whether to assign the object identifier of the object to the object or to assign an object identifier that is not yet used to the object; H) the warehouse node according to the determination, Assigning an object identifier to the object; I) the warehouse node extracting a second group of features from the object; J) a plurality of object features of the extracted second group. Fragmenting each of the object features into a plurality of object feature fragments; K) the warehouse node hashing each of the object feature fragments of the second group of the plurality of object features, wherein the hashing comprises: And L) wherein the warehouse node determines that the first portion of each of the hashed object feature fragments comprises: a first portion of each of the hashed object feature fragments; The plurality of indexes indicated by Transmitting the scan nodes to corresponding ones of each of said hashed object features fragments of the plurality of objects, wherein a fragment of said second group, M) said index node, the index
Using the second portion of the corresponding hashed object feature fragment to store data according to a local hash table located at the node. 2. The warehouse node further determines a measure of similarity between the accessed data and the object after returning a plurality of object identifiers of the first group. The method of claim 1, comprising: 3. The similarity measure is determined by a similarity function based on features possessed by both the accessed data and the object and based on features owned only by the object. The method according to claim 2, characterized in that: 4. A method of data mining using queries in a distributed computer database system in which a plurality of index nodes are connected by a network, the method comprising: A) a first of the plurality of index nodes. And B) extracting a plurality of sub-queries from a query by a user, wherein the home node extracts a plurality of sub-queries from a query by a user. C) fragmenting each of the subquery features into a plurality of subquery feature fragments; D) the home node. Each of the subqueries includes a subquery feature, a plurality of subqueries, and a calculation specification. Is the said each of the subquery feature fragments Hashing each of the subquery feature fragments, wherein each of the hashed subquery feature fragments has a first portion and a second portion; and E) the home node Transmitting the hashed subquery feature fragments to a corresponding one of the plurality of index nodes indicated in the first portion of each of the hashed subquery feature fragments; The node has the index
Using said second portion of each of said hashed subquery feature fragments to access data according to a local hash table located on a node; and G) said index node is said home node. Iteratively evaluating each subquery of said plurality of subqueries contained in said corresponding subquery sent by said index node, said index node serving as a home node of said subquery of said plurality of subqueries H) the index node comprises: the iterative search of each of the subqueries of the plurality of subqueries included in the accessed data and the corresponding subqueries sent by the home node. According to the information determined by the evaluation , Calculating the information in accordance with the calculated specifications subquery said corresponding transmitted by the home node, I) each of said index nodes, the home
Returning the information to a node. 5. The method of claim 4, further comprising, before the step of extracting the subquery from the query, receiving the query from the user at the home node. 6. A distributed computer database system for storing information objects or locations of information objects in a warehouse, the system comprising: A) a plurality of warehouse nodes and a plurality of index nodes. Wherein the plurality of warehouse nodes and the plurality of index nodes are connected by a network, and B) each of the warehouse nodes, when downloading an object, From first
Extracting a plurality of features of a group of the object features, fragmenting each of the object features into an object feature fragment, hashing each of the object feature fragments, and hashing the object feature having a first portion and a second portion. Transmitting each of said hashed object feature fragments to a corresponding one of said plurality of index nodes indicated in said first portion of said hashed object feature fragments; C) Each of the index nodes uses the second portion of the hashed object feature fragment to access data according to a local hash table located at the index node;
Returning a plurality of object identifiers corresponding to the accessed data to the warehouse node; and D) the warehouse node uses one of the object identifiers of the plurality of object identifiers or is still in use. Assigning any of the unidentified object identifiers to the object, extracting a second group of features from the object, and assigning each of the extracted features of the second group of features to a plurality of object features. Fragmenting into a fragment, hashing each of said object feature fragments of said plurality of object features of said second group into a hashed object feature having a first part and a second part, said hashed object The first of the feature fragments The plurality of indexes indicated by the part
Sending each of said hashed object feature fragments to a corresponding one of the nodes; E) each of said index nodes using said second portion of said hashed object feature fragment Storing an object or an object location according to a local hash table located on said index node. 7. The warehouse node determines a similarity measure for use in assigning an object identifier to the object, the similarity measure between the accessed data and the object. 7. The distributed computer database system according to claim 6, wherein: 8. The warehouse node measures similarity using a feature held by both the accessed data and the object and a similarity function determined by the feature held by the object. The method of claim 7, wherein 9. A distributed computer having a data mining tool for processing queries from users.
A database system comprising: A) a plurality of index nodes connected by a network; and B) each of said index nodes, referred to as a home node of said query, comprising a query from a user. Receiving a plurality of subqueries from the query and a plurality of subquery features from each of the subqueries, fragmenting each of the subquery features into a plurality of subquery feature fragments, and hashing the subquery features of the plurality of subqueries. Into a hashed subquery feature having a first portion and a second portion, and hashing the hashed subquery feature fragment to a corresponding one of the plurality of index nodes indicated in the first portion of the fragment. Of the subquery feature fragment Characterized by transmitting the s, C) In addition, each of the index node, the second of said hashed subqueries feature fragment
Accessing data according to a local hash table located on the index node, iteratively evaluating each subquery included in the corresponding subquery, using the accessed data and the A distributed computer database system comprising: calculating information according to information determined by iterative evaluation; and returning the information to the home node. 10. A distributed computer database system for warehousing and data mining, comprising: A) a plurality of warehouse nodes and a plurality of index nodes, wherein the plurality of warehouse nodes. And the plurality of index nodes are connected by a network. B) Each of the warehouse nodes, upon receiving the download command, responds to the download command by a predetermined one. C) queuing the task, C) queuing the download task in response to the download command, a first task from the object downloaded by the download command.
Extracting a plurality of features of the group of the first group, fragmenting each of the object features into a plurality of object feature fragments, hashing each of the object feature fragments of the plurality of object features of the first group to form a first portion And a retrieval message including each of the hashed object feature fragments as a hashed object feature fragment having a second portion and a second portion indicated by the first portion of the hashed object feature fragment. D) the index node, upon receiving the retrieval message, using the second portion of the hashed object feature fragment to transmit the index to the corresponding one of the index nodes. E) accessing the data according to a local hash table located on the warehouse node, and transmitting a message returning a plurality of object identifiers corresponding to the accessed data to the warehouse node; The warehouse node, upon receiving the plurality of object identifiers from the plurality of index nodes, assigns either one of the object identifiers of the plurality of object identifiers or an unused object identifier to the object. allocation,
Extracting a plurality of features of a second group from the object, fragmenting each of the object features of the plurality of object features of the second group into a plurality of object feature fragments, Hashing each of the object feature fragments of the feature fragment into a hashed object feature fragment having a first portion and a second portion, indicated by the first portion of the hashed object feature fragment Transmitting an insert message including each of the hashed object feature fragments to a corresponding one of the plurality of index nodes; F) the index node, upon receiving the insert message, Using the second portion of the stored object feature fragment to store data according to a local hash table located on the index node. 11. The warehouse node determines a similarity measure for use in assigning an object identifier to the object, the similarity measure between the accessed data and the object. The distributed computer database system according to claim 10, wherein: 12. The warehouse node determines similarity using a similarity function determined by a feature held by both the accessed data and the object and a feature held by the object. The method according to claim 11, wherein the measurement is performed. 13. Having a data mining tool,
A distributed computer database system for processing a query from a user, comprising: A) a plurality of index nodes connected by a network; B) each of said index nodes receiving a command from a user. The index node is referred to as the home node of the command and queues a predetermined task in response to the command; C) the queued query task is Extracting a plurality of subqueries from the query contained in the query command in response to the query command from
Extracting a plurality of subquery features from each of the extracted subqueries; fragmenting each of the subquery features into a plurality of subquery feature fragments; hashing each of the subquery feature fragments to form a first portion and a second portion. Each of the hashed subquery feature fragments to a corresponding one of the plurality of index nodes indicated by the first portion of the hashed subquery feature fragment D) the index node, upon receiving the subquery message, uses the second portion of the hashed subquery feature fragment to transmit the index query message. No Accessing data according to a local hash table located on the code, iteratively evaluating each subquery included in the corresponding subquery,
A distributed computer database system, comprising: calculating information according to the accessed data and the information determined by the iterative evaluation; and transmitting a message returning the information to the home node. 14. The query message according to claim 13, wherein the query message requests predetermined data from the index node in response to a query included in the query command from the user. The method described in. 15. An information retrieval apparatus for processing a query for retrieving information from a database, comprising: A) a mechanism for finding a plurality of features and feature fragments in an index; and B) a mechanism for finding the information. An evaluation mechanism for identifying a plurality of levels of subqueries included in the query and iteratively evaluating the subquery using each of the found features and feature fragments; C A) a mechanism coupled to the evaluation mechanism, wherein a plurality of results of the iterative evaluation of the query and the subquery are collected after a result calculated for the entire query is obtained and stored in a memory; And a mechanism for extracting the information. 16. A method for processing a query for retrieving information from a database, the method comprising: A) finding a plurality of features and feature fragments in an index; and B) finding a plurality of levels of the query. Identifying a plurality of subqueries and iteratively evaluating the subqueries using each of the found features and feature fragments; and C) obtaining the calculated results for the entire query and A method comprising collecting and storing a plurality of results of an iterative evaluation of a subquery. 17. A computer program product for processing a query for retrieving information from a database, the computer program product including a computer executable program embodied on a computer readable medium, The computer-executable program comprises: A) a first code portion for finding a plurality of features and feature fragments in an index; and B) identifying a plurality of levels of sub-queries included in the query. A second code portion for iteratively evaluating the sub-query using each of the found features and feature fragments; and C) the query after the results computed for the entire query are obtained. And a third for collecting and storing multiple results of the iterative evaluation of the subquery. A computer program product comprising a code part.