JP4502063B2 - Data warehouse system, query processing method used therein, data collection method and apparatus therefor, and billing system - Google Patents

Description

  The present invention relates to a data warehouse system in a distributed network computing environment (hereinafter referred to as a distributed environment), a query processing method used therein, a data collection method and apparatus therefor, and a charging system and method.

  Due to the spread of computer systems due to lower prices, the emergence of highly reliable software, and the demand for more efficient social systems, various information is being put online and used. For example, in business activities, various business data such as sales status in stores, product management information, and customer information are processed by a computer. The data warehouse system is used to effectively utilize the computer data used in these mission-critical operations not only for mission-critical operations but also for other purposes such as product sales trend surveys and customer preference analysis. Is actively used. As for the data warehouse, for example, Non-Patent Document 1 describes a configuration method and a usage method. As its name suggests, the data warehouse system is a data warehouse, and is spreading to store and manage enormous amounts of data used in mission-critical operations.

In recent years, it has been found that new information that has been overlooked so far can be obtained by examining the data stored and managed in this data warehouse from various angles. For example, when sales data was analyzed at a supermarket, the relationship between two seemingly unrelated relations, such as “men on the weekend to buy beer and diapers at the weekend,” became clear. There is an example that sales could be increased by arranging beer and diapers in close proximity. As described above, a method for finding useful information that has been overlooked in the data is called data mining.

  In parallel with the spread of computers, the progress of network technology represented by the Internet is also remarkable. This network technology, for example, uses a distributed network infrastructure technology represented by CORBA (Common Object Request Broker Architecture), which is described in Section 1 of Non-Patent Document 2, to network various information. It is becoming possible to use it via the Internet.

  From these backgrounds, we will try to obtain more useful information than before by integrating data on multiple databases and data warehouses on the network and using methods such as data mining. The attempt to do so is a natural outcome. Regarding the integrated use of databases, for example, as described in Non-Patent Document 3, etc., researches on heterogeneous databases, federal databases, multi-databases, etc. have been actively conducted mainly by academic societies. Many studies have been discussed, but most of the research focused on how to integrate and use different data in consideration of the heterogeneity of data.

However, when trying to build a data warehouse system in a distributed environment, there are many performance issues due to its large data scale and the complexity of query processing for data compared to conventional database search processing. The problem appears. For example, with regard to the data scale, a data warehouse of several TB (terabytes; 10 12 bytes) class has been constructed as of March 1998. With regard to the complexity of query processing, modeling support for data warehousing, data mining, etc., is a standard benchmark widely accepted in the industry, “TPC BENCHMARK D (Decision Support) Standard.
Specification ”, Revision 1.2.2, Trans
A suitable example is an action processing performance council query. For example, when a series of TPC-D queries are executed for large-scale data of 1 TB, even the world's fastest computer as of May 1998 takes several tens of minutes to several hours.

  As shown in FIG. 11, the data warehouse system is used by storing and managing data, managing the storage device 1105, and requesting the server 1102 for executing query processing, the client 1101 to perform processing. The client-server type that receives results is common.

  However, as shown in FIG. 14, an unspecified number of users (clients) 1401 to 1402 request inquiry processing to servers 1403 to 1404 such as an unspecified number of data warehouses and databases via a network 1405 ( 1406) Assuming a client-server usage mode in a distributed environment such as obtaining the result (1407), the server that accepts requests from an unspecified number of clients due to the high load of the query processing described above. It can be easily imagined that the load increases and the processing for the analysis request from the client is delayed.

  When data of a plurality of servers is to be analyzed, a module 1202 for managing server location information is sent from the client 1201 as an extension of the client-server type data warehouse system as shown in FIG. A method may be considered in which 1207 is transferred to the servers 1205 to 1206 via the network 1204 using the server location information 1203, and the query processing result 1208 is returned to the client. For example, INTERSOLV's Virtual Data Warehouse System (hereinafter referred to as VDW) is a suitable implementation. By managing the server location by the VDW, the client itself can handle a plurality of server data without being aware of it. However, VDW, like the client-server type data warehouse system in the distributed environment, increases the server load during query processing, and it is difficult to say that VDW is a suitable implementation example of the data warehouse system in the distributed environment.

Japanese Patent Application Laid-Open No. 2004-151867 discloses a method for processing queries to a plurality of databases or data warehouses in a distributed environment. This method forwards the query to the cluster server in order to reduce the processing load on the client. In response to the query, the cluster server transfers the query to an appropriate database, integrates the results obtained from the database, and returns the result to the client. In this method, since the query is eventually transferred to the server, it is impossible to reduce the load on the server.

With respect to the problem of server load reduction and processing time reduction, for example, as shown in FIG. 13, data 1307 to 1308 of servers 1305 to 1306 are copied to the module 1309 on the client side (processing 1311 and 1312), and the copy 131
There is a method of issuing a query 1313 to 0 and obtaining a result 1314. Hereinafter, this server data copy 1310 will be referred to as a replica. By executing the query processing on the replica, the query processing in the servers 1305 to 1306 can be avoided, the server load can be reduced, and access to the server via the network can be avoided, thereby reducing the query processing time. Can be shortened.

  However, when creating a replica for a plurality of servers in a distributed environment, a simple method requires a large-scale storage device 1315 to store the replica on the client side. For example, a client who intends to use 10 servers having data of about 300 GB (gigabytes; 10 9 bytes) needs to prepare a storage device of 300 GB × 10 units = 3 TB by simple calculation. Then, it is almost impossible to prepare this large-scale storage device on the client side. In addition, when a replica is created, a large amount of data is transferred from the server to the client via the network, which places a heavy load on the network. Furthermore, if the server-side data is updated after the replica is created, the corresponding replica must also be updated, and this update processing cost is proportional to the size of the replica. Costs cannot be ignored, and this method is not a suitable implementation example of a data warehouse system in a distributed environment.

  On the other hand, as described in Non-Patent Document 4, instead of creating a replica, the query and the query processing result are cached, and the cached result is reused for a new query. Thus, a method for reducing server load and query processing time has been proposed. This method is very effective in reducing server load and query processing time when the query result reuse rate is high, but when considering a data warehouse in a distributed environment, the target data scale and client Since the difference in the scale of storage devices that can be prepared on the side is extremely large, the reuse rate of cached data becomes extremely low, and there is a disadvantage that the efficiency is poor.

  Patent Document 2 discloses an information processing apparatus and system for acquiring a server file via a network and providing the file to a user. In this method, when the system receives a file reference request from the user, a replica is created, and when a query is issued from the user in the data warehouse system, it is necessary to first request a search from the server. Therefore, it is impossible to shorten the response time of the first inquiry. Furthermore, since the replica creation unit is a file in this method, it is difficult to create a record or object unit replica that matches the query condition of the database.

Regarding the method for propagating the data update in the server to the client (corresponding to the data collection means described later in the present invention), the server updates the server data at regular time intervals such as every hour, for example. There is a server-driven PUSH method in which data is transmitted to the client every time and a client-driven PULL method in which the client goes to acquire data at a certain time interval or when necessary. However, among the PUSH methods, there is a problem that the load of the server increases in the method in which the server distributes data for each client. In the method in which each client receives only necessary data by broadcasting or multicasting the data. Since there is a problem that it becomes difficult for the client to acquire data at an appropriate timing, efficient data delivery in a distributed environment is difficult only with the PUSH method. On the other hand, in the PULL method, when the data of the server is updated, the client needs to check the server data frequently when considering the severe application where the client data needs to be updated immediately. However, if a large number of clients frequently issue processing requests, there is a problem that the load for processing these processing requests in the server becomes high. Even with the PULL method alone, efficient data delivery in a distributed environment is not possible. It turns out to be difficult. A method using a combination of the PUSH method and the PULL method is described in, for example, Non-Patent Document 5 (hereinafter, CQ). In CQ, a query including a trigger condition from a client is registered on the CQ server, and data is transferred first using a client-driven PULL method and from the next time using a server-driven PUSH method according to the trigger condition included in the query. However, in CQ, since the PUSH method and the PULL method cannot be specified for each query, the data transfer eventually becomes the server-driven PUSH method, and the problem of the PUSH method that the server load becomes high appears.

W. H. Inmon, "Building the Data Walehouse Second Edition", John Wiley & Sons, Inc. , ISBN0-471-14161-5 Robert Orfali, by Dan Harkey, “Client / Server Programming with JAVA and CORBA Second Edition”, John Wiley & Sons, Inc. , ISBN0-471-24578-X A. Sheth, J.A. Larson, "Federated Database Systems for Managing Distributed, Heterogeneous, and Autonomous Databases," ACM Computer Surveys, Vol. 22, no. 3, pp. 183-236 and A.I. Sheth, G.M. By Karabatis, “Multidatabase Interdependencies in Industry”, Proc. of 1993 ACM Sigmad, Vol. 22, pp. 483-486 A. Keller, J .; Basu, “A Predicate-based Caching Scheme for Client-Server Database Architectures”, The VLDB Journal, Vol. 5, no. 1, pp. 35-47 C. Pu, L .; Liu, “Update Monitoring: The CQ Project”, Texture Notes in Computer Science, Vol. 1368, ISSN 0302-9743, pp. 396-411 Mary Campion, Katy Wallath, “The Java® Tutorial”, Addison-Wesley, ISBN 0-201-63454-6. Jeffrey D. By Ullman, “PRINCIPLES OF DATABASE AND KNOWLEDGE-BASE SYSTEMS”, VOLUME II, COMPUTER SCIENCE PRES, ISBN 0-7167-8162-X, Chapter 14 “Optimization for Conquest” JP-A-8-286960 JP-A-9-297702

  When there are a large number of servers such as databases and data warehouses, and an unspecified number of clients access these servers via the network and try to obtain useful information by integrating and using the data on the servers, In the method of transferring queries to the server, the load on the server is increased, the dependency on the network is high, and the response time to the query is increased. In the method of creating a replica on the client side, a large amount of data is transferred. Along with this, an increase in the load on the network, an increase in the storage device capacity on the client side, and an increase in the cost of replica update processing have been problems. In addition, since the hit rate is low in the cache-based method, the cached data reuse rate is extremely low, and as a result, it is difficult to construct an efficient data warehouse system in a distributed environment. was there.

  An object of the present invention is to provide an efficient data warehouse system in a distributed environment, a query processing method used there, and a data collection method and apparatus therefor.

  More specifically, the first objective is to reduce server load, the second objective is to reduce network dependency, the third objective is to reduce response time to an inquiry, and the fourth objective. Is the reduction of the network load, the fifth objective is to reduce the storage capacity of the client, the sixth objective is to reduce the replica update processing cost, and the seventh objective is to improve the hit rate of the replica .

  In order to achieve the above first to third objects, in the present invention, a data collection means for creating and managing a replica of server data used for client inquiry processing is provided. Run with. By executing the query process in the data collection means, the query transfer to the server is reduced, and the server load as the first object can be reduced. Further, by creating a replica of the server data by the data collection means, it becomes possible to process a query using the replica in the data collection means even when the network to the server is malfunctioning. The target network dependency can be reduced. In addition, a large amount of data is accessed by many clients, and a large amount of data is managed. In addition, a query is not processed at a server with a high query processing cost (which is expensive in a billing system). The query response time, which is the third object, can be shortened by processing inquiries from a number of clients in the data collection means using replicas of only necessary data.

  To explain the query processing more specifically, whether the data collection means can process the inquiry from the client by its own data collection means, can be processed by another cooperative data collection means, or needs to be transferred to the server If the data can be processed by the own data collection unit or the other data collection unit cooperating, the inquiry is processed by the corresponding data collection unit. By using the cooperative data collection means, the number of queries that can be processed by the data collection means increases, which contributes to the achievement of the first to third objects. Further, even when it is determined that the query needs to be transferred to the server, only the portion of the query that cannot be processed by the replica that can be used by the own data collecting means and the other data collecting means to cooperate with the server By transferring to, the amount of data to be processed by the server is reduced, and not only the server load can be reduced, but also the result data returned from the server to the client is reduced, so that the network load as the fourth object can be reduced.

Further, in order to solve the above-mentioned problems that occur when a replica is simply created and achieve the fourth to sixth objects, the clients that can share data are grouped, and the replica is set to the grouped client set. And the replica is shared among the clients, and the own data collection means cooperates with other data collection means to cooperate as necessary. By sharing the replica, the amount of data transferred from the server to the data collection means can be reduced, so that the network load can be reduced. In the present invention, since a replica is created in the data collection means, it is a matter of course that the storage capacity required by the client, which is the fifth object, can be reduced, and the storage capacity in the data collection means can be achieved by sharing the replica between the data collection means. Can also be reduced. Further, since the amount of replicas as a whole system can be reduced by sharing replicas, the replica update processing cost, which is the sixth object, can also be reduced.

  When creating a replica, it accepts a request to create a replica including conditions regarding the quality of data such as accuracy, freshness, and priority given by the user from the client, and conditions regarding the range of data to be collected. By holding and negotiating with the server that supplies data, taking into account the amount of resources that can be used by the data collection means such as storage capacity and CPU performance, some or all of the replica creation request is satisfied Create a replica to use. By receiving a replica creation request from the user, it becomes possible to collect data intended by the user, and the hit rate of the replica, which is the seventh object, can be improved. Further, by adjusting the data quality when creating the replica, it is possible to create a replica of an appropriate size according to the computer resources of the data collection means, and the fourth purpose is to reduce the network load and the fifth purpose. It is possible to reduce the storage device capacity of the client and the data collection means and the replica update processing cost, which is the sixth object.

  In replica update processing, by using a combination of the client-driven PULL method and the server-driven PUSH method, server load can be reduced while considering client data requirements, and server load during replica update can be reduced. it can. Thereby, the first object can be achieved.

  Based on the data warehouse configuration method of the present invention, for example, by grouping clients, data can be shared between clients, and the storage device capacity, update processing cost, and network load on the client side can be reduced. In addition, inquiries from users, taking into account the computer resources of the data collection means to create replicas by negotiation processing with the data supply source server based on the data requirements given by the user at the time of replica creation A replica with a high utilization rate can be created. By processing a query from a client using the replica, the server load can be reduced, and a practical data warehouse system and a query processing method using the same can be realized.

FIG. 1 shows a preferred implementation of a data warehouse system according to the present invention. The clients 103 and 104 are connected to the data collection unit 1 (101) via the internal network 128.
The internal network 128 may be a local area network connected by Ethernet (R), optical fiber, or FDDI, and the client may be any computer system such as a personal computer such as Hitachi FLORA or a Hitachi 3050 creative workstation. The data collection unit 1 manages a plurality of clients in a group, receives a replica creation request and a query issued from the client, and transfers the replica creation request to the replica creation request analysis unit 106 and the query to the query analysis unit 109. When the client management unit 105 determines whether or not to actually create a replica in response to a replica creation request from the client, and creates a replica, the replica creation management unit stores the replica description that is information about the replica to be created. A replica creation request analysis unit 106 that transfers to 107, a replica creation management unit 107 that stores and manages the replica 123 in the storage device 112 with reference to the replica management table 108, a query analysis unit 109 that analyzes a query from a client, Inquiries from clients The query processing unit 127 for processing the query if that can be processed by the collection means 1, and a communication control unit 110 via the network 113 to manage a communication process with the server or other data collector. In order to perform more advanced processing, the data collection unit may include a data collection unit negotiation processing unit 111. The data collection means negotiation processing unit will be described later. The data collection means may be an arbitrary computer system like the client, and the storage device 112 may be a magnetic storage device, an optical disk device, a magnetic tape, or the like. The network 113 may be the local area network or a wide area network connecting a plurality of geographically dispersed sites.

  The server 1 (114) includes a communication control unit 115 that accepts a replica creation request and inquiry from the data collection unit, an inquiry processing unit 117 that processes the inquiry, and a delivery data management unit 118 that manages data to be delivered to the data collection unit. And a delivery data management table 120 referred to by the delivery data management unit. In order to perform more advanced processing, a server negotiation processing unit 116 and a load management unit 119 may be provided, which will be described later. The server may be an arbitrary computer system similar to the data collection means, and the storage device 121 for storing the server data 124 is an arbitrary storage device such as a magnetic storage device, an optical disk device, a magnetic tape, a CD-ROM, or a storage thereof. A combination of devices may be used. Furthermore, the server data and the replica data of the data collection means may be managed by a general-purpose database management system such as a file system or HITACHI HiRDB.

  The client management unit in the data collection unit, the replica creation request analysis unit, the replica creation management unit, the inquiry processing unit, the communication control unit, the data collection unit negotiation processing unit, and the communication on the server The control unit, the inquiry processing unit, the server negotiation processing unit, the load management unit, and the delivery data management unit are not only configured as dedicated hardware, but also each of the storage devices 112 in the data collection unit, and The program may be a program stored locally in the storage device 121 in the server, or a program downloaded from a server storing a program on a network. As a mechanism for safely executing a program downloaded from a server in a heterogeneous distributed environment, for example, a programming language (hereinafter referred to as an Internet programming language) described in Section 4 of Non-Patent Document 6 can be used.

  In particular, as a preferred embodiment of the client, there is a form in which a web browser such as Microsoft Internet Explorer or Netscape Netscape Navigator is used on an arbitrary computer system such as a Hitachi FLORA personal computer or a Hitachi 3050 creative workstation. In this case, the client program can be dynamically changed by downloading a program module created using the Internet programming language to the browser.

  In this embodiment, the client is directly connected to the data collecting means 1 to n via the internal network. However, as shown in FIG. 18, the client is connected to the data collecting means via the LAN or the Internet. Needless to say, the present invention is effective even in the case of being connected to the network.

More specific features of the present invention are as follows: (1) A partial replica that can be shared by clients grouped in the data collection means is created. (2) To create a replica, a replica creation request from a user is accepted, data for actually creating a replica is determined by negotiation processing with a server, and a replica description describing the contents is created and managed. (3) The replica creation request includes not only conditions related to the range of data (data area conditions) but also conditions related to data quality (data quality conditions). (4) The replica update uses a data transfer method in which a server-driven PUSH method and a client-driven PULL method are combined using a delivery data management table held on the server side. (5) A query that can be processed using a replica in its own data collection means is processed using a replica of its own data collection means, and conversely, an impossible query is processed by another data collection means that can be processed, or It can be summarized into 5 points: transfer to the server and request an inquiry. Each will be described below using examples.
(1) In the prior art, it has been stated that it is difficult to realize a data warehouse system in a distributed environment with a simple client-server configuration as shown in FIG. Therefore, in the present invention, a data collecting means for collectively managing clients that can share data is provided. By using this data collection means, it is possible to suppress the creation of redundant replicas that overlap on the client side, and to prevent useless access to the server, thereby reducing the load on the server. For example, in FIG. 15, client 1 (1501) needs data (1503) of a product sales table with sales of 10,000 yen or more, and client 2 (1502) has data (1504) of the product sales table with sales of 5000 yen or more. ) Is required. If the product sales data 1509 is stored in the server 1505, the clients 1 and 2 need to acquire data from the server. In other words, the server is subjected to a load caused by inquiries from both the clients 1 and 2. However, the data collection means 1506 of the present invention is installed, the data collection means holds the replica creation request of the clients 1 and 2, and is the union of the conditions, “data with product sales of 5000 yen or more” (1507) ) As a replica description and the data is acquired from the server, and the replica 1508 is created, so that the query of the clients 1 and 2 can be processed using the data of the data collection means. Access can be omitted, and server load and network load due to data transfer can be reduced. Further, the union of replica creation requests requested by the clients 1 and 2 due to the capacity limitation of the storage device 1610 of the data collection means, the CPU processing capacity restriction of the data collection means 1606, or the restriction on the amount of data transferred over the network. 16 cannot be held, as shown in FIG. 16, from the respective replica creation requests (1603, 1604) of the client 1 (1601) and the client 2 (1602), “product sales is 10,000” If the processing request from the client 1 satisfies the replica creation request if the data is acquired from the server as a replica description and the data is obtained from the server as a replica description (1607), all the processing requests are processed using the replica. Can be processed, and client 2 cannot be processed using a replica. Processing request that is only queries that reference to "data of the product sales less than 10,000 yen or more 5000 yen" may be transferred to the server, it is possible to reduce the server load and network load. It is clear that the effects of server load reduction and network load reduction described with reference to FIGS. 15 and 16 increase as the number of clients increases.
(2) Receipt of a replica creation request and a replica creation method will be described with reference to FIG. 1, FIG. 2, and FIG. When the data collection unit 101 receives a replica creation request from the clients 103 to 104 (the replica creation request will be described in (3)) (process 202), the replica creation request analysis unit 106 analyzes the request. (Process 203) By communicating with the replica creation management unit 107, the contents are referred to the replica management table 108 shown in FIG. 8 (Process 204). The replica management table includes entries for a data area condition 801 relating to an area of data to be created and a data quality condition 802 relating to data quality. These two entries are called replica description 803. For one replica description entry, replica location information 804 that describes the name of the data collection means storing the replica, server location information 805 that describes the name of the server that is the data source of the replica, and the replica The data delivery condition 806 is a maintenance condition of For example, the first entry 807 in FIG. 8 is a replica of data obtained by extracting the three columns of order number, price, and customer number for records with a price of 10,000 yen or more among records stored in the order table. Is the data collection means 1, and the original data of the replica is the server 1, and the server 1 delivers data at 13:00 by the PUSH method, and represents that the replica is maintained. However, in the data delivery method in the second entry of FIG. 8, the description {1: 0, 13:00} and PUSH is data delivered by the PUSH method at both 1:00 and 13:00. Represents that.
When the requested replica can be created from an existing replica in the own data collection means (when Yes is selected in the determination process 205), the replica creation process is terminated without creating a new replica. For example, if the replica management table is as shown in FIG. 8 and the new replica creation request is the first entry 1901 in FIG. 19, the replica creation request is the data collection shown in 807 in FIG. Processing can be performed by using an existing replica in the means 1, and the determination processing 205 is Yes.

  When the requested replica cannot be created from the existing replica 123 in its own data collection means (when No is selected in the determination process 205), the existing replica of the data collection means 102 with which the requested replica cooperates It is determined whether or not it can be created from 126 (determination process 208). If creation is possible (when Yes is selected in the determination process 208), it is determined whether or not a replica is further created in the own data collection means (determination process 216). When replicas are not created redundantly in the own data collection means (when No is selected in the determination process 216), the replica creation process is terminated without creating a replica. When a duplicate replica is created in the own data collection means (when Yes is selected in the determination process 216), a replica is created from the replica of the cooperative data collection means 102 under the condition requested by the client ( Process 215), and the replica creation process ends (219).

  For example, if the replica management table is as shown in FIG. 8 and the new replica creation request is the second entry 1902 in FIG. 19, the new replica creation request in the existing replica in the data collection means 1 However, if a replica in the data collection means 2 shown at 808 in FIG. 8 is used, the new replica creation request can be processed. In such a case, if there is room for creating the replica in the storage device of the data collection unit 1, a new replica is created in the data collection unit 1 from the replica in the data collection unit 2. Further, when the priority of the replica creation request from the client to the data collection unit 1 is high, the replica having the low priority in the data collection unit 1 is deleted and a replica according to the replica creation request is created.

  Whether or not a replica based on a new replica creation request can be created from an existing replica is determined by comparing a replica management table describing the contents of the existing replica with a newly given replica creation request. That is, the data collection means holds the replica creation request, and when a new replica creation request arrives for the replica description generated from the set of replica creation requests and the combination of the replica creation requests, The data collection unit holds the replica creation request, compares the replica creation request with an entry in the replica management table, and determines an equivalence relation and an inclusion relation. For example, in the previous example, when a new replica creation request shown in FIG. 19 is given to each entry in the replica management table shown in FIG. 8, the new replica creation request represented by entry 1902 It is obvious that it is included in the existing replica represented by 808. In this embodiment, it is assumed that the replica creation request is very simple. However, in a more general case, for example, Non-Patent Document 7 uses a method called “Query Equivalence, Query Containment”. Discloses a method for examining equivalence relations and inclusion relations between conditions. By using this determination method, replica management using the replica management table according to the present invention can be realized even in a more general case.

  When the replica requested by the client cannot be created from the existing replica in the own data collection means or from the existing replica in the cooperative data collection means (when No is selected in the determination process 208), Negotiation processing is performed (processing 211). As a result of the negotiation process, if there is no replica creation condition that can be accepted by both the data collection means and the server (when No is selected in the determination process 212), the replica creation process is performed without performing the replica creation. The process ends (219). If there is a replica creation condition that both parties can accept as a result of the negotiation process (when Yes is selected in the determination process 212), a replica is created according to the replica creation condition that both parties have accepted (process 215). The process ends (219).

  The negotiation process in step 211 will be described with reference to FIGS. First, when a replica creation request is transferred from the data collection means 101 to the server 114 (process 302), if the server accepts the replica creation request (when Yes is selected in the determination process 303), a replica creation condition ( In the accounting system, the charge) is set as a replica creation condition requested by the data collection means (process 310), and based on the created condition, the replica creation management unit 107 of the data collection means manages the replica management table 108 and server delivery data management. The unit 118 updates the delivery data management table 120 and ends the negotiation process (311). When the server cannot accept the replica creation request transferred from the data collection unit (when No is selected in the determination process 303), there is no new condition that can be presented from the server to the data collection unit (determination process) If No is selected in 312), the negotiation process is terminated without setting information relating to the replica creation request (311). When the server cannot accept the replica creation request transferred from the data collection unit (when No is selected in the determination process 303), there is a new condition that can be presented from the server to the data collection unit (determination process) If Yes is selected in 312), the server transfers the new condition to the data collection means (process 306). When the data collection unit accepts the condition presented by the server (when Yes is selected in the determination process 307), the new condition is set as the replica creation condition (process 310), the replica management table and server on the data collection unit side The delivery data management table on the side is updated, and the negotiation process is terminated (311). If the data collection unit cannot accept the conditions presented by the server (when No is selected in the determination process 307), the negotiation process is terminated without setting information regarding the replica creation request.

  A specific example of the negotiation process will be described with reference to FIGS. It is assumed that the replica creation request from the clients 103 to 104 has a condition as shown in FIG. First, the data collection means 1 (101) transfers the replica creation request 1001 to the server 1 (114) via the network 113. The server 1 predicts the load of the local system when the replica creation request is received from the current load of the local system managed by the load management unit 119, and creates the replica when the predicted value is equal to or less than a threshold value. The request is accepted, and as shown in FIG. 10B, a response (1003) of acceptance is returned to the data collection means, and the replica creation request becomes a replica creation condition, and the negotiation process ends. In contrast, when the replica creation request 1002 is transferred to the server m (122), the server m cannot accept a replica creation request under the conditions indicated in the creation request 1002 as a result of the load prediction. If the new condition as shown in the request 1004 can be accepted, the server m returns a response that the condition is accepted to the data collecting means 1 and the new condition generated by the server. The data collection unit 1 returns the new condition to the client 103 or 104 that issued the replica creation request, and when the client can accept the new condition, the replica management table on the data collection unit side uses the new condition as the replica creation condition. And the delivery data management table on the server side is updated, and the negotiation process is terminated.

The following embodiments can be considered for the load measurement of the own system and the load prediction when the replica creation request is received. In general, a multitasking operating system has a queue of executable processes called “run queue”, and the time average of the average queue length of this queue is called a load average. Therefore, for example, the server considers the load average L of the own system as the load of the own system, estimates the load of the own system when the new replica creation request is accepted as L + 1, and the predicted load is less than a certain threshold L _MAX ( In other words, the replica creation request is accepted at L + 1 ≦ L _MAX . In the case of a server that cannot measure the load average, the average number of jobs N per fixed time is regarded as the load of the own system, and the load of the own system when the new replica creation request is accepted is estimated as N + 1. In the case where the predicted value is equal to or less than a certain threshold value N _MAX (that is, N + 1 ≦ N _MAX ), an embodiment may be considered in which the replica creation request is accepted.

  (3) First, a replica creation request will be described. For example, in the embodiment shown in FIG. 15, the clients 1 and 2 collect sales data of products and perform analysis. In data mining, data to be analyzed is often based on some restrictions, for example, a certain client sets sales data for only Tokyo as an analysis target. This restriction is based on the intention of a person who wants to perform an analysis using a client, and it is difficult and wasteful for a computer to capture this intention automatically. Therefore, in the present invention, an interface is provided on the client so that a user can give a replica creation request regarding data to be accessed to the system. The implementation of the interface is a program stored locally on the clients 1501 and 1502 in FIG. 15, or a program downloaded from a server storing a program on the network, as in the case of the data collection means and server. It may be. In accordance with a replica creation request issued by a user using the interface of the client, a replica 1508 that can be shared by the client is created in the data collection unit 1506.

  As shown in FIG. 6, the replica creation request includes a data area condition (601) indicating a data range, a data quality condition (603) indicating data quality, and a data delivery condition (604) indicating a data delivery method. ). The data quality condition includes, for example, data that has been updated on the server, but the replica assumes that the data before the update is the latest data within one hour, the freshness condition 605, and 10% of the order number in the order table. It includes a precision condition 606 for replicating sample data, and a priority condition 607 for targeting the top 10 data in the order table price. However, in consideration of compatibility with a conventional query language, the replica creation request also allows a notation that does not include a condition regarding data quality.

  If the client requests the latest data when the replica creation request does not include the freshness condition, the server data must be checked. If the replica creation request does not include the accuracy condition, the accuracy is 100% is assumed, and when priority conditions are not included, data for all ranks is obtained. As for accuracy conditions, as shown in FIG. 7, according to the type of data, sampling for relational database records, field segmentation, summary creation for documents, keyword segmentation, lossless for images, lossy compression, contour extraction, number of colors Various application methods such as reduction, resolution reduction, size reduction, reduction of the number of frames for moving images, compression of images within a frame, sound quality adjustment for sound, conversion to character data, and the like can be considered.

  By including the data quality condition in the replica creation request, the replica size can be reduced. Regarding the adjustment of accuracy, since a method disclosed in Japanese Patent Application Laid-Open No. 09-025863 “Aggregation Result Estimation Method in Database Processing System” can be used to estimate a result with high accuracy using small-scale sampling data, data It turns out that it is very effective for building a warehouse system. In addition, by using the freshness condition, it is possible to flexibly handle the update problem that is frequently discussed in the distributed system.

For example, as shown in FIG. 17, when the restriction on freshness is specified as one day, the last update time of the replica is 6:00 on October 23, 1997, and the current time is October of the same year. It is assumed that it is 22:00 on the 24th. The actual data on the server may be updated during this period. However, if the restriction on freshness is given as one day, the current time data D _A (T _C ) is changed to the data at the previous update time. By considering D _A (T _P ), it is possible to prevent data transfer through the network every time the server data is updated, and to reduce the network load. Of course, the freshness condition may not be specified for a usage mode in which the update needs to be reflected immediately when the server data is updated.

  Furthermore, when a priority constraint 607 as shown in FIG. 6 is applied, the data transfer amount can be reduced. This restriction 607 specifies that the top 10 items in the price of the order table are targeted for replicating, and the amount of transfer data is much smaller than when a replica of the entire order table is created.

  (4) Data delivery conditions at the time of replica update will be described with reference to FIGS. When the data 124 managed by the server 1 (114) is updated (process 502), the delivery data management unit 118 of the server refers to the delivery data management table 120 (process 503), and the updated data is the delivery data. If it is not registered in the management table (if No is selected in the determination process 504), the update process is terminated without delivering data (506). When the updated data is registered in the delivery data management table (when Yes is selected in the determination process 504), the data corresponding to the data delivery destination is delivered according to the data delivery condition of the delivery management table ( Process 505), and the update process ends.

  A specific example of the replica update process will be described with reference to FIGS. The server 1 (114) stores the order table data 124 in the storage device 121. It is assumed that new order data 907 is inserted into the order table. The delivery data management unit 118 checks the update data against the delivery data management table, and since the update data satisfies the entries 905 and 906 of the delivery data management table, the update data is collected according to the entry 905. The data is delivered to the data collecting means 2 at 1:00 and 13:00 according to the entry 906 in the server-driven PUSH method at 13:00 to the means 1 and according to the entry 906. At the time of data delivery at 13:00, the server 1 can perform data delivery for the data collection unit 1 and the data collection unit 2 collectively by performing multicasting, so that server load and network load are reduced. Can be reduced. Further, if the data delivery condition of entry 906 includes the data delivery condition of entry 905 such as once between 11:00 and 15:00, or once a day, entry 906 By matching the data transfer to the data transfer condition of the entry 905, the number of data transfers can be reduced.

  (5) Query processing using a replica will be described with reference to FIGS. When a query is issued from the client 103 to the data collection unit 101, the query is transferred to the query analysis unit 109 via the client management unit 105. The query analysis unit first communicates with the replica creation management unit 107 to refer to the replica management table (process 403). The query analysis unit determines whether or not the query can be processed in the own data collection unit from the replica management table, and if the query can be processed in the own data collection unit (Yes is selected in the determination process 404) In the case, the inquiry processing is performed in the own data collection means (processing 407), the solution is transferred to the client (processing 415), and the inquiry processing is terminated (416). When processing within the own data collection means is impossible (when No is selected in the determination process 404), it is determined whether or not the query can be processed by another cooperative data collection means, and processing is possible (When Yes is selected in the determination process 408), the inquiry is transferred to the other data collecting means to cooperate (process 411), the solution is received (process 412), and the solution is transferred to the client (415). The process ends. If the query cannot be processed either in the own data collection means or in the other data collection means that cooperates (when No is selected in the determination process 409), the inquiry is transferred to the server (process 413) and the solution is received (process 414). ) To the client (process 415), and the inquiry process is terminated (416).

  Whether or not a query given from a client can be processed using a replica is determined by comparing the given query with a replica management table describing the contents of an existing replica. That is, an entry in the replica management table (hereinafter referred to as an entry) is compared with the query, and if the query is equivalent to the entry or the query is included in the entry, the query is It can be processed. This determination is equivalent to whether or not the new replica creation request can be processed using an existing replica. As described in the processing of the new replica creation request, for example, non-patent document 7 By using the method called “Query Equivalence, Query Containment” disclosed in the above, it is possible to determine whether the query can be processed using the replica, so that the replica of the own data collection system according to the present invention cooperates. Query processing using another data collection system can be realized.

  A specific example of the inquiry process using the replica will be described with reference to FIGS. Assume that the client 103 issues an inquiry “find the order number of an order whose price is 20000 yen or more and the price of the order”. Upon receiving the inquiry from the client, the client management unit 105 of the data collection unit 1 transfers the inquiry to the inquiry analysis unit 101. The query analysis unit 101 refers to the replica management table 108, determines that the given query can be processed by the replica in the own data collection unit 1 corresponding to the entry 807, and performs query processing using the replica. . As a result, the inquiry process is not transferred to the server. Next, consider a case where the client 103 issues an inquiry “find the order number of an order whose price is 3000 yen or less”. In this case, by referring to the replica management table, it is determined that the query can be processed by the replica in the data collection unit 2 corresponding to the entry 808, and the query is transferred to the data collection unit 2. Finally, let us consider a case where an inquiry “A customer number for an order with a price of 7500 yen is requested” is issued. In this case, there is no entry for processing the query in the replica management table of FIG. 8, it is determined that the query cannot be processed using the replica, and the query is transferred to the server.

In the above description, it is assumed that the data collection means side can obtain a server name that can acquire necessary data by using the distributed network infrastructure technology such as CORBA. Needless to say, the present invention can be implemented by managing information about the server on the data collection means side.

The block block diagram of the 1st Example of the data warehouse system by this invention. The flowchart which shows a replica creation process procedure. The flowchart which shows a negotiation process procedure. The flowchart which shows an inquiry process procedure. The flowchart which shows an update process procedure. The figure which shows the replica creation request given from the client in the Example of this invention. The figure which shows the precision adjustment method according to the data type in the Example of this invention. The figure which shows the replica management table in the Example of this invention. The figure which shows the delivery data information in the Example of this invention, and update data. The figure which shows the information utilized at the time of the negotiation process in the Example of this invention. The figure which shows the utilization method of a data warehouse in a prior art. The figure which shows the structure method of the data warehouse in a distributed environment in a prior art. The figure which shows the structure method of the data warehouse in a distributed environment in a prior art. The figure which shows the many-to-many configuration method of the client and server in a distributed environment in a prior art. The figure which shows an example of the grouping method of the client in the data collection means by this invention. The figure which shows the other example of the grouping method of the client in the data collection means by this invention. The figure which shows the utilization method of the freshness conditions in the Example of this invention. The block block diagram of the 2nd Example of the data warehouse system by this invention. The figure which shows one example of the replica creation request | requirement in this invention.

Explanation of symbols

103, 104, 1101, 1201, 1301, 1401, 1402, 1501, 1502, 1601, 1602, 1806, 1807 ... client terminal, 112, 121, 1105, 1209, 1210, 1315, 1316, 1317, 1408, 1409, 1510 , 1511, 1610 ... storage device, 113, 1204, 1304, 1405, 1802 ... network, 114, 122, 1102, 1205, 1206, 1305, 1306, 1403, 1404, 1505, 1605, 1803, 1805 ... server, 101, 102, 1506, 1606, 1801, 1804 ... data collection means, 1202 ... server location information management module, 1203 ... server location information, 123, 126, 1310, 1508, 1608 ... replication , Data managed by the 124,125,1307,1308,1509,1609 ... server, 108 ... replica management table, 120 ... delivery data management table, 1507,1607 ... replica description.

Claims (3)

A client computer system that accepts processing requests from users;
A server computer system for searching the database according to the access request from said client computer system comprising a database, each,
A data collection unit provided between the client computer system and the server computer system and provided with a storage device;
The data collection means includes
A client management unit that manages the client computer system,
A query analyzing unit that analyzes an access request from said client computer system,
A communication control unit that determines an access procedure to the server computer system based on an analysis result in the query analysis unit;
Said receiving a replica creation request containing the requirement from the client computer system on the quality of the data, the replica creation request analysis unit for determining a generation condition of replicas by analyzing the replica creation request created based on the requirements,
Create a replica that creates a partial replica of the database in the storage device in accordance with the creation condition, and manages the replica using a replica management table that associates information about the creation condition and storage location of the created replica A management department,
The query analysis unit
In response to an access request from the client computer system, a determination processing unit that determines whether query processing using the replica is possible;
If the inquiry processing using the replica by the determination processing section is judged to be a query processor to query processing using the replica without transferring the inquiry to the server computing system,
A transfer processing unit that transfers the access request to the server computer system when it is determined by the determination processing unit that inquiry processing using the replica cannot be performed ,
Data warehouse system. The server computer system is
A communication control unit that receives the access request transferred from the data collection unit through the network;
The data warehouse system according to claim 1, further comprising: a query processing unit that searches the database included in the server computer system and creates a response to the received access request . The replica creation request analysis unit has a condition in which the request condition specifies data freshness, and the difference between the update time and the current time when the data is updated in the server computer system is smaller than a predetermined value. 3. The data warehouse system according to claim 2, further comprising a processing unit that handles data at the current time as data in the data collection means.

Images