KR101218712B1 - System and Method for data distributing - Google Patents

Abstract

The present invention relates to a data distribution system and a method for distributing data, by which a large amount of data can be distributed and stored in real time based on the performance of the data storage device, thereby efficiently constructing a system using a low capacity data storage device.
A data distribution system and a data distribution method according to the present invention include a plurality of data storage means for performing SQL and storing data, metadata about data stored in each data storage means, and priority information for each data storage means. Metadata storage means for storing the metadata including the; and calculating the priority based on the performance of the data storage means and storing the metadata in the metadata storage means, and analyzes the stored data applied from the terminal to distribute the mass In the case of data, the data storage means includes a data processing means for distributing and storing the data in a plurality of data storage means in a round-robin manner according to the priority of the data storage means stored in the metadata storage means.

Description

Data distribution system and its data distributing method {System and Method for data distributing}

The present invention relates to a data distribution system and a method for distributing data, by which a large amount of data can be distributed and stored in real time based on the performance of the data storage device, thereby efficiently constructing a system using a low capacity data storage device.

In general, a data distribution system includes a plurality of databases, that is, storage means. At this time, the storage means is operated through the management of the metadata consisting of the index information of the data stored in the storage means.

Recently, with the rapid development of the Internet, the type and amount of data stored in the data distribution system are also increasing.

Accordingly, the storage means constituting the data distribution system is also used to replace the storage means with a large capacity.

In particular, in a data distribution system, data distribution systems are generally focused on easily storing and distributing heterogeneous data stored therein, and rapidly generating large amounts of data such as network traffic data in real time. High performance and large storage means are required for distributed processing.

In other words, in order to expedite the processing of large amounts of data for the provision of Internet services, telecommunications operators should construct a new data distribution system by discarding previous low-capacity data storage means and replacing them with high-performance / high-capacity storage means that can accommodate a certain amount of capacity. do.

However, the data distribution system is mainly used for the purpose of distributed storage of data, and there is a cost loss in constructing the data distribution system by replacing it with a mass storage device to secure data storage capacity.

Accordingly, in the present invention, the priority is set based on the performance of the data storage means, and the data input in the round-robin manner is distributed and stored according to the priority, thereby easily using the low capacity storage means for the large data. It is a technical object to provide a data distribution system and a method for distributing data that allow distributed storage to be easily stored.

A data distribution system according to the first aspect of the present invention for achieving the above object comprises a plurality of data storage means for performing SQL and storing data, metadata for the data stored in each of the data storage means and each data storage means. Metadata storage means for storing metadata including priority information on the data, and storing the data in the metadata storage means and calculating the priority based on the performance of the data storage means. In the case of a large amount of distributed data by analyzing the data storage means in accordance with the priority of the data storage means stored in the metadata storage means comprises a data processing means for distributing and storing the data in a plurality of data storage means in a round-robin manner It is done.

In addition, the metadata for the data storage means stored in the metadata storage means is characterized in that it includes the current load information.

The priority information of the data storage means may be calculated based on the data processing speed, the storage capacity, and the current load of the data storage means.

In addition, the priority information of the data storage means is set an initial priority value according to the performance of each data storage means, the sum of the initial priority value of all the data storage means is set to 100, initially the initial priority value And set the current priority value to be the same, and reduce the current priority value of the data storage means by the first reference value set by the operator during data storage, and execute SQL in the data storage means having the smallest current priority value, And calculating a second reference value set by the operator to a current priority value of the data storage means on which SQL is performed.

In addition, the data processing means analyzes the stored data applied from the terminal, and if it is determined that the input stored data is not distributed data, it is characterized in that to store the stored data by performing SQL statements for all data storage means. .

In addition, the data processing means analyzes the read data applied from the terminal to read the distributed and stored data by performing SQL in all data storage means in the case of distributed data, and combines the read data and provides them to the corresponding terminal. When the read data applied from the terminal is not distributed data, the read data is performed by performing SQL through the data storage means having the least load amount, and then providing the read data to the corresponding terminal.

In addition, the data processing means includes a distributed data management block that provides various application programming interfaces (APIs) related to insertion / deletion / search query processing for distributed processing-related data and distributed table information by analyzing SQL statements to be processed. SQL Parser to check the query, and query router to determine the data storage means for processing the corresponding SQL statement based on the parsed SQL syntax information and metadata about each data storage means stored in the metadata storage means. And a distributed query processor for determining data storage means for storing or reading data in a round-robin manner based on priority information on the data storage means provided from the query router. Distributed data processing block configured to include the current status information including the current load of each data storage means. The distributed management unit calculates a priority value for distributed data storage based on a load balancer, data processing speed and storage capacity of each data storage means, and current load information provided from the load balancer. A storage means management block configured to include and storing metadata including priority information and current load information of the data storage means provided from the storage means management block, and the data stored in the data storage means. It is characterized in that it comprises a metadata processing block for storing the metadata.

In addition, the data distribution method according to the second aspect of the present invention for achieving the above object is a data distribution system comprising a plurality of data storage means and metadata storage means combined with the data processing means, wherein each of the data storage means A first step of calculating a priority value for each data storage means based on the performance of the data storage means and storing the priority value in the metadata storage means, and dividing the corresponding stored data into a plurality of pieces when a large amount of stored data is input from the terminal; And distributing and storing the divided data in a round-robin manner based on the priority value of the data storing means stored in the metadata storing means.

In addition, the first step is characterized in that the priority value is calculated based on the data processing speed, the storage capacity and the current load of the data storage means.

In addition, the first step is an eleventh step of setting the initial priority value based on the performance of each data storage means, the sum of the initial priority value of all the data storage means is set to 100, and initially the initial priority A twelfth step of setting the value equal to the current priority value; a thirteenth step of decreasing the current priority value of the corresponding data storage means by the first reference value set by the operator when storing data; and the smallest current priority value. Performing SQL in the data storage means, the priority value is calculated through the fourteenth step of adding the second reference value set by the operator to the current priority value of the data storage means performed SQL.

The first step may further include storing the current load of the data storing means in the metadata storing means.

The second step may further include a twenty-first step of storing SQL data on all data storage means when the stored data input from the terminal is not a large amount of data. .

In addition, if there is a read request for a large amount of distributed data from the terminal further comprises a fourth step of performing SQL in all data storage means to read the distributed and stored data, and combine the read data to provide to the terminal; It is characterized in that the configuration.

According to the present invention, the data processing means performs main processing for data distribution, and the data storage means performs only SQL execution and storage processing, and also stores and stores a large amount of divided data in consideration of the performance of the data storage means. By doing so, it is possible to easily process a large amount of distributed data using a low capacity data storage means.

1 is a view showing a schematic configuration of a data distribution system according to the present invention.
FIG. 2 is a view showing functionally separated internal structure of the data processing means 100 shown in FIG.
3 is a diagram showing functionally separated internal structure of the distributed data processing block 120 shown in FIG.
4 is a view showing functionally separated internal structure of the storage means management block 130 shown in FIG.
5 is a flowchart for explaining a data storage operation of the data distribution system shown in FIG.
FIG. 6 is a flowchart for explaining a data reading operation of the data distribution system shown in FIG.

Hereinafter, embodiments according to the present invention will be described with reference to the drawings. However, the embodiments described below are illustrative of one preferred embodiment of the present invention, and examples of such embodiments are not intended to limit the scope of the present invention. The present invention can be variously modified without departing from the technical idea thereof.

1 is a view showing a schematic configuration of a data distribution system according to the present invention.

As shown in FIG. 1, the data distribution system according to the present invention includes data processing means 100, a plurality of data storage means 200, metadata about data stored in each data storage means 200, and each data storage means. Metadata storage means 300, which stores metadata including priority information and current load information about 200, is stored in communication with each other via LAN 1.

In this case, the plurality of terminals 400 are coupled to the LAN 1. Here, the terminal 400 is to be connected to the data storage means 200, such as a user computer to be a communication terminal capable of various forms of Internet connection for storing data or reading data stored in the data storage means 200. Can be. For example, the terminal 400 may be a communication terminal providing a variety of data by wirelessly coupled to a mobile communication terminal capable of internet access.

In addition, the data processing means 100 inputs data distribution processing related information, such as priority condition information, addition and deletion of the data storage means 200, and also an operator for outputting data distribution processing related information. It is configured with a terminal (500).

In addition, the data processing means 100 analyzes the metadata of the data provided from the terminal 400 and selects and stores a suitable data storage means 200 among the plurality of data storage means 200. In this case, the data processing means 100 divides a large amount of data, such as network traffic data, into a plurality of data and stores the data in a plurality of data storage means 200. In this case, the data processing means 100 selects the data storage means 200 to store the divided data, and based on the state information including the data processing speed, the storage capacity, and the current load of the data storage means 200. Priority is set for the storage means 200, and the data are distributed and stored in a round-robin manner according to the priority.

In addition, the data processing means 100 may calculate the priority value for the data storage means 200, the following method may be applied.

1. All data storage means 200 has an initial priority and a current priority value.

2. The sum of the initial priority values of all the data storage means 200 is set to 100.

3. Initially, the initial priority value and the current priority value are set equal.

4. During data storage, the current priority value of the data storage means 200 is reduced by a first reference value set by the operator, for example, "100 / number of data storage means 200".

5. The SQL is executed in the data storage means 200 having the smallest current priority value.

6. The second reference value set by the operator is added to the current priority value of the data storage means 200 in which SQL is performed.

Here, the initial priority value may be set by the operator, which may be set according to the performance including the data processing speed and the storage capacity of the data storage means 200.

In addition, the data processing means 100 analyzes the metadata required to be read from the terminal 400, reads the corresponding data from the data storage means 200, and provides the same to the terminal 400. At this time, the data processing means 100 is provided to the terminal 400 by combining the divided data in the form of the original large-capacity data for the divided and stored divided data.

2 is a block diagram showing functional separation of the internal configuration of the data processing means 100 shown in FIG.

As shown in FIG. 2, the data processing means 100 includes a distributed data management block 110 that provides various application programming interfaces (APIs) related to insertion / deletion / search query processing for distributed processing-related data, and data storage means. Distributed data processing block 120 for performing data distributed storage and read processing by processing SQL (Structured Query Language) statements through selected data storage means 200 based on metadata information of 200, and storing each data In addition to managing the state of the means 200 and to perform overall management of the metadata storage means 300, such as the storage means management block 130 for calculating the priority value, the metadata storage means 300, etc. It comprises a metadata processing block 140, and an interface block 150 for communicating with the data storage means 200 and the metadata storage means 300.

Here, the distributed data processing block 120 interprets the SQL statement to be processed as shown in FIG. 3 to determine whether the distributed table information is included (SQL Parser 121), the parsed SQL syntax information and A query router (QR) 122 for determining the data storage means 200 to be processed the SQL statement based on the metadata for each data storage means 200 stored in the metadata storage means 300, Distributed to determine the data storage means 200 for storing or modifying / retrieving data, ie reading data, in a round-robin manner based on the priority information on the data storage means 200 provided from the query router 122. It is configured to include a distributed query processor (DQP, 123). Here, the query router 122 may change and set priority condition information for the data storage means 200 according to a request from the operator terminal 500. In addition, the initial priority value for each data storage means 200 may be arbitrarily set by the operator.

In addition, the storage means management block 130 is a load balancer (131) for confirming the current status information including the current load amount for each data storage means 200, as shown in FIG. Distributed to calculate the priority value of the data storage means 200 for data distributed storage based on the data processing speed and storage capacity for each data storage means 200 and the current load information provided from the load balancer 131 It is configured to include a management unit 132. At this time, the distributed management unit 132 stores the current load information and the priority information for each data storage means 200 in the metadata storage means 300 through the metadata processing block 140. At this time, the distribution management unit 132, when it is determined that the current load amount of the data storage means 200 is more than the preset amount, the corresponding data storage means 200 stored in the metadata storage means 300 through the metadata processing block 140 It is also possible to request that the current load quantity information of the " In addition, the distributed management unit 132 is stored in the metadata storage means 300 through the metadata processing block 140 when the data processing speed for the data storage means 200 and the load amount more than a certain ratio compared to the storage capacity is confirmed. It is also possible to request to update the current load information of the data storage means 200. In addition, the distributed management unit 132 performs a function of additional registration and deletion of the data storage means 200 by using a hash-partition technique.

In addition, the metadata processing block 140 updates the metadata storing means 300 based on the priority information and the current load information on the data storing means 200 provided from the storing means managing block 130. In addition to storing the metadata for the stored data provided from the distributed data processing block 120. In this case, the metadata for the large amount of data such as the network traffic data includes distribution table information for the split data.

In addition, the interface block 150 is a platform for communicating with various types of databases, such as ORACLE, MASQL, MYSQL, that is, the data storage means 200.

Next, the operation of the data distribution system having the above configuration will be described with reference to the flowcharts shown in Figs. 5 is a diagram showing a data storage operation of the data distribution system, and FIG. 6 is a diagram showing a data reading operation of the data distribution system.

1. Data storage 5 )

First, performance information and priority condition information including data processing speed and data storage capacity for each data storage means 200 are registered in the storage means management block 130 of the data processing means 100.

In addition, the storage means management block 130 of the data processing means 100 checks the current load amount of each data storage means 200 in real time as well as to each data storage means 200 based on priority condition information. The priority value is calculated and registered in the metadata storing means 300 through the metadata processing block 140.

As described above, when arbitrary data is applied from the terminal 400 to the data processing means 100 in a state in which the priority information for each data storage means 200 varies according to the current load amount (ST1), data processing The means 100 analyzes the SQL statement of the input data and checks whether it is stored data (ST2). At this time, the analysis of the SQL statement is performed by the SQL parser 121 of the distributed data processing block 120.

If it is determined that the data input in step ST2 is stored data, the data processing means 100 checks whether the corresponding stored data is a large amount of distributed data by analyzing the SQL statement (ST3). At this time, the SQL statement is interpreted by the SQL parser 121 of the distributed data processing block 120.

When it is determined that the stored data input in step ST3 is distributed data, the data processing means 100 uses SQL through the data storage means 200 selected by the priority-based round-robin method of each data storage means 200. The statement is executed and the stored data are distributed and stored (ST4). At this time, the data division of the stored data may be equally divided into predetermined size units or may be automatically or manually adjusted according to the load of the data storage means 200. In addition, the data processing means 100 stores the divided data in a round-robin manner based on the priority information for each data storage means 200 registered in the metadata storage means 300. In this case, the distributed information on the divided data is stored in the metadata storage means 300. Further, the selection of the data storage means 200 based on the priority information is performed in the query router 122 of the distributed data processing block 120, and the process of storing the data divided into the respective data storage means 200. Is done in distributed query processor 123.

On the other hand, when it is determined that the stored data input in step ST3 is not distributed data, the data processing means 100 performs SQL statements for all data storage means 200 to store the data in all data storage means 200. Save the data (ST5). At this time, the setting conditions of the data storage means 200 of the data other than the distributed data may be arbitrarily set by the operator.

2. Read data ( Figure 6 )

First, as described above, in a state in which divided data for a large amount of data is distributed and stored in a plurality of data storage means 200 in a priority-based round-robin manner, arbitrary data from the terminal 400 is transferred to the data processing means 100. If it is applied (ST11), the data processing means 100 analyzes the SQL statement of the input data and checks whether it is read data (ST12). In this case, the read data may be, for example, search data, and the analysis of the SQL statement is performed by the SQL parser 121 of the distributed data processing block 120.

If it is determined that the data input in step ST12 is read data, the data processing means 100 checks whether a distribution table exists in the read data by analyzing the SQL statement (ST13). At this time, the SQL statement is interpreted by the SQL parser 121 of the distributed data processing block 120.

When it is determined that the read data input in step ST13 is distributed data having a distribution table, the data processing means 100 performs an SQL statement for all data storage means 200 to store the data in which the corresponding distributed data exists. Search 200, read the distributed data from the plurality of searched data storage means 200 and combine them and provide them to the corresponding terminal 400 (ST14).

On the other hand, when it is determined that the read data input in step ST13 does not have a distribution table, the data processing means 100 searches the metadata storage means 300 to search for the data load means 200 having the smallest current load. Through the SQL statement to be performed, the corresponding data is read by the retrieved data storage means 200 and provided to the corresponding terminal 400 (ST15).

That is, according to the above embodiment, the data processing means performs the main processing for the data distribution processing, and the data storage means performs only the SQL execution and the storage processing, and in consideration of the performance and the current load of the data storage means. By distributing and storing the divided data for the large data, it is possible to easily perform the processing for the large-scale distributed data using a low performance / low capacity data storage means.

100: data processing means, 110: distributed data management block,
120: distributed data processing block, 130: storage means management block,
140: metadata processing block, 150: interface block,
200: data storage means, 300: metadata storage means,
400: terminal, 500: operator terminal.

Claims (13)

A plurality of data storage means for performing SQL and storing data;
Metadata storage means for storing metadata including metadata about data stored in each data storage means and priority information for each data storage means;
Priority is calculated based on the performance of the data storage means and stored in the metadata storage means, and the stored data applied from the terminal is analyzed and the data storage means stored in the metadata storage means in the case of a large amount of distributed data. It comprises a data processing means for distributing and storing the data in a plurality of data storage means in a round-robin manner according to the ranking,
Priority information of the data storage means is set an initial priority value according to the performance of each data storage means, the sum of the initial priority value of all the data storage means is set to 100, initially the initial priority value and the current Set the priority value equally, and reduce the current priority value of the data storage means by the first reference value set by the operator when storing data, and execute SQL in the data storage means having the smallest current priority value. And calculating a second reference value set by the operator to a current priority value of the data storage means performed. The method of claim 1,
And metadata about the data storage means stored in the metadata storage means includes current load information. The method of claim 1,
Priority information of the data storage means is calculated based on the data processing speed and storage capacity of the data storage means and the current load amount. delete The method of claim 1,
If the data processing means analyzes the stored data applied from the terminal and determines that the input stored data is not distributed data, the data processing means performs SQL statements for all data storage means and stores the stored data. system. The method of claim 1,
The data processing means analyzes the read data applied from the terminal, in the case of distributed data, executes SQL in all data storage means to read the distributed and stored data, combines the read data, and provides the read data to the corresponding terminal.
In the case of non-distributed data by analyzing the read data applied from the terminal, the data distribution system is configured to read data by performing SQL through the data storage means having the least load amount and then provide the data to the corresponding terminal. . The method of claim 1,
The data processing means includes a distributed data management block for providing various application programming interfaces (APIs) related to insertion / deletion / search query processing for distributed processing related data;
SQL parser that checks whether distributed table information is included by analyzing SQL statement to be processed, and the corresponding SQL statement based on the parsed SQL syntax information and metadata about each data storage means stored in metadata storage means. A query router for determining the data storage means to be processed, and a data storage means for storing or reading data in a round-robin manner based on priority information on the data storage means provided from the query router. Distributed data processing block including a distributed query processor for determining the
Load balancer to check the current status information including the current load amount of each data storage means, and data processing speed and storage capacity of each data storage means and the current load information provided from the load balancer A storage means management block comprising a distributed management unit for calculating a priority value for storing distributed data on the basis of;
And a metadata processing block for storing metadata about the data stored in the data storage means, in addition to storing metadata including priority information and current load information of the data storage means provided from the storage means management block. Data distribution system, characterized in that configured to. In the data distribution method of the data distribution system is configured by combining a plurality of data storage means and metadata storage means in the data processing means,
A first step of calculating a priority value for each data storage means based on the performance of each data storage means and storing it in the metadata storage means;
A second step of dividing the corresponding stored data into a plurality of pieces when a large amount of stored data is inputted from the terminal;
And a third step of distributing and storing the divided data in a round-robin manner based on the priority value for the data storing means stored in the metadata storing means,
The first step may include setting an initial priority value according to the performance of each data storage means, and setting the sum of the initial priority values of all data storage means to 100;
A twelfth step of initially setting the initial priority value and the current priority value equally;
A thirteenth step of reducing the current priority value of the corresponding data storage means by the first reference value set by the operator during data storage;
The SQL is executed in the data storage means having the smallest current priority value, and the priority value is calculated through step 14 of adding the second reference value set by the operator to the current priority value of the data storage means in which SQL is performed. Data distribution method, characterized in that. 9. The method of claim 8,
And the first step calculates a priority value based on the data processing speed, the storage capacity, and the current load of the data storage means. delete 10. The method according to claim 8 or 9,
The first step further comprises storing the current load of data storage means in the metadata storage means. 9. The method of claim 8,
The second step may further include a twenty-first step of storing the stored data by performing SQL statements for all data storage means when the stored data input from the terminal is not a large amount of data. Way. 9. The method of claim 8,
If there is a request for reading a large amount of distributed data from the terminal, and performing a SQL in all data storage means to read the distributed and stored data, and further comprising a fourth step of combining the read data to provide to the terminal Data distribution method, characterized in that.

Images