KR20180119888A - Hybrid Sharding system - Google Patents

Abstract

According to a preferred embodiment of the present invention, provided is a hybrid-sharding system including: at least one or more shard DBs for storing distributed data; and a hybrid-sharding unit. The hybrid-sharding unit includes a shard library and a meta node. Accordingly, the present invention can dispersedly process a database with a large capacity.

Description

[0001] Hybrid Sharding System [

The present invention relates to a method of sharding a database. More particularly, the present invention relates to a hybrid sharding system incorporating server side sharding technology and client side sharding technology.

With the explosion of data in the recent business environment, distributed database technology developed along with data distribution processing and storage technology is attracting attention. Of the various data processing technologies, Sharding technology easily divides large amount of data and can process large amount of data by expanding system of scale out method using low cost system without introducing high performance system.

In the database field, sharding refers to a method of distributing and querying data horizontally in a physically different database, and refers to the horizontal partitioning of one database with each individual partition, called a shard. When sharding is performed, each shard can receive more computational resources than a single large database. Therefore, if the data processing speed is fast and the redundancy technique is applied, even if a failure occurs in one shard, It is possible to provide a service in the mobile communication system.

KR 10-1544356 B1

In the case of the server side sharding system, there is a problem that performance increases can not be expected as the number of added data nodes increases. In the case of the client sharding system, there is a disadvantage that the system reconstruction cost is large when the data dispersion policy is changed.

According to a preferred embodiment of the present invention, there is provided a hybrid-sharding system, comprising: at least one shard DB for storing distributed data; and a hybrid shading unit, A meta node for determining whether the query is a shard query including a shard object, distributing data to each of the at least one shard DB based on a shard key in the case of a shard query, A coordinator between the application and the at least one shard DB, a user query is transmitted to the meta node, information of the at least one shard DB registered in the meta node is received, And the at least one shard DB And a shard library for storing the shard library.

Preferably, the hybrid shading unit further includes a selection unit selecting a server-side sharding mode or a client-side sharding mode.

Preferably, in the hybrid shading unit, the shard library requests table information in each of the shard DBs to the meta node when accessing the first query at the time of interpreting the user query, And receives a response from the meta node, so that the shard library directly accesses each of the shard DBs.

As another preferred embodiment of the present invention, a method of performing sharding in a hybrid-sharding system is installed in a library form in an application of a client terminal, and serves as a coordinator between the application and at least one shard DB Passing a user query to a meta node in a shard library; Receiving information of at least one shard DB registered in the meta node in the shard library and performing connection between the client terminal and at least one shard DB; Analyzing the user query in the meta node to determine whether the query is a shard query including a shard object; And distributing data to each of at least one shard DB based on a shard key in the case of a shard query as a result of determination by the meta node.

As a preferred embodiment of the present invention, the hybrid sharding apparatus has the effect of improving the storage capacity and the throughput per hour, and distributing a large-capacity database.

1 shows a hybrid shading system 100 as a preferred embodiment of the present invention.
2 shows an example of a system for performing server side sharding.
3 shows an example of a system for performing client side sharding.
FIG. 4 illustrates a method of operating a hybrid shading system according to an embodiment of the present invention.
Figure 5 shows a system diagram illustrating steps for performing sharding in a conventional sharding system.
6 shows a step for performing sharding in a hybrid sharding system, which is a preferred embodiment of the present invention.
7 shows a flowchart for performing sharding in a hybrid sharding system, which is a preferred embodiment of the present invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS The advantages and features of the present invention and the manner of achieving them will become apparent with reference to the embodiments described in detail below with reference to the accompanying drawings.

Sharding is a scale-out technique that distributes data stored in one database to multiple databases and stores it. Sharding technology easily divides large amount of data and can process large amount of data by expanding scale-out system using low cost system without introducing high performance system.

The sharding technique can be divided into a server-side sharding method for separating and processing data using a coordinator, and a client-side sharding method for separating and processing data in an application.

1 shows a hybrid shading system 100 as a preferred embodiment of the present invention. In a preferred embodiment of the present invention, the hybrid sharding system can simultaneously support the server-side sharding function and the client-side sharding function. In addition, it is possible to select only the server-side sharding function or only the client-side sharding function if necessary.

As a preferred embodiment of the present invention, in the hybrid sharding system 100, even if the number of the shard DBs 130, 132, 134, 136 increases, the server side sharding can be improved in the overall performance, There is an advantage that the application of the client terminal can be prevented from being modified.

As a preferred embodiment of the present invention, in the hybrid sharding system 100, the client side sharding can be implemented by only replacing the shard dedicated library without modifying the existing application source or existing SQL.

The hybrid sharding system 100 includes at least one application 112, 114, and 116 installed in the client terminal 110 and the client terminal 110 and at least one application 112, 114, 116, a Shard Library 113, 115, 117, a meta node 120, and at least one Shard DB 130, 132, 134, 136 for storing distributed data.

As a preferred embodiment of the present invention, the hybrid shading system 100 includes a hybrid shading unit. The hybrid sharding portion includes a meta node 120 and at least one shard library 113, 115, 117.

As a preferred embodiment of the present invention, the meta node 120 manages data nodes and sharding information, analyzes a user query, and provides integrated queries when performing a server side sharding function as in the example shown in FIG. 2 As a coordinator. It also performs the function of redistributing the data to shard DBs.

In one preferred embodiment of the present invention, at least one of the shard libraries 113, 115, and 117 is installed in the form of a library in a client terminal to perform a sharding function and provides the same API interface as the existing ODBC.

In one preferred embodiment of the present invention, at least one of the shard libraries 113, 115, and 117 also provides a coordinator role between the applications 112, 114, and 116 installed in the client terminal and the shard DBs 130, 132, .

In a preferred embodiment of the present invention, the server side sharding function and the client side sharding function can be simultaneously supported by using the hybrid sharding portion. Further, in addition to the hybrid shading section, the shard manager may further include a selection function for selecting either the server-side sharding function or the client-side sharding function.

2 shows an example of a system for performing server side sharding. The server side sharding as shown in FIG. 2 requires a coordinator 220 that integrates divided shard DBs 230, 232, and 234 for compatibility with applications installed in the client terminals 200, 212, and 214.

The coordinator 220 grasps the location of data corresponding to the query requested by the application, relays the data to the shard DBs 230, 232, and 234, re-executes the query, and returns the result to the application.

This server side sharding system has a merit that when the data distribution policy is changed, it is unnecessary to modify the application installed in the client terminal. However, there is a disadvantage that the performance increase is lowered as the shard DB node is added to the load of the coordinator.

3 shows an example of a system for performing client side sharding. Since the client side sharding as shown in FIG. 3 knows the shard DBs 330, 332 and 334 in which the data is located in the applications installed in the client terminals 310, 312 and 314, the shard DBs 330, 332 and 334 Connection is possible. Thus, client side sharding has the advantage that no separate coordinator is needed, and the overall throughput performance is increased as the number of shard DB nodes increases.

However, the client side sharding is difficult to create an application, and when the data distribution policy is changed, it is required to perform application modification and data redistribution on the client side, which has a disadvantage that the system reconstruction cost is large.

FIG. 4 illustrates a method of operating a hybrid shading system according to an embodiment of the present invention.

As a preferred embodiment of the present invention, the hybrid sharding system may implement both the server-side sharding function and the client-side sharding function.

First, the server side sharding function of the hybrid sharding system will be described with reference to FIG.

The server-side sharding of the hybrid sharding system has the advantage that it can be used without any modification to any application program installed on the client terminal. Shared connection management is implemented as follows when implementing the server side sharding function.

The application 412 installed in the client terminal 410 attempts to access the meta node 420 through the shard library 413. [ It is possible to connect in the same way as usual database connection method.

And creates a session in the meta node 420. The application 412 requests a user query including the shard object in the meta node 420

An example of determining whether a shard query includes a shard object is as follows.

/ * Create a table on each node after completing node configuration * /

CREATE TABLE t1 (id INTEGER, name VARCHAR (50));

/ * T1  of Shard  Set as table * /

EXEC DBMS_SHARD.SET_SHARD_TABLE ('SYS', 'T1', 'R', 'ID', 'NODE1');

EXEC DBMS_SHARD.SET_SHARD_RANGE ('SYS', 'T1', 3, 'NODE2');

EXEC DBMS_SHARD.SET_SHARD_RANGE ('SYS', 'T1', 6, 'NODE3');

/ * Enter data on each node * /

INSERT INTO t1 VALUES (1, 'Kim');

INSERT INTO t1 VALUES (2, 'Lee');

INSERT INTO t1 VALUES (3, 'Park');

INSERT INTO t1 VALUES (4, 'Choi');

INSERT INTO t1 VALUES (5, 'Jeong');

INSERT INTO t1 VALUES (6, 'Kang');

INSERT INTO t1 VALUES (7, 'Joe');

INSERT INTO t1 VALUES (8, 'Yoon');

INSERT INTO t1 VALUES (9, 'Jang');

/ * Test the query * /

iSQL > SELECT * FROM t1  WHERE id = 2;

Since it is possible to query only from a specific node, normal operation is performed.

ID NAME

-------------------------------------------------- -----------------

2 Lee

1 row selected.

iSQL > SELECT * FROM t1 ; - Shard  Because it is a table Single query  Error in view

[ERR-E1385: The shard table is only available inside the shard view.

0001: SELECT * FROM T1

]

iSQL > SHARD SELECT * FROM t1 ; - Use "SHARD" syntax when viewing all distributed data

ID NAME

-------------------------------------------------- -----------------

7 Joe

8 Yoon

9 Jang

1 Kim

2 Lee

3 Park

4 Choi

5 Jeong

6 Kang

9 rows selected.

iSQL > SELECT * FROM t1  WHERE id = 2 OR id = 3; - Normal operation is possible because it can be inquired only from specific node.

ID NAME

-------------------------------------------------- -----------------

2 Lee

3 Park

2 rows selected.

iSQL > SELECT COUNT (*) FROM t1 ; - We need to look up the sum of all nodes Single query  Error in use

[ERR-E1385: The shard table is only available inside the shard view.

0001: SELECT COUNT (*) FROM T1

]

iSQL > SHARD SELECT COUNT (*) FROM t1 ;

- You need to search by sum of all nodes, so use "SHARD" syntax

COUNT (*)

-----------------------

3

3

3

3 rows selected.

iSQL > SELECT SUM ( c1 ) FROM SHARD (SELECT COUNT (*) c1  FROM t1 );

- You need to search by sum of all nodes, so use "SHARD" syntax

SUM (C1)

-----------------------

9

1 row selected.

The meta node 420 generates a shard connection for each session for all the shard DBs 430, 432, 434, 436, and 438 registered in the meta node. When the session ends, the shard connection is also terminated.

The shard connection management is performed in the hybrid sharding system as described above (S410), and the input user query is analyzed as follows (S420).

The meta node 420 analyzes the user query requested by the application 412. When the user query is a shard query, the analysis result is generated, and the query tree is optimized by the analysis result to generate the plan tree. The meta node 420 can classify and process the case where the user query is a shard query and the case where the user query is not a shard query. A user query that is not a shard query processes the user query as a coordinator in the meta node 420.

When the shard query is performed, the meta node 420 performs the generated plan tree. When the plan is inquired after the query execution, the meta node 420 inquires the shard SQL plan performed by each shard DB 430, 432, 434, 436, can do. The meta node 420 returns the execution result of the shard query to the application 412.

Next, the client side sharding function of the hybrid sharding system will be described with reference to FIG.

As a preferred embodiment of the present invention, the client side sharding function of the hybrid sharding system does not require application modification or SQL modification by using the shard dedicated library 413 and the meta node 420.

In a preferred embodiment of the present invention, when the hybrid sharding system implements the client side sharding function, the meta node 420 includes the schema information of the shard DBs through analysis only when preparing the query for the first time in the application (442) And generates meta information. The application 412 obtains information on which tables are present in the shard DBs 430, 432, and 434 through a shard schema inquiry when accessing the meta node 420 for the first time. Only the first one analysis is required and no additional analysis is required.

The meta node 420 can repeatedly execute the query based only on the generated meta information and the bind information of the application 412. [ As a result, there is an advantage that the performance scalability of the client side sharding is maintained, but the application does not need to be modified or rewritten.

The meta node 420 analyzes the user query and transmits data based on the shard key 450 to at least one of the shard DBs 430, 432, 434, 436, and 438 in the case of the shard query including the shard object. And performs dispersion processing on each of them. In a preferred embodiment of the present invention, a method such as Range, List, Hash, etc. can be used in a method using a shard key 450.

When the hybrid sharding system implements the client side sharding function, the shard library 413 accesses the meta node 420 when the application 412 calls the SQLDriverConnect () function S414 in the meta node 420. The shard library 413 receives information of all the shard DBs 430, 432, 434, 436, and 438 that serve as data nodes registered in the meta node 420. Thereafter, accessing all the shard DBs 430, 432, 434, 436, and 438 informs the application 412 that the connection is successful. However, if one of the shard DBs 430, 432, 434, 436, and 438 fails to connect, the connection of the shard DBs that have already been successfully connected is terminated and the connection to the application 412 is notified.

When the shard connection is created, the application 412 calls the SQLPrepare () function (442). The shard library 413 passes the user query to the meta node 420. The meta node 420 analyzes whether the user query received in the application 412 is a shard query and transmits the analysis result to the shard library 413.

If the user query is a query that can not be performed in the shard library 413, an error message is transmitted to the application 412. The user query analysis result may include whether the user query is a shard query, a shard DB list in which the shard query can be performed in case of a shard query, a host variable related to the shard key, and an interpretation method for the bind value.

When the shard query is analyzed, the shard library 413 performs an SQLPrepare () 442 operation on the shard DBs included in the user query analysis result. When the application 412 calls the SQLBindParameter () function 444, the SQLBindParameter () 444 is performed on the shard DBs included in the user query analysis result.

 When executing SQLExecute () 446 in the application 412, the shard library 413 finds a value associated with the shard key among the bound values, and then interprets the bind value to determine the shard DB 430, 432, 434, 436, 438). Executes SQLExecute () 446 for the selected shard DB, and transfers the execution result to the application 412.

Figure 5 shows a system diagram for performing shading in a conventional sharding system.

The client 510 transmits a user query to the coordinator 520 in operation S510 and receives the result from the coordinator 520 in operation S511. The coordinator 520 transmits the user query to the data nodes 530, 532, 534 again The user query is transmitted (S512), and the resultant value is received (S513). Therefore, there has been an inconvenience that the network has to be passed twice every time data processing is performed.

However, as a preferred embodiment of the present invention shown in FIG. 6, the hybrid sharding system requires communication with the meta node only at the first time (S610-S611). When performing data processing thereafter, (S612, S613), the communication cost is reduced, and there is no restriction on performance enhancement even if a shadow node is added.

 FIG. 7 is a flowchart illustrating a method of performing shading in a hybrid-sharding system according to an embodiment of the present invention.

The shard library (see FIG. 4, 413) of the client terminal transmits the user query to the meta node (see FIG. 4, 420) (S710). The shard library receives information of at least one shard DB registered in the meta node and performs connection between the client terminal and at least one shard DB (S720). After that, the meta node analyzes the user query to determine whether the query is a shard query including the shard object, and then, when the query is a shard query, the data is distributed to each of at least one shard DB based on the shard key (S730 ).

The present invention can also be embodied as computer-readable codes on a computer-readable recording medium. A computer-readable recording medium includes all kinds of recording apparatuses in which data that can be read by a computer system is stored. Examples of the computer-readable recording medium include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage, and the like. The computer readable recording medium may also be distributed over a networked computer system so that computer readable code can be stored and executed in a distributed manner.

In the drawings and specification, there have been disclosed preferred embodiments. Although specific terms have been employed herein, they are used for purposes of illustration only and are not intended to limit the scope of the invention as defined in the claims or the claims. Therefore, those skilled in the art will appreciate that various modifications and equivalent embodiments are possible without departing from the scope of the present invention. Accordingly, the true scope of the present invention should be determined by the technical idea of the appended claims.

Claims (11)

With a hybrid-sharding system,
At least one shard DB for storing distributed data; and
And a hybrid shading unit, wherein the hybrid shading unit
A meta node for analyzing a user query to determine whether the query is a shard query including a shard object and distributing data to each of the at least one shard DB based on a shard key in the case of a shard query;
A plurality of shard DBs, which are installed in an application of a client terminal in a library form, and serve as a coordinator between the application and the at least one shard DB, transmit a user query to the meta node, And a shard library for receiving information from the client terminal and performing connection between the client terminal and the at least one shard DB. [2] The apparatus of claim 1, wherein the hybrid shading unit
Further comprising: a selection unit selecting a server-side sharding mode or a client-side sharding mode. The hybrid shading device according to claim 1,
The shard library requests table information in each of the shard DBs to the meta node when accessing the meta node for the first time when analyzing a user query, inquires where the data exists in the shard DB, Wherein the shard library directly accesses each of the shard DBs by receiving an answer thereto from the node. The method of claim 1, wherein when the hybrid sharding portion is implemented in a server-side sharding mode, the meta node may implement a compound syntax that includes additional grouping, ordering, data node joins, And the data transfer and data redistribution between the shard DBs can be implemented. 2. The method according to claim 1, wherein when the hybrid sharding part is implemented in a server side sharding mode, the application of the client terminal accesses the meta node, the meta node creates a session, And requesting the shard query, a shard connection is generated for each session for each of the at least one shard DB registered in the meta node. 6. The method of claim 5, wherein the meta node
A plan tree is generated by optimizing the shard query analysis result by separating the case where the user query is a shard query and the case where the user query is not a shard query, and the result of the execution is transmitted to an application of the client terminal And returns the hybrid sharding system. The method according to claim 1, wherein, when the hybrid sharding part is implemented in a client side sharding mode, a shard library installed in an application of the client terminal accesses the meta node, And generates a shard connection when connecting to each of the at least one shard DB. [8] The method of claim 7, wherein the meta node returns a shard query analysis result to the application of the client terminal when the user query is a shard query, and the shard query analysis result indicates whether the user query is a shard query, A shard DB list to which a query can be performed, a host variable associated with the shard key, and a value of a void. A method of performing shading in a hybrid-sharding system,
Transmitting a user query to a meta node in a shard library installed as a library in an application of a client terminal and serving as a coordinator between the application and at least one or more shard DBs;
Receiving information of at least one shard DB registered in the meta node in the shard library and performing connection between the client terminal and at least one shard DB;
Analyzing the user query in the meta node to determine whether the query is a shard query including a shard object; And
And distributing data to each of at least one shard DB based on a shard key in the case of a shard query as a result of determination by the meta node. 10. The system of claim 9, wherein the hybrid sharding system
And selecting a server-side sharding mode or a client-side sharding mode. 10. The method of claim 9,
The shard library requests table information in each of the shard DBs to the meta node when accessing the meta node for the first time when interpreting a user query and inquires of which of the at least one shard DB is data exists And receiving a response thereto from the meta node, the shard library then directly connects to each of the shard DBs.

Images