KR101919816B1 - Consistency checking in database sharding environment - Google Patents

Abstract

A consistency checking technique in a database sharding environment is provided. A consistency checking method according to an embodiment of the present invention can check consistency of a large amount of data by using each group identifier of a preset number of a plurality of groups generated by classifying shardkeys of sharded data.

Description

Checking Consistency in Database Sharding Environment {CONSISTENCY CHECKING IN DATABASE SHARDING ENVIRONMENT}

The following description relates to a consistency checking technique in a database sharding environment.

Database sharding is a technique for horizontally partitioning a database table to physically distribute and query the database table in order to process a large amount of data. For example, Korean Patent No. 10-1527634 discloses a method and apparatus for providing a sharding service.

At this time, it is very difficult to verify the consistency with a large amount of data. For example, it can be considered that data for each of users in a service used by one million users is distributed and stored in a plurality of databases. At this time, since each of the plurality of databases manages only the data stored therein, when selecting, updating and deleting data for a specific one of the one million users among users, There is a problem that it is difficult to confirm whether or not it is included in the database.

In order to clarify this, it is necessary to manage the consistency information of which one of the plurality of databases contains information about which one of the users of one million users, and to add the data to one new user, The information must be updated for each of the multiple databases. Such management and updating require a very large cost.

A computer apparatus for performing a consistency checking method and a consistency checking method capable of checking consistency with respect to a large amount of data using group identifiers for a predetermined number of groups in which groups of shard keys are grouped, There is provided a computer program stored in a computer-readable recording medium for causing a computer to execute a consistency checking method according to embodiments of the present invention, and a recording medium thereof.

A computer program product for performing a consistency checking method and a consistency checking method capable of checking consistency of a large amount of data only by checking whether a group of the group identifiers is included in a group of the group identifiers for a predetermined number of group identifiers And a computer program stored on a computer-readable recording medium for causing a computer to execute a consistency checking method according to embodiments of the present invention in combination with a computer, and a recording medium therefor.

A method for checking a consistency of a database server, the method comprising the steps of: determining, in association with group identifiers of a plurality of groups of a predetermined number generated by classifying shardkeys of sharded data, Managing group identifiers of groups of which the keys are classified; And checking the consistency of the data associated with the requested operation in the database server using the extracted group identifier corresponding to the shard key associated with the requested operation .

And a computer program for causing the computer to execute the consistency checking method.

There is provided a computer program stored in a computer-readable recording medium for causing a computer to execute the consistency checking method in combination with a computer.

CLAIMS What is claimed is: 1. A computer device for a database server, the computer device comprising: at least one processor configured to execute computer readable instructions, the at least one processor being configured to classify and generate shardkeys of sharded data; Managing group identifiers of groups in which shard keys managed by the database server are grouped in association with group identifiers of a predetermined number of the plurality of groups, And checks the consistency of the data associated with the requested operation in the database server using the identifier.

The consistency of a large amount of data can be checked using a group identifier for a plurality of groups of a predetermined number of grouped shard keys.

It is possible to check the consistency of a large amount of data only by checking whether or not the shard keys are included in the group of the group identifier for a predetermined number of group identifiers.

1 is a diagram showing an example of data stored in a database in an embodiment of the present invention.
2 is a diagram showing an example of shading of data in an embodiment of the present invention.
3 is a diagram illustrating an example of a consistency checking process according to an embodiment of the present invention.
4 is a diagram showing an example of checking an incorrect data inquiry in an embodiment of the present invention.
5 is a block diagram for explaining an internal configuration of a computer apparatus according to an embodiment of the present invention.
6 is a flowchart showing an example of a consistency checking method in an embodiment of the present invention.
7 is a flowchart illustrating a first example of managing group identifiers in an embodiment of the present invention.
8 is a flowchart showing a second example of managing group identifiers in an embodiment of the present invention.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings.

The consistency checking method according to embodiments of the present invention can be performed by a computer apparatus. In one example, a computer program may be installed and run in a computer device for controlling a computer device to perform a consistency checking method, the computer program being coupled to the computer device to cause the computer device to execute a consistency checking method, Can be stored in the medium.

FIG. 1 is a diagram illustrating an example of data stored in a database according to an embodiment of the present invention. FIG. 2 is a diagram illustrating an example of data sharding in an embodiment of the present invention. 1 and 2 show an example in which data is distributed to n shard nodes 210, 220 and 230 by database sharding of data stored in the database 110. [ For example, in FIG. 1 and FIG. 2, records of the same shard key are collected by using the identifiers 'id1', 'id2', 'id3', and 'id4' Shards 210, 220, and 230, respectively. In this case, the data in units of the shard key may mean data identified through the same shard key. For example, in FIG. 2, the data identified through the shard key 'id2' is indicated by a thick solid line 240.

The data of the shard key unit has a relatively small capacity and QPS (Query Per Second), but the number of the shard key is very large, and as a result, the database overall capacity and the QPS are very large. Further, the relation between the data identified through the shard key is meaningful, but the relationship between the data identified through the different shard keys is meaningless. In other words, the data identified through one shard key has no dependency on the data identified by the other shard keys, and it is possible to process independent individual operations (transactions) on a shard key basis.

Each of the n shard nodes 210,220 and 230 may include data corresponding to one shard key, as shown through the first shard node 210, As shown, it may include data corresponding to a plurality of shard keys.

At this time, in the embodiments of the present invention, the shard keys are classified into a plurality of groups of a predetermined number in relation to the data of the shard key described above, and the consistency check of the data is performed through the group identifiers of the plurality of groups have. At this time, each of the database servers can manage the group identifiers of the group in which the shard key managed by the database servers is classified.

3 is a diagram illustrating an example of a consistency checking process according to an embodiment of the present invention. Each of the database servers according to the present embodiments can perform consistency checking on sharding data when performing all DML (Data Manipulation Language) queries such as INSERT, UPDATE, and DELETE.

In the example of FIG. 3, the JDBC (Java DataBase Connectivity) 320 extracts the shard key 'id1' and hashs the shard key 'id1' to obtain the hash value '3' Respectively. At this time, the hash value '3' may be the group identifier of the group including the shard key 'id1'. For example, each of the shard keys may be used as a parameter of a predetermined hash function so that group identifiers for each shard key may be computed. In a more specific example, one million group identifiers may be computed through a hash function of one million shard keys, and grouped into 10,000 groups for each group identifier.

In this case, even if a new shard key is added, a group or group identifier is not added because it is classified as one of the 10,000 groups in the process of hashing through the hash function. Therefore, each database server does not manage the shard keys managed by itself among the 1 million shard keys but manages only the group identifier managed by itself among the 10,000 group identifiers, so that the specific shard key is managed by itself Group is included in the group.

At this time, the data movement between the database servers may be performed in units of data corresponding to the group identifier. For example, when database server a stores data for group identifier 1, the movement of data from database server a to database server b may be in units of data corresponding to group identifier 1.

3, the database server 330 includes a group identifier bit for indicating a group identifier corresponding to data managed by the group identifier and a group identifier corresponding to data not managed by the database server 330, The map 340 can be managed. For example, in FIG. 3, the database server 330 stores and manages data corresponding to the group identifiers 1, 3, and n in the database j 350 (the values of the group identifiers 1, 3, and n are set to 1 ), The data corresponding to the group identifier 2 is not managed (the value of the group identifier 2 is set to 0).

At this time, as described above, the JDBC 320 extracts the shard key 'id1' from the query statement 310 and extracts the group identifier '3' by hashing the extracted shard key 'id1' through the hash function have. At this time, the extracted group identifier '3' may be transmitted to the database server 330, and the database server 330 searches the group identifier bitmap 340 for the extracted group identifier '3' Is stored in the database J 350 and is managed by the database server 330. In other words, the database server 330 can check the consistency of the data associated with the requested operation. If the extracted group identifier is '2', the database server 330 can recognize that a wrong data change request has been input.

In addition, the database server 330 can perform consistency checking not only for data change request but also for data inquiry.

4 is a diagram showing an example of checking an incorrect data inquiry in an embodiment of the present invention. 4 shows the database server 1 410 and the database server 2 420 and the database 1 411 managed by the database server 1 410 and the database 2 421 managed by the database server 2 420, .

In FIG. 4, it is assumed that the data corresponding to the group identifier '3' of the database 1 (411) is transferred to the database 2 (421). It is assumed that the data corresponding to the group identifier '3' has not yet been deleted in the database 1 411 and the data corresponding to the group identifier '3' exists in both the database 1 411 and the database 2 412 do. It is also assumed that query statements 412 and 422 for retrieving mail for the shard key '1' are input to all database servers. At this time, there is a possibility that the result of the requested operation may include an incorrect result because it includes both the result of inquiry from the database 1 (411) and the result of inquiry from the database 2 (421). For example, when the data corresponding to the group identifier '3' is changed in the database 2 421, both the pre-change inquiry result and the post-change inquiry result are provided, which compromises the consistency of the data.

In this embodiment, to prevent this problem, group identifier bitmaps 413 and 423 may be utilized. Assuming that the hash value for the shard key '1' is '3', the group identifier '3' may be extracted, and the database server 1 410 may output the group identifier bitmap 413 The database server 1 410 can recognize that the database server 1 410 does not manage data corresponding to the group identifier '3'.

For example, when transferring data corresponding to the group identifier '3' to the database server 2 420, the database server 1 410 sets a value corresponding to the group identifier '3' in the group identifier bitmap 413 as It can be changed from '1' to '0'. Therefore, even if the data corresponding to the group identifier '3' still remains in the database 1 411, the database server 1 410 can grasp that the data corresponding to the group identifier '3' is not the data managed by itself have.

On the other hand, when the database server 2 420 migrates data corresponding to the group identifier '3' from the database server 1 410, the database server 2 420 deletes the value corresponding to the group identifier '3' from the group identifier bitmap 423 Can be changed from '0' to '1'. Accordingly, the database server 2 420 can know that the data corresponding to the group identifier '3' stored in the database 2 421 is the data managed by itself.

In other words, after the transfer of the data is completed, the database server 1 410 does not provide the result value for the query statements 412, 422 entered, and the database server 2 420 provides the result value. In other words, it is possible to check for incorrect data retrieval.

In the embodiments of FIGS. 3 and 4, an example of managing group identifiers in the form of a bitmap has been described. However, the present invention is not limited to this, as long as each database server can recognize group identifiers managed by the database servers . In the above embodiments, the hash function is used in classifying the shard keys into a plurality of groups. However, the present invention is not limited to this, as long as the shard keys can be classified into a plurality of groups. For example, the shard keys may be sequentially assigned to a plurality of groups according to the order of values. In a more detailed example, if the values of the shard keys are {1, 2, 3, 4, 5, 6, 7, 8, 9, 10} and the group identifiers of the plurality of groups are {a, b, c} 1, 4, 7, and 10 can be classified into group a, shard keys 2, 5, and 8 to group b, and shard keys 3, 6, and 9 to group c, respectively.

5 is a block diagram for explaining an internal configuration of a computer apparatus according to an embodiment of the present invention. The computer device 500 may include a memory 510, a processor 520, a communication interface 530, and an input / output interface 540. The memory 510 may be a computer-readable recording medium and may include a permanent mass storage device such as a random access memory (RAM), a read only memory (ROM), and a disk drive. The non-decaying mass storage device, such as a ROM and a disk drive, may be included in the computer device 500 as a separate persistent storage device separate from the memory 510. The memory 510 may also store an operating system and at least one program code. These software components may be loaded into the memory 510 from a computer readable recording medium separate from the memory 510. [ Such a computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD / CD-ROM drive, and a memory card. In other embodiments, the software components may be loaded into the memory 510 via the communication interface 530 rather than a computer-readable recording medium. For example, the software components may be loaded into the memory 510 of the computer device 500 based on the computer program installed by the files received via the network 560.

The processor 520 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and I / O operations. The instructions may be provided to the processor 520 by the memory 510 or the communication interface 530. For example, processor 520 may be configured to execute instructions received in accordance with program code stored in a recording device, such as memory 510. [

The communication interface 530 may provide functionality for the computer device 500 to communicate with another device (e. G., The storage devices described above) via the network 560. [ For example, a request or command, data, file, or the like, generated by the processor 520 of the computer device 500 according to the program code stored in the recording device such as the memory 510, 560 to other devices. Conversely, signals, commands, data, files, etc., from other devices may be received by the computer device 500 via the network 560 and the communication interface 530 of the computer device 500. Data or the like received via the communication interface 530 may be transferred to the processor 520 or the memory 510 and the file or the like may be transferred to a storage medium Permanent storage).

The input / output interface 540 may be a means for interfacing with the input / output device 550. For example, the input device may include a device such as a microphone, a keyboard or a mouse, and an output device may include a device such as a display, a speaker, and the like. As another example, the input / output interface 540 may be a means for interfacing with a device having integrated functions for input and output, such as a touch screen. The input / output device 550 may be composed of the computer device 500 and one device.

Also, in other embodiments, the computer device 500 may include fewer or more components than the components of FIG. However, there is no need to clearly illustrate most prior art components. For example, the computer device 500 may be implemented to include at least some of the input / output devices 550 described above, or may include other components such as a transceiver, Global Positioning System (GPS) module, camera, As shown in FIG.

Such a computer device 500 may correspond to the database servers 330, 410, and 420 described with reference to FIGS. 3 and 4, for example.

6 is a flowchart showing an example of a consistency checking method in an embodiment of the present invention. The consistency checking method according to the present embodiment can be performed by the computer apparatus 500 described above. For example, the processor 520 of the computer device 500 may be implemented to execute a control instruction according to code of an operating system loaded in the memory 510 or code of at least one program. Here, the processor 520 may control the computer apparatus 500 so that the computer apparatus 500 performs the steps 610 and 620 included in the consistency checking method of FIG. 6 according to this control instruction.

In step 610, the computer device 500 determines whether the computer device 500 is associated with group identifiers of each of a plurality of groups of a predetermined number generated by classifying the shardkeys of the sharded data, The group identifier of the group in which the shard key managed by the group key management unit is classified. At this time, the computer device 500 generates and manages a group identifier bitmap, which is set for each group identifier of a predetermined number of groups, in which information on whether or not a shard key group managed by the computer device 500 is grouped can do.

For example, the group identifier may include a hash value calculated by inputting each of the shard keys as a parameter of a predetermined hash function, and the shard keys having the same hash value may be classified into the same group. 3 and 4, '1' is set for group identifiers managed by the computer device 500 for all group identifiers, and '1' is set for group identifiers not managed by the computer device 500 A group identifier bitmap in which '0' is set has been described. In another embodiment, the computer device 500 may manage group identifiers in a form that includes only a collection of group identifiers that it manages.

In step 620, the computer device 500 may check the consistency of the data associated with the requested operation using the extracted group identifier in response to the shard key associated with the requested operation. In the embodiment of FIG. 3, an example has been described in which the JDBC 320 extracts and provides the group identifier '3' by hashing the shard key 'id1' included as a parameter in the query. The computer device 500 can check the consistency of the data using the group identifier extracted through the shard key associated with the requested operation.

For example, when the extracted group identifier is included in the group identifiers managed by the computer device 500, the computer device 500 determines that the consistency exists, and when the extracted group identifier is managed by the computer device 500 It can be determined that there is lack of consistency if it is not included in the group identifiers. Also, the computer device 500 may provide an error message to the user or the administrator when it is determined that the integrity is lacking.

7 is a flowchart illustrating a first example of managing group identifiers in an embodiment of the present invention. Steps 710 and 720 in FIG. 7 may be performed in step 610 described with reference to FIG.

In step 710, when the data corresponding to the first group identifier is moved to the computer device 500 as the data moves between the database servers, the first group identifier is stored in the group managed by the computer device It can be confirmed whether it is included in the identifiers.

In step 720, if the first group identifier is not included in the group identifiers managed in the computer device 500, the computer device 500 transmits the first group identifier to the group identifiers managed in the computer device 500 Can be added to manage more.

As described above, the movement of data between the database servers can be made in units of data corresponding to at least one group identifier. Thus, when data corresponding to a particular group identifier is moved to the computer device 500, the computer device 500 can further manage the group identifier.

8 is a flowchart showing a second example of managing group identifiers in an embodiment of the present invention. Step 810 of FIG. 8 may be performed in step 610 described above with reference to FIG.

In step 810, when the data corresponding to the first group identifier managed by the computer device 500 is moved to another database server in accordance with the movement of data between the database servers, the computer device 500 transmits, The first group identifier can be managed by excluding the first group identifier from the group identifiers managed by the second group identifier.

As described above, the movement of data between the database servers can be performed in units of data corresponding to at least one group identifier. Accordingly, when data corresponding to a specific group identifier is moved from the computer device 500 to another database server, the computer device 500 can manage the remaining group identifiers except for the corresponding group identifier.

In the embodiments of FIGS. 7 and 8, when the group identifier bitmap is used, the computer device 500 may change the value corresponding to the first group identifier to '1' or '0' It is possible to identify whether the data corresponding to the specific group identifier is managed through the map.

As described above, according to the embodiments of the present invention, it is possible to check the consistency of a large amount of data using a group identifier for a predetermined number of groups in which the shard keys are grouped. In addition, it is possible to check the consistency of a large amount of data by checking whether or not the shard keys are included in the group of the group identifier for a predetermined number of group identifiers.

The system or apparatus described above may be implemented as a hardware component, a software component or a combination of hardware components and software components. For example, the apparatus and components described in the embodiments may be implemented within a computer system, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA) , A programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For ease of understanding, the processing apparatus may be described as being used singly, but those skilled in the art will recognize that the processing apparatus may have a plurality of processing elements and / As shown in FIG. For example, the processing unit may comprise a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as a parallel processor.

The software may include a computer program, code, instructions, or a combination of one or more of the foregoing, and may be configured to configure the processing device to operate as desired or to process it collectively or collectively Device can be commanded. The software and / or data may be in the form of any type of machine, component, physical device, virtual equipment, computer storage media, or device As shown in FIG. The software may be distributed over a networked computer system and stored or executed in a distributed manner. The software and data may be stored on one or more computer readable recording media.

The method according to an embodiment may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The medium may be one that continues to store computer executable programs, or temporarily store them for execution or download. In addition, the medium may be a variety of recording means or storage means in the form of a combination of a single hardware or a plurality of hardware, but is not limited to a medium directly connected to a computer system, but may be dispersed on a network. Examples of the medium include a magnetic medium such as a hard disk, a floppy disk and a magnetic tape, an optical recording medium such as CD-ROM and DVD, a magneto-optical medium such as a floptical disk, And program instructions including ROM, RAM, flash memory, and the like. As another example of the medium, a recording medium or a storage medium managed by a site or a server that supplies or distributes an application store or various other software to distribute the application may be mentioned. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. For example, it is to be understood that the techniques described may be performed in a different order than the described methods, and / or that components of the described systems, structures, devices, circuits, Lt; / RTI > or equivalents, even if it is replaced or replaced.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (16)

A method for checking consistency of a database server,
A shard key managed by the database server is associated with a group identifier of a group in which a shard key is classified in association with each group identifier of a predetermined number of groups generated by sorting shardkeys of sharded data Managing; And
Checking the consistency of the data associated with the requested operation in the database server using the group identifier extracted corresponding to the shard key associated with the requested operation
And checking the consistency of the image. The method according to claim 1,
Wherein managing the group identifiers comprises:
Wherein information on whether or not the shard key group managed by the database server is generated and managed is set for each of the group identifiers of the predetermined number of groups. The method according to claim 1,
Wherein managing the group identifiers comprises:
Confirming whether the first group identifier is included in the managed group identifiers when data corresponding to the first group identifier is moved to the database server according to movement of data between the database servers; And
If the first group identifier is not included in the managed group identifiers, adding the first group identifier to the managed group identifiers and further managing the first group identifier
And checking the consistency of the image. The method according to claim 1,
Wherein managing the group identifiers comprises:
When data corresponding to a first group identifier managed by the database server is moved to another database server in accordance with movement of data between database servers, the first group identifier is excluded and managed from the managing group identifiers The method comprising the steps of: The method according to claim 3 or 4,
Wherein movement of data between the database servers is performed in units of data corresponding to at least one group identifier. The method according to claim 1,
The group identifier includes a hash value calculated by inputting each of the shard keys as a parameter of a predetermined hash function,
And the shard keys having the same hash value are classified into the same group. The method according to claim 1,
Wherein the step of checking consistency comprises:
Determining that there is consistency when the extracted group identifier is included in the managed group identifiers and determining that the extracted group identifier is not included in the managed group identifiers if the extracted group identifier is not included in the managed group identifiers . A computer program stored on a computer-readable medium for causing a computer to execute the method of any one of claims 1 to 4, 6, or 7 in combination with the computer. A computer-readable recording medium having recorded thereon a program for causing a computer to execute the method according to any one of claims 1 to 4, 6 or 7. A computer device for a database server,
At least one processor configured to execute computer readable instructions,
Lt; / RTI >
Wherein the at least one processor comprises:
A shard key managed by the database server is associated with a group identifier of a group in which a shard key is classified in association with each group identifier of a predetermined number of groups generated by sorting shardkeys of sharded data Management,
Checking the consistency of the data associated with the requested operation in the database server using the extracted group identifier corresponding to the shard key associated with the requested operation
The computer device comprising: 11. The method of claim 10,
Wherein the at least one processor comprises:
And generating and managing a group identifier bit map in which information on whether or not a shard key group managed by the database server is grouped is set for each of group identifiers of a predetermined number of groups
The computer device comprising: 11. The method of claim 10,
Wherein the at least one processor comprises:
Wherein the database management server determines whether the first group identifier is managed by the database server when the data corresponding to the first group identifier is moved to the database server according to the movement of data between the database servers,
Adding the first group identifier to the managed group identifiers and further managing the first group identifier when the first group identifier is not included in the managed group identifiers
The computer device comprising: 11. The method of claim 10,
Wherein the at least one processor comprises:
Managing data of the first group identifier managed by the database server according to the movement of data between the database servers, excluding the first group identifier from the managed group identifiers when the data corresponding to the first group identifier is moved to another database server
The computer device comprising: The method according to claim 12 or 13,
Wherein movement of data between the database servers is performed in units of data corresponding to at least one group identifier. 11. The method of claim 10,
The group identifier includes a hash value calculated by inputting each of the shard keys as a parameter of a predetermined hash function,
Wherein the shard keys having the same hash value are classified into the same group. 11. The method of claim 10,
Wherein the at least one processor comprises:
Determining that there is consistency when the extracted group identifier is included in the managed group identifiers and determining that the extracted group identifier is not included in the managed group identifiers if the extracted group identifier is not included in the managed group identifiers
The computer device comprising:

Images