CN106547471B - Method and device for expanding non-relational database - Google Patents
Method and device for expanding non-relational database Download PDFInfo
- Publication number
- CN106547471B CN106547471B CN201510596035.0A CN201510596035A CN106547471B CN 106547471 B CN106547471 B CN 106547471B CN 201510596035 A CN201510596035 A CN 201510596035A CN 106547471 B CN106547471 B CN 106547471B
- Authority
- CN
- China
- Prior art keywords
- server
- disk
- fragment
- relational database
- shard
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a method and a device for expanding a non-relational database. The method for expanding the non-relational database comprises the following steps: monitoring the data volume of the stored data of each first fragment of the non-relational database, wherein the first fragment is the fragment of the non-relational database at present; comparing the data volume of each first fragment with a preset threshold value; and adding a second fragment to the non-relational database under the condition that the first fragment with the data quantity larger than the preset threshold value is compared. By the method and the device, the technical problem that in the prior art, the complexity of artificially expanding the non-relational data is high is solved.
Description
Technical Field
The present application relates to the field of data processing, and in particular, to a method and an apparatus for expanding a non-relational database.
Background
With the rise of internet web2.0 websites, the traditional relational database is Not capable of dealing with the web2.0 websites, especially the Social Network Service (SNS) type web2.0 purely dynamic websites with ultra-large scale and high concurrency, and exposes many problems which are difficult to overcome, while the non-relational database (Not Only SQL, for short, NoSQL) is developed very rapidly due to its own characteristics. The generation of the NoSQL database aims to solve the challenges brought by large-scale data set multiple data types, especially the big data application problem.
In the process of storing data, a non-relational database needs to partition the data and disperse records on a plurality of nodes. The Mongo DB is a non-relational database which is very popular in the IT industry at present, and is taken as an example for illustration, and the MongoDB can realize distributed storage of data by establishing fragments on different servers. Meanwhile, horizontal expansion of data storage capacity can also be achieved by adding fragmentation.
In the current expansion method, after finding that the capacity of a disk is close to the limit, a maintainer manually adds a server, and deploys a new segment of the mongoDB on the new server to realize horizontal expansion. The manual expansion mode needs to manually check the disk state of the original system at regular intervals, manually deploy the mongoDB system on the new server, and establish the shards, so that the defect of high deployment complexity exists.
In view of the above problems, no effective solution has been proposed.
Disclosure of Invention
The embodiment of the application provides a method and a device for expanding a non-relational database, which are used for at least solving the technical problem that the complexity of artificially expanding non-relational data in the prior art is high.
According to an aspect of an embodiment of the present application, there is provided a method for expanding a non-relational database, including: monitoring the data volume of stored data of each first fragment of the non-relational database, wherein the first fragment is a fragment which the non-relational database currently has; comparing the data volume of each first fragment with a preset threshold value; and adding a second fragment to the non-relational database under the condition that the first fragment with the data quantity larger than the preset threshold value exists in comparison.
Further, in the case that there is a first tile whose data amount is greater than the preset threshold compared, adding a second tile to the non-relational database includes: detecting a first disk of a first server, wherein a target fragment is deployed on a second disk of the first server, the target fragment is a first fragment with a data volume larger than the preset threshold, and the first disk is a disk of the first server different from the second disk; creating the second shard on the first disk of the first server upon detecting that the first disk of the first server has remaining space; and under the condition that the first disk of the first server is detected not to have the residual space, searching a second server and creating the second fragment on the disk of the second server, wherein the second server is a server with the residual space in a plurality of servers of the non-relational database, and the second server and the first server are different servers.
Further, creating the second fragment on the disk of the second server comprises: and creating the second shard on a third disk of the second server, wherein the third disk is a disk of the second server where the first shard is not deployed, and the third disk has a remaining space.
Further, the searching for the second server includes: and traversing the plurality of servers, detecting the disk space of the traversed server until a server with a disk having a residual space is detected, and taking the server with the disk having the residual space as the second server.
Further, monitoring the data volume of each first shard of stored data of the non-relational database comprises: executing a monitoring program deployed on each server of the non-relational database; and acquiring a monitoring result of the monitoring program to obtain the data volume of the stored data of each first fragment, wherein the monitoring program acquires the data volume of the stored data of each first fragment on each server through a routing node connected with the non-relational database.
According to another aspect of the embodiments of the present application, there is also provided an expansion apparatus for a non-relational database, including: the monitoring unit is used for monitoring the data volume of the stored data of each first fragment of the non-relational database, wherein the first fragment is a fragment which the non-relational database currently has; the comparison unit is used for comparing the data volume of each first fragment with the size of a preset threshold; and the adding unit is used for adding a second fragment to the non-relational database under the condition that the comparing unit compares that the first fragment with the data quantity larger than the preset threshold exists.
Further, the adding unit includes: a detecting subunit, configured to detect a first disk of a first server when the comparing unit compares that there is a fragment with a data amount greater than the preset threshold, where a target fragment is deployed on a second disk of the first server, the target fragment is a first fragment with a data amount greater than the preset threshold, and the first disk is a disk of the first server that is different from the second disk; a first creating subunit, configured to create the second shard on the first disk of the first server if the detecting subunit detects that the first disk of the first server has remaining space; and a second creating subunit, configured to, when the detecting subunit detects that the first disk of the first server does not have the remaining space, find a second server, and create the second fragment on the disk of the second server, where the second server is a server having the remaining space among the servers of the non-relational database, and the second server and the first server are different servers.
Further, the second creating subunit includes: a creating module, configured to create the second shard on a third disk of the second server, where the third disk is a disk of the second server where the first shard is not deployed, and the third disk has a remaining space.
Further, the second creating subunit further includes: and the traversing module is used for traversing the plurality of servers, detecting the disk space of the traversed server until the server with the disk having the residual space is detected, and taking the server with the disk having the residual space as the second server.
Further, the monitoring unit includes: an execution subunit for executing a monitoring program deployed on each server of the non-relational database; and an obtaining subunit, configured to obtain a monitoring result of the monitoring program, and obtain a data amount of the stored data of each first fragment, where the monitoring program obtains the data amount of the stored data of the first fragment on each server through a routing node connected to the non-relational database.
In the embodiment of the application, the data volume of the data stored in each current existing fragment in the non-relational database is monitored, the data volume of the data stored in each current existing fragment is compared with the threshold, and when the data volume of the data stored in one or more fragments in the current existing fragments is found to be larger than the threshold through comparison, a new fragment is added into the database, so that the expansion method for automatically adding the fragments into the non-relational database is realized, the technical problem of high complexity in artificial expansion of the non-relational data in the prior art is solved, and the technical effect of automatically expanding the non-relational database is achieved.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the application and together with the description serve to explain the application and not to limit the application. In the drawings:
FIG. 1 is a flow chart of a method for expanding a non-relational database according to an embodiment of the present application; and
fig. 2 is a schematic diagram of an expansion apparatus of a non-relational database according to an embodiment of the present application.
Detailed Description
In order to make the technical solutions better understood by those skilled in the art, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only partial embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
It should be noted that the terms "first," "second," and the like in the description and claims of this application and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the application described herein are capable of operation in sequences other than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
In accordance with an embodiment of the present application, there is provided an embodiment of a method for expanding a non-relational database, it being noted that the steps illustrated in the flowchart of the figures may be performed in a computer system such as a set of computer-executable instructions and that, although a logical order is illustrated in the flowchart, in some cases the steps illustrated or described may be performed in an order different than here.
Fig. 1 is a method for expanding a non-relational database according to an embodiment of the present application. As shown in fig. 1, the method comprises the steps of:
step S102, monitoring the data volume of the stored data of each first fragment of the non-relational database, wherein the first fragment is the fragment of the non-relational database at present. That is, the data amount of the data currently stored in each shard in the non-relational database is monitored.
Step S104, comparing the data size of each first segment with a preset threshold, that is, comparing the data size of the data stored in each segment with the threshold, wherein the actual size of the threshold can be flexibly set according to the application requirement.
And step S106, adding a second fragment to the non-relational database under the condition that the first fragment with the data volume larger than the preset threshold value exists in comparison. That is, when the comparison shows that the data amount of the data stored in one or more of the currently existing shards is larger than the threshold value, a new shard is added to the non-relational database.
For example, a certain non-relational database has 10 shards, where the 10 shards are first shards, the data volume of data stored in the 10 shards is monitored, the data volume of data stored in the 10 shards is respectively compared with a threshold, and when the data volume of data stored in 1 shard exceeds the threshold, a new shard is added to the database, and the new shard is a second shard.
According to the expanding method of the non-relational database provided by the embodiment of the application, the data volume of the data stored in each current existing fragment in the non-relational database is monitored, the data volume of the data stored in each current existing fragment is compared with the threshold value, and when the data volume of the data stored in one or more fragments in the current existing fragments is found to be larger than the threshold value through comparison, a new fragment is added into the database, so that the expanding method for automatically adding the fragments into the non-relational database is realized, the technical problem that the complexity of artificially expanding the non-relational data in the prior art is high is solved, and the technical effect of automatically expanding the non-relational database is achieved.
Optionally, in a case where there is a first tile whose data amount is greater than a preset threshold compared, adding the second tile to the non-relational database includes: detecting a first disk of a first server, wherein a target fragment is deployed on a second disk of the first server, the target fragment is a first fragment with a data volume larger than a preset threshold, and the first disk is a disk of the first server different from the second disk; under the condition that the first disk of the first server is detected to have the residual space, creating a second fragment on the first disk of the first server; and in the case that the first disk of the first server is detected not to have the remaining space, searching a second server and creating a second fragment on the disk of the second server, wherein the second server is a server having the remaining space in the plurality of servers of the non-relational database, and the second server and the first server are different servers.
That is, if the data amount of data stored by a slice exceeds a threshold, then the slice is the target slice. The disk where the target fragment is deployed is a second disk, the server where the second disk is located is a first server, and disks other than the second disk on the first server are the first disk. Of the servers in the non-relational database, the non-first server having the remaining space is the second server. That is to say, the second disk is a disk on which the target fragment is deployed, and the first disk is a disk on which the target fragment is not deployed on the same server. The first server is a server where the disk with the target fragment is located, and the second server is a server which has residual space in the same non-relational database and does not contain the disk with the target fragment.
When a certain disk is found to be deployed with the target fragment through comparison, the disks which are not deployed with the target fragment on the server where the disk is located are detected, if the disk which is not deployed with the target fragment is detected to have the residual space, a new fragment is created on the disk, if the disk which is not deployed with the target fragment on the server is detected to have no residual space, a second server is searched, and the searched second server meets two characteristics: one is to not contain the disk on which the target fragment is deployed, and the other is to have the remaining space. After the second server is found, a new shard is created on the disk of the second server.
For example, there are 7 servers in a non-relational database, server 1, server 2, server 3, server 4, server 5, server 6, and server 7. There are 5 disks on each server. The 5 disks on the server i are respectively Di1, Di2, Di3, Di4 and Di 5.
On the server 1, 2 shards, i.e., shard P111 and shard P112, are deployed on the disk D11. On the disk D12, 2 shards are deployed, i.e., shard P121 and shard P122. No shards are deployed on disk D13, disk D14, and disk D15.
On the server 2, 1 shard is deployed on the disk D21, and is the shard P211. The disk D22 has 4 fragments, which are fragment P221, fragment P222, fragment P223, and fragment P224. The disk D23 has 3 fragments, which are fragment P231, fragment P232, and fragment P233. Disk D24 and disk D25 did not have any shards deployed.
On the server 3, a shard, shard P351, is deployed on the disk D35. No shards are deployed on disk D31, disk D32, disk D33, disk D34.
None of the shards are deployed on the disks of server 4, server 5, server 6, and server 7.
Slice P111, slice P112, slice P121, slice P122, slice P211, slice P221, slice P222, slice P223, slice P224, slice P231, slice P232, slice P233, and slice P351 are all first slices. The data volume of the data stored in the partition P112 is greater than the threshold, that is, the partition P112 is the target partition. The disk on which the target shard is deployed is the second disk, i.e., disk D11 is the second disk. Disk D12, disk D13, disk D14, and disk D15 are the first disks. And detecting that the disk D13 has the remaining space, and deploying the new fragment on the disk D13, wherein the newly deployed fragment is the second fragment.
For another example, there are 7 servers in a non-relational database, server 1, server 2, server 3, server 4, server 5, server 6, and server 7. There are 5 disks on each server. The 5 disks on the server i are respectively Di1, Di2, Di3, Di4 and Di 5.
On the server 1, 2 shards, i.e., shard P111 and shard P112, are deployed on the disk D11. On the disk D12, 2 shards are deployed, i.e., shard P121 and shard P122. The disk D13 has 1 fragment, which is the fragment D131, the disk D14 has 1 fragment, which is the fragment D141, and the disk D15 has 1 fragment, which is the fragment D151.
On the server 2, 1 shard is deployed on the disk D21, and is the shard P211. Disk D22, disk D23, disk D24, and disk D25 did not have any shards deployed.
None of the disks of server 3, server 4, server 5, server 6, and server 7 deploy any shards.
Slice P111, slice P112, slice P121, slice P122, slice P131, slice P141, slice P151, and slice P211 are all first slices. The data volume of the data stored in the partition P112 is greater than the threshold, that is, the partition P112 is the target partition. The disk on which the target shard is deployed is the second disk, i.e., disk D11 is the second disk. Server 1 is the first server. Disk D12, disk D13, disk D14, and disk D15 are the first disks. The test finds that none of the disk D12, the disk D13, the disk D14, and the disk D15 have the remaining space. Next, searching for the server 2, the server 3, the server 4, the server 5, the server 6, and the server 7, and finding that the server 2, the server 3, the server 4, the server 5, the server 6, and the server 7 all have the remaining space, all of these 6 servers are the second servers. A new fragment is added to the disk of one of the 6 servers, for example, the new fragment may be added to the disk D21 of the server 2, and the newly added fragment is the second fragment.
Optionally, creating the second fragment on the disk of the second server comprises: and creating a second fragment on a third disk of the second server, wherein the third disk is a disk of the second server where the first fragment is not deployed, and the third disk has a remaining space.
That is, after the second server is found, the disk on which the fragment is not deployed, that is, the third disk, is found, and after the third disk is found, a new fragment is deployed on the disk.
For example, a non-relational database has 3 servers, server 1, server 2, and server 3. There are 5 disks on each server. The 5 disks on the server i are respectively Di1, Di2, Di3, Di4 and Di 5.
On the server 1, 2 shards, i.e., shard P111 and shard P112, are deployed on the disk D11. 2 shards are deployed on the disk D12, namely shard P121 and shard P122. The disk D13 has 1 fragment, which is the fragment D131, the disk D14 has 1 fragment, which is the fragment D141, and the disk D15 has 1 fragment, which is the fragment D151.
On the server 2, 1 shard is deployed on the disk D21, and is the shard P211. Disk D22, disk D23, disk D24, and disk D25 did not have any shards deployed.
On the server 3, 2 shards, namely a shard P341 and a shard P342, are deployed on the disk D34. Disk D31, disk D32, disk D33, and disk D35 are not deployed with any shards.
Slice P111, slice P112, slice P121, slice P122, slice P131, slice P141, slice P151, slice P211, slice P341, slice P342 are all first slices. The data volume of the data stored in the partition P112 is greater than the threshold, that is, the partition P112 is the target partition. The disk on which the target shard is deployed is the second disk, i.e., disk D11 is the second disk. Server 1 is the first server. Disk D12, disk D13, disk D14, and disk D15 are the first disks. The test finds that none of the disk D12, the disk D13, the disk D14, and the disk D15 have the remaining space. Searching server 2 and server 3. It is found that both servers 2, 3 have space remaining, and both are then the second server.
On server 2, no first shard is deployed on disk D22, disk D23, disk D24, disk D25, so these 4 disks are the third disk. On the server 3, none of the disk D31, disk D32, disk D33, disk D35 deployed any first shard, so these 4 disks are also third disks. The new shard may be deployed on any one of 8 third disks, namely disk D22, disk D23, disk D24, disk D25, disk D31, disk D32, disk D33, and disk D35.
If a new fragment is created on the disk where the first fragment has been originally deployed, it is easy to cause the space of the disk to quickly reach the disk limit in a short time, so that both the first fragment on the disk and the second fragment newly deployed cannot be used quickly, and a new fragment needs to be created again.
Optionally, the searching for the second server comprises: traversing a plurality of servers, detecting the disk space of the traversed server until detecting the server with the disk having the residual space, stopping the continuous traversal of the server, and taking the detected server with the disk having the residual space as a second server. That is, in the process of traversing a plurality of servers, every time a server is traversed, the disk of the currently traversed server is detected, and if it is detected that the disk of the currently traversed server has a remaining space, the traversal is stopped, and the currently traversed server is determined to be the second server.
Wherein, the plurality of servers are traversed according to the sequence, and the traversing sequence can be specified in advance. And each time the server is traversed, detecting the disk of the server, traversing the next server if the disk of the server is detected to have no residual space, and if the disk of the server is detected to have the residual space, determining that the server is the second server and stopping traversing the residual servers.
For example, there are 7 servers in a non-relational database, server 1, server 2, server 3, server 4, server 5, server 6, and server 7. There are 5 disks on each server. The 5 disks on the server i are respectively Di1, Di2, Di3, Di4 and Di 5.
It is detected that the data amount of the data stored in the shard P112 is larger than the threshold, wherein the shard P112 is disposed on the disk D11 of the server 1. Slice P112 is the target slice and disk D11 is the second disk. The first disk in the server 1, i.e. disk D12, disk D13, disk D14, disk D15, is checked and none of the 4 disks is found to have any remaining space. The remaining servers are then traversed. The order of traversing the remaining servers is varied and can be pre-specified. For example, it may be specified to traverse in the order of server 2, server 3, server 4, server 5, server 6, and server 7, it may be specified to traverse in the order of server 7, server 6, server 5, server 4, server 3, and server 2, and so on. Assuming that traversal in the order of server 2, server 3, server 4, server 5, server 6, server 7 is specified, when traversing server 2, it is detected whether the disk on server 2 has remaining space, if it is detected that the disk on server 2 has remaining space, server 2 is the second server, traversal of server 3, server 4, server 5, server 6, server 7 is stopped, if it is detected that the disk on server 2 does not have remaining space, traversal of server 3 is continued, it is detected whether the disk on server 3 has remaining space, if it is detected that the disk on server 3 has remaining space, server 3 is the second server, traversal of server 4, server 5, server 6, server 7 is stopped, if it is detected that the disk on server 3 does not have remaining space, the traversal of server 4 continues, and so on, until a second server is found. After the second server is found, the traversal of the remaining servers is stopped.
Optionally, monitoring the data volume of the stored data of each first partition of the non-relational database comprises: executing a monitoring program deployed on each server of the non-relational database; and acquiring a monitoring result of the monitoring program to obtain the data volume of the stored data of each first fragment, wherein the monitoring program acquires the data volume of the stored data of the first fragment on each server through a routing node connected with the non-relational database.
That is, each server of the non-relational database is deployed with a monitoring program, the monitoring program obtains the data volume of the data stored in the existing fragment of each server through the routing node, executes the monitoring program, and obtains a monitoring result, where the monitoring result includes the data volume of the data stored in each existing fragment.
According to the embodiment of the application, the device for expanding the non-relational database is further provided. The expanding means of the non-relational database may execute the expanding means of the non-relational database, and the expanding means of the non-relational database may be implemented by the expanding means of the non-relational database.
Fig. 2 is a schematic diagram of an expanding apparatus of a non-relational database according to an embodiment of the application. As shown in fig. 2, the expanding means of the non-relational database includes: a monitoring unit 22, a comparing unit 24, an adding unit 26.
The monitoring unit 22 is configured to monitor a data amount of stored data of each first partition of the non-relational database, where the first partition is a partition that the non-relational database currently has. That is, the data amount of the data currently stored in each shard in the non-relational database is monitored.
And the comparison unit 24 is used for comparing the data volume of each first fragment with the size of a preset threshold value. That is, the data amount of the data stored in each slice currently existing is compared with the threshold, wherein the actual size of the threshold can be flexibly set according to the application requirements.
An adding unit 26, configured to add a second fragment to the non-relational database if the comparing unit 24 compares that there is a first fragment whose data amount is greater than a preset threshold. That is, when the comparison shows that the data amount of the data stored in one or more of the currently existing shards is larger than the threshold value, a new shard is added to the non-relational database.
For example, a certain non-relational database has 10 shards, where the 10 shards are first shards, the data volume of data stored in the 10 shards is monitored, the data volume of data stored in the 10 shards is respectively compared with a threshold, and when the data volume of data stored in 1 shard exceeds the threshold, a new shard is added to the database, and the new shard is a second shard.
According to the expanding device for the non-relational database provided by the embodiment of the application, the data volume of data stored in each current existing fragment in the non-relational database is monitored through the monitoring unit 22, the comparing unit 24 compares the data volume of the data stored in each current existing fragment with the threshold value, and when the data volume of the data stored in a certain fragment or a plurality of fragments in the current existing fragments is found to be larger than the threshold value through comparison, the adding unit 26 adds a new fragment in the database, so that the expanding method for automatically adding the fragments to the non-relational database is realized, the technical problem that the complexity of artificially expanding the non-relational database in the prior art is high is solved, and the technical effect of automatically expanding the non-relational database is achieved.
Optionally, the adding unit includes a detecting subunit, a first creating subunit, and a second creating subunit. The detecting subunit is configured to detect the first disk of the first server when the comparing unit 24 compares that there is a first fragment with a data amount greater than a preset threshold, where the target fragment is deployed on the second disk of the first server, the target fragment is the first fragment with the data amount greater than the preset threshold, and the first disk is a disk of the first server that is different from the second disk. The first creating subunit is configured to create the second fragment on the first disk of the first server if the detecting subunit detects that the first disk of the first server has the remaining space. The second creating subunit is configured to, in a case where the detecting subunit detects that the first disk of the first server does not have the remaining space, find a second server and create a second fragment on the disk of the second server, where the second server is a server having the remaining space among the servers of the non-relational database, and the second server and the first server are different servers.
That is, if the data amount of data stored by a slice exceeds a threshold, then the slice is the target slice. The disk where the target fragment is deployed is a second disk, the server where the second disk is located is a first server, and disks other than the second disk on the first server are the first disk. Of the servers in the non-relational database, the non-first server having the remaining space is the second server. That is to say, the second disk is a disk on which the target fragment is deployed, and the first disk is a disk on which the target fragment is not deployed on the same server. The first server is a server where the disk with the target fragment is located, and the second server is a server which has residual space in the same non-relational database and does not contain the disk with the target fragment.
When a certain disk is found to be deployed with the target fragment through comparison, the disks which are not deployed with the target fragment on the server where the disk is located are detected, if the disk which is not deployed with the target fragment is detected to have the residual space, a new fragment is created on the disk, if the disk which is not deployed with the target fragment on the server is detected to have no residual space, a second server is searched, and the searched second server meets two characteristics: one is to not contain the disk on which the target fragment is deployed, and the other is to have the remaining space. After the second server is found, a new shard is created on the disk of the second server.
For example, there are 7 servers in a non-relational database, server 1, server 2, server 3, server 4, server 5, server 6, and server 7. There are 5 disks on each server. The 5 disks on the server i are respectively Di1, Di2, Di3, Di4 and Di 5.
On the server 1, 2 shards, i.e., shard P111 and shard P112, are deployed on the disk D11. On the disk D12, 2 shards are deployed, i.e., shard P121 and shard P122. No shards are deployed on disk D13, disk D14, and disk D15.
On the server 2, 1 shard is deployed on the disk D21, and is the shard P211. The disk D22 has 4 fragments, which are fragment P221, fragment P222, fragment P223, and fragment P224. The disk D23 has 3 fragments, which are fragment P231, fragment P232, and fragment P233. Disk D24 and disk D25 did not have any shards deployed.
On the server 3, a shard, shard P351, is deployed on the disk D35. No shards are deployed on disk D31, disk D32, disk D33, disk D34.
None of the shards are deployed on the disks of server 4, server 5, server 6, and server 7.
Slice P111, slice P112, slice P121, slice P122, slice P211, slice P221, slice P222, slice P223, slice P224, slice P231, slice P232, slice P233, and slice P351 are all first slices. The data volume of the data stored in the partition P112 is greater than the threshold, that is, the partition P112 is the target partition. The disk on which the target shard is deployed is the second disk, i.e., disk D11 is the second disk. Disk D12, disk D13, disk D14, and disk D15 are the first disks. And detecting that the disk D13 has the remaining space, and deploying the new fragment on the disk D13, wherein the newly deployed fragment is the second fragment.
For another example, there are 7 servers in a non-relational database, server 1, server 2, server 3, server 4, server 5, server 6, and server 7. There are 5 disks on each server. The 5 disks on the server i are respectively Di1, Di2, Di3, Di4 and Di 5.
On the server 1, 2 shards, i.e., shard P111 and shard P112, are deployed on the disk D11. On the disk D12, 2 shards are deployed, i.e., shard P121 and shard P122. The disk D13 has 1 fragment, which is the fragment D131, the disk D14 has 1 fragment, which is the fragment D141, and the disk D15 has 1 fragment, which is the fragment D151.
On the server 2, 1 shard is deployed on the disk D21, and is the shard P211. Disk D22, disk D23, disk D24, and disk D25 did not have any shards deployed.
None of the disks of server 3, server 4, server 5, server 6, and server 7 deploy any shards.
Slice P111, slice P112, slice P121, slice P122, slice P131, slice P141, slice P151, and slice P211 are all first slices. The data volume of the data stored in the partition P112 is greater than the threshold, that is, the partition P112 is the target partition. The disk on which the target shard is deployed is the second disk, i.e., disk D11 is the second disk. Server 1 is the first server. Disk D12, disk D13, disk D14, and disk D15 are the first disks. The test finds that none of the disk D12, the disk D13, the disk D14, and the disk D15 have the remaining space. Next, searching for the server 2, the server 3, the server 4, the server 5, the server 6, and the server 7, and finding that the server 2, the server 3, the server 4, the server 5, the server 6, and the server 7 all have the remaining space, all of these 6 servers are the second servers. A new fragment is added to the disk of one of the 6 servers, for example, the new fragment may be added to the disk D21 of the server 2, and the newly added fragment is the second fragment.
Optionally, the second creating sub-unit comprises a creating module. The creating module is configured to create a second shard on a third disk of the second server, where the third disk is a disk of the second server where the first shard is not deployed, and the third disk has a remaining space.
That is, after the second server is found, the disk on which the fragment is not deployed, that is, the third disk, is found, and after the third disk is found, a new fragment is deployed on the disk.
For example, a non-relational database has 3 servers, server 1, server 2, and server 3. There are 5 disks on each server. The 5 disks on the server i are respectively Di1, Di2, Di3, Di4 and Di 5.
On the server 1, 2 shards, i.e., shard P111 and shard P112, are deployed on the disk D11. 2 shards are deployed on the disk D12, namely shard P121 and shard P122. The disk D13 has 1 fragment, which is the fragment D131, the disk D14 has 1 fragment, which is the fragment D141, and the disk D15 has 1 fragment, which is the fragment D151.
On the server 2, 1 shard is deployed on the disk D21, and is the shard P211. Disk D22, disk D23, disk D24, and disk D25 did not have any shards deployed.
On the server 3, 2 shards, namely a shard P341 and a shard P342, are deployed on the disk D34. Disk D31, disk D32, disk D33, and disk D35 are not deployed with any shards.
Slice P111, slice P112, slice P121, slice P122, slice P131, slice P141, slice P151, slice P211, slice P341, slice P342 are all first slices. The data volume of the data stored in the partition P112 is greater than the threshold, that is, the partition P112 is the target partition. The disk on which the target shard is deployed is the second disk, i.e., disk D11 is the second disk. Server 1 is the first server. Disk D12, disk D13, disk D14, and disk D15 are the first disks. The test finds that none of the disk D12, the disk D13, the disk D14, and the disk D15 have the remaining space. Searching server 2 and server 3. It is found that both servers 2, 3 have space remaining, and both are then the second server.
On server 2, no first shard is deployed on disk D22, disk D23, disk D24, disk D25, so these 4 disks are the third disk. On the server 3, none of the disk D31, disk D32, disk D33, disk D35 deployed any first shard, so these 4 disks are also third disks. The new shard may be deployed on any one of 8 third disks, namely disk D22, disk D23, disk D24, disk D25, disk D31, disk D32, disk D33, and disk D35.
If a new fragment is created on the disk where the first fragment has been originally deployed, it is easy to cause the space of the disk to quickly reach the disk limit in a short time, so that both the first fragment on the disk and the second fragment newly deployed cannot be used quickly, and a new fragment needs to be created again.
Optionally, the second creating subunit further comprises a traversing module. The traversal module is used for traversing a plurality of servers, detecting the disk space of the traversed server until detecting the server with the disk having the residual space, stopping the continuous traversal of the server, and taking the detected server with the disk having the residual space as a second server. That is, in the process of traversing a plurality of servers, every time a server is traversed, the disk of the currently traversed server is detected, and if it is detected that the disk of the currently traversed server has a remaining space, the traversal is stopped, and the currently traversed server is determined to be the second server.
Wherein, the plurality of servers are traversed according to the sequence, and the traversing sequence can be specified in advance. And each time the server is traversed, detecting the disk of the server, traversing the next server if the disk of the server is detected to have no residual space, and if the disk of the server is detected to have the residual space, determining that the server is the second server and stopping traversing the residual servers.
For example, there are 7 servers in a non-relational database, server 1, server 2, server 3, server 4, server 5, server 6, and server 7. There are 5 disks on each server. The 5 disks on the server i are respectively Di1, Di2, Di3, Di4 and Di 5.
It is detected that the data amount of the data stored in the shard P112 is larger than the threshold, wherein the shard P112 is disposed on the disk D11 of the server 1. Slice P112 is the target slice and disk D11 is the second disk. The first disk in the server 1, i.e. disk D12, disk D13, disk D14, disk D15, is checked and none of the 4 disks is found to have any remaining space. The remaining servers are then traversed. The order of traversing the remaining servers is varied and can be pre-specified. For example, it may be specified to traverse in the order of server 2, server 3, server 4, server 5, server 6, and server 7, it may be specified to traverse in the order of server 7, server 6, server 5, server 4, server 3, and server 2, and so on. Assuming that traversal in the order of server 2, server 3, server 4, server 5, server 6, server 7 is specified, when traversing server 2, it is detected whether the disk on server 2 has remaining space, if it is detected that the disk on server 2 has remaining space, server 2 is the second server, traversal of server 3, server 4, server 5, server 6, server 7 is stopped, if it is detected that the disk on server 2 does not have remaining space, traversal of server 3 is continued, it is detected whether the disk on server 3 has remaining space, if it is detected that the disk on server 3 has remaining space, server 3 is the second server, traversal of server 4, server 5, server 6, server 7 is stopped, if it is detected that the disk on server 3 does not have remaining space, the traversal of server 4 continues, and so on, until a second server is found. After the second server is found, the traversal of the remaining servers is stopped.
Optionally, the monitoring unit includes an execution subunit and an acquisition subunit. The execution subunit is configured to execute a monitoring program deployed on each server of the non-relational database. The acquisition subunit is configured to acquire a monitoring result of the monitoring program, and acquire a data amount of the stored data of each first segment, where the monitoring program acquires the data amount of the stored data of the first segment on each server through a routing node connected to the non-relational database.
That is, each server of the non-relational database is deployed with a monitoring program, the monitoring program obtains the data volume of the data stored in the existing fragment of each server through the routing node, executes the monitoring program, and obtains a monitoring result, where the monitoring result includes the data volume of the data stored in each existing fragment.
The above-mentioned serial numbers of the embodiments of the present application are merely for description and do not represent the merits of the embodiments.
In the above embodiments of the present application, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.
In the embodiments provided in the present application, it should be understood that the disclosed technology can be implemented in other ways. The above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units may be a logical division, and in actual implementation, there may be another division, for example, multiple units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, units or modules, and may be in an electrical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be substantially implemented or contributed to by the prior art, or all or part of the technical solution may be embodied in a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a removable hard disk, a magnetic or optical disk, and other various media capable of storing program codes.
The foregoing is only a preferred embodiment of the present application and it should be noted that those skilled in the art can make several improvements and modifications without departing from the principle of the present application, and these improvements and modifications should also be considered as the protection scope of the present application.
Claims (6)
1. A method for expanding a non-relational database, comprising:
monitoring the data volume of stored data of each first fragment of the non-relational database, wherein the first fragment is a fragment which the non-relational database currently has;
comparing the data volume of each first fragment with a preset threshold value; and
adding a second fragment to the non-relational database under the condition that the first fragment with the data volume larger than the preset threshold value exists in comparison;
wherein, in the case that there is a first shard whose data amount is greater than the preset threshold value compared, adding a second shard to the non-relational database includes:
detecting a first disk of a first server, wherein a target fragment is deployed on a second disk of the first server, the target fragment is a first fragment with a data volume larger than the preset threshold, and the first disk is a disk of the first server different from the second disk;
creating the second shard on the first disk of the first server upon detecting that the first disk of the first server has remaining space; and
under the condition that the first disk of the first server is detected not to have the remaining space, searching a second server and creating the second fragment on the disk of the second server, wherein the second server is a server with the remaining space in a plurality of servers of the non-relational database, and the second server and the first server are different servers;
wherein creating the second fragment on the disk of the second server comprises:
and creating the second shard on a third disk of the second server, wherein the third disk is a disk of the second server where the first shard is not deployed, and the third disk has a remaining space.
2. The method of claim 1, wherein locating a second server comprises:
and traversing the plurality of servers, detecting the disk space of the traversed server until a server with a disk having a residual space is detected, and taking the server with the disk having the residual space as the second server.
3. The method of claim 1 or 2, wherein monitoring the data volume of each first shard of stored data of the non-relational database comprises:
executing a monitoring program deployed on each server of the non-relational database; and
and acquiring a monitoring result of the monitoring program to obtain the data volume of the stored data of each first fragment, wherein the monitoring program acquires the data volume of the stored data of each first fragment on each server through a routing node connected with the non-relational database.
4. An expansion device for a non-relational database, comprising:
the monitoring unit is used for monitoring the data volume of the stored data of each first fragment of the non-relational database, wherein the first fragment is a fragment which the non-relational database currently has;
the comparison unit is used for comparing the data volume of each first fragment with the size of a preset threshold; and
the adding unit is used for adding a second fragment to the non-relational database under the condition that the comparing unit compares that the first fragment with the data quantity larger than the preset threshold exists;
wherein, in the case that there is a first shard whose data amount is greater than the preset threshold value compared, adding a second shard to the non-relational database includes:
detecting a first disk of a first server, wherein a target fragment is deployed on a second disk of the first server, the target fragment is a first fragment with a data volume larger than the preset threshold, and the first disk is a disk of the first server different from the second disk;
creating the second shard on the first disk of the first server upon detecting that the first disk of the first server has remaining space; and
under the condition that the first disk of the first server is detected not to have the remaining space, searching a second server and creating the second fragment on the disk of the second server, wherein the second server is a server with the remaining space in a plurality of servers of the non-relational database, and the second server and the first server are different servers;
wherein the second creating subunit includes:
a creating module, configured to create the second shard on a third disk of the second server, where the third disk is a disk of the second server where the first shard is not deployed, and the third disk has a remaining space.
5. The apparatus of claim 4, wherein the second creating subunit further comprises:
and the traversing module is used for traversing the plurality of servers, detecting the disk space of the traversed server until the server with the disk having the residual space is detected, and taking the server with the disk having the residual space as the second server.
6. The apparatus of claim 4 or 5, wherein the monitoring unit comprises:
an execution subunit for executing a monitoring program deployed on each server of the non-relational database; and
and the acquisition subunit is configured to acquire a monitoring result of the monitoring program, and acquire a data amount of the stored data of each first fragment, where the monitoring program acquires the data amount of the stored data of the first fragment on each server through a routing node connected to the non-relational database.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510596035.0A CN106547471B (en) | 2015-09-17 | 2015-09-17 | Method and device for expanding non-relational database |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510596035.0A CN106547471B (en) | 2015-09-17 | 2015-09-17 | Method and device for expanding non-relational database |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106547471A CN106547471A (en) | 2017-03-29 |
| CN106547471B true CN106547471B (en) | 2020-03-03 |
Family
ID=58362895
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510596035.0A Active CN106547471B (en) | 2015-09-17 | 2015-09-17 | Method and device for expanding non-relational database |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106547471B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109828960B (en) * | 2018-12-14 | 2024-05-28 | 平安科技（深圳）有限公司 | Log library capacity expansion method, system, computer device and readable storage medium |
| CN113521745B (en) * | 2021-06-17 | 2024-01-09 | 广州三七极耀网络科技有限公司 | Data storage method, device and equipment of AI model training architecture of FPS game |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2014107901A1 (en) * | 2013-01-14 | 2014-07-17 | 华为技术有限公司 | Data storage method, database storage node failure processing method and apparatus |
| CN103136336B (en) * | 2013-01-31 | 2015-09-16 | 中国农业银行股份有限公司 | The integrated system and method for a kind of mass data |
| CN104123388A (en) * | 2014-08-07 | 2014-10-29 | 武汉大学 | Massive-sensing-network-data-oriented high-concurrency real-time access system and method |
| CN104615785A (en) * | 2015-03-04 | 2015-05-13 | 浪潮集团有限公司 | Data storage method and device based on TYKY cNosql |
-
2015
- 2015-09-17 CN CN201510596035.0A patent/CN106547471B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN106547471A (en) | 2017-03-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104978253B (en) | A kind of method and apparatus of file destination verification | |
| CN106649362B (en) | Webpage crawling method and device | |
| CN108470028B (en) | Picture auditing method and device | |
| CN106326222B (en) | A kind of data processing method and device | |
| CN106570025B (en) | Data filtering method and device | |
| CN110083651B (en) | Data loading method and device | |
| CN106547471B (en) | Method and device for expanding non-relational database | |
| CN110389940B (en) | Data equalization method and device and computer readable storage medium | |
| JP6855428B2 (en) | Methods, devices, and computer devices that scan the information to be scanned. | |
| CN109379396B (en) | Method, medium, device and block chain system for preventing hard branching phenomenon in block chain system | |
| CN107391303B (en) | Data processing method, device, system, server and computer storage medium | |
| US9917697B2 (en) | Performing incremental upgrade on APK base file corresponding to APK eigenvalue value | |
| CN114528350A (en) | Cluster split brain processing method, device and equipment and readable storage medium | |
| CN104778252A (en) | Index storage method and index storage device | |
| CN104735041A (en) | Method and device for having access to offline resources | |
| CN109120579B (en) | Malicious domain name detection method and device and computer readable storage medium | |
| JP6294145B2 (en) | Monitoring method, monitoring device and monitoring control program | |
| CN103399890B (en) | At the method and apparatus that input method client collects words | |
| CN103152381B (en) | A kind of method that browser crash data is processed and server system | |
| CN106789889B (en) | Network monitoring method and device based on mesh structure | |
| JP2017530477A (en) | System and method for processing graphs | |
| CN111324504A (en) | Method and device for monitoring running state of power communication network | |
| CN114911515A (en) | Configuration management method, device, equipment and medium | |
| CN113806204B (en) | Method, device, system and storage medium for evaluating message segment correlation | |
| AU2016372750A1 (en) | Verification method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: 100083 No. 401, 4th Floor, Haitai Building, 229 North Fourth Ring Road, Haidian District, Beijing Applicant after: Beijing Guoshuang Technology Co.,Ltd. Address before: 100086 Cuigong Hotel, 76 Zhichun Road, Shuangyushu District, Haidian District, Beijing Applicant before: Beijing Guoshuang Technology Co.,Ltd. |
|
| CB02 | Change of applicant information | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |