CN106682199B

CN106682199B - Method and device for realizing automatic expansion of Mongos cluster

Info

Publication number: CN106682199B
Application number: CN201611250080.1A
Authority: CN
Inventors: 李丹
Original assignee: Beijing Qihoo Technology Co Ltd
Current assignee: Beijing Qihoo Technology Co Ltd
Priority date: 2016-12-29
Filing date: 2016-12-29
Publication date: 2020-09-01
Anticipated expiration: 2036-12-29
Also published as: CN106682199A

Abstract

The invention discloses a method and a device for realizing Mongos cluster automatic expansion, wherein the method comprises the following steps: respectively deploying clients for initializing Mongos cluster fragments on each server in the database system; receiving service capacity expansion demand information needing to use more Mongos cluster resources, and generating a Mongos cluster capacity expansion task according to the service capacity expansion demand information; selecting a certain number of available servers from a database system, and distributing the tasks to clients initializing Mongos cluster shards on the selected servers; and the client side for initializing the Mongos cluster fragments initializes new fragments on the server according to the received Mongos cluster capacity expansion task, deletes non-system libraries in the new fragments, and adds the new fragments into the Mongos cluster. By the scheme, the capacity expansion of the database can be automatically completed, the capacity expansion efficiency of the database is improved, and the running stability of the database system is ensured.

Description

Method and device for realizing automatic expansion of Mongos cluster

Technical Field

The invention relates to the technical field of computers, in particular to a method and a device for realizing Mongos cluster automatic expansion.

Background

With the increase of the data storage, the database for storing data needs to be continuously expanded. Therefore, when the capacity expansion/reduction of each database system is performed manually, the efficiency is low, the labor cost is high, the database resources are not fully utilized, and the stability of the overall operation of the database system is also affected. Therefore, there is a need for a method or apparatus for automatically expanding a database to overcome the above problems.

Disclosure of Invention

In view of the above, the present invention has been made to provide a method and a corresponding apparatus for implementing automated montos clustering that overcome or at least partially solve the above problems.

According to one aspect of the present invention, there is provided a method for implementing automated montos cluster expansion, including:

respectively deploying clients for initializing Mongos cluster fragments on each server in the database system;

receiving service capacity expansion demand information needing to use more Mongos cluster resources, and generating a Mongos cluster capacity expansion task according to the service capacity expansion demand information;

selecting a certain number of available servers from a database system, and distributing the Mongos cluster expansion tasks to clients initializing Mongos cluster fragmentation on the selected servers;

the client side for initializing the Mongos cluster fragments initializes new fragments on the server according to the received Mongos cluster capacity expansion task, deletes non-system libraries in the new fragments, and adds the new fragments into the Mongos cluster.

Optionally, the receiving the service capacity expansion requirement information that needs to use more Mongos cluster resources includes:

and providing an interface for submitting the service requirement information on a specified cloud platform, and receiving the service capacity expansion requirement information needing to use more Mongos cluster resources through the interface.

Optionally, the montos cluster expansion task includes: the number of required servers and relevant parameters for deploying Mongos cluster shards;

the selecting an available number of servers from the database system comprises: a corresponding number of available servers is selected from the database system based on the number of required servers.

Optionally, the generating a Mongos cluster expansion task according to the service expansion demand information includes:

determining the number of required servers and role information of databases to be deployed on the determined number of servers in the fragments according to the service capacity expansion demand information;

and generating the determined number of Mongos cluster expansion tasks respectively corresponding to the role information according to the role information of the determined number of databases to be deployed on each server in the fragments.

Optionally, the selecting an available certain number of servers in the database system, and distributing the montos cluster expansion task to the client initializing the montos cluster sharding on each selected server includes:

and selecting a server with corresponding data volume from the database system according to the determined required server data volume, and randomly distributing the determined number of Mongos cluster capacity expansion tasks respectively corresponding to the role information to the clients initializing the Mongos cluster shards on the selected servers with corresponding number one by one.

Optionally, the method further comprises:

and starting the data migration operation of the Mongos cluster, and migrating data to the added new fragment.

Optionally, the method further comprises:

receiving initialization result information returned by the client side which initializes the Mongos cluster fragments on each selected server;

and sending the received initialization result information to a service party submitting the service capacity expansion demand information through a specified communication channel.

Optionally, the method further comprises:

respectively deploying fragmented offline clients of the Mongos cluster on each server in the database system;

the fragment offline client determines a database of the fragments needing offline on a server where the client is located, and performs offline processing on the determined database of the fragments needing offline.

Optionally, the determining, by the segment offline client, a database of the segment that needs to be offline on the server includes:

the service capacity expansion demand information comprises the expiration time of capacity expansion fragments and the Mongos cluster capacity expansion task also comprises the expiration time of the capacity expansion fragments; the fragment offline client scans resource information on a server where the fragment offline client is located to obtain expiration time of capacity expansion fragments, when the expiration time of the capacity expansion fragments reaches, the corresponding fragments subjected to capacity expansion deployment are determined to be fragments needing offline, and a database contained in the fragments needing offline is a database needing offline;

alternatively, the first and second electrodes may be,

the fragment offline client acquires access information of fragments deployed on a server where the fragment offline client is located, the fragments with the access quantity lower than a preset value in a latest period of time are determined as the fragments needing offline, and a database contained in the fragments needing offline is a database needing offline.

Optionally, the method further comprises:

respectively deploying a fragmentation migration client of a Mongos cluster on each server of the database system;

receiving information that the designated fragments are no longer available, and generating a fragment newly-built task and a fragment migration task according to the information;

selecting a certain number of available servers from a database system, distributing the fragmentation newly-built task to clients initializing Mongos cluster fragmentation on each selected server, and distributing the fragmentation migration task to fragmentation migration clients on each selected server;

the client side for initializing the Mongos cluster fragments initializes new fragments on the server according to the received fragment newly-built task, deletes non-system libraries in the new fragments, and adds the new fragments into the Mongos cluster; and migrating the data in the unavailable designated fragment to a new fragment by the fragment migration client.

Optionally, the method further comprises:

and when the data in the unavailable designated fragments are migrated, deleting the unavailable designated fragments from the Mongos cluster.

According to another aspect of the present invention, there is provided an apparatus for implementing automated montos cluster expansion, including:

the client deployment unit is suitable for respectively deploying clients initializing the Mongos cluster fragments on each server in the database system;

the demand receiving unit is suitable for receiving service capacity expansion demand information needing to use more Mongos cluster resources;

the capacity expansion processing unit is suitable for generating a Mongos cluster capacity expansion task according to the service capacity expansion demand information; selecting a certain number of available servers from a database system, and distributing the Mongos cluster expansion tasks to clients initializing Mongos cluster fragmentation on the selected servers;

Optionally, the requirement receiving unit is adapted to provide an interface for submitting the service requirement information on a specified cloud platform, and receive the service capacity expansion requirement information that needs to use more Mongos cluster resources through the interface.

and the capacity expansion processing unit is suitable for selecting a corresponding number of available servers from the database system according to the number of the required servers.

Optionally, the capacity expansion processing unit is adapted to determine, according to the service capacity expansion requirement information, the number of servers needed and role information of the databases to be deployed on the determined number of servers in the shards; and generating the determined number of Mongos cluster expansion tasks respectively corresponding to the role information according to the role information of the determined number of databases to be deployed on each server in the fragments.

Optionally, the capacity expansion processing unit is adapted to select a server with a corresponding data size from a database system according to the determined required server data size, and randomly distribute the determined number of the montos cluster capacity expansion tasks corresponding to the role information to the clients initializing the montos cluster shards on the selected server with the corresponding number one by one.

Optionally, the capacity expansion processing unit is further adapted to start a data migration operation of the montos cluster, and migrate data to the added new segment.

Optionally, the capacity expansion processing unit is further adapted to receive initialization result information returned by the client that initializes the Mongos cluster shards on each selected server; and sending the received initialization result information to a service party submitting the service capacity expansion demand information through a specified communication channel.

Optionally, the client deployment unit is further adapted to deploy a fragmented offline client of a Mongos cluster on each server in the database system;

Optionally, the service capacity expansion demand information includes expiration time of capacity expansion fragments, and the Mongos cluster capacity expansion task also includes expiration time of capacity expansion fragments; the fragment offline client acquires the expiration time of the capacity expansion fragments by scanning the resource information on the server where the fragment offline client is located, determines the corresponding fragments to be subjected to capacity expansion deployment as the fragments needing to be offline when the expiration time of the capacity expansion fragments is reached, and the database contained in the fragments needing to be offline is the database needing to be offline;

alternatively, the first and second electrodes may be,

Optionally, the client deployment unit is further adapted to deploy a sharding migration client of a Mongos cluster on each server of the database system;

the requirement receiving unit is further suitable for receiving the information that the designated fragments are no longer available;

the capacity expansion processing unit is further suitable for generating a fragment newly-built task and a fragment migration task according to the information; selecting a certain number of available servers from a database system, distributing the fragmentation newly-built task to clients initializing Mongos cluster fragmentation on each selected server, and distributing the fragmentation migration task to fragmentation migration clients on each selected server;

the client side for initializing the Mongos cluster fragments initializes new fragments on the server according to the received fragment newly-built task, deletes non-system libraries in the new fragments, and adds the new fragments into the Mongos cluster; and migrating the unavailable data of the specified fragment to a new fragment by the fragment migration client.

Optionally, the capacity expansion processing unit is further adapted to delete the unavailable designated segment from the montos cluster after migration of data in the unavailable designated segment is completed.

According to the technical scheme of the invention, each server in the database system is provided with a client for initializing the Mongos cluster fragment, after receiving service capacity expansion requirement information of Mongos cluster resources of a service party, the service capacity expansion requirement information is generated into a capacity expansion service, a certain number of available servers are selected, a capacity expansion task is sent to the client for initializing the Mongos cluster fragment on the selected server, and the client can initialize a new fragment according to the capacity expansion task to complete the Mongos cluster capacity expansion. Therefore, the invention can automatically complete the capacity expansion of the database according to the service capacity expansion requirement, improve the capacity expansion efficiency of the database and ensure the stability of the operation of the database system under the condition of meeting the service requirement.

The foregoing description is only an overview of the technical solutions of the present invention, and the embodiments of the present invention are described below in order to make the technical means of the present invention more clearly understood and to make the above and other objects, features, and advantages of the present invention more clearly understandable.

Drawings

Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the invention. Also, like reference numerals are used to refer to like parts throughout the drawings. In the drawings:

FIG. 1 shows a flow diagram of a method of implementing automated Mongos cluster expansion, according to one embodiment of the present invention;

fig. 2 shows a schematic structural diagram of an apparatus for implementing automated montos cluster expansion according to an embodiment of the present invention.

Detailed Description

Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

FIG. 1 shows a flow diagram of a method of implementing automated Mongos cluster expansion, according to one embodiment of the present invention; as shown in fig. 1, the method includes:

step S110, respectively deploying clients initializing Mongos cluster fragments on each server in the database system.

The Mongos cluster is a database cluster of a distributed storage architecture and is suitable for storing massive data. The Mongos cluster comprises a plurality of routing nodes, a plurality of fragmentation nodes and a plurality of metadata nodes. The routing node is used for directing the requirements of the service party to corresponding fragments, the positions of the stored data in each fragment are recorded in the metadata, and the fragment node is used for storing a large amount of data. When a user inquires data, the corresponding data can be inquired through the storage position of the corresponding data in the metadata node. When the service capacity expansion requirement for more Mongos cluster resources is met, the capacity expansion of the whole Mongos cluster can be carried out by increasing the fragments in the cluster.

In order to realize the automatic expansion of the Mongos cluster, a client for initializing the Mongos cluster fragment needs to be deployed on each server in the database system, and the client for initializing the Mongos cluster fragment is the client for initializing the Mongos cluster fragment, so that the expansion of the Mongos cluster fragment in the service requirement is completed.

And step S120, receiving service capacity expansion requirement information needing to use more Mongos cluster resources, and generating a Mongos cluster capacity expansion task according to the service capacity expansion requirement information. The client side for initializing the Mongos cluster fragments initializes new fragments on the server according to the received Mongos cluster capacity expansion task, deletes non-system libraries in the new fragments, and adds the new fragments into the Mongos cluster.

When the service party has service capacity expansion requirements of Mongos cluster resources, the requirement information can be provided through the corresponding requirement interface. The requirement interface can be in the form of a cloud platform, the cloud platform is provided with a service requirement editing interface, and a service party edits and submits information corresponding to the service requirement.

And when the service capacity expansion demand information is received through the demand inlet, generating a corresponding Mongos cluster capacity expansion task according to the corresponding service demand information, so that the server performs corresponding capacity expansion operation of the initialization fragment according to the Mongos cluster capacity expansion task.

And step S130, selecting a certain number of available servers from the database system, and distributing the Mongos cluster expansion task to the client side which initializes the Mongos cluster fragmentation on each selected server.

On one hand, the server which can be used for deploying the service is selected according to the availability ratio of the resources of each server and whether the servers are available, so that the waste of the resources is reduced; on the other hand, the number of servers for deploying the service also needs to be selected according to the service demand information so as to meet the service demand. For example, the service requirement needs to be increased by 10 pieces, but the space available for each server can be increased by only 2 pieces, and then 5 available servers are needed to expand the service.

And after a certain number of available servers are selected, the Mongos cluster expansion task is distributed to the client side which initializes the Mongos cluster fragmentation on each selected server.

After the selected client side for initializing the Mongos cluster fragments on each server receives the Mongos cluster expansion task, the client side for initializing the Mongos cluster fragments initializes a new fragment on the server according to the received Mongos cluster expansion task, deletes a non-system library in the new fragment, and adds the new fragment into the Mongos cluster.

The non-system database is not self-contained in the server, so that a user can use the non-system database, a non-system database needs to be created in a newly-built fragment, but the non-system database is redundant when existing data is migrated into a new fragment, so that the non-system database in the fragment needs to be deleted when the new fragment is initialized, then the new fragment is added into the Mongos cluster, and finally the expansion of the Mongos cluster is completed.

Because the Mongos cluster capacity expansion tasks need to be sent to each selected server, the number of the Mongos cluster capacity expansion tasks for one service requirement is the same as that of the selected servers, and then the number of the database deployment tasks is sent to the selected servers.

Therefore, the invention can automatically complete the capacity expansion of the database according to the service capacity expansion requirement, improve the capacity expansion efficiency of the database and ensure the stability of the operation of the database system under the condition of meeting the service requirement.

In an embodiment of the present invention, the receiving the service capacity expansion requirement information that needs to use more Mongos cluster resources in step S110 includes: and providing an interface for submitting the service requirement information on a specified cloud platform, and receiving the service capacity expansion requirement information needing to use more Mongos cluster resources through the interface.

When the service side has more service capacity expansion requirements of Mongos cluster resources, the requirement information can be provided through a requirement interface provided on the appointed cloud platform. For example, the cloud platform provides a web page, and the business party edits information of corresponding business requirements on the web page. After the service party edits and submits, the service requirement is sent to the back end through the requirement interface of the cloud platform to perform the processing operation of Mongos cluster expansion.

In an embodiment of the present invention, the montos cluster expansion task in step S120 includes: the number of required servers and relevant parameters for deploying Mongos cluster shards; the selecting of the available number of servers from the database system in step S130 includes: a corresponding number of available servers is selected from the database system based on the number of required servers.

After a service party submits a service capacity expansion requirement of more Mongos cluster resources, in order to complete the processing of the requirement, a corresponding capacity expansion task needs to be generated according to the service capacity expansion requirement, and the task contains the number of servers required for repair, so that the selection of available servers is facilitated; the method also includes relevant parameters of the expanded Mongos cluster shards, such as: required disk space, etc.

After the capacity expansion task is generated, the selection is performed according to the number of the required servers included in the capacity expansion task, and when the server is selected, except the number and the related parameters, the requirement of service capacity expansion needs to be met, for example, the number of the required servers in the database deployment task is 5, then 5 available servers need to be selected during the selection; the selection is also performed according to the conditions of each server, such as the availability, availability and importance of the service in the database, so as to reduce the waste of the server resources. After the two aspects are considered, the corresponding number of servers can be selected finally.

Because the added fragments can also be in the form of a master-slave database when the Mongos cluster is expanded, the master-slave database comprises a master database and one or more slave databases, and the separation of database reading and writing functions is realized. The master database is responsible for data writing operation, and the slave database is responsible for data backup of the master database, data query and other operations. In order to implement the capacity expansion of the master-slave database, in an embodiment of the present invention, the step S120 of generating the Mongos cluster capacity expansion task according to the service capacity expansion requirement information includes: determining the number of required servers and role information of databases to be deployed on the determined number of servers in the shards according to the service capacity expansion demand information, for example, whether the servers are master databases or slave databases; and generating the determined number of Mongos cluster expansion tasks respectively corresponding to the role information according to the role information of the determined number of databases to be deployed on each server in the fragments.

For example, when capacity expansion is performed, 4 servers are selected, 1 master database and 3 slave databases are needed, and then a montos cluster capacity expansion task of 1 master database and a corresponding montos cluster capacity expansion task of 3 slave databases are generated.

Specifically, the selecting a certain number of available servers in the database system, and distributing the montos cluster expansion task to the client initializing the montos cluster fragmentation on each selected server includes: and selecting a server with corresponding data volume from the database system according to the determined required server data volume, and randomly distributing the determined number of Mongos cluster capacity expansion tasks respectively corresponding to the role information to the clients initializing the Mongos cluster shards on the selected servers with corresponding number one by one.

The determined number of database deployment tasks corresponding to the role information is randomly sent to the selected servers, that is, each selected server receives a Mongos cluster expansion task corresponding to the role information, but the specific Mongos cluster expansion task of which role is uncertain. That is to say, the sending of the tasks is distributed randomly, it is not limited that a certain Mongos cluster capacity expansion task must be distributed to a designated server, and it is only required to ensure that each selected server is distributed with a Mongos cluster capacity expansion task after distribution. For example, if the number of the determined servers is 3, three servers, namely server 1, server 2 and server 3, need to be selected, and finally, a montos cluster capacity expansion task of 1 master database and a corresponding montos cluster capacity expansion task of 2 slave databases are generated, and when the tasks are sent, the montos cluster capacity expansion task of the master database can be distributed to the client for initializing the montos cluster shards on the server 1, the client for initializing the montos cluster shards on the server 2 and the client for initializing the montos cluster shards on the server 3; in addition, 2 Mongos cluster capacity expansion tasks of the slave database can be distributed to clients initializing Mongos cluster fragments on the server 1, clients initializing Mongos cluster fragments on the server 2 and clients initializing Mongos cluster fragments on the server 3, and after final distribution, the Mongos cluster capacity expansion tasks are guaranteed to be available for the server 1, the server 2 and the server 3.

In a specific example, the capacity needs to be expanded in 4 servers, one of which is a master database, and the other three of which are slave databases. After the Mongos cluster capacity expansion tasks are generated, parameter information contained in one Mongos cluster capacity expansion task is as follows: p1-mdm, P2-30G, and P3-200G. Wherein P1 represents the database role, P2 represents the memory space, and P3 represents the disk space. When a client of a selected server which initializes the Mongos cluster fragment receives the Mongos cluster expansion task, a main database is established on the server according to parameter information in the task, wherein the database has a 30G memory and a 200G disk space.

When the data in the database of the Mongos framework is written, the situation that a large amount of data is stored in the same fragment and the amount of data in other fragments is small is not excluded, in order to avoid the situation, the Mongos framework has a data balancing function, and a part of data in the fragments with more stored data is moved to be stored in other relatively idle fragments, so that the effect of balancing the amount of data in all the fragments is achieved. Therefore, after performing the montos clustering expansion, in an embodiment of the present invention, the method shown in fig. 1 further includes: and starting the data migration operation of the Mongos cluster, and migrating data to the added new fragment.

Each segment in the montos cluster is divided into a plurality of data blocks (also called chunck or bucket), a data migration operation of the montos cluster is started, data is migrated to the added new segment, that is, a data block is created in the added new segment, and the data is migrated to the data block. For example, the original Mongos cluster has 4 fragments, each fragment has a data size of 6 data blocks, after the Mongos cluster expands the capacity, the Mongos cluster newly adds 2 fragments, and becomes 6 fragments, then after the data equalizer is started, the newly added 2 fragments are respectively migrated by 4 data blocks, and finally each fragment has a data size of 4 data blocks.

When the montos cluster is expanded, in an embodiment of the present invention, the method shown in fig. 1 further includes: receiving initialization result information returned by the client side which initializes the Mongos cluster fragments on each selected server; and sending the received initialization result information to a service party submitting service capacity expansion requirement information through a specified communication channel.

For the case of more traffic, when the montos cluster no longer needs to be expanded, in order to reduce the waste of database resources, the expanded fragments need to be offline processed, in an embodiment of the present invention, the method shown in fig. 1 further includes:

respectively deploying fragmented offline clients of the Mongos cluster on each server in the database system; the fragment offline client determines a database of the fragments needing offline on a server where the client is located, and performs offline processing on the determined database of the fragments needing offline. And when the requirement of service capacity expansion is finished, the database offline client determines the database of the fragments needing to be offline on the server, and performs offline processing on the determined database of the fragments needing to be offline.

When offline processing is performed on the fragmented database, if the service capacity expansion requirement information includes the database expiration time of the capacity expansion fragments and the montos cluster capacity expansion task also includes the database expiration time of the capacity expansion fragments, the fragmented offline client determines that the database of the fragmented on the server that needs offline includes: the service capacity expansion demand information comprises the expiration time of capacity expansion fragments, and the Mongos cluster capacity expansion task also comprises the expiration time of the capacity expansion fragments; the fragment offline client scans resource information on a server where the fragment offline client is located to obtain expiration time of capacity expansion fragments, when the expiration time of the capacity expansion fragments reaches, the corresponding fragments subjected to capacity expansion deployment are determined to be the fragments needing offline, and a database contained in the fragments needing offline is a database needing offline. For example, when a service side puts forward a service capacity expansion requirement, the expiration time of a database of a capacity expansion fragment is preset to be 10 days, and then after the database of the capacity expansion fragment of the service is used for 10 days, a fragment offline client in a server determines that the database of the capacity expansion fragment has reached the expiration time for 10 days, and then offline processing is performed on the database of the fragment.

When the offline processing of the partitioned database is performed, the partitioned database needing to be offline is not necessarily the newly expanded partitioned database, and the partitioned database needing to be offline can also be determined after the whole database of the whole Mongos cluster of the service is analyzed. In this case, the method for determining the database of the segment needing to be offline is as follows: the fragment offline client acquires access information of fragments deployed on a server where the fragment offline client is located, the fragments with the access quantity lower than a preset value in a latest period of time are determined as the fragments needing offline, and a database contained in the fragments needing offline is a database needing offline. For example, if the preset value is set to 100, a database with the number of fragments that are accessed less than 100 times in a period of time may be regarded as a database of fragments that need to be offline.

In the use process of the fragments of the Mongos cluster, the situation that the fragments are in problem and are no longer available is inevitable, and if no corresponding countermeasure is provided, the data in the fragments in problem are lost. In order to prevent the loss of data, a deployment fragment migration client is required to perform migration of data. In one embodiment of the invention, the method further comprises: and respectively deploying the fragmentation migration client of the Mongos cluster on each server of the database system.

When the fragments are not available any more, the client receives the information that the specified fragments are not available any more, and generates a fragment newly-building task and a fragment migration task according to the information; selecting a certain number of available servers from a database system, distributing the fragment new task to the client side of the initial Mongos cluster fragment on each selected server, and distributing the fragment migration task to the fragment migration client side on each selected server, wherein the server is selected according to the previous detailed description, and the details are not repeated herein.

After the fragment migration client on each selected server receives the fragment migration task, the client initializing the Mongos cluster fragment initializes a new fragment on the server according to the received fragment newly-built task, deletes a non-system library in the new fragment, and adds the new fragment into the Mongos cluster; the fragment migration client migrates the data in the unavailable designated fragment to a new fragment.

Because the state that the partition of the data needs to be migrated is no longer available, to prevent the waste of resources, the method further comprises: and when the data in the unavailable designated fragments are migrated, deleting the unavailable designated fragments from the Mongos cluster.

Fig. 2 shows a schematic structural diagram of an apparatus for implementing automated montos cluster expansion according to an embodiment of the present invention. As shown in fig. 2, the apparatus 200 for implementing the montos cluster automation extension includes:

the client deployment unit 210 is adapted to deploy clients initializing the Mongos cluster shards on each server in the database system.

The demand receiving unit 220 is adapted to receive service capacity expansion demand information that needs to use more Mongos cluster resources.

The capacity expansion processing unit 230 is adapted to generate a montos cluster capacity expansion task according to the service capacity expansion demand information; and selecting a certain number of available servers from the database system, and distributing the Mongos cluster expansion tasks to the clients initializing the Mongos cluster shards on the selected servers. The client side for initializing the Mongos cluster fragments initializes new fragments on the server according to the received Mongos cluster capacity expansion task, deletes non-system libraries in the new fragments, and adds the new fragments into the Mongos cluster.

In an embodiment of the present invention, the requirement receiving unit 220 is adapted to provide an interface for submitting the service requirement information on a specified cloud platform, and receive the service capacity expansion requirement information that needs to use more Mongos cluster resources through the interface.

In an embodiment of the present invention, the montos cluster capacity expansion task includes: the number of servers required and the relevant parameters for deploying the Mongos cluster shards.

The capacity expansion processing unit 230 is adapted to select a corresponding number of available servers from the database system according to the number of required servers.

Because the added fragments can also be in the form of a master-slave database when the Mongos cluster is expanded, the master-slave database comprises a master database and one or more slave databases, and the separation of database reading and writing functions is realized. The master database is responsible for data writing operation, and the slave database is responsible for data backup of the master database, data query and other operations. In order to implement the capacity expansion of the master-slave database, in an embodiment of the present invention, the capacity expansion processing unit 230 is adapted to determine, according to the service capacity expansion requirement information, the number of required servers and role information of the databases to be deployed on the determined number of servers in the shards, for example, whether the servers are master databases or slave databases; and generating the determined number of Mongos cluster expansion tasks respectively corresponding to the role information according to the role information of the determined number of databases to be deployed on each server in the fragments.

Specifically, the capacity expansion processing unit 230 is adapted to select a server with a corresponding data size from the database system according to the determined required server data size, and randomly distribute the determined number of the montos cluster capacity expansion tasks corresponding to the role information to the clients initializing the montos cluster shards on the selected server with the corresponding number one by one.

When the data in the database of the Mongos framework is written, the situation that a large amount of data is stored in the same fragment and the amount of data in other fragments is small is not excluded, in order to avoid the situation, the Mongos framework has a data balancing function, and a part of data in the fragments with more stored data is moved to be stored in other relatively idle fragments, so that the effect of balancing the amount of data in all the fragments is achieved. Therefore, after the montos cluster expansion is performed, in an embodiment of the present invention, the expansion processing unit 230 is further adapted to start a data migration operation of the montos cluster, and migrate data to the added new segment.

After the montos cluster is expanded, in an embodiment of the present invention, the expansion processing unit 230 is further adapted to receive initialization result information returned by the client that initializes the montos cluster fragments on each selected server; and sending the received initialization result information to a service party submitting service capacity expansion requirement information through a specified communication channel.

In the service capacity expansion, for the case of more services, when the montos cluster does not need capacity expansion any more, in order to reduce the waste of database resources, the capacity-expanded fragments need to be offline processed, in an embodiment of the present invention, the client deployment unit 210 is further adapted to respectively deploy the fragment offline clients of the montos cluster on each server in the database system; the fragment offline client determines a database of the fragments needing offline on a server where the client is located, and performs offline processing on the determined database of the fragments needing offline. And when the requirement of service capacity expansion is finished, the database offline client determines the database of the fragments needing to be offline on the server, and performs offline processing on the determined database of the fragments needing to be offline.

When offline processing of the fragmented databases is performed, if the service capacity expansion demand information contains the database expiration time of capacity expansion fragmentation and the Mongos cluster capacity expansion task also contains the database expiration time of capacity expansion fragmentation, then the service capacity expansion demand information contains the expiration time of capacity expansion fragmentation and the Mongos cluster capacity expansion task also contains the expiration time of capacity expansion fragmentation; the fragment offline client acquires the expiration time of the capacity expansion fragments by scanning the resource information on the server where the fragment offline client is located, when the expiration time of the capacity expansion fragments reaches, the corresponding fragments subjected to capacity expansion deployment are determined to be the fragments needing offline, and the database contained in the fragments needing offline is the database needing offline. For example, when a service side puts forward a service capacity expansion requirement, the expiration time of a database of a capacity expansion fragment is preset to be 10 days, and then after the database of the capacity expansion fragment of the service is used for 10 days, a fragment offline client in a server determines that the database of the capacity expansion fragment has reached the expiration time for 10 days, and then offline processing is performed on the database of the fragment.

In the use process of the fragments of the Mongos cluster, the situation that the fragments are in problem and are no longer available is inevitable, and if no corresponding countermeasure is provided, the data in the fragments in problem are lost. In order to prevent the loss of data, a deployment fragment migration client is required to perform migration of data. In an embodiment of the present invention, the client deploying unit 210 is further adapted to deploy the fragmentation migration clients of the Mongos cluster on each server of the database system.

The requirement receiving unit 220 is further adapted to receive information specifying that the fragments are no longer available when a situation occurs in which the fragments are no longer available.

The capacity expansion processing unit 230 is further adapted to generate a fragment newly-building task and a fragment migration task according to the information; selecting a certain number of available servers from a database system, distributing the fragment new task to the client side of the initial Mongos cluster fragment on each selected server, and distributing the fragment migration task to the fragment migration client side on each selected server, wherein the server is selected according to the previous detailed description, and the details are not repeated herein.

After the fragment migration client on each selected server receives the fragment migration task, the client initializing the Mongos cluster fragment initializes a new fragment on the server according to the received fragment newly-built task, deletes a non-system library in the new fragment, and adds the new fragment into the Mongos cluster; the shard migration client migrates the unavailable shard-specific data to a new shard.

Because the partition that needs to migrate data is already in a state of no longer being available, in order to prevent the waste of resources, the capacity expansion processing unit 230 is further adapted to delete the unavailable designated partition from the montos cluster after the migration of the data in the unavailable designated partition is completed.

In summary, according to the technical solution of the present invention, each server in the database system is provided with a client for initializing the Mongos cluster fragment, after receiving the service capacity expansion requirement information of the Mongos cluster resource of the service party, the service capacity expansion requirement information is generated into a capacity expansion service, a certain number of available servers are selected, and a capacity expansion task is sent to the client for initializing the Mongos cluster fragment on the selected server, and the client initializes a new fragment according to the capacity expansion task, thereby completing the capacity expansion of the Mongos cluster. Therefore, the invention can automatically complete the capacity expansion of the database according to the service capacity expansion requirement, improve the capacity expansion efficiency of the database and ensure the stability of the operation of the database system under the condition of meeting the service requirement.

It should be noted that:

the algorithms and displays presented herein are not inherently related to any particular computer, virtual machine, or other apparatus. Various general purpose devices may be used with the teachings herein. The required structure for constructing such a device will be apparent from the description above. Moreover, the present invention is not directed to any particular programming language. It is appreciated that a variety of programming languages may be used to implement the teachings of the present invention as described herein, and any descriptions of specific languages are provided above to disclose the best mode of the invention.

In the description provided herein, numerous specific details are set forth. It is understood, however, that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.

Similarly, it should be appreciated that in the foregoing description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. However, the disclosed method should not be interpreted as reflecting an intention that: that the invention as claimed requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of this invention.

Those skilled in the art will appreciate that the modules in the device in an embodiment may be adaptively changed and disposed in one or more devices different from the embodiment. The modules or units or components of the embodiments may be combined into one module or unit or component, and furthermore they may be divided into a plurality of sub-modules or sub-units or sub-components. All of the features disclosed in this specification (including any accompanying claims, abstract and drawings), and all of the processes or elements of any method or apparatus so disclosed, may be combined in any combination, except combinations where at least some of such features and/or processes or elements are mutually exclusive. Each feature disclosed in this specification (including any accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.

Furthermore, those skilled in the art will appreciate that while some embodiments described herein include some features included in other embodiments, rather than other features, combinations of features of different embodiments are meant to be within the scope of the invention and form different embodiments. For example, in the following claims, any of the claimed embodiments may be used in any combination.

The various component embodiments of the invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art will appreciate that a microprocessor or Digital Signal Processor (DSP) may be used in practice to implement some or all of the functions of some or all of the components of an apparatus implementing the montos cluster automation expansion according to embodiments of the present invention. The present invention may also be embodied as apparatus or device programs (e.g., computer programs and computer program products) for performing a portion or all of the methods described herein. Such programs implementing the present invention may be stored on computer-readable media or may be in the form of one or more signals. Such a signal may be downloaded from an internet website or provided on a carrier signal or in any other form.

It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by one and the same item of hardware. The usage of the words first, second and third, etcetera do not indicate any ordering. These words may be interpreted as names.

Claims

1. A method for realizing Mongos cluster automatic expansion comprises the following steps:

the client side for initializing the Mongos cluster fragments initializes new fragments on the server according to the received Mongos cluster capacity expansion task, deletes non-system libraries in the new fragments, and adds the new fragments into the Mongos cluster;

the generating of the Mongos cluster capacity expansion task according to the service capacity expansion requirement information includes: determining the number of required servers and role information of databases to be deployed on the determined number of servers in the fragments according to the service capacity expansion demand information; and generating the determined number of Mongos cluster expansion tasks respectively corresponding to the role information according to the role information of the determined number of databases to be deployed on each server in the fragments.

2. The method of claim 1, wherein the receiving the service capacity expansion requirement information requiring use of more Mongos cluster resources comprises:

3. The method of claim 1, wherein,

the Mongos cluster capacity expansion task comprises the following steps: the number of required servers and relevant parameters for deploying Mongos cluster shards;

4. The method of claim 1, wherein selecting a number of servers available in the database system and distributing the Mongos cluster expansion tasks to clients initializing Mongos cluster shards on the selected servers comprises:

5. The method of claim 1, wherein the method further comprises:

6. The method of claim 1, wherein the method further comprises:

7. The method of claim 1, wherein the method further comprises:

8. The method of claim 7, wherein the shard downline client determining the database of shards that need to be downline on the server comprises:

alternatively, the first and second electrodes may be,

9. The method of claim 1, wherein the method further comprises:

10. The method of claim 9, wherein the method further comprises:

11. A device for realizing Mongos cluster automatic expansion comprises:

the capacity expansion processing unit is further adapted to determine the number of servers required according to the service capacity expansion demand information, and role information of the databases to be deployed on the determined number of servers in the shards; and generating the determined number of Mongos cluster expansion tasks respectively corresponding to the role information according to the role information of the determined number of databases to be deployed on each server in the fragments.

12. The apparatus of claim 11, wherein,

the demand receiving unit is suitable for providing an interface for submitting business demand information on a specified cloud platform, and receiving the business capacity expansion demand information needing to use more Mongos cluster resources through the interface.

13. The apparatus of claim 11, wherein,

14. The apparatus of claim 11, wherein,

the capacity expansion processing unit is suitable for selecting a server with corresponding data volume from a database system according to the determined required server data volume, and randomly distributing the determined number of Mongos cluster capacity expansion tasks respectively corresponding to the role information to the clients initializing the Mongos cluster shards on the selected server with the corresponding number one by one.

15. The apparatus of claim 11, wherein,

the capacity expansion processing unit is further suitable for starting data migration operation of the Mongos cluster and migrating data to the added new fragments.

16. The apparatus of claim 11, wherein,

the capacity expansion processing unit is further suitable for receiving initialization result information returned by the client side which initializes the Mongos cluster fragments on each selected server; and sending the received initialization result information to a service party submitting the service capacity expansion demand information through a specified communication channel.

17. The apparatus of claim 11, wherein,

the client deployment unit is further suitable for respectively deploying the fragmentation offline client of the Mongos cluster on each server in the database system;

18. The apparatus of claim 17, wherein,

the service capacity expansion demand information comprises the expiration time of capacity expansion fragments and the Mongos cluster capacity expansion task also comprises the expiration time of the capacity expansion fragments; the fragment offline client acquires the expiration time of the capacity expansion fragments by scanning the resource information on the server where the fragment offline client is located, determines the corresponding fragments to be subjected to capacity expansion deployment as the fragments needing to be offline when the expiration time of the capacity expansion fragments is reached, and the database contained in the fragments needing to be offline is the database needing to be offline;

alternatively, the first and second electrodes may be,

19. The apparatus of claim 11, wherein,

the client deployment unit is further suitable for respectively deploying the fragmentation migration clients of the Mongos cluster on each server of the database system;

20. The apparatus of claim 19, wherein,

the capacity expansion processing unit is further adapted to delete the unavailable designated fragments from the Mongos cluster after the data in the unavailable designated fragments are migrated.