CN112083885B - Data migration method and device, electronic equipment and storage medium - Google Patents

Data migration method and device, electronic equipment and storage medium Download PDF

Info

Publication number
CN112083885B
CN112083885B CN202010936715.3A CN202010936715A CN112083885B CN 112083885 B CN112083885 B CN 112083885B CN 202010936715 A CN202010936715 A CN 202010936715A CN 112083885 B CN112083885 B CN 112083885B
Authority
CN
China
Prior art keywords
data
storage node
migration
service request
sub
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202010936715.3A
Other languages
Chinese (zh)
Other versions
CN112083885A (en
Inventor
王志勇
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Didi Infinity Technology and Development Co Ltd
Original Assignee
Beijing Didi Infinity Technology and Development Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Didi Infinity Technology and Development Co Ltd filed Critical Beijing Didi Infinity Technology and Development Co Ltd
Priority to CN202010936715.3A priority Critical patent/CN112083885B/en
Publication of CN112083885A publication Critical patent/CN112083885A/en
Application granted granted Critical
Publication of CN112083885B publication Critical patent/CN112083885B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/0604—Improving or facilitating administration, e.g. storage management
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/14—Error detection or correction of the data by redundancy in operation
    • G06F11/1402—Saving, restoring, recovering or retrying
    • G06F11/1446—Point-in-time backing up or restoration of persistent data
    • G06F11/1448—Management of the data involved in backup or backup restore
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/21—Design, administration or maintenance of databases
    • G06F16/214—Database migration support
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
    • G06F3/0647—Migration mechanisms
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
    • G06F3/0652—Erasing, e.g. deleting, data cleaning, moving of data to a wastebasket
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Abstract

The application relates to a data migration method, a data migration device, an electronic device and a storage medium. The data migration method comprises the following steps: if an event meeting the data migration condition is detected, a first storage node for performing data migration and a second storage node for receiving migrated data are determined, a data migration instruction is sent to the first storage node, the first data sub-database is backed up, a second data sub-database which stores the same data as the first data sub-database is generated, the second data sub-database is processed, a third data sub-database only containing migrated data is obtained, the first storage node is controlled, and the third data sub-database is integrally transmitted to the second storage node through a network.

Description

Data migration method and device, electronic equipment and storage medium
Technical Field
The present application relates to the field of storage service technologies, and in particular, to a data migration method and apparatus, an electronic device, and a storage medium.
Background
A distributed storage system is a storage system which dispersedly stores data on a plurality of independent storage nodes, generally, the distributed network storage system adopts an expandable system structure, a plurality of storage nodes are used for sharing storage load, a position server is used for positioning storage information, the reliability, the availability and the access efficiency of the system are improved, and the system is easy to expand.
When a service request peak comes, the storage system needs to be capable of quickly expanding capacity to reduce the pressure of a single machine, so flexible and quick capacity expansion is an important capacity of a distributed storage system, and at present, the capacity expansion logic mainly has the following two modes: the method comprises the steps that on a storage node needing to migrate data, a read interface of a storage database (RocksDB) is called, the data needing to migrate are read, then write commands are constructed one by one, each command comprises one or more pieces of data, finally the command is sent to a node newly added in, and the new node receives the command, analyzes the data in the command and finally writes the data into the RocksDB. In a second mode, on a node needing to migrate data, a read interface of a rocksDB is called first to read the data needing to migrate, then the data are constructed in situ into a Serialized Certificate storage File (SST) File of the rocksDB, finally the whole SST File is sent to the newly added node, and the SST File is imported into the rocksDB after the newly added node receives the SST File.
The two modes are that data needing to be migrated needs to be read from the rocksDB, the reading operation consumes the performance of a hard disk relatively, the speed cannot be too fast, and the stability of the whole system is affected if the speed is too fast; in addition, after receiving a write command or an SST file, a new node needs to write data into its rocksDB, and the write operation cannot be too fast, so that the data migration efficiency is low, and the capacity expansion efficiency is restricted by the above two points.
Disclosure of Invention
In view of this, embodiments of the present application at least provide a data migration method, an apparatus, an electronic device, and a storage medium, which can improve efficiency of data migration.
The application mainly comprises the following aspects:
in a first aspect, an embodiment of the present application provides a data migration method, which is applied to a central control node in a storage system, where the storage system further includes at least two storage nodes, each storage node includes a data sub-database, and the data migration method includes:
if an event meeting the data migration condition is detected to occur, determining a first storage node for data migration and a second storage node for receiving the migrated data;
sending a data migration instruction to the first storage node to enable the first storage node to backup a first data sub-library stored by the first storage node, generate a second data sub-library storing the same data as the first data sub-library, and process the second data sub-library to obtain a third data sub-library only containing the migration data;
and controlling the first storage node, and transmitting the third data sub-database to the second storage node through the network integrally.
In one possible implementation, the first storage node to perform the data migration is determined according to the following steps:
and determining the storage node with the current service performance value smaller than or equal to a first preset performance threshold value and/or the storage node with the current service request amount larger than or equal to a first preset access threshold value as the first storage node.
In one possible embodiment, the second storage node receiving the migration data is determined according to the following steps:
the current service performance value is larger than or equal to a storage node of a second preset performance threshold value, and/or; determining the storage node with the current service request amount smaller than or equal to a second preset access threshold value as the second storage node receiving the migration data;
the first preset performance threshold is smaller than a second preset performance threshold, and the first preset access threshold is larger than the second preset access threshold.
In one possible embodiment, the migration data is determined according to the following steps:
determining a current service request amount for the first storage node;
determining a plurality of service requests which need to be processed by the second storage node based on the current service request quantity, the first preset access threshold and the request number of each service request;
determining data corresponding to the plurality of service requests as the migration data.
In a possible implementation manner, the determining, based on the current service request amount, the first preset access threshold and the request number of each service request, a plurality of service requests that need to be processed by the second storage node includes:
determining a target service request quantity which needs to be processed by the second storage node according to the current service request quantity and the first preset access threshold;
and determining the plurality of service requests which need to be processed by the second storage node according to the target service request quantity and the request number of each service request.
In a possible implementation manner, after the controlling the first storage node and the transmitting the third data sub-library to the second storage node in its entirety through the network, the data migration method further includes:
and sending the service request corresponding to the migration data stored in the third data sub-base to the second storage node.
In a possible implementation manner, in the process of controlling the first storage node and transmitting the third data sub-library to the second storage node through the network as a whole, the data migration method further includes:
and sending the service request corresponding to the data stored in the first data sub-database to the first storage node.
In a possible implementation manner, after the controlling the first storage node and the transmitting the third data sub-library to the second storage node in its entirety through the network, the data migration method further includes:
sending a data deletion instruction to the first storage node to enable the first storage node to delete the migration data stored in the first data sub-database to obtain a fourth data sub-database;
and sending the service request corresponding to the data stored in the fourth data sub-database to the first storage node.
In one possible embodiment, the service request comprises at least one of the following requests:
data read requests, data write requests, data delete requests.
In a second aspect, an embodiment of the present application further provides a data migration apparatus, which is applied to a central control node in a storage system, where the storage system further includes at least two storage nodes, each storage node includes a data sub-base, and the data migration apparatus includes:
the determining module is used for determining a first storage node for data migration and a second storage node for receiving migration data if an event meeting a data migration condition is detected to occur;
the generating module is used for sending a data migration instruction to the first storage node so that the first storage node backs up a first data sub-base stored by the first storage node, generates a second data sub-base storing the same data as the first data sub-base, and processes the second data sub-base to obtain a third data sub-base only containing the migration data;
and the transmission module is used for controlling the first storage node and transmitting the whole third data sub-database to the second storage node through a network.
In a possible implementation manner, the determining module is configured to determine the first storage node performing data migration according to the following steps:
and determining the storage node with the current service performance value smaller than or equal to a first preset performance threshold value and/or the storage node with the current service request amount larger than or equal to a first preset access threshold value as the first storage node.
In a possible implementation manner, the determining module is configured to determine the second storage node receiving the migration data according to the following steps:
the current service performance value is larger than or equal to a storage node of a second preset performance threshold value, and/or; determining the storage node with the current service request amount smaller than or equal to a second preset access threshold value as the second storage node receiving the migration data;
the first preset performance threshold is smaller than a second preset performance threshold, and the first preset access threshold is larger than the second preset access threshold.
In one possible embodiment, the determining module includes:
a first determination unit, configured to determine a current service request amount for the first storage node;
a second determining unit, configured to determine, based on the current service request amount, the first preset access threshold, and a request number of each service request, a plurality of service requests that need to be processed by the second storage node;
a third determining unit, configured to determine data corresponding to the plurality of service requests as the migration data.
In a possible implementation manner, the second determining unit is configured to determine a plurality of service requests that need to be processed by the second storage node according to the following steps:
determining a target service request quantity which needs to be processed by the second storage node according to the current service request quantity and the first preset access threshold;
and determining the plurality of service requests which need to be processed by the second storage node according to the target service request quantity and the request number of each service request.
In a possible implementation manner, the data migration apparatus further includes a first sending module; after the transmission module controls the first storage node and transmits the third data sub-database to the second storage node through the network in its entirety,
the first sending module is configured to send the service request corresponding to the migration data stored in the third data sub-base to the second storage node.
In a possible implementation manner, the data migration apparatus further includes a second sending module: in the process of controlling the first storage node and transmitting the third data sub-database to the second storage node through the network,
the second sending module is configured to send the service request corresponding to the data stored in the first data sub-database to the first storage node.
In a possible implementation manner, the data migration apparatus further includes a third sending module; after the control of the first storage node and the transmission of the third data bank to the second storage node through the network in its entirety,
the third sending module is configured to send a data deletion instruction to the first storage node, so that the first storage node deletes the migration data stored in the first data sub-database, and a fourth data sub-database is obtained;
and sending the service request corresponding to the data stored in the fourth data sub-database to the first storage node.
In one possible embodiment, the service request comprises at least one of the following requests:
data read requests, data write requests, data delete requests.
In a third aspect, an embodiment of the present application further provides an electronic device, including: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device runs, the processor and the memory communicate with each other through the bus, and the machine-readable instructions are executed by the processor to perform the steps of the data migration method according to the first aspect or any one of the possible implementation manners of the first aspect.
In a fourth aspect, this application provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the computer program performs the steps of the data migration method described in the first aspect or any one of the possible implementation manners of the first aspect.
According to the data migration method, the data migration device, the electronic device and the storage medium, the first storage node for data migration is adopted to backup the first data sub-library, the second data sub-library which stores the same data as the first data sub-library is generated, the second data sub-library is processed, the third data sub-library only containing migration data is obtained, the first storage node is controlled through the central control node, the third data sub-library is integrally transmitted to the second storage node through the network, and compared with the prior art that data are transmitted to the second storage node one by one or data are transmitted through a serialized certificate storage file, the data migration method, the data migration device, the electronic device and the storage medium can improve the efficiency of data migration by integrally transmitting the third data sub-library only containing migration data to the second storage node.
Further, the data migration method provided in this embodiment of the present application may further determine a current service request amount for the first storage node, determine, based on the current service request amount, the first preset access threshold, and the request number of each service request, a plurality of service requests that need to be processed by the second storage node, and determine, as the migration data, data corresponding to the plurality of service requests, so that the migration data that needs to be migrated may be quickly determined, and data migration efficiency may be further improved.
In order to make the aforementioned objects, features and advantages of the present application more comprehensible, preferred embodiments accompanied with figures are described in detail below.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are required to be used in the embodiments will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present application and therefore should not be considered as limiting the scope, and for those skilled in the art, other related drawings can be obtained from the drawings without inventive effort.
Fig. 1 is a flowchart illustrating a data migration method provided in an embodiment of the present application;
FIG. 2 is a functional block diagram of a data migration apparatus according to an embodiment of the present application;
FIG. 3 illustrates a functional block diagram of a determination module provided by an embodiment of the present application;
FIG. 4 is a second functional block diagram of a data migration apparatus according to an embodiment of the present application;
fig. 5 shows a schematic structural diagram of an electronic device provided in an embodiment of the present application.
Description of the main element symbols:
in the figure: 200-a data migration apparatus; 210-a determination module; 211-a first determination unit; 212-a second determination unit; 213-a third determination unit; 220-a generation module; 230-a transmission module; 240-a first sending module; 250-a second sending module; 260-a third sending module; 500-an electronic device; 510-a processor; 520-a memory; 530-bus.
Detailed Description
To make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it should be understood that the drawings in the present application are for illustrative and descriptive purposes only and are not used to limit the scope of protection of the present application. Additionally, it should be understood that the schematic drawings are not necessarily drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of the present application. It should be understood that the operations of the flow diagrams may be performed out of order, and that steps without logical context may be performed in reverse order or concurrently. One skilled in the art, under the guidance of this application, may add one or more other operations to, or remove one or more operations from, the flowchart.
In addition, the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. The components of the embodiments of the present application, generally described and illustrated in the figures herein, can be arranged and designed in a wide variety of different configurations. Thus, the following detailed description of the embodiments of the present application, presented in the accompanying drawings, is not intended to limit the scope of the claimed application, but is merely representative of selected embodiments of the application. All other embodiments, which can be derived by a person skilled in the art from the embodiments of the present application without making any creative effort, shall fall within the protection scope of the present application.
To enable those skilled in the art to utilize the present disclosure, the following embodiments are presented in conjunction with a specific application scenario "data migration," and it will be apparent to those skilled in the art that the general principles defined herein may be applied to other embodiments and application scenarios without departing from the spirit and scope of the present disclosure.
The method, apparatus, electronic device or computer-readable storage medium described in the embodiments of the present application may be applied to any scenario that requires data migration, and the embodiments of the present application do not limit specific application scenarios, and any scheme that uses the data migration method and apparatus provided in the embodiments of the present application is within the scope of protection of the present application.
It should be noted that, before the present application is proposed, when a service request peak comes, a storage system needs to be quickly expanded to reduce a single-machine pressure, so flexible and quick expansion is an important capability of a distributed storage system, and at present, the following two ways are mainly used to implement expansion logic: the method comprises the steps of calling a read interface of a rocksDB on a storage node needing to migrate data, reading the data needing to migrate, constructing write commands one by one, wherein each command comprises one or more pieces of data, finally sending the command to a newly added node, and after receiving the command, analyzing the data in the command by the new node and finally writing the data into the rocksDB of the new node. And secondly, on the node needing to migrate data, firstly calling a read interface of the rocksDB to read the data needing to migrate, then constructing the data into an SST file of the rocksDB in situ, finally sending the whole SST file to the newly added node, and after receiving the SST file, the newly added node introduces the SST file into the rocksDB.
The two modes are that data needing to be migrated needs to be read from the rocksDB, the reading operation consumes the performance of a hard disk relatively, the speed cannot be too fast, and the stability of the whole system is affected if the speed is too fast; in addition, after receiving a write command or an SST file, a new node needs to write data into its rocksDB, and the write operation cannot be too fast, so that the data migration efficiency is low, and the capacity expansion efficiency is restricted by the above two points.
It should be noted that a storage database (RocksDB) is an open-source high-performance Key-Value pair (Key-Value) storage engine, and is widely applied to building a distributed storage system, and many non-relational databases (Not Only SQL, NoSql) and new relational database management system (NewSql) storage products use RocksDB as a bottom storage engine, and the RocksDB provides interfaces for reading data, writing data, loading SST files, generating snapshot data, deleting SST files within a certain range, and the like, and provides external calls. The SST file is a data file in RocksDB, and can be recognized by RocksDB and loaded into a database.
In view of the above problems, in the embodiment of the present application, if an event meeting a data migration condition is detected to occur, a first storage node performing data migration is determined, and a second storage node receiving migration data is received, a data migration instruction is sent to the first storage node, the first data sub-database is backed up, a second data sub-database storing the same data as the first data sub-database is generated, the second data sub-database is processed, a third data sub-database only containing migration data is obtained, the first storage node is controlled, and the third data sub-database is integrally transmitted to the second storage node through a network.
For the convenience of understanding of the present application, the technical solutions provided in the present application will be described in detail below with reference to specific embodiments.
Fig. 1 is a flowchart of a data migration method according to an embodiment of the present application. As shown in fig. 1, the data migration method provided in the embodiment of the present application is applied to a central control node in a storage system, where the storage system further includes at least two storage nodes, and each storage node includes a data sub-database, and includes the following steps:
s101: and if the event meeting the data migration condition is detected to occur, determining a first storage node for data migration and a second storage node for receiving the migrated data.
In this step, when detecting that an event meeting the data migration condition occurs in the storage system, the central control node first determines a first storage node performing data migration and determines a second storage node receiving migrated data in the first storage node.
Here, the event that satisfies the data migration condition may be an event that a service request for a certain storage node in the storage system has a peak, that is, the service request amount for the storage node is high, and the storage node cannot handle a large concurrent service request; the event satisfying the data migration condition may be an event when the service performance of a certain storage node in the storage system is low, such as a case where the storage node fails. Wherein the service request comprises at least one of the following requests: data read requests, data write requests, data delete requests.
Further, determining a first storage node for data migration according to the following steps:
and determining the storage node with the current service performance value smaller than or equal to a first preset performance threshold value and/or the storage node with the current service request amount larger than or equal to a first preset access threshold value as the first storage node.
In this step, a storage node in the storage system whose current service performance value is less than or equal to the first preset performance threshold may be determined as a first storage node that needs to perform data migration, or a storage node in the storage system whose current service request amount is greater than or equal to the first preset access threshold may be determined as a first storage node that needs to perform data migration, or both of these storage nodes may be used as first storage nodes, where the number of the first storage nodes is not limited.
Here, if the current service performance value of a certain storage node is less than or equal to the first preset performance threshold, it indicates that the storage node has low efficiency in processing a service request from a client, for example, it takes 0.01s to process a service request normally, and it takes 0.1s to process a service request currently, where the current service performance value of a certain storage node may reflect the capability of the storage node in processing a service request, and the higher the current service performance value is, the higher the capability of processing a service request is, and the lower the current service performance value is, the lower the capability of processing a service request is. The first preset performance threshold may be set according to actual service requirements.
Here, if the current service request amount of a certain storage node is greater than or equal to the first preset access threshold, it indicates that the storage node needs to process a larger request amount of service requests from the client, when the current service request amount of the storage node is greater than or equal to the first preset access threshold, the storage node cannot process the service requests in time, the current service request amount already exceeds the processing capability of the storage node, and the first preset access threshold may be set according to an actual service requirement.
It should be noted that when it is detected that a first storage node that needs to perform data migration exists in the storage system, it is indicated that the pressure of the first storage node in processing the service request is large, at this time, other storage nodes in the storage system are required to share the pressure of the first storage node, and specifically, part of data in the first storage node may be migrated to other storage nodes, so that the service request corresponding to the migrated data may be forwarded to other storage nodes and is not processed by the first storage node any more, so as to achieve the purpose of reducing the pressure of the first storage node.
Here, the data migration manner is substantially used for capacity expansion of the storage system, where the capacity expansion is to reduce the pressure of a single storage node, for example, one storage node can only process 100 data read/write requests, and currently 200 data read/write requests come, the storage node cannot process the request, and needs to add another storage node to share 100 data read/write requests, which needs 2 storage nodes, and then 1000 data read/write requests come, and the current 2 data storage nodes also do not process the request, and need to add new storage nodes, and continue to share the pressure.
Further, determining a second storage node receiving the migrated data according to the following steps:
the current service performance value is larger than or equal to a storage node of a second preset performance threshold value, and/or; determining the storage node with the current service request amount smaller than or equal to a second preset access threshold value as the second storage node receiving the migration data; the first preset performance threshold is smaller than a second preset performance threshold, and the first preset access threshold is larger than the second preset access threshold.
In this step, after the central control node detects that a first storage node requiring data migration appears in the storage system, a second storage node receiving the migrated data needs to be determined, where a storage node whose current service performance value is greater than or equal to a second preset performance threshold is determined as the second storage node receiving the migrated data, or a storage node whose current service request amount is less than or equal to the second preset access threshold is determined as the second storage node receiving the migrated data, or both storage nodes may be determined as the second storage node.
Here, if the current service performance value of a certain storage node is greater than or equal to a second preset performance threshold, it indicates that the efficiency of the storage node in processing the service request from the client is high, where the current service performance value of a certain storage node may reflect the capability of the storage node in processing the service request, the higher the current service performance value is, the higher the capability in processing the service request is, the lower the current service performance value is, the lower the capability in processing the service request is, and the second preset performance threshold may be set according to the actual service requirement, where the first preset performance threshold is usually much smaller than the second preset performance threshold.
Here, if the current service request amount of a certain storage node is less than or equal to the second preset access threshold, it is indicated that the request amount of the service request from the client that needs to be processed by the storage node is small, when the current service request amount of the storage node is less than or equal to the second preset access threshold, the storage node may process the service requests in time, the current service request amount does not reach the processing capability of the storage node, the second preset access threshold may be set according to an actual service requirement, wherein the first preset access threshold is greater than the second preset access threshold.
It should be noted that when it is detected that a first storage node that needs to perform data migration exists in the storage system, it is described that the pressure of the first storage node for processing the service request is relatively large, at this time, other storage nodes in the storage system are required to share the pressure of the first storage node, specifically, a part of data in the first storage node may be migrated to other storage nodes, and a second storage node that receives the migrated data is determined according to the current service performance value and the current service request amount, so that the service request corresponding to the migrated data may be forwarded to the second storage node and is no longer processed by the first storage node, so as to achieve the purpose of reducing the pressure of the first storage node, and the second storage node may help process the part of service request.
Further, the migration data may be determined according to the following steps:
determining a current service request amount for the first storage node; determining a plurality of service requests which need to be processed by the second storage node based on the current service request quantity, the first preset access threshold and the request number of each service request; determining data corresponding to the plurality of service requests as the migration data.
In this step, after determining a first storage node that needs to perform data migration and a second storage node that receives the migrated data, the central control node needs to determine which data in the first storage node needs to be migrated to the second storage node, specifically, first determines a current service request amount for the first storage node, then determines a plurality of service requests that need to be processed by the second storage node based on the current service request amount of the first storage node, a first preset access threshold, and a request number of each service request, and further determines data corresponding to the plurality of service requests as migrated data.
Further, the determining a plurality of service requests which need to be processed by the second storage node based on the current service request amount, the first preset access threshold and the request number of each service request includes the following steps:
determining a target service request quantity which needs to be processed by the second storage node according to the current service request quantity and the first preset access threshold; and determining the plurality of service requests which need to be processed by the second storage node according to the target service request quantity and the request number of each service request.
In this step, a target service request amount that needs to be processed by the second storage node may be determined according to the current service request amount of the first storage node and a first preset access threshold, for example, if the current service request amount of the first storage node is 100, and the first preset access threshold is 60, the target service request amount that needs to be processed by the second storage node is 100-60 — 40; after the target service request amount required to be processed by the second storage node is determined, a plurality of service requests required to be processed by the second storage node may be determined according to the target service request amount and the request number of each service request, for example, the target service request amount required to be processed by the second storage node is 40, the service requests corresponding to the request numbers 1 to 40 may be selected as the service requests required to be processed by the second storage node, and then, the determined data corresponding to the service requests required to be processed by the second storage node is determined as migration data.
It should be noted that, the central control node may pre-set a routing rule, for example, a storage node a is just started, the routing rule is that 100 service requests are all sent to the storage node a, when the pressure of the storage node a exceeds a threshold, a storage node B is added to share the pressure, the routing rule may change to number 0-50, the 50 service requests are sent to the storage node a, and the 50 service requests of number 50-100 are sent to the storage node B, but data corresponding to the service requests of number 50-100 need to be migrated from the storage node a to the storage node B in advance, where the routing rule may be flexibly set according to the situation.
S102: and sending a data migration instruction to the first storage node to enable the first storage node to backup a first data sub-library stored by the first storage node, generate a second data sub-library storing the same data as the first data sub-library, and process the second data sub-library to obtain a third data sub-library only containing the migration data.
In a specific implementation, after determining a first storage node that needs to perform data migration and a second storage node that receives migration data, a central control node sends a data migration instruction to the first storage node, where the data migration instruction includes a data identifier of the migration data, where the data identifier may be a data number, and after receiving the data migration instruction sent by the central control node, the first storage node backs up a first data sub-library stored by the first storage node, so that a second data sub-library that is the same as the first data sub-library in storage data can be generated, that is, the second data sub-library is completely the same as the first data sub-library, and further, the second data sub-library is processed to obtain a third data sub-library that only includes the migration data, specifically, data other than the migration data in the second data sub-library can be deleted, a third data sub-base is obtained that only contains the migrated data.
Here, describing the process of the first storage node backing up the first data sub-base stored by itself, first, each data sub-base (RocksDB) has the following functions: the method comprises the steps of making a snapshot, generating a snapshot data, starting a new rocksDB by using the snapshot data later, deleting the data in a certain range, calling a snapshot function of a first database to generate the snapshot data for only one first database in a first storage node of the application, and starting a new second database by using the snapshot data, wherein the data of the first database and the data of the second database are identical. And then calling a data deleting interface of the second data sub-library to delete the data which is not in the migration range, so that the residual data in the second data sub-library is required to be migrated, finally calling an interface of the first data sub-library to delete the data which is required to be migrated, only the data which is not required to be migrated is left in the first data sub-library, and the whole process is like splitting from the final effect. Therefore, the backup of the first data sub-library actually calls the snapshot interface of the first data sub-library to generate a snapshot data, and then the snapshot data is backed up.
S103: and controlling the first storage node, and transmitting the third data sub-database to the second storage node through the network integrally.
In specific implementation, after the first storage node stores the first data sub-database and the third data sub-database only containing migration data, the central control node controls the first storage node and transmits the whole third data sub-database to the second storage node through the network, so that the migration process of the data in the first storage node is completed.
Here, the central control node may also issue a data transmission instruction to the second storage node, so that the second storage node acquires the third data sub-database from the first storage node.
It should be noted that, in the related art, when data migration is performed, data to be migrated needs to be read from RocksDB first, and this read operation consumes hard disk performance, which cannot be too fast, and affects the stability of the entire system if it is fast; in addition, after receiving a write command or an SST file, a new node needs to write data into its rocksDB, and the write operation cannot be too fast, so that the data migration efficiency is low, and the capacity expansion efficiency is restricted by the above two points. According to the data migration method and the data migration system, the third data which only contain migration data are transmitted to the second storage node in a sub-database mode, and the data migration scheme does not involve data reading and writing, so that the data migration efficiency can be improved, and further the capacity expansion efficiency of the storage system is improved to a certain extent. The method comprises the steps of performing one-time online splitting operation on an original first data sub-database, changing the operation into generation of a second data sub-database, processing the second data sub-database to obtain a third data sub-database only containing data needing to be migrated, transmitting the whole first data sub-database to a second storage node through a network, and after receiving the third data sub-database, the second storage node mounts the ready-made second data sub-database into a process of the second storage node and can directly use the second data sub-database.
The splitting operation comprises the following specific steps: the method comprises the steps of calling a snapshot data generation interface of a first data sub-library to generate a cloned brand new second data sub-library, then calling a file deletion interface of the second data sub-library, deleting data which does not need to be migrated, and avoiding the defect of reading data because the snapshot data generation interface and the file deletion interface of the data sub-libraries do not consume the performance of a hard disk.
The data sub-library is in the form of a file directory, and then a stack of various data files is stored in the data sub-library, and the file directory is transmitted to another storage node through network transmission in an intact mode. The data sub-library is provided with an open interface, the second storage node calls the open interface after completely receiving the file directory, and the path of the data directory in the interface is appointed.
It should be noted that, the storage system includes a central control node and a plurality of storage nodes, where the central control node is responsible for distributing to each storage node data read-write task, that is, the central control node needs to play a role of scheduling, when a client initiates a write service request, a first step is to obtain a file identifier from the central control node, and the central control node selects an appropriate node and backup according to the size, backup number and other parameters of the read-write file of the client and the state and weight of the current system node, and returns a file identifier to the client, where the file identifier includes location information of a plurality of copies of the file; the storage nodes are responsible for storing data, typically in a data vault (RocksDB) of the storage nodes, one per storage node.
Further, in the process of controlling the first storage node and transmitting the entire third data sub-library to the second storage node through the network in step S103, the method further includes the following steps:
and sending the service request corresponding to the data stored in the first data sub-database to the first storage node.
In this step, in the process that the first storage node transmits the third data sub-library only containing the migration data to the second storage node, it is indicated that at this time, the data migration has not been completed, and the second storage node has not all the migration data yet, in this process, the service request corresponding to the migration data may still be processed by the first storage node, that is, the central control node still sends the service request containing the migration data stored in the first data sub-library to the first storage node for processing by the first storage node, so that it may be ensured that the service request may be processed.
Further, after the controlling the first storage node and the transmitting the entire third data sub-library to the second storage node through the network in step S103, the method further includes the following steps:
and sending the service request corresponding to the migration data stored in the third data sub-base to the second storage node.
In this step, after the first storage node completely transmits the third data sub-library only containing the migration data to the second storage node, the service request corresponding to the migration data stored in the third data sub-library may not be sent to the first storage node any more, and instead, this part of service request is sent to the second storage node, so that the pressure of the first storage node in processing the service request may be greatly reduced by not sending the service request corresponding to the migration data to the first storage node any more.
Further, after the controlling the first storage node and the transmitting the entire third data sub-library to the second storage node through the network in step S103, the method further includes the following steps:
sending a data deletion instruction to the first storage node to enable the first storage node to delete the migration data stored in the first data sub-database to obtain a fourth data sub-database; and sending the service request corresponding to the data stored in the fourth data sub-database to the first storage node.
In this step, after the first storage node completely transmits the third data sub-library containing only the migration data to the second storage node, since the service request corresponding to the migration data has been sent to the second storage node, so, the migration data in the first data sub-base is not useful for the first storage node, at this time, the central control node may send a data deletion instruction to the first storage node, so that the first storage node deletes the migration data stored in the first data sub-database to obtain a fourth data sub-database, furthermore, the central control node sends only the service request corresponding to the data stored in the fourth data sub-base to the first storage node, so that, by only sending the service request corresponding to the data stored in the fourth data sub-database to the first storage node, the pressure of the first storage node in processing the service request can be greatly reduced.
Here, the data migration scheme in the present application is explained in detail in an exemplary form, and includes the following steps:
step (1): when only one storage node A (containing 0-100 data) is originally in the storage system, a storage node B is newly added, and a part of data (0-50) in the storage node A needs to be migrated to an empty storage node B.
Step (2): splitting a rocksDB1 in a storage node A to generate a new rocksDB2, wherein the rocksDB2 only contains data (0-50) needing to be moved, and the rocksDB1 contains all the data, and a central control node control process (Proxy) of the storage system forwards a read-write request to the rocksDB 1;
the rocksDB2 is cloned, the rocksDB2 which is cloned just now contains all data, an operation of deleting the data is needed, the data which do not need to be migrated is deleted, finally, the rocksDB2 only contains the data which need to be migrated, all the data are reserved because the read-write request is still in the rocksDB1, and after the last part of the read-write request is transferred to the rocksDB2, redundant data in the rocksDB1 can be deleted.
And (3): the rocksd db2 in the storage node a is sent to the new storage node B through a network transport protocol (such as an HTTP protocol).
And (4): and after receiving the RocksDB2, the new storage node B mounts the RocksDB2 into the progress of the new storage node B and directly uses the RocksDB 2.
And (5): the process forwards the read-write request belonging to the RocksDB2 to the storage node B, the process contains a routing table, and the information in the process can determine which requests are forwarded to the storage node B;
and (6): the rocksDB1 in the storage node A deletes the moved data, so that the data of 0-100 are completely split into two rocksDBs, and the capacity expansion process is realized.
In the embodiment of the application, if an event meeting a data migration condition is detected to occur, a first storage node for performing data migration is determined, a second storage node for receiving migrated data is received, a data migration instruction is sent to the first storage node, the first data sub-database is backed up, a second data sub-database which stores the same data as the first data sub-database is generated, the second data sub-database is processed, a third data sub-database only containing migrated data is obtained, the first storage node is controlled, and the third data sub-database is integrally transmitted to the second storage node through a network.
Based on the same application concept, a data migration apparatus corresponding to the data migration method provided in the foregoing embodiment is also provided in the embodiment of the present application, and because the principle of solving the problem of the apparatus in the embodiment of the present application is similar to that of the data migration method in the foregoing embodiment of the present application, the implementation of the apparatus may refer to the implementation of the method, and repeated details are not described again.
As shown in fig. 2 to 4, fig. 2 is a functional block diagram of a data migration apparatus 200 according to an embodiment of the present application; FIG. 3 illustrates a functional block diagram of the determination module 210 provided by an embodiment of the present application; fig. 4 shows a second functional block diagram of a data migration apparatus 200 according to an embodiment of the present application.
As shown in fig. 2, the data migration apparatus 200 includes:
a determining module 210, configured to determine, if an event meeting a data migration condition is detected to occur, a first storage node performing data migration and a second storage node receiving migration data;
a generating module 220, configured to send a data migration instruction to the first storage node, so that the first storage node backs up a first data sub-library stored in the first storage node, generates a second data sub-library that stores the same data as the first data sub-library, and processes the second data sub-library to obtain a third data sub-library that only includes the migration data;
a transmission module 230, configured to control the first storage node, and transmit the entire third data sub-database to the second storage node through a network.
In one possible implementation, as shown in fig. 2, the determining module 210 is configured to determine a first storage node for data migration according to the following steps:
and determining the storage node with the current service performance value smaller than or equal to a first preset performance threshold value and/or the storage node with the current service request amount larger than or equal to a first preset access threshold value as the first storage node.
In one possible implementation, as shown in fig. 2, the determining module 210 is configured to determine the second storage node receiving the migration data according to the following steps:
the current service performance value is larger than or equal to a storage node of a second preset performance threshold value, and/or; determining the storage node with the current service request amount smaller than or equal to a second preset access threshold value as the second storage node receiving the migration data;
the first preset performance threshold is smaller than a second preset performance threshold, and the first preset access threshold is larger than the second preset access threshold.
In one possible implementation, as shown in fig. 3, the determining module 210 includes:
a first determining unit 211, configured to determine a current service request amount for the first storage node;
a second determining unit 212, configured to determine, based on the current service request amount, the first preset access threshold, and a request number of each service request, a plurality of service requests that need to be processed by the second storage node;
a third determining unit 213, configured to determine data corresponding to the plurality of service requests as the migration data.
In a possible implementation manner, as shown in fig. 3, the second determining unit 212 is configured to determine a plurality of service requests that need to be processed by the second storage node according to the following steps:
determining a target service request quantity which needs to be processed by the second storage node according to the current service request quantity and the first preset access threshold;
and determining the plurality of service requests which need to be processed by the second storage node according to the target service request quantity and the request number of each service request.
In one possible embodiment, as shown in fig. 4, the data migration apparatus 200 further includes a first sending module 240; after the transmission module 230 controls the first storage node, transmits the third data sub-library to the second storage node through the network as a whole,
the first sending module 240 is configured to send the service request corresponding to the migration data stored in the third data sub-base to the second storage node.
In a possible implementation manner, as shown in fig. 4, the data migration apparatus 200 further includes a second sending module 250: in the process of controlling the first storage node and transmitting the third data sub-database to the second storage node through the network,
the second sending module 250 is configured to send a service request corresponding to data stored in the first data sub-database to the first storage node.
In one possible embodiment, as shown in fig. 4, the data migration apparatus 200 further includes a third sending module 260; after the control of the first storage node and the transmission of the third data bank to the second storage node through the network in its entirety,
the third sending module 260 is configured to send a data deletion instruction to the first storage node, so that the first storage node deletes the migration data stored in the first data sub-database, and a fourth data sub-database is obtained;
and sending the service request corresponding to the data stored in the fourth data sub-database to the first storage node.
In one possible embodiment, the service request comprises at least one of the following requests:
data read requests, data write requests, data delete requests.
In the embodiment of the application, if an event meeting a data migration condition is detected to occur, a first storage node for performing data migration is determined, a second storage node for receiving migrated data is received, a data migration instruction is sent to the first storage node, the first data sub-database is backed up, a second data sub-database which stores the same data as the first data sub-database is generated, the second data sub-database is processed, a third data sub-database only containing migrated data is obtained, the first storage node is controlled, and the third data sub-database is integrally transmitted to the second storage node through a network.
Based on the same application concept, referring to fig. 5, a schematic structural diagram of an electronic device 500 provided in the embodiment of the present application includes: a processor 510, a memory 520, and a bus 530, the memory 520 storing machine-readable instructions executable by the processor 510, the processor 510 and the memory 520 communicating via the bus 530 when the electronic device 500 is operating, the machine-readable instructions being executable by the processor 510 to perform the steps of the data migration method as in any of the above embodiments.
In particular, the machine readable instructions, when executed by the processor 510, may perform the following:
if an event meeting the data migration condition is detected to occur, determining a first storage node for data migration and a second storage node for receiving the migrated data;
sending a data migration instruction to the first storage node to enable the first storage node to backup a first data sub-library stored by the first storage node, generate a second data sub-library storing the same data as the first data sub-library, and process the second data sub-library to obtain a third data sub-library only containing the migration data;
and controlling the first storage node, and transmitting the third data sub-database to the second storage node through the network integrally.
In the embodiment of the application, if an event meeting a data migration condition is detected to occur, a first storage node for performing data migration is determined, a second storage node for receiving migrated data is received, a data migration instruction is sent to the first storage node, the first data sub-database is backed up, a second data sub-database which stores the same data as the first data sub-database is generated, the second data sub-database is processed, a third data sub-database only containing migrated data is obtained, the first storage node is controlled, and the third data sub-database is integrally transmitted to the second storage node through a network.
Based on the same application concept, embodiments of the present application further provide a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the data migration method provided in the foregoing embodiments are performed.
Specifically, the storage medium may be a general-purpose storage medium, such as a mobile disk, a hard disk, or the like, and when a computer program on the storage medium is executed, the data migration method may be executed, and the efficiency of data migration may be improved by integrally transmitting the third data sub-library including only migration data to the second storage node.
It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the system and the apparatus described above may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again. In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus and method may be implemented in other ways. The above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units is only one logical division, and there may be other divisions when actually implemented, and for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection of devices or units through some communication interfaces, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit.
The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a non-volatile computer-readable storage medium executable by a processor. Based on such understanding, the technical solutions of the present application may be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the methods described in the embodiments of the present application. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
The above description is only for the specific embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present application, and shall be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims (20)

1. A data migration method is applied to a central control node in a storage system, the storage system further comprises at least two storage nodes, each storage node comprises a data sub-base, and the data migration method comprises the following steps:
if an event meeting the data migration condition is detected to occur, determining a first storage node for data migration, and determining a second storage node for receiving the migrated data based on the current service request quantity and/or the current service performance value of each storage node;
sending a data migration instruction to the first storage node to enable the first storage node to backup a first data sub-library stored by the first storage node, generate a second data sub-library storing the same data as the first data sub-library, and process the second data sub-library to obtain a third data sub-library only containing the migration data;
and controlling the first storage node, and transmitting the third data sub-database to the second storage node through the network integrally.
2. The data migration method according to claim 1, wherein the first storage node performing the data migration is determined according to the following steps:
and determining the storage node with the current service performance value smaller than or equal to a first preset performance threshold value and/or the storage node with the current service request amount larger than or equal to a first preset access threshold value as the first storage node.
3. The data migration method according to claim 2, wherein the second storage node receiving the migration data is determined according to the following steps:
the current service performance value is larger than or equal to a storage node of a second preset performance threshold value, and/or; determining the storage node with the current service request amount smaller than or equal to a second preset access threshold value as the second storage node receiving the migration data;
the first preset performance threshold is smaller than a second preset performance threshold, and the first preset access threshold is larger than the second preset access threshold.
4. The data migration method of claim 2, wherein the migration data is determined according to the following steps:
determining a current service request amount for the first storage node;
determining a plurality of service requests which need to be processed by the second storage node based on the current service request quantity, the first preset access threshold and the request number of each service request;
determining data corresponding to the plurality of service requests as the migration data.
5. The data migration method according to claim 4, wherein the determining, based on the current service request amount, the first preset access threshold and the request number of each service request, a plurality of service requests that need to be processed by the second storage node comprises:
determining a target service request quantity which needs to be processed by the second storage node according to the current service request quantity and the first preset access threshold;
and determining the plurality of service requests which need to be processed by the second storage node according to the target service request quantity and the request number of each service request.
6. The data migration method according to claim 1, wherein after the controlling the first storage node and the transmitting the third data sub-library as a whole to the second storage node through the network, the data migration method further comprises:
and sending the service request corresponding to the migration data stored in the third data sub-base to the second storage node.
7. The data migration method according to claim 1, wherein in the process of controlling the first storage node and transmitting the entire third data sub-library to the second storage node through the network, the data migration method further comprises:
and sending the service request corresponding to the data stored in the first data sub-database to the first storage node.
8. The data migration method according to claim 1, wherein after the controlling the first storage node and the transferring the entire third data sub-library to the second storage node via the network, the data migration method further comprises:
sending a data deletion instruction to the first storage node to enable the first storage node to delete the migration data stored in the first data sub-database to obtain a fourth data sub-database;
and sending the service request corresponding to the data stored in the fourth data sub-database to the first storage node.
9. The data migration method of claim 4, wherein the service request comprises at least one of:
data read requests, data write requests, data delete requests.
10. A data migration device is applied to a central control node in a storage system, the storage system further comprises at least two storage nodes, each storage node comprises a data sub-base, and the data migration device comprises:
the determining module is used for determining a first storage node for data migration if an event meeting a data migration condition is detected to occur, and determining a second storage node for receiving migration data based on the current service request quantity and/or the current service performance value of each storage node;
the generating module is used for sending a data migration instruction to the first storage node so that the first storage node backs up a first data sub-base stored by the first storage node, generates a second data sub-base storing the same data as the first data sub-base, and processes the second data sub-base to obtain a third data sub-base only containing the migration data;
and the transmission module is used for controlling the first storage node and transmitting the whole third data sub-database to the second storage node through a network.
11. The data migration apparatus according to claim 10, wherein the determining module is configured to determine the first storage node performing data migration according to the following steps:
and determining the storage node with the current service performance value smaller than or equal to a first preset performance threshold value and/or the storage node with the current service request amount larger than or equal to a first preset access threshold value as the first storage node.
12. The data migration apparatus according to claim 11, wherein said determining module is configured to determine the second storage node receiving the migration data according to the following steps:
the current service performance value is larger than or equal to a storage node of a second preset performance threshold value, and/or; determining the storage node with the current service request amount smaller than or equal to a second preset access threshold value as the second storage node receiving the migration data;
the first preset performance threshold is smaller than a second preset performance threshold, and the first preset access threshold is larger than the second preset access threshold.
13. The data migration apparatus of claim 11, wherein the determining module comprises:
a first determination unit, configured to determine a current service request amount for the first storage node;
a second determining unit, configured to determine, based on the current service request amount, the first preset access threshold, and a request number of each service request, a plurality of service requests that need to be processed by the second storage node;
a third determining unit, configured to determine data corresponding to the plurality of service requests as the migration data.
14. The data migration apparatus according to claim 13, wherein said second determining unit is configured to determine a plurality of service requests that need to be processed by said second storage node according to the following steps:
determining a target service request quantity which needs to be processed by the second storage node according to the current service request quantity and the first preset access threshold;
and determining the plurality of service requests which need to be processed by the second storage node according to the target service request quantity and the request number of each service request.
15. The data migration apparatus according to claim 10, wherein said data migration apparatus further comprises a first sending module; after the transmission module controls the first storage node and transmits the third data sub-database to the second storage node through the network in its entirety,
the first sending module is configured to send the service request corresponding to the migration data stored in the third data sub-base to the second storage node.
16. The data migration apparatus according to claim 10, wherein the data migration apparatus further comprises a second sending module: in the process of controlling the first storage node and transmitting the third data sub-database to the second storage node through the network,
the second sending module is configured to send the service request corresponding to the data stored in the first data sub-database to the first storage node.
17. The data migration apparatus according to claim 10, wherein said data migration apparatus further comprises a third sending module; after the control of the first storage node and the transmission of the third data bank to the second storage node through the network in its entirety,
the third sending module is configured to send a data deletion instruction to the first storage node, so that the first storage node deletes the migration data stored in the first data sub-database, and a fourth data sub-database is obtained;
and sending the service request corresponding to the data stored in the fourth data sub-database to the first storage node.
18. The data migration apparatus of claim 13, wherein the service request comprises at least one of:
data read requests, data write requests, data delete requests.
19. An electronic device, comprising: a processor, a memory and a bus, the memory storing machine-readable instructions executable by the processor, the processor and the memory communicating over the bus when the electronic device is operating, the machine-readable instructions being executable by the processor to perform the steps of the data migration method of any of claims 1 to 9.
20. A computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, which computer program, when being executed by a processor, performs the steps of the data migration method according to any one of claims 1 to 9.
CN202010936715.3A 2020-09-08 2020-09-08 Data migration method and device, electronic equipment and storage medium Active CN112083885B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010936715.3A CN112083885B (en) 2020-09-08 2020-09-08 Data migration method and device, electronic equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010936715.3A CN112083885B (en) 2020-09-08 2020-09-08 Data migration method and device, electronic equipment and storage medium

Publications (2)

Publication Number Publication Date
CN112083885A CN112083885A (en) 2020-12-15
CN112083885B true CN112083885B (en) 2021-10-15

Family

ID=73732146

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010936715.3A Active CN112083885B (en) 2020-09-08 2020-09-08 Data migration method and device, electronic equipment and storage medium

Country Status (1)

Country Link
CN (1) CN112083885B (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116414788A (en) * 2021-12-31 2023-07-11 华为技术有限公司 Database system updating method and related device

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108319596A (en) * 2017-01-16 2018-07-24 阿里巴巴集团控股有限公司 A kind of expansion method and device of database

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6366988B1 (en) * 1997-07-18 2002-04-02 Storactive, Inc. Systems and methods for electronic data storage management
WO2004109556A1 (en) * 2003-05-30 2004-12-16 Arkivio, Inc. Operating on migrated files without recalling data
CN105468473B (en) * 2014-07-16 2019-03-01 北京奇虎科技有限公司 Data migration method and data migration device
CN106777225B (en) * 2016-12-26 2021-04-06 腾讯科技（深圳）有限公司 Data migration method and system
CN109683826B (en) * 2018-12-26 2023-08-29 北京百度网讯科技有限公司 Capacity expansion method and device for distributed storage system

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108319596A (en) * 2017-01-16 2018-07-24 阿里巴巴集团控股有限公司 A kind of expansion method and device of database

Also Published As

Publication number Publication date
CN112083885A (en) 2020-12-15

Similar Documents

Publication Publication Date Title
US11809726B2 (en) Distributed storage method and device
JP4809040B2 (en) Storage apparatus and snapshot restore method
JP5727020B2 (en) Cloud computing system and data synchronization method thereof
US20060047926A1 (en) Managing multiple snapshot copies of data
JP2019101703A (en) Storage system and control software arrangement method
US20150213100A1 (en) Data synchronization method and system
JP2019519025A (en) Division and movement of ranges in distributed systems
JP2019135637A (en) Information processing method, information processing device, server, and computer-readable recording medium
US11032156B1 (en) Crash-consistent multi-volume backup generation
CN113282564B (en) Data storage method, system, node and storage medium
CN109407975B (en) Data writing method, computing node and distributed storage system
CN115098299B (en) Backup method, disaster recovery method, device and equipment of virtual machine
JP4755244B2 (en) Information generation method, information generation program, and information generation apparatus
CN104583966A (en) Backup and restore system for a deduplicated file system and corresponding server and method
CN112083885B (en) Data migration method and device, electronic equipment and storage medium
CN108604201A (en) A kind of snapshot rollback method, device, storage control and system
WO2021082925A1 (en) Transaction processing method and apparatus
KR20200102293A (en) Blockchain software capable of operation corresponding sql query, blockchain system, and method of thereof
CN115087966A (en) Data recovery method, device, equipment, medium and program product
JP4512386B2 (en) Backup system and method
CN111400273A (en) Database capacity expansion method and device, electronic equipment and machine-readable storage medium
EP3830709B1 (en) Distributed recovery of server information
CN111147226A (en) Data storage method, device and storage medium
CN109254870B (en) Data backup method and device
WO2020151337A1 (en) Distributed file processing method and apparatus, computer device and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant