CN113946293A - Cluster data migration method and device, electronic equipment and storage medium - Google Patents

Cluster data migration method and device, electronic equipment and storage medium Download PDF

Info

Publication number
CN113946293A
CN113946293A CN202111258167.4A CN202111258167A CN113946293A CN 113946293 A CN113946293 A CN 113946293A CN 202111258167 A CN202111258167 A CN 202111258167A CN 113946293 A CN113946293 A CN 113946293A
Authority
CN
China
Prior art keywords
data
cluster
node
migration
source
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202111258167.4A
Other languages
Chinese (zh)
Inventor
陈春斌
胡德祺
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Dajia Internet Information Technology Co Ltd
Original Assignee
Beijing Dajia Internet Information Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Dajia Internet Information Technology Co Ltd filed Critical Beijing Dajia Internet Information Technology Co Ltd
Priority to CN202111258167.4A priority Critical patent/CN113946293A/en
Publication of CN113946293A publication Critical patent/CN113946293A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/0604—Improving or facilitating administration, e.g. storage management
    • G06F3/0605—Improving or facilitating administration, e.g. storage management by facilitating the interaction with a user or administrator
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0638—Organizing or formatting or addressing of data
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
    • G06F3/0647—Migration mechanisms
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Abstract

The disclosure relates to a cluster data migration method, a cluster data migration device, an electronic device and a storage medium, wherein the method comprises the following steps: in the process of migrating the target data from the source data cluster to the target data cluster, setting the read-write address of the target data as the data address corresponding to the source data cluster, and after the data migration is completed, updating the read-write address of the target data as the data address corresponding to the target data cluster, so as to indicate the positions of the data to be written or read by the producer and the consumer in the data migration process and after the data migration is completed. The method can enable the consumers and the producers to have no sense on the data migration service in the process of executing respective service in the process of executing data migration, thereby not needing the cooperation of service parties to carry out data migration, reducing the labor cost, improving the efficiency of data migration, and finishing the seamless migration of data among various clusters.

Description

Cluster data migration method and device, electronic equipment and storage medium
Technical Field
The present disclosure relates to the field of big data technologies, and in particular, to a cluster data migration method and apparatus, an electronic device, and a storage medium.
Background
Due to the increase of business data and the limitation of cluster expansion, some data need to be migrated from one data cluster to another data cluster in a data migration mode, in the related art, when data migration is performed, a business party needs to cooperate with the data migration in a manual migration mode or a dual-consumption cluster switching mode, so that the business of a consumer party and the business of a producer party are affected, labor cost and time cost are consumed, and the efficiency of the data migration is low.
Disclosure of Invention
The disclosure provides a cluster data migration method, a cluster data migration device, an electronic device and a storage medium, so as to at least solve the problems that data migration in the related art consumes labor cost and time cost and the efficiency of data migration is low. The technical scheme of the disclosure is as follows:
according to a first aspect of the embodiments of the present disclosure, a cluster data migration method is provided, where the method includes:
in the process of migrating target data to a target data cluster in a source data cluster, setting a read-write address corresponding to the target data as address information corresponding to the source data cluster;
under the condition that the migration time of the target data is greater than a preset migration time, closing a data writing function corresponding to the source data cluster, wherein the preset migration time is consumption delay time of data in the source data cluster;
and under the condition that the target data is completely migrated to the target data cluster, updating the read-write address corresponding to the target data into the address information corresponding to the target data cluster.
As an optional embodiment, before the target data is completely migrated to the target data cluster, the method further includes:
receiving a first address acquisition request sent by a consumer and forwarded based on a source data node in the source data cluster;
determining, in the source data cluster, first node address information of to-be-processed data corresponding to the first address acquisition request based on address information corresponding to the source data cluster when the migration state information of the target data is in a data migration process;
forwarding the first node address information to the consumer based on the source data node, where the first node address information is used to instruct the consumer to read the to-be-processed data from a first data node corresponding to the first node address information.
As an optional embodiment, after sending the first node address information to the data node of the source data cluster, and forwarding the first node address information to the consumer through the source data node, the method further includes:
receiving a first progress acquisition request sent by the consumer and forwarded based on the first data node;
determining a first data processing progress corresponding to the consumer from the source data cluster under the condition that the migration state information of the data to be processed is in the data migration process;
forwarding the first data processing progress to the consumer based on the first data node, the first data processing progress being used for instructing the consumer to determine the data to be processed from the data stored by the first data node.
As an optional embodiment, after the target data is completely migrated to the target data cluster, the method further includes:
receiving a second address acquisition request sent by the consumer based on the forwarding of the source data node;
determining second node address information of the data to be processed corresponding to the second address acquisition request in the target data cluster based on address information corresponding to the target data cluster under the condition that the migration state information of the target data is data migration completion;
and forwarding the second node address information to the consumer based on the source data node, wherein the second node address information is used for indicating the consumer and a second data node corresponding to the second node address information to establish a subscription relationship, and reading the data to be processed from the second data node.
As an optional embodiment, after sending the second node address information to the data node of the source data cluster, and forwarding the second node address information to the consumer through the source data node, the method further includes:
receiving a second progress acquisition request sent by the consumer and forwarded based on the second data node;
determining a second data processing progress corresponding to the consumer from the source data cluster under the condition that the migration state information of the data to be processed is data migration completion;
forwarding the second data processing progress to the consumer based on the second data node, the second data processing progress being used to instruct the consumer to determine the data to be processed from the data stored by the second data node.
As an optional embodiment, before closing the data writing function corresponding to the source data cluster, the method further includes:
receiving a third address acquisition request transmitted by a producer and forwarded based on the source data node;
determining, in the target data cluster, third node address information corresponding to the third address acquisition request based on address information corresponding to the source data cluster when the migration state information of the target data is in a data migration process;
and forwarding the third node address information to the producer based on the source data node, wherein the third node address information is used for indicating the producer to write data into a third data node corresponding to the third node address information.
As an optional embodiment, after closing the data writing function corresponding to the source data cluster, the method further includes:
receiving a fourth address acquisition request transmitted by the producer and forwarded based on a source data node;
determining fourth node address information corresponding to the fourth address acquisition request in the target data cluster based on address information corresponding to the target data cluster under the condition that the migration state information of the target data is data migration completion;
forwarding the fourth node address information to the producer based on the source data node, where the fourth node address information is used to indicate that the producer and a fourth data node corresponding to the fourth node address information establish a distribution relationship, and performing data writing to the fourth data node.
As an optional embodiment, in the case that the time for migrating the target data is greater than a preset migration time, the closing the data writing function corresponding to the source data cluster includes:
determining data processing states respectively corresponding to the source data cluster and the target data cluster under the condition that the migration time of the target data is longer than the preset migration time;
determining a data migration state corresponding to the target data cluster;
and when the data processing state is a target processing state and the data migration state is a target migration state, closing a data writing function corresponding to the source data cluster. According to a second aspect of the embodiments of the present disclosure, there is provided a cluster data migration method, the method including;
acquiring current migration data from target data in a source data cluster based on a data migration progress corresponding to the migrated data, wherein the current migration data is difference data between the data migration progress and a data writing progress corresponding to the source data cluster;
updating the migrated data and the data migration progress based on the current migration data, wherein in the target data migration process, the read-write address corresponding to the target data is address information corresponding to the source data cluster, and after the target data migration is finished, the read-write address corresponding to the target data is address information corresponding to the target data cluster.
According to a third aspect of the embodiments of the present disclosure, there is provided a cluster data migration apparatus, the apparatus comprising;
the data address setting module is configured to set a read-write address corresponding to target data as address information corresponding to a source data cluster in the process of migrating the target data in the source data cluster to the target data cluster;
a write function closing module configured to close a data write function corresponding to the source data cluster when the time for migrating the target data is longer than a preset migration time, where the preset migration time is a consumption delay time of data in the source data cluster;
and the data address updating module is configured to update the read-write address corresponding to the target data to the address information corresponding to the target data cluster under the condition that the target data is completely migrated to the target data cluster.
As an optional embodiment, the apparatus further comprises:
a first address request receiving module configured to perform receiving a first address obtaining request sent by a consumer and forwarded based on a source data node in the source data cluster;
a first node address determining module, configured to determine, in the source data cluster, first node address information of to-be-processed data corresponding to the first address obtaining request based on address information corresponding to the source data cluster when the migration state information of the target data is in a data migration process;
a first node address information sending module configured to perform forwarding the first node address information to the consumer based on the source data node, where the first node address information is used to instruct the consumer to read the to-be-processed data from a first data node corresponding to the first node address information.
As an optional embodiment, the apparatus further comprises:
a first progress acquisition request receiving module configured to perform receiving a first progress acquisition request sent by the consumer and forwarded based on the first data node;
a first data processing progress acquisition module configured to determine a first data processing progress corresponding to the consumer from the source data cluster when the migration status information of the to-be-processed data is in a data migration process;
a first data processing progress sending module configured to execute forwarding the first data processing progress to the consumer based on the first data node, where the first data processing progress is used to instruct the consumer to determine the data to be processed from the data stored in the first data node.
As an optional embodiment, the apparatus further comprises:
a second address request receiving module configured to perform receiving a second address acquisition request sent by the consumer and forwarded based on the source data node;
a second node address determining module, configured to execute, when the migration status information of the target data is data migration completed, determining, in the target data cluster, second node address information of to-be-processed data corresponding to the second address acquisition request based on address information corresponding to the target data cluster;
a second node address information sending module configured to forward the second node address information to the consumer based on the source data node, where the second node address information is used to indicate that the consumer and a second data node corresponding to the second node address information establish a subscription relationship, and read the to-be-processed data from the second data node.
As an optional embodiment, the apparatus further comprises:
a second progress acquisition request receiving module configured to perform receiving a second progress acquisition request sent by the consumer and forwarded based on the second data node;
a second data processing progress acquisition module configured to determine a second data processing progress corresponding to the consumer from the source data cluster when the migration status information of the to-be-processed data indicates that data migration is completed;
a second data processing progress sending module configured to perform forwarding the second data processing progress to the consumer based on the second data node, where the second data processing progress is used to instruct the consumer to determine the to-be-processed data from the data stored in the second data node.
As an optional embodiment, the apparatus further comprises:
a third address request receiving module configured to execute receiving a third address obtaining request sent by the producer and forwarded based on the source data node;
a third node address determining module, configured to determine, in the target data cluster, third node address information corresponding to the third address obtaining request based on address information corresponding to the source data cluster when the migration state information of the target data is in a data migration process;
a third node address information sending module configured to perform forwarding the third node address information to the producer based on the source data node, where the third node address information is used to instruct the producer to perform data writing to a third data node corresponding to the third node address information.
As an optional embodiment, the apparatus further comprises:
a fourth address request receiving module configured to perform receiving a fourth address obtaining request sent by the producer and forwarded based on a source data node;
a fourth node address determining module, configured to determine, in the target data cluster, fourth node address information corresponding to the fourth address obtaining request based on address information corresponding to the target data cluster when the migration state information of the target data is that data migration is completed;
a fourth node address information sending module configured to forward the fourth node address information to the producer based on the source data node, where the fourth node address information is used to indicate that the producer and a fourth data node corresponding to the fourth node address information establish a distribution relationship, and perform data writing to the fourth data node.
As an alternative embodiment, the write function shutdown module includes:
a data processing state determining unit, configured to determine data processing states corresponding to the source data cluster and the target data cluster, respectively, when a data amount of synchronized data in the target data cluster is greater than a preset data amount;
a data synchronization state determination unit configured to perform determining a data synchronization state corresponding to the target data cluster;
and a write function closing unit configured to close the data write function corresponding to the source data cluster when the data processing state is a target processing state and the data synchronization state is a target synchronization state.
According to a fourth aspect of the embodiments of the present disclosure, there is provided a cluster data migration apparatus, the apparatus comprising;
the current migration data determining module is configured to execute data migration progress corresponding to migrated data, and acquire current migration data from target data in a source data cluster, where the current migration data is difference data between the data migration progress and a data writing progress corresponding to the source data cluster;
and the data migration module is configured to update the migrated data and the data migration progress based on the current migration data, in the target data migration process, the read-write address corresponding to the target data is address information corresponding to the source data cluster, and after the target data migration is finished, the read-write address corresponding to the target data is address information corresponding to the target data cluster.
According to a fifth aspect of the embodiments of the present disclosure, there is provided a cluster data migration system, the system including:
the system comprises: a server cluster, a source data cluster and a target data cluster;
the server cluster comprises a plurality of servers and a control server, wherein the control server is electrically connected with each server in the plurality of servers;
the server cluster is electrically connected with the source data cluster based on the control server, and the server cluster is electrically connected with the target data cluster based on the control server;
the target data cluster is configured to execute target data acquisition from the source data cluster in response to a data migration instruction;
the control server is configured to set a read-write address corresponding to target data as address information corresponding to a source data cluster in a process of migrating the target data in the source data cluster to the target data cluster;
the control server is configured to execute a data writing function corresponding to the source data cluster when the time for migrating the target data is greater than a preset migration time, where the preset migration time is a consumption delay time of data in the source data cluster;
the control server is configured to update the read-write address corresponding to the target data to the address information corresponding to the target data cluster when the target data is completely migrated to the target data cluster.
As an optional embodiment, the system further comprises: a consumer;
the source data cluster comprises a plurality of source data nodes and a source cluster control node, and the source cluster control node is electrically connected with each source data node in the plurality of source data nodes;
the control server comprises a first address request receiving module, a first node address determining module and a first node address information sending module, wherein the first address request receiving module is electrically connected with the first node address determining module, and the first node address determining module is electrically connected with the first node address information sending module;
the consumer is configured to perform sending a first address fetch request to the source data node
The source data node is configured to perform forwarding of the first address acquisition request to the source cluster control node;
the source cluster control node is configured to perform sending the first address acquisition request to the control server;
the first address request receiving module is configured to execute receiving a first address obtaining request sent by the consumer and forwarded based on a source data node in the source data cluster;
the first node address determining module is configured to determine, in the source data cluster, first node address information of to-be-processed data corresponding to the first address obtaining request based on address information corresponding to the source data cluster when the migration state information of the target data is in a data migration process;
the first node address information sending module is configured to perform sending the first node address information to the source cluster control node;
the source cluster control node is configured to perform forwarding of the first node address information to the source data node;
the source data node is configured to perform forwarding of the first node address information to the consumer;
the consumer is configured to read the data to be processed from the first data node corresponding to the first node address information.
As an optional embodiment, the control server further includes a first progress acquisition request receiving module, a first data processing progress acquisition module, and a first data processing progress sending module;
the first progress acquisition request receiving module is electrically connected with the first data processing progress acquisition module;
the first data processing progress acquisition module is electrically connected with the first data processing progress sending module:
the consumer is configured to perform sending a first progress acquisition request to the first data node;
the first data node is configured to execute a first progress acquisition request forwarded to the source cluster control node;
the source cluster control node is configured to perform forwarding of the first progress acquisition request to the control server;
the first progress acquisition request receiving module is configured to execute receiving a first progress acquisition request sent by the consumer and forwarded based on the first data node;
the first data processing progress acquisition module is configured to determine a first data processing progress corresponding to the consumer from the source data cluster when the migration state information of the data to be processed is in a data migration process;
the first data processing progress sending module is configured to execute sending the first data processing progress to the source cluster control node;
the source cluster control node is configured to perform sending the first data processing schedule to the first data node;
the first data node is configured to perform forwarding the first data processing schedule to the consumer;
the consumer is configured to perform determining the pending data from the data stored by the first data node based on the first data processing schedule.
As an optional embodiment, the control server further includes: the second node address information sending module is used for sending the second node address information to the second node address request receiving module;
the second address request receiving module is electrically connected with the second node address determining module;
the second node address determining module is electrically connected with the second node address information sending module;
the target data cluster comprises a plurality of target data nodes and a target data cluster control node, and the target data cluster control node is electrically connected with each target data node in the plurality of target data nodes;
the consumer is configured to perform sending a second address fetch request to the source data node;
the source data node is configured to perform receiving a second address fetch request sent by the consumer and forwarding the second address fetch request to the source cluster control node;
the source cluster control node is configured to perform forwarding the second address fetch request to the control server;
the second address request receiving module is configured to execute receiving a second address acquisition request sent by the consumer and forwarded based on the source data node;
the second node address determining module is configured to determine, in the target data cluster, second node address information of the to-be-processed data corresponding to the second address obtaining request based on address information corresponding to the target data cluster, when migration state information of data corresponding to the to-be-processed data is data migration completion;
the second node address information sending module is configured to perform sending the second node address information to the source cluster control node;
the source cluster control node is configured to perform forwarding the second node address information to the source data node;
the source data node is configured to perform forwarding the second node address information to the consumer;
the consumer is configured to execute a second data node corresponding to the second node address information to establish a subscription relationship, and read the data to be processed from the second data node.
As an optional embodiment, the control server further includes: the system comprises a second progress acquisition request receiving module, a second data processing progress acquisition module and a second data processing progress sending module;
the second progress acquisition request receiving module is electrically connected with the second data processing progress acquisition module;
the second data processing progress acquisition module is electrically connected with the second data processing progress sending module;
the consumer is configured to perform sending a second progress acquisition request to the second data node;
the second data node is configured to execute forwarding of the second progress acquisition request to the target cluster control node;
the target cluster control node is configured to execute forwarding of the second progress acquisition request to a control server;
the second progress acquisition request receiving module is configured to execute receiving a second progress acquisition request sent by the consumer and forwarded based on the second data node;
the second data processing progress acquisition module is configured to determine a second data processing progress corresponding to the consumer from the source data cluster when the migration status information of the to-be-processed data indicates that data migration is completed;
the second data processing progress sending module is configured to execute sending the second data processing progress to the target cluster control node;
the target cluster control node is configured to perform forwarding the second data processing schedule to the second data node;
the second data node is configured to perform forwarding the second data processing progress to the consumer;
the consumer is configured to perform determining the pending data from the data stored by the second data node based on the second data processing schedule.
As an optional embodiment, the system further comprises: a producer;
the control server further includes: the third address request receiving module, the third node address determining module and the third node address information sending module;
the third address request receiving module is electrically connected with the third node address determining module;
the third node address determining module is electrically connected with the third node address information sending module;
the producer is configured to perform sending a third address fetch request to the source data node;
the source data node is configured to perform forwarding the third address acquisition request to the source cluster control node;
the source cluster control node is configured to perform forwarding the third address acquisition request to the control server;
the third address request receiving module is configured to execute receiving a third address acquisition request sent by the producer and forwarded based on a source data node;
the third node address determining module is configured to determine, in the target data cluster, third node address information corresponding to the third address obtaining request based on address information corresponding to the source data cluster when the migration state information of the target data is in a data migration process;
the third node address information sending module is configured to perform sending the third node address information to the source cluster control node;
the source cluster control node is configured to perform forwarding of the third node address information to the source data node;
the source data node is configured to perform forwarding of the third node address information to the producer;
the producer is configured to perform data writing to a third data node corresponding to the third node address information.
As an optional embodiment, the control server further includes: the fourth node address information sending module is used for sending the fourth node address information to the fourth node address determining module;
the fourth address request receiving module is electrically connected with the fourth node address determining module;
the fourth node address determining module is electrically connected with the fourth node address information sending module;
the producer is configured to perform sending a fourth address fetch request to the source data node;
the source data node is configured to perform forwarding the fourth address acquisition request to the source cluster control node;
the source cluster control node is configured to perform forwarding the fourth address acquisition request to the control server;
the fourth address request receiving module is configured to execute receiving a fourth address obtaining request sent by the producer and forwarded based on a source data node;
the fourth node address determining module is configured to determine, in the target data cluster, fourth node address information corresponding to the fourth address obtaining request based on address information corresponding to the target data cluster when the migration state information of the target data is that data migration is completed;
the fourth node address information sending module is configured to perform sending the fourth node address information to the source data control node;
the source data control node is configured to perform forwarding of the fourth node address information to the source data node;
the source data node is configured to perform forwarding the fourth node address information to the producer;
the production is configured to execute establishing a distribution relationship with a fourth data node corresponding to the fourth node address information, and execute data writing to the fourth data node.
As an optional embodiment, the system further comprises:
the target data cluster is configured to execute data migration progress-based execution, and obtain current migration data from the target data, where the current migration data is difference data between the data migration progress in the target data cluster and the data writing progress corresponding to the data in the source data cluster.
As an optional embodiment, the control server further includes: a write function shutdown module, the write function shutdown module comprising: the device comprises a data processing state determining unit, a data synchronization state determining unit and a writing function closing unit;
the data processing state determining unit is electrically connected with the data synchronization state determining unit;
the data synchronization state determining unit is electrically connected with the writing function closing unit;
the data processing state determining unit is configured to determine data processing states respectively corresponding to the source data cluster and the target data cluster when the time for migrating the target data is longer than a preset migration time;
the data synchronization state determining unit is configured to determine a data migration state corresponding to the target data cluster;
and a write function closing unit configured to close the data write function corresponding to the source data cluster when the data processing state is a target processing state and the data migration state is a target migration state.
As an alternative embodiment, the target data cluster is configured to execute the step of obtaining the target data from the source data cluster in response to the data migration instruction, including:
the target data cluster is configured to execute determining current migration data from the target data based on a data migration progress in response to a data migration instruction, wherein the current migration data is difference data between the data migration progress in the target data cluster and a data writing progress corresponding to data in the source data cluster;
the target data cluster is configured to perform the migration of the current migration data from the source data cluster into the target data cluster.
According to a sixth aspect of embodiments of the present disclosure, there is provided an electronic apparatus including:
a processor;
a memory for storing the processor-executable instructions;
wherein the processor is configured to execute the instructions to implement the cluster data migration method described above.
According to a seventh aspect of embodiments of the present disclosure, there is provided a computer-readable storage medium, wherein instructions of the computer-readable storage medium, when executed by a processor of an electronic device, enable the electronic device to perform the cluster data migration method as described above.
According to an eighth aspect of the embodiments of the present disclosure, there is provided a computer program product comprising a computer program which, when executed by a processor, implements the cluster data migration method described above.
The technical scheme provided by the embodiment of the disclosure at least brings the following beneficial effects:
in the process of migrating the target data from the source data cluster to the target data cluster, setting the read-write address of the target data as the data address corresponding to the source data cluster, and after the data migration is completed, updating the read-write address of the target data as the data address corresponding to the target data cluster, so as to indicate the positions of the data to be written or read by the producer and the consumer in the data migration process and after the data migration is completed. The method can enable the consumers and the producers to have no sense on the data migration service in the process of executing respective service in the process of executing data migration, thereby not needing the cooperation of service parties to carry out data migration, reducing the labor cost, improving the efficiency of data migration, and finishing the seamless migration of data among various clusters.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the disclosure and are not to be construed as limiting the disclosure.
FIG. 1 is a schematic diagram illustrating an application scenario of a cluster data migration method according to an exemplary embodiment;
FIG. 2 is a flow diagram illustrating a method of cluster data migration in accordance with an exemplary embodiment.
FIG. 3 is a flow diagram illustrating a consumer requesting an address before target data has been migrated in a method of cluster data migration in accordance with an illustrative embodiment.
Fig. 4 is a schematic diagram illustrating an interaction when a consumer or a producer requests a read/write address in a cluster data migration method according to an exemplary embodiment.
FIG. 5 is a flow diagram illustrating a consumer requesting a first data processing schedule in a method for clustered data migration in accordance with an exemplary embodiment.
FIG. 6 is a schematic diagram illustrating interactions when a consumer requests progress of data processing in a cluster data migration method according to an example embodiment.
Fig. 7 is a flowchart illustrating a method for migrating cluster data, according to an exemplary embodiment, to close a data writing function corresponding to a source data cluster.
FIG. 8 is a flow diagram illustrating a consumer requesting read and write addresses after target data has been migrated in a method for cluster data migration in accordance with an illustrative embodiment.
FIG. 9 is a flowchart illustrating a consumer requesting a first data processing schedule in a method for clustered data migration in accordance with an exemplary embodiment.
Fig. 10 is a flowchart illustrating a consumer requesting a read/write address before closing a data write function corresponding to a source data cluster in a cluster data migration method according to an exemplary embodiment.
Fig. 11 is a flowchart illustrating a consumer requesting a read/write address after closing a data write function corresponding to a source data cluster in a cluster data migration method according to an exemplary embodiment.
Fig. 12 is a schematic diagram illustrating data migration in a cluster data migration method according to an exemplary embodiment.
Fig. 13 is an interaction flow diagram illustrating a target data cluster, a control server and a source data cluster in a cluster data migration method according to an example embodiment.
Fig. 14 is a diagram illustrating a federated framework for a cluster in a cluster data migration method according to an example embodiment.
FIG. 15 is a block diagram of a cluster data migration apparatus on the control server side, shown in accordance with an example embodiment.
FIG. 16 is a block diagram illustrating a cluster data migration apparatus on a target data cluster side in accordance with an exemplary embodiment.
FIG. 17 is a block diagram illustrating a cluster data migration system in accordance with an exemplary embodiment.
FIG. 18 is a block diagram illustrating a server-side electronic device in accordance with an example embodiment.
Detailed Description
In order to make the technical solutions of the present disclosure better understood by those of ordinary skill in the art, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.
It should be noted that the terms "first," "second," and the like in the description and claims of the present disclosure and in the above-described drawings are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the disclosure described herein are capable of operation in sequences other than those illustrated or otherwise described herein. The implementations described in the exemplary embodiments below are not intended to represent all implementations consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.
Fig. 1 is a schematic diagram illustrating an application scenario of a cluster data migration method according to an exemplary embodiment, where the application scenario includes a consumer 110, a producer 120, a source data cluster 130, a target data cluster 140, and a server 150. The server 150 migrates the target data from the source data cluster 130 to the target data cluster 140, and sets the read-write address corresponding to the target data as the address information corresponding to the source data cluster. Based on the address information corresponding to the source data cluster, the producer 120 writes data to the source data cluster 130, and the consumer 110 reads data from the source data cluster 130. After the target data is migrated, the server 150 updates the read-write address corresponding to the target data to the address information corresponding to the target data cluster. Based on the address information corresponding to the target data cluster, the producer 120 writes data to the target data cluster 130, and the consumer 110 reads data from the target data cluster 130.
In the embodiment of the present disclosure, the client 110 includes a physical device of a smart phone, a desktop computer, a tablet computer, a notebook computer, a digital assistant, a smart wearable device, and the like, and may also include software running in the physical device, such as an application program and the like. The operating system running on the entity device in the embodiment of the present application may include, but is not limited to, an android system, an IOS system, linux, Unix, windows, and the like. Client 110 includes a consumer 110 and a producer 120, consumer 110 reads data from source data cluster 130 or target data cluster 140, and producer writes data to source data cluster 130 or target data cluster 140.
In the disclosed embodiment, the server 150 may include a server operating independently, or a distributed server, or a server cluster composed of a plurality of servers. The server 150 may include a network communication unit, a processor, a memory, and the like. Specifically, the server 150 may be configured to determine a read/write address of the target data and determine a data processing progress.
Fig. 2 is a flowchart illustrating a cluster data migration method according to an exemplary embodiment, where the method is applied to a server, as shown in fig. 2, a server to which the method is applied may be a server cluster, and the method may be performed by a control server in the server cluster. The method comprises the following steps.
S210, in the process of migrating target data to a target data cluster in a source data cluster, setting a read-write address corresponding to the target data as address information corresponding to the source data cluster;
as an alternative embodiment, the target data includes historical write data of the source data cluster and data written by the producer to the source data cluster during the data migration process. And under the condition that the target data is migrated from the source data cluster to the target data cluster, setting the read-write address corresponding to the target data as the address information corresponding to the source data cluster. When the read-write address corresponding to the target data is the address information corresponding to the source data cluster, it indicates that the target data is still stored in the source data cluster during the migration of the target data, and the consumer needs to read the target data from the source data cluster, and at the same time, the producer also needs to write the target data from the source data cluster.
As an optional embodiment, when performing data migration, a data topic corresponding to target data may be identified as a migration status, and then the target data corresponding to the data topic is data in migration or completed in migration, that is, migration status information corresponding to the target data may be obtained, and a source data node, a producer, and a consumer cannot determine whether the target data is in migration or completed in migration according to the migration status information, and only can obtain a read-write address corresponding to the target data according to the migration status information through a server, so as to determine whether the target data is currently in a source data cluster or in a target data cluster, that is, determine whether the target data is in migration or completed in migration.
As an alternative embodiment, referring to fig. 3, before the target data is completely migrated to the target data cluster, the method further includes:
s310, receiving a first address acquisition request sent by a consumer and forwarded based on a source data node in a source data cluster;
s320, under the condition that the migration state information of the target data is in the data migration process, determining first node address information of the to-be-processed data corresponding to the first address acquisition request in the source data cluster based on the address information corresponding to the source data cluster;
s330, forwarding first node address information to the consumer based on the source data node, wherein the first node address information is used for indicating the consumer to read the data to be processed from the first data node corresponding to the first node address information.
As an alternative embodiment, before the target data is completely migrated to the target data cluster, the server receives a first address obtaining request forwarded based on the source data node in the source data cluster, where the first address obtaining request is sent to the source data node by the consumer. In a time period before the data writing function corresponding to the source data cluster is closed, or in a time period after the data writing function corresponding to the source data cluster is closed until the data migration is completed, the target data is in a migration state, the target data cluster continues to perform the data migration, these two time periods are therefore the time periods before the target data is completely migrated to the target data cluster, at which point, the data to be processed requested to be read by the consumer is the data in the target data set with the migration state information, the consumer and the source data node cannot determine whether the data to be processed is located in the source data cluster or the target data cluster, the server can determine whether the target data is located in the source data cluster or the target data cluster through the read-write address corresponding to the target data, so that the source data node forwards the first address acquisition request to the server.
When the migration state information of the target data is in the data migration process, the read-write address corresponding to the target data is address information corresponding to the source data cluster, and indicates that the target data is stored in the source data cluster, the server may determine, in the source data cluster, first node address information of the to-be-processed data corresponding to the first address acquisition request, based on the address information corresponding to the source data cluster. The server sends the first node address information to the source data node, the source data node forwards the first node address information to the consumer, and the consumer can read the data to be processed from the first data node corresponding to the first node address information. Because the first data node is located in the source data cluster, and the consumer and the source partition corresponding to the first data node have a subscription relationship, the consumer can directly obtain the data to be processed from the first data node and process the data to be processed.
As an optional embodiment, when a consumer, a source data node, and a server interact with each other at three ends, and when data in a source data cluster is undergoing data migration, please refer to fig. 4, as shown in fig. 4, the consumer sends a first address acquisition request to the source data node of the source data cluster, and the source data node recognizes that data to be processed corresponding to the first address acquisition request is data with migration status information, forwards the first address acquisition request to the server, and the server determines that a read-write address corresponding to target data is address information corresponding to the source data cluster. The server determines first node address information corresponding to target data based on address information corresponding to a source data cluster, and sends the first node address information to a source data node, and the source data node forwards the first node address information to a consumer, so that the consumer accesses the corresponding first data node to read data based on the first node address information.
As an optional embodiment, when the source data node interacts with the server, the source data node may interact with the control server in the server cluster through a source cluster control node in the source data cluster, that is, the source data node forwards the first address obtaining request to the source cluster control node, and the source cluster control node forwards the first address obtaining request to the control server. The control server sends the first node address information to the source cluster control node, the source cluster control node forwards the first node address information to the source data node, and the source data node forwards the first node address information to the consumer.
Before the target data is completely migrated to the target data cluster, when a consumer reads data marked with migration state information, the first address acquisition request can be forwarded to the server through the source data node, and the server determines that the data is stored in the source data cluster at the moment, so that the consumer can acquire correct address information in the data migration process, and data reading is normally performed, the data migration is insensitive to the service of the consumer, and the stability and the continuity of the consumer service in the data migration process are improved.
As an alternative embodiment, referring to fig. 5, after sending the first node address information to the data node of the source data cluster, and forwarding the first node address information to the consumer through the source data node, the method further includes:
s510, receiving a first progress acquisition request transmitted by a consumer based on the first data node;
s520, under the condition that the migration state information of the data to be processed is in the data migration process, determining a first data processing progress corresponding to a consumer from the source data cluster;
s530, forwarding a first data processing progress to the consumer based on the first data node, wherein the first data processing progress is used for indicating the consumer to determine data to be processed from the data stored in the first data node.
As an optional embodiment, after the consumer receives the address information of the first node, the server receives a first progress obtaining request forwarded based on the first data node, where the first progress obtaining request is sent by the consumer and is used to obtain a target data processing progress corresponding to the to-be-processed data. The first data node is the data node corresponding to the first node address information. The method includes that a consumer and a source partition corresponding to a data node in a source data cluster have a subscription relationship, when migration state information of data to be processed is in a data migration process, the consumer still reads the data to be processed from the source data cluster, a server can determine a first data processing progress corresponding to the consumer from the data node subscribed by the consumer in the source data cluster, and forwards the first data processing progress to the consumer based on the first data node, and the consumer can obtain the data to be processed from the data stored in the first data node based on the first data processing progress and process the data to be processed.
As an alternative embodiment, since each consumer has a subscription relationship with a data node in the source data cluster, the consumer has a corresponding relationship with the source data cluster, and each source data cluster may correspond to at least one consumer. When a consumer acquires data from a subscribed data node and performs data processing, the data node records the data processing progress of the consumer, and therefore the data processing progress is stored in a source data cluster. When the server receives a first progress acquisition request corresponding to a consumer, the server acquires preset mapping information from a database, wherein the preset mapping information is a corresponding relation between the consumer and a source data cluster, so that a target source data cluster corresponding to the consumer is determined, and the server acquires a first data processing progress from a data node which has a subscription relation with the consumer in the target source data cluster.
As an optional embodiment, when a consumer, a first data node, and a server interact with each other at three ends, please refer to fig. 6, as shown in fig. 6, after receiving address information of a first node, the consumer sends a first progress acquisition request to the first data node corresponding to the address information of the first node, the first data node forwards the first progress acquisition request to the server, the server obtains a corresponding relationship between the consumer and a source data cluster from a database, determines a target source data cluster corresponding to the consumer, the server obtains a first data processing progress from the target source data cluster, and sends the first data processing progress to the first data node, and the first data node forwards the first data processing progress to the consumer. If the server cannot determine the target source data cluster corresponding to the consumer in the preset mapping information, the server may query the target source data cluster corresponding to the consumer according to a preset cluster query manner, establish a corresponding relationship between the consumer and the queried target source data cluster, and update the preset mapping information based on the corresponding relationship.
As an optional embodiment, when the first data node interacts with the server, the first data node may interact with the control server in the server cluster through a source cluster control node in the source cluster, that is, the first data node forwards the first progress acquisition request to the source cluster control node, and the source cluster control node forwards the first progress acquisition request to the control server. The control server sends the first data processing progress to the source cluster control node, the source cluster control node forwards the first data processing progress to the first data node, and the first data node forwards the first data processing progress to the consumer.
When the consumer obtains the data processing progress, whether the data processing progress is in the data migration process or after the data migration is completed, the data processing progress is stored in the source data cluster and is forwarded to the consumer after being obtained by the server, and in the data migration process, the consumer consumes in the source data cluster based on the data processing progress, so that the data processing progress is not lost, the problems that the consumer has repeated consumption and the consumption is missed are avoided, and the continuity of the data processing progress of the consumer before and after the data migration is improved.
S220, under the condition that the migration time of the target data is greater than the preset migration time, closing a data writing function corresponding to the source data cluster, wherein the preset migration time is the consumption delay time of the data in the source data cluster;
as an optional embodiment, there is at least one consumer reading data in the source data cluster, where at least one consumer corresponds to different consumption delay times, and the target data cluster needs to determine a target delay time from the different consumption delay times as a preset migration time, where the target delay time may be a maximum value of the different consumption delay times. For example, there are consumer a, consumer B and consumer C in the source data cluster, where the consumption delay time of consumer a is 4 hours, the consumption delay time of consumer B is 1 hour, and the consumption delay time of consumer C is 5 hours, then at least 5 hours of data in the target data cluster should be synchronized, if the target data cluster only synchronizes 1 hour of data, the subsequent steps are started, then the delayed 4 hours of data cannot be read in the middle of consumer C, and therefore at least 5 hours of data should be synchronized.
As an optional embodiment, the data writing function corresponding to the source data cluster is closed, so that no new data exists in the source data cluster, and in the subsequent step, the data with the difference between the source data cluster and the target data cluster is migrated to the target data cluster, so as to complete the data migration.
As an alternative embodiment, please refer to fig. 7, where the time for migrating the target data is longer than the preset migration time, the closing the data writing function corresponding to the source data cluster includes:
s710, determining data processing states respectively corresponding to a source data cluster and a target data cluster under the condition that the migration time of target data is longer than preset migration time;
s720, determining a data migration state corresponding to the target data cluster;
and S730, under the condition that the data processing state is the target processing state and the data migration state is the target migration state, closing the data writing function corresponding to the source data cluster.
As an alternative embodiment, the target processing state is that the data in the source data cluster and the data in the target data cluster are in the enqueue state, and no data in the source data cluster and the data in the target data cluster is dequeued. The enqueuing state is that the copies of each data partition in the source data cluster or the target data cluster are synchronous with the data of the main partition when backing up the data of the main partition, and the dequeuing state is that the copies of each data partition in the source data cluster or the target data cluster are asynchronous with the data of the main partition when backing up the data of the main partition, and when dequeuing occurs, the data volume of the copies subjected to dequeuing is greatly different from the data volume in the main partition. The target migration state is that no delay occurs in migrating the target data from the source data cluster to the target data cluster. And under the conditions of no consumer queue loss and no delay in data migration, closing the data writing function corresponding to the source data cluster, so that no new data exists in the source data cluster.
If the data processing state is that the data in the source data cluster is dequeued or the data in the target data cluster is dequeued, the data writing function corresponding to the source data cluster is closed, so that the copy in the source data cluster, in which the dequeue occurs, is not in time with the data which is different between the backup and the main partition, or the copy in the target data cluster, in which the dequeue occurs, is not in time with the data which is different between the backup and the main partition.
If a delay occurs when migrating the target data from the source data cluster to the target data cluster, and the data writing function corresponding to the source data cluster is closed, it takes a longer time to migrate the remaining data in the target data from the source data cluster to the target data cluster.
Under the condition that no data in the cluster is fallen behind and data migration is not delayed, the data writing function corresponding to the source data cluster is closed, data loss can be avoided, the problem that the data migration time is too long can be avoided, and therefore the efficiency and the accuracy of the data migration can be improved.
As an optional embodiment, in the case that the target data is completely migrated to the target data cluster, a data writing function and a data reading function of the target data cluster are started;
as an alternative embodiment, in the case that the target data is completely migrated to the target data cluster, the data writing function and the data reading function of the target data cluster are started, so that the consumer can read the data from the target data cluster, and the producer can write the data into the target data cluster. The data writing function of the source data cluster may be turned on or may remain turned off, and no matter whether the data writing function of the source data cluster is turned on or turned off, the producer does not write data into the source data cluster any more.
And S230, under the condition that the target data are completely migrated to the target data cluster, updating the read-write address corresponding to the target data into the address information corresponding to the target data cluster.
As an optional embodiment, when the target data is completely migrated to the target data cluster, the read-write address of the target data is updated to address information corresponding to the target data cluster, and when the read-write address corresponding to the target data is address information corresponding to the target data cluster, it indicates that the target data has been migrated to the target data cluster after the target data migration is completed, a data reading function corresponding to data consumption and a data writing function corresponding to data production are performed by the target data cluster, and a consumer needs to read data from the target data cluster, and at the same time, a producer also needs to write the data from the target data cluster.
As an alternative embodiment, after the target data is completely migrated to the target data of the target data cluster, please refer to fig. 8, the method further includes:
s810, receiving a second address acquisition request transmitted by a consumer based on the forwarding of a source data node;
s820, under the condition that the migration state information of the target data is data migration completion, determining second node address information of data to be processed corresponding to a second address acquisition request in the target data cluster based on address information corresponding to the target data cluster;
and S830, forwarding second node address information to the consumer based on the source data node, wherein the second node address information is used for indicating the consumer and a second data node corresponding to the second node address information to establish a subscription relationship, and reading the data to be processed from the second data node.
As an optional embodiment, after the target data is completely migrated to the target data cluster, the migration state information of the data in the target data cluster is set to the migration complete state. The server receives a second address retrieval request forwarded based on a source data node in the source data cluster, the second address retrieval request being sent to the source data node by the consumer. At this time, the server may determine, according to the migration completion state corresponding to the to-be-processed data, that the to-be-processed data requested to be read by the consumer is already migrated, and need to acquire node address information corresponding to the migrated data, so that the source data node forwards the second address acquisition request to the server.
When the migration state information of the target data is the data migration completion state, the read-write address corresponding to the target data is address information corresponding to the target data cluster, and indicates that the target data is stored in the target data cluster, the server may determine, in the target data cluster, second node address information of the to-be-processed data corresponding to the second address acquisition request based on the address information corresponding to the target data cluster. The server sends the second node address information to the source data node, the source data node forwards the second node address information to the consumer, and the consumer can read the data to be processed from the second data node corresponding to the second node address information. Because the second data node is located in the target data cluster, the target partition corresponding to the second data node is a data partition under the target data topic, not a source partition, and the consumer does not have a subscription relationship with the target partition corresponding to the second data node, the consumer can acquire the data to be processed from the second data node and process the data to be processed only by establishing the subscription relationship with the target partition corresponding to the second data node.
As an alternative embodiment, in a case that data migration is completed, please refer to fig. 4, as shown in fig. 4, a consumer sends a second address obtaining request to a source data node of a source data cluster, the source data node forwards the second address obtaining request to a server, and the server determines that a read-write address corresponding to target data is address information corresponding to a target data cluster. The server determines second node address information corresponding to the target data based on the address information corresponding to the target data cluster, and sends the second node address information to the source data node, and the source data node forwards the second node address information to the consumer, so that the consumer accesses the corresponding second data node to read data based on the second node address information.
As an optional embodiment, when the source data node interacts with the server, the source data node may interact with the control server in the server cluster through a source cluster control node in the source cluster, that is, the source data node forwards the second address obtaining request to the source cluster control node, and the source cluster control node forwards the second address obtaining request to the control server. The control server sends the second node address information to the source cluster control node, the source cluster control node forwards the second node address information to the source data node, and the source data node forwards the second node address information to the consumer.
As an alternative embodiment, the consumer sends an address fetch request to the source data node before each data processing. Under the condition that the data in the source data cluster is not provided with the migration state information, when a consumer of the source data cluster sends an address acquisition request to a source data node in the source data cluster, the source data node can directly send corresponding node address information to the consumer without forwarding to the server. Under the condition that the data in the source data cluster is provided with the migration state information, no matter whether the migration state information indicates that the data migration is in the data migration process or indicates that the data migration is completed, the source data node needs to forward the received address acquisition request to the server. In this case, the received address fetch request may include the first address fetch request or the second address fetch request.
After the target data is completely migrated to the target data cluster, when a consumer reads the data marked with migration state information, the consumer can forward a second address acquisition request to the server through the source data node, and the server determines that the data is stored in the target data cluster at the moment, so that the consumer can acquire correct address information after the data migration is completed, the data reading is normally performed, the service of the consumer side after the data migration can be smoothly transited, and the stability and the continuity of the service of the consumer after the data migration are improved.
As an alternative embodiment, referring to fig. 9, after sending the second node address information to the data node of the source data cluster, and forwarding the second node address information to the consumer through the source data node, the method further includes:
s910, receiving a second progress acquisition request transmitted by a consumer based on the second data node;
s920, under the condition that the migration state information of the data to be processed is data migration completion, determining a second data processing progress corresponding to a consumer from the source data cluster;
and S930, forwarding a second data processing progress to the consumer based on the second data node, wherein the second data processing progress is used for indicating the consumer to determine data to be processed from the data stored in the second data node.
As an optional embodiment, after the consumer receives the address information of the second node, the server may receive a second progress obtaining request forwarded based on the second data node, where the second progress obtaining request is sent by the consumer and is used to obtain a target data processing progress corresponding to the to-be-processed data. The second data node is the data node corresponding to the second node address information. The consumer has a subscription relationship with the data nodes in the source data cluster before data migration, and when migration state information of the to-be-processed data is data migration completion, the consumer reads the to-be-processed data from the target data cluster, but the consumer reads the to-be-processed data from the source data cluster before, so that the data processing progress corresponding to the consumer is also stored in the source data server, the server can determine a second data processing progress corresponding to the consumer from the data nodes subscribed by the consumer in the source data cluster, and forward the second data processing progress to the consumer based on the second data node, and the consumer can obtain the to-be-processed data from the data stored in the second data node based on the second data processing progress and process the to-be-processed data. That is to say, the server obtains a second data processing progress corresponding to the consumer from the source data cluster, and the consumer reads data in a second data node in the target data cluster based on the second data processing progress.
Because the data in the source data cluster is not updated any more, as the consumer reads the data in the target data cluster, the data processing progress corresponding to the consumer will gradually increase until the data processing progress corresponds to the updated data in the target data cluster after the data migration is completed, and at this time, the data processing progress corresponding to the consumer will be transferred to the target data cluster. And when the next data migration is completed, the server acquires a second data processing progress corresponding to the consumer from the new source data cluster, and the consumer reads data from a second data node in the new target data cluster based on the second data processing progress.
As an optional embodiment, when a consumer, a source data node, and a server interact with each other at three ends, please refer to fig. 6, as shown in fig. 6, after receiving second node address information, the consumer sends a second progress acquisition request to a second data node corresponding to the second node address information, the second data node forwards the second progress acquisition request to the server, the server then acquires a corresponding relationship between the consumer and the source data cluster from a database, determines a target source data cluster corresponding to the consumer, the server acquires a second data processing progress from the target source data cluster, and sends the second data processing progress to the second data node, and the second data node forwards the second data processing progress to the consumer. If the server cannot determine the target source data cluster corresponding to the consumer in the preset mapping information, the server may query the target source data cluster corresponding to the consumer according to a preset cluster query manner, establish a corresponding relationship between the consumer and the queried target source data cluster, and update the preset mapping information based on the corresponding relationship.
As an optional embodiment, when the second data node interacts with the server, the second data node may interact with the control server in the server cluster through the target cluster control node in the target data cluster, that is, the second data node forwards the second progress acquisition request to the target cluster control node, and the target cluster control node forwards the second progress acquisition request to the control server. And the control server sends the second data processing progress to the target cluster control node, the target cluster control node forwards the second data processing progress to the second data node, and the second data node forwards the second data processing progress to the consumer.
When the consumer obtains the data processing progress, the data processing progress is stored in the source data cluster whether in the data migration process or after the data migration is completed, the data processing progress is forwarded to the consumer after being obtained by the server, and after the data migration is completed, the consumer consumes in the target data cluster based on the data processing progress, so that the data processing progress is not lost, the problems that the consumer consumes repeatedly and loses consumption are avoided, and the continuity of the data processing progress of the consumer before and after the data migration is improved.
As an alternative embodiment, referring to fig. 10, before the data writing function corresponding to the source data cluster is turned off, the method further includes:
s1010, receiving a third address acquisition request transmitted by a producer and forwarded based on a source data node;
s1020, under the condition that the migration state information of the target data is in the data migration process, determining third node address information corresponding to a third address acquisition request in the target data cluster based on the address information corresponding to the source data cluster;
and S1030, forwarding third node address information to the producer based on the source data node, wherein the third node address information is used for indicating the producer to write data into a third data node corresponding to the third node address information.
As an alternative embodiment, before closing the data writing function corresponding to the source data cluster, the server receives a third address obtaining request forwarded based on the source data node in the source data cluster, where the third address obtaining request is sent to the source data node by the producer. At this time, because the target data is provided with the migration status information, the consumer and the source data node cannot determine whether the object requested to be written by the producer is the source data cluster or the target data cluster, and the server can determine whether the object requested to be written by the producer is the source data cluster or the target data cluster according to the read-write address corresponding to the target data, so that the source data node forwards the third address acquisition request to the server.
When the migration state information of the target data is in the data migration process, the read-write address corresponding to the target data is address information corresponding to the source data cluster, and indicates that the target data is stored in the source data cluster, and then the object requested to be written by the producer is also the source data cluster, the server may determine, in the source data cluster, address information of a third node corresponding to the third address acquisition request based on the address information corresponding to the source data cluster. The server sends the address information of the third node to the source data node, the source data node forwards the address information to the producer, and the producer can read the data to be processed from the third data node corresponding to the address information of the third node. Because the third data node is located in the source data cluster, and the producer and the source partition corresponding to the third data node have a publishing relationship, the producer can directly obtain the data to be processed from the third data node and process the data to be processed.
As an optional embodiment, when a producer, a source data node and a server interact with each other at three ends, and when data in a source data cluster is undergoing data migration, please refer to fig. 4, as shown in fig. 4, the producer sends a third address acquisition request to the source data node of the source data cluster, and the source data node determines that data in the source data cluster is provided with migration status information, so that it is recognized that a data write object corresponding to the third address acquisition request is a cluster undergoing data migration, the third address acquisition request is forwarded to the server, and the server determines that a read-write address of target data is address information corresponding to the source data cluster. The server determines third node address information corresponding to a third address acquisition request based on address information corresponding to the source data cluster, and sends the third node address information to the source data node, and the source data node forwards the third node address information to a producer, so that the producer writes data into the corresponding third data node based on the third node address information.
As an optional embodiment, when the source data node interacts with the server, the source data node may interact with the control server in the server cluster through a source cluster control node in the source data cluster, that is, the source data node forwards the third address obtaining request to the source cluster control node, and the source cluster control node forwards the third address obtaining request to the control server. The control server sends the address information of the third node to the source cluster control node, the source cluster control node forwards the address information of the third node to the source data node, and the source data node forwards the address information of the third node to the producer.
As an alternative embodiment, the producer may send an address fetch request to the source data node before each data write. Under the condition that the data in the source data cluster is not provided with the migration state information, when a producer of the source data cluster sends an address acquisition request to a source data node in the source data cluster, the source data node can directly send corresponding node address information to the producer, and the corresponding node address information does not need to be forwarded to the server. Under the condition that the data in the source data cluster is provided with the migration state information, no matter whether the migration state information indicates that the data migration is in the data migration process or indicates that the data migration is completed, the source data node needs to forward the received address acquisition request to the server. In this case, the address fetch request received at the time of completion of the data migration may be a third address fetch request or a fourth address fetch request.
Before closing the data writing function corresponding to the source data cluster, when the written object is the cluster for data migration, the producer may forward a third address acquisition request to the server through the source data node, and the server determines that the written object is in the source data cluster at this time, so that before closing the data writing function corresponding to the source data cluster, the producer can obtain correct address information, thereby performing data writing normally, and enabling smooth transition of the service of the producer side after data migration, thereby improving stability and continuity of the producer service after data migration.
As an alternative embodiment, referring to fig. 11, after the data writing function corresponding to the source data cluster is turned off, the method further includes:
s1110, receiving a fourth address acquisition request transmitted by a producer and forwarded based on a source data node;
s1120, under the condition that the migration state information of the target data is data migration completion, determining fourth node address information corresponding to a fourth address acquisition request in the target data cluster based on the address information corresponding to the target data cluster;
s1130, fourth node address information is forwarded to the producer based on the source data node, the fourth node address information is used for indicating the producer and a fourth data node corresponding to the fourth node address information to establish a publishing relationship, and data writing is performed on the fourth data node.
As an alternative embodiment, after the data writing function corresponding to the source data cluster is closed, the producer who is writing data in the source data cluster may receive a preset prompt instruction, where the preset prompt instruction indicates that the data in the source data cluster is abnormal. And the producer responds to the preset prompt instruction to regenerate the address acquisition request, wherein the address acquisition request is the fourth address acquisition request. The server receives a fourth address acquisition request forwarded based on the source data node in the source data cluster, wherein the fourth address acquisition request is sent to the source data node by the producer.
When the migration state information of the target data is the data migration completion state, the read-write address corresponding to the target data is address information corresponding to the target data cluster, and indicates that the target data is stored in the target data cluster, and then the object requested to be written by the producer is also the target data cluster, and the server may determine, in the target data cluster, fourth node address information corresponding to a fourth address acquisition request based on the address information corresponding to the target data cluster. The server sends the fourth node address information to the source data node, the source data node forwards the fourth node address information to the producer, and the producer can read the data to be processed from the fourth data node corresponding to the fourth node address information. Because the fourth data node is located in the target data cluster, and the producer and the target partition corresponding to the fourth data node do not have a subscription relationship, the consumer needs to establish a publishing relationship with the target partition corresponding to the fourth data node to write data into the fourth data node.
As an alternative embodiment, in the case that the data migration of the source data cluster is completed, please refer to fig. 4, as shown in fig. 4, after the data writing function corresponding to the source data cluster is closed. And the producer generates a fourth address acquisition request based on a preset prompt instruction. The producer sends the fourth address acquisition request to a source data node of the source data cluster, the source data node determines that the data in the source data cluster is provided with migration state information, therefore, the data write object corresponding to the fourth address obtaining request is identified as the cluster undergoing data migration, the fourth address acquisition request is forwarded to the server, the server determines that the read-write address corresponding to the target data is the address information corresponding to the target data cluster, the server determines the fourth node address information corresponding to the fourth address acquisition request based on the address information corresponding to the target data cluster, and transmits the fourth node address information to the source data node, which forwards the fourth node address information to the producer, and enabling the producer to write data into the corresponding fourth data node based on the fourth node address information.
As an optional embodiment, when the source data node interacts with the server, the source data node may interact with the control server in the server cluster through a source cluster control node in the source data cluster, that is, the source data node forwards the fourth address obtaining request to the source cluster control node, and the source cluster control node forwards the fourth address obtaining request to the control server. The control server sends the fourth node address information to the source cluster control node, the source cluster control node forwards the fourth node address information to the source data node, and the source data node forwards the fourth node address information to the producer.
After the data writing function corresponding to the source data cluster is closed, when a producer writes in a cluster for data migration, the producer can forward a fourth address acquisition request to the server through the source data node, and the server determines that the written object is in the target data cluster, so that after the data migration is completed, the producer can obtain correct address information, thereby normally writing in the data, smoothly transitioning the business of the producer after the data migration, and improving the stability and the continuity of the business of the producer after the data migration.
As an alternative embodiment, a cluster data migration method is shown, which is applied to a target data cluster side, and includes:
acquiring current migration data from target data in the source data cluster based on the data migration progress corresponding to the migrated data, wherein the current migration data is difference data between the data migration progress and the data writing progress corresponding to the source data cluster;
updating the migrated data and the data migration progress based on the current migration data, wherein in the target data migration process, the read-write address corresponding to the target data is address information corresponding to the source data cluster, and after the target data migration is finished, the read-write address corresponding to the target data is address information corresponding to the target data cluster.
As an alternative embodiment, a target data cluster is created, and target data is migrated from a source data cluster to the target data cluster, where the target data includes historical write data of the source data cluster and data written by a producer to the source data cluster during data migration. The source data cluster is a cluster for producing and consuming target data before data migration, and the target data cluster is a cluster for producing and consuming target data after data migration. The target data may be obtained from the source data cluster by using the target data cluster as a copy (follower) of the source data cluster based on a copy mechanism of the consumption queue. During the data migration process, the producer may continue to write data to the source data cluster until the data writing function of the source data cluster is turned off.
As an optional embodiment, in the data migration process, the producer also continuously writes data into the source data cluster, the data writing progress corresponding to the target data in the source data cluster is continuously increased, and when the data writing progress is increased each time, a progress difference between the data migration progress in the target data cluster and the data writing progress in the source data cluster is determined, and difference data corresponding to the progress difference is current migration data.
As an alternative embodiment, the target data cluster is used as a copy of the source data cluster, end displacement (LEO) of data in the target data cluster is used as data migration progress, the LEO is used to indicate an end position of the data migrated into the target data cluster, and the end position of the data in the source data cluster is used as data writing progress. Determining the data progress corresponding to the LEO in the source data cluster, so as to obtain difference data corresponding to a progress difference between the data progress and the data writing progress, that is, comparing the data with a difference between the tail position of the data in the source data cluster and the tail position of the data in the target data cluster, thereby obtaining the current migration data. Because the data writing progress and the data migration progress corresponding to the data written into the source data cluster by the producer are corresponding, after the migration is completed, when the producer writes data into the target data cluster, the data can be written from the data corresponding to the data migration progress.
As an optional embodiment, in the source data cluster, there is a corresponding relationship between the target data and the source data topics, each source data topic corresponds to at least one data partition, and the target data is stored on the data node corresponding to the target data partition. When data migration is performed, a target data topic having the same identification information as a source data topic corresponding to target data can be created in a target data node, and the target data topic is the same as the source data topic. And pulling data from the source partition corresponding to the source data topic in the source data cluster. And after the current migration data is migrated from the source data cluster to the target data cluster, the current migration data is correspondingly stored in a target partition corresponding to the target data subject. The source data topic and the target data topic may be data topics (topics) in a card queue, and the source partition and the target partition may be data partitions (partitions) in the card queue.
As an alternative embodiment, please refer to fig. 12, as shown in fig. 12, a cluster B is a target data cluster, a cluster a is a source data cluster, a cluster B is a copy of the cluster a, and a target partition B1 corresponding to a target data topic B of the cluster B obtains data from a source partition a1 corresponding to the source data topic a of the cluster a and correspondingly stores the data in a target partition B1 corresponding to the target data topic B.
The target data is migrated from the source data cluster to the target data cluster based on a copy mechanism, so that the target data cluster is a complete mirror image cluster of the source data cluster, and the data migration progress corresponds to the data writing progress, thereby avoiding the problem of data repetition or data loss during data migration, and improving the stability and consistency of data migration.
As an alternative embodiment, as shown in fig. 13, there is an interaction flowchart of a target data cluster, a control server and a source data cluster, where the control server belongs to a server cluster, and the method includes:
s1310, the target data cluster responds to the data migration instruction and obtains target data from the source data cluster;
s1320, the control server sets a read-write address corresponding to the target data as address information corresponding to the source data cluster in the process of migrating the target data in the source data cluster to the target data cluster;
s1330, the control server closes the data writing function corresponding to the source data cluster under the condition that the migration time of the target data is longer than the preset migration time, wherein the preset migration time is the consumption delay time of the data in the source data cluster;
s1340, the control server updates the read-write address corresponding to the target data into the address information corresponding to the target data cluster under the condition that the target data are completely migrated to the target data cluster.
As an alternative embodiment, the server may be a server corresponding to a federation architecture (federation), where the federation architecture includes a plurality of data clusters and service clusters, and the service clusters include a data migration service, a metadata service, and a global scheduling service. In the process of executing data migration, a target data cluster is created in a federation framework, a data cluster storing target data to be migrated is used as a source data cluster, a target data subject corresponding to the target data is marked as a migration state, and migration state information corresponding to the target data is obtained.
And the control server in the service cluster migrates the target data from the source data cluster to the target data cluster, and sets the read-write address of the target data as the address information corresponding to the source data cluster. The data address corresponding to the source data cluster may be stored in the metadata service. And under the condition that the data volume of the migrated data in the target data cluster is greater than the preset data volume, closing the data writing function corresponding to the source data cluster, and waiting for the completion of data migration between the target data cluster and the source data cluster. And after the data migration is finished, starting a data writing function and a data reading function of the target data cluster, and updating the reading and writing address of the target data into address information corresponding to the target data cluster. Address information corresponding to the target data cluster may be stored in the metadata service. After the data migration is completed, the control server in the service cluster may also update the migration state information corresponding to the target data to be the migration completed state.
Before target data is completely migrated to a target data cluster, and after the target data is completely migrated to the target data cluster, when a consumer reads data, the read data is data with migration state information, so that when a source data node receives a first address acquisition request or a second address acquisition request sent by the consumer, the first address acquisition request or the second address acquisition request is forwarded to a control server in a service cluster, and the control server determines a read-write address corresponding to the target data from a metadata service, so as to determine node address information of the data read by the consumer. The control server forwards the node address information to the consumer via the source data node.
When a consumer receives node address information for reading data, a data node corresponding to the node address information sends a first progress acquisition request or a second progress acquisition request, the corresponding data node forwards the first progress acquisition request or the second progress acquisition request to a control server in a service cluster, the control server starts a global scheduling service, and the global scheduling service determines a source data cluster with a first data processing progress or a source data cluster with a second data processing progress from a database to obtain the first data processing progress or the second data processing progress. And the control server forwards the first data processing progress or the second data processing progress to the consumer through the corresponding data node.
Before closing a data writing function corresponding to a source data cluster and after closing the data writing function corresponding to the source data cluster, when consumer written data is generated, because a written object is a cluster which is in data migration, when a source data node receives a third address acquisition request or a fourth address acquisition request sent by a producer, the source data node forwards the third address acquisition request or the fourth address acquisition request to a control server of a server cluster, and the control server determines a read-write address corresponding to target data from a metadata service, so that node address information written by the producer is determined. The control server forwards the node address information to the producer via the source data node.
As an alternative embodiment, please refer to fig. 14, each data cluster in the federation includes a plurality of data nodes and a cluster control node, the cluster control node is configured to forward a request received by each data node to a server, the service cluster also includes a plurality of servers and a control server, the servers other than the control server may be sub-servers, the control server may be a metadata control server, the metadata control server is configured to maintain metadata information of all data clusters in the federation, and the metadata control server may synchronize the metadata information to each sub-server, so as to ensure consistency of metadata management.
The embodiment of the disclosure provides a cluster data migration method, which includes: in the process of migrating the target data from the source data cluster to the target data cluster, the read-write address of the target data is set as the address information corresponding to the source data cluster, and after the data migration is completed, the read-write address of the target data is updated to the address information corresponding to the target data cluster, so that the positions of the data written in or read from the producer and the consumer are indicated in the data migration process and after the data migration is completed. The method can enable the consumer and the producer to have no sense on the data migration service in the process of executing respective service in the process of executing data migration, so that the data migration is not needed to be carried out by the cooperation of a service party, the server can execute the data migration, manual operation is not needed, the labor cost is reduced, the data migration efficiency is improved, and the seamless migration of data among clusters can be completed.
FIG. 15 is a block diagram illustrating a cluster data migration apparatus, according to an example embodiment. Referring to fig. 15, the apparatus includes;
a data address setting module 1510 configured to set a read-write address corresponding to target data as address information corresponding to a source data cluster in a process of migrating the target data in the source data cluster to the target data cluster;
a write function closing module 1520, configured to close a data write function corresponding to the source data cluster when the time for migrating the target data is greater than a preset migration time, where the preset migration time is a consumption delay time of the data in the source data cluster;
the data address updating module 1530 is configured to update the read/write address corresponding to the target data to the address information corresponding to the target data cluster when the target data is completely migrated to the target data cluster.
As an optional embodiment, the apparatus further comprises:
a first address request receiving module configured to execute receiving a first address obtaining request sent by a consumer and forwarded based on a source data node in a source data cluster;
the first node address determining module is configured to determine, in the source data cluster, first node address information of to-be-processed data corresponding to the first address obtaining request, based on address information corresponding to the source data cluster, when the migration state information of the target data is in a data migration process;
the first node address information sending module is configured to forward first node address information to the consumer based on the source data node, wherein the first node address information is used for indicating the consumer to read the to-be-processed data from the first data node corresponding to the first node address information.
As an optional embodiment, the apparatus further comprises:
the first progress acquisition request receiving module is configured to execute receiving of a first progress acquisition request transmitted by a consumer and forwarded based on a first data node;
the first data processing progress acquisition module is configured to determine a first data processing progress corresponding to a consumer from the source data cluster under the condition that the migration state information of the data to be processed is in the data migration process;
the first data processing progress sending module is configured to forward a first data processing progress to the consumer based on the first data node, and the first data processing progress is used for instructing the consumer to determine data to be processed from the data stored in the first data node.
As an optional embodiment, the apparatus further comprises:
a second address request receiving module configured to execute receiving a second address acquisition request sent by a consumer and forwarded based on a source data node;
the second node address determining module is configured to execute, based on address information corresponding to the target data cluster, determining second node address information of the to-be-processed data corresponding to the second address acquisition request in the target data cluster when the migration state information of the target data is the completion of data migration;
and the second node address information sending module is configured to execute forwarding of second node address information to the consumer based on the source data node, wherein the second node address information is used for indicating the consumer and a second data node corresponding to the second node address information to establish a subscription relationship, and reading the data to be processed from the second data node.
As an optional embodiment, the apparatus further comprises:
the second progress acquisition request receiving module is configured to execute receiving of a second progress acquisition request transmitted by the consumer and forwarded based on the second data node;
the second data processing progress acquisition module is configured to determine a second data processing progress corresponding to the consumer from the source data cluster under the condition that the migration state information of the data to be processed is data migration completion;
and the second data processing progress sending module is configured to execute forwarding of a second data processing progress to the consumer based on the second data node, wherein the second data processing progress is used for instructing the consumer to determine data to be processed from the data stored in the second data node.
As an optional embodiment, the apparatus further comprises:
a third address request receiving module configured to execute receiving a third address obtaining request sent by the producer and forwarded based on the source data node;
a third node address determining module, configured to determine, in the target data cluster, third node address information corresponding to a third address obtaining request based on the address information corresponding to the source data cluster when the migration state information of the target data is in the data migration process;
and the third node address information sending module is configured to execute forwarding of third node address information to the producer based on the source data node, wherein the third node address information is used for indicating the producer to execute data writing to a third data node corresponding to the third node address information.
As an optional embodiment, the apparatus further comprises:
the fourth address request receiving module is configured to execute receiving of a fourth address acquisition request sent by the producer and forwarded based on the source data node;
a fourth node address determining module, configured to determine fourth node address information corresponding to a fourth address obtaining request in the target data cluster based on the address information corresponding to the target data cluster when the migration state information of the target data is that data migration is completed;
and the fourth node address information sending module is configured to forward fourth node address information to the producer based on the source data node, wherein the fourth node address information is used for indicating the producer and a fourth data node corresponding to the fourth node address information to establish a publishing relationship, and data writing is performed on the fourth data node.
As an alternative embodiment, the write function shutdown module includes:
the data processing state determining unit is configured to determine data processing states respectively corresponding to the source data cluster and the target data cluster under the condition that the data volume of the synchronized data in the target data cluster is larger than a preset data volume;
a data synchronization state determination unit configured to perform determining a data synchronization state corresponding to the target data cluster;
and the writing function closing unit is configured to close the data writing function corresponding to the source data cluster under the condition that the data processing state is the target processing state and the data synchronization state is the target synchronization state.
FIG. 16 illustrates a cluster data migration apparatus, according to an example embodiment, the apparatus including;
a current migration data determining module 1610, configured to execute obtaining current migration data from target data in the source data cluster based on a data migration progress corresponding to the migrated data, where the current migration data is difference data between the data migration progress and a data write progress corresponding to the source data cluster;
and the data migration module 1620 is configured to update the migrated data and the data migration progress based on the current migration data, in the target data migration process, the read-write address corresponding to the target data is address information corresponding to the source data cluster, and after the target data migration is finished, the read-write address corresponding to the target data is address information corresponding to the target data cluster.
With regard to the apparatus in the above-described embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.
FIG. 17 is a block diagram illustrating a cluster data migration system in accordance with an exemplary embodiment. Referring to fig. 17, the system includes;
the system comprises: server cluster 1710, source cluster 1720, and target cluster 1730;
the server cluster 1710 includes a plurality of servers and a control server electrically connected to each of the plurality of servers;
the server cluster is electrically connected with the source data cluster based on the control server, and the server cluster is electrically connected with the target data cluster based on the control server;
the target data cluster is configured to execute the target data acquisition from the source data cluster in response to the data migration instruction;
the control server is configured to set a read-write address corresponding to target data as address information corresponding to the source data cluster in the process of migrating the target data in the source data cluster to the target data cluster;
the control server is configured to close a data writing function corresponding to the source data cluster under the condition that the migration time of the target data is greater than a preset migration time, wherein the preset migration time is consumption delay time of the data in the source data cluster;
the control server is configured to update the read-write address corresponding to the target data to the address information corresponding to the target data cluster under the condition that the target data is completely migrated to the target data cluster.
As an optional embodiment, the system further comprises: a consumer;
the source data cluster comprises a plurality of source data nodes and a source cluster control node, and the source cluster control node is electrically connected with each source data node in the plurality of source data nodes;
the control server comprises a first address request receiving module, a first node address determining module and a first node address information sending module, wherein the first address request receiving module is electrically connected with the first node address determining module, and the first node address determining module is electrically connected with the first node address information sending module;
the consumer is configured to perform sending a first address fetch request to the source data node
The source data node is configured to perform forwarding of the first address acquisition request to the source cluster control node;
the source cluster control node is configured to execute sending a first address acquisition request to the control server;
the first address request receiving module is configured to execute receiving a first address acquisition request sent by a consumer and forwarded based on a source data node in a source data cluster;
the first node address determining module is configured to determine, in the source data cluster, first node address information of data to be processed corresponding to the first address obtaining request, based on address information corresponding to the source data cluster, when the migration state information of the target data is in a data migration process;
the first node address information sending module is configured to execute sending the first node address information to the source cluster control node;
the source cluster control node is configured to perform forwarding of the first node address information to the source data node;
the source data node is configured to perform forwarding of the first node address information to the consumer;
the consumer is configured to read the data to be processed from the first data node corresponding to the first node address information.
As an optional embodiment, the control server further includes a first progress acquisition request receiving module, a first data processing progress acquisition module, and a first data processing progress sending module;
the first progress acquisition request receiving module is electrically connected with the first data processing progress acquisition module;
the first data processing progress acquisition module is electrically connected with the first data processing progress sending module:
the consumer is configured to perform sending a first progress acquisition request to the first data node;
the first data node is configured to execute a first progress acquisition request forwarded to the source cluster control node;
the source cluster control node is configured to perform forwarding of the first progress acquisition request to the control server;
the first progress acquisition request receiving module is configured to execute receiving of a first progress acquisition request transmitted by a consumer and forwarded based on the first data node;
the first data processing progress acquisition module is configured to determine a first data processing progress corresponding to a consumer from the source data cluster under the condition that the migration state information of the data to be processed is in the data migration process;
the first data processing progress sending module is configured to execute sending the first data processing progress to the source cluster control node;
the source cluster control node is configured to perform sending a first data processing schedule to the first data node;
the first data node is configured to perform forwarding the first data processing progress to the consumer;
the consumer is configured to perform determining data to be processed from the data stored at the first data node based on the first data processing schedule.
As an optional embodiment, the control server further comprises: the second node address information sending module is used for sending the second node address information to the second node address request receiving module;
the second address request receiving module is electrically connected with the second node address determining module;
the second node address determining module is electrically connected with the second node address information sending module;
the target data cluster comprises a plurality of target data nodes and a target data cluster control node, and the target data cluster control node is electrically connected with each target data node in the plurality of target data nodes;
the consumer is configured to perform sending a second address fetch request to the source data node;
the source data node is configured to execute receiving a second address acquisition request sent by a consumer and forward the second address acquisition request to the source cluster control node;
the source cluster control node is configured to perform forwarding of the second address fetch request to the control server;
the second address request receiving module is configured to execute receiving a second address acquisition request sent by a consumer and forwarded based on the source data node;
the second node address determining module is configured to determine second node address information of the to-be-processed data corresponding to the second address acquisition request in the target data cluster based on address information corresponding to the target data cluster under the condition that migration state information of the data corresponding to the to-be-processed data is data migration completed;
the second node address information sending module is configured to execute sending the second node address information to the source cluster control node;
the source cluster control node is configured to perform forwarding of the second node address information to the source data node;
the source data node is configured to perform forwarding of the second node address information to the consumer;
the consumer is configured to establish a subscription relationship with a second data node corresponding to the second node address information, and read the data to be processed from the second data node.
As an optional embodiment, the control server further comprises: the system comprises a second progress acquisition request receiving module, a second data processing progress acquisition module and a second data processing progress sending module;
the second progress acquisition request receiving module is electrically connected with the second data processing progress acquisition module;
the second data processing progress acquisition module is electrically connected with the second data processing progress sending module;
the consumer is configured to execute sending a second progress acquisition request to the second data node;
the second data node is configured to execute forwarding of a second progress acquisition request to the target cluster control node;
the target cluster control node is configured to execute forwarding of the second progress acquisition request to the control server;
the second progress acquisition request receiving module is configured to execute receiving of a second progress acquisition request transmitted by the consumer and forwarded based on the second data node;
the second data processing progress acquisition module is configured to determine a second data processing progress corresponding to the consumer from the source data cluster under the condition that the migration state information of the data to be processed is data migration completion;
the second data processing progress sending module is configured to execute sending of a second data processing progress to the target cluster control node;
the target cluster control node is configured to perform forwarding of the second data processing schedule to the second data node;
the second data node is configured to perform forwarding the second data processing progress to the consumer;
the consumer is configured to perform determining data to be processed from the data stored by the second data node based on the second data processing schedule.
As an optional embodiment, the system further comprises: a producer;
the control server further includes: the third address request receiving module, the third node address determining module and the third node address information sending module;
the third address request receiving module is electrically connected with the third node address determining module;
the third node address determining module is electrically connected with the third node address information sending module;
the producer is configured to perform sending a third address acquisition request to the source data node;
the source data node is configured to perform forwarding of the third address acquisition request to the source cluster control node;
the source cluster control node is configured to perform forwarding of the third address acquisition request to the control server;
the third address request receiving module is configured to execute receiving a third address acquisition request sent by the producer and forwarded based on the source data node;
the third node address determining module is configured to determine, in the target data cluster, third node address information corresponding to a third address obtaining request based on the address information corresponding to the source data cluster, when the migration state information of the target data is in the data migration process;
the third node address information sending module is configured to execute sending the third node address information to the source cluster control node;
the source cluster control node is configured to perform forwarding of the third node address information to the source data node;
the source data node is configured to perform forwarding of the third node address information to the producer;
the producer is configured to perform data writing to a third data node corresponding to the third node address information.
As an optional embodiment, the control server further comprises: the fourth node address information sending module is used for sending the fourth node address information to the fourth node address determining module;
the fourth address request receiving module is electrically connected with the fourth node address determining module;
the fourth node address determining module is electrically connected with the fourth node address information sending module;
the producer is configured to perform sending a fourth address acquisition request to the source data node;
the source data node is configured to perform forwarding of the fourth address acquisition request to the source cluster control node;
the source cluster control node is configured to perform forwarding of the fourth address acquisition request to the control server;
the fourth address request receiving module is configured to execute receiving a fourth address acquisition request sent by the producer and forwarded based on the source data node;
the fourth node address determining module is configured to determine fourth node address information corresponding to a fourth address obtaining request in the target data cluster based on the address information corresponding to the target data cluster under the condition that the migration state information of the target data is that data migration is completed;
the fourth node address information sending module is configured to execute sending the fourth node address information to the source data control node;
the source data control node is configured to perform forwarding of the fourth node address information to the source data node;
the source data node is configured to perform forwarding of the fourth node address information to the producer;
the production is configured to perform the establishment of the publishing relationship with the fourth data node corresponding to the fourth node address information, and perform the data writing to the fourth data node.
As an optional embodiment, the system further comprises:
the target data cluster is configured to execute data migration progress-based execution and obtain current migration data from the target data, wherein the current migration data is difference data between the data migration progress in the target data cluster and the data writing progress corresponding to the data in the source data cluster.
As an optional embodiment, the control server further comprises: a write function shutdown module, the write function shutdown module comprising: the device comprises a data processing state determining unit, a data synchronization state determining unit and a writing function closing unit;
the data processing state determining unit is electrically connected with the data synchronization state determining unit;
the data synchronization state determining unit is electrically connected with the writing function closing unit;
the data processing state determining unit is configured to determine data processing states respectively corresponding to the source data cluster and the target data cluster under the condition that the migration time of the target data is greater than the preset migration time;
the data synchronization state determining unit is configured to determine a data migration state corresponding to the target data cluster;
and the writing function closing unit is configured to close the data writing function corresponding to the source data cluster when the data processing state is the target processing state and the data migration state is the target migration state.
As an alternative embodiment, the target data cluster is configured to perform the obtaining the target data from the source data cluster in response to the data migration instruction, including:
the target data cluster is configured to execute data migration instruction responding, and current migration data is determined from the target data based on the data migration progress, wherein the current migration data is difference data between the data migration progress in the target data cluster and the data writing progress corresponding to the data in the source data cluster;
the target data cluster is configured to perform migration of current migration data from the source data cluster into the target data cluster.
With regard to the system in the above embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.
FIG. 18 is a block diagram illustrating an electronic device, which may be a server, for cluster data migration according to an example embodiment, an internal structure of which may be as shown in FIG. 18. The electronic device includes a processor, a memory, and a network interface connected by a system bus. Wherein the processor of the electronic device is configured to provide computing and control capabilities. The memory of the electronic equipment comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The network interface of the electronic device is used for connecting and communicating with an external terminal through a network. The computer program is executed by a processor to implement a cluster data migration method.
Those skilled in the art will appreciate that the configuration shown in fig. 18 is a block diagram of only a portion of the configuration associated with the disclosed aspects and does not constitute a limitation on the electronic device to which the disclosed aspects apply, and that a particular electronic device may include more or less components than those shown in the figures, or combine certain components, or have a different arrangement of components.
In an exemplary embodiment, a computer-readable storage medium comprising instructions, such as the memory 1804 comprising instructions, executable by the processor 1820 of the electronic device 1800 to perform the above-described method is also provided. Alternatively, the computer readable storage medium may be a ROM, a Random Access Memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.
In an exemplary embodiment, a computer program product is also provided, comprising a computer program which, when executed by a processor, implements the cluster data migration method described above.
Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This application is intended to cover any variations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.
It will be understood that the present disclosure is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims (10)

1. A method for cluster data migration, the method comprising:
in the process of migrating target data to a target data cluster in a source data cluster, setting a read-write address corresponding to the target data as address information corresponding to the source data cluster;
under the condition that the migration time of the target data is greater than a preset migration time, closing a data writing function corresponding to the source data cluster, wherein the preset migration time is consumption delay time of data in the source data cluster;
and under the condition that the target data is completely migrated to the target data cluster, updating the read-write address corresponding to the target data into the address information corresponding to the target data cluster.
2. The method of cluster data migration according to claim 1, wherein before said target data is completely migrated to said target data cluster, said method further comprises:
receiving a first address acquisition request sent by a consumer and forwarded based on a source data node in the source data cluster;
determining, in the source data cluster, first node address information of to-be-processed data corresponding to the first address acquisition request based on address information corresponding to the source data cluster when the migration state information of the target data is in a data migration process;
forwarding the first node address information to the consumer based on the source data node, where the first node address information is used to instruct the consumer to read the to-be-processed data from a first data node corresponding to the first node address information.
3. The cluster data migration method of claim 2, wherein after sending the first node address information to the data node of the source data cluster and forwarding the first node address information to the consumer through the source data node, the method further comprises:
receiving a first progress acquisition request sent by the consumer and forwarded based on the first data node;
determining a first data processing progress corresponding to the consumer from the source data cluster under the condition that the migration state information of the data to be processed is in the data migration process;
forwarding the first data processing progress to the consumer based on the first data node, the first data processing progress being used for instructing the consumer to determine the data to be processed from the data stored by the first data node.
4. A method for cluster data migration, the method comprising:
acquiring current migration data from target data in a source data cluster based on a data migration progress corresponding to the migrated data, wherein the current migration data is difference data between the data migration progress and a data writing progress corresponding to the source data cluster;
updating the migrated data and the data migration progress based on the current migration data, wherein in the target data migration process, the read-write address corresponding to the target data is address information corresponding to the source data cluster, and after the target data migration is finished, the read-write address corresponding to the target data is address information corresponding to the target data cluster.
5. A cluster data migration control apparatus, the apparatus comprising:
the data address setting module is configured to set a read-write address corresponding to target data as address information corresponding to a source data cluster in the process of migrating the target data in the source data cluster to the target data cluster;
a write function closing module configured to close a data write function corresponding to the source data cluster when the time for migrating the target data is longer than a preset migration time, where the preset migration time is a consumption delay time of data in the source data cluster;
and the data address updating module is configured to update the read-write address corresponding to the target data to the address information corresponding to the target data cluster under the condition that the target data is completely migrated to the target data cluster.
6. An apparatus for cluster data migration, the apparatus comprising:
the current migration data determining module is configured to execute data migration progress corresponding to migrated data, and acquire current migration data from target data in a source data cluster, where the current migration data is difference data between the data migration progress and a data writing progress corresponding to the source data cluster;
and the data migration module is configured to update the migrated data and the data migration progress based on the current migration data, in the target data migration process, the read-write address corresponding to the target data is address information corresponding to the source data cluster, and after the target data migration is finished, the read-write address corresponding to the target data is address information corresponding to the target data cluster.
7. A cluster data migration system, the system comprising: a server cluster, a source data cluster and a target data cluster;
the server cluster comprises a plurality of servers and a control server, wherein the control server is electrically connected with each server in the plurality of servers;
the server cluster is electrically connected with the source data cluster based on the control server, and the server cluster is electrically connected with the target data cluster based on the control server;
the target data cluster is configured to execute target data acquisition from the source data cluster in response to a data migration instruction;
the control server is configured to set a read-write address corresponding to target data as address information corresponding to a source data cluster in a process of migrating the target data in the source data cluster to the target data cluster;
the control server is configured to execute a data writing function corresponding to the source data cluster when the time for migrating the target data is greater than a preset migration time, where the preset migration time is a consumption delay time of data in the source data cluster;
the control server is configured to update the read-write address corresponding to the target data to the address information corresponding to the target data cluster when the target data is completely migrated to the target data cluster.
8. An electronic device, comprising:
a processor;
a memory for storing the processor-executable instructions;
wherein the processor is configured to execute the instructions to implement the cluster data migration control method of any of claims 1 to 4.
9. A computer-readable storage medium, wherein instructions in the computer-readable storage medium, when executed by a processor of an electronic device, enable the electronic device to perform the cluster data migration control method of any of claims 1 to 4.
10. A computer program product comprising a computer program, characterized in that the computer program realizes the cluster data migration control method of any one of claims 1 to 4 when executed by a processor.
CN202111258167.4A 2021-10-27 2021-10-27 Cluster data migration method and device, electronic equipment and storage medium Pending CN113946293A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202111258167.4A CN113946293A (en) 2021-10-27 2021-10-27 Cluster data migration method and device, electronic equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202111258167.4A CN113946293A (en) 2021-10-27 2021-10-27 Cluster data migration method and device, electronic equipment and storage medium

Publications (1)

Publication Number Publication Date
CN113946293A true CN113946293A (en) 2022-01-18

Family

ID=79332805

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202111258167.4A Pending CN113946293A (en) 2021-10-27 2021-10-27 Cluster data migration method and device, electronic equipment and storage medium

Country Status (1)

Country Link
CN (1) CN113946293A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114615263A (en) * 2022-02-10 2022-06-10 深圳市小满科技有限公司 Cluster online migration method, device, equipment and storage medium

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH10105522A (en) * 1996-06-27 1998-04-24 Hewlett Packard Co <Hp> Multicomputer system
CN102917072A (en) * 2012-10-31 2013-02-06 北京奇虎科技有限公司 Device, system and method for carrying out data migration between data server clusters
CN102982085A (en) * 2012-10-31 2013-03-20 北京奇虎科技有限公司 System and method of data migration
CN108628874A (en) * 2017-03-17 2018-10-09 北京京东尚科信息技术有限公司 Method, apparatus, electronic equipment and the readable storage medium storing program for executing of migrating data
CN113438275A (en) * 2021-05-27 2021-09-24 众安在线财产保险股份有限公司 Data migration method and device, storage medium and data migration equipment

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH10105522A (en) * 1996-06-27 1998-04-24 Hewlett Packard Co <Hp> Multicomputer system
CN102917072A (en) * 2012-10-31 2013-02-06 北京奇虎科技有限公司 Device, system and method for carrying out data migration between data server clusters
CN102982085A (en) * 2012-10-31 2013-03-20 北京奇虎科技有限公司 System and method of data migration
CN108628874A (en) * 2017-03-17 2018-10-09 北京京东尚科信息技术有限公司 Method, apparatus, electronic equipment and the readable storage medium storing program for executing of migrating data
CN113438275A (en) * 2021-05-27 2021-09-24 众安在线财产保险股份有限公司 Data migration method and device, storage medium and data migration equipment

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114615263A (en) * 2022-02-10 2022-06-10 深圳市小满科技有限公司 Cluster online migration method, device, equipment and storage medium

Similar Documents

Publication Publication Date Title
US10719260B2 (en) Techniques for storing and retrieving data from a computing device
CN105426439B (en) Metadata processing method and device
ES2703901T3 (en) Distributed file system through consensus nodes
CN108829720B (en) Data processing method and device
CN111368002A (en) Data processing method, system, computer equipment and storage medium
US20110066595A1 (en) Database server, replication server and method for replicating data of a database server by at least one replication server
CN105493474B (en) System and method for supporting partition level logging for synchronizing data in a distributed data grid
US20190347167A1 (en) Primary Node-Standby Node Data Transmission Method, Control Node, and Database System
CN110099084B (en) Method, system and computer readable medium for ensuring storage service availability
CN111212100A (en) Synchronization method and system for publishing and subscribing message queue
CN112612799B (en) Data synchronization method and terminal
CA3131954C (en) Information synchronization method, system, apparatus, computer device and storage medium
CN103986694A (en) Control method of multi-replication consistency in distributed computer data storing system
CN113055430A (en) Data synchronization method and related equipment
WO2023185934A1 (en) Data processing method and device
CN113946293A (en) Cluster data migration method and device, electronic equipment and storage medium
CN112416884A (en) Data synchronization method and system
CN112069152A (en) Database cluster upgrading method, device, equipment and storage medium
CN107404511B (en) Method and device for replacing servers in cluster
CN115629909A (en) Service data processing method and device, electronic equipment and storage medium
CN114936095A (en) Partition expansion and reduction method and system
CN112052104A (en) Message queue management method based on multi-computer-room realization and electronic equipment
CN112559568A (en) Virtual article determination method and device and computer readable storage medium
CN111638980A (en) Message processing method, device and system based on memory mapping and storage medium
CN116991815B (en) Log collection method, device, equipment and medium of distributed storage system

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination