CN113051103A

CN113051103A - Data processing method and device and electronic equipment

Info

Publication number: CN113051103A
Application number: CN201911372918.8A
Authority: CN
Inventors: 宋迎春
Original assignee: China Mobile Communications Group Co Ltd; China Mobile Group Henan Co Ltd
Current assignee: China Mobile Communications Group Co Ltd; China Mobile Group Henan Co Ltd
Priority date: 2019-12-27
Filing date: 2019-12-27
Publication date: 2021-06-29
Anticipated expiration: 2039-12-27
Also published as: CN113051103B

Abstract

The embodiment of the invention provides a data processing method, a data processing device and electronic equipment, wherein the method is applied to a target data processing node, and comprises the following steps: under the condition of receiving node change information of a data processing node sent by a distributed coordination server, carrying out fragmentation processing on target data to be processed based on the node change information to obtain a first fragmentation result, wherein the first data processing node is different from the target data processing node and receives a second fragmentation result sent by the first data processing node, the second fragmentation result is a result of the first data processing node carrying out fragmentation processing on the target data based on the node change information, and the first data is processed under the condition that the first fragmentation result is matched with the second fragmentation result. By the method, repeated processing or missing processing of the target data can be avoided, and the data processing efficiency is improved.

Description

Data processing method and device and electronic equipment

Technical Field

The present invention relates to the field of computer technologies, and in particular, to a data processing method and apparatus, and an electronic device.

Background

With the continuous development of mobile communication technology, the amount of mobile communication services is increasing, and in order to ensure the efficiency of service processing, operators generally adopt an asynchronous scheduling mechanism to process services, that is, to perform distribution scheduling on data to be processed through a distributed coordination server.

At present, when the distributed scheduling is performed through the distributed coordination server, a hot backup mechanism may be adopted to ensure the continuity of data processing. For example, the service data to be processed may be sent to a plurality of main devices for processing based on the distributed coordination server, and meanwhile, for each main device, there may also be one backup device, and when the main device fails, the distributed coordination server may start the backup device to replace the main device for processing the data.

However, when the hot backup mechanism is used to process the service data, the following problems exist: secondly, if the signal between the main device and the distributed coordination server is disconnected, the distributed coordination server can start the backup device to continue processing the data, but the main device may not be in failure at this time and still continues processing the data, so that when a hot backup mechanism is adopted to process the service data, the problems of high risk of data repeated processing and low data processing efficiency exist.

Disclosure of Invention

Embodiments of the present invention provide a data processing method, an apparatus, and an electronic device, so as to solve the problems that when a hot backup mechanism is used to process business data in the prior art, the risk of data duplicate processing is high, and the data processing efficiency is low.

To solve the above technical problem, the embodiment of the present invention is implemented as follows:

in a first aspect, an embodiment of the present invention provides a data processing method, where the method is applied to a target data processing node, and the method includes:

under the condition that node change information of a data processing node sent by a distributed coordination server is received, based on the node change information, fragmentation processing is carried out on target data to be processed to obtain a first fragmentation result, wherein the first fragmentation result comprises first data to be processed locally and second data to be processed by the first data processing node, and the first data processing node is different from the target data processing node;

receiving a second fragmentation result sent by the first data processing node, where the second fragmentation result is a result of the first data processing node performing fragmentation processing on the target data based on the node change information, and the second fragmentation result includes third data to be processed by the first data processing node and fourth data to be processed by the target data processing node;

and processing the first data under the condition that the first slicing result is matched with the second slicing result.

Optionally, the performing, based on the node change information, fragmentation processing on the target data to be processed to obtain a first fragmentation result includes:

determining a second data processing node currently used for processing the target data based on the node change information;

determining data processing nodes except the target data processing node in the second data processing nodes as the first data processing node;

determining first location information of the target data processing node in the second data processing node and second location information of the first data processing node in the second data processing node;

and based on the first position information and the second position information, carrying out fragmentation processing on the target data to be processed to obtain the first fragmentation result.

Optionally, the processing the first data in the case that the first slicing result matches the second slicing result includes:

receiving third position information sent by the first data processing node, wherein the third position information is position information of the first data processing node in the second data processing node, which is determined by the first data processing node based on the node change information;

and processing the first data under the condition that the second position information is matched with the third position information and the first slicing result is matched with the second slicing result.

Optionally, the determining first location information of the target data processing node in the second data processing node and second location information of the first data processing node in the second data processing node includes:

acquiring a self-increment value of each second data processing node, wherein the self-increment value is determined by the starting time of the data processing node;

determining first location information of the target data processing node in the second data processing node based on the self-increment value of the target data processing node and the self-increment values of other second data processing nodes;

determining second location information of the first data processing node in the second data processing node based on the self-increment value of the first data processing node and the self-increment values of the other second data processing nodes.

Optionally, the performing, based on the first location information of the target data processing node and the second location information of the first data processing node, fragmentation processing on the target data to be processed to obtain the first fragmentation result includes:

performing modulo processing on the number of the target data to be processed based on the number of the second data processing nodes;

and performing fragmentation processing on the target data to be processed based on the modulus processing result, the first position information of the target data processing node and the second position information of the first data processing node to obtain the first fragmentation result.

Optionally, the number of the first data processing nodes is smaller than a preset node number threshold.

Optionally, the target data processing node and the first data processing node are data processing nodes with data processing amount smaller than a preset data processing amount threshold.

In a second aspect, an embodiment of the present invention provides an apparatus for processing data, where the apparatus includes:

the system comprises a fragmentation module, a data processing node and a target data processing node, wherein the fragmentation module is used for carrying out fragmentation processing on target data to be processed based on node change information under the condition of receiving the node change information of the data processing node sent by a distributed coordination server to obtain a first fragmentation result, the first fragmentation result comprises first data to be processed locally and second data to be processed of the first data processing node, and the first data processing node is different from the target data processing node;

a receiving module, configured to receive a second fragmentation result sent by the first data processing node, where the second fragmentation result is a result of the first data processing node performing fragmentation processing on the target data based on the node change information, and the second fragmentation result includes third data to be processed by the first data processing node and fourth data to be processed by the target data processing node;

and the processing module is used for processing the first data under the condition that the first slicing result is matched with the second slicing result.

Optionally, the slicing module includes:

a first determining unit configured to determine, based on the node change information, a second data processing node currently used for processing the target data;

a second determining unit, configured to determine, as the first data processing node, a data processing node other than the target data processing node in the second data processing nodes;

a third determining unit, configured to determine first location information of the target data processing node in the second data processing node and second location information of the first data processing node in the second data processing node;

and the slicing unit is used for carrying out slicing processing on the target data to be processed based on the first position information and the second position information to obtain the first slicing result.

Optionally, the processing module includes:

a location receiving unit, configured to receive third location information sent by the first data processing node, where the third location information is location information of the first data processing node in the second data processing node, which is determined by the first data processing node based on the node change information;

and the data processing unit is used for processing the first data under the condition that the second position information is matched with the third position information and the first slicing result is matched with the second slicing result.

Optionally, the third determining unit is configured to:

Optionally, the slicing unit is configured to:

In a third aspect, an embodiment of the present invention provides an electronic device, which includes a processor, a memory, and a computer program stored on the memory and executable on the processor, where the computer program, when executed by the processor, implements the steps of the data processing method provided in the foregoing embodiments.

In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the computer program implements the steps of the data processing method provided in the foregoing embodiment.

As can be seen from the above technical solutions provided by the embodiments of the present invention, in the case of receiving node change information of a data processing node sent by a distributed coordination server, based on the node change information, the embodiments of the present invention perform fragmentation processing on target data to be processed to obtain a first fragmentation result, where the first fragmentation result includes first data to be processed locally and second data to be processed by a first data processing node, and the first data processing node is different from the target data processing node, and receives a second fragmentation result sent by the first data processing node, where the second fragmentation result is a result of the first data processing node performing fragmentation processing on the target data based on the node change information, and the second fragmentation result includes third data to be processed by the first data processing node and fourth data to be processed by the target data processing node, and in the case that the first fragmentation result matches the second fragmentation result, the first data is processed. Therefore, through the matching detection of the first slicing result and the second slicing result, the data processed by the target data processing node is different from the data processed by the first data processing node, the problem of repeated data processing is avoided, meanwhile, the processing integrity of the target data can be ensured, namely, the problem of data omission processing is avoided, in addition, the data needing to be processed locally can be determined immediately through the node change information, the data processing is carried out, and the data processing efficiency is improved.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below, it is obvious that the drawings in the following description are only some embodiments described in the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.

FIG. 1 is a schematic flow chart of a data processing method according to the present invention;

FIG. 2 is a schematic diagram of a node variation according to the present invention;

FIG. 3 is a flow chart illustrating another data processing method according to the present invention;

FIG. 4 is a schematic diagram of a data processing apparatus according to the present invention;

fig. 5 is a schematic structural diagram of an electronic device according to the present invention.

Detailed Description

The embodiment of the invention provides a data processing method and device and electronic equipment.

In order to make those skilled in the art better understand the technical solution of the present invention, the technical solution in the embodiment of the present invention will be clearly and completely described below with reference to the drawings in the embodiment of the present invention, and it is obvious that the described embodiment is only a part of the embodiment of the present invention, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

Example one

As shown in fig. 1, an execution subject of the method may be a target data processing node, and the target data processing node may be a server, where the server may be an independent server, or a server cluster composed of multiple servers. The method may specifically comprise the steps of:

in S102, under the condition that the node change information of the data processing node sent by the distributed coordination server is received, based on the node change information, the target data to be processed is fragmented, and a first fragmentation result is obtained.

The first fragmentation result may include first data to be processed locally and second data to be processed by a first data processing node, the first data processing node may be different from the target data processing node, the target data processing node and the first data processing node are data processing nodes connected to the distributed coordination server, and may cooperate to complete a processing task on the target data, and the target data may include multiple sets of data to be processed.

In implementation, with the continuous development of mobile communication technology, the amount of mobile communication traffic is increasing, and in order to ensure the efficiency of service processing, operators generally use an asynchronous scheduling mechanism to process services, that is, to perform allocation scheduling on data to be processed through a distributed coordination server. At present, when the distributed scheduling is performed through the distributed coordination server, a hot backup mechanism may be adopted to ensure the continuity of data processing. For example, the service data to be processed may be sent to a plurality of main devices for processing based on the distributed coordination server, and meanwhile, for each main device, there may also be one backup device, and when the main device fails, the distributed coordination server may start the backup device to replace the main device for processing the data.

However, when the hot backup mechanism is used to process the service data, the following problems exist: secondly, if the signal between the main device and the distributed coordination server is disconnected, the distributed coordination server may start the backup device to continue processing data, but the main device may not be in failure at this time, and still continue processing data, so that when a hot backup mechanism is used to process service data, there are problems of high risk of data reprocessing and low data processing efficiency, and therefore, another implementation scheme provided by the embodiment of the present invention may specifically include the following contents:

the distributed coordination server can be connected with a plurality of data processing nodes, and when detecting that a newly added data processing node or a disconnected data processing node exists, the distributed coordination server can send node change information to each data processing node connected with the distributed coordination server, that is, after the data processing node changes, a target data processing node can receive the node change information of the data processing node sent by the distributed coordination server.

For example, as shown in fig. 2, the distributed coordination server may be connected to the data processing node 1, the data processing node 2, and the data processing node 3, wherein the target data processing node may be any one of the three data processing nodes, and it is assumed that there is a data processing node 4 added to the current data processing task, that is, the data processing node 4 may also establish a connection with the distributed coordination server. At this time, the distributed coordination server may send the node change information (i.e., the new data processing node 4) to the data processing nodes 1, 2, and 3, respectively, i.e., the target data processing node (assumed to be the data processing node 1) may receive the node change information sent by the distributed coordination server. The target data processing node may be the data processing node 4, that is, the distributed coordination server may transmit node change information (i.e., the additional data processing node 1, the data processing node 2, and the data processing node 3) with respect to the data processing node 4.

After receiving the node change information, the target data processing node may perform fragment processing on the target data to be processed. During the fragmentation processing, the target data may be fragmented based on a preset fragmentation processing rule, for example, the target data may be equally divided into multiple groups of data according to the number of the first data processing nodes and the target data processing nodes, and the multiple groups of data are allocated to the target data processing nodes and the first data processing nodes. If the target data includes 6 sets of data and the number of the first data processing nodes is 2, the 6 sets of data can be equally divided into 3, wherein the target data processing nodes can process the data (i.e. the first data) of the 1 st and 2 nd sets, the first data processing node 1 can process the data (i.e. the second data 1) of the 3 rd and 4 th sets, and the first data processing node 2 can process the data (i.e. the second data 2) of the 5 th and 6 th sets, that is, the first fragmentation result is obtained.

In addition, the target data can be subjected to fragmentation processing according to the attribute value of the specified attribute. For example, the target data may include multiple groups of communication numbers of the users, and the target data may be fragmented according to a last digit value of the communication numbers to obtain a first fragmentation result. For example, the target data with the last bit of the communication number being 0 or 1 may be determined as the first data to be processed locally, and the target data with the last bit of the communication number being 2 or 3 may be determined as the second data to be processed by the first data processing node.

The method for performing fragment processing on the target data is an optional and realizable fragment processing method, and in an actual application scenario, there may be a plurality of different fragment processing methods, and different fragment processing methods may be selected according to different actual application scenarios, which is not specifically limited in the embodiment of the present invention.

In addition, the distributed coordination server may further receive data processing conditions sent by the target data processing node and the first data processing node, summarize the received data processing conditions, and send the summarized data processing conditions to the target data processing node and the first data processing node, so that the target data processing node (or the first data processing node) determines target data to be processed according to the received data processing conditions.

For example, before a data processing node changes, a total of 100 pieces of data need to be processed, and the 100 pieces of data may be sent to the data processing node 1, the data processing node 2, the data processing node 3, and the data processing node 4 respectively for processing, where each data processing node may process 25 pieces of data. When a data processing node 2 fails, assuming that each data processing node has completed processing 10 pieces of data and sends the data processing condition to the distributed coordination server, the distributed coordination server may send the data processing condition and node change information to a target data processing node (such as data processing node 1), and the target data processing node may determine, based on the data processing condition, that the target data is the remaining 60 pieces of data, and then perform fragmentation processing on the 60 pieces of data to obtain a first fragmentation result.

In S104, the second fragmentation result sent by the first data processing node is received.

The second fragmentation result may be a result of the first data processing node performing fragmentation processing on the target data based on the node change information, where the second fragmentation result includes third data to be processed by the first data processing node and fourth data to be processed by the target data processing node.

In implementation, as in the method of performing fragmentation processing on target data in S102, the first data processing node may perform fragmentation processing on the target data and obtain a second fragmentation result when receiving the node change information sent by the distributed coordination server, and then send the second fragmentation result to the target data processing node, that is, the target data processing node may receive the second fragmentation result sent by the first data processing node.

In S106, the first data is processed in the case where the first fragmentation result matches the second fragmentation result.

In implementation, when the first data matches with the third data and the second data matches with the fourth data, the target data processing node may process the first data (i.e., the third data), and the first data processing node may also process the second data (i.e., the fourth data), so as to ensure complete processing of the target data and avoid repeated processing of the target data.

The embodiment of the invention provides a data processing method, which comprises the steps of carrying out fragmentation processing on target data to be processed based on node change information under the condition of receiving the node change information of a data processing node sent by a distributed coordination server to obtain a first fragmentation result, wherein the first fragmentation result comprises the first data to be processed locally and the second data to be processed by a first data processing node, the first data processing node is different from the target data processing node, receiving a second fragmentation result sent by the first data processing node, the second fragmentation result is a result of the first data processing node carrying out fragmentation processing on the target data based on the node change information, the second fragmentation result comprises the third data to be processed by the first data processing node and the fourth data to be processed by the target data processing node, and under the condition that the first fragmentation result is matched with the second fragmentation result, the first data is processed. Therefore, through the matching detection of the first slicing result and the second slicing result, the data processed by the target data processing node is different from the data processed by the first data processing node, the problem of repeated data processing is avoided, meanwhile, the processing integrity of the target data can be ensured, namely, the problem of data omission processing is avoided, in addition, the data needing to be processed locally can be determined immediately through the node change information, the data processing is carried out, and the data processing efficiency is improved.

Example two

As shown in fig. 3, an execution subject of the method may be a target data processing node, and the target data processing node may be a server, where the server may be an independent server, or a server cluster composed of multiple servers. The method may specifically comprise the steps of:

in S302, upon receiving the node change information of the data processing node sent by the distributed coordination server, a second data processing node currently used for processing the target data is determined based on the node change information.

In an implementation, as shown in fig. 2, based on the node change information, it may be determined that the second data processing node currently used for processing the target data may include data processing node 1, data processing node 2, data processing node 3, and data processing node 4, where the target data processing node may be any one of the four data processing nodes.

In S304, the data processing node other than the target data processing node among the second data processing nodes is determined as the first data processing node.

The number of the first data processing nodes may be less than a preset node number threshold, and the target data processing node and the first data processing node may be data processing nodes with data processing amount less than a preset data processing amount threshold.

In an implementation, as shown in fig. 2, the second data processing node may include a data processing node 1, a data processing node 2, a data processing node 3, and a data processing node 4, and if the data processing node 1 is a target data processing node, the data processing node 2, the data processing node 3, and the data processing node 4 may be the first data processing node.

In addition, at present, repeated processing of data can be avoided through a message system (such as a third-party message system) mode, but because the message system mode needs more devices to be configured, if the data volume of target data to be processed is small, the problem of high resource consumption exists when the message system mode is adopted for data processing, and meanwhile, the problem of long time delay of data processing exists, and the data processing efficiency is low.

In order to ensure the data processing efficiency of the target data processing node and the first data processing node, the target data processing node and the first data processing node may be data processing nodes with data processing capacities less than a preset data processing capacity threshold, for example, the processing magnitudes of the target data processing node and the first data processing node may be less than 1 million, that is, the target data processing node and the first data processing node may be lightweight data processing nodes.

In addition, since the start time of the second data processing nodes (including the target data processing node and the first data processing node) is different, and the time for each second data processing node to acquire the fragmentation results of other second data processing nodes is different, the longer the number of first data processing nodes is, the longer the data processing time is, so the number of first data processing nodes may be less than the preset node number threshold.

In S306, first location information of the target data processing node in the second data processing node and second location information of the first data processing node in the second data processing node are determined.

In practical applications, the processing manner of S306 may be various, and an alternative implementation manner is provided below, which may specifically refer to the following processing from step one to step three.

Step one, the self-increment value of each second data processing node is obtained.

Wherein the self-increment value can be determined by the start-up time of the data processing node.

In an implementation, the self-increment value of each second data processing node may be obtained according to the start-up time of each second data processing node (including the target data processing node and the first data processing node).

For example, as shown in fig. 2, the second data processing node may include data processing node 1, data processing node 2, data processing node 3, and data processing node 4, and the start time of each data processing node is as follows: the start time of the data processing node 1 (i.e., the target data processing node) is 00:00, the start time of the data processing node 2 is 00:05, the start time of the data processing node 3 is 00:01, and the start time of the data processing node 4 is 00:08, and the corresponding self-increment value can be determined according to the start time of each second data processing node. The self-increment value of the data processing node 1 may be 0 (i.e., the self-increment value of the target data processing node), the self-increment value of the data processing node 2 may be 5, the self-increment value of the data processing node 3 may be 1, and the self-increment value of the data processing node 4 may be 8.

And step two, determining first position information of the target data processing node in the second data processing node based on the self-increment value of the target data processing node and the self-increment values of other second data processing nodes.

And step three, determining second position information of the first data processing node in the second data processing node based on the self-increment value of the first data processing node and the self-increment values of other second data processing nodes.

In an implementation, the second data processing nodes may be sorted based on the self-increment value of each second data processing node, and the first location information of the target data processing node and the second location information of the first data processing node may be obtained.

For example, the self-increment value of the data processing node 1 may be 0 (i.e., the self-increment value of the target data processing node), the self-increment value of the data processing node 2 may be 5, the self-increment value of the data processing node 3 may be 1, and the self-increment value of the data processing node 4 may be 8, then the first location information of the target data processing node in the second data processing node may be location 1, the second location information of the first data processing node 1 (i.e., the data processing node 2) may be location 3, the second location information of the first data processing node 2 (i.e., the data processing node 3) may be location 2, and the second location information of the first data processing node 3 (i.e., the data processing node 4) may be location 4.

In addition, assuming that the data processing node 2 fails in the current second data processing node and the data processing node 5 is newly added to the data processing task, the new second data processing node may include the data processing node 1, the data processing node 3, the data processing node 4 and the data processing node 5, assuming that the start time of the data processing node 5 is 00:10, the self-increment value of the data processing node 5 may be 10, the 4 data processing nodes may be sorted according to the self-increment values of the 4 data processing nodes, and then the first location information of the target data processing node and the second location information of the first data processing node are obtained.

In S308, based on the first location information and the second location information, the target data to be processed is sliced to obtain a first slicing result.

In an implementation, for example, the target data may be split according to the location information of the second data processing node (including the target data processing node and the first data processing node), where, for example, the first location information of the target data processing node is location 2, the first data processing node includes the first data processing node 1 and the first data processing node 2, where the second location information of the first data processing node 1 is location 1, the second location information of the first data processing node 2 is location 3, and the target data is 13 pieces of data to be processed, the first splitting result may be: the first data of the target data processing node may be 2 nd to 4 th data in the target data, the second data 1 to be processed of the first data processing node 1 may be 1 st data in the target data, and the second data 2 to be processed of the first data processing node 2 may be 5 th to 13 th data in the target data.

In practical applications, the processing manner of S308 may be various, and an alternative implementation manner is provided below, which may specifically refer to the following processing from step one to step two.

And step one, performing modular processing on the number of target data to be processed based on the number of second data processing nodes.

And secondly, based on the result of the modulus processing, the first position information of the target data processing node and the second position information of the first data processing node, performing fragmentation processing on the target data to be processed to obtain a first fragmentation result.

In an implementation, for example, there may be 3 second data processing nodes, including a target data processing node, a first data processing node 1, and a first data processing node 2, respectively. Assuming that the target data includes 100 pieces of data, modulo processing may be performed on the 100 pieces of data based on data sequence numbers of the 100 pieces of data, and the 100 pieces of data are divided into 3 sets of data according to results of the modulo processing, respectively, and then a first fragmentation result is determined according to first position information of the target data processing node and second position information of the first data processing node. For example, if the first location information of the target data processing node is location 1, the second location information of the first data processing node 1 is location 2, and the second location information of the first data processing node 2 is location 3, the target data with a modulo result of 0 may be used as the first data to be processed by the target data processing node, the target data with a modulo result of 1 may be used as the second data to be processed by the first data processing node 1, and the target data with a modulo result of 2 may be used as the second data to be processed by the first data processing node 2.

In S310, the second fragmentation result sent by the first data processing node is received.

For the specific processing procedure of S310, reference may be made to relevant contents in S104 in the first embodiment, which is not described herein again.

In S312, the third location information sent by the first data processing node is received.

The third location information may be the location information of the first data processing node in the second data processing node determined by the first data processing node based on the node change information.

In S314, the first data is processed in a case where the second position information matches the third position information, and the first fragmentation result matches the second fragmentation result.

In implementation, through the matching detection of the second position information and the third position information and the matching detection of the first slicing result and the second slicing result, repeated processing on the target data can be avoided.

EXAMPLE III

Based on the same idea, the foregoing data processing method provided in the embodiment of the present invention further provides a data processing apparatus, as shown in fig. 4.

The data processing device comprises: a fragmentation module 401, a receiving module 402 and a processing module 403, wherein:

the fragmentation module 401 is configured to, when node change information of a data processing node sent by a distributed coordination server is received, perform fragmentation processing on target data to be processed based on the node change information to obtain a first fragmentation result, where the first fragmentation result includes first data to be processed locally and second data to be processed by the first data processing node, and the first data processing node is different from the target data processing node;

a receiving module 402, configured to receive a second fragmentation result sent by the first data processing node, where the second fragmentation result is a result of the first data processing node performing fragmentation processing on the target data based on the node change information, and the second fragmentation result includes third data to be processed by the first data processing node and fourth data to be processed by the target data processing node;

a processing module 403, configured to process the first data when the first slicing result matches the second slicing result.

In this embodiment of the present invention, the fragmentation module 401 includes:

In this embodiment of the present invention, the processing module 403 includes:

In an embodiment of the present invention, the third determining unit is configured to:

In an embodiment of the present invention, the fragmentation unit is configured to:

In this embodiment of the present invention, the number of the first data processing nodes is smaller than a preset node number threshold.

In this embodiment of the present invention, the target data processing node and the first data processing node are data processing nodes whose data processing capacities are smaller than a preset data processing capacity threshold.

The embodiment of the invention provides a data processing device, which performs fragmentation processing on target data to be processed based on node change information when receiving the node change information of a data processing node sent by a distributed coordination server, to obtain a first fragmentation result, wherein the first fragmentation result comprises first data to be processed locally and second data to be processed by a first data processing node, the first data processing node is different from the target data processing node, and receives a second fragmentation result sent by the first data processing node, the second fragmentation result is a result of the first data processing node performing fragmentation processing on the target data based on the node change information, the second fragmentation result comprises third data to be processed by the first data processing node and fourth data to be processed by the target data processing node, and when the first fragmentation result is matched with the second fragmentation result, the first data is processed. Therefore, through the matching detection of the first slicing result and the second slicing result, the data processed by the target data processing node is different from the data processed by the first data processing node, the problem of repeated data processing is avoided, meanwhile, the processing integrity of the target data can be ensured, namely, the problem of data omission processing is avoided, in addition, the data needing to be processed locally can be determined immediately through the node change information, the data processing is carried out, and the data processing efficiency is improved.

Example four

Figure 5 is a schematic diagram of a hardware configuration of an electronic device implementing various embodiments of the invention,

the electronic device 500 includes, but is not limited to: a radio frequency unit 501, a network module 502, an audio output unit 503, an input unit 504, a sensor 505, a display unit 506, a user input unit 507, an interface unit 508, a memory 509, a processor 510, and a power supply 511. Those skilled in the art will appreciate that the electronic device configuration shown in fig. 5 does not constitute a limitation of the electronic device, and that the electronic device may include more or fewer components than shown, or some components may be combined, or a different arrangement of components.

The processor 510 is configured to, when node change information of a data processing node sent by a distributed coordination server is received, perform fragmentation processing on target data to be processed based on the node change information to obtain a first fragmentation result, where the first fragmentation result includes first data to be processed locally and second data to be processed by a first data processing node, and the first data processing node is different from the target data processing node;

the processor 510 is further configured to receive a second fragmentation result sent by the first data processing node, where the second fragmentation result is a result of the first data processing node performing fragmentation processing on the target data based on the node change information, and the second fragmentation result includes third data to be processed by the first data processing node and fourth data to be processed by the target data processing node;

processor 510 is further configured to process the first data if the first sliced result matches the second sliced result.

Further, processor 510 is further configured to determine, based on the node change information, a second data processing node currently used for processing the target data;

in addition, the processor 510 is further configured to determine, as the first data processing node, a data processing node other than the target data processing node in the second data processing node;

further, processor 510 is configured to determine first location information of the target data processing node in the second data processing node and second location information of the first data processing node in the second data processing node;

in addition, the processor 510 is further configured to perform fragmentation processing on target data to be processed based on the first location information and the second location information, so as to obtain the first fragmentation result.

Further, processor 510 is further configured to receive third location information sent by the first data processing node, where the third location information is location information of the first data processing node in the second data processing node, which is determined by the first data processing node based on the node change information;

in addition, the processor 510 is further configured to process the first data when the second position information matches the third position information and the first slicing result matches the second slicing result.

Further, the processor 510 is further configured to obtain a self-increment value of each of the second data processing nodes, where the self-increment value is determined by a start-up time of the data processing node;

additionally, the processor 510 further determines first location information of the target data processing node in the second data processing node based on the self-increment value of the target data processing node and the self-increment values of other second data processing nodes;

further, processor 510 is configured to determine second location information of the first data processing node in the second data processing node based on the self-increment value of the first data processing node and the self-increment values of the other second data processing nodes.

In addition, the processor 510 is further configured to perform modulo processing on the number of the target data to be processed based on the number of the second data processing nodes;

in addition, the processor 510 is further configured to perform fragmentation processing on the target data to be processed based on the result of the modulo processing, the first location information of the target data processing node, and the second location information of the first data processing node, so as to obtain the first fragmentation result.

In addition, the number of the first data processing nodes is smaller than a preset node number threshold value.

In addition, the target data processing node and the first data processing node are data processing nodes with data processing capacity smaller than a preset data processing capacity threshold value.

The embodiment of the invention provides an electronic device, which performs fragmentation processing on target data to be processed based on node change information when receiving the node change information of a data processing node sent by a distributed coordination server, to obtain a first fragmentation result, wherein the first fragmentation result comprises first data to be processed locally and second data to be processed by a first data processing node, the first data processing node is different from the target data processing node, receives a second fragmentation result sent by the first data processing node, the second fragmentation result is a result of the first data processing node performing fragmentation processing on the target data based on the node change information, the second fragmentation result comprises third data to be processed by the first data processing node and fourth data to be processed by the target data processing node, and when the first fragmentation result is matched with the second fragmentation result, the first data is processed. Therefore, through the matching detection of the first slicing result and the second slicing result, the data processed by the target data processing node is different from the data processed by the first data processing node, the problem of repeated data processing is avoided, meanwhile, the processing integrity of the target data can be ensured, namely, the problem of data omission processing is avoided, in addition, the data needing to be processed locally can be determined immediately through the node change information, the data processing is carried out, and the data processing efficiency is improved.

It should be understood that, in the embodiment of the present invention, the radio frequency unit 501 may be used for receiving and sending signals during a message sending and receiving process or a call process, and specifically, receives downlink data from a base station and then processes the received downlink data to the processor 510; in addition, the uplink data is transmitted to the base station. In general, radio frequency unit 501 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier, a duplexer, and the like. In addition, the radio frequency unit 501 can also communicate with a network and other electronic devices through a wireless communication system.

The electronic device provides wireless broadband internet access to the user via the network module 502, such as assisting the user in sending and receiving e-mails, browsing web pages, and accessing streaming media.

The input unit 504 is used to receive an audio or video signal. The input Unit 504 may include a Graphics Processing Unit (GPU) 5041 and a microphone 5042. The processed image frames may be displayed on the display unit 506. The image frames processed by the graphic processor 5041 may be stored in the memory 509 (or other storage medium) or transmitted via the radio frequency unit 501 or the network module 502. The microphone 5042 may receive sounds and may be capable of processing such sounds into audio data. The processed audio data may be converted into a format output transmittable to a mobile communication base station via the radio frequency unit 501 in case of the phone call mode.

The display unit 506 is used to display information input by the user or information provided to the user. The Display unit 506 may include a Display panel 5061, and the Display panel 5061 may be configured in the form of a Liquid Crystal Display (LCD), an Organic Light-Emitting Diode (OLED), or the like.

The user input unit 507 may be used to receive input numeric or character information and generate key signal inputs related to user settings and function control of the electronic device. Specifically, the user input unit 507 includes a touch panel 5071 and other input devices 5072. The touch panel 5071 may include two parts of a touch detection device and a touch controller. The touch detection device detects the touch direction of a user, detects a signal brought by touch operation and transmits the signal to the touch controller; the touch controller receives touch information from the touch sensing device, converts the touch information into touch point coordinates, sends the touch point coordinates to the processor 510, and receives and executes commands sent by the processor 510. Further, the touch panel 5071 may be overlaid on the display panel 5061, and when the touch panel 5071 detects a touch operation thereon or nearby, the touch operation is transmitted to the processor 510 to determine the type of the touch event, and then the processor 510 provides a corresponding visual output on the display panel 5061 according to the type of the touch event. Although in fig. 5, the touch panel 5071 and the display panel 5061 are two independent components to implement the input and output functions of the electronic device, in some embodiments, the touch panel 5071 and the display panel 5061 may be integrated to implement the input and output functions of the electronic device, and is not limited herein.

The interface unit 508 is an interface for connecting an external device to the electronic apparatus 500. For example, the external device may include a wired or wireless headset port, an external power supply (or battery charger) port, a wired or wireless data port, a memory card port, a port for connecting a device having an identification module, an audio input/output (I/O) port, a video I/O port, an earphone port, and the like. The interface unit 508 may be used to receive input (e.g., data information, power, etc.) from external devices and transmit the received input to one or more elements within the electronic apparatus 500 or may be used to transmit data between the electronic apparatus 500 and external devices.

The memory 509 may be used to store software programs as well as various data. The memory 509 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required by at least one function (such as a sound playing function, an image playing function, etc.), and the like; the storage data area may store data (such as audio data, a phonebook, etc.) created according to the use of the cellular phone, and the like. Further, the memory 509 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid-state storage device.

The processor 510 is a control center of the electronic device, connects various parts of the whole electronic device by using various interfaces and lines, performs various functions of the electronic device and processes data by running or executing software programs and/or modules stored in the memory 509 and calling data stored in the memory 509, thereby performing overall monitoring of the electronic device. Processor 510 may include one or more processing units; preferably, the processor 510 may integrate an application processor, which mainly handles operating systems, user interfaces, application programs, etc., and a modem processor, which mainly handles wireless communications. It will be appreciated that the modem processor described above may not be integrated into processor 510.

The electronic device 500 may further include a power supply 511 (e.g., a battery) for supplying power to various components, and preferably, the power supply 511 may be logically connected to the processor 510 via a power management system, so as to implement functions of managing charging, discharging, and power consumption via the power management system.

Preferably, an embodiment of the present invention further provides an electronic device, which includes a processor 510, a memory 509, and a computer program that is stored in the memory 509 and can be run on the processor 510, and when the computer program is executed by the processor 510, the processes of the data processing method embodiment are implemented, and the same technical effect can be achieved, and details are not described here to avoid repetition.

EXAMPLE five

The embodiment of the present invention further provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the computer program implements each process of the data processing method embodiment, and can achieve the same technical effect, and in order to avoid repetition, details are not repeated here. The computer-readable storage medium may be a Read-only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk.

The embodiment of the present invention provides a computer-readable storage medium, where, in a case where node change information of a data processing node sent by a distributed coordination server is received, a first fragmentation result is obtained by performing fragmentation processing on target data to be processed based on the node change information, where the first fragmentation result includes first data to be processed locally and second data to be processed by a first data processing node, the first data processing node is different from the target data processing node, and a second fragmentation result is sent by the first data processing node and is a result of the first data processing node performing fragmentation processing on the target data based on the node change information, and the second fragmentation result includes third data to be processed by the first data processing node and fourth data to be processed by the target data processing node, where, in a case where the first fragmentation result matches the second fragmentation result, the first data is processed. Therefore, through the matching detection of the first slicing result and the second slicing result, the data processed by the target data processing node is different from the data processed by the first data processing node, the problem of repeated data processing is avoided, meanwhile, the processing integrity of the target data can be ensured, namely, the problem of data omission processing is avoided, in addition, the data needing to be processed locally can be determined immediately through the node change information, the data processing is carried out, and the data processing efficiency is improved.

As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium.

Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include transitory computer readable media (transmyedia) such as modulated data signals and carrier waves.

It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.

The above description is only an example of the present invention, and is not intended to limit the present invention. Various modifications and alterations to this invention will become apparent to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A method of processing data, the method being applied to a target data processing node, the method comprising:

2. The method according to claim 1, wherein the performing fragmentation processing on the target data to be processed based on the node change information to obtain a first fragmentation result comprises:

3. The method of claim 2, wherein the processing the first data in the case that the first sliced result matches the second sliced result comprises:

4. The method of claim 2, wherein said determining first location information of said target data processing node in said second data processing node and second location information of said first data processing node in said second data processing node comprises:

5. The method according to claim 4, wherein the performing fragmentation processing on the target data to be processed based on the first location information of the target data processing node and the second location information of the first data processing node to obtain the first fragmentation result comprises:

6. The method according to any one of claims 1 to 5, wherein the number of first data processing nodes is less than a preset node number threshold.

7. The method according to any one of claims 1 to 5, wherein the target data processing node and the first data processing node are data processing nodes having a data throughput less than a preset data throughput threshold.

8. An apparatus for processing data, the apparatus comprising:

9. An electronic device, comprising a processor, a memory and a computer program stored on the memory and executable on the processor, the computer program, when executed by the processor, implementing the steps of the method of processing data according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, which computer program, when being executed by a processor, carries out the steps of the method of processing data according to any one of claims 1 to 7.