CN113687846B - Method, device, device and readable storage medium for processing data - Google Patents

Abstract

The present disclosure provides methods, apparatuses, devices and readable storage media for processing data, relating to the field of data processing technology, and in particular to the field of big data and intelligent searching. The specific implementation scheme is as follows: obtaining, at a first server, allocation information of new version data generated by a data source, the allocation information including a plurality of storage addresses of a plurality of data fragments into which the new version data is divided and operation information corresponding to a plurality of ordered operations for the new version data; configuring operation information using a storage address of a data fragment of the plurality of data fragments to generate configured operation information for the data fragment; determining target state information corresponding to a target operation to be completed in the plurality of ordered operations; and sending the configured operation information and the target state information for the data shard to a second server for processing the data shard. By the method, the final state consistency of the data distribution stage can be realized, and the complexity of a data distribution system is reduced.

Description

Method, device, device and readable storage medium for processing data

technical field

The present disclosure relates to the technical field of data processing, and in particular to methods, devices, devices, and readable storage media for processing data in fields such as big data and intelligent search.

Background technique

With the development of the computer Internet, complex systems composed of large-scale computer programs have gradually begun to be used. With the development of such a complex system, more and more subsystems and data to be processed are related to the system. However, due to the limited capacity of computing devices, people usually store the calculation of the program in each component subsystem and the data used for the program separately. In the storage separation technical solution, the cloud program service running on the cloud server usually includes program files and data files. As the scale of the program service grows, the data information contained in the data file also gradually increases accordingly. However, there are many technical problems that need to be solved in the process of providing data files for program services

Contents of the invention

The present disclosure provides a method, device, device and storage medium for processing data.

According to a first aspect of the present disclosure, a method for processing data is provided. The method includes: acquiring, at the first server, distribution information of new version data generated by a data source, the distribution information including multiple storage addresses of multiple data fragments into which the new version data is divided, and multiple storage addresses for the new version data The operation information corresponding to an ordered operation; use the storage address configuration operation information of the data fragments in the multiple data fragments to generate the configured operation information for the data fragments; determine the information to be completed in the multiple ordered operations target state information corresponding to the target operation; and sending the configured operation information and target state information for the data slice to the second server for processing the data slice.

According to a second aspect of the present disclosure, a method for processing data is provided. The method includes receiving, at a second server, from a first server configured operational information and target state information for a data slice from a plurality of data generated by dividing a new version of data generated by a data source Fragmentation, the configured operation information is related to a plurality of ordered operations for data fragmentation, and the target state corresponds to a target operation to be completed in the plurality of ordered operations; First heartbeat information of the slice, sending operation information and target state information to the first computing device, the first heartbeat information including the current state for the data slice; and updating the current state of the first computing device with the current state of the data slice information.

According to a third aspect of the present disclosure, a method for processing data is provided. The method includes: obtaining an identifier of a data fragment to be processed at the first computing device, and the data fragment comes from a plurality of data fragments generated by dividing the new version data generated by the data source; The heartbeat message of data sharding is used to receive the operation information and target status information for data sharding. The operation information is related to multiple ordered operations for data sharding, and the target state is related to the goals to be completed in multiple ordered operations The operations correspond; comparing the current state of the data shard to the target state; and if the current state is determined to be different from the target state, proceeding to perform a plurality of sequential operations to complete the target operation.

According to a fourth aspect of the present disclosure, an apparatus for processing data is provided. The device includes: a distribution information acquisition module configured to obtain distribution information of new version data generated by the data source at the first server, the distribution information including multiple storage addresses of multiple data fragments into which the new version data is divided and operation information corresponding to a plurality of ordered operations for the new version data; an operation information configuration module configured to use the storage address configuration operation information of the data fragments in the plurality of data fragments to generate the data fragments The configured operation information; the target state information determination module configured to determine the target state information corresponding to the target operation to be completed in the plurality of ordered operations; and the sending module configured to send the target state information to the second server Shard's configured operational information and target state information for processing data shards.

According to a fifth aspect of the present disclosure, an apparatus for processing data is provided. The apparatus includes: an operation information and target state information receiving module configured to receive, at the second server, configured operation information and target state information for data slices from the first server, the data slices being generated from the data source Multiple data shards generated by dividing the new version of the data, the configured operation information is related to multiple ordered operations for data shards, and the target state corresponds to the target operation to be completed in the multiple ordered operations; The first operation information and target state information sending module is configured to send operation information and target state information to the first computing device in response to receiving first heartbeat information for data fragmentation from the first computing device, the first heartbeat information A current state for the data slice is included; and an update module configured to update the current state information of the first computing device with the current state of the data slice.

According to a sixth aspect of the present disclosure, an apparatus for processing data is provided. The apparatus includes: an identification acquisition module configured to acquire the identification of the data fragment to be processed at the first computing device, the data fragment comes from a plurality of data fragments generated by dividing the new version data generated by the data source The heartbeat information sending module is configured to send a heartbeat message for the data fragmentation to the second server for receiving operation information and target state information for the data fragmentation, the operation information and a plurality of ordered operations for the data fragmentation Regarding, the target state corresponds to the target operation to be completed in a plurality of ordered operations; the comparison module is configured to compare the current state of the data slice with the target state; and the operation execution module is configured to determine if the current state Unlike the goal state, multiple sequential operations continue to be performed to complete the goal operation.

According to a seventh aspect of the present disclosure, an electronic device is provided. The electronic device includes at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can A method according to the first aspect of the present disclosure is performed.

According to an eighth aspect of the present disclosure, there is provided an electronic device. The electronic device includes at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can A method according to the second aspect of the present disclosure is performed.

According to a ninth aspect of the present disclosure, there is provided an electronic device. The electronic device includes at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can A method according to the third aspect of the present disclosure is performed.

According to a tenth aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions for causing a computer to execute the method according to the first aspect of the present disclosure.

According to an eleventh aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions for causing a computer to execute the method according to the second aspect of the present disclosure.

According to a twelfth aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions for causing a computer to execute the method according to the third aspect of the present disclosure.

According to a thirteenth aspect of the present disclosure there is provided a computer program product comprising a computer program which, when executed by a processor, implements the steps of the method according to the first aspect of the present disclosure.

According to a fourteenth aspect of the present disclosure there is provided a computer program product comprising a computer program which, when executed by a processor, implements the steps of the method according to the second aspect of the present disclosure.

According to a fifteenth aspect of the present disclosure there is provided a computer program product comprising a computer program which, when executed by a processor, implements the steps of the method according to the third aspect of the present disclosure.

It should be understood that what is described in this section is not intended to identify key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will be readily understood through the following description.

Description of drawings

The accompanying drawings are used to better understand the present solution, and do not constitute a limitation to the present disclosure. in:

FIG. 1 shows a schematic diagram of an environment 100 in which various embodiments of the present disclosure can be implemented;

FIG. 2 shows a flowchart of a method 200 for processing data according to some embodiments of the present disclosure;

FIG. 3 shows a schematic diagram of an example 300 of a fragmentation mode according to some embodiments of the present disclosure;

FIG. 4 shows a schematic diagram of an example 400 of multiple distribution stages according to some embodiments of the present disclosure;

FIG. 5 shows a flowchart of a method 500 for processing data according to some embodiments of the present disclosure;

FIG. 6 shows a schematic diagram of a structure 600 of hosting services and proxies according to some embodiments of the present disclosure;

FIG. 7 shows a flowchart of a method 700 for processing data according to some embodiments of the present disclosure;

FIG. 8 shows a schematic diagram of an example 800 of multiple sequential operations for data sharding according to some embodiments of the present disclosure;

FIG. 9 shows a schematic diagram of an example 900 of multiple sequential operations for data sharding according to some embodiments of the present disclosure;

FIG. 10 shows a schematic diagram of an example 1000 of multiple ordered operations for data sharding according to some embodiments of the present disclosure;

FIG. 11 shows a schematic diagram of an example 1100 of migrating data shards for data shards according to some embodiments of the present disclosure;

FIG. 12 shows a schematic diagram of an example system 1200 for processing data according to some embodiments of the present disclosure;

FIG. 13 shows a block diagram of an apparatus 1300 for processing data according to some embodiments of the present disclosure; and

FIG. 14 shows a block diagram of an apparatus 1400 for processing data according to some embodiments of the present disclosure; and

FIG. 15 shows a block diagram of an apparatus 1500 for processing data according to some embodiments of the present disclosure; and

FIG. 16 shows a block diagram of a device 1600 capable of implementing various embodiments of the present disclosure.

Detailed ways

Exemplary embodiments of the present disclosure are described below in conjunction with the accompanying drawings, which include various details of the embodiments of the present disclosure to facilitate understanding, and they should be regarded as exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the disclosure. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

In the description of the embodiments of the present disclosure, the term "comprising" and its similar expressions should be interpreted as an open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be read as "at least one embodiment". The terms "first", "second", etc. may refer to different or the same object. Other definitions, both express and implied, may also be included below.

As the scale of the program service becomes larger, the data information contained in the data file gradually increases accordingly, usually exceeding the storage capacity of a computer's physical memory. At this time, the large data file needs to be cut and sharded, and a large data file is split into hundreds or thousands of data shards. Therefore, in distributed system solutions, how to distribute a large number of data fragments to distributed computer programs has become a technical problem that needs to be solved.

In order to solve the above problems, in a traditional solution, a central control service program is used to notify and push the delivery address on the target computer, and after the data is obtained on the computer, it is placed on the file path used by the computer program to realize data collection. delivered one by one. However, this solution requires a central push system in the entire data distribution mechanism to distribute and push data. After the single-point computer push fails, only limited re-push can be done, and the state cannot be consistent in the end. At the same time, for the new computer nodes generated during the distribution and push process, the latest data files cannot be obtained in time, and the central push system needs to sense the existence of new nodes and initiate additional new pushes, which are obvious in terms of timeliness and fault tolerance. lack of.

Another traditional solution is to split an overly large data file into multiple small data files, and establish corresponding data distribution one by one, so as to realize the data distribution function of multiple small data files, wherein multiple small data files The data distribution of each is independent of each other. However, when data exceeding the capacity of a single computer occurs, the data needs to be split into small data files, and independent data distribution tasks corresponding to the small files need to be created. At this time, it is necessary to manually disassemble and add new distribution. This method has a very high maintenance cost for the expansion of the number of small data files. At the same time, the data distribution of each small data is independent of each other, and it is impossible to do collaborative distribution between small data files. It is not applicable to strictly consistent large data files. .

There is also a traditional solution that does not make an obvious split between programs and data. When there is new data to be delivered, the program package is iteratively upgraded by changing the computer program package. By upgrading computer programs one by one, the updating of data files used in the programs is realized. However, for this solution, the small data files after the big data split need to be bound to the computer program upgrade. Every time the data is updated, it is necessary to iteratively change the computer program and upgrade one by one. It is impossible to achieve computer program and The flexible decoupling of data files is also difficult to achieve the technical decoupling of computer programs and data files.

In order to at least solve the above problems, according to an embodiment of the present disclosure, an improved solution for processing data is proposed. In this scheme, the allocation information of the new version data generated by the data source is acquired at the first server, the allocation information includes multiple storage addresses of the multiple data fragments into which the new version data is divided and the Operation information corresponding to multiple sequential operations. Then the first server uses the storage address configuration operation information of the data fragments in the plurality of data fragments to generate configured operation information for the data fragments, and determines that it corresponds to the target operation to be completed in the plurality of sequential operations target state information. The first server sends configured operational information and target state information for the data slice to the second server for processing the data slice. Through this method, the final state consistency in the data distribution stage can be realized, the complexity of the data distribution system can be significantly reduced, and the stability of data processing can be improved.

Figure 1 shows a schematic diagram of an environment 100 in which various embodiments of the present disclosure can be implemented. The example environment 100 includes a server 106 , a server 112 and a computing device 114 .

Server 106 and server 112 can be various suitable computing devices, and can also be cloud servers, also known as cloud computing servers or cloud hosts, which are a host product in the cloud computing service system to solve the problem of traditional physical hosts and VPS In the service ("Virtual Private Server", or "VPS" for short), there are defects such as difficult management and weak business scalability. The server can also be a server of a distributed system, or a server combined with a blockchain. For ease of description, server 106 may be referred to as a first server and server 112 may be referred to as a second server.

The server 106 can be used to monitor whether the version of the data produced by the data source changes. If the new version data 102 is detected, the new version data 102 is acquired. Configuration information corresponding to a data source exists for each application that processes data generated by the data source. The configuration information includes at least the number of data fragments into which the new version data is divided, and multiple sequential operations for the new version data. Alternatively or additionally, the configuration information further includes multiple publishing stages for releasing the new version data to computing devices, cluster identifiers of computing device clusters used in each publishing stage, and the like.

The plurality of sequential operations can also be viewed as a state machine path, comprising a plurality of operations performed in sequence on a new version of data. The plurality of sequential operations includes at least download and load operations. In some embodiments, the multiple sequential operations also include configuration operations, such as creating file directories and the like. Alternatively or additionally, the plurality of sequential operations may also include subsequent operations after the load operation, such as processing operations on previously accepted data. The above examples are only used to describe the present disclosure, rather than to specifically limit the present disclosure.

The server 106 obtains the configuration information. The new version data 102 is then divided into multiple data fragments 104 according to the number of data fragments. The server 106 can generate allocation information according to the configuration information and related information of the data slice 104 . The relevant information of the data slice includes, for example, the storage address of the data slice, the data version, and the like. The allocation information may include the storage address of the data fragment, the destination address of the data fragment in the destination computing device, multiple release stages for releasing new version data to the computing device, and the cluster of computing device clusters used in each release stage identification, multiple sequential operations on this new version of the data, etc.

For each data shard, the server 106 generates operation information 108 corresponding to a plurality of sequential operations for each data shard. For example, the storage address of each data fragment is used to configure the download address in the download operation among the multiple sequential operations of the data fragment. The server 106 also determines target operations to be performed for these data fragments to form target state information 110 . Alternatively or additionally, the target status information 110 is generated for the identification and target operation of the computing device cluster in different publishing phases, for example, the computing device cluster identification is used as a field in the target status information.

In FIG. 1, the process of generating allocation information, configuring multiple ordered operations and generating target state information is performed in the server 106, which is only an example, rather than a specific limitation of this disclosure. The above-mentioned processes can be implemented on different servers. , those skilled in the art can set as needed.

Server 112 receives operational information 108 and object status information 110 . When the server 112 receives the heartbeat information sent by the computing device 114 , it sends the operation information and target state information of the data slices processed on the computing device to the computing device 114 . In some embodiments, the heartbeat information includes the identification of the data segment, and the server 112 searches for the corresponding operation information according to the identification of the data segment, and then delivers the operation information and target status information for the data segment. Alternatively or additionally, the heartbeat information includes the cluster ID of the device cluster to which the computing device 114 belongs, and the server 112 matches the cluster ID of the device cluster to which the computing device 114 belongs with the cluster ID in the target state information. The operation information and the target state information are delivered to the computing device 114 only when the two match, which ensures that only the computing device corresponding to each publishing stage executes the application for processing data fragments. The above examples are only used to describe the present disclosure, rather than to specifically limit the present disclosure.

After obtaining the identifier of the data segment to be processed, the computing device 114 sends heartbeat information including the identifier of the data segment to the server 112 to obtain the operation information 108 and target state information 110 of the segment from the server 112 . After the computing device 114 acquires the operation information 108 and the target status information 110 , it compares the target status in the target status information 110 with the current status in the computing device 114 . If the target state does not match the current state, it indicates that the current operation has not yet reached the target operation, and the computing device continues to perform operations in multiple sequential operations. If they match, it indicates that the operation performed in the computing device 114 is already the target operation, and there is no need to continue to perform subsequent operations.

Computing devices 114 include, but are not limited to, personal computers, server computers, handheld or laptop devices, mobile devices (such as mobile phones, personal digital assistants (PDAs), media players, etc.), multiprocessor systems, consumer electronics, small Computers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The server can be a cloud server, also known as a cloud computing server or a cloud host. ), there are defects such as high management difficulty and weak business scalability. The server can also be a server of a distributed system, or a server combined with a blockchain.

FIG. 1 shows one server 112 and one computing device 114 , which is only an example and not a specific limitation to the present disclosure. The environment 100 may also include multiple servers 112 and multiple computing devices. Where multiple servers 112 receive operational information and object status information from one server 106 , and multiple computing devices 114 receive data information from one server 112 .

Through this method, the final state consistency in the data distribution stage can be realized, the complexity of the data distribution system can be significantly reduced, and the stability of data processing can be improved.

The environment 100 in which multiple embodiments of the present disclosure can be implemented has been described above with reference to FIG. 1 . The following describes a flow chart of a method 200 for processing data according to some embodiments of the present disclosure with reference to FIG. 2 . Method 200 in FIG. 2 may be performed by server 106 in FIG. 1 or any suitable computing device.

At block 202, the allocation information of the new version data generated by the data source is obtained at the first server, the allocation information includes multiple storage addresses of the multiple data fragments into which the new version data is divided and the Operation information corresponding to multiple sequential operations. As shown in FIG. 1, the server 106 obtains distribution information of new version data generated by the data source. In some embodiments, server 106 receives allocation information from other computing devices. In some embodiments, server 106 generates allocation information. The above examples are only used to describe the present disclosure, rather than to specifically limit the present disclosure.

In some embodiments, the server 106 monitors whether the data generated by the data source changes, for example, whether the data generated by the data source changes or is updated. If the data changes, the data version is considered to have changed. In some embodiments, the data version information is represented by time or various appropriate identification information set by the user. When data changes, new data version information is generated. The server checks the version information to determine if the data version has changed. The above examples are only used to describe the present disclosure, rather than to specifically limit the present disclosure. If it is determined that the version of the data has changed, the new version data is divided into multiple data fragments 104 . In this way, data shards can be quickly divided.

In some embodiments, server 106 obtains a predetermined number of data shards to generate. For example, the server 106 obtains the predetermined number from the configuration information for the data of the data source. The server 106 then divides the new version data according to a predetermined number. In this manner, data can be quickly divided. By dividing the new version data 102 into multiple data fragments 104, data processing can be accelerated and data processing efficiency is improved.

FIG. 3 shows a schematic diagram of an example 300 of a fragmentation scheme according to some embodiments of the present disclosure. In FIG. 3 , in the traditional single mode, different versions of data will generate a data file, such as data version 1 302 , data version 2 304 and data version 3 306 . In an embodiment of the present disclosure, each version of data may be divided into a predetermined number of data fragments. In some embodiments, for example, data version 1 may be divided into three data shards: shards 308 , 310 , and 312 . Similarly, subsequent data versions are also divided into the same number of data shards. In some embodiments, data may be partitioned into any suitable number of data shards. The foregoing are examples only, not specific descriptions of the present disclosure.

Now return to FIG. 2 to continue the description. In some embodiments, the server 106 generates allocation information according to the address and version of the data fragment and in combination with the configuration information corresponding to the data source. The configuration information includes at least the number of data fragments to be divided into new version data, multiple release stages for releasing the new version data 102 to computing devices, cluster identifiers of computing device clusters used in each release stage, and Multiple sequential operations on data. Assignment information is generated based on the above information. In some embodiments, server 106 receives sub-configuration information from other other servers. The above examples are only used to describe the present disclosure, rather than to specifically limit the present disclosure.

At block 204, operational information is configured using storage addresses of data slices of the plurality of data slices to generate configured operational information for the data slices. For example, the server 106 in FIG. 1 uses the storage address of the data slice to configure the operation information for each data slice. Alternatively or additionally, other information in the allocation information is utilized to configure the operation information. For example, a storage address of the data slice in the computing device may also be configured.

In some embodiments, the server 106 obtains the storage address for the data slice from the allocation information. Then, the server 106 associates the storage address with the partial information in the operation information to generate configured operation information for the data slice, the partial information corresponds to the download operation in the plurality of sequential operations. In this way, the storage address of the data fragment can be quickly determined.

At block 206, target state information corresponding to a target operation to be completed in the plurality of ordered operations is determined. For example, server 106 in FIG. 1 determines target state information corresponding to a target operation to be completed in a plurality of sequential operations.

In some embodiments, there is only one phase of data release, and the target state information is used for all computing devices running applications that process data, and target operations to be completed are set in the target state information.

In some embodiments, the data publishing includes multiple phases, wherein the distribution information further includes target cluster identifications of target computing device clusters corresponding to each of the multiple publishing phases of the new version of the data. At this time, for each release stage, the server 106 determines the target cluster identifier of the computing device cluster corresponding to the release stage. The server may also determine a target operation to complete among multiple sequential operations. Then the server generates target status information based on the target operation and the target cluster ID, for example, the target status information includes the operation ID corresponding to the target operation and the target cluster ID. In this manner, target state information for computing devices in different stages can be quickly generated. Multiple publishing stages are described below in conjunction with FIG. 4 , which shows a schematic diagram of an example 400 of multiple distribution stages according to some embodiments of the present disclosure.

In FIG. 4 , data release includes four stages, stage S0 402 , stage S1 404 , stage S2 406 and stage S3 408 . Each stage is executed by a different computing device selected to process the application of the data. For example, in stage S1 404, union clusters Union_ig1 and Union_ig2 are selected to run applications for processing data fragmentation, each union cluster includes multiple computing device clusters, for example, union cluster Union_ig1 includes computing device clusters ig1, ig2, ig3 and ig4 . Each computing device cluster includes multiple computing device instances. Therefore, for each release phase, the server 106 puts the identifications of different computing device clusters into the target state information for delivering operation information and target state information to computing devices in the target cluster.

Returning to FIG. 2 to continue the description, at block 208 , the configured operation information and target state information for the data shards are sent to the second server for processing the data shards. For example, as shown in FIG. 1 , server 106 sends configured operational information and target state information for a data segment to server 112 for processing the data segment.

In some embodiments, the server 106 also obtains current state information of the plurality of computing devices corresponding to the target cluster identifier, where the current state information indicates operations in the plurality of sequential operations completed by the plurality of computing devices. The server 106 then determines whether the current state information matches the target state information. If it is determined that the current state information matches the target state information, the target state information is updated to correspond to the next operation of the target operation. Server 106 then sends the updated target state to server 112 for use in completing the next operation. If the two state information do not match, it indicates that there are computing devices that have not yet performed the target operation, and then obtain the current state information of multiple computing devices at a predetermined time interval to detect and wait for all computing devices to complete the target operation. In this way, consistency in the data processing process can be achieved.

As an example, as shown in FIG. 4, in the publishing phase S1 404, the server 106 regularly queries the state of each computing device instance, and if the state of each instance in the computing device cluster matches the target state, the corresponding cluster's The state is set to the state where the target operation is completed, also known as state recycling. If the state of each computing device cluster is the state of completing the target operation, the state of the joint cluster can be set to the state of completing the target operation, which can also be called state recursion, and then the execution of the target state can be advanced in parallel through the joint cluster. After the states of all computing devices at this stage are the target state, the server adjusts the target operation in the target state information to be the next operation among the multiple sequential operations.

In some embodiments, if the server 106 receives release information of an updated version of the data again, it waits for the orderly operation of the current data version to be completed before performing the release operation of the updated version of the data.

Through this method, the final state consistency in the data distribution stage can be realized, the complexity of the data distribution system can be significantly reduced, and the stability of data processing can be improved.

The flow chart of the method 200 for processing data according to multiple embodiments of the present disclosure is described above with reference to FIGS. 2-4 . A flow chart of a method 500 for processing data according to some embodiments of the present disclosure is described below with reference to FIG. 5 . Method 500 in FIG. 5 may be performed by server 112 in FIG. 1 or any suitable computing device.

At block 502, configured operational information and target state information for data shards from the first server are received at the second server from multiple For data sharding, the configured operation information is related to multiple ordered operations for data sharding, and the target state corresponds to the target operation to be completed in the multiple ordered operations. For example, server 112 in FIG. 1 receives configured operational information and target state information for data shards from server 106 . The server 112 stores the operation information and target state information for data distribution after receiving it.

At block 504, it is determined whether first heartbeat information for the data shard is received from the first computing device. For example, server 112 monitors whether heartbeat information is received from computing device 114 . If first heartbeat information for the data slice is received from the first computing device 114, at block 506, operational information and target state information are sent to the first computing device, the first heartbeat information including the current state for the data slice. For example, the heartbeat information sent by computing device 114 to server 112 includes a current status indicating operations that have been completed on computing device 114 for data shards.

In some embodiments, the first heartbeat information further includes the reference cluster identifier of the reference computing device cluster where the first computing device is located, and the target state information includes a release stage corresponding to each of the multiple release stages of the new version data. The target cluster ID of the target computing device cluster. When the server 112 sends the operation information and the target state information to the first computing device 114, it needs to first match the reference cluster ID with the target cluster ID. If it is determined that the reference cluster ID matches the target cluster ID, indicating that the first computing device is the selected computing device in the publishing phase, then the operation information and the target status information are sent to the first computing device. If not, it indicates that the first computing device was not the computing device selected at the publishing stage. Therefore, there is no need to send operational information and target status information. In this way, the operation of each computing device can be precisely controlled.

Using the above heartbeat information to obtain operation information and target status information, a declarative distribution mechanism is essentially established to ensure the consistency of data processing. The declarative distribution mechanism is implemented by a master service on the server 112 and a proxy service running on the computing device 114, and the master service and the proxy service have a one-to-many relationship. As shown in Figure 6, multiple proxy services will communicate with the main control service over the network to synchronize heartbeat information and obtain the latest version of data files. The main control service maintains the latest version of the data file, and at the same time maintains the current stage of the latest version for the proxy service to query and obtain. Specifically, the master service 602 running on the second server includes a plurality of sequential operations, corresponding to a plurality of stages. Multiple agents associated with hosting service 602 run on multiple different computing devices, eg, agents 604, 606, and 608 run on three computing devices. Each agent is responsible for managing a number of different stages of operations for data shards 610 , 612 and 614 .

Returning to FIG. 5 to continue the description, at block 508, the current state information of the first computing device is updated with the current state of the data slice. For example, the server 112 in FIG. 1 uses the current state of the data slice to update the current state information of the first computing device, such as storing the current state in the state information list of the computing device.

In some embodiments, if the target state information is updated in server 106, server 112 receives updated target state information from server 106, wherein the updated target state information corresponds to the next operation of the target operation. Then, after receiving the second heartbeat information for the data segment, the server 112 sends the operation information and the updated target state information to the first computing device for completing the next operation. In this way, consistency of operation can be achieved.

In some embodiments, the server 112 also receives a third heartbeat message from the first computing device 114 including an identification of the transferred data segment, in the first state information of the partial data segment associated with the reference computing device cluster The operation information and target state information of the transferred data fragments are searched, and the transferred data fragments are transferred from the second computing device to the first computing device. If the operation information and target state of transferring data fragments are not found in the first state information, the operation information and target state of transferring data fragments are searched in the second state information corresponding to multiple data fragments. Then the server 112 sends the found operation information and target status of the transferred data fragments to the first computing device 114 . In this way, the transfer of data fragments can be realized quickly.

Through this method, the final state consistency in the data distribution stage can be achieved, the complexity of the data distribution system can be significantly reduced, and the stability of data processing can be improved.

The method 500 for processing data according to multiple embodiments of the present disclosure has been described above with reference to FIGS. 5-6 . The following describes a flow chart of a method 700 for processing data according to some embodiments of the present disclosure with reference to FIG. 7 . Method 700 in FIG. 7 may be performed by computing device 114 in FIG. 1 or any suitable computing device.

At block 702, an identification of a data slice to be processed is obtained at a first computing device, the data slice being from a plurality of data slices generated by dividing new version data generated by a data source. For example, computing device 114 obtains an identification of data shard 104 to process.

In some examples, the application control system connected to the application running on the computing device 114 can obtain the information of all the data slices and the identification of the computing devices that can run the application that processes the data slices, and then assign the available data to each computing device. Processed data shards. Accordingly, computing device 114 may obtain from the application control system an identification of a data slice that it is to process.

At block 704, a heartbeat message for data fragmentation is sent to the second server for receiving operation information and target state information for data fragmentation, the operation information is related to multiple sequential operations for data fragmentation, and the target state Corresponds to the target operation to be completed in multiple ordered operations. For example, computing device 114 sends a heartbeat message for the data shard to server 112 for receiving operational information and target status information for the data shard.

Multiple ordered operations per data shard include multiple operations. The multiple sequential operations include at least a download operation and a load operation, which are fixed operations for processing data fragments. Users can set multiple custom operation stages before and after these two operations, such as custom operations before downloading, such as creating file directories, and adding custom operations after loading operations. All operations form a data analysis Information about the operation of the slice. The proxy service reports the current status of different data units to the main control service on the second server 112 through the heartbeat information, and the main control service will issue operation information and target status according to the heartbeat information received from the proxy service. 8 shows a schematic diagram of an example 800 of multiple sequential operations on data sharding according to some embodiments of the present disclosure.

As shown in Figure 8. Data slice 1 802 is a data slice generated by the first data source processed by the computing device 114 , and data slice 2 806 is a data slice generated by the second data source processed on the computing device 114 . There are multiple sequential operations 804 for data fragment 1802, and multiple sequential operations 808 for data fragment 2 806, where stage 0, stage 3, and stage 4 correspond to other user-defined operations. The proxy service running on the computing device 114 can send the state information of the data slice 1 to the main control service of the server 112 through the heartbeat information, for example, the state information corresponding to the current operation completed by the computing device 114 . Similarly, for the data slice 2 806 , the status information corresponding to the current operation on the data slice completed by the computing device 114 may also be sent to the main control service in the server 112 . Then the operation information and target state information for data slice 1 and the operation information and target state information for data slice 2 are received from the master service. The computing device 114 processes the data slices according to the received operation information and target state information until the operation corresponding to the target state information is completed.

FIG. 8 above shows a schematic diagram of multiple sequential operations for data fragments from different data sources located on the same computing device. The following describes multiple sequential operations for a data fragment in conjunction with FIG. 9 .

As shown in FIG. 9 , the proxy service on computing device 114 obtains operational information and target state information for data shard 902 . The operation information 904 includes a plurality of ordered operations, wherein stage 0, stage 3 and stage 4 correspond to a plurality of custom operations. The computing device 114 compares the target state information with the current state of the data slice on the computing device 114. If the state is different, it needs to continue to perform subsequent operations on the data slice to finally achieve the current target state of the data file. For backward operations, post-failure processing will be performed according to the defined retry times and operation timeout time of each stage, so as to maximize the smooth achievement of the target operation and achieve the final consistency of the version stage of the data file. Specifically, for each operation, if the operation is unsuccessful, it can be retried. If the specified number of retries is unsuccessful or exceeds a certain time limit, it will be fed back that the processing of the data fragment is unsuccessful. For example, the loading phase can be retried 20 times and completed within 1200s. If it is not implemented, it indicates that the operation failed. Therefore, through this mechanism, the processing for data fragmentation can be completed as accurately as possible. Then, by controlling the execution of data sharding, it is possible to control all data shards generated by the data source.

When a new computer or container is added to the service cluster of the computer program, it will obtain the latest target stage according to the interaction between the started agent service and the main control service, complete the target stage of data distribution in turn, and finally communicate with other computing device clusters The service on the computing device achieves the consistency of data distribution.

In some embodiments, the heartbeat message includes determining the current state for the data shards. Computing device 114 generates heartbeat information using the identification and current state of the data slice. In this way, accurate heartbeat information can be generated.

In some embodiments, the heartbeat information is sent periodically for updating the current state information of the first computing device processing the slice, for example, the heartbeat information is sent every 5s. In this way, the status of the computing device can be quickly updated.

At block 706, the current state of the data shard is compared to the target state. For example, in FIG. 1 computing device 114 compares the current state of the data slice to the obtained target state. At block 708, if it is determined that the current state is different from the target state, a plurality of sequential operations are continued to complete the target operation. If the current state is the same as the target state, it means that the scheduled operation has been performed, and no subsequent operation is required.

As shown in FIG. 10 , large data 1002 is divided into multiple data slices, such as data slice 1 1004 , data slice 2 1006 and data slice 3 1008 . Use the target operation in the multiple ordered operations that control multiple data fragments to make the proxy service execute according to the target state, for example, execute at the same time until the data download is completed. After collecting all the data fragments and completing the current target operation, the main The control service makes changes to the target state of the next stage of multiple data units at the same time, so that multiple data units of distributed data can be loaded and other operations performed by computer programs at the same time.

In some embodiments, the plurality of sequential operations include a download operation, and the download operation includes storage addresses of data segments. Computing device 114 also performs a download operation to obtain data slices from storage addresses. In this manner, the data fragments can be quickly obtained from the storage locations of the data fragments.

In some embodiments, after the data slices are allocated to each computing device running the program for processing the data slices, the processing of the data slices on different computing devices is dynamically adjusted according to the processing conditions of the computing devices. If it is determined that the data slice assigned to the computing device is to be executed by an application on the third computing device, the identification of the data slice needs to be deleted from the description file of the program running on the computing device, so that the data slice is no longer processed. deal with. Through this method, data transfer can be quickly realized.

In some embodiments, if the processing capability of the computing device is relatively strong and there is no data fragment processing, data fragments allocated to other computing devices may be adjusted to be processed by the computing device. For example, a program control system managing the programs run by each computing device would coordinate the data slices for processing by computing device 114 . If it is determined that the identification of the transferred data segment is received, the heartbeat information including the identification of the transferred data segment is sent to the second server for acquiring operation information and target status for the transferred data segment. Computing device 114 processes the transfer data segment based on the operational information and the target state of the transfer data segment. Through this method, the processing of transferring data fragments can be realized quickly.

In one example, the computing device 114 obtains an identifier of the transferred data segment, and then adds the identifier into a data description file of an application running on the computing device 114 that processes the data segment. Then the agent will obtain the identifier from the data description file, and then search for corresponding operation information and target status information by sending heartbeat information including the identifier of the transfer data fragment to the main control service.

As shown in FIG. 11 , an application 1114 for processing data slices is running in computing device 1 1110 , and an application 1128 for processing data slices is running in counting device 2 . Proxy 1118 for application 1114 and proxy 1124 for application 1128 communicate with hosting service 1102 . The data slice 1 1120 and the data slice 2 1122 are allocated to the application 1114 in the computing device 1 for processing, and the data slice 3 1130 is allocated to the application 1128 in the computing device 2 for processing. In the computer application 1114, the identifiers of the sliced data files used by the program are recorded in the data description files 1116 and 1126 for the application.

When the program application control system changes the data fragment 2 1122 from being processed by the application 1114 to being processed by the application 1128, first add a new fragment description in the data description file 1126 for the application 1128, and at this time the content of the data description file of the application 1128 Change to Data Shard 2 and Data Shard 3. The agent 1124 of the application 1128 uses the data description information to perform heartbeat communication with the main control service 1102. At this time, the main control service first searches the key information ig_key2 1106 of the computing device cluster corresponding to the distribution version information to find the corresponding data slice 2 through a first-level query. As a result, the operation information and target state information of data slice 2 are not queried, because it is assigned to the computing device cluster whose key information is ig_key1 1104 . At this time, the main control service performs a second-level query, searches the stored information Field_key 1106 of the dictionary information dimension that contains global information, and can query the current version target stage information and operation information of the data distribution of data fragment 2, and then deliver Give the proxy service the correct data shard 2 information. After the proxy service obtains the operation information and target status information of the data shard, it will complete the operation on the data shard. Alternatively or additionally, the identification of data slice 2 1122 is deleted in the data description file 1116 for the application 1114 .

Through this method, the final state consistency in the data distribution stage can be realized, the complexity of the data distribution system can be significantly reduced, and the stability of data processing can be improved.

The method 700 for processing data according to multiple embodiments of the present disclosure has been described above with reference to FIGS. 7-11 . A schematic diagram of a system 1200 for processing data according to some embodiments of the present disclosure is described below with reference to FIG. 12 . As shown in FIG. 12 , the system includes building blocks 1204 . The building block 1204 is responsible for checking the update of the data source, periodically inspecting whether each data file has the latest content change, making a comparison and judgment of the information of the historical version, and generating a new version of the data file if the inspection finds that there is new data content, Assembled with the template configuration information of data distribution, a distributed data distribution to be executed is generated, and delivered to the state machine driver 1206 for execution. The state machine driver 1206 is responsible for recommending the state of distributed data distribution, performing different driver controls according to the single mode and fragmentation mode, coordinating the distribution target machines in each stage, and realizing state coordination and progress control of fragmented data. After the collaborative operation of distributed data is realized, the status of the current stage is recovered and checked, and the execution is promoted to the next stage until the end of the overall data distribution.

System 1200 also includes a master service module 1208 . The main control service module 1208 receives and updates the data file version information of each operation, maintains the current version information and maintains communication with the proxy service, accepts the heartbeat information of the proxy service, and transmits operation information and target status information to the proxy service. At the same time, the data distribution status of computing devices is summarized, and the instance version information of the program is recorded.

System 1200 also includes an agent module 1212 . The proxy module 1212 is responsible for declarative data distribution execution, and updates the status of the data version according to the information obtained from the main control service and the final status. Collect the computer environment information and current data status where the agent service is located, determine the data collected by it, and obtain the final data issued by the main control service. Likewise, the host service 1208 also processes other data 1210 and data 1214 .

Therefore, for a distribution process of a new version of data, first, the data source generates new content, the building module senses the data, assembles and makes decisions with the distribution module, and generates a data distribution to initiate execution. Each stage of data distribution is controlled by the state driver and sent to the main control service module. When the main control service module receives the heartbeat report information of the proxy service, it notifies and sends the new version information of the data file. In the agent service, according to the version information and final state of the data file, the consistency catch-up and data distribution are completed.

Fig. 13 shows a schematic block diagram of an apparatus 1300 for processing data according to an embodiment of the present disclosure. As shown in FIG. 13 , the apparatus 1300 includes: a distribution information acquisition module 1302 configured to obtain, at the first server, the distribution information of the new version data generated by the data source, and the distribution information includes a plurality of data into which the new version data is divided. A plurality of storage addresses of the fragments and operation information corresponding to a plurality of sequential operations for the new version data; the operation information configuration module 1304 is configured to use the storage addresses of the data fragments among the plurality of data fragments to configure all The above operation information is used to generate configured operation information for data fragmentation; the target state information determination module 1306 is configured to determine the target state information corresponding to the target operation to be completed in the plurality of ordered operations; and the sending module 1308 , configured to send the configured operation information and target state information for the data shards to the second server for processing the data shards.

In some embodiments, the operation information configuration module 1304 includes: a storage address obtaining module, configured to obtain the storage address for the data slice from the allocation information; and a storage address association module, configured to associate the storage address with the operation Part of the information in the information is used to generate configured operation information for data fragments, and the part of information corresponds to a download operation among the multiple sequential operations.

In some embodiments, the allocation information further includes a target cluster identifier of a target computing device cluster corresponding to each of the multiple release stages of the new version data, wherein the target state information determining module 1306 includes: the target cluster identifier The determination module is configured to determine the target cluster identifier corresponding to the release phase for the release phase; the target operation determination module is configured to determine the target operation to be completed in a plurality of sequential operations; and the target state information generation module is Configured to generate target status information based on the target operation and the target cluster ID.

In some embodiments, the apparatus 1300 further includes: a current status information acquisition module configured to acquire the current status information of multiple computing devices corresponding to the target cluster identifier, the current status information indicating that the multiple computing devices have completed multiple An operation in an ordered operation; a matching determination module configured to determine whether the current state information matches the target state information; an update module configured to update the target state information to match the target state information if it is determined that the current state information matches the target state information corresponding to the next operation of the operation; and an update sending module configured to send the updated target state to the second server for completing the next operation.

In some embodiments, the apparatus 1300 further includes: a monitoring module configured to monitor whether the version of the data generated by the data source changes; and a first division module configured to convert the new version Data is divided into multiple data shards.

In some embodiments, the first division module includes: a predetermined data acquisition module configured to acquire a predetermined number of data fragments to be generated; and a second division module configured to divide the new version data based on the predetermined number.

Fig. 14 shows a schematic block diagram of an apparatus 1400 for processing data according to an embodiment of the present disclosure. As shown in FIG. 14 , the apparatus 1400 includes: an operation information and target state information receiving module 1402 configured to receive, at the second server, configured operation information and target state information for data fragments from the first server, the Data sharding comes from multiple data shards generated by dividing the new version data generated by the data source. The configured operation information is related to multiple ordered operations for data shards, and the target state is related to multiple ordered operations. corresponding to the target operation to be completed; the first operation information and target state information sending module 1404 is configured to send the operation to the first computing device in response to receiving the first heartbeat information for data fragmentation from the first computing device information and target state information, the first heartbeat information includes a current state for the data slice; and an update module 1406 configured to update the current state information of the first computing device with the current state of the data slice.

In some embodiments, the apparatus 1400 further includes: an updated target state information receiving module configured to receive updated target state information from the first server, the updated target state information corresponds to the next operation of the target operation and the operation information and the updated target state information sending module, configured to send the operation information and the updated target state information to the first computing device in response to receiving the second heartbeat information for the data fragmentation to complete the next operation.

In some embodiments, the first heartbeat information further includes the reference cluster identifier of the reference computing device cluster where the first computing device is located, and the target state information includes information about each of the multiple release stages of the new version data. The target cluster identifier of the corresponding target computing device cluster; wherein the first operation information and target status information sending module 1404 includes: a matching module configured to match the reference cluster identifier with the target cluster identifier; and the second operation information and the target cluster identifier The status information sending module is configured to send the operation information and the target status information to the first computing device if it is determined that the reference cluster ID matches the target cluster ID.

In some embodiments, the apparatus 1400 further includes: a first search module configured to, in response to receiving from the first computing device third heartbeat information including an identification of the transfer data segment, in the cluster associated with the reference computing device Find the operation information and target state information of the transfer data fragment in the first state information of the partial data fragment, and transfer the data fragment from the second computing device to the first computing device; the second search module is configured to if in The operation information and target state of transferring data fragments are not found in the first state information, and the operation information and target state of transferring data fragments are found in the second state information corresponding to multiple data fragments; and the search result A sending module configured to send the found operation information and target status of the transferred data fragments to the first computing device.

Fig. 15 shows a schematic block diagram of an apparatus 1500 for processing data according to an embodiment of the present disclosure. As shown in FIG. 15 , the apparatus 1500 includes: an identification acquisition module 1502 configured to acquire, at the first computing device, the identification of the data fragment to be processed, and the data fragment comes from dividing the new version data generated by the data source. A plurality of generated data fragments; the heartbeat information sending module 1504, configured to send a heartbeat message for the data fragments to the second server for receiving operation information and target state information for the data fragments, and the operation information is related to the data The multiple ordered operations of the slice are related, and the target state corresponds to the target operation to be completed in the multiple ordered operations; the comparison module 1506 is configured to compare the current state of the data slice with the target state; and the operation execution Module 1508, configured to continue executing multiple sequential operations to complete the target operation if it is determined that the current state is different from the target state.

In some embodiments, the heartbeat information sending module 1504 includes: a current state determination module configured to determine the current state of the data fragmentation; Status generates heartbeat information.

In some embodiments, the heartbeat information sending module 1504 further includes: a periodic module configured to periodically send heartbeat information for updating the current state information of the first computing device processing the slice.

In some embodiments, wherein the plurality of ordered operations includes a download operation, and the download operation includes a storage address of the data slice; the device further includes: a download operation execution module configured to execute the download operation to obtain the data slice from the storage address .

In some embodiments, the apparatus 1500 further includes: a deletion module configured to delete the identifier of the data slice if it is determined that the data slice is to be executed by an application on the third computing device.

In some embodiments, the apparatus 1500 further includes: a sending module configured to, if it is determined that the identifier of the transferred data fragment is received, send heartbeat information including the identifier of the transferred data fragment to the second server for obtaining operation information and target state of the slice, the transfer data slice is transferred from the second computing device to the first computing device; and the transfer data slice processing module is configured to transfer the data slice based on the operation information and the target state to handle transferring data shards.

In the technical solution of the present disclosure, the acquisition, storage and application of the user's personal information involved are in compliance with relevant laws and regulations, and do not violate public order and good customs.

According to the embodiments of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

FIG. 16 shows a schematic block diagram of an example electronic device 1600 that may be used to implement embodiments of the present disclosure. The example electronic device 1600 may be used to implement server 106 , server 112 , and computing device 114 in FIG. 1 . Electronic device is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are by way of example only, and are not intended to limit implementations of the disclosure described and/or claimed herein.

As shown in FIG. 16, the device 1600 includes a computing unit 1601 that can execute according to a computer program stored in a read-only memory (ROM) 1602 or loaded from a storage unit 1608 into a random access memory (RAM) 1603. Various appropriate actions and treatments. In the RAM 1603, various programs and data necessary for the operation of the device 1600 can also be stored. The calculation unit 1601 , the ROM 1602 and the RAM 1603 are connected to each other through a bus 1604 . An input/output (I/O) interface 1605 is also connected to the bus 1604 .

Multiple components in the device 1600 are connected to the I/O interface 1605, including: an input unit 1606, such as a keyboard, a mouse, etc.; an output unit 1607, such as various types of displays, speakers, etc.; a storage unit 1608, such as a magnetic disk, an optical disk, etc. ; and a communication unit 1609, such as a network card, a modem, a wireless communication transceiver, and the like. The communication unit 1609 allows the device 1600 to exchange information/data with other devices through a computer network such as the Internet and/or various telecommunication networks.

The computing unit 1601 may be various general-purpose and/or special-purpose processing components with processing and computing capabilities. Some examples of computing units 1601 include, but are not limited to, central processing units (CPUs), graphics processing units (GPUs), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processing processor (DSP), and any suitable processor, controller, microcontroller, etc. The calculation unit 1601 executes various methods and processes described above, such as methods 200 , 500 and 700 . For example, in some embodiments, methods 200 , 500 , and 700 may be implemented as a computer software program tangibly embodied on a machine-readable medium, such as storage unit 1608 . In some embodiments, part or all of the computer program may be loaded and/or installed on the device 1600 via the ROM 1602 and/or the communication unit 1609 . When a computer program is loaded into RAM 1603 and executed by computing unit 1601, one or more steps of methods 200, 500 and 700 described above may be performed. Alternatively, in other embodiments, the computing unit 1601 may be configured to execute the methods 200, 500 and 700 in any other suitable manner (for example, by means of firmware).

Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips Implemented in a system of systems (SOC), load programmable logic device (CPLD), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include being implemented in one or more computer programs executable and/or interpreted on a programmable system including at least one programmable processor, the programmable processor Can be special-purpose or general-purpose programmable processor, can receive data and instruction from storage system, at least one input device, and at least one output device, and transmit data and instruction to this storage system, this at least one input device, and this at least one output device an output device.

Program codes for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special purpose computer, or other programmable data processing devices, so that the program codes, when executed by the processor or controller, make the functions/functions specified in the flow diagrams and/or block diagrams Action is implemented. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.

In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media would include one or more wire-based electrical connections, portable computer discs, hard drives, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), optical fiber, compact disk read only memory (CD-ROM), optical storage, magnetic storage, or any suitable combination of the foregoing.

To provide for interaction with the user, the systems and techniques described herein can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user. ); and a keyboard and pointing device (eg, a mouse or a trackball) through which a user can provide input to the computer. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and can be in any form (including Acoustic input, speech input or, tactile input) to receive input from the user.

The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., as a a user computer having a graphical user interface or web browser through which a user can interact with embodiments of the systems and techniques described herein), or including such backend components, middleware components, Or any combination of front-end components in a computing system. The components of the system can be interconnected by any form or medium of digital data communication, eg, a communication network. Examples of communication networks include: Local Area Network (LAN), Wide Area Network (WAN) and the Internet.

A computer system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, a server of a distributed system, or a server combined with a blockchain.

It should be understood that steps may be reordered, added or deleted using the various forms of flow shown above. For example, each step described in the present disclosure may be executed in parallel, sequentially, or in a different order, as long as the desired result of the technical solution disclosed in the present disclosure can be achieved, no limitation is imposed herein.

The specific implementation manners described above do not limit the protection scope of the present disclosure. It should be apparent to those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made depending on design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present disclosure shall be included within the protection scope of the present disclosure.

Claims (30)

1. A method for processing data, comprising:

obtaining, at a first server, allocation information of a new version of data generated by a data source, the allocation information including a plurality of storage addresses of a plurality of data slices into which the new version of data is partitioned and operation information corresponding to a plurality of ordered operations for the new version of data, and a target cluster identification of a target computing device cluster corresponding to each of a plurality of release phases of the new version of data;

Configuring the operation information using a storage address of a data shard of the plurality of data shards to generate configured operation information for the data shard;

determining target state information corresponding to a target operation to be completed in the plurality of ordered operations; and

sending the configured operational information and the target state information for the data shard to a second server for processing the data shard;

wherein determining the target state information comprises:

determining a target cluster identifier corresponding to each release phase;

determining a target operation to be completed in the plurality of ordered operations; and

and generating the target state information based on the target operation and the target cluster identification.

2. The method of claim 1, wherein configuring the operation information comprises:

acquiring a storage address for the data fragment from the allocation information; and

the memory address is associated into partial information in the operation information to generate the configured operation information for the data slice, the partial information corresponding to a download operation of the plurality of ordered operations.

3. The method of claim 1, further comprising:

obtaining current state information of a plurality of computing devices corresponding to the target cluster identification, the current state information indicating operations of the plurality of ordered operations that the plurality of computing devices have completed;

determining whether the current state information matches the target state information;

if the current state information is determined to be matched with the target state information, updating the target state information to correspond to the next operation of the target operation; and

the updated target state information is sent to a second server for completion of the next operation.

4. The method of claim 1, further comprising:

monitoring whether a version of data generated by the data source changes; and

and if the version of the data is determined to change, dividing the new version of data into the plurality of data fragments.

5. The method of claim 4, wherein dividing the new version of data into a plurality of data slices comprises:

acquiring a predetermined number of data fragments to be generated; and

the new version data is partitioned based on the predetermined number.

6. A method for processing data, comprising:

receiving, at a second server, from a first server, configured operation information and target state information for a data shard from a plurality of data shards generated by partitioning new version data generated by a data source, the configured operation information relating to a plurality of ordered operations for the data shard, the target state information corresponding to a target operation to be completed among the plurality of ordered operations;

in response to receiving first heartbeat information for the data shard from a first computing device, sending the operation information and the target state information to the first computing device, the first heartbeat information including a reference cluster identity for a current state of the data shard and a reference computing device cluster in which the first computing device is located, the target state information including a target cluster identity of a target computing device cluster corresponding to each of a plurality of release phases of the new version data; and

updating current state information of the first computing device with the current state of the data slice;

Wherein sending the operation information and the target state information to the first computing device comprises:

matching the reference cluster identifier with the target cluster identifier; and

and if the reference cluster identity is determined to match the target cluster identity, transmitting the operation information and the target state information to the first computing device.

7. The method of claim 6, further comprising:

receiving updated target state information from the first server, the updated target state information corresponding to a next operation of the target operation; and

in response to receiving second heartbeat information for the data slice, the operation information and the updated target state information are sent to the first computing device for completion of the next operation.

8. The method of claim 6, further comprising:

in response to receiving third heartbeat information from the first computing device including an identification of a transferred data shard, looking up operation information and target state information of the transferred data shard in first state information of partial data shards associated with the reference computing device cluster, the transferred data shard transferred from a second computing device onto the first computing device;

If the operation information and the target state information of the transferred data fragments are not found in the first state information, the operation information and the target state information of the transferred data fragments are found in the second state information corresponding to the plurality of data fragments; and

and sending the searched operation information and target state information of the transferred data fragment to the first computing equipment.

9. A method for processing data, comprising:

obtaining, at a first computing device, an identification of a data shard to be processed from a plurality of data shards generated by partitioning new version data generated by a data source;

sending a heartbeat message for the data shard to a second server for receiving operation information and target state information for the data shard, the operation information relating to a plurality of ordered operations for the data shard, the target state information corresponding to a target operation to be completed in the plurality of ordered operations;

wherein sending the heartbeat message includes:

determining a current state for the data slice;

generating the heartbeat information based on the identification of the data fragment and the current state; and

Periodically transmitting the heartbeat information for updating current state information of a first computing device processing the shard;

comparing the current state of the data fragment with the target state information; and

if it is determined that the current state is different from the target state information, continuing to perform the plurality of ordered operations to complete the target operation.

10. The method of claim 9, wherein the plurality of ordered operations comprises a download operation comprising a storage address of the data chunk; the method further comprises the steps of:

and executing the downloading operation to acquire the data fragments from the storage address.

11. The method of claim 9, further comprising:

if it is determined that the data shard is to be executed by an application on a third computing device, the identification of the data shard is deleted.

12. The method of claim 9, further comprising:

if it is determined that the identification of the transferred data fragment is received, sending heartbeat information including the identification of the transferred data fragment to a second server for obtaining operation information and target state information for the transferred data fragment, the transferred data fragment being transferred from a second computing device to the first computing device; and

The transferred data shard is processed based on the operation information and the target state information of the transferred data shard.

13. An apparatus for processing data, comprising:

an allocation information acquisition module configured to acquire allocation information of a new version of data generated by a data source at a first server, the allocation information including a plurality of storage addresses of a plurality of data slices into which the new version of data is divided and operation information corresponding to a plurality of ordered operations for the new version of data, and a target cluster identification of a target computing device cluster corresponding to each of a plurality of release phases of the new version of data;

an operation information configuration module configured to configure the operation information using a storage address of a data shard of the plurality of data shards to generate configured operation information for the data shard;

a target state information determination module configured to determine target state information corresponding to a target operation to be completed among the plurality of ordered operations; and

a transmitting module configured to transmit the configured operation information and the target state information for the data shard to a second server for processing the data shard;

Wherein the target state information determination module comprises:

a target cluster identity determination module configured to determine, for each publication phase, a target cluster identity corresponding to the publication phase;

a target operation determination module configured to determine a target operation to be completed among the plurality of ordered operations; and

a target state information generation module configured to generate the target state information based on the target operation and the target cluster identity.

14. The apparatus of claim 13, wherein the operation information configuration module comprises:

a storage address acquisition module configured to acquire a storage address for the data fragment from the allocation information; and

a storage address association module configured to associate the storage address into partial information in the operation information to generate the configured operation information for the data fragment, the partial information corresponding to a download operation of the plurality of ordered operations.

15. The apparatus of claim 13, further comprising:

a current state information acquisition module configured to acquire current state information of a plurality of computing devices corresponding to the target cluster identification, the current state information indicating an operation of the plurality of ordered operations that the plurality of computing devices have completed;

A match determination module configured to determine whether the current state information matches the target state information;

an updating module configured to update the target state information to correspond to a next operation of the target operation if it is determined that the current state information matches the target state information; and

and an update transmission module configured to transmit the updated target state information to a second server for completing the next operation.

16. The apparatus of claim 13, further comprising:

a monitoring module configured to monitor whether a version of data generated by the data source has changed; and

and the first dividing module is configured to divide the new version data into the plurality of data fragments if the version of the data is determined to change.

17. The device of claim 16, wherein the first partitioning module comprises:

a predetermined data acquisition module configured to acquire a predetermined number of data slices to be generated; and

and a second dividing module configured to divide the new version data based on the predetermined number.

18. An apparatus for processing data, comprising:

An operation information and target state information receiving module configured to receive, at a second server, configured operation information and target state information for a data shard from a first server, the data shard being from a plurality of data shards generated by dividing new version data generated by a data source, the configured operation information being related to a plurality of ordered operations for the data shard, the target state information corresponding to a target operation to be completed among the plurality of ordered operations;

a first operation information and target state information sending module configured to send the operation information and the target state information to a first computing device in response to receiving first heartbeat information for the data shard from the first computing device, the first heartbeat information including a reference cluster identity for a current state of the data shard and a reference computing device cluster in which the first computing device is located, the target state information including a target cluster identity of a target computing device cluster corresponding to each of a plurality of release phases of the new version data; and

an update module configured to update current state information of the first computing device with a current state of the data slice;

Wherein the first operation information and target state information transmitting module includes:

a matching module configured to match the reference cluster identity with the target cluster identity; and

and a second operation information and target state information sending module configured to send the operation information and the target state information to the first computing device if it is determined that the reference cluster identity matches the target cluster identity.

19. The apparatus of claim 18, further comprising:

an updated target state information receiving module configured to receive the updated target state information from the first server, the updated target state information corresponding to a next operation of the target operation; and

an operation information and updated target state information transmission module configured to transmit the operation information and updated target state information to the first computing device for completing the next operation in response to receiving second heartbeat information for the data slice.

20. The apparatus of claim 18, further comprising:

a first lookup module configured to, in response to receiving third heartbeat information from the first computing device including an identification of a transferred data patch, lookup operation information and target state information of the transferred data patch in first state information of a portion of the data patch associated with the reference computing device cluster, the transferred data patch transferred from a second computing device onto the first computing device;

A second search module configured to search, if the operation information and the target state information of the transferred data patch are not found in the first state information, the operation information and the target state information of the transferred data patch in second state information corresponding to the plurality of data patches; and

and the searching result sending module is configured to send the searched operation information and target state information of the transfer data fragments to the first computing device.

21. An apparatus for processing data, comprising:

an identification acquisition module configured to acquire, at a first computing device, an identification of a data shard to be processed from a plurality of data shards generated by partitioning new version data generated by a data source;

a heartbeat information sending module configured to send a heartbeat message for the data shard to a second server for receiving operation information for the data shard and target state information, the operation information relating to a plurality of ordered operations for the data shard, the target state information corresponding to a target operation to be completed among the plurality of ordered operations;

Wherein the heartbeat information sending module includes:

a current state determination module configured to determine a current state for the data slice; and

a heartbeat information generating module configured to generate the heartbeat information based on the identification of the data slice and the current state;

a periodicity module configured to periodically send the heartbeat information for updating current state information of the first computing device processing the shards

A comparison module configured to compare a current state of the data slice with the target state information; and

and an operation execution module configured to continue executing the plurality of ordered operations to complete the target operation if the current state is determined to be different from the target state information.

22. The apparatus of claim 21, wherein the plurality of ordered operations comprises a download operation comprising a storage address of the data chunk; the apparatus further comprises:

and the download operation execution module is configured to execute the download operation to acquire the data fragments from the storage address.

23. The apparatus of claim 21, further comprising:

And a deletion module configured to delete the identification of the data shard if it is determined that the data shard is to be executed by an application on a third computing device.

24. The apparatus of claim 21, further comprising:

a sending module configured to send, if it is determined that an identification of a transferred data patch is received, heartbeat information including the identification of the transferred data patch to a second server for obtaining operation information and target state information for the transferred data patch, the transferred data patch being transferred from a second computing device onto the first computing device; and

and a transfer data fragment processing module configured to process the transfer data fragment based on the operation information and the target state information of the transfer data fragment.

25. An electronic device, comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein,,

the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-5.

26. An electronic device, comprising:

At least one processor; and

a memory communicatively coupled to the at least one processor; wherein,,

the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 6-8.

27. An electronic device, comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein,,

the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 9-12.

28. A non-transitory computer readable storage medium storing computer instructions for causing the computer to perform the method of any one of claims 1-5.

29. A non-transitory computer readable storage medium storing computer instructions for causing the computer to perform the method of any one of claims 6-8.

30. A non-transitory computer readable storage medium storing computer instructions for causing the computer to perform the method of any one of claims 9-12.