CN115563217A - Data processing method, system, device, medium, and program product - Google Patents

Abstract

The invention relates to the field of computer data processing, in particular to a data processing method, which comprises a data acquisition step, a data processing step and a data processing step, wherein the data acquisition step is used for acquiring data from a data source; a data distribution step, distributing the data based on the unique identifier of the data; and a data transmission step of transmitting the data and updating site information based on a processing result of the data, the site information being a boundary point between processed and unprocessed data. The invention also relates to a data processing system, device, medium and program product. The invention ensures the high-efficiency synchronization of the data by adopting a multithread memory caching strategy, simultaneously ensures the orderliness of the data during the synchronization by using the same type of data distribution, and updates the site information after the data processing is finished so as to ensure the reliability of the data.

Description

Data processing method, system, device, medium, and program product

Technical Field

The present invention relates to the field of computer data processing, and in particular, to a data processing method, system, device, medium, and program product.

Background

In an actual service scene, how to ensure high efficiency and reliability simultaneously in the process of data synchronization becomes a problem which needs to be continuously solved by the industry at present. The data are guaranteed not to be lost under any condition while the data are efficiently synchronized, and the method becomes a core problem to be solved urgently in the industry.

At present, if data reliability is required to be ensured by data synchronization tools on the market, a disk cache mode is adopted, and because disk reading and writing are required, the synchronization efficiency is extremely low. If high-efficiency synchronization is needed, a memory caching mode is adopted, but data loss exists after downtime, and the memory caching mode and the data loss cannot obtain a balance point between performance and reliability.

The invention provides a high-performance and high-reliability data synchronization technology, which can ensure that data can be synchronized efficiently without data loss under any condition.

Disclosure of Invention

The invention aims to provide a data processing method, a system, equipment, a medium and a program product, which solve the difficult problem in data synchronization by processing data information and recording site information, support the high performance and integrity of data synchronization, reduce the delay of data synchronization, ensure the integrity of data and avoid the loss of any piece of data, thereby ensuring the correctness, real-time property and integrity of background calculation.

The embodiment of the invention discloses data processing, and the method comprises the following steps:

a data acquisition step of acquiring data from a data source;

a data distribution step, distributing the data based on the unique identifier of the data;

and a data transmission step of transmitting the data and updating site information based on a processing result of the data, the site information being a boundary point between processed and unprocessed data.

Optionally, the data offloading step further includes performing hash calculation on the unique identifier, and adding data with the same unique identifier into the same memory queue.

Optionally, in the data sending step, updating the location information based on the processing result of the data includes updating the location information in the memory queue when the processing of the data is completed, and resending the data when the processing of the data is not completed.

An embodiment of the present invention discloses a data processing system, the system comprising:

the data acquisition module acquires data from a data source;

the data distribution module distributes the data based on the unique identifier of the data;

and the data sending module is used for sending the data and updating site information based on the processing result of the data, wherein the site information is a boundary point between the processed data and the unprocessed data.

Optionally, the data offloading module further includes a computing module, where the computing module performs hash computation on the unique identifier, and adds data with the same unique identifier to the same memory queue.

Optionally, there are multiple data sending modules, and the multiple data sending modules correspond to the memory queues one to one.

The invention discloses an electronic device, which is characterized by comprising a memory and a processor, wherein the memory stores computer-executable instructions, and the processor is configured to execute the instructions to implement the data processing method.

The embodiment of the invention discloses a computer-readable storage medium, which is characterized in that at least one computer instruction is stored in the computer-readable storage medium, and the at least one instruction is loaded and executed by a processor to realize the data processing method.

The embodiment of the invention discloses a computer program product, which is characterized by comprising computer instructions, wherein the computer instructions are executed to realize the data processing method.

The invention provides a data synchronization technology, which ensures the orderliness of data by processing of shunting data and updating site information so as to support the high performance and integrity of data synchronization, reduce the delay of data synchronization, ensure the integrity of data and avoid the loss of any piece of data.

Compared with the prior art, the implementation mode of the invention has the main differences and the effects that: on the premise of efficiently synchronizing data, the problem of data loss is avoided under any condition, and meanwhile, the data order is ensured in the synchronizing process.

In the prior art, the synchronous data adopts a disk cache mode, and the synchronization efficiency is extremely low because disk reading and writing are required. If high-efficiency synchronization is needed, a memory caching mode is adopted, but data loss exists after downtime, and the memory caching mode and the data loss cannot obtain a balance point between performance and reliability.

The data processing method has the technical effects that the data with the unique identifier are shunted to ensure the orderliness of the data, the site information in the cache is updated based on whether the data is processed or not, and the site information is a dividing point of the processed data and the unprocessed data, so that the data can be efficiently synchronized, and the problem that the data can be lost in any link in the data synchronization process is solved.

Drawings

Fig. 1 is a schematic view of a data processing method applied in accordance with an embodiment of the present invention;

FIG. 2 is a flow chart of a method of data processing according to an embodiment of the present invention;

FIG. 3 is a schematic diagram of streaming data of a data processing method according to an embodiment of the present invention;

fig. 4 is a schematic diagram of transmission data of a data processing method according to an embodiment of the present invention;

FIG. 5 is a block diagram of a data processing system according to an embodiment of the present invention;

fig. 6 is a block diagram of a hardware configuration of an electronic device according to an embodiment of the present invention.

Detailed Description

The invention is further illustrated with reference to the following specific embodiments and the accompanying drawings. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not to be construed as limiting the invention. In addition, for convenience of description, only a part of structures or processes related to the present invention, not all of them, is illustrated in the drawings. It should be noted that in this specification, like reference numerals and letters refer to like items in the following drawings.

It will be understood that, although the terms "first", "second", etc. may be used herein to describe various features, these features should not be limited by these terms. These terms are used merely for distinguishing and are not intended to indicate or imply relative importance. For example, a first feature may be termed a second feature, and, similarly, a second feature may be termed a first feature, without departing from the scope of example embodiments.

In order to make the objects, technical solutions and advantages of the present invention more apparent, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

Fig. 1 is a schematic view of a scenario in which a data processing method according to an embodiment of the present invention is applied.

As shown in fig. 1, the data processing method includes a data source 100, a server 200, and a data receiver 300, where the server 200 includes a data pulling end 210, a stream splitter 220, a memory queue 230, and a sending end 240. The data source 100 and the server side 200 communicate with each other via a wired/wireless network connection, and also enable user access via a wired or wireless network.

Fig. 1 shows one data source 100, and it should be noted that, as those skilled in the art can understand, the number of the data sources 100 is not limited to one, and may be one data source 100, or may be several data sources 100, the number of the data sources 100 should not be taken as a limitation of the present invention, and meanwhile, the data source 100 may be any client or server that provides data, and a data processing method of the present invention may be applied to an electronic device that needs to synchronize data while ensuring that no data is lost.

The server 200 can synchronize data and ensure the data order, so that no data is lost in the data synchronization process.

The data receiver 300 may be any client or server that receives data.

At present, the synchronous data in the market all adopt a disk cache mode, and the synchronization efficiency is extremely low because disk reading and writing are required. If high-efficiency synchronization is needed, a memory caching mode is adopted, but data loss exists after downtime, and the memory caching mode and the data loss cannot obtain a balance point between performance and reliability.

In view of the above problems, the present invention provides a data processing method. The method is described in detail below with reference to the accompanying drawings.

Fig. 2 is a flow chart of a data processing method according to an embodiment of the present invention.

As shown in fig. 2, a data processing method according to an embodiment of the present invention requires cooperation of a data source 100, a data pull end 210, a stream splitter 220, a memory queue 230, a sending end 240, and a data receiving end 300 in steps S100 to S300, and the method includes:

step S100: data from a data source 100 is acquired.

The big data acquisition mode can adopt the modes of database acquisition, system log acquisition, network data acquisition and sensing equipment data acquisition. The traditional relational databases MySQL and Oracle can store data, in the big data era, noSQL databases such as Redis, mongoDB, HBase and the like are also commonly used for data acquisition, and the big data acquisition work is completed by deploying a large number of databases at the acquisition end and performing load balancing and fragmentation among the databases; the system log collection is mainly used for collecting a large amount of log data generated by a data platform in daily life and providing the log data for an offline and online big data analysis system. High availability, high reliability and expandability are basic characteristics of the log collection system. The system log acquisition tools all adopt a distributed architecture, and can meet the requirements of acquisition and transmission of log data of hundreds of MB per second; network data acquisition refers to a process of acquiring data information from a website in a manner of a web crawler or a website public API or the like. The web crawler obtains contents on each webpage from the URLs of one or a plurality of initial webpages, continuously extracts new URLs from the current webpage and puts the new URLs into a queue in the process of capturing the webpage until the set stop condition is met, so that unstructured data and semi-structured data can be extracted from the webpage and stored in a local storage system; the data acquisition of the sensing equipment refers to acquiring data by automatically acquiring signals, pictures or videos through a sensor, a camera and other intelligent terminals. The big data intelligent perception system needs to realize intelligent identification, positioning, tracking, access, transmission, signal conversion, monitoring, primary processing, management and the like of structured, semi-structured and unstructured mass data.

The collected data can be used as the data source 110 acquired by the server 200 of the data processing method from various data platforms.

Step S200: the data is streamed based on the unique identifier of the data.

The hash algorithm is a generalized algorithm, and can improve the utilization rate of a storage space, improve the query efficiency of data and ensure the safety of data transmission by making a digital signature. The hash algorithm is widely used in internet applications.

The hash operation is performed on the unique identifier of the data obtained by the data source 100, so that the data with the same unique identifier can conveniently enter the same memory queue 230.

In one embodiment, the data pull end 210 obtains data in the data source 100, and the splitter 220 performs a hash operation on the unique identifier of the data, and adds the data with the same unique identifier to the same memory queue 230 through the hash operation.

The splitter is an operation program or device that identifies, operates, and processes a unique identifier of data.

In one example, the unique identifier of the data is composed of a plurality of characters, and the splitter refers to an algorithm program or device having a function of performing a hash operation on the unique identifier of the data, and finally sending the operation result to the memory queue 230 according to an allocation rule, for example, the allocation rule may be based on the same unique identifier.

It should be noted that, as those skilled in the art can understand, there are a plurality of memory queues, and the number of the memory queues is set based on the type of the identifier, so the number of the memory queues is not a limitation condition of a data processing method of the present invention.

In one embodiment, there are several pieces of data with unique identifiers, wherein the data with the same unique identifier is grouped into one memory queue 230, and there are several memory queues 230 that can hold all data, and each memory queue 230 temporarily stores the data with the same unique identifier.

Step S300: and sending the data to a data receiving end, and updating site information based on the processing result of the data, wherein the site information is a boundary point between the processed data and the unprocessed data.

The sending of data is performed by the sender 240 in the server side 200.

In one embodiment, the initiator 240 is in one-to-one correspondence with the memory queues 230, that is, each memory queue 230 that stores the same unique identifier has one initiator 240 corresponding thereto. Those skilled in the art can understand that the number of the sending end 240 may be one or several, and the number of the sending end 240 is based on the number of the memory queue 230, and therefore, the number of the sending end 240 is not a limitation condition of a data processing method of the present invention.

As an embodiment, the sending end 240 obtains data with the same unique identifier from the memory queue 230 corresponding to the sending end, and sends the data with the unique identifier to the data receiving end 300, the data receiving end 300 processes the data, and the sending end 240 updates the location information based on the processing result of the data.

In an example, the sending end 240 obtains data with the same unique identifier from the memory queue 230 corresponding to the sending end 240, and sends the data with the unique identifier to the data receiving end 300, the data receiving end 300 processes the data, and the data receiving end 300 completes processing the data, at this time, the sending end 240 receives a command that the task is successful, and the sending end 240 updates the location information in the memory queue 230.

In another example, the sending end 240 obtains data with the same unique identifier from the corresponding memory queue 230, and sends the data with the unique identifier to the data receiving end 300, the data receiving end 300 processes the data, and the data receiving end 300 does not complete processing the data, at this time, the sending end 240 receives an instruction of a task failure, and the sending end 240 resends the data to the data receiving end 300 until the data receiving end 300 completes processing the data, and the sending end 240 receives the instruction of a task success, and the sending end 240 updates the location information in the memory queue 230.

The site information is a demarcation point between processed data and unprocessed data, is position identification information, does not change the original data and the data position thereof, and can ensure the orderliness of the data.

Preferably, the location information is in the memory queue 230.

According to the data processing method, the data with the unique identifier is shunted to ensure the orderliness of the data, the site information in the cache is updated based on whether the data is processed or not, the data can be efficiently synchronized, and the difficult problem that the data may be lost in any link in the data synchronization process can be solved.

Fig. 3 is a schematic diagram of streaming data of a data processing method according to an embodiment of the present invention.

As shown in fig. 3, the data pulling end 210 sends data to the stream splitter 220, the stream splitter performs hash operation on the unique identifier of the data, the operation rule is that data with the same unique identifier is added to the same memory queue 230, the number of the memory queues 230 is several, based on that all data with different unique identifiers are actually and completely accommodated, one end of each of the different memory queues 230 has a unique sending end 240, and the number of the sending ends 230 corresponds to that of the memory queue 230.

The data pulling end 210 obtains data from cached location information, where the location information is located in the data source 100, and the location information is location information describing a boundary between processed data and unprocessed data.

Fig. 4 is a diagram illustrating transmitted data of a data processing method according to an embodiment of the present invention.

As shown in fig. 4, the sender 240 continuously obtains data from the memory queue 230, and as can be seen from fig. 3, each sender 240 corresponds to one memory queue 230 with the same unique identifier, so that the ordering of data can be ensured.

The sending end 240 sends the data to the data receiving end 300, the sending end 240 judges the actual condition of the task executed by the data receiving end 300, and if the data receiving end finishes processing the data, the sending end 240 updates the site information of the memory queue 230; if the data receiving end does not complete the processing of the data, the sending end 240 does not update the location point information of the memory queue 230, and the sending end 240 resends the data to the data receiving end 300 until the data receiving end 300 completes the processing task of the data, at this time, the sending end 240 updates the location point information of the memory queue 230.

FIG. 5 is a block diagram of a data processing system according to an embodiment of the present invention.

As shown in fig. 5, the system 400 includes a data obtaining module 410, a data splitting module 420, and a data sending module 430;

a data acquisition module 410 for acquiring data from a data source;

a data splitting module 420 for splitting the data based on the unique identifier of the data;

and a data transmission module 430 for transmitting the data and updating the site information based on the processing result of the data, wherein the site information is a boundary point between the processed data and the unprocessed data.

In one embodiment, the data distribution module 420 further includes a calculation module, which can perform a hash operation on the unique identifier of the data and add the data with the same unique identifier to the same memory queue 230.

In one embodiment, the number of the data sending modules is multiple, and the multiple data sending modules correspond to the memory queues one by one.

This embodiment is a method embodiment corresponding to the above embodiment, and this embodiment can be implemented in cooperation with the above embodiment. The related technical details mentioned in the foregoing embodiments are still valid in this embodiment, and are not described herein again in order to reduce repetition. Accordingly, the related technical details mentioned in the present embodiment can also be applied to the foregoing embodiments.

According to some embodiments of the invention, an electronic device is disclosed, the device comprising a memory storing computer-executable instructions and a processor configured to execute the instructions to implement a data processing method.

Fig. 6 is a block diagram of a hardware configuration for implementing an electronic device according to an embodiment of the present invention.

As shown in fig. 6, electronic device 600 may include one or more processors 602, a system motherboard 608 connected to at least one of processors 602, system memory 605 connected to system motherboard 608, non-volatile memory (NVM) 606 connected to system motherboard 608, and a network interface 610 connected to system motherboard 608.

Processor 602 may include one or more single-core or multi-core processors. The processor 602 may include any combination of general-purpose processors and dedicated processors (e.g., graphics processors, application processors, baseband processors, etc.). In an embodiment of the invention, the processor 602 may be configured to perform a method according to the method as shown in fig. 2.

In some embodiments, the system motherboard 608 may include any suitable interface controllers to provide any suitable interface to at least one of the processors 602 and/or any suitable device or component in communication with the system motherboard 608.

In some embodiments, the system motherboard 608 may include one or more memory controllers to provide an interface to the system memory 605. System memory 605 may be used to load and store data and/or instructions. In some embodiments, system memory 605 of electronic device 600 may include any suitable volatile memory, such as suitable Dynamic Random Access Memory (DRAM).

NVM 606 may include one or more tangible, non-transitory computer-readable media for storing data and/or instructions. In some embodiments, NVM 606 may include any suitable non-volatile memory such as flash memory and/or any suitable non-volatile storage device, such as at least one of a HDD (Hard Disk Drive), CD (Compact Disc) Drive, DVD (Digital Versatile Disc) Drive.

NVM 606 may include a portion of a storage resource installed on a device of electronic device 600, or it may be accessible by, but not necessarily a part of, the device. For example, NVM 606 may be accessed over a network via network interface 610.

In particular, system memory 605 and NVM 606 may each include: a temporary copy and a permanent copy of instructions 620. The instructions 620 may include: instructions that, when executed by at least one of the processors 602, cause the electronic device 600 to implement the method shown in fig. 2. In some embodiments, the instructions 620, hardware, firmware, and/or software components thereof may additionally/alternatively be located in the system motherboard 608, the network interface 610, and/or the processor 602.

The network interface 610 may include a transceiver to provide a radio interface for the electronic device 600 to communicate with any other suitable devices (e.g., front end modules, antennas, etc.) over one or more networks. In some embodiments, the network interface 610 may be integrated with other components of the electronic device 600. For example, network interface 610 may be integrated with at least one of processor 602, system memory 605, NVM 606, and a firmware device (not shown) having instructions that, when executed by at least one of processors 602, electronic device 600 implements one or more of the various embodiments shown in fig. 2.

The network interface 610 may further include any suitable hardware and/or firmware to provide a multiple-input multiple-output radio interface. For example, network interface 610 may be a network adapter, a wireless network adapter, a telephone modem, and/or a wireless modem.

In one embodiment, at least one of the processors 602 may be packaged together with one or more controllers for a system motherboard 608 to form a System In Package (SiP). In one embodiment, at least one of the processors 602 may be integrated on the same die with one or more controllers for a system motherboard 608 to form a system on a chip (SoC).

The electronic device 600 may further include: input/output (I/O) devices 612 are connected to the system motherboard 608. The I/O device 612 may include a user interface to enable a user to interact with the electronic device 600; the design of the peripheral component interface enables peripheral components to also interact with the electronic device 600. In some embodiments, the electronic device 600 further comprises a sensor for determining at least one of environmental conditions and location information related to the electronic device 600.

In some embodiments, I/O devices 612 may include, but are not limited to, a display (e.g., a liquid crystal display, a touch screen display, etc.), a speaker, a microphone, one or more cameras (e.g., still image cameras and/or video cameras), a flashlight (e.g., a light emitting diode flash), and a keyboard.

In some embodiments, the peripheral component interfaces may include, but are not limited to, a non-volatile memory port, an audio jack, and a power interface.

It is to be understood that the illustrated structure of the embodiment of the invention is not intended to limit the electronic device 600. In other embodiments of the present application, the electronic device 600 may include more or fewer components than illustrated, or combine certain components, or split certain components, or a different arrangement of components. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

Program code may be applied to input instructions to perform the functions described in this disclosure and to generate output information. The output information may be applied to one or more output devices in a known manner. For purposes of this application, a system for processing instructions that includes processor 602 includes any system having a processor such as a Digital Signal Processor (DSP), a microcontroller, an Application Specific Integrated Circuit (ASIC), or a microprocessor.

The program code may be implemented in a high level procedural or object oriented programming language to communicate with a processing system. The program code can also be implemented in assembly or machine language, if desired. Indeed, the mechanisms described in this disclosure are not limited in scope to any particular programming language. In any case, the language may be a compiled or interpreted language.

According to an embodiment of the present invention, a computer-readable storage medium is further provided, in which at least one computer instruction is stored, and the at least one instruction is loaded and executed by a processor to implement the foregoing method.

According to an embodiment of the present invention, a computer program product is also proposed, which comprises computer instructions that, when executed, implement the aforementioned method.

Illustrative embodiments of the invention include, but are not limited to, a data processing method, system, device, medium, and program product.

Various aspects of the illustrative embodiments will be described using terms commonly employed by those skilled in the art to convey the substance of their work to others skilled in the art. It will be apparent, however, to one skilled in the art that some alternative embodiments may be practiced using the features described in part. For purposes of explanation, specific numbers and configurations are set forth in order to provide a more thorough understanding of the illustrative embodiments. It will be apparent, however, to one skilled in the art that alternative embodiments may be practiced without the specific details. In some other instances, well-known features are omitted or simplified in order not to obscure the illustrative embodiments of the present invention.

Moreover, various operations will be described as multiple operations separate from one another in a manner that is most helpful in understanding the illustrative embodiments; however, the order of description should not be construed as to imply that these operations are necessarily order dependent, and that many of the operations can be performed in parallel, concurrently, or simultaneously. In addition, the order of the operations may be re-arranged. The process may be terminated when the described operations are completed, but may have additional steps not included in the figure. The processes may correspond to methods, functions, procedures, subroutines, and the like.

References in the specification to "one example," "an example," "one embodiment," "an implementation," etc., indicate that the embodiment described may include a particular feature, structure, or characteristic, but every embodiment may or may not necessarily include the particular feature, structure, or characteristic. Moreover, such phrases are not necessarily referring to the same embodiment. Furthermore, when a particular feature is described in connection with a particular embodiment, the knowledge of one skilled in the art can affect such feature in combination with other embodiments, whether or not such embodiments are explicitly described.

The terms "comprising," "having," and "including" are synonymous, unless the context dictates otherwise. The phrase "A and/or B" means "(A), (B) or (A and B)".

As used herein, the term "module" may refer to, be a part of, or include: memory (shared, dedicated, or group) for executing one or more software or firmware programs, an Application Specific Integrated Circuit (ASIC), an electronic circuit and/or processor (shared, dedicated, or group), a combinational logic circuit, and/or other suitable components that provide the described functionality.

In the drawings, some features of structures or methods may be shown in a particular arrangement and/or order. However, it should be understood that such specific arrangement and/or ordering is not required. Rather, in some embodiments, these features may be described in a manner and/or order different from that shown in the illustrative figures. Additionally, the inclusion of structural or methodical features in a particular figure does not imply that all embodiments need to include such features, and in some embodiments, may not include such features or may be combined with other features.

In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.

Similarly, it should be appreciated that in the foregoing description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the invention and aiding in the understanding of one or more of the various inventive aspects. However, the disclosed method should not be interpreted as reflecting an intention that: that the invention as claimed requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of this invention.

Those skilled in the art will appreciate that the modules in the device in an embodiment may be adaptively changed and disposed in one or more devices different from the embodiment. The modules or units or components of the embodiments may be combined into one module or unit or component, and furthermore they may be divided into a plurality of sub-modules or sub-units or sub-components. All of the features disclosed in this specification (including any accompanying claims, abstract and drawings), and all of the processes or elements of any method or apparatus so disclosed, may be combined in any combination, except combinations where at least some of such features and/or processes or elements are mutually exclusive. Each feature disclosed in this specification (including any accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.

Moreover, those skilled in the art will appreciate that although some embodiments described herein include some features included in other embodiments, not others, combinations of features of different embodiments are meant to be within the scope of the invention and form different embodiments. For example, in the claims, any of the claimed embodiments may be used in any combination.

Claims (9)

1. A data processing method for an electronic device, comprising:

a data acquisition step of acquiring data from a data source;

a data distribution step, distributing the data based on the unique identifier of the data;

and a data transmission step of transmitting the data and updating site information based on a processing result of the data, the site information being a boundary point between processed and unprocessed data.

2. The data processing method according to claim 1, wherein the data splitting step further includes performing hash calculation on the unique identifier, and adding data with the same unique identifier to the same memory queue.

3. The data processing method according to claim 1, wherein in the data sending step, updating the location information based on the processing result of the data includes updating the location information in the memory queue in a case where the processing of the data is completed, and resending the data in a case where the processing of the data is not completed.

4. A data processing system, comprising:

the data acquisition module acquires data from a data source;

the data distribution module distributes the data based on the unique identifier of the data;

and the data sending module is used for sending the data and updating site information based on the processing result of the data, wherein the site information is a boundary point between the processed data and the unprocessed data.

5. The data processing system of claim 4, wherein the data offload module further comprises a computation module that performs a hash computation on the unique identifier and adds data with the same unique identifier to the same memory queue.

6. The data processing system of claim 4, wherein there are a plurality of said data sending modules, and a plurality of said data sending modules are in one-to-one correspondence with said memory queues.

7. An electronic device, characterized in that the device comprises a memory storing computer executable instructions and a processor configured to execute the instructions to implement the data processing method according to any one of claims 1-3.

8. A computer-readable storage medium having stored therein at least one computer instruction, which is loaded and executed by a processor, to implement the data processing method of any one of claims 1-3.

9. A computer program product, characterized in that the computer program product comprises computer instructions which, when executed, implement the data processing method according to any one of claims 1-3.

Images