CN115629918A - Data processing method and device, electronic equipment and storage medium - Google Patents
Data processing method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN115629918A CN115629918A CN202211305396.1A CN202211305396A CN115629918A CN 115629918 A CN115629918 A CN 115629918A CN 202211305396 A CN202211305396 A CN 202211305396A CN 115629918 A CN115629918 A CN 115629918A
- Authority
- CN
- China
- Prior art keywords
- data
- standby
- water level
- processed
- level time
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/16—Error detection or correction of the data by redundancy in hardware
- G06F11/1658—Data re-synchronization of a redundant component, or initial sync of replacement, additional or spare unit
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1448—Management of the data involved in backup or backup restore
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/16—Error detection or correction of the data by redundancy in hardware
- G06F11/20—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
- G06F11/202—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where processing functionality is redundant
Abstract
The disclosure provides a data processing method and device, electronic equipment and a storage medium, and relates to the technical field of big data. The implementation scheme is as follows: responding to the stop of a main device from processing a data stream, and acquiring first data which is processed by the main device in the data stream; based on the first data, carrying out deduplication on second data to be processed by standby equipment in the data stream; processing the second data after the duplication removal; and in response to the data processing progress of the standby device leading the primary device, treating the standby device as the primary device for processing the data stream.
Description
Technical Field
The present disclosure relates to the field of computer technologies, and in particular, to the field of big data technologies, and in particular, to a data processing method and apparatus, an electronic device, a computer-readable storage medium, and a computer program product.
Background
A data stream is also called an event stream, and is a borderless data set. Over time, new data is continuously added to the data stream, causing the data in the data stream to grow continuously and indefinitely. Common data streams include behavioral data (e.g., browsing, clicking, paying, etc.) of a user in an application (App), streaming data of a network switch, data collected by a sensor, and the like.
The approaches described in this section are not necessarily approaches that have been previously conceived or pursued. Unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section. Similarly, unless otherwise indicated, the problems mentioned in this section should not be considered as having been acknowledged in any prior art.
Disclosure of Invention
The present disclosure provides a data processing method and apparatus, an electronic device, a computer-readable storage medium, and a computer program product.
According to an aspect of the present disclosure, there is provided a data processing method applied to a standby device, where the standby device and a main device are configured to process the same data stream, and a data processing progress of the standby device lags behind that of the main device, the method including: responding to the stop of the main device from processing the data stream, and acquiring first data which is processed by the main device in the data stream; based on the first data, carrying out deduplication on second data to be processed by the standby equipment in the data stream; processing the second data after the duplication removal; and in response to the data processing progress of the standby device leading the primary device, treating the standby device as the primary device for processing the data stream.
According to an aspect of the present disclosure, there is provided a data processing apparatus applied to a standby device, the standby device and a main device being configured to process a same data stream, a data processing progress of the standby device lagging behind that of the main device, the apparatus including: an obtaining module configured to obtain first data in the data stream, which has been processed by the master device, in response to the master device stopping processing the data stream; a deduplication module configured to deduplicate second data to be processed by the standby device in the data stream based on the first data; the processing module is configured to process the second data after the duplication removal; and the main/standby switching module is configured to take the standby equipment as the main equipment for processing the data stream in response to that the data processing progress of the standby equipment is ahead of that of the main equipment.
According to an aspect of the present disclosure, there is provided an electronic device including: at least one processor; and a memory communicatively coupled to the at least one processor; the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the data processing method described above.
According to an aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions for causing a computer to execute the above-described data processing method.
According to an aspect of the present disclosure, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the above-described data processing method.
According to one or more embodiments of the present disclosure, when a data processing system is subjected to active/standby switching, the accurate consistency of the data state of an end-to-end system (from the data processing system to a downstream system) can be ensured.
It should be understood that the statements in this section do not necessarily identify key or critical features of the embodiments of the present disclosure, nor do they limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate exemplary embodiments of the embodiments and, together with the description, serve to explain the exemplary implementations of the embodiments. The illustrated embodiments are for purposes of example only and do not limit the scope of the claims. Throughout the drawings, identical reference numbers designate similar, but not necessarily identical, elements.
Fig. 1 illustrates a schematic diagram of an exemplary system in which various methods described herein may be implemented, in accordance with some embodiments of the present disclosure;
FIG. 2 illustrates a flow diagram of a data processing method according to some embodiments of the present disclosure;
FIG. 3 illustrates a schematic diagram of high and low water level times, according to some embodiments of the present disclosure;
FIG. 4 illustrates a schematic diagram of first data, according to some embodiments of the present disclosure;
FIG. 5 shows a block diagram of a data processing apparatus, according to some embodiments of the present disclosure; and
FIG. 6 illustrates a block diagram of an exemplary electronic device that can be used to implement some embodiments of the present disclosure.
Detailed Description
Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, in which various details of the embodiments of the disclosure are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, it will be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
In the present disclosure, unless otherwise specified, the use of the terms "first", "second", and the like to describe various elements is not intended to limit the positional relationship, the temporal relationship, or the importance relationship of the elements, and such terms are used only to distinguish one element from another. In some examples, a first element and a second element may refer to the same instance of the element, while in some cases they may refer to different instances based on the context of the description.
The terminology used in the description of the various described examples in this disclosure is for the purpose of describing the particular examples only and is not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the elements may be one or more. Furthermore, the term "and/or" as used in this disclosure is intended to encompass any and all possible combinations of the listed items.
In the technical scheme of the disclosure, the processes of collecting, storing, using, processing, transmitting, providing, disclosing and the like of the personal information of the related user all accord with the regulations of related laws and regulations, and do not violate the common customs of public order.
The streaming processing system is used for processing data streams. The streaming processing system receives a data stream from an upstream system (source, e.g., a service system), processes the data stream, and outputs a processing result to a downstream system (sink, e.g., a storage system, an application system, etc.). In a streaming computing scenario, end-to-end state consistency includes state consistency between the streaming processing system and the downstream system, i.e., consistency of data states between the downstream system and the streaming processing system after the streaming processing system experiences an exception and returns to normal. The degree of state consistency between a streaming processing system and a downstream system can be represented by the following three semantics:
1. at least once (at least once): the data is processed at least once. When an exception occurs and the streaming system returns to normal, some data may be processed repeatedly.
2. At most once (at most once): the data is processed at most once. And when the streaming processing system generates an exception, not performing any processing. After the system returns to normal, some data may be lost.
3. Exact once (exact once): the data is processed and only once. When the stream processing system is abnormal and returns to normal, the data is not repeatedly processed and lost.
It is understood that "exactly once" is the most stringent state consistency requirement among the three semantics described above.
For important streaming services (e.g., billing services), in order to ensure high availability of the services, a streaming system typically employs multiple computing modules (e.g., a primary device and a standby device) for redundancy, and ensures that only one computing module (i.e., the primary device) distributes the computing results to downstream systems. When the main device is abnormal, the main device and the standby device need to be switched, and the standby device replaces the current main device to provide service. For important streaming services such as billing services, end-to-end semantic changes (i.e., duplication or loss of data) are unacceptable. Therefore, in the process of active/standby switching, it is necessary to ensure the precise consistency between the streaming processing system and the downstream system, i.e. the consistency of "precise once".
In the related art, precise consistency between a streaming processing system and a downstream system is generally achieved by idempotent writing by the downstream system. Idempotent writing refers to data that can be processed multiple times by a streaming system, but the state in downstream systems is updated only once. The scheme depends on idempotent writing capability of a downstream system, and is poor in universality. If the downstream system does not have idempotent write capability, the data may be processed repeatedly and end-to-end exact consistency cannot be guaranteed.
In view of the foregoing problems, embodiments of the present disclosure provide a data processing method, which optimizes a primary/standby switching process of a data processing system to ensure accurate end-to-end consistency from the data processing system to a downstream system.
Embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.
Fig. 1 illustrates a schematic diagram of an exemplary system 100 in which various methods and apparatus described herein may be implemented, according to some embodiments of the present disclosure. Referring to FIG. 1, the system 100 includes one or more client devices 111, 112, 113, 114, 115, and 116, an upstream business system 120, a data processing system 130, and a downstream storage system 140. Client devices 111-116, upstream business system 120, data processing system 130, and downstream storage system 140 may communicate via a network.
Client devices 111-116 may provide an interface that enables a user of the client device to interact with the client device. The client device may also output information to the user via the interface. Although fig. 1 depicts only six client devices, those skilled in the art will appreciate that any number of client devices may be supported by the present disclosure.
Client devices 111-116 may include various types of computer devices, such as portable handheld devices, general purpose computers (such as personal computers and laptop computers), workstation computers, wearable devices, smart screen devices, self-service terminal devices, service robots, in-vehicle devices, gaming systems, thin clients, various messaging devices, sensors or other sensing devices, and so forth. These computer devices may run various types and versions of software applications and operating systems, such as MICROSOFT Windows, APPLE iOS, UNIX-like operating systems, linux, or Linux-like operating systems (e.g., GOOGLE Chrome OS); or include various Mobile operating systems, such as MICROSOFT Windows Mobile OS, iOS, windows Phone, android. Portable handheld devices may include cellular telephones, smart phones, tablet computers, personal Digital Assistants (PDAs), and the like. Wearable devices may include head-mounted displays (such as smart glasses) and other devices. The gaming system may include a variety of handheld gaming devices, internet-enabled gaming devices, and the like. The client device is capable of executing a variety of different applications, such as various Internet-related applications, communication applications (e.g., email applications), short Message Service (SMS) applications, and may use a variety of communication protocols.
Client devices 111-116 may be configured to execute one or more application programs. Accordingly, upstream business system 120 may be a server to which these applications correspond.
In some embodiments, the client devices 111-116 send data (or referred to as "events") to the upstream business system 120 based on user interaction, such as user browsing, clicking, downloading, payment, etc. behavior data. Over time, the upstream traffic system 120 receives data from time to time, the received data forming a pending data stream. Each piece of data in the data stream includes an event time (event time) and a globally unique identification. Wherein the event time represents a production time of the corresponding data.
The data stream generated by the upstream traffic system 120 is sent to the data processing system 130 for processing. The data processing system 130 is a streaming processing system.
In the embodiment of the present disclosure, the data processing system 130 includes the primary device 132 and the standby device 134, and both the primary device 132 and the standby device 134 are configured to process the data stream, so that the availability and disaster tolerance capability of the data processing system 130 can be improved.
It is noted that data processing system 130 may include one or more primary devices 132 and may also include one or more backup devices 134. Where the data processing system 130 includes a plurality of master devices 132, the plurality of master devices 132 form a master cluster. Where data processing system 130 includes multiple standby devices 134, the multiple standby devices 134 form a standby cluster.
In an embodiment of the present disclosure, data streams generated by upstream traffic system 120 are sent to primary device 132 and standby device 134 simultaneously. The data received by primary device 132 and backup device 134 are the same (both complete data streams). The primary device 132 and the standby device 134 process the data streams separately, and the standby device 134 acts as a backup to the primary device 132 with a data processing progress that lags behind that of the primary device 132.
It should be noted that there may be some degree of misordering of the data streams received by both the primary device 132 and the standby device 134. For example, due to a network or the like, data a with a later event time may arrive at primary device 132 or standby device 134 earlier than data B with a earlier event time.
In an embodiment of the present disclosure, master 132 processes the data stream and writes the data processing results to downstream storage system 140. The downstream storage system 140 may be a temporary storage system (e.g., a message queue, etc.) or a persistent storage system (e.g., a distributed file system, an object storage system, etc.).
According to some embodiments, the downstream storage system 140 may include multiple packets (i.e., storage locations). For example, where the downstream storage system 140 is a message queue, the packet may be a partition within a certain message topic (topic). Where downstream storage system 140 is a distributed file system, the grouping may be a certain file path or a certain file slice in a file path.
According to some embodiments, each packet of the downstream storage system 140 may correspond to a different hash value. Master device 132 may calculate a hash value for the identification of the data and store the data in a corresponding packet based on the hash value. Determining the packets based on the hash values enables relatively even distribution of data among the packets, thereby improving the utilization efficiency of the downstream storage system 140.
According to some embodiments, system 100 also includes a backup storage system 150. The standby device 134 processes the data stream and writes the data processing results to the standby storage system 150. The backup storage system 150 may be a temporary storage system (e.g., a message queue, etc.) or a persistent storage system (e.g., a distributed file system, an object storage system, etc.). In the event that the system 100 does not include a backup storage system 150, the backup device 134 may also discard its data processing results.
According to some embodiments, the backup storage system 150 may also include a plurality of packets, similar to the downstream storage system 140. The backup device 134 may determine the group to which the data corresponds based on the identification of the data and write the processing results of the data to the corresponding group of the backup storage system 150. The process of the backup device 134 writing the data processing result into the packet of the backup storage system 150 is similar to the process of the master device 132 writing the data processing result into the packet of the downstream storage system 140, and is not described again here.
In the embodiment of the present disclosure, data processing system 130 may perform a primary-standby switch, that is, using original standby device 134 as a primary device for processing data streams and original main device 132 as a standby device for processing data streams. In the event that data processing system 130 needs to perform a primary-secondary switch (e.g., in the event that primary device 132 fails or processing performance fails to meet expectations), primary device 132 stops processing the data stream, and backup device 134 may execute data processing method 200 of the embodiments of the present disclosure to implement the primary-secondary switch and ensure end-to-end precise consistency (i.e., exact once consistency) of data processing system 130 to downstream storage system 140.
FIG. 2 shows a flow diagram of a data processing method 200 according to an embodiment of the present disclosure. As described above, the method 200 is applied to a standby device, i.e., the execution subject of the method 200 is the standby device (e.g., the standby device 134 in fig. 1). As described above, the standby device and the main device are configured to process the same data stream, and the data processing progress of the standby device lags behind that of the main device.
As shown in FIG. 2, the method 200 includes steps S210-S240.
In step S210, in response to the master device stopping processing the data stream, first data in the data stream that has been processed by the master device is acquired.
In step S220, second data to be processed by the standby device in the data stream is deduplicated based on the first data.
In step S230, the second data after deduplication is processed.
In step S240, in response to the data processing progress of the standby device leading the primary device, the standby device is taken as the primary device for processing the data stream.
According to the embodiment of the disclosure, when the data processing system is subjected to active-standby switching, the end-to-end accurate consistency can be ensured. After the main device stops processing the data stream, the standby device obtains the data (i.e., the first data) processed by the main device, and performs deduplication on the data (i.e., the second data) to be processed by the standby device, so that it can be ensured that the data processed by the standby device after the main device stops processing is not repeated with the data processed by the main device, and is not lost.
The various steps of method 200 are described in detail below.
According to some embodiments, the operational status of the master device may be monitored during operation of the data processing system. When monitoring that the main equipment fails or the data processing performance cannot meet the expectation, triggering the main-standby switching, and stopping processing the data stream by the main equipment to wait for being taken over by the standby equipment.
With respect to step S210, the standby device acquires first data that has been processed by the main device in the data stream in response to the main device stopping processing the data stream. The first data is a data set obtained by tracing back a data of a certain time from the latest piece of data processed by the main device. It can be understood that if the backtracking time window is too short, it cannot be guaranteed that the data to be processed by the standby device is completely deduplicated; if the backtracking time window is too long, the data volume of the first data is too large, which occupies more memory of the standby device, and backtracking the data takes more time, thereby prolonging the time of stopping the data processing service and reducing the availability of the data processing system.
According to some embodiments, the first data may be determined according to the following steps S212-S216:
in step S212, a high water level (high water) time of the master is acquired, which indicates the maximum value of the event time for which the master has processed data. In other words, the high water level time is the event time of the latest piece of data that the master has processed.
In step S214, a low watermark (low watermark) time of the standby device is obtained, where the low watermark time indicates that data with an event time less than or equal to the low watermark time in the data stream has been processed by the standby device.
In step S216, data whose event time is greater than or equal to the low water level time of the backup apparatus and less than or equal to the high water level time of the master apparatus is taken as the first data.
The first data determined in steps S212 to S216 is the minimum set of data that needs to be deduplicated, which can ensure that the data to be processed by the standby device is completely deduplicated, and can also ensure the data processing efficiency, shorten the time for stopping the data processing service as much as possible, and implement the host/standby switching that is insensitive to the downstream system.
It should be noted that the above embodiment only uses the high water level time of the main device and the low water level time of the standby device when determining the first data. According to other embodiments, the high water level time of the main device and the low water level time of the standby device can be acquired simultaneously. Although the low water level time of the master device and the high water level time of the standby device are not used to determine the first data, both may be used as a reference for the system operation state.
According to some embodiments, for step S212, in case of multiple masters (multiple masters forming a master cluster), the high water level time of a master may be determined according to the following steps: acquiring the maximum event time of data processed by each of a plurality of main devices to obtain a plurality of first maximum event times; and taking the maximum value of the first maximum event times as the high water level time of the main device. Further, the minimum value of the plurality of first maximum event times may be set as a low water level time of the master device.
Similarly, according to some embodiments, for step S214, in the case that there are multiple standby devices (multiple standby devices form a standby cluster), the low water time of the standby device may be determined according to the following steps: acquiring the maximum event time of the data processed by each standby device in the plurality of standby devices to obtain a plurality of second maximum event times; and taking the minimum value in the second maximum event times as the low water level time of the standby equipment. Further, a maximum value of the plurality of second maximum event times may be set as a high water level time of the standby device.
According to the above embodiments, for a cluster (e.g., a master cluster, a standby cluster), a high water time of the cluster refers to a fastest data processing progress of each device within the cluster, and a low water time refers to a slowest data processing progress of each device within the cluster. According to some embodiments, since the standby cluster is a backup of the main cluster, the fastest data processing progress of the standby cluster cannot be greater than the slowest data processing progress of the main cluster, that is, the high water time of the standby cluster is less than or equal to the low water time of the main cluster.
Fig. 3 illustrates a schematic of high and low water level times according to some embodiments of the present disclosure. In the embodiment shown in fig. 3, the master cluster includes four masters, master 1-master 4. Rectangular boxes 311 to 314 in fig. 3 represent the data processing progress of the master 1 to the master 4, respectively. As can be seen by rectangular boxes 311-314, the maximum event times (i.e., first maximum event times) of the data processed by master 1-master 4 are t, respectively 7 、t 4 、t 5 、t 6 。t 4 -t 7 Maximum value of (1) 7 High water level time of the master (master cluster), minimum t 4 Is the low water time of the master (master cluster).
The standby cluster comprises three devices, standby device 1-standby device 3. Rectangular boxes 321-323 in fig. 3 represent the data processing progress of the standby device 1-the standby device 3, respectively. As can be seen by the rectangular boxes 321-323, the maximum event times (i.e., the second maximum event times) of the data processed by the standby devices 1-4 are t, respectively 1 、t 3 、t 2 。t 1 -t 3 Maximum value of (1) 3 High water level time for standby equipment (standby cluster), minimum value t 1 Low water time for standby devices (standby clusters).
After determining the high water level time of the main cluster and the low water level time of the standby device based on steps S212 and S214, data having an event time greater than or equal to the low water level time of the standby device and less than or equal to the high water level time of the main device may be taken as the first data in step S216. The first data is the minimum set of data needing to be deduplicated, so that the data to be processed of the standby equipment can be completely deduplicated, the data processing efficiency can be ensured, the time for stopping the data processing service is shortened as much as possible, and the primary-standby switching which is insensitive to a downstream system is realized.
The rationality of having data with an event time greater than or equal to the low water level time of the backup apparatus and less than or equal to the high water level time of the main apparatus as the first data is analyzed below in conjunction with fig. 4.
In fig. 4, rectangular boxes 411, 412 represent the fastest data processing progress and the slowest data processing progress of the master cluster, respectively, and as shown by the rectangular boxes 411, 412, the high water level time and the low water level time of the master device are t 4 、t 3 . The rectangular blocks 421 and 422 respectively represent the fastest data processing progress and the slowest data processing progress of the standby cluster, and as shown by the rectangular blocks 421 and 422, the high water level time and the low water level time of the standby device are t 2 、t 1 . In some embodiments, the high water time of the standby device may be equal to the low water time of the primary device, i.e., t 2 ＝t 3 。
For data with event time T before T1 (i.e., T < T1): for the main cluster and the standby cluster, the data are processed, so that the backtracking of the data is not needed during the main-standby switching.
For data with event times T between T1-T3 (i.e., T1 ≦ T ≦ T3): for the primary cluster, both of these data have been processed. For the standby cluster, one part of the data is processed, and the other part is not processed. Data that has been processed by the master cluster but not by the standby cluster needs to be deduplicated to prevent the standby cluster from repeatedly sending these data to downstream systems after taking over the master cluster. Therefore, the standby cluster needs to trace back the data processed by the main cluster in the time period t1-t 3.
For data with event times T between T3-T4 (i.e., T3 ≦ T ≦ T4): for the primary cluster, some of the data has been processed and another has not. For the standby cluster, none of the data is processed. The backup cluster needs to deduplicate the data processed by the main cluster, so the backup cluster needs to trace back the data processed by the main cluster in the time period from t3 to t 4.
In summary, the standby cluster needs to trace back the data processed by the main cluster in the time period from t1 to t4, that is, the data with the event time greater than or equal to the low water level time of the standby device and less than or equal to the high water level time of the main device is used as the first data.
According to some embodiments, first data may be retrieved from respective downstream storage locations, the retrieving the first data comprising: determining a storage location of the first data based on the identification of the first data; and retrieving the first data from the storage location.
According to the above-described embodiments, the storage location of data in the downstream storage system is determined based on the identification of the data, and thus each piece of data has a fixed storage location in the downstream storage system. The first data is acquired from the fixed storage position, so that the full backtracking of all storage positions (namely all groups) of the downstream storage system can be avoided, and the time consumption and memory occupation of backtracking of the first data are reduced.
According to some embodiments, the storage location of the first data may be determined based on the identified hash value of the first data. The storage positions are determined based on the hash values, so that the data can be distributed relatively uniformly in each storage position, the utilization efficiency of a downstream storage system is improved, and the data backtracking pressure of certain storage positions is avoided.
According to some embodiments, the method 200 further comprises: storing the first data to a memory; and in response to the data processing progress of the standby device ahead of the primary device, clearing the first data from the memory.
According to the embodiment, after the standby device acquires the first data, the first data is stored in the local memory. And when the data processing progress of the standby equipment is ahead of that of the main equipment, the standby equipment takes over the main equipment to complete the main-standby switching. At this time, the first data is cleared from the memory of the standby device, so that the memory space of the standby device is released, and the memory is saved for other use.
In step S240, in response to that the data processing progress of the standby device is ahead of that of the main device, the standby device is used as a main device for processing a data stream, and the original main device is used as a standby device for processing a data stream, thereby completing the main-standby switching. After the main-standby switching is completed, the new standby device and the new main device can both process the data stream, and the data processing progress of the new standby device lags behind that of the new main device.
According to some embodiments, in a case where there are a plurality of the master device and the standby device, respectively, it may be determined that the data processing progress of the standby device is ahead of that of the master device in response to the low water time of the standby device being greater than the high water time of the master device. That is, when the slowest data processing progress of the standby cluster exceeds the fastest data processing progress of the main cluster, the main/standby switching is performed.
According to the embodiment of the disclosure, a data processing device is also provided.
Fig. 5 shows a block diagram of a data processing apparatus 500 according to an embodiment of the present disclosure. The apparatus 500 is applied to a standby device. The standby device and the primary device are configured to process the same data stream, and the data processing progress of the standby device lags behind that of the primary device. As shown in fig. 5, the apparatus 500 includes an obtaining module 510, a deduplication module 520, a processing module 530, and a main/standby switching module 540.
The obtaining module 510 is configured to obtain first data in the data stream, which has been processed by the master device, in response to the master device stopping processing the data stream;
the deduplication module 520 is configured to perform deduplication on second data to be processed by the standby device in the data stream based on the first data;
the processing module 530 is configured to process the de-duplicated second data; and
the primary-standby switching module 540 is configured to take the standby device as the primary device for processing the data stream in response to the data processing progress of the standby device leading the primary device.
According to the embodiment of the disclosure, the accurate consistency of end to end can be ensured when the data processing system is subjected to active-standby switching. After the main device stops processing the data stream, the standby device obtains the data (i.e., the first data) processed by the main device, and performs deduplication on the data to be processed (i.e., the second data), so that it can be ensured that the data processed by the standby device after the main device stops serving is not repeated with the data processed by the main device, and is not lost.
According to some embodiments, the data stream comprises a plurality of pieces of data, each of the plurality of pieces of data comprising an event time, the obtaining module 510 comprises: a first acquisition unit configured to acquire a high water level time of the master, wherein the high water level time indicates a maximum value of an event time for which the master has processed data; a second obtaining unit configured to obtain a low water level time of the standby device, wherein the low water level time indicates that data of which event time is less than or equal to the low water level time in the data stream have been processed by the standby device; and a first determination unit configured to take data having an event time greater than or equal to the low water level time and less than or equal to the high water level time as the first data.
According to some embodiments, the master device is plural, and the first obtaining unit is further configured to: acquiring the maximum event time of data processed by each of a plurality of main devices to obtain a plurality of first maximum event times; and taking a maximum value of the first maximum event times as the high water level time.
According to some embodiments, the standby device is plural, and the second obtaining unit is further configured to: acquiring the maximum event time of the data processed by each standby device in the plurality of standby devices to obtain a plurality of second maximum event times; and taking the minimum value of the second maximum event times as the low water level time.
According to some embodiments, the obtaining module 510 comprises: a second determination unit configured to determine a storage location of the first data based on the identification of the first data; and a third acquisition unit configured to acquire the first data from the storage location.
According to some embodiments, the second determination unit is further configured to: determining a storage location of the first data based on the identified hash value of the first data.
According to some embodiments, the apparatus 500 further comprises: a storage module configured to store the first data to a memory; and the clearing module is configured to clear the first data from the memory in response to the data processing progress of the standby device leading the primary device.
According to some embodiments, the apparatus 500 further comprises: a determination module configured to determine that a data processing progress of the standby device precedes the main device in response to the low water level time being greater than the high water level time.
It should be understood that the various modules or units of the apparatus 500 shown in fig. 5 may correspond to the various steps in the method 200 described with reference to fig. 2. Thus, the operations, features and advantages described above with respect to the method 200 are equally applicable to the apparatus 500 and the modules and units comprised thereby. Certain operations, features and advantages may not be described in detail herein for the sake of brevity.
Although specific functionality is discussed above with reference to particular modules, it should be noted that the functionality of the various modules discussed herein may be divided into multiple modules and/or at least some of the functionality of multiple modules may be combined into a single module.
It should also be appreciated that various techniques may be described herein in the general context of software hardware elements or program modules. The various modules described above with respect to fig. 5 may be implemented in hardware or in hardware in combination with software and/or firmware. For example, the modules may be implemented as computer program code/instructions configured to be executed in one or more processors and stored in a computer-readable storage medium. Alternatively, the modules may be implemented as hardware logic/circuitry. For example, in some embodiments, one or more of the modules 510-540 may be implemented together in a System on a Chip (SoC). The SoC may include an integrated circuit chip (which includes one or more components of a Processor (e.g., a Central Processing Unit (CPU), microcontroller, microprocessor, digital Signal Processor (DSP), etc.), memory, one or more communication interfaces, and/or other circuitry), and may optionally execute received program code and/or include embedded firmware to perform functions.
There is also provided, in accordance with an embodiment of the present disclosure, a data processing system including a master device and a slave device for processing a same data stream, a data processing progress of the standby device lagging behind that of the master device, wherein the master device is configured to: responding to the data processing system needing to carry out active-standby switching, and stopping processing the data stream; the backup device is configured to: responding to the main device stopping processing the data stream, and acquiring first data processed by the main device in the data stream; based on the first data, carrying out deduplication on second data to be processed by the standby equipment in the data stream; processing the second data after the duplication removal; and in response to the data processing progress of the standby device leading the primary device, treating the standby device as the primary device for processing the data stream.
According to an embodiment of the present disclosure, there is also provided an electronic apparatus including: at least one processor; and a memory communicatively coupled to the at least one processor, the memory storing instructions executable by the at least one processor, the instructions being executable by the at least one processor to enable the at least one processor to perform the data processing method of the disclosed embodiments.
There is also provided, in accordance with an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform a data processing method of an embodiment of the present disclosure.
According to an embodiment of the present disclosure, there is also provided a computer program product comprising a computer program which, when executed by a processor, implements the data processing method of an embodiment of the present disclosure.
Referring to fig. 6, a block diagram of a structure of an electronic device 600, which may be a server or a client of the present disclosure, which is an example of a hardware device that may be applied to aspects of the present disclosure, will now be described. Electronic device is intended to represent various forms of digital electronic computer devices, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the disclosure described and/or claimed herein.
As shown in fig. 6, the electronic device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a Read Only Memory (ROM) 602 or a computer program loaded from a storage unit 608 into a Random Access Memory (RAM) 603. In the RAM603, various programs and data necessary for the operation of the electronic apparatus 600 can also be stored. The calculation unit 601, the ROM 602, and the RAM603 are connected to each other via a bus 604. An input/output (I/O) interface 605 is also connected to bus 604.
Various components in the electronic device 600 are connected to the I/O interface 605, including: an input unit 606, an output unit 607, a storage unit 608, and a communication unit 609. The input unit 606 may be any type of device capable of inputting information to the electronic device 600, and the input unit 606 may receive input numeric or character information and generate key signal inputs related to user settings and/or function controls of the electronic device, and may include, but is not limited to, a mouse, a keyboard, a touch screen, a track pad, a track ball, a joystick, a microphone, and/or a remote control. Output unit 607 may be any type of device capable of presenting information and may include, but is not limited to, a display, speakers, a video/audio output terminal, a vibrator, and/or a printer. The storage unit 608 may include, but is not limited to, a magnetic disk, an optical disk. The communication unit 609 allows the electronic device 600 to exchange information/data with other devices via a computer network, such as the internet, and/or various telecommunications networks, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and/or a chipset, such as bluetooth TM Devices, 802.11 devices, wi-Fi devices, wiMAX devices, cellular communication devices, and/or the like.
Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuitry, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), system on a chip (SOCs), complex Programmable Logic Devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.
Program code for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the program codes, when executed by the processor or controller, cause the functions/operations specified in the flowchart and/or block diagram to be performed. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.
In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), the Internet, and blockchain networks.
The computer system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server may be a cloud server, a server of a distributed system, or a server with a combined blockchain.
It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present disclosure may be performed in parallel, sequentially or in different orders, and are not limited herein as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved.
While embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it is to be understood that the above-described methods, systems, and apparatus are merely illustrative embodiments or examples and that the scope of the disclosure is not limited by these embodiments or examples, but only by the claims as issued and their equivalents. Various elements in the embodiments or examples may be omitted or may be replaced with equivalents thereof. Further, the steps may be performed in an order different from that described in the present disclosure. Further, the various elements in the embodiments or examples may be combined in various ways. It is important that as technology evolves, many of the elements described herein may be replaced by equivalent elements that appear after the present disclosure.
Claims (20)
1. A data processing method applied to a standby device, wherein the standby device and a main device are configured to process the same data stream, and the data processing progress of the standby device lags behind that of the main device, the method comprising:
responding to the stop of the main device from processing the data stream, and acquiring first data which is processed by the main device in the data stream;
based on the first data, carrying out deduplication on second data to be processed by the standby equipment in the data stream;
processing the second data after the duplication removal; and
in response to the data processing progress of the standby device leading the primary device, treating the standby device as the primary device for processing the data stream.
2. The method of claim 1, wherein the data stream comprises a plurality of pieces of data, each piece of data of the plurality of pieces of data comprising an event time, the obtaining first data of the data stream that has been processed by the master device comprising:
acquiring a high water level time of the main device, wherein the high water level time indicates a maximum value of an event time of processed data of the main device;
acquiring a low water level time of the standby equipment, wherein the low water level time indicates that data with event time smaller than or equal to the low water level time in the data stream are processed by the standby equipment; and
and taking data with the event time greater than or equal to the low water level time and less than or equal to the high water level time as the first data.
3. The method of claim 2, wherein there are a plurality of master devices, and the obtaining the high water level time of the master devices comprises:
acquiring the maximum event time of data processed by each of a plurality of main devices to obtain a plurality of first maximum event times; and
taking a maximum of the first maximum event times as the high water level time.
4. The method of claim 2 or 3, wherein the standby device is multiple, and the obtaining the low water level time of the standby device comprises:
acquiring the maximum event time of the processed data of each standby device in the plurality of standby devices to obtain a plurality of second maximum event times; and
and taking the minimum value of the second maximum event times as the low water level time.
5. The method of any of claims 1-4, wherein the obtaining first data in the data stream that has been processed by the master device comprises:
determining a storage location of the first data based on the identity of the first data; and
the first data is retrieved from the storage location.
6. The method of claim 5, wherein the determining a storage location of the first data based on the identification of the first data comprises:
determining a storage location of the first data based on the identified hash value of the first data.
7. The method of any of claims 1-6, further comprising:
storing the first data to a memory; and
and in response to the data processing progress of the standby equipment ahead of the main equipment, clearing the first data from the memory.
8. The method of any of claims 2-4, further comprising:
and in response to the low water level time being larger than the high water level time, determining that the data processing progress of the standby equipment is ahead of that of the main equipment.
9. A data processing apparatus, applied to a standby device, the standby device and a main device being configured to process the same data stream, a data processing progress of the standby device lagging behind that of the main device, the apparatus comprising:
an obtaining module configured to obtain first data in the data stream, which has been processed by the master device, in response to the master device stopping processing the data stream;
a deduplication module configured to deduplicate second data to be processed by the standby device in the data stream based on the first data;
the processing module is configured to process the second data after the duplication removal; and
and the main/standby switching module is configured to take the standby device as a main device for processing the data stream in response to that the data processing progress of the standby device is ahead of that of the main device.
10. The apparatus of claim 9, wherein the data stream comprises a plurality of pieces of data, each of the plurality of pieces of data comprising an event time, the obtaining module comprising:
a first acquisition unit configured to acquire a high water level time of the master, wherein the high water level time indicates a maximum value of an event time for which the master has processed data;
a second obtaining unit configured to obtain a low water level time of the standby device, wherein the low water level time indicates that data of which event time is less than or equal to the low water level time in the data stream have been processed by the standby device; and
a first determination unit configured to take data having an event time greater than or equal to the low water level time and less than or equal to the high water level time as the first data.
11. The apparatus of claim 11, wherein there are a plurality of the master devices, the first obtaining unit is further configured to:
acquiring the maximum event time of data processed by each of a plurality of main devices to obtain a plurality of first maximum event times; and
taking a maximum value of the first maximum event times as the high water level time.
12. The apparatus according to claim 10 or 11, wherein there are a plurality of the standby devices, and the second obtaining unit is further configured to:
acquiring the maximum event time of the processed data of each standby device in the plurality of standby devices to obtain a plurality of second maximum event times; and
taking a minimum value of the second maximum event times as the low water level time.
13. The apparatus of any of claims 9-12, wherein the means for obtaining comprises:
a second determination unit configured to determine a storage location of the first data based on the identification of the first data; and
a third acquisition unit configured to acquire the first data from the storage location.
14. The apparatus of claim 13, wherein the second determining unit is further configured to:
determining a storage location of the first data based on the identified hash value of the first data.
15. The apparatus of any of claims 9-14, further comprising:
a storage module configured to store the first data to a memory; and
a purging module configured to purge the first data from the memory in response to a data processing progress of the standby device leading the primary device.
16. The apparatus of any of claims 10-12, further comprising:
a determination module configured to determine that a data processing progress of the standby device precedes the main device in response to the low water level time being greater than the high water level time.
17. A data processing system comprising a master device and a slave device for processing the same data stream, the data processing progress of the standby device lagging behind that of the master device, wherein,
the master device is configured to:
responding to the data processing system needing to carry out primary-standby switching, and stopping processing the data stream;
the backup device is configured to:
responding to the stop of the main device from processing the data stream, and acquiring first data which is processed by the main device in the data stream;
based on the first data, carrying out deduplication on second data to be processed by the standby equipment in the data stream;
processing the second data after the duplication removal; and
in response to the data processing progress of the standby device leading the primary device, treating the standby device as the primary device for processing the data stream.
18. An electronic device, comprising:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein, the first and the second end of the pipe are connected with each other,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-8.
19. A non-transitory computer readable storage medium having stored thereon computer instructions for causing a computer to perform the method of any one of claims 1-8.
20. A computer program product comprising a computer program, wherein the computer program realizes the method of any one of claims 1-8 when executed by a processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211305396.1A CN115629918B (en) | 2022-10-24 | 2022-10-24 | Data processing method, device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211305396.1A CN115629918B (en) | 2022-10-24 | 2022-10-24 | Data processing method, device, electronic equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115629918A true CN115629918A (en) | 2023-01-20 |
| CN115629918B CN115629918B (en) | 2023-06-27 |
Family
ID=84906343
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211305396.1A Active CN115629918B (en) | 2022-10-24 | 2022-10-24 | Data processing method, device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115629918B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20060129718A1 (en) * | 2002-07-19 | 2006-06-15 | Thomas Stromqvist | Method and apparatus for pipelined processing of data packets |
| US20080133561A1 (en) * | 2006-12-01 | 2008-06-05 | Nec Laboratories America, Inc. | Methods and systems for quick and efficient data management and/or processing |
| CN106031130A (en) * | 2014-02-19 | 2016-10-12 | 第三雷沃通讯有限责任公司 | Content delivery network architecture with edge proxy |
| CN113377809A (en) * | 2021-06-23 | 2021-09-10 | 北京百度网讯科技有限公司 | Data processing method and apparatus, computing device, and medium |
| CN113568938A (en) * | 2021-08-04 | 2021-10-29 | 北京百度网讯科技有限公司 | Data stream processing method and device, electronic equipment and storage medium |
-
2022
- 2022-10-24 CN CN202211305396.1A patent/CN115629918B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20060129718A1 (en) * | 2002-07-19 | 2006-06-15 | Thomas Stromqvist | Method and apparatus for pipelined processing of data packets |
| US20080133561A1 (en) * | 2006-12-01 | 2008-06-05 | Nec Laboratories America, Inc. | Methods and systems for quick and efficient data management and/or processing |
| CN106031130A (en) * | 2014-02-19 | 2016-10-12 | 第三雷沃通讯有限责任公司 | Content delivery network architecture with edge proxy |
| CN113377809A (en) * | 2021-06-23 | 2021-09-10 | 北京百度网讯科技有限公司 | Data processing method and apparatus, computing device, and medium |
| CN113568938A (en) * | 2021-08-04 | 2021-10-29 | 北京百度网讯科技有限公司 | Data stream processing method and device, electronic equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115629918B (en) | 2023-06-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10382380B1 (en) | Workload management service for first-in first-out queues for network-accessible queuing and messaging services | |
| JP6325001B2 (en) | Method and system using recursive event listeners in nodes of hierarchical data structures | |
| US20190278659A1 (en) | Enhanced application write performance | |
| US9483352B2 (en) | Process control systems and methods | |
| CN109308170B (en) | Data processing method and device | |
| CN111782365B (en) | Timed task processing method, device, equipment and storage medium | |
| US11023284B2 (en) | System and method for optimization and load balancing of computer clusters | |
| CN113193947B (en) | Method, apparatus, medium, and program product for implementing distributed global ordering | |
| CN113986788A (en) | Data processing method and device, chip, electronic device and medium | |
| CN115168256A (en) | Interrupt control method, interrupt controller, electronic device, medium, and chip | |
| CN112181942A (en) | Time sequence database system and data processing method and device | |
| US10372542B2 (en) | Fault tolerant event management system | |
| CN115629918B (en) | Data processing method, device, electronic equipment and storage medium | |
| CN113448770A (en) | Method, electronic device and computer program product for recovering data | |
| CN113641693B (en) | Data processing method and device of streaming computing system, electronic equipment and medium | |
| CN115544042A (en) | Cached information updating method and device, equipment and medium | |
| CN114697398A (en) | Data processing method and device, electronic equipment, storage medium and product | |
| CN113867891A (en) | Method, device, equipment and storage medium for migrating disk across available area cluster | |
| CN113821232A (en) | Model updating method and device | |
| CN112817701A (en) | Timer processing method and device, electronic equipment and computer readable medium | |
| CN111142795A (en) | Control method, control device and control equipment for write operation of distributed storage system | |
| CN112463514A (en) | Monitoring method and device for distributed cache cluster | |
| CN111694801A (en) | Data deduplication method and device applied to fault recovery | |
| WO2019126720A1 (en) | A system and method for optimization and load balancing of computer clusters | |
| CN117193670B (en) | Method and device for clearing cache, storage medium and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |