CN111190707A - Data processing method and device - Google Patents

Data processing method and device Download PDF

Info

Publication number
CN111190707A
CN111190707A CN201910712832.9A CN201910712832A CN111190707A CN 111190707 A CN111190707 A CN 111190707A CN 201910712832 A CN201910712832 A CN 201910712832A CN 111190707 A CN111190707 A CN 111190707A
Authority
CN
China
Prior art keywords
data
processing
group
unprocessed
state
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201910712832.9A
Other languages
Chinese (zh)
Other versions
CN111190707B (en
Inventor
罗洪林
肖晨光
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Tencent Technology Shenzhen Co Ltd
Original Assignee
Tencent Technology Shenzhen Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Tencent Technology Shenzhen Co Ltd filed Critical Tencent Technology Shenzhen Co Ltd
Priority to CN201910712832.9A priority Critical patent/CN111190707B/en
Publication of CN111190707A publication Critical patent/CN111190707A/en
Application granted granted Critical
Publication of CN111190707B publication Critical patent/CN111190707B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/48—Program initiating; Program switching, e.g. by interrupt
    • G06F9/4806—Task transfer initiation or dispatching
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Debugging And Monitoring (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

A data processing method and device comprises the following steps: acquiring a plurality of data, and dividing the data into a plurality of data groups; distributing the data groups to corresponding processing processes for processing, storing the group identification of the data groups to the local, and synchronizing the processing results of the processing processes to the monitoring process; when the processing process is abnormal, triggering the monitoring process to determine unprocessed data corresponding to the abnormal processing process based on the monitoring side record file; when the abnormal processing process is not recovered in a preset time period, distributing unprocessed data to other processing processes for processing; in the process of distributing unprocessed data to other processing processes, if the exception type of the exception processing process is a preset exception type and the exception processing process is recovered, checking the current group identifier of the unprocessed data; if the verification fails, the abnormal processing process is stopped to continue processing the unprocessed data, and other processing processes process the unprocessed data, so that the processing integrity is improved.

Description

Data processing method and device
Technical Field
The present application relates to the field of communications, and in particular, to a data processing method and apparatus.
Background
The data processing process in the communication system comprises the following steps: a series of data flows from an upstream system to a data processing system, flows to a downstream system through the data processing system, and reaches the process that the downstream system acquires the series of data. The circulation process is divided into two types, the first type is that data is published by an upstream system (Publish), a receiver of a downstream system subscribes, a data processing system records one or more subscribers, and data conforming to the rules is sent to the subscribers according to the subscription rules; the second is that data is produced by an upstream system (product), consumed by a downstream system consumer (Consume), received and stored by a data processing system, and sent to the consumer at the request of the consumer.
At present, no matter what kind of flow process is adopted, in order to ensure the high availability of the data processing system, a processing mode of a multi-node deployment and arbitration mechanism is generally adopted for processing, and the significance is that a plurality of nodes receive requests of users, and processing progress information of data is updated in a mode of mutually master-slave or election decision makers (leaders). When a certain node fails, the election is initiated again, and a new Leader is elected, so that the high availability of the data processing system is ensured, but when the failed node is the Leader, the original data to be processed is not processed, and the integrity of data processing is further reduced.
Disclosure of Invention
The embodiment of the application provides a data processing method and device, which can improve the integrity of data processing.
An embodiment of the present application provides a data processing method, including:
acquiring a plurality of data to be processed from a data source, dividing the plurality of data into a plurality of data groups, and storing the group identification of each data group to the local;
allocating each data group to a corresponding processing process in the processing process groups, and updating the data state of the data in the data source to be a to-be-processed state;
processing the data in the target data group according to the processing process and the data state of the data in the distributed target data group; synchronizing a processing result of the processing process to a monitoring process for monitoring the processing process of the processing process, wherein the processing result comprises a data state of processed data;
when the processing process is detected to be abnormal, triggering the monitoring process to determine unprocessed data in a data group corresponding to the abnormal processing process based on a monitoring side record file, wherein the monitoring side record file comprises a synchronized data state and a group identification of the monitoring side;
updating the data state of the unprocessed data in the data source to be an unprocessed state through a monitoring process;
when the abnormal processing process is detected not to be recovered to be normal in a preset time period, distributing the unprocessed data to other processing processes for processing, and adjusting the group identifier of the unprocessed data to the group identifier of the data group corresponding to the other processing processes, wherein the other processing processes are processing processes except the abnormal processing process in the processing process group;
in the process of distributing the unprocessed data to other processing processes, if the exception type of the exception processing process is a preset exception type and the exception processing process is recovered to be normal, checking the current group identifier of the unprocessed data;
if the verification fails, the abnormal processing process is stopped to continue processing the unprocessed data, and the other processing processes process the unprocessed data.
In some embodiments, the method further comprises:
when the abnormal processing process is detected to be recovered to be normal in a preset time period, distributing the unprocessed data to the abnormal processing process; updating the data state of the unprocessed data into a state to be processed;
and reprocessing the unprocessed data based on the exception handling process and the updated data state of the unprocessed data.
In some embodiments, the allocating the unprocessed data to another processing process for processing and adjusting the group identifier of the unprocessed data to the group identifier of the data group corresponding to the other processing process includes:
updating the data state of the unprocessed data into a state to be processed, and adjusting the group identifier of the unprocessed data into the group identifier of the data group corresponding to the other processing process;
and triggering other processing processes to process the unprocessed data according to the data state and the group identification of the unprocessed data.
In some embodiments, the triggering the monitor process to determine unprocessed data in a data group corresponding to an exception handling process based on a monitor-side log file includes:
and triggering the monitoring process to determine first data with a data state being a processing state and second data with a data state being a to-be-processed state in a data group corresponding to the abnormal processing process as unprocessed data based on the monitoring side record file.
In some embodiments, in the process of allocating the unprocessed data to the other processing processes, if the exception type of the exception handling process is a preset exception type and the exception handling process returns to normal, checking the current group identifier of the unprocessed data includes:
in the process of distributing the unprocessed data to other processing processes, if the exception type of the exception processing process is a preset exception type and the exception processing process is recovered to be normal, comparing the current group identification of the first data with the group identification corresponding to the exception processing process;
if the check fails, terminating the exception handling process to continue processing the unprocessed data, and processing the unprocessed data by the other handling processes, including:
and when the current group identification of the first data is different from the group identification corresponding to the abnormal processing process, stopping the abnormal processing process from processing the first data, and processing the first data by the other processing processes.
In some embodiments, the method further comprises:
and when the current group identification of the first data is the same as the group identification corresponding to the abnormal processing process, refusing to distribute the first data to other processing processes, and triggering the abnormal processing process to continue processing the first data.
In some embodiments, said assigning each data group to a respective process in the group of processes comprises:
carrying out group marking on each data group to obtain a group identifier corresponding to each data group;
determining a target process identifier corresponding to the group identifier from a process identifier set, wherein the process identifier set comprises process identifiers of processing processes in the processing process group;
and distributing the data group corresponding to the group identification to the processing process corresponding to the target process identification.
In some embodiments, the processing the data in the target data group according to the processing progress and the data state of the data in the allocated target data group includes:
determining target data which needs to be processed currently in the processing process from a target data group;
acquiring data processing expectation information corresponding to the processing process and data attribute information of the target data, wherein the data processing expectation information comprises a data group which is expected to be processed by the processing process and a data state which is expected to be processed, and the data attribute information comprises a current data state of the target data and a data group to which the target data belongs;
verifying the target data based on the data processing expectation information and the data attribute information;
and when the verification is passed, triggering a processing process to process the target data.
Correspondingly, an embodiment of the present application further provides a data processing apparatus, including:
the data dividing unit is used for acquiring a plurality of data to be processed from a data source, dividing the plurality of data into a plurality of data groups, and storing the group identification of each data group to the local;
the first data distribution unit is used for distributing each data group to a corresponding processing process in the processing process groups and updating the data state of the data in the data source into a state to be processed;
the first data processing unit is used for processing the data in the target data group according to the processing process and the data state of the data in the distributed target data group; synchronizing a processing result of the processing process to a monitoring process for monitoring the processing process of the processing process, wherein the processing result comprises a data state of processed data;
the data determining unit is used for triggering the monitoring process to determine unprocessed data in a data group corresponding to the abnormal processing process based on a monitoring side record file when the processing process is detected to be abnormal, wherein the monitoring side record file comprises a synchronized data state and a group identification of the monitoring side;
the data updating unit is used for updating the data state of the unprocessed data in the data source into an unprocessed state through a monitoring process;
a second data allocation unit, configured to, when it is detected that the abnormal processing process does not return to normal within a preset time period, allocate the unprocessed data to another processing process for processing, and adjust the group identifier of the unprocessed data to the group identifier of the data group corresponding to the other processing process, where the other processing process is a processing process in the processing process group other than the abnormal processing process;
the data verification unit is used for verifying the current group identifier of the unprocessed data if the exception type of the exception handling process is a preset exception type and the exception handling process is recovered to be normal in the process of distributing the unprocessed data to other handling processes;
and the termination processing unit is used for terminating the exception handling process to continue processing the unprocessed data if the verification fails, and processing the unprocessed data by the other processing processes.
In some embodiments, the data processing apparatus further comprises:
a third data allocation subunit, configured to, when it is detected that the exception handling process returns to normal within a preset time period, allocate the unprocessed data to the exception handling process; updating the data state of the unprocessed data into a state to be processed;
and the data processing unit is used for reprocessing the unprocessed data based on the exception handling process and the updated data state of the unprocessed data.
Correspondingly, an embodiment of the present application further provides a storage medium, where the storage medium stores instructions, and the instructions, when executed by a processor, implement the data processing method provided in any embodiment of the present application.
According to the data processing method provided by the embodiment of the application, when the data in the data groups are sent to the corresponding processing processes to be processed, the group identification of each data group is stored to the local, the processing result of each data group is synchronized to the monitoring process in the processing process, and if the processing process is abnormal, the data which are not processed in the abnormal processing process can be found according to the local and monitoring side record files; updating the data state of the unprocessed data into an unprocessed state to wait for processing through the monitoring process, and distributing the unprocessed data to other processing processes for processing when the abnormal processing process is not recovered within a preset time period, so as to avoid missing processing; in the process of distributing unprocessed data to other processing processes, if the exception type of the exception processing process is a preset exception type and the exception processing process returns to normal, the current group identifier of the unprocessed data is verified, if the verification is passed, the unprocessed data is not distributed to other processing processes, the unprocessed data can be continuously processed by the exception processing process, and if the verification is not passed, the unprocessed data is distributed to other processing processes, the exception processing process does not need to process the unprocessed data, so that repeated processing is avoided, and the processing integrity is improved.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
Fig. 1a is a schematic diagram of a first scenario of a data processing system according to an embodiment of the present application.
Fig. 1b is a schematic flowchart of a first data processing method according to an embodiment of the present disclosure.
Fig. 2 is a second flowchart of the data processing method according to the embodiment of the present application.
Fig. 3a is a schematic diagram of a second scenario of a data processing system according to an embodiment of the present application.
Fig. 3b is a third schematic flow chart of the data processing method according to the embodiment of the present application
Fig. 4a is a schematic diagram of a first interaction between a processing server, a monitoring server, and a data source in the data processing method according to the embodiment of the present application.
Fig. 4b is a schematic diagram of a second interaction between a processing server, a monitoring server, and a data source in the data processing method according to the embodiment of the present application.
Fig. 5a is a first flowchart of a processing server and a listening server when processing an exception according to an embodiment of the present application.
Fig. 5b is a second flowchart of the processing server and the listening server when processing an exception according to the embodiment of the present application.
Fig. 5c is a third flowchart of the processing server and the listening server when processing an exception according to the embodiment of the present application.
Fig. 5d is a fourth flowchart of the processing server and the listening server when processing an exception according to the embodiment of the present application.
Fig. 6a is a schematic structural diagram of a data processing apparatus according to an embodiment of the present application.
Fig. 6b is a schematic structural diagram of a second data processing apparatus according to an embodiment of the present application.
Fig. 7 is a schematic structural diagram of a server according to an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. It is to be understood that the embodiments described are only a few embodiments of the present application and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
The embodiment of the application provides a data processing method, a data processing device, a storage medium and a server. Wherein the data processing apparatus may be integrated in a server, which may refer to a single independent server, e.g. the data processing apparatus may be integrated in a data processing server; in addition, a server may also refer to a server cluster constructed by a plurality of servers, for example, a data processing device may be integrated in a data processing server or a monitoring server, that is, the data processing server or the monitoring server may implement the method of the present application,
Taking an example that a data processing apparatus is integrated in a single server, referring to fig. 1a, an embodiment of the present application provides a data processing system, including: a data source 10, a processing server 20 and a terminal 30, wherein the data source 10 may also be a third party service or a database provided in the terminal 30, which is not limited herein. The terminal 30 may be a mobile phone, a tablet computer, a notebook computer, etc., and fig. 1a illustrates the terminal 30 as a mobile phone. The terminal 30 may be installed with various applications required by the user, such as applications with entertainment functions (e.g., video applications, audio playing applications, game applications, and reading software), and applications with service functions (e.g., map navigation applications, dining applications, etc.).
Based on the system shown in fig. 1a, taking the user to use the shopping software to perform the robbery as an example, when a large number of users initiate the robbery request at the same time, the data source 10 includes request data initiated by a large number of users through the terminal 30, the processing server 20 can obtain data to be processed from the data source 10, and after the processing by the processing server 20 is completed, the processing server 20 resends the processing result of each piece of data to the data source 10, where the processing server 20 includes a processing process and a monitoring process for monitoring the processing process. Specifically, the processing server 20 obtains a plurality of data to be processed from the data source 10, and divides the plurality of data into a plurality of data groups; allocating each data group to a corresponding processing process in the processing process group, storing the group identification of each data group to the local, and updating the data state of the data in the data source 10 to a state to be processed; processing the data in the target data group according to the processing process and the data state of the data in the distributed target data group; and synchronously processing the processing result of the process to the monitoring process, wherein the processing result comprises the data state of the processed data; when the processing progress is detected to be abnormal, triggering the monitoring progress to determine unprocessed data in a data group corresponding to the abnormal processing progress based on the monitoring side record file, wherein the monitoring side record file comprises a synchronized data state and a group identification of the monitoring side; updating the data state of the unprocessed data in the data source 10 to an unprocessed state through the monitoring process; when the abnormal processing process is detected not to be recovered to be normal in the preset time period, distributing the unprocessed data to other processing processes for processing, and adjusting the group identifier of the unprocessed data to the group identifier of the data group corresponding to other processing processes, wherein the other processing processes are processing processes except the abnormal processing process in the processing process group; in the process of distributing unprocessed data to other processing processes, if the exception type of the exception processing process is a preset exception type and the exception processing process is recovered to be normal, checking the current group identifier of the unprocessed data; if the verification fails, the normal processing process is terminated to continue processing the unprocessed data, and other processing processes process the unprocessed data.
Specifically, after the data acquired from the data source 10 is processed by the processing server 20, the processed data may be selectively sent to the terminal 30 or stored in the processing server 20 according to different processing flow processes, which is not limited herein.
The above example of fig. 1a is only one example of a system architecture for implementing the embodiment of the present application, and the embodiment of the present application is not limited to the system architecture shown in fig. 1a, and various embodiments of the present application are proposed based on the system architecture.
In some embodiments, there is provided a data processing method, executable by a processor of a server, as shown in fig. 1b, the data processing method comprising:
101. the method comprises the steps of obtaining a plurality of data to be processed from a data source, and dividing the plurality of data into a plurality of data groups.
The data source is a source of data, and is a device or original medium providing some required data. The data source stores a plurality of data, which may include, for example, request data sent by a user for a first purchase or other data corresponding to the request data. The multiple data to be processed can be uniformly acquired in a set form, for example, acquiring a data set { data 1, data 2 … … data n }.
The data groups may be divided in various ways, for example, a plurality of data to be processed are divided by the total number of processing processes or the number of idle processing processes, for example, the total number of processing processes or the number of idle processing processes is 3, and n data to be processed are divided into 3 data groups on average.
102. And distributing each data group to a corresponding processing process in the processing process group, storing the group identification of each data group to the local, and updating the data state of the data in the data source to be the state to be processed.
The data group comprises data in at least one slave data source, the processing process group comprises a plurality of processing processes, and the processing processes are processes for processing the data, such as response processes responding to request data. Data state according to the processing process of the data by the processing process, the data state of the data can be divided into 4 types: the first type is an unprocessed state, which refers to a state when data is not acquired by a processing process in a data source, and when the processing process is abnormal, the processing state of the data is also updated from other states to the unprocessed state; the second type is a to-be-processed state, wherein the to-be-processed state refers to a state that data is loaded by a processing process and is to be processed by the processing process; the third is the in-process state, which refers to the state when a piece of data is being processed by the processing process for a single piece of data; the fourth is a processing completion state, which is a state after the data is processed by the processing process. The data state of the data can be marked by the state of each piece of data, so that each piece of data has one data state.
The group identifier corresponding to each data group may be stored locally, which may be a cache, a network disk, or the like. The information stored locally is not limited to the group identifier, but also can include information such as processing time for processing the group data.
Specifically, in order to send a data group to a corresponding process, therefore, assigning each data group to a corresponding process in a process group, the method includes:
carrying out group marking on each data group to obtain a group identifier corresponding to each data group;
determining a target process identifier corresponding to the group identifier from a process identifier set, wherein the process identifier set comprises process identifiers of processing processes in the processing process group;
and distributing the data group corresponding to the group identification to the processing process corresponding to the target process identification.
Specifically, the group marking of each data group includes the group marking of each piece of data in the data group. The corresponding relation between the data group and the processing process can be realized by marking the data in the data group with a group identification corresponding to the processing process, and sending the data with the same identification as the corresponding identification of the processing process to the processing process for processing through the group identification. The processing process processes each data according to the time sequence when processing the data. Data sets, data line locks, lock-free algorithms (CAS), etc. may be used to ensure that each data is exclusively owned by only one processing thread, avoiding other processing threads from dirty reading And thus repeatedly processing the same data.
103. Processing the data in the target data group according to the processing process and the data state of the data in the distributed target data group; and synchronizing the processing result of the processing process to the monitoring process of the processing process of the monitoring processing process.
After receiving the data in the data groups, each processing process judges the state of each data, the processing process only receives and processes the data with the data state of unprocessed data, and when the data in each data group is processed, the state of the data in the data group is updated to be the state to be processed. The pending state means that the data in the data group has been sent to the corresponding processing process to await processing by the processing process.
Wherein. The processing result includes the current data state of each piece of data. It can be understood that, when the processing process processes the data in the data group, the processing result of the data is synchronized to the monitoring process, the processing result may be the data state of the data, and the monitoring process records the data state of the data in the data group corresponding to the processing process. For example: the data group comprises data 1, data 2 and data 3, after the processing process finishes processing the data 1, the data state of the data 1 is updated to the processing finished state and is synchronized to the monitoring process, the monitoring process records the data state of the data 1 in the data group to be the processing finished state, the record can be stored in a record file on the monitoring side, and the record file can also be stored to a cache, a network disk and the like.
104. When the processing process is detected to be abnormal, the monitoring process is triggered to determine unprocessed data in the data group corresponding to the abnormal processing process based on the monitoring process and the monitoring side record file.
Specifically, according to the actual abnormal conditions occurring in the processing process, the abnormalities are divided into two types: the first type is to handle the process itself exception, which specifically includes: when the processing process processes data, the processing process crashes, is manually closed or quits the processing process, and the like; the second type is communication exception, i.e. exception caused by network such as communication disconnection between the processing process and the monitoring process. When an exception occurs, the monitoring process can be triggered to search whether data which is not processed yet exists in the group identifier of the data group corresponding to the exception handling process according to the group identifier recorded locally. The specific determination method comprises the following steps: the monitoring process searches data in the data group corresponding to the abnormal processing process from the data source, and then finds data with a data state except a processing completion state by combining a monitoring side record file generated by the monitoring processing process. Namely, the data state of each data in the corresponding data group of the abnormal processing process is verified. And if the data state of the data in the data group is a processing completion state, determining that the data is completed by the data, and if the data state of the data is a processing state or a to-be-processed state, determining that the data is unprocessed data. For example: the data group corresponding to the exception handling process comprises data 1, data 2 and data 3, the data state of the data 1 is a handling completion state, the data state of the data 2 and the data 3 is a to-be-handled state, and it can be determined that the data 2 and the data 3 are not processed when an exception occurs, so that the data 2 and the data 3 are determined to be unprocessed data in the data group corresponding to the exception handling process; or the data 1 is in the processing state, the data states of the data 2 and the data 3 are in the pending state, and the data 1, the data 2 and the data 3 are also determined as the unprocessed state. If the processing process is abnormal while processing the data 2, determining the data 2 with the data state being processing as unprocessed data, so that the unprocessed data is determined from the target data group according to the current data state of the data in the target data group, and the method comprises the following steps:
and determining first data with a data state being a processing state and second data with a data state being a to-be-processed state in a data group corresponding to the exception handling process as unprocessed data.
105. And updating the data state of the unprocessed data in the data source to be an unprocessed state through the monitoring process.
When the monitoring process determines that the data is not processed, the data state of the data is updated from the to-be-processed state or the in-process state to the unprocessed state in the data source.
106. When the abnormal processing process is detected not to be recovered to be normal in the preset time period, distributing the unprocessed data to other processing processes for processing, and adjusting the group identification of the unprocessed data to the group identification of the data group corresponding to other processing processes.
Specifically, in order to avoid that the unprocessed data in the data group corresponding to the abnormal processing process cannot be processed due to the fact that the abnormal processing process cannot be recovered for a long time, a preset time period may be set, whether the abnormal processing process is recovered to be normal within the preset time period is judged, if yes, the unprocessed data is allocated to the processing process recovering to be normal to continue processing, the data state of the unprocessed data is updated to be a to-be-processed state, and the unprocessed data is reprocessed based on the processing process and the updated data state of the unprocessed data. Thus, the method further comprises:
when the abnormal processing process is detected to be recovered to be normal in a preset time period, distributing unprocessed data to the abnormal processing process; updating the data state of the unprocessed data into a state to be processed;
and reprocessing the unprocessed data based on the exception handling process and the updated data state of the unprocessed data.
If the abnormal processing process does not recover to be normal in the preset time period, distributing the unprocessed data to other processing processes for processing, wherein after the unprocessed data are distributed to other processing processes, the other processing processes can adjust the group identifier of the processed data to the group identifier of the data group corresponding to the other processing processes; it will be appreciated that the other processing processes are processing processes in the processing process group other than the exception handling process.
The unprocessed data is reprocessed based on the other processing processes and the updated data state of the unprocessed data.
107. In the process of distributing unprocessed data to other processing processes, if the exception type of the exception processing process is a preset exception type and the exception processing process is recovered to be normal, checking the current group identifier of the unprocessed data
Specifically, the preset exception type is communication exception, when the exception type of the exception handling process is communication exception, it indicates that the exception handling process is in a state capable of handling data, when the exception occurs, the exception handling process is handling a certain piece of data, and based on the handling mechanism, the piece of data is handled after the exception occurs, but because the communication between the exception handling process and the monitoring process is not recovered, the unprocessed data may have been already distributed to other handling processes, so that the same piece of data is handled by the exception handling process and the other handling processes, that is, the data is repeatedly handled. Therefore, in the process of distributing unprocessed data to other processing processes, if the exception type of the exception processing process is the preset exception type and the exception processing process is recovered to be normal, the current group identifier of the unprocessed data needs to be checked;
when an exception occurs, the exception handling process is processing a certain piece of data, the current group identifier of the piece of data can be compared with the group identifier corresponding to the exception handling process, that is, the second data can be distributed to other handling processes, and the first data determines whether to process the piece of data or not through the group identifier. Therefore, in the process of allocating unprocessed data to other processing processes, if the exception type of the exception handling process is the preset exception type and the exception handling process returns to normal, checking the current group identifier of the unprocessed data includes:
in the process of distributing unprocessed data to other processing processes, if the exception type of the exception processing process is a preset exception type and the exception processing process is recovered to be normal, comparing the current group identifier of the first data with the group identifier corresponding to the exception processing process.
108. If the verification fails, the abnormal processing process is stopped to continue processing the unprocessed data, and other processing processes process the unprocessed data.
Specifically, if the check fails, it indicates that the unprocessed data has been allocated to another processing process, the exception handling process is terminated to continue to process the unprocessed data, and the other processing process processes the unprocessed data.
And when the current group identifier of the first data is different from the group identifier corresponding to the abnormal processing process, stopping the abnormal processing process to process the first data, and processing the first data by other processing processes. Of course, since the group identifier of the first data has been changed, it means that the group identifier of the second data is also changed at the same time, and the second data is also distributed to other processing procedures for processing.
In addition, because the processing process performs the group identification on the data group before processing the data in the data group, if the group identification check passes, it indicates that none of the unprocessed data is allocated to other processing processes, the exception handling process may continue to process the unprocessed data. Thus, the method further comprises:
if the verification is passed, the unprocessed data is refused to be distributed to other processing processes, and the abnormal processing process is triggered to continue processing the unprocessed data. The following steps can be also included: when the current group identifier of the first data is the same as the group identifier corresponding to the abnormal processing process, refusing to allocate the first data to other processing processes, and triggering the abnormal processing process to continue processing the first data, wherein the group identifier of the second data is not changed because the group identifier of the first data is not changed, but the unprocessed data is allocated to other processing processes, so that the second data can be processed by other processing processes, or refusing to allocate the second data to other processing processes, and the recovered abnormal processing process continues processing, which is not limited herein.
Specifically, after the processing process completes processing of data in the target data group, the data is sent to the downstream system in a group or strip form, and if the downstream system is unavailable due to some reason, retry is performed for multiple times to ensure that the processed data is not missed when being sent to the downstream system. And, after processing a data set each time, the processing process can delete the group identification saved to the local, and the monitoring side can delete the record information about the data set in the monitoring side record file at the same time.
As can be seen from the above, in the embodiment of the present application, while data in data groups is sent to a corresponding processing process for processing, a group identifier of each data group is stored to the local, in the processing process, a processing result of each data group is synchronized to a monitoring process, and if an exception occurs in the processing process, it is ensured that data that has not been processed by an exception processing process can be found according to local and monitoring-side record files; updating the data state of the unprocessed data into an unprocessed state to wait for processing through the monitoring process, and distributing the unprocessed data to other processing processes for processing when the abnormal processing process is not recovered within a preset time period, so as to avoid missing processing; in the process of distributing unprocessed data to other processing processes, if the exception type of the exception processing process is a preset exception type and the exception processing process returns to normal, the current group identifier of the unprocessed data is verified, if the verification is passed, the unprocessed data is not distributed to other processing processes, the unprocessed data can be continuously processed by the exception processing process, and if the verification is not passed, the unprocessed data is distributed to other processing processes, the exception processing process does not need to process the unprocessed data, so that repeated processing is avoided, and the processing integrity is improved.
In some embodiments, taking the data processing apparatus as an example, which is based on a single server, as shown in fig. 2, the data processing method includes:
201. the server acquires a plurality of data to be processed from the data source and divides the plurality of data into a plurality of data groups.
The data source stores a plurality of data, and the data may include request data sent by a user for a first purchase or data corresponding to the request data. The multiple data to be processed can be uniformly acquired in a set form, for example, acquiring a data set { data 1, data 2 … … data n }.
The processing server is provided with a processing progress group comprising a plurality of processing progresses, and after receiving a plurality of data, the processing server divides the plurality of data to be processed into a plurality of data groups according to the plurality of processing progresses in the processing progress group.
The dividing manner may divide the plurality of data to be processed by the total number of processing processes or the number of idle processing processes, such as: the total number of processing processes or the number of idle processing processes is 3, and n data to be processed are averagely divided into 3 data groups.
202. And the server marks each data group in a group manner to obtain a group identifier corresponding to each data group.
Specifically, when each data group is marked with a group identity, it is marked with a plurality of processing processes in the corresponding processing process group, And it can be ensured that each data is exclusively occupied by only one processing thread by using data sets, data line locks, And lock-free algorithms (CAS), etc., to avoid that other processing threads dirty read And process the same data repeatedly.
203. And the server determines a target process identifier corresponding to the group identifier from the process identifier set.
It is understood that, in order to determine that each group-marked data group can be allocated to a corresponding processing process, a process identification set may be pre-established, and the process identification set may include a process identification of each processing process, for example: if the process group includes process 1 and process 2 … …, the process identifier set includes {1, 2 … … n }, and in step 203, the process identifier set may directly mark 1, 2 … … n in the process identifier set to the corresponding data group or mark the identifier corresponding to the process identifier to the data group, which is not limited herein.
204. And the server distributes the data groups corresponding to the group identification to the processing process corresponding to the target process identification, and stores the group identification of each data group to the local.
As can be seen from step 204, since the process identifier corresponding to the processing process or the identifier corresponding to the process identifier is marked to the data group when the group identifier is marked for each data group, the processing process corresponding to each data group can be found according to the group identifier obtained by the group identifier and the target process identifier, and the data in the data group is allocated to the processing process corresponding to the target process identifier.
The group identifier corresponding to each data group may be stored locally, which may be a cache, a network disk, or the like. The information stored locally is not limited to the group identification, but also can comprise information such as processing time for processing the group data
205. The server determines the target data which needs to be processed currently by the processing process from the target data group.
It can be understood that after the data group is assigned to the corresponding processing process, each data process in the data group needs to be verified to determine whether it is the target data that the corresponding processing process currently needs to process.
The method specifically comprises the following steps: in step 206, the server obtains the data processing expectation information corresponding to the processing process and the data attribute information of the target data. Wherein, the data processing expected information may include: information such as group identification of a data group which is expected to be processed by the processing process, data state of the data which is expected to be processed, whether the data state of the data is rolled back, and the like; the data attribute information may include: the actual group identifier of the data, the data state, whether the data state is rolled back, and the like, wherein the rolling back is as follows: the data state is updated from the other state to unprocessed.
Step 207, the server verifies the target data based on the data processing expectation information and the data attribute information. If the verification is passed, step 208 is performed.
Specifically, the verification method may be to compare the data processing expected information with the data attribute information, and compare the group identifier of the data group expected to be processed by the processing process with the actual group identifier of the data in the data attribute information; the data state of the data expected to be processed by the processing process and the actual data state of the data in the data attribute information are processed; whether the data which the processing process expects to process is rolled back or not is compared with whether the data in the data attribute information is actually rolled back or not. Step 208 is performed only if each verification condition has passed.
And step 208, the server triggers a processing process to process the target data and updates the data state of the corresponding data in the data source.
And when the verification is passed, if the target data is the data which needs to be processed currently in the processing process, updating the data state of the target data in the data source. The updated data state is a pending state, meaning that the pending process processes the data. The updated data state may also be notified to the data source, and the data state of the data is updated by the data source, which is not described herein again. If the verification fails, the data are ignored if the processing process does not need to process the data.
Step 209 synchronizes the processing results of the processing process to the listening process of the listening processing process.
Wherein. The processing results include the data state of the processed data. It can be understood that after each piece of data is processed, the processing process synchronizes a processing result of the piece of data to the monitoring process, where the processing result may be a data state of the piece of data, and the monitoring process records the data state of the piece of data in the data group corresponding to the processing process. For example: the data group comprises data 1, data 2 and data 3, after the processing process finishes processing the data 1, the data state of the data 1 is updated to the processing finished state and is synchronized to the monitoring process, the monitoring process records the data state of the data 1 in the data group to be the processing finished state, the record can be stored in a record file on the monitoring side, and the record file can also be stored to a cache, a network disk and the like.
Step 210, when the server detects that the processing process is abnormal, triggering the monitoring process to determine, based on the monitoring-side record file, first data in which the data state in the data group corresponding to the abnormal processing process is the processing state and second data in which the data state is the to-be-processed state as unprocessed data.
Specifically, according to the actual abnormal conditions occurring in the processing process, the abnormalities are divided into two types: the first type is to handle the process itself exception, which specifically includes: when the processing process processes data, the processing process crashes, is manually closed or quits the processing process, and the like; the second type is communication exception, i.e. exception caused by network such as communication disconnection between the processing process and the monitoring process. When an exception occurs, the monitoring process can be triggered to search whether data which is not processed yet exists in the group identifier of the data group corresponding to the exception handling process according to the group identifier recorded locally. The specific determination method comprises the following steps: the monitoring process searches data in the data group corresponding to the abnormal processing process from the data source, and then finds data with a data state except a processing completion state by combining a monitoring side record file generated by the monitoring processing process. Namely, the data state of each data in the corresponding data group of the abnormal processing process is verified. And if the data state of the data in the data group is a processing completion state, determining that the data is completed by the data, and if the data state of the data is a processing state or a to-be-processed state, determining that the data is unprocessed data. For example: the data group corresponding to the exception handling process comprises data 1, data 2 and data 3, the data state of the data 1 is a handling completion state, the data state of the data 2 and the data 3 is a to-be-handled state, and it can be determined that the data 2 and the data 3 are not processed when an exception occurs, so that the data 2 and the data 3 are determined to be unprocessed data in the data group corresponding to the exception handling process; or the data 1 is in the processing state, the data states of the data 2 and the data 3 are in the pending state, and the data 1, the data 2 and the data 3 are also determined as the unprocessed state. If the processing process is abnormal while processing the data 2, the data 2 with the data state of processing is also determined as unprocessed data
It is understood that, when processing each piece of data, the processing process updates the data whose data state is the pending state to the in-process state, indicating that the piece of data is being processed. If the processing progress is abnormal, if the data in the processing state is not processed, determining the first data in the processing state and the second data in the pending state as unprocessed data.
If the processing process is not abnormal in the processing process, after the processing process finishes processing the data in the data group, the data state of the data in the data group is updated to be finished in the data source, and the information stored to the local in the step 203 is deleted.
And step 211, updating the data state of the unprocessed data in the data source to an unprocessed state through the monitoring process.
When the monitoring process determines that the data is not processed, the data state of the data is updated from the to-be-processed state or the in-process state to the unprocessed state in the data source. Specifically, the monitoring process may also be instructed to update the data state of the unprocessed data at the data source, and count the number of times that the data state of the data is changed into the unprocessed state, that is, the number of rollback times, where the number of rollback times is increased by 1 when the data state of the data is changed into the unprocessed state once.
Step 212, the server determines whether the processing procedure is normal within a preset time period, if so, step 213 is executed; if not, go to step 215.
In order to ensure the data processing efficiency, it is impossible to wait for the recovery of the abnormal processing process and then process the corresponding data. Therefore, a preset time period can be preset to determine whether the processing process is normal within the preset time period. The preset time period can be 2s, 1min and the like, and different time is set according to different application scenes.
Step 213, the server allocates the unprocessed data to the processing process; and updating the data state of the unprocessed data to be a to-be-processed state.
When the abnormal processing process returns to normal within the preset time period, the data which is not processed by the abnormal processing process can be allocated to the processing process again for processing, and after the unprocessed data is allocated, the data state of the unprocessed data is updated from the unprocessed state to the state to be processed again.
Step 214, the server reprocesses the unprocessed data based on the processing progress and the updated data state of the unprocessed data.
Specifically, unprocessed data is redistributed, and the unprocessed data is reprocessed according to the updated data state of the unprocessed data. Unprocessed data may also be retrieved from a newly derived data source and redistributed.
Step 215, the server distributes the unprocessed data to other processing processes and updates the data state of the unprocessed data to the pending state.
Specifically, if the abnormal processing process does not return to normal within a preset time period, allocating the unprocessed data to other processing processes for processing, wherein after the unprocessed data is allocated to other processing processes, the other processing processes adjust the group identifier of the processed data to the group identifier of the data group corresponding to the other processing processes; it will be appreciated that the other processing processes are processing processes in the processing process group other than the exception handling process.
Step 216, in the process of distributing the unprocessed data to other processing processes, if the exception type of the exception processing process is the preset exception type and the exception processing process is recovered to be normal, comparing the current group identifier of the first data with the group identifier corresponding to the exception processing process.
Specifically, the preset exception type is communication exception, when the exception type of the exception handling process is communication exception, it indicates that the exception handling process is in a state capable of handling data, when the exception occurs, the exception handling process is handling a certain piece of data, and based on the handling mechanism, the piece of data is handled after the exception occurs, but because the communication between the exception handling process and the monitoring process is not recovered, the unprocessed data may have been already distributed to other handling processes, so that the same piece of data is handled by the exception handling process and the other handling processes, that is, the data is repeatedly handled. When an exception occurs and the exception handling process is handling a certain piece of data, comparing the current group identification of the piece of data with the group identification corresponding to the exception handling process, namely, the second data can be distributed to other handling processes, and the first data determines whether to handle the piece of data or not through the group identification.
And step 217, when the current group identifier of the first data is different from the group identifier corresponding to the exception handling process, terminating the exception handling process to process the first data, and processing unprocessed data by other handling processes.
Specifically, if the comparison result is that the current group identifier of the first data is not the same as the group identifier corresponding to the abnormal processing process, it is determined that the first data has been allocated to another processing process, the abnormal processing process is terminated to continue processing the unprocessed data, and the other processing processes process the unprocessed data.
Step 218, when the current group identifier of the first data is the same as the group identifier corresponding to the exception handling process, refusing to allocate the first data to other handling processes, and triggering the exception handling process to continue to process the first data.
Specifically, if the comparison result is that the current group identifier of the first data is the same as the group identifier corresponding to the abnormal processing process, it is determined that the first data is not yet allocated to another processing process, the first data is rejected to be allocated to another processing process, and the abnormal processing process is triggered to continue processing the first data.
If other processing processes are processing other data and meanwhile receive unprocessed data, the unprocessed data may be preferentially processed, or the unprocessed data may be processed after the other data is processed, which is not limited herein.
Referring to fig. 3a, taking the data processing apparatus integrated in the data processing server and the monitoring server, that is, taking the data processing server and the monitoring server as an example, the embodiment of the present application further provides a data processing system, including: data source 10, processing server 20, terminal 30 and listening server 40. The data source 10 may also be a third party service or a database provided in the terminal 30, which is not limited herein. The terminal 30 may be a mobile phone, a tablet computer, a notebook computer, etc., and fig. 3a illustrates the terminal 30 as a mobile phone. The terminal 30 may be installed with various applications required by the user, such as applications with entertainment functions (e.g., video applications, audio playing applications, game applications, and reading software), and applications with service functions (e.g., map navigation applications, dining applications, etc.). The listening server 40 is used for listening to the processing procedure of the processing server 20, and the listening server 40 and the processing server 20 may be connected through a network protocol (TCP/IP). The processing server 20 is connected to the listening server 40 before processing data or may be connected to the listening server 40 regardless of whether the processing server 20 processes data, and when the processing server 20 is disconnected from the listening server 40, the processing server 20 does not process unprocessed data. The processing server 20 includes a processing process group having a plurality of processing processes, the monitoring server 40 includes a monitoring process group having a plurality of monitoring processes, and the connection between the processing processes and the monitoring processes may be one-to-one, one-to-many, or many-to-many.
Specifically, the processing server 20 obtains a plurality of data to be processed from the data source 10, and divides the plurality of data into a plurality of data groups; distributing each data group to a corresponding processing process in the processing process groups, storing the group identification of each data group to the local, and updating the data state of the data in the data source to be a to-be-processed state; processing the data in the target data group according to the processing process and the data state of the data in the distributed target data group, updating the data state of the corresponding data in the data source 10 and synchronizing the processing result of the processing process to the monitoring server 40, wherein the processing result comprises the data state of the processed data; when the processing server 20 detects that the processing process is abnormal, the monitoring server 40 is triggered to determine unprocessed data in the data group corresponding to the abnormal processing process based on the monitoring side record file including the synchronized data state and the group identification of the monitoring side, the data state of the unprocessed data in the data source 10 is updated to an unprocessed state through the monitoring server 40, and the processing server 20 reprocesses the unprocessed data based on the updated data state of the unprocessed data.
Therefore, please refer to fig. 3b, which illustrates a data processing method of the present application, further comprising:
step 301, the processing server obtains a plurality of data to be processed from the data source, and divides the plurality of data into a plurality of data groups.
The data source is a source of data, and is a device or original medium providing some required data. The data source stores a plurality of data, which may include, for example, request data sent by a user for a first purchase or other data corresponding to the request data. The multiple data to be processed can be uniformly acquired in a set form, for example, acquiring a data set { data 1, data 2 … … data n }.
The data groups may be divided in various ways, for example, a plurality of data to be processed are divided by the total number of processing processes or the number of idle processing processes, for example, the total number of processing processes or the number of idle processing processes is 3, and n data to be processed are divided into 3 data groups on average.
Step 302, allocating each data group to a corresponding processing process in the processing process group, and updating the data state of the data in the data source to a to-be-processed state.
The data group comprises data in at least one slave data source, the processing process group comprises a plurality of processing processes, and the processing processes are processes for processing the data, such as response processes responding to request data. Data state according to the processing process of the data by the processing process, the data state of the data can be divided into 4 types: the first type is an unprocessed state, which refers to a state when data is not acquired by a processing process in a data source, and when the processing process is abnormal, the processing state of the data is also updated from other states to the unprocessed state; the second type is a to-be-processed state, wherein the to-be-processed state refers to a state that data is loaded by a processing process and is to be processed by the processing process; the third is the in-process state, which refers to the state when a piece of data is being processed by the processing process for a single piece of data; the fourth is a processing completion state, which is a state after the data is processed by the processing process. The data state of the data can be marked by the state of each piece of data, so that each piece of data has one data state.
Specifically, in order to send a data group to a corresponding process, therefore, assigning each data group to a corresponding process in a process group, the method includes:
carrying out group marking on each data group to obtain a group identifier corresponding to each data group;
determining a target process identifier corresponding to the group identifier from a process identifier set, wherein the process identifier set comprises process identifiers of processing processes in the processing process group;
and distributing the data group corresponding to the group identification to the processing process corresponding to the target process identification.
And 303, processing the data in the target data group according to the processing process and the data state of the data in the distributed target data group, and updating the data state of the corresponding data in the data source and the processing result of the synchronous processing process to the monitoring server.
After receiving the data in the data groups, each processing process judges the state of each data, the processing process only receives and processes the data with the data state of unprocessed data, and when the data in each data group is processed, the state of the data in the data group is updated to be the state to be processed. The pending state means that the data in the data group has been sent to the corresponding processing process to wait for the processing of the processing process.
After a piece of data is processed by the processing process, the data source will update the data state of the piece of data to a processing completion state, and the processing result of the piece of data will be synchronized to the monitoring server, where the processing result may include: data state of processed data.
And step 304, when the processing process is detected to be abnormal, determining unprocessed data in the data group corresponding to the abnormal processing process.
When the processing server detects that the processing process is abnormal, the processing server determines which data in the data group corresponding to the abnormal processing process are not processed. Similarly, when detecting the occurrence of an exception, the monitoring server may also determine unprocessed data in the data group corresponding to the exception handling process. Specifically, the determination mode may be determined by a processing result of the synchronization from the processing server to the monitoring server to determine whether the processing of the data in the data group is completed.
For example: the monitoring server monitors the data 1 and the data 2 … … n in the data group a, when the processing process is abnormal, the monitoring server checks the state information of each data, and if the data 1 is in a processing completion state and the data 2 is in a processing state … …, the data n is in a pending state, the data 2 to the data n are determined as unprocessed data.
Therefore, specifically, when detecting that an exception occurs in a processing process, determining unprocessed data in a data group corresponding to the exception processing process includes:
and when the processing process is detected to be abnormal, triggering the monitoring server to determine unprocessed data in the data group corresponding to the abnormal processing process based on the synchronized data state.
Specifically, determining unprocessed data in the data group corresponding to the exception handling process includes:
and determining first data with a data state being a processing state and second data with a data state being a to-be-processed state in a data group corresponding to the exception handling process as unprocessed data.
It is understood that, when processing each piece of data, the processing process updates the data whose data state is the pending state to the in-process state, indicating that the piece of data is being processed. If the processing progress is abnormal, if the data in the processing state is not processed, determining the first data in the processing state and the second data in the pending state as unprocessed data.
And 305, updating the data state of the unprocessed data in the data source to an unprocessed state through a monitoring process.
When the interception server determines that the data is not processed, the data state of the data is updated to be the unprocessed state in the data source.
Step 306, in the process of distributing the unprocessed data to other processing processes, if the exception type of the exception processing process is communication exception and the exception processing process is recovered to normal, comparing the current group identifier of the first data with the group identifier corresponding to the exception processing process. If not, go to step 307; if so, go to step 308.
Specifically, the communication exception refers to disconnection of network communication between the processing server and the monitoring server, that is, TCP/IP disconnection, because the processing process in the processing server has no problem, and only the network between the processing server and the monitoring process is disconnected, in order not to affect the normal data processing progress in the process of being disconnected and recovering to normal in the second preset time period, if the processing process is processing data in the data state of being processed during disconnection, the data will be processed continuously, and the data in the subsequent data state of being to be processed will not be processed. Therefore, the first data with the data state being the processing state and the second data with the data state being the to-be-processed state in the data group corresponding to the abnormal processing process are determined as unprocessed data, and then the current group identifier of the first data is compared with the group identifier corresponding to the abnormal processing process to judge whether the first data is allocated to other processes for processing. The time of the second preset time period is longer than that of the first preset time period, namely the exception handling process is not recovered in the first preset time period and is recovered in the second preset time period.
And 307, when the current group identifier of the first data is different from the group identifier corresponding to the exception handling process, terminating the exception handling process to process the first data.
It can be understood that, if the current group identifier of the first data is not the same as the group identifier corresponding to the exception handling process, it indicates that the group identifier of the first data has been updated by another process, and in order to avoid the exception handling process and the other handling process from repeatedly handling the first data, the exception handling process is terminated to continue to handle the first data.
And 308, when the current group identification of the first data is the same as the group identification corresponding to the exception handling process, refusing to distribute the first data to other handling processes, and triggering the exception handling process to continue to handle the first data process.
It can be understood that if the current group identifier of the first data is the same as the group identifier corresponding to the exception handling process, it indicates that the group identifier of the first data is not updated by other processes, and the exception handling process recovered in the second preset time period is triggered to continue to process the first data process.
Specifically, after the processing process completes processing of data in the data group, the data is sent to the downstream system in a group or strip form, and if the downstream system is unavailable for some reason, the data is retried for multiple times to ensure that the processed data is not missed when being sent to the downstream system.
Please refer to fig. 4a and 4b for an interaction method of the processing server, the monitoring server, and the data source.
In the data loading phase: before processing each group of data, the processing server stores the group identifier of the data group to the local, the monitoring server and the data source of the monitoring processing server also synchronize the group identifier of the data group, and the data source returns the group of data to the processing server according to the group identifier;
in the data processing stage: the processing server processes the received data, and synchronizes the processing result to the monitoring server and the data source when processing one piece of data, and the processing server and the monitoring server can both update the data state of the processed data to the processing completion state in the data source; judging whether the data in the current data group are completely processed or not, if not, continuing the processing, if so, deleting the group identification stored to the local, and ending the processing;
in the abnormal rollback phase: if the processing process is abnormal, the monitoring server finds the group identification of the data group processed corresponding to the abnormal processing process, and updates the data state of the data which is not processed by the abnormal processing process into the unprocessed state in the data source;
in the data loading phase: the processing server acquires unprocessed data from the data source again, circularly verifies the group identification and the data state of each piece of data in the data group, judges whether the data is processed, skips the piece of data if the current data is processed, continuously judges whether other pieces of data are processed, processes the piece of data if the current data is not processed, synchronizes the processing result of the piece of data to the monitoring server, and updates the data state of the piece of data in the data source to be processed. And if the data in the data group corresponding to the abnormal process is completely processed, the monitoring server completes the processing of the data group corresponding to the record group identification, and the processing is finished.
For a clearer understanding of the data processing method described in the present application, please refer to fig. 5a, fig. 5b, fig. 5c, and fig. 5d, and fig. 5a to fig. 5d illustrate the data processing method from the perspective of the processing process and the listening process.
The BATCH is a group identifier, disconnection indicates that the processing process is abnormal, the monitoring process A is used for monitoring the processing process A, the processing process B is other processing processes, and the monitoring process B is other monitoring processes.
Specifically, as shown in fig. 5a, when the processing process a is abnormal, the monitoring process a of the monitoring processing process a rolls back the data whose group is identified as a and whose data state is to be processed. If the unprocessed data is loaded by the processing process B and the BATCH is updated to be the BATCH, if the abnormal processing process returns to normal in the second preset time period, due to the processing mechanism, the processing process a needs to connect a monitoring process, specifically, taking the connection to the monitoring process B as an example, after the monitoring process B establishes a connection with the processing process a, the processing process a triggers the monitoring process B to roll back the data of the BATCH as a according to the information such as the group identifier stored locally, and since the group identifier is already loaded and changed by the processing process B before, the group identifier of the unprocessed data at this time is the group identifier B, and therefore, the number of lines actually affected by the roll-back operation of the monitoring process B at this time is zero. When the processing process A judges that the current group identifier of the unprocessed data and the group identifier of the corresponding data group are updated to be B, the processing process A does not process the unprocessed data any more, the processing process B processes the unprocessed data, and after the processing process B finishes processing the data, the data state of the data is updated to be a processed state.
Similarly, as shown in fig. 5b, when the processing process a is abnormal, the monitoring process a of the monitoring processing process a rolls back the data whose group is identified as a and whose data state is to be processed. If the processing process a is reconnected, the processing process a triggers the monitoring process B to roll back the BATCH-a data according to the information such as the group identifier stored locally, because the data state of the unprocessed data is rolled back to the unprocessed state before, and the data state of the data expected to be rolled back by the monitoring process B is the pending state, the number of lines actually affected by the roll-back operation of the monitoring process B is zero due to the unmatched data state, if the unprocessed data is still acquired by the processing process B, the group identifier of the unprocessed data is updated to the BATCH-B, because the group identifier of the unprocessed data is updated to the B, the processing process a does not process the unprocessed data any more, the processing process B processes the unprocessed data, and after the processing process B completes the data processing, the data state of the data is updated to the processed state, and after the processing process B completes the data processing, updating the data state of the data into a processing completion state.
Similarly, as shown in fig. 5c, when the processing process a is abnormal, the monitoring process a of the monitoring processing process a rolls back the data whose group is identified as a and whose data state is to be processed. If the unprocessed data is loaded by the processing process B and the BATCH is updated to be the BATCH, if the abnormal processing process returns to normal in the second preset time period, due to the processing mechanism, the processing process a needs to connect a monitoring process, specifically, taking the connection to the monitoring process B as an example, after the monitoring process B establishes a connection with the processing process a, the processing process a triggers the monitoring process B to roll back the data of the BATCH as a according to the information such as the group identifier stored locally, and since the group identifier is already loaded and changed by the processing process B before, the group identifier of the unprocessed data at this time is the group identifier B, and therefore, the number of lines actually affected by the roll-back operation of the monitoring process B at this time is zero. When the processing process A judges that the current group identifier of the unprocessed data and the group identifier of the corresponding data group are updated to be B, the processing process A does not process the unprocessed data any more, the processing process B processes the unprocessed data, and after the processing process B finishes processing the data, the data state of the data is updated to be a processed state.
Similarly, as shown in fig. 5d, when the processing process a is abnormal, the monitoring process a of the monitoring processing process a rolls back the data whose group is identified as a and whose data state is to be processed. If the processing process a is reconnected, the processing process a triggers the monitoring process B to roll back the data of BATCH ═ a according to the information such as the group identifier stored locally, since the data state of the unprocessed data has been previously rolled back to an unprocessed state, while the data state of the data for which the snooping process B expects a roll-back is a pending state, therefore, the rollback operation of the snooping process B actually affects zero number of lines due to the mismatch of the data states, and if the unprocessed data is still acquired by the processing process B, the group identifier of the unprocessed data is updated to BATCH ═ B, because the group identifier of the unprocessed data is updated to B, the processing process A can not process the unprocessed data any more, but the processing process B can process the unprocessed data, and after the processing process B completes the data processing, updating the data state of the data into a processing completion state.
If the reconnection and rollback are performed before the other processing processes process the load, the rolled-back unprocessed data may be preferentially allocated to the reconnected processing processes for processing, or the rolled-back unprocessed data may be allocated to the other processing processes, which is not limited herein.
As can be seen from the above, in the embodiment of the present application, while data in data groups is sent to a corresponding processing process for processing, a group identifier of each data group is stored to the local, in the processing process, a processing result of each data group is synchronized to a monitoring process, and if an exception occurs in the processing process, it is ensured that data that has not been processed by an exception processing process can be found according to local and monitoring-side record files; updating the data state of the unprocessed data into an unprocessed state to wait for processing through the monitoring process, and distributing the unprocessed data to other processing processes for processing when the abnormal processing process is not recovered within a preset time period, so as to avoid missing processing; in the process of distributing unprocessed data to other processing processes, if the exception type of the exception processing process is a preset exception type and the exception processing process returns to normal, the current group identifier of the unprocessed data is verified, if the verification is passed, the unprocessed data is not distributed to other processing processes, the exception processing process can continue processing, and if the verification is failed, the unprocessed data is distributed to other processing processes, the exception processing process does not need to process the unprocessed data, so that repeated processing is avoided, the processing integrity is improved, and the data expected to be processed by the processing processes can be screened out through the verification of the data processing expected information and the data attribute information, so that the processing of the unnecessary data is avoided.
In order to better implement the data processing method provided by the embodiment of the present application, in an embodiment, a data processing apparatus is further provided. The terms are the same as those in the data processing method, and details of implementation can be referred to the description in the method embodiment.
In some embodiments, there is also provided a data processing apparatus, as shown in fig. 6a and 6b, the data processing apparatus may include: a data dividing unit 401, a first data allocating unit 402, a first data processing unit 403, a data determining unit 404, a data updating unit 405, a second data allocating unit 406, a data verifying unit 407, and a termination processing unit 408.
A data dividing unit 401, configured to obtain multiple data to be processed from a data source, and divide the multiple data into multiple data groups;
a first data allocation unit 402, configured to allocate each data group to a corresponding processing process in a processing process group, store a group identification of each data group to a local, and update a data state of the data in a data source to a to-be-processed state;
a first data processing unit 403, configured to process data in the target data group according to the processing procedure and the data state of the data in the allocated target data group; synchronizing a processing result of the processing process to a monitoring process for monitoring the processing process of the processing process, wherein the processing result comprises a data state of processed data;
a data determining unit 404, configured to, when it is detected that the processing process is abnormal, trigger the monitoring process to determine unprocessed data in a data group corresponding to the abnormal processing process based on a monitoring-side log file, where the monitoring-side log file includes a synchronized data state and a group identifier of a monitoring side;
a data updating unit 405, configured to update a data state of the unprocessed data in the data source to an unprocessed state through a monitoring process;
a second data allocating unit 406, configured to, when it is detected that the abnormal processing process does not return to normal within a preset time period, allocate the unprocessed data to another processing process for processing, and adjust the group identifier of the unprocessed data to the group identifier of the data group corresponding to the other processing process, where the other processing process is a processing process in the processing process group other than the abnormal processing process.
A data checking unit 407, configured to, in the process of allocating the unprocessed data to another processing process, check a current group identifier of the unprocessed data if the exception type of the exception handling process is a preset exception type and the exception handling process returns to normal.
A termination processing unit 408, configured to terminate the exception handling process to continue to process the unprocessed data if the check fails, and process the unprocessed data by the other handling processes.
In some embodiments, the data processing apparatus further comprises:
a third data allocation unit 409, configured to allocate the unprocessed data to the exception handling process when it is detected that the exception handling process returns to normal within a preset time period; updating the data state of the unprocessed data into a state to be processed;
a data processing unit 410, configured to reprocess the unprocessed data based on the exception handling process and the updated data state of the unprocessed data.
In some embodiments, the first data allocation unit 402 may include:
the marking subunit 4021 is configured to mark each data group to obtain a group identifier corresponding to each data group;
a first determining subunit 4022, configured to determine a target process identifier corresponding to the group identifier from a process identifier set, where the process identifier set includes process identifiers of processing processes in the processing process group;
an allocating subunit 4023, configured to allocate the data group corresponding to the group identifier to the processing process corresponding to the target process identifier.
In some embodiments, the second data allocation unit 406 may further include:
the data adjusting subunit 4061 is configured to update the data state of the unprocessed data to a to-be-processed state, and adjust the group identifier of the unprocessed data to the group identifier of the data group corresponding to the other processing process.
The data processing subunit 4062 is configured to trigger other processing processes to process the unprocessed data according to the data state and the group identifier of the unprocessed data.
In some embodiments, the data determination unit 404 may further include:
the second data determining subunit 4041 is configured to trigger the monitoring process to determine, based on the monitored side record file, the first data in the data group corresponding to the abnormal processing process, whose data state is the processing state, and the second data in the data state which is the to-be-processed state, as unprocessed data.
In some embodiments, the second data allocation unit 406 may further include:
a data comparing subunit 4063, configured to, in the process of allocating the unprocessed data to another processing process, compare the current group identifier of the first data with the group identifier corresponding to the abnormal processing process if the abnormal type of the abnormal processing process is the preset abnormal type and the abnormal processing process is recovered to be normal
The termination processing unit 408 may further include:
a termination processing subunit 4081, configured to terminate the processing of the first data by the exception handling process when the current group identifier of the first data is different from the group identifier corresponding to the exception handling process, and process the unprocessed data by the other handling processes.
A refusal allocation subunit 4082, configured to refuse to allocate the first data to the other processing processes when the current group identifier of the first data is the same as the group identifier corresponding to the abnormal processing process, and trigger the abnormal processing process to continue processing the first data.
In some embodiments, the first data processing unit 403 may further include:
a third determining subunit 4031, configured to determine, from the target data group, target data that needs to be processed currently by the processing process.
An obtaining subunit 4032, configured to obtain data processing expectation information corresponding to the processing process, where the data processing expectation information includes a data group that the processing process expects to process and a data state that the processing process expects to process, and data attribute information includes a current data state of the target data and a data group to which the target data currently belongs.
A verification subunit 4033, configured to verify the target data based on the data processing expected information and the data attribute information.
And the triggering subunit 4034 is configured to trigger the processing process to process the target data when the verification passes.
In a specific implementation, the above units may be implemented as independent entities, or may be combined arbitrarily to be implemented as the same or several entities, and the specific implementation of the above units may refer to the foregoing method embodiments, which are not described herein again.
As can be seen from the above, in the data processing method in the embodiment of the present application, the data dividing unit 401 is adopted to obtain a plurality of data to be processed from the data source, and divide the plurality of data into a plurality of data groups; the first data allocation unit 402 allocates each data group to a corresponding processing process in the processing process group, stores the group identifier of each data group to the local, and updates the data state of the data in the data source to a to-be-processed state; the first data processing unit 403 processes the data in the target data group according to the processing process and the data state of the data in the allocated target data group; synchronizing a processing result of the processing process to a monitoring process for monitoring the processing process of the processing process, wherein the processing result comprises a data state of processed data; when detecting that the processing process is abnormal, the data determining unit 404 triggers the monitoring process to determine unprocessed data in a data group corresponding to the abnormal processing process based on a monitoring side record file, where the monitoring side record file includes a synchronized data state and a group identification of the monitoring side; the data updating unit 405 updates the data state of the unprocessed data in the data source to an unprocessed state through a monitoring process; when detecting that the abnormal processing process does not return to normal within a preset time period, the second data allocation unit 406 allocates the unprocessed data to other processing processes for processing, and adjusts the group identifier of the unprocessed data to the group identifier of the data group corresponding to the other processing processes; in the process of distributing unprocessed data to other processing processes, if the exception type of the exception processing process is a preset exception type and the exception processing process is recovered to normal, the data verification unit 407 verifies the current group identifier of the unprocessed data; if the verification fails, the termination processing unit 408 terminates the exception handling process to continue processing the unprocessed data, and processes the unprocessed data by other processing processes. Data omission in the processing process can be avoided, each piece of data is guaranteed to be processed, and the integrity of the processed data is improved.
Referring to fig. 7, embodiments of the present application provide a server 800, which may include one or more processors 801 of a processing core, one or more memories 802 of a computer-readable storage medium, a Radio Frequency (RF) circuit 803, a power supply 804, an input unit 805, and a display unit 806. Those skilled in the art will appreciate that the server architectures depicted in FIGS. 1a and 3a are not meant to be limiting, and may include more or fewer components than those shown, or some components in combination, or a different arrangement of components. Wherein:
the processor 801 is a control center of the server, connects various parts of the entire server using various interfaces and lines, and performs various functions of the server and processes data by running or executing software programs and/or modules stored in the memory 802 and calling data stored in the memory 802, thereby performing overall monitoring of the server. Alternatively, processor 801 may include one or more processing cores; preferably, the processor 801 may integrate an application processor, which mainly handles operating systems, user interfaces, application programs, etc., and a modem processor, which mainly handles wireless communications. It will be appreciated that the modem processor described above may not be integrated into the processor 801.
The memory 802 may be used to store software programs and modules, and the processor 801 executes various functional applications and data processing by operating the software programs and modules stored in the memory 802.
The RF circuit 803 may be used for receiving and transmitting signals during transmission and reception of information.
The server also includes a power source 804 (e.g., a battery) for powering the various components, which may preferably be logically connected to the processor 801 via a power management system to manage charging, discharging, and power consumption management functions via the power management system.
The server may also include an input unit 805, the input unit 805 may be used to receive input numeric or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.
The server may also include a display unit 806, and the display unit 806 may be used to display information input by or provided to the user and various graphical user interfaces of the server, which may be made up of graphics, text, icons, video, and any combination thereof. Specifically, in this embodiment, the processor 801 in the server loads the executable file corresponding to the process of one or more application programs into the memory 802 according to the following instructions, and the processor 801 runs the application programs stored in the memory 802, thereby implementing various functions as follows:
acquiring a plurality of data to be processed from a data source, dividing the plurality of data into a plurality of data groups, and storing the group identification of each data group to the local; distributing each data group to a corresponding processing process in the processing process groups, and updating the data state of the data in the data source to be a to-be-processed state; processing the data in the target data group according to the processing process and the data state of the data in the distributed target data group; synchronizing the processing result of the processing process to the monitoring process of the monitoring processing process; when the processing progress is detected to be abnormal, triggering the monitoring progress to determine unprocessed data in a data group corresponding to the abnormal processing progress based on the monitoring side record file, wherein the monitoring side record file comprises a synchronized data state and a group identification of the monitoring side; updating the data state of unprocessed data in the data source to be an unprocessed state through the monitoring process; when the abnormal processing process is detected not to be recovered to be normal within the preset time period, distributing the unprocessed data to other processing processes for processing, and adjusting the group identification of the unprocessed data to the group identification of the data group corresponding to the other processing processes; in the process of distributing unprocessed data to other processing processes, if the exception type of the exception processing process is a preset exception type and the exception processing process is recovered to be normal, checking the current group identifier of the unprocessed data; if the verification fails, the abnormal processing process is stopped to continue processing the unprocessed data, and other processing processes process the unprocessed data.
As can be seen from the above, the server in the embodiment of the present application may store the group identifier of each data group to the local while sending the data in the data group to the corresponding processing process for processing, synchronize the processing result of each data group to the monitoring process during the processing process, and if the processing process is abnormal, ensure that the data that has not been processed by the abnormal processing process can be found according to the local and monitoring side record files; updating the data state of the unprocessed data into an unprocessed state to wait for processing through the monitoring process, and distributing the unprocessed data to other processing processes for processing when the abnormal processing process is not recovered within a preset time period, so as to avoid missing processing; in the process of distributing unprocessed data to other processing processes, if the exception type of the exception processing process is a preset exception type and the exception processing process returns to normal, the current group identifier of the unprocessed data is verified, if the verification is passed, the unprocessed data is not distributed to other processing processes, the unprocessed data can be continuously processed by the exception processing process, and if the verification is not passed, the unprocessed data is distributed to other processing processes, the exception processing process does not need to process the unprocessed data, so that repeated processing is avoided, and the processing integrity is improved.
Those skilled in the art will appreciate that all or part of the steps in the methods of the above embodiments may be implemented by associated hardware instructed by a program, which may be stored in a computer-readable storage medium, and the storage medium may include: read Only Memory (ROM), Random Access Memory (RAM), magnetic or optical disks, and the like.
In the foregoing embodiments, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.
The foregoing describes a data processing method, an apparatus, a storage medium, and a server provided in the embodiments of the present application in detail, and a specific example is applied in the present application to explain the principle and the implementation of the present application, and the description of the foregoing embodiments is only used to help understand the technical solutions and the core ideas of the present application; those of ordinary skill in the art will understand that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; such modifications or substitutions do not depart from the spirit and scope of the present disclosure as defined by the appended claims.

Claims (10)

1. A data processing method, comprising:
acquiring a plurality of data to be processed from a data source, and dividing the plurality of data into a plurality of data groups;
distributing each data group to a corresponding processing process in the processing process groups, storing the group identification of each data group to the local, and updating the data state of the data in the data source to be a to-be-processed state;
processing the data in the target data group according to the processing process and the data state of the data in the distributed target data group; synchronizing a processing result of the processing process to a monitoring process for monitoring the processing process of the processing process, wherein the processing result comprises a data state of processed data;
when the processing process is detected to be abnormal, triggering the monitoring process to determine unprocessed data in a data group corresponding to the abnormal processing process based on a monitoring side record file, wherein the monitoring side record file comprises a synchronized data state and a group identification of the monitoring side;
updating the data state of the unprocessed data in the data source to be an unprocessed state through a monitoring process;
when the abnormal processing process is detected not to be recovered to be normal in a preset time period, distributing the unprocessed data to other processing processes for processing, and adjusting the group identifier of the unprocessed data to the group identifier of the data group corresponding to the other processing processes, wherein the other processing processes are processing processes except the abnormal processing process in the processing process group;
in the process of distributing the unprocessed data to other processing processes, if the exception type of the exception processing process is a preset exception type and the exception processing process is recovered to be normal, checking the current group identifier of the unprocessed data;
if the verification fails, the abnormal processing process is stopped to continue processing the unprocessed data, and the other processing processes process the unprocessed data.
2. The data processing method of claim 1, wherein the method further comprises:
when the abnormal processing process is detected to be recovered to be normal in a preset time period, distributing the unprocessed data to the abnormal processing process; updating the data state of the unprocessed data into a state to be processed;
and reprocessing the unprocessed data based on the exception handling process and the updated data state of the unprocessed data.
3. The data processing method according to claim 1, wherein the allocating the unprocessed data to another processing process for processing and adjusting the group identifier of the unprocessed data to the group identifier of the data group corresponding to the other processing process comprises:
updating the data state of the unprocessed data into a state to be processed, and adjusting the group identifier of the unprocessed data into the group identifier of the data group corresponding to the other processing process;
and triggering other processing processes to process the unprocessed data according to the data state and the group identification of the unprocessed data.
4. The data processing method according to claim 3, wherein the triggering the monitor process to determine unprocessed data in the data group corresponding to the exception handling process based on the monitor-side log file comprises:
and triggering the monitoring process to determine first data with a data state being a processing state and second data with a data state being a to-be-processed state in a data group corresponding to the abnormal processing process as unprocessed data based on the monitoring side record file.
5. The data processing method according to claim 4, wherein, in the process of allocating the unprocessed data to the other processing processes, if the exception type of the exception handling process is a preset exception type and the exception handling process returns to normal, checking the current group identifier of the unprocessed data includes:
in the process of distributing the unprocessed data to other processing processes, if the exception type of the exception processing process is a preset exception type and the exception processing process is recovered to be normal, comparing the current group identification of the first data with the group identification corresponding to the exception processing process;
if the check fails, terminating the exception handling process to continue processing the unprocessed data, and processing the unprocessed data by the other handling processes, including:
and when the current group identification of the first data is different from the group identification corresponding to the abnormal processing process, stopping the abnormal processing process from processing the first data, and processing the first data by the other processing processes.
6. The data processing method of claim 5, wherein the method further comprises:
and when the current group identification of the first data is the same as the group identification corresponding to the abnormal processing process, refusing to distribute the first data to other processing processes, and triggering the abnormal processing process to continue processing the first data.
7. The data processing method according to any one of claims 1 to 6, wherein said assigning each data group to a respective process in a group of processes comprises:
carrying out group marking on each data group to obtain a group identifier corresponding to each data group;
determining a target process identifier corresponding to the group identifier from a process identifier set, wherein the process identifier set comprises process identifiers of processing processes in the processing process group;
and distributing the data group corresponding to the group identification to the processing process corresponding to the target process identification.
8. The data processing method according to claim 1, wherein the processing the data in the target data group according to the processing progress and the data state of the data in the allocated target data group comprises:
determining target data which needs to be processed currently in the processing process from a target data group;
acquiring data processing expectation information corresponding to the processing process and data attribute information of the target data, wherein the data processing expectation information comprises a data group which is expected to be processed by the processing process and a data state which is expected to be processed, and the data attribute information comprises a current data state of the target data and a data group to which the target data belongs;
verifying the target data based on the data processing expectation information and the data attribute information;
and when the verification is passed, triggering a processing process to process the target data.
9. A data processing apparatus, comprising:
the data dividing unit is used for acquiring a plurality of data to be processed from a data source and dividing the plurality of data into a plurality of data groups;
the first data allocation unit is used for allocating each data group to a corresponding processing process in the processing process groups, storing the group identification of each data group to the local, and updating the data state of the data in the data source to be a to-be-processed state;
the first data processing unit is used for processing the data in the target data group according to the processing process and the data state of the data in the distributed target data group; synchronizing a processing result of the processing process to a monitoring process for monitoring the processing process of the processing process, wherein the processing result comprises a data state of processed data;
the data determining unit is used for triggering the monitoring process to determine unprocessed data in a data group corresponding to the abnormal processing process based on a monitoring side record file when the processing process is detected to be abnormal, wherein the monitoring side record file comprises a synchronized data state and a group identification of the monitoring side;
the data updating unit is used for updating the data state of the unprocessed data in the data source into an unprocessed state through a monitoring process;
a second data allocation unit, configured to, when it is detected that the abnormal processing process does not return to normal within a preset time period, allocate the unprocessed data to another processing process for processing, and adjust the group identifier of the unprocessed data to the group identifier of the data group corresponding to the other processing process, where the other processing process is a processing process in the processing process group other than the abnormal processing process;
the data verification unit is used for verifying the current group identifier of the unprocessed data if the exception type of the exception handling process is a preset exception type and the exception handling process is recovered to be normal in the process of distributing the unprocessed data to other handling processes;
and the termination processing unit is used for terminating the exception handling process to continue processing the unprocessed data if the verification fails, and processing the unprocessed data by the other processing processes.
10. The data processing apparatus of claim 9, further comprising:
the third data distribution unit is used for distributing the unprocessed data to the abnormal processing process when the abnormal processing process is detected to be recovered to be normal within a preset time period; updating the data state of the unprocessed data into a state to be processed;
and the data processing unit is used for reprocessing the unprocessed data based on the exception handling process and the updated data state of the unprocessed data.
CN201910712832.9A 2019-08-02 2019-08-02 Data processing method and device Active CN111190707B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201910712832.9A CN111190707B (en) 2019-08-02 2019-08-02 Data processing method and device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201910712832.9A CN111190707B (en) 2019-08-02 2019-08-02 Data processing method and device

Publications (2)

Publication Number Publication Date
CN111190707A true CN111190707A (en) 2020-05-22
CN111190707B CN111190707B (en) 2023-04-28

Family

ID=70707168

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201910712832.9A Active CN111190707B (en) 2019-08-02 2019-08-02 Data processing method and device

Country Status (1)

Country Link
CN (1) CN111190707B (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114257573A (en) * 2020-09-22 2022-03-29 中国移动通信集团广东有限公司 Method and device for detecting VOLTE voice function abnormity
CN118134208A (en) * 2024-05-06 2024-06-04 国网山东省电力公司滨州市沾化区供电公司 Power transmission and transformation operation detection data processing method, system, terminal and storage medium

Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH07121419A (en) * 1993-10-22 1995-05-12 Hitachi Ltd Data base restoration management system
JPH0830470A (en) * 1994-07-20 1996-02-02 Chubu Nippon Denki Software Kk Exclusive execution control system
CN101599039A (en) * 2008-06-03 2009-12-09 华为技术有限公司 Abnormality eliminating method and device under the embedded type C language environment
CN103369000A (en) * 2012-03-29 2013-10-23 北京智慧风云科技有限公司 Data transmission method and data transmission system
US8892719B2 (en) * 2007-08-30 2014-11-18 Alpha Technical Corporation Method and apparatus for monitoring network servers
CN105303107A (en) * 2014-06-06 2016-02-03 中兴通讯股份有限公司 Abnormal process detection method and apparatus
CN106713944A (en) * 2016-12-30 2017-05-24 北京奇虎科技有限公司 Method and apparatus for processing streaming data task
US20190129887A1 (en) * 2015-07-27 2019-05-02 Sas Institute Inc. Distributed data storage grouping
US20190155539A1 (en) * 2017-11-23 2019-05-23 Beijing Baidu Netcom Science And Technology Co., Ltd. Method and apparatus for processing data based on physical host
US20190179726A1 (en) * 2016-12-08 2019-06-13 Tencent Technology (Shenzhen) Company Limited Monitoring method and apparatus of server, and storage medium

Patent Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH07121419A (en) * 1993-10-22 1995-05-12 Hitachi Ltd Data base restoration management system
JPH0830470A (en) * 1994-07-20 1996-02-02 Chubu Nippon Denki Software Kk Exclusive execution control system
US8892719B2 (en) * 2007-08-30 2014-11-18 Alpha Technical Corporation Method and apparatus for monitoring network servers
CN101599039A (en) * 2008-06-03 2009-12-09 华为技术有限公司 Abnormality eliminating method and device under the embedded type C language environment
CN103369000A (en) * 2012-03-29 2013-10-23 北京智慧风云科技有限公司 Data transmission method and data transmission system
CN105303107A (en) * 2014-06-06 2016-02-03 中兴通讯股份有限公司 Abnormal process detection method and apparatus
US20190129887A1 (en) * 2015-07-27 2019-05-02 Sas Institute Inc. Distributed data storage grouping
US20190179726A1 (en) * 2016-12-08 2019-06-13 Tencent Technology (Shenzhen) Company Limited Monitoring method and apparatus of server, and storage medium
CN106713944A (en) * 2016-12-30 2017-05-24 北京奇虎科技有限公司 Method and apparatus for processing streaming data task
US20190155539A1 (en) * 2017-11-23 2019-05-23 Beijing Baidu Netcom Science And Technology Co., Ltd. Method and apparatus for processing data based on physical host

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114257573A (en) * 2020-09-22 2022-03-29 中国移动通信集团广东有限公司 Method and device for detecting VOLTE voice function abnormity
CN114257573B (en) * 2020-09-22 2023-09-19 中国移动通信集团广东有限公司 Method and device for detecting VOLTE voice function abnormality
CN118134208A (en) * 2024-05-06 2024-06-04 国网山东省电力公司滨州市沾化区供电公司 Power transmission and transformation operation detection data processing method, system, terminal and storage medium
CN118134208B (en) * 2024-05-06 2024-07-23 国网山东省电力公司滨州市沾化区供电公司 Power transmission and transformation operation detection data processing method, system, terminal and storage medium

Also Published As

Publication number Publication date
CN111190707B (en) 2023-04-28

Similar Documents

Publication Publication Date Title
CN113014634B (en) Cluster election processing method, device, equipment and storage medium
KR101871383B1 (en) Method and system for using a recursive event listener on a node in hierarchical data structure
CN107453929B (en) Cluster system self-construction method and device and cluster system
CN110224871A (en) A kind of high availability method and device of Redis cluster
CN112328421B (en) System fault processing method and device, computer equipment and storage medium
CN112671928B (en) Equipment centralized management architecture, load balancing method, electronic equipment and storage medium
CN109802986B (en) Equipment management method, system, device and server
EP3813335B1 (en) Service processing methods and systems based on a consortium blockchain network
CN105069152B (en) data processing method and device
CN111190707B (en) Data processing method and device
CN113553179A (en) Distributed key value storage load balancing method and system
CN112231108A (en) Task processing method and device, computer readable storage medium and server
CN109388481A (en) A kind of transmission method of transaction information, device, calculates equipment and medium at system
WO2023071999A1 (en) User matching method and apparatus, and device and storage medium
CN112650812A (en) Data fragment storage method and device, computer equipment and storage medium
CN110635941A (en) Database node cluster fault migration method and device
CN114265753A (en) Management method and management system of message queue and electronic equipment
CN111181765A (en) Task processing method and device
CN110083653B (en) Order data operation method and device, computer equipment and storage medium
CN110417833B (en) Data processing method and device based on block chain and storage medium
CN114363350A (en) Service management system and method
CN105323271B (en) Cloud computing system and processing method and device thereof
CN107645396B (en) Cluster capacity expansion method and device
CN112631756A (en) Distributed regulation and control method and device applied to space flight measurement and control software
CN104754029B (en) Determine the methods, devices and systems of master management server

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant