CN117667374A - Data processing method, device and system - Google Patents
Data processing method, device and system Download PDFInfo
- Publication number
- CN117667374A CN117667374A CN202211052797.0A CN202211052797A CN117667374A CN 117667374 A CN117667374 A CN 117667374A CN 202211052797 A CN202211052797 A CN 202211052797A CN 117667374 A CN117667374 A CN 117667374A
- Authority
- CN
- China
- Prior art keywords
- data
- node
- processing
- information
- task
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 45
- 238000012545 processing Methods 0.000 claims abstract description 759
- 238000000034 method Methods 0.000 claims abstract description 125
- 230000008569 process Effects 0.000 claims abstract description 59
- 230000002776 aggregation Effects 0.000 claims description 56
- 238000004220 aggregation Methods 0.000 claims description 56
- 238000012546 transfer Methods 0.000 claims description 45
- 230000015654 memory Effects 0.000 claims description 36
- 238000001914 filtration Methods 0.000 claims description 27
- 238000005070 sampling Methods 0.000 claims description 21
- 230000001502 supplementing effect Effects 0.000 claims description 18
- 238000004590 computer program Methods 0.000 claims description 10
- 230000010365 information processing Effects 0.000 claims description 7
- 238000007726 management method Methods 0.000 description 142
- 238000012544 monitoring process Methods 0.000 description 47
- 206010047289 Ventricular extrasystoles Diseases 0.000 description 33
- 238000005129 volume perturbation calorimetry Methods 0.000 description 33
- 238000004891 communication Methods 0.000 description 19
- 230000006870 function Effects 0.000 description 17
- 238000010586 diagram Methods 0.000 description 10
- 230000005540 biological transmission Effects 0.000 description 8
- 238000013075 data extraction Methods 0.000 description 7
- 238000000586 desensitisation Methods 0.000 description 7
- 239000007787 solid Substances 0.000 description 4
- 238000004140 cleaning Methods 0.000 description 3
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 238000006243 chemical reaction Methods 0.000 description 2
- 230000000295 complement effect Effects 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000011068 loading method Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 230000008439 repair process Effects 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 238000012935 Averaging Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000000802 evaporation-induced self-assembly Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 230000037361 pathway Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
Abstract
A data processing method, device and system belong to the technical field of computers. The method comprises the following steps: after acquiring the information of a first acquisition task and the information of a first processing task of the first data, the management node sends the information of the first acquisition task and the information of a first subtask in the first processing task to the first proxy node and sends the information of a second subtask in the first processing task to the processing node; the first proxy node acquires first data according to the information of the first acquisition task, processes the first data according to the information of the first subtask, and sends the processed data to the processing node; and the processing node processes the data sent by the first proxy node according to the information of the second subtask. The method and the device can reduce the data quantity required to be processed by the processing node, and are used for processing the data.
Description
Technical Field
The present disclosure relates to the field of computer technologies, and in particular, to a data processing method, apparatus, and system.
Background
A data processing system generally includes: a plurality of agent nodes and a processing node. The agent node is used for acquiring data according to the information of the pre-configured acquisition task and transmitting the acquired data to the processing node. The processing node is used for processing the data sent by the proxy node according to the information of the pre-configured processing task.
Illustratively, the proxy node is connected to the data source, and the proxy node may collect data on the connected data source according to information of a pre-configured collection task. The proxy node is also connected with the message queue node, and the proxy node can transmit the acquired data to the message queue node so that the data is stored in the message queue by the message queue node. And then the processing node reads the data from the message queue and processes the data sent by the proxy node according to the information of the pre-configured processing task.
However, at present, the processing node needs to process the data collected by the plurality of proxy nodes, so that the processing node needs to process more data, and the load of the processing node is higher.
Disclosure of Invention
The application provides a data processing method, device and system, which can solve the problem of higher load of processing nodes, and the technical scheme is as follows:
in a first aspect, there is provided a data processing method for a data processing system comprising a management node, a first proxy node and a processing node, the method comprising: after acquiring information of a first acquisition task and information of a first processing task of first data, the management node sends the information of the first acquisition task and the information of a first subtask used for processing the first data in the first processing task to the first proxy node; the management node also sends information of a second subtask in the first processing task to the processing node, wherein the first subtask and the second subtask are tasks which are executed in sequence; illustratively, the first subtask includes: at least one of a data filtering task, a data extracting task, a data supplementing task, a data desensitizing task and a first data aggregation task; the second subtask includes: and a second data aggregation task.
After receiving the information of the first acquisition task, the first proxy node can acquire first data according to the information of the first acquisition task; after receiving the information of the first subtask, the first proxy node can process the first data according to the information of the first subtask and send the processed data to the processing node; after receiving the information of the second subtask, the processing node can process the data sent by the first proxy node according to the information of the second subtask.
In the data processing method provided by the application, the management node can allocate a first subtask in a first processing task to be executed to the first proxy node, and allocate a second subtask in the first processing task to the processing node, so that the first proxy node and the processing node execute certain data processing. Therefore, the first proxy node can share the processing which needs to be executed by some processing nodes, so that the load of the processing nodes is reduced, and the processing performance of the processing nodes is ensured. And the first proxy node can execute certain data processing by dividing the first processing task by the management node without modifying the configuration of the first proxy node. When the data processing system comprises a plurality of first proxy nodes, the plurality of first proxy nodes can respectively execute certain data processing without modifying the configuration of the plurality of first proxy nodes. In addition, when the first proxy node processes the first data to obtain data smaller than the data amount of the first data, the network load between the first proxy node and the processing node can be reduced.
Optionally, the data processing system comprises: one or more first proxy nodes. When the data processing system comprises a plurality of first proxy nodes, the plurality of first proxy nodes can share the load of the processing nodes, so that the load of the processing nodes is further reduced. In addition, the number of the first proxy nodes is large, so that the data processing efficiency of the data processing system can be improved.
Optionally, the information of the first acquisition task carries: the data sent by the first proxy node to the processing node carries the target label, and the information of the second subtask also carries the target label; before the processing node processes the data sent by the first proxy node according to the information of the second subtask, the processing node determines the information of the second subtask carrying the target label according to the target label carried by the data sent by the first proxy node. In this way, the first subtask and the second subtask in the first processing task are associated through the target label, so that the first proxy node and the processing node jointly complete the first processing task. When the data processing system comprises a plurality of proxy nodes, the processing nodes receive the data sent by the plurality of proxy nodes, and through the target labels, the processing nodes can find the information of the tasks required to process each data. It may be understood that the information of the first acquisition task and the data sent by the first proxy node to the processing node may not carry the target tag, which is not limited in this application.
The management node transmits the information of the first acquisition task and the information of the first subtask to the first proxy node in various manners. In an optional implementation manner, the information of the first acquisition task carries a type identifier of the first proxy node; the data processing system further comprises a transit node; the management node may send the information of the first acquisition task and the information of the first subtask to the transfer node when sending the information of the first acquisition task and the information of the first subtask for processing the first data in the first processing task to the first proxy node; the transit node can determine the address of the first proxy node corresponding to the type identifier of the first proxy node according to the corresponding relation between the address of each proxy node and the type identifier; the transfer node may further send, to the first proxy node, information of the first acquisition task and information of the first subtask according to the address of the first proxy node.
In another alternative implementation, the data processing system may not include a transit node, and the management node may not send the information of the first acquisition task and the information of the first subtask through the transit node.
The manner in which the first proxy node sends processed data to the processing node is varied. Illustratively, the data processing system further comprises a message queue node; when the first proxy node sends the processed data to the processing node, the first proxy node can send the processed data to the message queue node; then, the message queue node adds the data sent by the first proxy node into a message queue; and the processing node reads the data sent by the first proxy node from the message queue. It will be appreciated that the data processing system may not include a message queue node, and the first proxy node may not send data to the processing node through the message queue node, for example, the first proxy node may send data directly to the processing node, which is not limited in this application.
Optionally, the information of the first subtask carries a first alarm condition, and the first proxy node performs a first alarm when the data obtained by processing according to the information of the first subtask meets the first alarm condition. It can be seen that the first proxy node may not only process the data, but also perform the first alarm when the processed data meets the first alarm condition. The staff can accurately locate the data with problems as the first data processed by the first proxy node according to the first alarm, so that the node related to the first data can be repaired.
Optionally, the information of the second subtask may also carry a second alarm condition, and when the data obtained by processing according to the information of the second subtask meets the second alarm condition, the processing node performs a second alarm. The staff can analyze the first data with problems according to the second alarm, and repair the node related to the first data.
Optionally, the data processing system may comprise one or more second proxy nodes in addition to the first proxy nodes described above. The method further comprises the steps of: the management node acquires information of a second acquisition task of second data and information of a second processing task, wherein the data processed by the second processing task comprises the second data; the second processing task may be any data processing task, for example, the second processing task may include: invoking a chain error sampling task (a sampling task of an erroneous call chain); the management node sends the information of the second acquisition task to the second proxy node; the management node sends the information of the second processing task to the processing node; the second agent node acquires the second data according to the information of the second acquisition task and sends the second data to the processing node; and the processing node processes the second data sent by the second proxy node according to the information of the second processing task. It can be seen that the second proxy node is configured to collect the second data, but is not configured to process the second data, and the processing task of the second data is completed by the processing node.
In a second aspect, there is provided a data processing system comprising: the system comprises a management node, a first proxy node and a processing node.
The management node is used for: acquiring information of a first acquisition task of first data and information of a first processing task, wherein the data used for processing by the first processing task comprises the first data; transmitting information of the first acquisition task and information of a first subtask used for processing the first data in the first processing task to the first proxy node; and sending information of a second subtask in the first processing task to a processing node, wherein the first subtask and the second subtask are tasks which are executed sequentially; illustratively, the first subtask includes: at least one of a data filtering task, a data extracting task, a data supplementing task, a data desensitizing task and a first data aggregation task; the second subtask includes: and a second data aggregation task.
The first proxy node is configured to collect the first data according to the information of the first collecting task, process the first data according to the information of the first subtask, and send the processed data to the processing node; and the processing node is used for processing the data sent by the first proxy node according to the information of the second subtask.
Optionally, the data processing system comprises: one or more first proxy nodes.
Optionally, the information of the first acquisition task carries: the data sent by the first proxy node to the processing node carries the target label, and the information of the second subtask also carries the target label; the processing node is further configured to determine, according to the target tag carried by the data sent by the first proxy node, information of the second subtask carrying the target tag.
The management node transmits the information of the first acquisition task and the information of the first subtask to the first proxy node in various manners. In an optional implementation manner, the information of the first acquisition task carries a type identifier of the first proxy node; the data processing system further comprises: a transit node; the management node is used for sending the information of the first acquisition task and the information of the first subtask to the transfer node; the transfer node is used for determining the address of the first proxy node corresponding to the type identifier of the first proxy node according to the corresponding relation between the address of each proxy node and the type identifier; the transfer node is further configured to send, to the first proxy node, information of the first acquisition task and information of the first subtask according to the address of the first proxy node.
In another alternative implementation, the data processing system may also not include a transit node, where the management node may be configured to send the information of the first acquisition task and the information of the first subtask to the first proxy node without passing through the transit node.
The manner in which the first proxy node sends processed data to the processing node is varied. Illustratively, the data processing system further comprises: a message queue node; the first proxy node is used for sending the processed data to the message queue node; the message queue node is used for adding the data sent by the first proxy node into a message queue; the processing node is used for reading the data sent by the first proxy node from the message queue.
Optionally, the information of the first subtask carries a first alarm condition; the first proxy node is further configured to perform a first alarm when the data obtained by processing the information of the first subtask meets the first alarm condition.
Optionally, the information of the second subtask carries a second alarm condition; and the processing node is also used for carrying out a second alarm when the data obtained by the information processing of the second subtask meets the second alarm condition.
Optionally, the data processing system may comprise one or more second proxy nodes in addition to the first proxy nodes described above. The management node is further configured to obtain information of a second acquisition task of second data and information of a second processing task, where the data processed by the second processing task includes the second data; the second processing task may be any data processing task, for example, the second processing task may include: invoking a chain error sampling task (a sampling task of an erroneous call chain); the management node is further used for sending the information of the second acquisition task to the second proxy node, and the management node is further used for sending the information of the second processing task to the processing node; the second agent node is used for acquiring the second data according to the information of the second acquisition task and sending the second data to the processing node; and the processing node is also used for processing the second data sent by the second proxy node according to the information of the second processing task.
In a third aspect, a data processing method is provided, the method being performed by a management node in a data processing system, the data processing system further comprising: a first proxy node and a processing node, the method comprising: the method comprises the steps that a management node obtains information of a first acquisition task of first data and information of a first processing task, wherein the data used for processing by the first processing task comprises the first data; then, the management node sends the information of the first acquisition task and the information of a first subtask used for processing the first data in the first processing task to the first proxy node; and the management node sends information of a second subtask in the first processing task to the processing node, wherein the first subtask and the second subtask are tasks which are executed in sequence. Illustratively, the first subtask includes: at least one of a data filtering task, a data extracting task, a data supplementing task, a data desensitizing task and a first data aggregation task; the second subtask includes: and a second data aggregation task.
Optionally, the data processing system comprises: one or more first proxy nodes.
Optionally, the information of the first acquisition task carries: and the information of the second subtask also carries the target label.
The management node transmits the information of the first acquisition task and the information of the first subtask to the first proxy node in various manners. In an optional implementation manner, the information of the first acquisition task carries a type identifier of the first proxy node; when the management node sends the information of the first acquisition task and the information of a first subtask used for processing the first data in the first processing task to the first proxy node, the management node can send the information of the first acquisition task and the information of the first subtask to the transfer node; the transit node is used for: determining the address of the first proxy node corresponding to the type identifier of the first proxy node according to the corresponding relation between the address of each proxy node and the type identifier; and sending the information of the first acquisition task and the information of the first subtask to the first proxy node according to the address of the first proxy node.
Optionally, the information of the first subtask carries a first alarm condition.
Optionally, the information of the second subtask carries a second alarm condition.
Optionally, the data processing system may comprise one or more second proxy nodes in addition to the first proxy nodes described above. The method further comprises the steps of: the management node acquires information of a second acquisition task of second data and information of a second processing task, wherein the data processed by the second processing task comprises the second data; the second processing task may be any data processing task, for example, the second processing task may include: invoking a chain error sampling task (a sampling task of an erroneous call chain); and then, the management node sends the information of the second acquisition task to the second proxy node and sends the information of the second processing task to the processing node.
In a fourth aspect, a data processing apparatus is provided, the data processing apparatus belonging to a management node in a data processing system, the data processing system further comprising: the data processing device comprises a first proxy node and a processing node, and the data processing device comprises: the device comprises a first acquisition module, a first sending module and a second sending module. The first acquisition module is used for acquiring information of a first acquisition task of first data and information of a first processing task, and the data which is used for processing by the first processing task comprises the first data; the first sending module is used for sending the information of the first acquisition task and the information of a first subtask used for processing the first data in the first processing task to the first proxy node; and the second sending module is used for sending information of a second subtask in the first processing task to the processing node, wherein the first subtask and the second subtask are tasks which are executed in sequence. Illustratively, the first subtask includes: at least one of a data filtering task, a data extracting task, a data supplementing task, a data desensitizing task and a first data aggregation task; the second subtask includes: and a second data aggregation task.
Optionally, the data processing system comprises: one or more first proxy nodes.
Optionally, the information of the first acquisition task carries: and the information of the second subtask also carries the target label.
The management node transmits the information of the first acquisition task and the information of the first subtask to the first proxy node in various manners. In an optional implementation manner, the data processing system further includes a transit node, and the information of the first acquisition task carries a type identifier of the first proxy node; the first sending module is used for: transmitting the information of the first acquisition task and the information of the first subtask to the transfer node; the transit node is used for: determining the address of the first proxy node corresponding to the type identifier of the first proxy node according to the corresponding relation between the address of each proxy node and the type identifier; and sending the information of the first acquisition task and the information of the first subtask to the first proxy node according to the address of the first proxy node.
Optionally, the information of the first subtask carries a first alarm condition.
Optionally, the information of the second subtask carries a second alarm condition.
Optionally, the data processing system may comprise one or more second proxy nodes in addition to the first proxy nodes described above. The data processing apparatus further includes: the device comprises a second acquisition module and a third sending module. The second acquisition module is used for acquiring information of a second acquisition task of second data and information of a second processing task, and the data processed by the second processing task comprises the second data; the second processing task may be any data processing task, for example, the second processing task may include: invoking a chain error sampling task (a sampling task of an erroneous call chain); and the third sending module is used for sending the information of the second acquisition task to the second proxy node and sending the information of the second processing task to the processing node.
In a fifth aspect, there is provided a data processing method performed by a first proxy node in a data processing system, the data processing system further comprising a management node and a processing node, the method comprising: the method comprises the steps that a first proxy node receives information of a first acquisition task of first data sent by a management node and information of a first subtask used for processing the first data in a first processing task; the first processing task further comprises a second subtask, and the first subtask and the second subtask are tasks which are executed sequentially; illustratively, the first subtask includes: at least one of a data filtering task, a data extracting task, a data supplementing task, a data desensitizing task and a first data aggregation task; the second subtask includes: and a second data aggregation task. Then, the first proxy node acquires the first data according to the information of the first acquisition task; and processing the first data according to the information of the first subtask, and sending the processed data to the processing node.
Optionally, the data processing system comprises: one or more of the first proxy nodes.
Optionally, the information of the first acquisition task carries: and the target label is carried by the data sent to the processing node by the first proxy node.
The management node transmits the information of the first acquisition task and the information of the first subtask to the first proxy node in various manners. In an optional implementation manner, the information of the first acquisition task carries a type identifier of the first proxy node; the data processing system further comprises a transit node; when receiving information of a first acquisition task of first data sent by the management node and information of a first subtask used for processing the first data in a first processing task, a first proxy node can receive an address of the relay node according to the first proxy node and send the information of the first acquisition task and the information of the first subtask to the first proxy node; the management node is used for sending the information of the first acquisition task and the information of the first subtask to the transfer node; the transfer node is used for determining the address of the first proxy node corresponding to the type identifier of the first proxy node according to the corresponding relation between the address of each proxy node and the type identifier.
The manner in which the first proxy node sends processed data to the processing node is varied. Illustratively, the data processing system further comprises a message queue node; when the first proxy node sends the processed data to the processing node, the first proxy node can send the processed data to the message queue node; the message queue node is used for adding the data sent by the first proxy node into a message queue; the processing node is used for reading the data sent by the first proxy node from the message queue.
Optionally, the information of the first subtask carries a first alarm condition, and the first proxy node performs a first alarm when the data obtained by processing according to the information of the first subtask meets the first alarm condition.
In a sixth aspect, there is provided a data processing apparatus belonging to a first proxy node in a data processing system, the data processing system further comprising a management node and a processing node, the data processing apparatus comprising: the device comprises a receiving module, an acquisition module, a processing module and a sending module.
The receiving module is used for receiving information of a first acquisition task of first data sent by the management node and information of a first subtask used for processing the first data in a first processing task, the data processed by the first processing task comprises the first data, the first processing task further comprises a second subtask, and the first subtask and the second subtask are sequentially executed tasks; illustratively, the first subtask includes: at least one of a data filtering task, a data extracting task, a data supplementing task, a data desensitizing task and a first data aggregation task; the second subtask includes: and a second data aggregation task.
The acquisition module is used for acquiring the first data according to the information of the first acquisition task; the processing module is used for processing the first data according to the information of the first subtask; and the sending module is used for sending the processed data to the processing node.
Optionally, the data processing system comprises: one or more of the first proxy nodes.
Optionally, the information of the first acquisition task carries: and the target label is carried by the data sent to the processing node by the first proxy node.
The management node transmits the information of the first acquisition task and the information of the first subtask to the first proxy node in various manners. In an optional implementation manner, the information of the first acquisition task carries a type identifier of the first proxy node; the data processing system further comprises a transit node; the receiving module is used for: receiving the address of the transfer node according to the first proxy node, and sending the information of the first acquisition task and the information of the first subtask to the first proxy node; the management node is used for sending the information of the first acquisition task and the information of the first subtask to the transfer node; the transfer node is used for determining the address of the first proxy node corresponding to the type identifier of the first proxy node according to the corresponding relation between the address of each proxy node and the type identifier.
The manner in which the first proxy node sends processed data to the processing node is varied. Illustratively, the data processing system further comprises a message queue node; the sending module is used for: transmitting the processed data to the message queue node; the message queue node is used for adding the data sent by the first proxy node into a message queue; the processing node is used for reading the data sent by the first proxy node from the message queue.
Optionally, the information of the first subtask carries a first alarm condition, and the data processing apparatus further includes: and the alarm module is used for carrying out first alarm when the data obtained by the processing module according to the information processing of the first subtask meets the first alarm condition.
In a seventh aspect, a data relay method is provided, the method being performed by a relay node in a data processing system, the data processing system further comprising: the method comprises the steps of: the transfer node receives information of a first acquisition task of first data sent by the management node and information of a first subtask used for processing the first data in a first processing task; the information of the first acquisition task carries a type identifier of the first proxy node; the first processing task further comprises a second subtask, and the first subtask and the second subtask are tasks which are executed sequentially; for example, the first subtask includes: at least one of a data filtering task, a data extracting task, a data supplementing task, a data desensitizing task and a first data aggregation task; the second subtask includes: and a second data aggregation task. Then, the transit node determines the address of the first proxy node corresponding to the type identifier of the first proxy node according to the corresponding relation between the address of each proxy node and the type identifier; and sending the information of the first acquisition task and the information of the first subtask to the first proxy node according to the address of the first proxy node.
Optionally, the data processing system comprises: one or more of the first proxy nodes.
Optionally, the information of the first acquisition task carries: target tags.
Optionally, the information of the first subtask carries a first alarm condition.
An eighth aspect provides a data transfer device, the data transfer device belonging to a transfer node in a data processing system, the data processing system further comprising: the data transfer device comprises a management node, a first proxy node and a processing node, and comprises: the device comprises a receiving module, a determining module and a sending module.
The receiving module is used for receiving information of a first acquisition task of the first data sent by the management node and information of a first subtask used for processing the first data in a first processing task; the information of the first acquisition task carries a type identifier of the first proxy node; the first processing task further comprises a second subtask, and the first subtask and the second subtask are tasks which are executed sequentially; for example, the first subtask includes: at least one of a data filtering task, a data extracting task, a data supplementing task, a data desensitizing task and a first data aggregation task; the second subtask includes: and a second data aggregation task. The determining module is used for determining the address of the first proxy node corresponding to the type identifier of the first proxy node according to the corresponding relation between the address of each proxy node and the type identifier; and the sending module is used for sending the information of the first acquisition task and the information of the first subtask to the first proxy node according to the address of the first proxy node.
Optionally, the data processing system comprises: one or more of the first proxy nodes.
Optionally, the information of the first acquisition task carries: target tags.
Optionally, the information of the first subtask carries a first alarm condition.
In a ninth aspect, a cluster of computing devices is provided, comprising at least one computing device, each computing device comprising a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method of any one of the designs of the first aspect.
In a tenth aspect, a cluster of computing devices is provided, comprising at least one computing device, each computing device comprising a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method of any one of the designs of the third aspect, the fifth aspect or the seventh aspect.
In an eleventh aspect, there is provided a computer readable storage medium comprising computer program instructions which, when executed by a cluster of computing devices, perform the method of any of the designs of the first aspect.
In a twelfth aspect, there is provided a computer readable storage medium comprising computer program instructions which, when executed by a cluster of computing devices, perform the method of any one of the designs of the third, fifth or seventh aspects.
In a thirteenth aspect, there is provided a computer program product comprising instructions which, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method of any of the designs of the first aspect.
In a fourteenth aspect, there is provided a computer program product comprising instructions which, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method of any one of the designs of the third, fifth or seventh aspects.
The effects of the second aspect to the fourteenth aspect may refer to the effects of the corresponding aspects of the first aspect, and embodiments of the present application are not described herein.
Drawings
FIG. 1 is a schematic diagram of a data processing system according to an embodiment of the present application;
FIG. 2 is a schematic diagram of another data processing system according to an embodiment of the present application;
FIG. 3 is a flowchart of a data processing method according to an embodiment of the present application;
FIG. 4 is a schematic diagram of a further data processing system according to an embodiment of the present application;
fig. 5 is an application scenario schematic diagram of a data processing method according to an embodiment of the present application;
FIG. 6 is a schematic diagram of a data processing apparatus according to an embodiment of the present application;
FIG. 7 is a schematic diagram of another data processing apparatus according to an embodiment of the present disclosure;
fig. 8 is a schematic structural diagram of a data transfer device according to an embodiment of the present application;
FIG. 9 is a schematic diagram of a computing device provided in an embodiment of the present application;
FIG. 10 is a schematic diagram of a computing device cluster provided in an embodiment of the present application;
fig. 11 is a schematic diagram of another computing device cluster provided in an embodiment of the present application.
Detailed Description
In order to make the principles and technical solutions of the present application more apparent, the embodiments of the present application will be described in further detail below with reference to the accompanying drawings.
An embodiment of the present application provides a data processing system, as shown in fig. 1, including: proxy node 01 and processing node 02. The data processing system may comprise one or more proxy nodes 01, only one proxy node 01 being shown in fig. 1. The nodes in the data processing system, such as proxy node 01 and processing node 02, may be servers or clusters of servers.
The proxy node 01 is connected to a data source (not shown in fig. 1), which may or may not belong to the data processing system, which is not limited in this embodiment of the present application. The agent node 01 may be preconfigured with information of a collection task of data on the connected data source, and the agent node 01 may collect the data on the data source according to the information of the collection task.
Proxy node 01 may also transmit the collected data to processing node 02. Illustratively, proxy node 01 may be connected to processing node 02 through a message queue node (not shown in fig. 1). The agent node 01 may transmit the collected data to the message queue node, so that the data is stored in the message queue (e.g. Kafka (an open source software) message queue) by the message queue node, and then the processing node 02 reads the data from the message queue, so that the agent node 01 transmits the collected data to the processing node 02.
The processing node 02 may be preconfigured with information of a processing task for the data collected by the proxy node 01, and the processing node 02 is configured to execute the processing task according to the information of the processing task, so as to process the data sent by the proxy node 03. The processing may include, for example, a first stage of processing and a second stage of processing. Wherein the first stage of processing may include: data cleansing (such as data filtering, data extraction, data shuffling and data desensitization), data aggregation, and the like. The second stage of processing may include: the processing node 02 displays the processing result of the first stage to the user; alternatively, processing node 02 may also be coupled to a database node (not shown in FIG. 1), and the second stage of processing may include: the processing node 02 stores the processing result of the first stage to a database node; alternatively, the second stage of processing includes: processing node 02 transmits the processing results of the first stage to other nodes (not shown in fig. 1) for use by the other nodes. When the processing node 02 transmits the processing result of the first stage to other nodes, the processing result of the first stage may be transmitted to another message queue node, so that the processing result of the first stage is stored in the message queue by the message queue node, and then the other nodes read the processing result of the first stage from the message queue. The processing in the second stage includes: processing node 02 stores the results of the first stage of processing to a database node, or processing node 02 transmits the results of the first stage of processing to other nodes, the second stage of processing may be referred to as data loading. In this case, if the processing of the first stage includes data extraction and data conversion (such as data patching, data desensitization), the processing of the data by the processing node 01 may be referred to as extraction, conversion and loading (ETL) processing. The database node may be an elastic search (elastic search) node.
The processing node 02 may be a Real-time computing node, and the duration of processing data by the processing node may be in milliseconds or microseconds. When the processing node 02 is a real-time computing node, the processing node 02 may be run with a real-time computing engine, such as an apache link, an apache store, an apache spark, etc., where the apache link, the apache store, and the apache spark are three open-source real-time computing engines.
The above describes the process by which a proxy node 01 collects data and the data is processed in a processing node 02. It will be appreciated that where the data processing system comprises a plurality of proxy nodes 01, the process by which each proxy node 01 of the plurality of proxy nodes 01 collects data and the collected data is processed in the processing node 02 may refer to the process described above. The data collected by the plurality of proxy nodes 01 may be the same or different, and the data sources where the data collected by the plurality of proxy nodes 01 are located may be the same or different.
From the foregoing, it can be seen that the processing node 02 needs to process the data collected by the plurality of proxy nodes 01, and therefore, the processing node 02 needs to execute more processing tasks, resulting in a higher load of the processing node 02. Moreover, the processing resources of the processing node 02 are limited, and when the load of the processing node 02 is high, the processing performance of the processing node 02 cannot be ensured, so that the instantaneity of data processing is affected. In addition, the amount of data transferred from the plurality of proxy nodes 01 to the processing node 02 is large, resulting in a high load on the network between the proxy nodes 01 and the processing node 02, for example, a high load on the message queue nodes between the proxy nodes 01 and the processing node 02 in fig. 1.
The embodiment of the application provides another data processing system, which can reduce processing tasks required to be executed by processing nodes, reduce the load of the processing nodes, ensure the processing performance of the processing nodes and reduce the network load between proxy nodes and the processing nodes. Such a data processing system may be as shown in fig. 2, which data processing system further comprises a management node 11 on the basis of fig. 1, the management node 11 being connected to both the agent node 01 and the processing node 02. The management node 11 is used for managing the proxy node 01 and the processing node 02. The function of the data processing system will be further described below in connection with the data processing method provided in the embodiments of the present application.
Fig. 3 is a flowchart of a data processing method according to an embodiment of the present application, where the data processing method may be used in any data processing system including a management node according to an embodiment of the present application. It should be noted that the data processing system may include one or more first proxy nodes, and in this embodiment of the present application, the data processing method provided in this embodiment of the present application is described by taking one first proxy node as an example. When the data processing system includes a plurality of first proxy nodes, each of the data processing methods associated with the first proxy nodes may refer to the data processing method associated with the one first proxy node.
As shown in fig. 3, the data processing method includes:
s101, a management node acquires information of a first acquisition task of first data and information of a first processing task, wherein the data for processing of the first processing task comprises the first data.
The management node obtains the information of the first acquisition task and the information of the first processing task in various modes. Illustratively, the management node may receive information of a first acquisition task and information of a first processing task input by a worker; and/or the management node can display information of various acquisition tasks and information of various processing tasks, and receive information of a first acquisition task and information of a first processing task selected by a worker from the information of the acquisition tasks and the information of the processing tasks; and/or the management node can receive the information of the first acquisition task and the information of the first processing task sent by other nodes; and/or the management node may generate the information of the first acquisition task and the information of the first processing task according to some information.
The first acquisition task is an acquisition task of first data. The first data is data used for collecting by the first proxy node. The first proxy node is various in type and the first data is various in type.
Illustratively, the first proxy node is a proxy node of an application class, such as a java (a computer programming language) application class. In this case, the first proxy node corresponds to an application instance, and is co-process deployed with the application instance, where the first data collected by the first proxy node includes first data of a log class, where the first data includes: during the running process of the application instance, the application instance monitors the log.
Also illustratively, the first proxy node is a log (log) class proxy node. In this case, the first proxy node is deployed in the virtual machine node, and the first data collected by the first proxy node includes first data of a log class, where the first data includes: logs in the virtual machine node, such as monitoring logs of applications running on the virtual machine node, and the like.
Further by way of example, the first proxy node is a hypertext transfer protocol (hyper text transfer protocol, http) class proxy node. In this case, the first proxy node is connected to a third party monitoring component, such as zabbix or promethaus, which are two open source components, through an http interface. The third party monitoring component is used for acquiring monitoring data, the first data acquired by the first proxy node comprises http-type first data, and the first data comprises: monitoring data acquired by a third party monitoring component acquired by the first proxy node through an http interface.
In the case where the types of the first data are diverse, information of the first acquisition task of the first data is also diverse. As an example, according to the above three examples, the first data used for collection by the first proxy node of the application class or the log class includes first data of the log class, and the first data used for collection by the first proxy node of the http class includes first data of the http class. Wherein, for the first data of the log class, the information of the first acquisition task of the first data may include: an internet protocol (internet protocol, IP) address (or domain name) of a node on which the first data is generated, a storage path of the first data on the node, a collection time interval of the first data, and the like. For the first data of the http class, the information of the first acquisition task of the first data may include: the address of the http interface for collecting the first data, the http method for collecting the first data, and the format of the message used for collecting the first data.
In this embodiment, the data processing system includes a first proxy node, and the data processed by the first processing task includes first data collected by the first proxy node. Optionally, when the data processing system includes a plurality of first proxy nodes, the data for processing by the first processing task may also include first data for acquisition by the plurality of first proxy nodes.
The first processing tasks may include a first stage of processing tasks and a second stage of processing tasks. Wherein the processing tasks of the first stage may include at least one of a data cleansing task and a data aggregation task. The second stage of processing task is used for processing the processing result obtained according to the first stage of processing task, for example, the second stage of processing task includes: and performing at least one of the tasks of storing, forwarding and displaying the processing result of the first stage.
The data cleaning task is used for cleaning the data to obtain the data with the format meeting the requirements. For example, the data cleansing task may include at least one of a data filtering task, a data extraction task, a data flush task, and a data desensitization task. The data filtering task is used for filtering data meeting the requirements and filtering data not meeting the requirements; the data extraction task is used for extracting a designated field in the data; the data complement task is used for complement the missing field in the data; the data desensitization task is used to desensitize sensitive fields in data, for example, replace sensitive fields with "x".
The data aggregation task is used for calculating data to obtain characteristics of the data, such as: at least one characteristic selected from the group consisting of number, maximum value (max), minimum value (min), average value (avg), error rate (err_rate), sum (sum), and quantile value. The data aggregation task for calculating the number of data is also called a count task, the data aggregation task for calculating the maximum value of the data is also called a maximum value task, the data aggregation task for calculating the minimum value of the data is also called a minimum value task, the data aggregation task for calculating the average value of the data is also called an average value task, the data aggregation task for calculating the error rate of the data is also called an error rate task, the data aggregation task for calculating the sum of the data is also called a summation task, and the data aggregation task for calculating the quantile value of the data is also called a quantile value task.
In the case that the management node receives the information of the first acquisition task and the information of the first processing task input by the staff, the staff can define the data cleaning task through a regular expression or a grovy (a mature object-oriented programming language) script, and define the data aggregation task through checking.
S102, the management node sends information of a first acquisition task and information of a first subtask used for processing the first data in a first processing task to a first proxy node.
After acquiring the information of the first processing task, the management node can identify a first subtask and a second subtask which are sequentially executed in the first processing task, and acquire the information of the first subtask and the information of the second subtask. The first subtask is a task for processing first data, and the second subtask is a task for processing data obtained through information processing according to the first subtask. The first processing task comprises a first subtask and a second subtask which are sequentially executed; processing the first data according to the information of the first processing task includes: first, first data are processed according to the information of the first subtask, and then data obtained according to the information processing of the first subtask are processed according to the information of the second subtask.
Optionally, the first subtask includes: at least one of a data filtering task, a data extracting task, a data supplementing task, a data desensitizing task and a first data aggregation task; the second subtask includes: and a second data aggregation task. The data characteristics of the first data aggregation task and the second data aggregation task, which are used for calculation, can be the same or different, and the data aimed by the first data aggregation task and the second data aggregation task are different. By way of example, assume that the first processing task includes: the data aggregation system comprises a data filtering task, a data extracting task, a data supplementing task, a data desensitizing task and a target data aggregation task which are sequentially executed, wherein the target data aggregation task comprises a first data aggregation task and a second data aggregation task which are sequentially executed. Then, the management node may determine that the first subtask includes: the data filtering task, the data extracting task, the data supplementing task, the data desensitizing task and the first data aggregation task, and the second subtask comprises: and a second data aggregation task.
It will be appreciated that the second sub-task may also include the processing tasks of the second stage described above.
The management node may send the information of the first acquisition task and the information of the first subtask to the first proxy node at the same time, for example, the information of the first acquisition task and the information of the first subtask are carried in the same information and are sent to the first proxy node. The management node may also sequentially send the information of the first acquisition task and the information of the first subtask to the first proxy node. In this embodiment, taking an example that the management node sends the information of the first acquisition task and the information of the first subtask to the first proxy node at the same time.
The management node transmits the information of the first acquisition task and the information of the first subtask to the first proxy node in various manners.
In an alternative implementation, as shown in fig. 4, the data processing system further comprises a transit node 12 on the basis of the data processing system shown in fig. 2, and the management node 11 is connected to the proxy node 01 through the transit node 12. The information of the first acquisition task may carry a type identifier of the first proxy node, and the management node 11 may send the information of the first acquisition task and the information of the first subtask to the transit node 12. After receiving the information of the first acquisition task and the information of the first subtask, the transit node 12 can extract the type identifier of the first proxy node carried in the information of the first acquisition task; then, the transit node 12 may determine, according to the correspondence between the address of each proxy node 01 and the type identifier in the data processing system, the address of the first proxy node corresponding to the type identifier of the first proxy node; finally, the transit node 12 may send the information of the first acquisition task and the information of the first subtask to the first proxy node according to the determined address of the first proxy node.
By way of example, assume that a data processing system includes three proxy nodes 01A, 01B, and 01C, whose addresses and type identifiers correspond as shown in Table 1. If the type identifier of the first proxy node is X, the transit node 12 may determine that the proxy nodes 01A and 01B are both first proxy nodes according to the type identifier X, send the information of the first acquisition task and the information of the first subtask to the proxy node 01A according to the address 1, and send the information of the first acquisition task and the information of the first subtask to the proxy node 01B according to the address 2. If the type identifier of the first proxy node is Y, the transit node 12 may determine that the proxy node 01C is the first proxy node according to the type identifier Y, and send the information of the first acquisition task and the information of the first subtask to the proxy node 01C according to the address 3.
TABLE 1
| Proxy node | Proxy node address | Type identification of proxy node |
| 01A | Address 1 | X |
| 01B | Address 2 | Y |
| 01C | Address 3 | Y |
Optionally, the transit node may also determine the proxy node according to the type identifier and other information. For example, when the type identifier of the first proxy node 01 is an identifier of an application class, the information of the first acquisition task may also carry an application name of an application instance corresponding to the first proxy node 01. The transfer node determines a group of proxy nodes with type identifiers carried by the information of the first acquisition task according to the type identifiers of all proxy nodes 01 in the data processing system; then, the transit node searches the first proxy node of which the application name of the corresponding application instance is the application name carried by the information of the first acquisition task in the proxy nodes; and finally, sending the information of the first acquisition task and the information of the first subtask to the first proxy node according to the address of the first proxy node.
In another alternative implementation, the data processing system may not include a transit node, where the management node may not send the information of the first acquisition task and the information of the first subtask to the first proxy node through the transit node in S102.
Alternatively, the transit node may be referred to as a manager of the proxy node, and the transit node may be responsible for managing a lifecycle of each proxy node in the data processing system, and managing operation information of each proxy node.
S103, the management node sends information of a second subtask in the first processing task to the processing node, wherein the first subtask and the second subtask are sequentially executed tasks.
The management node is connected with the processing node, and after determining the information of the second subtask, the management node can send the information of the second subtask to the processing node through the connection.
S104, the first proxy node collects first data according to the information of the first collection task.
After receiving the information of the first acquisition task, the first proxy node may execute the first acquisition task according to the information of the first acquisition task to acquire the first data.
For example, for first data of the log class, the information of the first acquisition task of the first data may include: the IP address (or domain name) of the node generating the first data, the storage path of the first data on the node, the acquisition time interval of the first data, and the like. The first proxy node may periodically collect the first data at the node generating the first data according to the IP address (or domain name) and the storage path of the first data at the collection time interval.
For another example, for the first data of the http class, the information of the first acquisition task of the first data may include: the address of the http interface for collecting the first data, the http method for collecting the first data, and the format of the message used for collecting the first data. The first proxy node may adopt an http method for collecting the first data, and send the request message in the format to the third party monitoring component through the http interface according to the address of the http interface, so that the third party monitoring component sends the first data to the first proxy node according to the request message.
S105, the first proxy node processes the first data according to the information of the first subtask.
After the first proxy node collects the first data, the first proxy node can execute the first subtask according to the information of the first subtask sent by the management node so as to process the first data.
S106, the first proxy node sends the processed data to the processing node.
After the first proxy node processes the first data according to the information of the first subtask, the data obtained by processing can be sent to the processing node, so that the processing node can further process the data in a subsequent step.
Optionally, the information of the first acquisition task sent by the management node to the first proxy node carries a target tag, and when the first proxy node sends the processed data to the processing node, the first proxy node may carry the target tag in the data. It may be understood that the information of the first acquisition task and the data sent by the first proxy node to the processing node may also not carry the target tag, which is not limited in the embodiment of the present application.
The manner in which the first proxy node sends processed data to the processing node is varied. Illustratively, as shown in FIG. 4, the data processing system further comprises a message queue node 13 on the basis of the data processing system shown in FIG. 2. Proxy node 01 may be connected to processing node 02 through message queue node 13. The first proxy node (e.g., proxy node 01 in fig. 4) may send the processed data to message queue node 13; the message queue node 13 may add the data sent by the first proxy node to the message queue after receiving the data; thereafter, processing node 02 may read the data sent by the first proxy node from the message queue (or message queue node 13 may send the data sent by the first proxy node in the message queue to processing node 02). It will be appreciated that the data processing system may not include a message queue node, and the first proxy node may not send data to the processing node through the message queue node, for example, the first proxy node may send data directly to the processing node, which is not limited in the embodiment of the present application.
Optionally, the information of the first subtask sent by the management node to the first proxy node further carries information for indicating the reporting time, and the first proxy node may determine the reporting time according to the information and send the processed data to the processing node at the reporting time. It may be appreciated that the information of the first subtask may not carry information for indicating the reporting time, in which case the first proxy node may determine the reporting time in other manners, for example, the first proxy node may receive the reporting time configured by the staff member, or the first proxy node may determine the reporting time randomly, or the like.
And S107, the processing node processes the data sent by the first proxy node according to the information of the second subtask.
After the processing node obtains the data processed by the first proxy node, the processing node can execute the second subtask according to the information of the second subtask so as to further process the data.
Optionally, the information of the first acquisition task sent by the management node to the first proxy node carries a target tag, and the information of the second subtask sent by the management node to the processing node also carries the target tag. When the first proxy node sends the processed data to the processing node, the target label can be carried in the data. The processing node can find the information of the second subtask carrying the target label according to the target label, and then execute the second subtask according to the information of the second subtask so as to process the data.
In this way, the first subtask and the second subtask in the first processing task are associated through the target label, so that the first proxy node and the processing node jointly complete the first processing task. It will be appreciated that when the data processing system includes a plurality of proxy nodes, the processing node may receive data sent by the plurality of proxy nodes, and through the target tag, the processing node may find information about tasks on which each data needs to be processed.
It may be understood that the information of the first acquisition task sent by the management node to the first proxy node may not carry the target tag, and the information of the second subtask sent by the management node to the processing node may not carry the target tag.
Optionally, when the processing node needs to execute the plurality of second subtasks, the processing node may start the plurality of processes to execute the plurality of second subtasks in parallel, so as to improve the efficiency of executing the second subtasks by the processing node.
In summary, in the data processing method provided in the embodiment of the present application, the management node may allocate a first subtask in a first processing task to be executed to the first proxy node, and allocate a second subtask in the first processing task to the processing node, so that both the first proxy node and the processing node execute certain data processing. Therefore, the first proxy node can share the processing which needs to be executed by some processing nodes, so that the load of the processing nodes is reduced, and the processing performance of the processing nodes is ensured.
In the data processing method provided by the embodiment of the invention, the first proxy node can execute certain data processing by dividing the first processing task by the management node without modifying the configuration of the first proxy node. When the data processing system comprises a plurality of first proxy nodes, the plurality of first proxy nodes can respectively execute certain data processing without modifying the configuration of the plurality of first proxy nodes.
In addition, when the first proxy node processes the first data to obtain data smaller than the data amount of the first data, the network load between the first proxy node and the processing node can be reduced.
When the data processing system comprises a plurality of first proxy nodes, the plurality of first proxy nodes can share the load of the processing nodes, so that the load of the processing nodes is further reduced. In addition, the number of the first proxy nodes is large, so that the data processing efficiency of the data processing system can be improved. By way of example, when the number of the first proxy nodes is more than 10 times of the number of the processing nodes, the computing pressure of the processing nodes can be effectively dispersed, the data volume transmitted between the first proxy nodes and the processing tasks required to be executed by the processing nodes are remarkably compressed, and the processing performance of the processing nodes is improved by more than 5 times.
In the above embodiments, the data processing system includes one or more first proxy nodes, for example, it may be understood that the data processing system may include one or more second proxy nodes in addition to the first proxy nodes.
When the data processing system includes the second proxy node, the management node may further acquire information of a second acquisition task of the second data, and information of a second processing task, where the data for processing by the second processing task includes the second data. And then, the management node can send the information of the second acquisition task to the second proxy node and send the information of the second processing task to the processing node, so that the second proxy node acquires second data according to the information of the second acquisition task and sends the second data to the processing node, and the processing node processes the second data sent by the second proxy node according to the information of the second processing task. It can be seen that the second proxy node is configured to collect the second data, but is not configured to process the second data, and the processing task of the second data is completed by the processing node.
The process of the management node sending the information of the second acquisition task to the second proxy node may refer to the process of the management node sending the information of the first acquisition task and the information of the first subtask to the first proxy node in S102. The process in which the management node transmits the information of the second processing task to the processing node may refer to the process in which the management node transmits the information of the second subtask to the processing node in S103 described above. The process of the second proxy node collecting the second data according to the information of the second collecting task may refer to the process of the first proxy node collecting the first data according to the information of the first collecting task in S104. The process of sending the second data to the processing node by the second proxy node may refer to the process of sending the processed data to the processing node by the first proxy node in S106. The process of the processing node processing the second data sent by the second proxy node according to the information of the second processing task may refer to the process of the processing node processing the data sent by the first proxy node according to the information of the second subtask. The embodiments of the present application are not described herein.
The second processing task may be any data processing task, which is not limited in this embodiment of the present application. For example, the second processing task may include: the call chain error sampling task (the sampling task of the wrong call chain). It will be appreciated that one application will call a plurality of other applications in sequence during execution, the call chain indicating the order in which the plurality of other applications are called in sequence. And, when each other application is called, a call chain log recording the call process is generated, and the plurality of other applications sequentially generate a plurality of call chain logs. When the second processing task is a call chain error sampling task, the second data includes a call chain log, and the processing node may determine whether the call chain is error according to the plurality of call chain logs. The plurality of call chain logs may be collected by one or more second proxy nodes.
From the foregoing, it will be appreciated that, in the data processing tasks performed by the data processing system, a first processing task is performed by the first proxy node and the processing node together, and a second processing task is performed by the processing node. The management node needs to identify the first processing task and the second processing task, and to identify a first subtask and a second subtask in the first processing task.
Illustratively, as shown in Table 2 below, the data filtering task, the data extraction task, the data flush task, the data desensitization task, and the call chain error sampling task are all data cleansing tasks. The data filtering task, the data extracting task, the data supplementing task and the data desensitizing task can be all executed by the agent node by taking the management node as a first subtask, and the calling chain error sampling task can be executed by the processing node by taking the management node as a second processing task.
TABLE 2
| Task name | Proxy node execution | Processing node execution |
| Data filtering task | Data filtering task | / |
| Data extraction task | Data extraction task | / |
| Data patch task | Data patch task | / |
| Data desensitization task | Data desensitization task | / |
| Invoking a chain error sampling task | / | Invoking a chain error sampling task |
As another example, as shown in table 3 below, the count task, the maximum task, the minimum task, the average task, the error rate task, the sum task, and the quantile task are all data aggregation tasks.
TABLE 3 Table 3
The counting task may be divided by the management node into a first subtask (counting task) performed by the first proxy node and a second subtask (summing task) performed by the processing node. The first proxy node performs a counting task on the collected first data to obtain the number of the collected first data; and the processing nodes sum the processing results sent by all the first proxy nodes to obtain the total number of the first data acquired by all the first proxy nodes in the data processing system.
The maximizing task may be divided by the managing node into a first sub-task (maximizing task) performed by the first proxy node and a second sub-task (maximizing task) performed by the processing node. The first proxy node performs a maximum value task on the collected first data to obtain the maximum value in the collected first data; and the processing node executes a maximum value task on the processing results sent by all the first proxy nodes to obtain the maximum value of the first data acquired by all the first proxy nodes in the data processing system.
The minimum task may be divided by the management node into a first subtask (minimum task) performed by the first proxy node and a second subtask (minimum task) performed by the processing node. The first proxy node performs a minimum value task on the collected first data to obtain a minimum value in the collected first data; and the processing node executes a minimum value task on the processing results sent by all the first proxy nodes to obtain the minimum value of the first data acquired by all the first proxy nodes in the data processing system.
The averaging task may be divided by the management node into a first subtask (summation task + counting task) performed by the first proxy node and a second subtask (first summation task + second summation task + first quotient task) performed by the processing node. The first proxy node performs a summation task on the collected first data to obtain the sum of all the collected first data, and performs a counting task on the collected first data to obtain the number of the collected first data; the processing node executes a first summation task on the sum of the results of the summation tasks sent by all the first proxy nodes to obtain the sum of first data acquired by all the first proxy nodes in the data processing system; the processing node executes a second summation task on the results of the counting tasks sent by all the first proxy nodes to obtain the total number of the first data acquired by all the first proxy nodes in the data processing system; and the processing node executes a first quotient-solving task on the result obtained by the first summation task and the result obtained by the second summation task to obtain an average value of first data acquired by the first proxy node in the data processing system.
The error rate task may be divided by the management node into a first subtask (error count task + total count task) performed by the first proxy node and a second subtask (third summation task + fourth summation task + second quotient task) performed by the processing node. The first proxy node performs an error counting task on the collected first data to obtain the error number of the collected first data, and performs a total counting task on the collected first data to obtain the total number of the collected first data; the processing node executes a third summation task on the results of the error counting tasks sent by all the first proxy nodes to obtain the sum of the error numbers of the first data acquired by all the first proxy nodes in the data processing system; the processing node executes a third summation task on the sum of the results of the total counting tasks sent by all the first proxy nodes to obtain the sum of the total number of the first data acquired by all the first proxy nodes in the data processing system; and the processing node executes a second quotient-solving task on the result obtained by the third summation task and the result obtained by the fourth summation task to obtain the error rate of the first data acquired by the first proxy node in the data processing system.
The summation task may be divided by the management node into a first sub-task (summation task) performed by the first proxy node and a second sub-task (summation task) performed by the processing node. The first proxy node performs a summation task on the collected first data to obtain the sum of the collected first data; and the processing node executes a summation task on the processing results sent by all the first proxy nodes to obtain the sum of the first data acquired by all the first proxy nodes in the data processing system.
The quantisation task may be divided by the management node into a first sub-task (counting tasks in time units) performed by the first proxy node and a second sub-task (quantisation task from counts in time units) performed by the processing node. The method comprises the steps that a first proxy node executes counting tasks in time units to obtain the number of first data acquired in each time unit; the processing node performs a task of calculating a bit value on the processing results sent by all the first proxy nodes, and first obtains the sum of the numbers of the first data collected by all the first proxy nodes in each time unit, and then processes the sum of the numbers in each time unit by adopting a linear interpolation method to obtain a bit value (such as a 90 bit value, a 95 bit value or a 99 bit value) in the sum of the numbers.
Further, the information of the first subtask may carry a first alarm condition, and after the first proxy node processes the first data according to the information of the first subtask, if the processed data meets the first alarm condition, the first proxy node may perform the first alarm. For example, the first subtask may include a first data aggregation task, where the first data aggregation task may be configured to calculate at least one feature of the data, where each feature may correspond to a feature threshold, and where the first alert condition may be determined to be met when the feature calculated by the first proxy node is greater than (or less than) the feature threshold.
The first agent node may be able to make the first alert in a variety of ways.
For example, the data processing system further includes an alarm node, and the first proxy node may generate alarm information and send the alarm information to the message queue node; the message queue node can add the alarm information into another message queue and send the alarm information in the message queue to the alarm node; the alarm node can display the alarm information in a display mode, a voice broadcast mode and/or an indicator lamp prompt mode.
For another example, the first proxy node may send the alert information to the management node (e.g., the first proxy node sends the alert information to the management node through the transit node), so that the management node may display the alert information in a manner of displaying, voice broadcasting, and/or indicator light prompting.
It can be seen that the first proxy node may not only process the data, but also perform the first alarm when the processed data meets the first alarm condition. The staff can accurately locate the data with problems as the first data processed by the first proxy node according to the first alarm, so that the node related to the first data can be repaired.
Optionally, the information of the second subtask may also carry a second alarm condition, and after the processing node processes the data obtained according to the information of the second subtask, if the data meets the second alarm condition, the processing node performs a second alarm. The second alarm condition may refer to the first alarm condition, and the manner in which the processing node performs the second alarm may refer to the manner in which the first proxy node performs the first alarm. The staff can analyze the first data with problems according to the second alarm, and repair the node related to the first data.
The data processing method provided in the embodiment of the present application will be illustrated by taking the application scenario shown in fig. 5 as an example. Assuming that the first data is a monitoring log of an application instance, and the data processing system needs to collect monitoring logs of three application instances, the monitoring logs of the three application instances can be collected by three first proxy nodes (01A, 01B and 01C) in fig. 5 in a one-to-one correspondence manner, and the first processing task is an error rate task for obtaining error rates of the monitoring logs in the monitoring logs of the three application instances.
The management node 11 may acquire information of the acquisition tasks of the monitoring logs (three kinds of monitoring logs) of the three application instances, which may include: the three types of monitoring logs include the IP address (IP address 1, IP address 2 and IP address 3) of the node where the monitoring log is located, the collecting path (path 1) of the monitoring log, the collecting time interval (10 seconds) of the monitoring log, and the type (log class data) of the monitoring log. Wherein the management node 11 may send information of the first acquisition task of the monitoring log of the application instance for acquisition to each first proxy node. Illustratively, the information may include: the IP address of the node where the monitoring log is located (e.g. IP address 1, 2 or 3), the acquisition path of the monitoring log (path 1), the acquisition time interval of the monitoring log (10 seconds), and the type of the monitoring log (log class data).
The management node 11 may send information of the first acquisition task of the monitoring log of the application instance for acquisition to each first proxy node through the transit node 12. The first proxy nodes 01A, 01B and 01C are respectively connected with different data sources 21, and the three first proxy nodes can collect the monitoring data of the application instance in the connected data sources 21 according to the received information of the first collecting task and process the collected monitoring data according to the received information of the first subtask.
The monitoring log may include: a timestamp field, a log level field, a log path field, and a log description information field. If the monitoring log is an error log, the content of a log level field in the monitoring log is an error level; if the monitoring log is not an error log, the contents of the log level field in the monitoring log are not error level, e.g., the contents of the log level field are correct level. The first processing task may include a first subtask for execution on a first proxy node, the first subtask including: and filtering to obtain an error log in the monitoring log, and extracting a time stamp field, a log level field and a log description information field in the error log.
Illustratively, assume that the monitoring log collected by the first proxy node 01A includes monitoring logs 1, 2, and 3. Wherein, monitor log 1 includes: timestamp field 1, log level field 1, log path field 1, and log description information field 1; the monitoring log 2 includes: timestamp field 2, log level field 2, log path field 1, and log description information field 1; the monitoring log 1 includes: timestamp field 3, log level field 1, log path field 1, and log description information field 1. The content of the log level field 1 is a correct level, and the content of the log level field 2 is an error level. The first proxy node 01A may filter out the error log (monitoring log 2) of the three monitoring logs when executing the first subtask. And, the first proxy node 01A may also extract the timestamp field 2, the log level field 2 and the log description information field 1 in the monitoring log 2.
The first subtask for execution on the first proxy node in the first processing task may further include: error count tasks and total count tasks. The first proxy node performs an error counting task to obtain the number of error logs, and the first proxy node performs a total counting task to obtain the total number of monitoring logs.
The first proxy node may also send the processed data to the processing node 02 via the message queue node 13. The management node 11 may also send information of the second subtask in the first processing task to the processing node 02. The processing node 02 may process the data sent by the first proxy node 01A and the first proxy node 01B according to the received information of the second subtask. For example, the processing node 02 may aggregate data sent by the first proxy node 01A and the first proxy node 01B, and store the result obtained by the aggregation to the database node.
By way of example, the second subtask for execution on the processing node in the first processing task described above may comprise: executing a third summation task on the results of the error counting tasks sent by the three first proxy nodes to obtain the sum of the numbers of error logs collected by the three first proxy nodes; the processing node executes a third summation task on the sum of the results of the total counting tasks sent by the three first proxy nodes to obtain the sum of the total number of the monitoring logs collected by the three first proxy nodes; and the processing node executes a second quotient-solving task on the result obtained by the third summation task and the result obtained by the fourth summation task to obtain the error rate of the monitoring log acquired by the first proxy node. Assuming that the total number of the monitoring logs collected by the three first proxy nodes within 1 minute is 300, and the number of error logs in the 300 monitoring logs collected by the three first proxy nodes is 21, 9 and 30 respectively, the error rate calculated by the processing node is (21+9+30)/(300+300+300) =6.67%.
In addition, the second alarm condition may be that the error rate calculated by the processing node exceeds a threshold (e.g. 20%), and if the error rate calculated by the processing node exceeds the threshold, the processing node will perform the second alarm.
In this example, the three first proxy nodes can process 300×3=900 monitoring logs, and the processing node only needs to process three processing results of the three first proxy nodes, so that it can be seen that processing tasks required to be executed by the processing node are greatly reduced, resource consumption of the processing node is reduced, time delay of processing data is reduced, and performance of the processing node is improved.
The present application also provides a data processing system, as shown in fig. 2, including: a management node 11, a first proxy node 01 and a processing node 02.
The management node 11 is configured to: acquiring information of a first acquisition task of first data and information of a first processing task, wherein the data used for processing by the first processing task comprises the first data; transmitting information of the first acquisition task and information of a first subtask used for processing the first data in the first processing task to the first proxy node 01; and sending information of a second subtask in the first processing task to a processing node 02, wherein the first subtask and the second subtask are tasks which are executed sequentially; illustratively, the first subtask includes: at least one of a data filtering task, a data extracting task, a data supplementing task, a data desensitizing task and a first data aggregation task; the second subtask includes: and a second data aggregation task.
The first proxy node 01 is configured to collect the first data according to the information of the first collecting task, process the first data according to the information of the first subtask, and send the processed data to the processing node; the processing node 02 is configured to process data sent by the first proxy node according to the information of the second subtask.
Optionally, the data processing system comprises: one or more first proxy nodes.
Optionally, the information of the first acquisition task carries: the data sent by the first proxy node 01 to the processing node 02 carries the target tag, and the information of the second subtask also carries the target tag; the processing node 02 is further configured to determine, according to the target tag carried by the data sent by the first proxy node 01, information of the second subtask carrying the target tag.
The manner in which the management node 11 transmits the information of the first acquisition task and the information of the first subtask to the first proxy node 01 is varied. In an optional implementation manner, the information of the first acquisition task carries a type identifier of the first proxy node; the data processing system further comprises: a transit node (not shown in fig. 2); the management node 11 is configured to send information of the first acquisition task and information of the first subtask to the transit node; the transfer node is used for determining the address of the first proxy node corresponding to the type identifier of the first proxy node 01 according to the corresponding relation between the address of each proxy node and the type identifier; the transfer node is further configured to send, to the first proxy node 01, information of the first acquisition task and information of the first subtask according to the address of the first proxy node 01.
In another alternative implementation, the data processing system may also not include a transit node, in which case the management node 11 may be configured to send the information of the first acquisition task and the information of the first subtask to the first proxy node 01 without passing through the transit node.
The manner in which the first proxy node 01 transmits the processed data to the processing node 02 is varied. Illustratively, the data processing system further comprises: message queue nodes (not shown in fig. 2); the first proxy node 01 is used for sending the processed data to a message queue node; the message queue node is used for adding the data sent by the first proxy node 01 into a message queue; the processing node 02 is configured to read, from the message queue, data sent by the first proxy node 01.
Optionally, the information of the first subtask carries a first alarm condition; the first proxy node 01 is further configured to perform a first alarm when the data obtained by processing the information of the first subtask meets the first alarm condition.
Optionally, the information of the second subtask carries a second alarm condition; the processing node 02 is further configured to perform a second alarm when the data obtained by processing according to the information of the second subtask meets the second alarm condition.
Optionally, the data processing system may comprise one or more second proxy nodes (not shown in fig. 2) in addition to the first proxy node 01 described above. The management node 11 is further configured to obtain information of a second acquisition task of second data, and information of a second processing task, where the data processed by the second processing task includes the second data; the second processing task may be any data processing task, for example, the second processing task may include: invoking a chain error sampling task (a sampling task of an erroneous call chain); the management node 11 is further configured to send information of the second acquisition task to the second proxy node, and the management node 11 is further configured to send information of the second processing task to the processing node 02; the second proxy node is configured to collect the second data according to the information of the second collection task, and send the second data to the processing node 02; the processing node 02 is further configured to process the second data sent by the second proxy node according to the information of the second processing task.
The management node 11, the first proxy node 01, and the processing node 02 may each be implemented by software, or may be implemented by hardware. Illustratively, an implementation of the management node 11 is described next. Similarly, the implementation of the first proxy node 01 and the processing node 02 may refer to the implementation of the management node 11.
Module as an example of a software functional unit, the management node 11 may comprise code running on a computing instance. Wherein the computing instance may be at least one of a physical host (computing device), a virtual machine, a container, etc. computing device. Further, the computing device may be one or more. For example, the management node 11 may include code running on multiple hosts/virtual machines/containers. It should be noted that, multiple hosts/virtual machines/containers for running the application may be distributed in the same region (region), or may be distributed in different regions. Multiple hosts/virtual machines/containers for running the code may be distributed in the same availability zone (availability zone, AZ) or may be distributed in different AZs, each AZ comprising a data center or multiple geographically close data centers. Wherein typically one region may comprise a plurality of AZs.
Also, multiple hosts/virtual machines/containers for running the code may be distributed in the same virtual private cloud (virtual private cloud, VPC) or in multiple VPCs. Wherein typically one VPC is arranged in one area. The inter-regional communication between two VPCs in the same area and between VPCs in different areas needs to set a communication gateway in each VPC, and the interconnection between the VPCs is realized through the communication gateway.
Modules as an example of hardware functional units, the management node 11 may comprise at least one computing device, such as a server or the like. Alternatively, the management node 11 may be a device implemented by an application specific integrated circuit (application specific integrated circuit, ASIC), or a programmable logic device (programmable logic device, PLD), or the like. The PLD may be a complex programmable logic device (complex programmable logic device, CPLD), a field programmable gate array (field programmable gate array, FPGA), a general-purpose array logic (generic array logic, GAL) device, or any combination thereof.
The plurality of computing devices included in the management node 11 may be distributed in the same area or may be distributed in different areas. The plurality of computing devices comprised by the management node 11 may be distributed in the same AZ or may be distributed in different AZs. Likewise, multiple computing devices included in management node 11 may be distributed in the same VPC or may be distributed among multiple VPCs. Wherein the plurality of computing devices may be any combination of computing devices such as servers, ASIC, PLD, CPLD, FPGA, and GAL.
The application also provides a data processing device, which belongs to a management node in a data processing system, and the data processing system further comprises: the first proxy node and the processing node, as shown in fig. 6, the data processing apparatus includes: a first acquisition module 601, a first transmission module 602, and a second transmission module 603.
A first obtaining module 601, configured to obtain information of a first acquisition task of first data, and information of a first processing task, where data used for processing by the first processing task includes the first data; the first sending module is used for sending the information of the first acquisition task and the information of a first subtask used for processing the first data in the first processing task to the first proxy node;
and a second sending module 602, configured to send, to the processing node, information of a second subtask in the first processing task, where the first subtask and the second subtask are tasks that are executed sequentially.
Illustratively, the first subtask includes: at least one of a data filtering task, a data extracting task, a data supplementing task, a data desensitizing task and a first data aggregation task; the second subtask includes: and a second data aggregation task.
Optionally, the data processing system comprises: one or more first proxy nodes.
Optionally, the information of the first acquisition task carries: and the information of the second subtask also carries the target label.
The management node transmits the information of the first acquisition task and the information of the first subtask to the first proxy node in various manners. In an optional implementation manner, the data processing system further includes a transit node, and the information of the first acquisition task carries a type identifier of the first proxy node; the first sending module 601 is configured to: transmitting the information of the first acquisition task and the information of the first subtask to the transfer node; the transit node is used for: determining the address of the first proxy node corresponding to the type identifier of the first proxy node according to the corresponding relation between the address of each proxy node and the type identifier; and sending the information of the first acquisition task and the information of the first subtask to the first proxy node according to the address of the first proxy node.
Optionally, the information of the first subtask carries a first alarm condition.
Optionally, the information of the second subtask carries a second alarm condition.
Optionally, the data processing system may comprise one or more second proxy nodes in addition to the first proxy nodes described above. The data processing apparatus further includes: a second acquisition module and a third transmission module (not shown in fig. 6). The second acquisition module is used for acquiring information of a second acquisition task of second data and information of a second processing task, and the data processed by the second processing task comprises the second data; the second processing task may be any data processing task, for example, the second processing task may include: invoking a chain error sampling task (a sampling task of an erroneous call chain); and the third sending module is used for sending the information of the second acquisition task to the second proxy node and sending the information of the second processing task to the processing node.
The respective modules in the data processing apparatus (such as the first acquisition module 601, the first transmission module 602, the second transmission module 603, the second acquisition module, and the third transmission module) may be implemented by software, or may be implemented by hardware. By way of example, the implementation of the first acquisition module 601 will be described below using the first acquisition module 601 as an example. Similarly, the implementation of the first sending module 602 and the second sending module 603, the second obtaining module and the third sending module may refer to the implementation of the first obtaining module 601.
Module as an example of a software functional unit, the first acquisition module 601 may comprise code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container, among others. Further, the above-described computing examples may be one or more. For example, the first acquisition module 601 may include code running on multiple hosts/virtual machines/containers. It should be noted that, multiple hosts/virtual machines/containers for running the code may be distributed in the same area, or may be distributed in different areas. Further, multiple hosts/virtual machines/containers for running the code may be distributed among the same AZ, or may be distributed among different AZs, each AZ including one data center or multiple geographically proximate data centers. Wherein typically one region may comprise a plurality of AZs.
Also, multiple hosts/virtual machines/containers for running the code may be distributed in the same VPC, or may be distributed among multiple VPCs. In general, one VPC is disposed in one area, and a communication gateway is disposed in each VPC for implementing inter-connection between VPCs in the same area, and inter-area communication between two VPCs in different areas.
Module as an example of a hardware functional unit, the first acquisition module 601 may include at least one computing device, such as a server or the like. Alternatively, the first acquisition module 601 may be a device implemented by ASIC or PLD. Wherein the PLD may be CPLD, FPGA, GAL device or any combination thereof.
The plurality of computing devices included in the first acquisition module 601 may be distributed in the same area or may be distributed in different areas. The plurality of computing devices included in the first acquisition module 601 may be distributed in the same AZ or may be distributed in different AZ. Likewise, the multiple computing devices included in the first acquisition module 601 may be distributed in the same VPC, or may be distributed in multiple VPCs. Wherein the plurality of computing devices may be any combination of computing devices such as servers, ASIC, PLD, CPLD, FPGA, and GAL devices.
In other embodiments, the first obtaining module 601 may be used to perform any step in a data processing method used by a management node to perform, the first sending module 602 may be used to perform any step in a data processing method used by a management node to perform, the second sending module 603 may be used to perform any step in a data processing method used by a management node to perform, the second obtaining module may be used to perform any step in a data processing method used by a management node to perform, and the third sending module may be used to perform any step in a data processing method used by a management node to perform. The steps of the first acquiring module 601, the first sending module 602, the second sending module 603, the second acquiring module and the third sending module responsible for implementation can be specified according to needs, and different steps in the data processing method used for being executed by the management node are respectively implemented through the first acquiring module 601, the first sending module 602, the second sending module 603, the second acquiring module and the third sending module to realize all functions of the data processing device belonging to the management node.
The present application further provides a data processing apparatus, where the data processing apparatus belongs to a first proxy node in a data processing system, and the data processing system further includes a management node and a processing node, as shown in fig. 7, and the data processing apparatus includes: a receiving module 701, an acquisition module 702, a processing module 703 and a transmitting module 704.
The receiving module 701 is configured to receive information of a first acquisition task of first data sent by the management node, and information of a first subtask used for processing the first data in a first processing task, where the data processed by the first processing task includes the first data, and the first processing task further includes a second subtask, and the first subtask and the second subtask are tasks that are executed sequentially; illustratively, the first subtask includes: at least one of a data filtering task, a data extracting task, a data supplementing task, a data desensitizing task and a first data aggregation task; the second subtask includes: and a second data aggregation task.
The acquisition module 702 is configured to acquire the first data according to the information of the first acquisition task; a processing module 703, configured to process the first data according to the information of the first subtask; and the sending module 704 is configured to send the processed data to the processing node.
Optionally, the data processing system comprises: one or more of the first proxy nodes.
Optionally, the information of the first acquisition task carries: and the target label is carried by the data sent to the processing node by the first proxy node.
The management node transmits the information of the first acquisition task and the information of the first subtask to the first proxy node in various manners. In an optional implementation manner, the information of the first acquisition task carries a type identifier of the first proxy node; the data processing system further comprises a transit node; the receiving module 701 is configured to: receiving the address of the transfer node according to the first proxy node, and sending the information of the first acquisition task and the information of the first subtask to the first proxy node; the management node is used for sending the information of the first acquisition task and the information of the first subtask to the transfer node; the transfer node is used for determining the address of the first proxy node corresponding to the type identifier of the first proxy node according to the corresponding relation between the address of each proxy node and the type identifier.
The manner in which the first proxy node sends processed data to the processing node is varied. Illustratively, the data processing system further comprises a message queue node; the sending module 704 is configured to: transmitting the processed data to the message queue node; the message queue node is used for adding the data sent by the first proxy node into a message queue; the processing node is used for reading the data sent by the first proxy node from the message queue.
Optionally, the information of the first subtask carries a first alarm condition, and the data processing apparatus further includes: and the alarm module (not shown in fig. 7) is used for carrying out a first alarm when the data obtained by the processing module according to the information processing of the first subtask meets the first alarm condition.
The various modules in the data processing apparatus (such as the receiving module 701, the collecting module 702, the processing module 703, the transmitting module 704 and the alarm module) may be implemented by software, or may be implemented by hardware. Illustratively, the implementation of the receiving module 701 is described next as an example of the receiving module 701. Similarly, the implementation of the acquisition module 702, the processing module 703, the sending module 704 and the alarm module may refer to the implementation of the receiving module 701.
Module as an example of a software functional unit, the receiving module 701 may comprise code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container, among others. Further, the above-described computing examples may be one or more. For example, the receiving module 701 may include code running on multiple hosts/virtual machines/containers. It should be noted that, multiple hosts/virtual machines/containers for running the code may be distributed in the same area, or may be distributed in different areas. Further, multiple hosts/virtual machines/containers for running the code may be distributed among the same AZ, or may be distributed among different AZs, each AZ including one data center or multiple geographically proximate data centers. Wherein typically one region may comprise a plurality of AZs.
Also, multiple hosts/virtual machines/containers for running the code may be distributed in the same VPC, or may be distributed among multiple VPCs. In general, one VPC is disposed in one area, and a communication gateway is disposed in each VPC for implementing inter-connection between VPCs in the same area, and inter-area communication between two VPCs in different areas.
Module as an example of a hardware functional unit, the receiving module 701 may include at least one computing device, such as a server or the like. Alternatively, the receiving module 701 may be a device implemented by ASIC or PLD. Wherein the PLD may be CPLD, FPGA, GAL device or any combination thereof.
The plurality of computing devices included in the receiving module 701 may be distributed in the same area or may be distributed in different areas. The plurality of computing devices included in the receiving module 701 may be distributed in the same AZ or may be distributed in different AZ. Also, the plurality of computing devices included in the receiving module 701 may be distributed in the same VPC or may be distributed in a plurality of VPCs. Wherein the plurality of computing devices may be any combination of computing devices such as servers, ASIC, PLD, CPLD, FPGA, and GAL devices.
In other embodiments, the receiving module 701 may be configured to perform any step in a data processing method used by the first proxy node, the collecting module 702 may be configured to perform any step in a data processing method used by the first proxy node, the processing module 703 may be configured to perform any step in a data processing method used by the first proxy node, the sending module 704 may be configured to perform any step in a data processing method used by the first proxy node, and the alarm module may be configured to perform any step in a data processing method used by the first proxy node. The steps that the receiving module 701, the collecting module 702, the processing module 703, the sending module 704 and the alarm module are responsible for implementing may be specified according to needs, and all functions of the data processing apparatus belonging to the first proxy node are implemented by implementing different steps in the data processing method that the first proxy node is used to execute through the receiving module 701, the collecting module 702, the processing module 703, the sending module 704 and the alarm module.
The application also provides a data transfer device, which belongs to a transfer node in a data processing system, the data processing system further comprises a management node, a first proxy node and a processing node, as shown in fig. 8, and the data transfer device comprises: a receiving module 801, a determining module 802 and a transmitting module 803.
The receiving module 801 is configured to receive information of a first acquisition task of first data sent by the management node, and information of a first subtask used for processing the first data in a first processing task; the information of the first acquisition task carries a type identifier of the first proxy node; the first processing task further comprises a second subtask, and the first subtask and the second subtask are tasks which are executed sequentially; for example, the first subtask includes: at least one of a data filtering task, a data extracting task, a data supplementing task, a data desensitizing task and a first data aggregation task; the second subtask includes: and a second data aggregation task.
A determining module 802, configured to determine, according to a correspondence between an address of each proxy node and a type identifier, an address of the first proxy node corresponding to the type identifier of the first proxy node;
And a sending module 803, configured to send, to the first proxy node, the information of the first acquisition task and the information of the first subtask according to the address of the first proxy node.
Optionally, the data processing system comprises: one or more of the first proxy nodes.
Optionally, the information of the first acquisition task carries: target tags.
Optionally, the information of the first subtask carries a first alarm condition.
Each of the modules in the data relay device (such as the receiving module 801, the determining module 802, and the transmitting module 803) may be implemented by software, or may be implemented by hardware. Illustratively, an implementation of the receiving module 701 is described next with reference to the receiving module 801. Similarly, the implementation of the determining module 802 and the transmitting module 803 may refer to the implementation of the receiving module 801.
Module as an example of a software functional unit, the receiving module 801 may comprise code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container, among others. Further, the above-described computing examples may be one or more. For example, the receiving module 801 may include code running on multiple hosts/virtual machines/containers. It should be noted that, multiple hosts/virtual machines/containers for running the code may be distributed in the same area, or may be distributed in different areas. Further, multiple hosts/virtual machines/containers for running the code may be distributed among the same AZ, or may be distributed among different AZs, each AZ including one data center or multiple geographically proximate data centers. Wherein typically one region may comprise a plurality of AZs.
Also, multiple hosts/virtual machines/containers for running the code may be distributed in the same VPC, or may be distributed among multiple VPCs. In general, one VPC is disposed in one area, and a communication gateway is disposed in each VPC for implementing inter-connection between VPCs in the same area, and inter-area communication between two VPCs in different areas.
Module as an example of a hardware functional unit, the receiving module 801 may include at least one computing device, such as a server or the like. Alternatively, the receiving module 801 may be a device implemented by ASIC or PLD. Wherein the PLD may be CPLD, FPGA, GAL device or any combination thereof.
The plurality of computing devices included in the receiving module 801 may be distributed in the same area or may be distributed in different areas. The plurality of computing devices included in the receiving module 801 may be distributed in the same AZ or may be distributed in different AZ. Likewise, multiple computing devices included in the receiving module 801 may be distributed in the same VPC or may be distributed among multiple VPCs. Wherein the plurality of computing devices may be any combination of computing devices such as servers, ASIC, PLD, CPLD, FPGA, and GAL devices.
In other embodiments, the receiving module 801 may be configured to perform any step in a data processing method for execution by a relay node, the determining module 802 may be configured to perform any step in a data processing method for execution by a relay node, and the sending module 803 may be configured to perform any step in a data processing method for execution by a relay node. The steps that the receiving module 801, the determining module 702 and the transmitting module 803 are responsible for implementing may be specified as required, and all functions of the data transfer device belonging to the transfer node are implemented by implementing different steps in the data processing method that the transfer node is used to execute through the receiving module 801, the determining module 702 and the transmitting module 803, respectively.
Embodiments of the present application also provide a computing device 100. As shown in fig. 9, the computing device 100 includes: bus 1002, processor 1004, memory 1006, and communication interface 1008. Communication between the processor 1004, memory 1006 and communication interface 1008 is via bus 1002. Computing device 100 may be a server or a terminal device. It should be understood that the present application is not limited to the number of processors, memories in computing device 100.
Bus 1002 may be a peripheral component interconnect standard (peripheral component interconnect, PCI) bus or an extended industry standard architecture (extended industry standard architecture, EISA) bus, among others. The buses may be divided into address buses, data buses, control buses, etc. For ease of illustration, only one line is shown in fig. 9, but not only one bus or one type of bus. Bus 1004 may include a pathway to transfer information between various components of computing device 100 (e.g., memory 1006, processor 1004, communication interface 1008).
The processor 1004 may include any one or more of a central processing unit (central processing unit, CPU), a graphics processor (graphics processing unit, GPU), a Microprocessor (MP), or a digital signal processor (digital signal processor, DSP).
The memory 1006 may include volatile memory (RAM), such as random access memory (random access memory). The processor 1004 may also include a non-volatile memory (non-volatile memory), such as read-only memory (ROM), flash memory, a mechanical hard disk (HDD) or a solid state disk (solid state drive, SSD).
As shown in fig. 9, the memory 1006 stores executable program codes, and the processor 1004 executes the executable program codes to implement the functions of the first acquisition module 601, the first transmission module 602, and the second transmission module 603, respectively, so as to implement the method performed by the management node provided in the embodiment of the present application. That is, the memory 1006 has instructions stored thereon for performing the methods performed by the management node provided by embodiments of the present application.
Alternatively, the memory 1006 stores executable program codes, and the processor 1004 executes the executable program codes to implement the functions of the foregoing receiving module 701, the collecting module 702, the processing module 703 and the transmitting module 704 (not shown in fig. 9), so as to implement the method performed by the first proxy node provided in the embodiment of the present application. That is, the memory 1006 has stored thereon instructions for performing the methods performed by the first proxy node provided in the embodiments of the present application.
Alternatively, the memory 1006 stores executable program codes, and the processor 1004 executes the executable program codes to implement the functions of the foregoing receiving module 801, determining module 702 and transmitting module 803 (not shown in fig. 9), respectively, so as to implement the method performed by the transit node provided in the embodiment of the present application. That is, the memory 1006 has stored thereon instructions for performing the methods performed by the transit node provided in the embodiments of the present application.
Alternatively, the memory 1006 stores executable code, and the processor 1004 executes the executable code to implement the functions of the aforementioned management node 11, the first proxy node 01, and the processing node 02 (not shown in fig. 9), respectively, so as to implement the method performed by the data processing system provided in the embodiment of the present application. That is, the memory 1006 has instructions stored thereon for performing the methods provided by embodiments of the present application that are performed by a data processing system.
Communication interface 1003 enables communication between computing device 100 and other devices or communication networks using a transceiver module such as, but not limited to, a network interface card, transceiver, or the like.
The embodiment of the application also provides a computing device cluster. The cluster of computing devices includes at least one computing device. The computing device may be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device may also be a terminal device such as a desktop, notebook, or smart phone.
As shown in fig. 10, the cluster of computing devices includes at least one computing device 100. The same instructions may be stored in memory 1006 in one or more computing devices 100 in the computing device cluster. The instruction is used to execute the data processing method for the data processing system, the management node, the first proxy node or the transit node provided in the embodiment of the present application, and fig. 10 is taken as an example of the instruction used to execute the data processing method for the management node provided in the embodiment of the present application.
In some possible implementations, part of the instructions for performing the data processing method for the data processing system, the management node, the first proxy node, or the transit node provided in the embodiments of the present application may also be stored in the memory 1006 of one or more computing devices 100 in the computing device cluster, respectively. In other words, a combination of one or more computing devices 100 may collectively execute instructions for a data processing method for a data processing system, a management node, a first proxy node, or a transit node provided by embodiments of the present application.
The memories 1006 in different computing devices 100 in the computing device cluster may store different instructions for performing part of the functions of the data processing means belonging to the management node, the data processing means belonging to the first proxy node, or the data forwarding means belonging to the transit node, respectively. That is, instructions stored in memory 1006 in different computing devices 100 may implement the functions of one or more modules in the apparatus.
In some possible implementations, one or more computing devices in a cluster of computing devices may be connected through a network. Wherein the network may be a wide area network or a local area network, etc. Fig. 11 shows one possible implementation. As shown in fig. 11, two computing devices 100A and 100B are connected by a network. Specifically, the connection to the network is made through a communication interface in each computing device. In this type of possible implementation, instructions to perform the functions of a portion of the modules of the apparatus are stored in memory 106 in computing device 100A. Meanwhile, the memory 106 in the computing device 100B stores therein instructions for performing functions of another part of the modules of the apparatus.
The connection between clusters of computing devices shown in fig. 11 may be implemented by computing device 100B in consideration of the large amount of data that needs to be transmitted by the data processing method provided in the present application, and thus, the functions implemented by first transmitting module 602 and second transmitting module 603 are considered to be performed.
It should be appreciated that the functionality of computing device 100A shown in fig. 11 may also be performed by multiple computing devices 100. Likewise, the functionality of computing device 100B may also be performed by multiple computing devices 100.
In fig. 11, the data processing apparatus belonging to the management node is taken as an example, and it is understood that, for the data processing apparatus belonging to the first proxy node or the data relay apparatus of the relay node, an instruction for executing a part of the functions of the apparatus may be stored in the memory 106 in the computing device 100A, and an instruction for executing another part of the functions of the apparatus may be stored in the memory 106 in the computing device 100B.
The memory 1006 in different computing devices 100 in a cluster of computing devices may store different instructions for performing part of the functions of the data processing system. That is, the instructions stored by the memory 1006 in the different computing devices 100 may implement the functionality of one or more of a management node, a first proxy node, a processing node, a transit node, and a message queue node.
Embodiments of the present application also provide a computer readable storage medium comprising computer program instructions which, when executed by a cluster of computing devices, perform any of the methods performed by a data processing system as provided by the embodiments of the present application. The computer readable storage medium may be any available medium that can be stored by a computing device or a data storage device such as a data center containing one or more available media. The usable medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk), etc.
The embodiments also provide a computer readable storage medium comprising computer program instructions which, when executed by a cluster of computing devices, perform any of the methods performed by a management node, a first proxy node, a processing node or a transit node in a data processing system as provided by the embodiments of the present application. The computer readable storage medium may be any available medium that can be stored by a computing device or a data storage device such as a data center containing one or more available media. The usable medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk), etc.
Embodiments of the present application also provide a computer program product comprising instructions, which may be a software or program product comprising instructions, capable of running on a computing device or stored in any available medium. The instructions, when executed by a cluster of computing devices, cause the cluster of computing devices to perform any of the methods performed by a data processing system as provided by embodiments of the present application.
Embodiments of the present application also provide a computer program product comprising instructions, which may be a software or program product comprising instructions, capable of running on a computing device or stored in any available medium. The instructions, when executed by a cluster of computing devices, cause the cluster of computing devices to perform any of the methods performed by a management node, a first proxy node, a processing node, or a transit node in a data processing system as provided by embodiments of the present application.
The data and information related to the application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of the data and information need to comply with relevant laws and regulations and standards of relevant countries and regions.
In this application, the terms "first" and "second" and the like are used for descriptive purposes only and are not to be construed as indicating or implying relative importance. The term "at least one" means one or more, "a plurality" means two or more, unless expressly defined otherwise. The term "and/or" is merely an association relationship describing an associated object, meaning that there may be three relationships, e.g., a and/or B, may represent: a exists alone, A and B exist together, and B exists alone.
Different types of embodiments, such as a method embodiment and a device embodiment, provided in the embodiments of the present application may be mutually referred to, and the embodiments of the present application are not limited to this. The sequence of operations in the method embodiment provided by the embodiment of the invention can be properly adjusted, the operations can be correspondingly increased or decreased according to the situation, and any method which is easily conceivable to be changed by a person skilled in the art within the technical scope of the disclosure of the invention is covered in the protection scope of the invention, so that the description is omitted.
In the corresponding embodiments provided in the present application, it should be understood that the disclosed system and apparatus may be implemented in other structural manners. For example, the apparatus embodiments described above are merely illustrative, e.g., the division of modules is merely a logical function division, and there may be additional divisions of actual implementation, e.g., multiple modules may be combined or integrated into another system, or some features may be omitted or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or modules, which may be in electrical or other forms.
Elements illustrated as separate elements may or may not be physically separate, and elements described as elements may or may not be physically located, or may be distributed over several apparatuses. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
While the invention has been described with reference to certain preferred embodiments, it will be understood by those skilled in the art that various changes and substitutions of equivalents may be made and equivalents will be apparent to those skilled in the art without departing from the scope of the invention. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.
Claims (30)
1. A data processing method for a data processing system, the data processing system comprising a management node, a first proxy node, and a processing node, the method comprising:
the method comprises the steps that a management node obtains information of a first acquisition task of first data and information of a first processing task, wherein the data used for processing by the first processing task comprises the first data;
The management node sends information of the first acquisition task and information of a first subtask used for processing the first data in the first processing task to the first proxy node;
the management node sends information of a second subtask in the first processing task to a processing node, wherein the first subtask and the second subtask are tasks which are executed in sequence;
the first proxy node acquires the first data according to the information of the first acquisition task, processes the first data according to the information of the first subtask, and sends the processed data to the processing node;
and the processing node processes the data sent by the first proxy node according to the information of the second subtask.
2. The method of claim 1, wherein the data processing system comprises: and a plurality of the first proxy nodes.
3. The method according to claim 1 or 2, wherein the information of the first acquisition task carries: the data sent by the first proxy node to the processing node carries the target label, and the information of the second subtask also carries the target label;
Before the processing node processes the data sent by the first proxy node according to the information of the second subtask, the method further includes:
and the processing node determines the information of the second subtask carrying the target label according to the target label carried by the data sent by the first proxy node.
4. A method according to any one of claims 1 to 3, wherein the information of the first acquisition task carries a type identifier of the first proxy node; the data processing system further comprises a transit node;
the management node sending information of the first acquisition task and information of a first subtask used for processing the first data in the first processing task to the first proxy node, wherein the information comprises:
the management node sends the information of the first acquisition task and the information of the first subtask to the transfer node;
the transit node determines the address of the first proxy node corresponding to the type identifier of the first proxy node according to the corresponding relation between the address of each proxy node and the type identifier;
and the transfer node sends the information of the first acquisition task and the information of the first subtask to the first proxy node according to the address of the first proxy node.
5. The method of any of claims 1 to 4, wherein the data processing system further comprises a message queue node; the first proxy node sends the processed data to the processing node, including:
the first proxy node sends the processed data to the message queue node;
the message queue node adds the data sent by the first proxy node into a message queue;
and the processing node reads the data sent by the first proxy node from the message queue.
6. The method according to any one of claims 1 to 5, wherein the information of the first subtask carries a first alarm condition, the method further comprising:
and the first proxy node carries out first alarm when the data obtained through information processing according to the first subtask meets the first alarm condition.
7. The method according to any of claims 1 to 6, wherein the information of the second subtask carries a second alarm condition, the method further comprising:
and the processing node carries out a second alarm when the data obtained by processing the information of the second subtask meets the second alarm condition.
8. The method of any of claims 1 to 7, wherein the first subtask comprises: at least one of a data filtering task, a data extracting task, a data supplementing task, a data desensitizing task and a first data aggregation task;
the second subtask includes: and a second data aggregation task.
9. The method according to any of claims 1 to 8, wherein the data processing system further comprises a second proxy node; the method further comprises the steps of:
the management node acquires information of a second acquisition task of second data and information of a second processing task, wherein the data processed by the second processing task comprises the second data;
the management node sends the information of the second acquisition task to the second proxy node;
the management node sends information of the second processing task to the processing node;
the second agent node acquires the second data according to the information of the second acquisition task and sends the second data to the processing node;
and the processing node processes the second data sent by the second proxy node according to the information of the second processing task.
10. The method of claim 9, wherein the second processing task comprises: a chain error sampling task is invoked.
11. A data processing system, the data processing system comprising: the system comprises a management node, a first proxy node and a processing node;
the management node is used for acquiring information of a first acquisition task of first data and information of a first processing task, and the data processed by the first processing task comprises the first data;
the management node is further configured to send information of the first acquisition task and information of a first subtask used for processing the first data in the first processing task to the first proxy node;
the management node is further configured to send information of a second subtask in the first processing task to a processing node, where the first subtask and the second subtask are tasks that are executed sequentially;
the first proxy node is configured to collect the first data according to the information of the first collecting task, process the first data according to the information of the first subtask, and send the processed data to the processing node;
And the processing node is used for processing the data sent by the first proxy node according to the information of the second subtask.
12. The data processing system of claim 11, wherein the data processing system comprises: and a plurality of the first proxy nodes.
13. The data processing system of claim 11 or 12, wherein the information of the first acquisition task carries: the data sent by the first proxy node to the processing node carries the target label, and the information of the second subtask also carries the target label;
the processing node is further configured to determine, according to the target tag carried by the data sent by the first proxy node, information of the second subtask carrying the target tag.
14. The data processing system according to any of claims 11 to 13, wherein the information of the first acquisition task carries a type identifier of the first proxy node; the data processing system further comprises: a transit node;
the management node is used for sending the information of the first acquisition task and the information of the first subtask to the transfer node;
The transfer node is used for determining the address of the first proxy node corresponding to the type identifier of the first proxy node according to the corresponding relation between the address of each proxy node and the type identifier;
the transfer node is further configured to send, to the first proxy node, information of the first acquisition task and information of the first subtask according to the address of the first proxy node.
15. The data processing system of any of claims 11 to 14, wherein the data processing system further comprises: a message queue node;
the first proxy node is used for sending the processed data to the message queue node;
the message queue node is used for adding the data sent by the first proxy node into a message queue;
the processing node is used for reading the data sent by the first proxy node from the message queue.
16. A data processing system according to any one of claims 11 to 15, wherein the information of the first subtask carries a first alarm condition;
the first proxy node is further configured to perform a first alarm when the data obtained by processing the information of the first subtask meets the first alarm condition.
17. A data processing system according to any of claims 11 to 16, wherein the information of the second subtask carries a second alarm condition;
and the processing node is also used for carrying out a second alarm when the data obtained by the information processing of the second subtask meets the second alarm condition.
18. A data processing system according to any of claims 11 to 17, wherein the first subtask comprises: at least one of a data filtering task, a data extracting task, a data supplementing task, a data desensitizing task and a first data aggregation task;
the second subtask includes: and a second data aggregation task.
19. A data processing system according to any of claims 11 to 18, wherein the data processing system further comprises a second proxy node;
the management node is further configured to obtain information of a second acquisition task of second data and information of a second processing task, where the data processed by the second processing task includes the second data;
the management node is further used for sending information of the second acquisition task to the second proxy node;
the management node is further configured to send information of the second processing task to the processing node;
The second agent node is used for acquiring the second data according to the information of the second acquisition task and sending the second data to the processing node;
and the processing node is also used for processing the second data sent by the second proxy node according to the information of the second processing task.
20. The data processing system of claim 19, wherein the second processing task comprises: a chain error sampling task is invoked.
21. A data processing method, the method performed by a management node in a data processing system, the data processing system further comprising: a first proxy node and a processing node, the method comprising:
acquiring information of a first acquisition task of first data and information of a first processing task, wherein the data used for processing by the first processing task comprises the first data;
transmitting information of the first acquisition task and information of a first subtask used for processing the first data in the first processing task to the first proxy node;
and sending information of a second subtask in the first processing task to the processing node, wherein the first subtask and the second subtask are tasks which are executed sequentially.
22. A data processing apparatus, the data processing apparatus belonging to a management node in a data processing system, the data processing system further comprising: the data processing device comprises a first proxy node and a processing node, and the data processing device comprises:
the first acquisition module is used for acquiring information of a first acquisition task of first data and information of a first processing task, wherein the data for processing by the first processing task comprises the first data;
the first sending module is used for sending the information of the first acquisition task and the information of a first subtask used for processing the first data in the first processing task to the first proxy node;
and the second sending module is used for sending information of a second subtask in the first processing task to the processing node, wherein the first subtask and the second subtask are tasks which are executed in sequence.
23. A data processing method, the method performed by a first proxy node in a data processing system, the data processing system further comprising a management node and a processing node, the method comprising:
receiving information of a first acquisition task of first data sent by the management node and information of a first subtask used for processing the first data in a first processing task; the first processing task further comprises a second subtask, and the first subtask and the second subtask are tasks which are executed sequentially;
Acquiring the first data according to the information of the first acquisition task;
processing the first data according to the information of the first subtask;
and sending the processed data to the processing node.
24. A data processing apparatus, the data processing apparatus belonging to a first proxy node in a data processing system, the data processing system further comprising a management node and a processing node, the data processing apparatus comprising:
the system comprises a receiving module, a processing module and a processing module, wherein the receiving module is used for receiving information of a first acquisition task of first data sent by the management node and information of a first subtask used for processing the first data in a first processing task, the data processed by the first processing task comprises the first data, the first processing task further comprises a second subtask, and the first subtask and the second subtask are sequentially executed tasks;
the acquisition module is used for acquiring the first data according to the information of the first acquisition task;
the processing module is used for processing the first data according to the information of the first subtask;
and the sending module is used for sending the processed data to the processing node.
25. A method of data transfer, the method performed by a transfer node in a data processing system, the data processing system further comprising: the method comprises the steps of:
receiving information of a first acquisition task of first data sent by the management node and information of a first subtask used for processing the first data in a first processing task; the information of the first acquisition task carries a type identifier of the first proxy node; the first processing task further comprises a second subtask, and the first subtask and the second subtask are tasks which are executed sequentially;
determining the address of the first proxy node corresponding to the type identifier of the first proxy node according to the corresponding relation between the address of each proxy node and the type identifier;
and sending the information of the first acquisition task and the information of the first subtask to the first proxy node according to the address of the first proxy node.
26. A data transfer device, wherein the data transfer device belongs to a transfer node in a data processing system, the data processing system further comprising: the data transfer device comprises a management node, a first proxy node and a processing node, and comprises:
The receiving module is used for receiving information of a first acquisition task of the first data sent by the management node and information of a first subtask used for processing the first data in a first processing task; the information of the first acquisition task carries a type identifier of the first proxy node; the first processing task further comprises a second subtask, and the first subtask and the second subtask are tasks which are executed sequentially;
the determining module is used for determining the address of the first proxy node corresponding to the type identifier of the first proxy node according to the corresponding relation between the address of each proxy node and the type identifier;
and the sending module is used for sending the information of the first acquisition task and the information of the first subtask to the first proxy node according to the address of the first proxy node.
27. A cluster of computing devices, comprising at least one computing device, each computing device comprising a processor and a memory;
the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method of any one of claims 1 to 10.
28. A cluster of computing devices, comprising at least one computing device, each computing device comprising a processor and a memory;
the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method of claim 21, 23, or 25.
29. A computer readable storage medium comprising computer program instructions which, when executed by a cluster of computing devices, perform the method of any of claims 1 to 10.
30. A computer readable storage medium comprising computer program instructions which, when executed by a cluster of computing devices, perform the method of claim 21, 23 or 25.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211052797.0A CN117667374A (en) | 2022-08-31 | 2022-08-31 | Data processing method, device and system |
| PCT/CN2023/087209 WO2024045621A1 (en) | 2022-08-31 | 2023-04-10 | Data processing method, apparatus and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211052797.0A CN117667374A (en) | 2022-08-31 | 2022-08-31 | Data processing method, device and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117667374A true CN117667374A (en) | 2024-03-08 |
Family
ID=90073884
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211052797.0A Pending CN117667374A (en) | 2022-08-31 | 2022-08-31 | Data processing method, device and system |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN117667374A (en) |
| WO (1) | WO2024045621A1 (en) |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP2028893A1 (en) * | 2007-06-27 | 2009-02-25 | Nokia Siemens Networks Oy | Proxy network element and method of performance management in a network |
| US10305759B2 (en) * | 2015-01-05 | 2019-05-28 | Cisco Technology, Inc. | Distributed and adaptive computer network analytics |
| CN107451223A (en) * | 2017-07-17 | 2017-12-08 | 广州特道信息科技有限公司 | The big data acquisition system and method for a kind of high concurrent parallel computation |
| CN111090687B (en) * | 2019-12-24 | 2023-03-10 | 腾讯科技（深圳）有限公司 | Data processing method, device and system and computer readable storage medium |
| CN111400326B (en) * | 2020-02-28 | 2023-09-12 | 深圳市赛为智能股份有限公司 | Smart city data management system and method thereof |
-
2022
- 2022-08-31 CN CN202211052797.0A patent/CN117667374A/en active Pending
-
2023
- 2023-04-10 WO PCT/CN2023/087209 patent/WO2024045621A1/en unknown
Also Published As
| Publication number | Publication date |
|---|---|
| WO2024045621A1 (en) | 2024-03-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111913818B (en) | Method for determining dependency relationship between services and related device | |
| CN112311617A (en) | Configured data monitoring and alarming method and system | |
| CN112737800B (en) | Service node fault positioning method, call chain generating method and server | |
| CN108521339A (en) | A kind of reaction type node failure processing method and system based on cluster daily record | |
| CN114090366A (en) | Method, device and system for monitoring data | |
| CN113452607A (en) | Distributed link acquisition method and device, computing equipment and storage medium | |
| CN113806191A (en) | Data processing method, device, equipment and storage medium | |
| CN110806960A (en) | Information processing method and device and terminal equipment | |
| CN110932933A (en) | Network condition monitoring method, computing device and computer storage medium | |
| CN117176802B (en) | Full-link monitoring method and device for service request, electronic equipment and medium | |
| CN110572284A (en) | Method, device and system for upgrading virtual network element | |
| CN110609761B (en) | Method and device for determining fault source, storage medium and electronic equipment | |
| CN110198246B (en) | Method and system for monitoring flow | |
| CN111431733A (en) | Service alarm coverage information evaluation method and device | |
| CN116136801B (en) | Cloud platform data processing method and device, electronic equipment and storage medium | |
| CN117667374A (en) | Data processing method, device and system | |
| CN115102730A (en) | Integrated monitoring method for multiple devices | |
| US10296967B1 (en) | System, method, and computer program for aggregating fallouts in an ordering system | |
| CN109388546B (en) | Method, device and system for processing faults of application program | |
| CN116886445B (en) | Processing method and device of filtering result, storage medium and electronic equipment | |
| CN110543470A (en) | Message early warning method and device, storage medium and electronic device | |
| CN111130919B (en) | Interface monitoring method, device and system and storage medium | |
| CN111740871A (en) | Data acquisition method and device | |
| CN115396280B (en) | Alarm data processing method, device, equipment and storage medium | |
| CN115604668B (en) | Short message sending and pushing monitoring method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication |