CN112988739B

CN112988739B - Data management and processing method, device, computer system and readable storage medium

Info

Publication number: CN112988739B
Application number: CN201911312980.8A
Authority: CN
Inventors: 王煜骢; 酆晓杰
Original assignee: Alibaba Group Holding Ltd
Current assignee: Alibaba Group Holding Ltd
Priority date: 2019-12-18
Filing date: 2019-12-18
Publication date: 2024-04-19
Anticipated expiration: 2039-12-18
Also published as: CN112988739A

Abstract

The embodiment of the disclosure discloses a data management and processing method, a device, a computer system and a readable storage medium. The data management method comprises the following steps: acquiring logic topology structures of a plurality of data acquisition devices, wherein the logic topology structures are determined according to grouping information of the plurality of data acquisition devices; acquiring a resource-node correspondence between computing resources and nodes in the logical topology structure; and determining a designated computing resource for processing the data acquired by the designated data acquisition device in the plurality of data acquisition devices according to the logic topology structure and the resource-node corresponding relation.

Description

Data management and processing method, device, computer system and readable storage medium

Technical Field

The present disclosure relates to the field of databases, and in particular, to a method, an apparatus, a computer system, and a readable storage medium for managing and processing data.

Background

The data generated in the operation of the database such as the log, the performance and the like has great value for each business on the cloud, the data needs to be collected, processed and stored, and various data services such as inquiry, analysis and the like are finally provided.

The prior business service system for analyzing big data of the log is characterized in that the log is transmitted to a data pipeline for caching through a network after being preprocessed, and the data is processed in the corresponding pipeline in a form of sub-business sub-queues.

Disclosure of Invention

To solve the problems in the related art, embodiments of the present disclosure provide a data management and processing method, apparatus, computer system, and readable storage medium.

In a first aspect, a data management method is provided in an embodiment of the present disclosure.

Specifically, the data management method includes:

Acquiring logic topology structures of a plurality of data acquisition devices, wherein the logic topology structures are determined according to grouping information of the plurality of data acquisition devices;

acquiring a resource-node correspondence between computing resources and nodes in the logical topology structure;

and determining a designated computing resource for processing the data acquired by the designated data acquisition device in the plurality of data acquisition devices according to the logic topology structure and the resource-node corresponding relation.

With reference to the first aspect, in a first implementation manner of the first aspect, the data acquisition device includes an acquisition entity; and/or

The data acquisition device comprises an acquisition agent and one or more acquisition entities; and/or

The packet information includes physical packet information and/or logical packet information of the data acquisition device.

With reference to the first aspect, in a second implementation manner of the first aspect, the logical topology includes a tree structure; and/or

The computing resources include data pipes.

With reference to the first aspect, in a third implementation manner of the first aspect, the determining, according to the logical topology and the resource-node correspondence, a specified computing resource for processing the data acquired by the specified data acquiring device includes:

traversing from the node of the logic topology structure corresponding to the specified data acquisition device from bottom to top along the logic topology structure, determining a first specified node with corresponding computing resources for processing the corresponding service data of the specified data acquisition device, and taking the computing resources corresponding to the specified node as the specified computing resources for processing the data acquired by the specified data acquisition device.

With reference to the first aspect, in a fourth implementation manner of the first aspect, the present disclosure further includes:

Acquiring state information of the computing resource;

determining one or more downstream nodes of the node corresponding to the computing resource under the condition that the state information of the computing resource meets a first preset condition;

and establishing a corresponding relation between the downstream node and another computing resource and updating the resource-node corresponding relation.

With reference to the fourth implementation manner of the first aspect, in a fifth implementation manner of the first aspect, the disclosure includes that the state information of the computing resource meets a first preset condition, including that a load of the computing resource reaches or exceeds a preset threshold; and/or

The other computing resource is different from the computing resources corresponding to other nodes in the logic topology structure, or the other computing resource is the computing resource meeting the second preset condition in the computing resources corresponding to other nodes in the logic topology structure.

With reference to the fifth implementation manner of the first aspect, in a sixth implementation manner of the first aspect, the second preset condition is that a load of a computing resource corresponding to the other node is smaller than a predetermined threshold.

With reference to the fourth implementation manner of the first aspect, in a seventh implementation manner of the first aspect, the determining, according to the logical topology and the resource-node correspondence, a specified computing resource for processing data acquired by a specified data acquisition device in the plurality of data acquisition devices includes:

and determining a designated computing resource for processing the data acquired by the designated data acquisition device according to the logic topology structure and the updated resource-node corresponding relation.

In a second aspect, a data processing method is provided in an embodiment of the present disclosure.

Specifically, the data processing method includes: acquiring data by a designated data acquisition device;

Determining a designated computing resource for processing the data, the designated computing resource being determined from a logical topology of a plurality of data acquisition devices and a resource-node correspondence of computing resources to nodes in the logical topology;

Transmitting the data to the designated computing resource for processing,

Wherein the logical topology is determined based on grouping information of the plurality of data acquisition devices.

With reference to the second aspect, in a first implementation manner of the second aspect, the data acquisition device includes an acquisition entity; and/or

With reference to the second aspect, in a second implementation manner of the second aspect, the logical topology includes a tree structure; and/or

The computing resources include data pipes.

With reference to the second aspect, in a third implementation manner of the second aspect, the determining the specified computing resource according to a logical topology structure of the plurality of data acquisition devices and a resource-node correspondence relationship between the computing resource and a node in the logical topology structure includes:

In a third aspect, a data management apparatus is provided in an embodiment of the present disclosure.

Specifically, the data management apparatus includes:

the logic topology acquisition module is used for acquiring the logic topologies of the plurality of data acquisition devices, wherein the logic topologies are determined according to the grouping information of the plurality of data acquisition devices;

The corresponding relation acquisition module is used for acquiring the resource-node corresponding relation between the computing resource and the nodes in the logic topological structure;

and the first computing resource determining module is used for determining the designated computing resource for processing the data acquired by the designated data acquisition device in the plurality of data acquisition devices according to the logic topological structure and the resource-node corresponding relation.

With reference to the third aspect, in a first implementation manner of the third aspect, the data acquisition device includes an acquisition entity; and/or

With reference to the third aspect, in a second implementation manner of the third aspect, the logical topology includes a tree structure; and/or

The computing resources include data pipes.

With reference to the third aspect, in a third implementation manner of the third aspect, the determining, according to the logical topology and the resource-node correspondence, a specified computing resource for processing the data acquired by the specified data acquiring device includes:

With reference to the third aspect, in a fourth implementation manner of the third aspect, the disclosure further includes:

a computing resource state information acquisition module, configured to acquire state information of the computing resource;

The node determining module is used for determining one or more downstream nodes of the nodes corresponding to the computing resources under the condition that the state information of the computing resources meets a first preset condition;

And the corresponding relation updating module is used for establishing a corresponding relation between the downstream node and another computing resource and updating the resource-node corresponding relation.

With reference to the fourth implementation manner of the third aspect, in a fifth implementation manner of the third aspect, the state information of the computing resource meets a first preset condition, including that a load of the computing resource reaches or exceeds a preset threshold; and/or

With reference to the fifth implementation manner of the third aspect, in a sixth implementation manner of the third aspect, the second preset condition is that a load of a computing resource corresponding to the other node is smaller than a predetermined threshold.

With reference to the fourth implementation manner of the third aspect, in a seventh implementation manner of the third aspect, the determining, according to the logical topology and the resource-node correspondence, a specified computing resource for processing data acquired by a specified data acquisition device in the plurality of data acquisition devices includes:

In a fourth aspect, a data processing apparatus is provided in an embodiment of the present disclosure.

Specifically, the data processing apparatus includes:

the data acquisition module is used for acquiring data through the designated data acquisition device;

A second computing resource determining module configured to determine a specified computing resource for processing the data, the specified computing resource being determined according to a logical topology of a plurality of data acquisition devices and a resource-node correspondence of the computing resource with a node in the logical topology;

A data transmission module for transmitting the data to the appointed computing resource for processing,

With reference to the fourth aspect, in a first implementation manner of the fourth aspect, the data acquisition device includes an acquisition entity; and/or

With reference to the fourth aspect, in a second implementation manner of the fourth aspect, the logical topology includes a tree structure; and/or

The computing resources include data pipes.

With reference to the fourth aspect, in a third implementation manner of the fourth aspect, the determining the specified computing resource according to a logical topology structure of the plurality of data acquisition devices and a resource-node correspondence relationship between the computing resource and a node in the logical topology structure includes:

In a fifth aspect, a computer system is provided in an embodiment of the present disclosure. Specifically, the computer system includes: a processor; a memory storing executable instructions which, when executed by a processor, perform the method steps of:

With reference to the fifth aspect, in a first implementation manner of the fifth aspect, the data acquisition device includes an acquisition entity; and/or

With reference to the fifth aspect, in a second implementation manner of the fifth aspect, the logical topology includes a tree structure; and/or

The computing resources include data pipes.

With reference to the fifth aspect, in a third implementation manner of the fifth aspect, the determining, according to the logical topology and the resource-node correspondence, a specified computing resource for processing the data acquired by the specified data acquiring device includes:

With reference to the fifth aspect, in a fourth implementation manner of the fifth aspect, the present disclosure further implements the following method steps when the executable instructions are executed by a processor:

Acquiring state information of the computing resource;

With reference to the fourth implementation manner of the fifth aspect, in a fifth implementation manner of the fifth aspect, the state information of the computing resource meets a first preset condition, including that a load of the computing resource reaches or exceeds a preset threshold; and/or

With reference to the fifth implementation manner of the fifth aspect, in a sixth implementation manner of the fifth aspect, the second preset condition is that a load of computing resources corresponding to the other nodes is smaller than a predetermined threshold.

With reference to the fourth implementation manner of the fifth aspect, in a seventh implementation manner of the fifth aspect, the determining, according to the logical topology and the resource-node correspondence, a specified computing resource for processing data acquired by a specified data acquisition device in the plurality of data acquisition devices includes:

In a sixth aspect, a computer system is provided in an embodiment of the present disclosure. Specifically, the computer system includes: a processor; a memory storing executable instructions which, when executed by a processor, perform the method steps of:

Acquiring data by a designated data acquisition device;

Transmitting the data to the designated computing resource for processing,

With reference to the sixth aspect, in a first implementation manner of the sixth aspect, the data acquisition device includes an acquisition entity; and/or

With reference to the sixth aspect, in a second implementation manner of the sixth aspect, the logical topology includes a tree structure; and/or

The computing resources include data pipes.

With reference to the sixth aspect, in a third implementation manner of the sixth aspect, the determining the specified computing resource according to a logical topology of the plurality of data acquisition devices and a resource-node correspondence of the computing resource with a node in the logical topology includes:

In a seventh aspect, a computer-readable storage medium is provided in an embodiment of the present disclosure.

In particular, the computer readable storage medium stores executable instructions that, when executed by a processor, implement the method according to any one of the first aspect, the first implementation of the first aspect, the seventh implementation of the first aspect, the second aspect, the first implementation of the second aspect, and the third implementation of the second aspect.

Drawings

Other objects and advantages of the present disclosure will become more apparent from the following detailed description of non-limiting embodiments, taken in conjunction with the accompanying drawings. In the drawings:

FIG. 1 shows a schematic diagram of an application scenario according to an embodiment of the present disclosure;

FIG. 2A illustrates a flow chart of a data management method according to an embodiment of the present disclosure;

FIG. 2B illustrates a schematic diagram of a logical topology of a plurality of data acquisition devices according to an embodiment of the present disclosure;

FIG. 3A illustrates a flow chart of a data management method according to an embodiment of the present disclosure;

FIG. 3B shows a schematic diagram of mounting a new pipe according to an embodiment of the present disclosure;

FIG. 4 shows a flow chart of a data processing method according to an embodiment of the present disclosure;

FIG. 5 shows a block diagram of a data management device according to an embodiment of the present disclosure;

FIG. 6 shows a block diagram of a data management device according to an embodiment of the present disclosure;

FIG. 7 shows a block diagram of a data processing apparatus according to an embodiment of the present disclosure;

FIG. 8 shows a block diagram of a computer system according to an embodiment of the present disclosure;

FIG. 9 illustrates a block diagram of a computer architecture suitable for use in implementing methods for data management and/or data processing in accordance with embodiments of the present disclosure.

Detailed Description

Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily implement them. In addition, for the sake of clarity, portions irrelevant to description of the exemplary embodiments are omitted in the drawings.

In this disclosure, it should be understood that terms such as "comprises" or "comprising," etc., are intended to indicate the presence of features, numbers, steps, acts, components, portions, or combinations thereof disclosed in this specification, and are not intended to exclude the possibility that one or more other features, numbers, steps, acts, components, portions, or combinations thereof are present or added.

In addition, it should be noted that, without conflict, the embodiments of the present disclosure and features of the embodiments may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings in conjunction with embodiments.

In the process of providing the disclosure, the inventor finds that in the existing business service system facing to log big data analysis, data is processed in corresponding pipelines in the form of sub-business sub-queues, management granularity is coarse, and the pipeline is easy to be fully loaded due to the increase of business data volume. Moreover, the existing business service system for log big data analysis does not have the capacity of expanding and splitting the pipeline, and has limited expansibility. The system has only task management and scheduling capability and lacks task monitoring capability. The present disclosure has been made in order to at least partially solve the above-mentioned problems.

Fig. 1 shows a schematic diagram of an application scenario according to an embodiment of the present disclosure. It is to be understood that the application scenario shown in fig. 1 is merely for the purpose of illustrating the concepts and principles of the present disclosure, and is not meant to suggest that the present disclosure is applicable only to such an application scenario.

As shown in fig. 1, an exemplary application scenario includes a configuration center 10, an acquisition end 20, a pipe management system 30, a data service system 40, a message channel 50, and a coordinator 60.

The configuration center 10 is used for maintaining meta information such as pipeline division, data routing, etc., and provides centralized configuration services for the acquisition end 20, the pipeline control system 30, the data service system 40, etc.

The acquisition end 20 is used for acquiring data.

The pipe management system 30 is used to provide data pipes and to enable creation, updating, splitting, destruction, etc. of data pipes and to monitor pipe status.

The data service system 40 is used to provide data query, analysis, etc. services.

The message channel 50 is used for uniformly collecting the state information of the collecting end 20 and the pipeline control system 30 and sending the state information to the coordinator 60, so as to realize decoupling of the collecting end 20, the pipeline control system 30 and the coordinator 60.

The coordinator 60 is used for collecting the states of the collection end 20 and the pipeline management system 30, and performing proper handling operations on errors and abnormal situations.

In fig. 1, the solid line link is a data link, and is used for data acquisition, storage and later acquisition and analysis; the broken line link is a control link, and the control function of the system is realized.

In step S101, the configuration center 10 provides a centralized configuration service to the acquisition end 20, the pipe management system 30, and the data service system 40.

In step S102, the collecting end 20 sends status information to the message channel 50;

in step S103, the pipe management and control system 30 transmits status information to the message channel 50;

in step S104, the coordinator 60 acquires status information from the message channel 50;

In step S105, the coordinator 60 issues a control instruction to the acquisition end 20 and the pipe management and control system 30 according to the status information.

Fig. 2A shows a flow chart of a data management method according to an embodiment of the present disclosure.

In step S201, a logical topology of a plurality of data acquisition devices is acquired, the logical topology being determined from grouping information of the plurality of data acquisition devices.

In step S202, a resource-node correspondence between computing resources and nodes in the logical topology is obtained.

In step S203, a specified computing resource for processing the data acquired by the specified data acquisition device of the plurality of data acquisition devices is determined according to the logical topology and the resource-node correspondence.

According to an embodiment of the present disclosure, the data management method may be implemented by the configuration center 10 shown in fig. 1, for example, the data acquisition device may include the acquisition end shown in fig. 1, and the computing resource may be any resource capable of processing data, for example, including a data pipe provided by the pipe management system 30 shown in fig. 1. Or the computing resources may also include, for example, any one or more of computing devices, processors, controllers, computing capabilities allocated in a cloud computing platform.

According to the embodiment of the disclosure, the data acquisition device acquires data, such as performance, log and other information from the running database host. The computing resources are used to process the data acquired by the data acquisition device. The data acquisition device groups according to a preset rule to form a specific logic topology structure. The computing resources are distributed in the logic topology structure and are mounted on nodes of the logic topology structure. According to the logic topology structure of the data acquisition device and the corresponding relation between the computing resources and the mounting nodes, the designated computing resources for processing the data acquired by the designated data acquisition device are determined, and then the routing information from the designated data acquisition device to the designated computing resources can also be determined.

According to an embodiment of the present disclosure, the logical topology includes a tree structure; and/or the computing resource comprises a data pipeline. For example, logical topologies include tree structures, as well as other topologies such as graphs. The computing resources include data pipes, as well as other data processing means.

Fig. 2B shows a schematic diagram of a logical topology of a plurality of data acquisition devices according to an embodiment of the present disclosure.

According to an embodiment of the present disclosure, the data acquisition device comprises an acquisition entity; and/or the data acquisition device comprises an acquisition agent and one or more acquisition entities; and/or the packet information comprises physical packet information and/or logical packet information of the data acquisition device.

For example, as shown in fig. 2B, the data acquisition device may include acquisition entities, such as acquisition entities A1, A2, B1, B2, C1, C2. On the same database server, one acquisition entity may be included, or a plurality of acquisition entities may be included, for the same or different types of data acquisition.

Or the data acquisition device may also include an acquisition agent and one or more acquisition entities. On the same database server, there may be one collection agent for data summarization of all collection entities on the database server. For example, acquisition entities A1, A2 and acquisition agent a may constitute one data acquisition device, acquisition entities B1, B2 and acquisition agent B may constitute one data acquisition device, and acquisition entities C1, C2 and acquisition agent C may constitute one data acquisition device.

According to the embodiment of the disclosure, the logic topology structure of a plurality of data acquisition devices can be determined according to grouping information of the plurality of data acquisition devices. The packet information of the data acquisition device includes physical packet information and/or logical packet information. For example, data acquisition devices located on the same physical host may be grouped into one physical packet and data acquisition devices located on the same physical host may be grouped into the same or different logical packets.

The logical topology may include a plurality of nodes determined by grouping information of the plurality of data acquisition devices.

As shown in fig. 2B, if the acquisition entity is used as the data acquisition means, the grouping information may include physical grouping information and logical grouping information. For example, acquisition entity A1 and acquisition entity A2 located on host a belong to physical group 1, acquisition entity B1 and acquisition entity B2 located on host B belong to physical group 2, and acquisition entity C1 and acquisition entity C2 located on host C belong to physical group 3. The logical grouping may be a grouping of physical groupings and/or data acquisition devices, as shown in fig. 2B, where clusters 1,2, 3,4 are logical groupings, where cluster 1 may include physical grouping 1, cluster 2 may include physical grouping 2, cluster 3 may include acquisition entity C1, and cluster 4 may include acquisition entity C2. Data subdomains 1,2 are upper level logical groupings of clusters 1,2, 3,4, data subdomain 1 may include cluster 1 and cluster 2, and data subdomain 2 may include cluster 3 and cluster 4. Logical groupings at the upper layer of the data subdomain are available areas, regions, and cells in sequence. The usable area 1 may include a data subfield 1 and the usable area 2 may include a data subfield 2. Zone 1 may include an available zone 1 and zone 2 may include an available zone 2. The cell may include region 1 and region 2. Accordingly, the logical topology shown in fig. 2B includes nodes from top to bottom: unit, area 1, area 2, available area 1, available area 2, data subdomain 1, data subdomain 2, cluster 1, cluster 2, cluster 3, cluster 4, physical group 1, physical group 2, acquisition entity C1, acquisition entity C2.

Alternatively, the physical grouping may not be performed for the data acquisition device, but may be performed for only the data acquisition device. For example, as shown in fig. 2B, cluster 1 may comprise acquisition entities A1, A2, cluster 2 may comprise acquisition entities B1, B2, cluster 3 may comprise acquisition entity C1, and cluster 4 may comprise acquisition entity C2. Data subdomains 1,2 are upper level logical groupings of clusters 1,2, 3, 4, data subdomain 1 may include cluster 1 and cluster 2, and data subdomain 2 may include cluster 3 and cluster 4. Logical groupings at the upper layer of the data subdomain are available areas, regions, and cells in sequence. The usable area 1 may include a data subfield 1 and the usable area 2 may include a data subfield 2. Zone 1 may include an available zone 1 and zone 2 may include an available zone 2. The cell may include region 1 and region 2. Accordingly, the logical topology shown in fig. 2B includes nodes from top to bottom: unit, area 1, area 2, available area 1, available area 2, data subdomain 1, data subdomain 2, cluster 1, cluster 2, cluster 3, cluster 4, acquisition entity A1, acquisition entity A2, acquisition entity B1, acquisition entity B2, acquisition entity C1, acquisition entity C2.

As shown in fig. 2B, if the data acquisition device includes an acquisition agent and an acquisition entity located on the same host, the packet information may include logical packet information. For example, as shown in fig. 2B, clusters 1,2,3, and 4 are logical groupings, where cluster 1 may include a logical acquisition device made up of acquisition entities A1, A2 and acquisition agent a, cluster 2 may include a logical acquisition device made up of acquisition entities B1, B2 and acquisition agent B, cluster 3 may include acquisition entity C1, and cluster 4 may include acquisition entity C2. Data subdomains 1,2 are upper level logical groupings of clusters 1,2,3,4, data subdomain 1 may include cluster 1 and cluster 2, and data subdomain 2 may include cluster 3 and cluster 4. Logical groupings at the upper layer of the data subdomain are available areas, regions, and cells in sequence. The usable area 1 may include a data subfield 1 and the usable area 2 may include a data subfield 2. Zone 1 may include an available zone 1 and zone 2 may include an available zone 2. The cell may include region 1 and region 2. Accordingly, the logical topology shown in fig. 2B includes nodes from top to bottom: unit, area 1, area 2, available area 1, available area 2, data subdomain 1, data subdomain 2, cluster 1, cluster 2, cluster 3, cluster 4, data acquisition device composed of acquisition entities A1, A2 and acquisition agent a, data acquisition device composed of acquisition entities B1, B2 and acquisition agent B, acquisition entity C1, acquisition entity C2.

It is noted that in case the data acquisition device comprises an acquisition agent and an acquisition entity, the data acquisition device may comprise one or more acquisition entities. The data acquisition device may comprise other components than the acquisition agent and the acquisition entity in addition to or in addition to the acquisition entity. Or the data acquisition device may take other forms as long as the data acquisition device is capable of performing a data acquisition function.

According to embodiments of the present disclosure, a computing resource (e.g., a data pipe) may have a correspondence with a node in a logical topology. For example, the performance data pipe 1 and the log data pipe 1 are mounted on the area 1, and the performance data pipe 2 and the log data pipe 2 are mounted on the area 2.

According to an embodiment of the present disclosure, determining a specified computing resource for processing data acquired by the specified data acquisition device according to the logical topology and the resource-node correspondence, includes: traversing from the node of the logic topology structure corresponding to the specified data acquisition device from bottom to top along the logic topology structure, determining a first specified node with corresponding computing resources for processing the corresponding service data of the specified data acquisition device, and taking the computing resources corresponding to the specified node as the specified computing resources for processing the data acquired by the specified data acquisition device. For example, in the tree structure, the specified data acquisition means is located on a specified leaf node of the lowest level. In order to determine which computing resource the data acquired by the designated data acquisition device is stored in, the configuration center may traverse from the designated leaf node in the tree structure from the bottom up along the tree structure to find the first node on which the data pipe for processing the service corresponding to the designated data acquisition device is mounted, and use the pipe as the designated computing resource for processing the data acquired by the designated data acquisition device. For example, in fig. 2B, for the log data acquired by the designated data acquisition device (for example, the acquisition entity A1), the log data pipe 1 mounted in the found area 1 may be processed by traversing up the tree structure from the leaf node where the acquisition entity A1 is located, and then the log data pipe 1 is used as a designated computing resource for processing the log data acquired by the acquisition entity 1.

According to the embodiment of the present disclosure, after determining a node of a data pipe that processes a corresponding service of a designated data acquisition device, routing information for transmitting data to the pipe may be determined accordingly.

Unlike the prior art, the present disclosure does not merely divide pipelines by traffic, but rather defines a more general data processing scheme based on the logical topology of the data acquisition device, which is more convenient for expansion, organization, and management of computing resources.

Fig. 3A shows a flow chart of a data management method according to an embodiment of the present disclosure.

Fig. 3A includes steps S204 to S206 in addition to steps S201 to S203 similar to fig. 2.

In step S204, status information of the computing resource is obtained;

In step S205, determining one or more downstream nodes of the nodes corresponding to the computing resource, where the state information of the computing resource meets a first preset condition;

in step S206, a correspondence between the downstream node and another computing resource is established and the resource-node correspondence is updated.

According to the embodiment of the disclosure, the state information of the computing resource is acquired, one or more downstream nodes of the node corresponding to the computing resource are determined under the condition that the state information of the computing resource meets a first preset condition, a corresponding relation between the downstream nodes and another computing resource is established, and the resource-node corresponding relation is updated. For example, the coordinator may obtain status information of the data pipe through a message channel. And when the state information meets a first preset condition, determining one or more downstream nodes of the node on which the data pipeline is mounted, mounting another data pipeline on the downstream nodes, and updating the resource-node corresponding relation.

According to the embodiment of the disclosure, the state information of the computing resource meets a first preset condition, including that the load of the computing resource reaches or exceeds a preset threshold; and/or the other computing resource is different from the computing resource corresponding to other nodes in the logic topological structure, or the other computing resource is the computing resource meeting the second preset condition in the computing resource corresponding to other nodes in the logic topological structure. For example, the first preset condition may be that the data pipe is overloaded, or other preset conditions. At this point, another data pipe is mounted on one or more downstream nodes of the node on which the overloaded data pipe is mounted. For example, assuming that the load of the node N on which the overloaded data pipe is mounted is W, another data pipe may be mounted on the downstream node M whose load of the node N is greater than or equal to W/2 to reduce the load of the node N.

The other data pipe mounted to the downstream node may be different from the data pipe mounted on the other nodes in the tree topology, i.e. the newly built data pipe. Or the other data pipe may be a data pipe mounted on another node of the tree topology, which satisfies the second preset condition. Or the further data pipe may be a data pipe that fulfils other conditions.

According to an embodiment of the present disclosure, the second preset condition is that a load of a computing resource corresponding to the other node is smaller than a predetermined threshold. For example, the second preset condition may be that the load of the data pipe mounted by the other node is smaller than a predetermined threshold, i.e. an already existing light-load data pipe. At this time, the light-load data pipeline can be mounted on the downstream node, the load on the overload node is shunted, and the resource-node corresponding relation is correspondingly updated.

Fig. 3B shows a schematic diagram of mounting a new pipe according to an embodiment of the present disclosure.

In fig. 3B, it is assumed that the state of the log data pipe 2 is overloaded, and the cluster 4 is found as a hot spot node with larger data traffic at the lower layer of the overloaded node. A new log data pipe 3 may be mounted on the cluster 4, and the data in the collecting entity C2 is stored in the log data pipe 3, i.e. the route information from the collecting entity C2 to the log data pipe 2 is updated as follows: from the collection entity C2 to the log data pipe 3.

According to an embodiment of the present disclosure, the determining, according to the logical topology and the resource-node correspondence, a specified computing resource for processing data acquired by a specified data acquisition device of the plurality of data acquisition devices includes: and determining a designated computing resource for processing the data acquired by the designated data acquisition device according to the logic topology structure and the updated resource-node corresponding relation. Since the resource-node correspondence is changed, it is possible to determine a specified computing resource for processing the data acquired by the specified data acquisition means based on the logical topology and the updated resource-node correspondence, and to determine the corresponding routing information.

According to the embodiment of the disclosure, the state information of the computing resources is used for monitoring, and the new computing resources are correspondingly set or the existing computing resources are reallocated and utilized, so that the timely and convenient computing resource expansion is realized, the problem of limited expansibility is solved, and the reliability of data processing and the system stability are ensured.

Fig. 4 shows a flow chart of a data processing method according to an embodiment of the present disclosure.

In step S401, data is acquired by the specified data acquisition means;

In step S402, determining a specified computing resource for processing the data, the specified computing resource being determined according to a logical topology of a plurality of data acquisition devices and a resource-node correspondence of the computing resource with a node in the logical topology;

In step S403, the data is transmitted to the specified computing resource for processing.

For example, the specified data acquisition device acquires performance or log data on the database host. The computing resource end determines a designated data pipeline for processing performance or log data through configuration information of the configuration center. The specified data pipe is determined based on the logical topology and resource-node correspondence of the plurality of data acquisition devices. The performance or log data is then transferred to the designated data pipe for processing.

According to an embodiment of the present disclosure, the logical topology includes a tree structure; and/or the computing resource comprises a data pipeline. For example, the logical topology includes a tree structure, but may be other topologies such as a graph. The computing resources include data pipes, as well as other data processing means.

According to an embodiment of the disclosure, determining the specified computing resource according to a logical topology of a plurality of data acquisition devices and a resource-node correspondence of the computing resource with nodes in the logical topology includes: traversing from the node of the logic topology structure corresponding to the specified data acquisition device from bottom to top along the logic topology structure, determining a first specified node with corresponding computing resources for processing the corresponding service data of the specified data acquisition device, and taking the computing resources corresponding to the specified node as the specified computing resources for processing the data acquired by the specified data acquisition device.

For example, in a tree structure, the specified data acquisition device is located on a specified leaf node at the lowest level of the tree. In order to determine which computing resource the data acquired by the designated data acquisition device is stored in, the configuration center may traverse from the designated leaf node in the tree structure from the bottom up along the tree structure to find the first node on which the data pipe for processing the service corresponding to the designated data acquisition device is mounted, and use the pipe as the designated computing resource for processing the data acquired by the designated data acquisition device.

According to the embodiment of the disclosure, the pipelining scheme based on the tree structure is maintained by the configuration center, pipeline division information and routing information are issued to the acquisition end and the pipeline management and control system, and the convenience expansion of the pipeline is realized by combining the coordinator, so that error processing and abnormal recovery capability is provided, and the expandability and reliability of the whole data link are greatly improved.

Fig. 5 illustrates a block diagram of a data management apparatus according to an embodiment of the present disclosure.

In fig. 5, the data management apparatus 500 includes:

A logic topology acquisition module 501, configured to acquire a logic topology of a plurality of data acquisition devices, where the logic topology is determined according to grouping information of the plurality of data acquisition devices;

a correspondence acquiring module 502, configured to acquire a resource-node correspondence between a computing resource and a node in the logical topology;

A first computing resource determining module 503, configured to determine a specified computing resource for processing the data acquired by the specified data acquiring device in the plurality of data acquiring devices according to the logical topology and the resource-node correspondence.

According to an embodiment of the present disclosure, the logical topology includes a tree structure; and/or the computing resource comprises a data pipeline.

According to an embodiment of the present disclosure, determining a specified computing resource for processing data acquired by the specified data acquisition device according to the logical topology and the resource-node correspondence, includes: traversing from the node of the logic topology structure corresponding to the specified data acquisition device from bottom to top along the logic topology structure, determining a first specified node with corresponding computing resources for processing the corresponding service data of the specified data acquisition device, and taking the computing resources corresponding to the specified node as the specified computing resources for processing the data acquired by the specified data acquisition device.

Fig. 6 illustrates a block diagram of a data management apparatus according to an embodiment of the present disclosure.

In fig. 6, the data management apparatus 600 includes, in addition to the same logical topology acquisition module 501, correspondence acquisition module 502, and first computing resource determination module 503 as in fig. 5, a logic topology acquisition module that includes:

a computing resource status information acquisition module 504, configured to acquire status information of the computing resource;

a node determining module 505, configured to determine one or more downstream nodes of a node corresponding to the computing resource, where the state information of the computing resource meets a first preset condition;

the correspondence updating module 506 is configured to establish a correspondence between the downstream node and another computing resource and update the resource-node correspondence.

According to the embodiment of the disclosure, the state information of the computing resource meets a first preset condition, including that the load of the computing resource reaches or exceeds a preset threshold; and/or the other computing resource is different from the computing resource corresponding to other nodes in the logic topological structure, or the other computing resource is the computing resource meeting the second preset condition in the computing resource corresponding to other nodes in the logic topological structure.

According to an embodiment of the present disclosure, the second preset condition is that a load of a computing resource corresponding to the other node is smaller than a predetermined threshold.

According to an embodiment of the present disclosure, the determining, according to the logical topology and the resource-node correspondence, a specified computing resource for processing data acquired by a specified data acquisition device of the plurality of data acquisition devices includes: and determining a designated computing resource for processing the data acquired by the designated data acquisition device according to the logic topology structure and the updated resource-node corresponding relation.

Fig. 7 shows a block diagram of a data processing apparatus according to an embodiment of the present disclosure.

In fig. 7, a data processing apparatus 700 includes:

A data acquisition module 701 for acquiring data by designating a data acquisition device;

A second computing resource determining module 702 configured to determine a specified computing resource for processing the data, the specified computing resource being determined according to a logical topology of a plurality of data acquisition devices and a resource-node correspondence of the computing resource with a node in the logical topology;

a data transmission module 703, configured to transmit the data to the specified computing resource for processing.

Fig. 8 shows a block diagram of a computer system according to an embodiment of the present disclosure.

As shown in fig. 8, the computer system 800 may include one or more processors 801 and one or more memories 802. The one or more memories 802 are configured to store one or more executable instructions that, when executed by the one or more processors 801, may implement the steps of:

A method of data management, comprising: acquiring logic topology structures of a plurality of data acquisition devices, wherein the logic topology structures are determined according to grouping information of the plurality of data acquisition devices; acquiring a resource-node correspondence between computing resources and nodes in the logical topology structure; and determining a designated computing resource for processing the data acquired by the designated data acquisition device in the plurality of data acquisition devices according to the logic topology structure and the resource-node corresponding relation.

According to an embodiment of the present disclosure, when the executable instructions are executed by a processor, the following steps are also implemented: acquiring state information of the computing resource; determining one or more downstream nodes of the node corresponding to the computing resource under the condition that the state information of the computing resource meets a first preset condition; and establishing a corresponding relation between the downstream node and another computing resource and updating the resource-node corresponding relation.

According to an embodiment of the present disclosure, the one or more memories 802 are configured to store one or more executable instructions that, when executed by the one or more processors 801, may implement the steps of:

A method of data processing, comprising: acquiring data by a designated data acquisition device; determining a designated computing resource for processing the data, the designated computing resource being determined from a logical topology of a plurality of data acquisition devices and a resource-node correspondence of computing resources to nodes in the logical topology; transmitting the data to the designated computing resource for processing, wherein: the logical topology is determined based on grouping information of the plurality of data acquisition devices.

According to embodiments of the present disclosure, the data management and/or processing apparatus described above may be implemented in a distributed database system. The distributed database system may be implemented using multiple computers.

As shown in fig. 9, the computer system 900 includes a processor (CPU) 901 that can execute the above-described method in accordance with a program stored in a Read Only Memory (ROM) 902 or a program loaded from a storage section 908 into a Random Access Memory (RAM) 903. In the RAM 903, various programs and data necessary for the operation of the system 900 are also stored. The CPU 901, ROM 902, and RAM 903 are connected to each other through a bus 904. An input/output (I/O) interface 905 is also connected to the bus 904.

The following components are connected to the I/O interface 905: an input section 906 including a keyboard, a mouse, and the like; an output portion 907 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and a speaker; a storage portion 908 including a hard disk or the like; and a communication section 909 including a network interface card such as a LAN card, a modem, or the like. The communication section 909 performs communication processing via a network such as the internet. The drive 910 is also connected to the I/O interface 905 as needed. A removable medium 911 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is installed as needed on the drive 910 so that a computer program read out therefrom is installed into the storage section 908 as needed.

In addition, the methods described above may be implemented as computer software programs according to embodiments of the present disclosure. For example, embodiments of the present disclosure include a computer program product comprising a computer program tangibly embodied on a medium readable thereby, the computer program comprising program code for performing the method described above. In such an embodiment, the computer program may be downloaded and installed from the network through the communication section 909, and/or installed from the removable medium 911.

According to embodiments of the present disclosure, one computer architecture as described above may be employed to implement a method according to embodiments of the present disclosure, and a plurality of computer architectures as described above may be employed to cooperate with one another to implement a method according to embodiments of the present disclosure.

The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The units or modules referred to in the embodiments of the present disclosure may be implemented in software or in programmable hardware. The units or modules described may also be provided in a processor, the names of which in some cases do not constitute a limitation of the unit or module itself.

As another aspect, the present disclosure also provides a computer-readable storage medium, which may be a computer-readable storage medium included in the apparatus described in the above embodiment; or may be a computer-readable storage medium, alone, that is not assembled into a device. The computer-readable storage medium stores one or more programs for use by one or more processors in performing the methods described in the present disclosure.

The foregoing description is only of the preferred embodiments of the present disclosure and description of the principles of the technology being employed. It will be appreciated by those skilled in the art that the scope of the invention referred to in this disclosure is not limited to the specific combination of features described above, but encompasses other embodiments in which any combination of features described above or their equivalents is contemplated without departing from the inventive concepts described. Such as those described above, are mutually substituted with the technical features having similar functions disclosed in the present disclosure (but not limited thereto).

Claims

1. A method of data management, comprising:

Determining a specified computing resource for processing the data acquired by the specified data acquisition device in the plurality of data acquisition devices according to the logic topology structure and the resource-node correspondence;

Wherein the computing resource comprises a data pipeline.

2. The method according to claim 1, characterized in that:

the data acquisition device comprises an acquisition entity; and/or

3. The method according to claim 1, characterized in that:

The logical topology includes a tree structure.

4. The method of claim 1, wherein determining a specified computing resource for processing the data acquired by the specified data acquisition device based on the logical topology and the resource-node correspondence comprises:

5. The method as recited in claim 1, further comprising:

Acquiring state information of the computing resource;

6. The method according to claim 5, wherein:

The state information of the computing resource meets a first preset condition, wherein the load of the computing resource reaches or exceeds a preset threshold value; and/or

7. The method of claim 6, wherein the second preset condition is that a load of computing resources corresponding to the other nodes is less than a predetermined threshold.

8. The method of claim 5, wherein determining a specified computing resource for processing data acquired by a specified data acquisition device of the plurality of data acquisition devices based on the logical topology and the resource-node correspondence comprises:

9. A method of data processing, comprising:

Acquiring data by a designated data acquisition device;

Transmitting the data to the designated computing resource for processing,

Wherein the logical topology is determined from grouping information of the plurality of data acquisition devices; the computing resources include data pipes.

10. The method according to claim 9, wherein:

the data acquisition device comprises an acquisition entity; and/or

11. The method according to claim 9, wherein:

The logical topology includes a tree structure.

12. The method of claim 9, wherein determining the specified computing resources from a logical topology of a plurality of data acquisition devices and a resource-node correspondence of computing resources to nodes in the logical topology comprises:

13. A data management apparatus, comprising:

A first computing resource determining module configured to determine a specified computing resource for processing data acquired by a specified data acquiring device of the plurality of data acquiring devices according to the logical topology and the resource-node correspondence;

Wherein the computing resource comprises a data pipeline.

14. The data management apparatus according to claim 13, further comprising:

15. A data processing apparatus, comprising:

16. A computer system, comprising:

A processor;

Memory storing executable instructions which, when executed by a processor, implement the data management method according to any one of claims 1 to 8 or implement the data processing method according to any one of claims 9 to 12.

17. A computer readable storage medium storing executable instructions which, when executed by a processor, implement the data management method according to any one of claims 1 to 8 or implement the data processing method according to any one of claims 9 to 12.