CN112988739A

CN112988739A - Data management and processing method, device, computer system and readable storage medium

Info

Publication number: CN112988739A
Application number: CN201911312980.8A
Authority: CN
Inventors: 王煜骢; 酆晓杰
Original assignee: Alibaba Group Holding Ltd
Current assignee: Alibaba Group Holding Ltd
Priority date: 2019-12-18
Filing date: 2019-12-18
Publication date: 2021-06-18
Anticipated expiration: 2039-12-18
Also published as: CN112988739B

Abstract

The embodiment of the disclosure discloses a data management and processing method, a data management and processing device, a computer system and a readable storage medium. The data management method comprises the following steps: acquiring logical topologies of a plurality of data acquisition devices, wherein the logical topologies are determined according to grouping information of the plurality of data acquisition devices; acquiring a resource-node corresponding relation between a computing resource and a node in the logical topological structure; determining a designated computing resource for processing data acquired by a designated data acquisition device of the plurality of data acquisition devices based on the logical topology and the resource-node correspondence.

Description

Data management and processing method, device, computer system and readable storage medium

Technical Field

The present disclosure relates to the field of databases, and in particular, to a data management and processing method, apparatus, computer system, and readable storage medium.

Background

Data generated in database operation such as logs, performance and the like has great value on each service on the cloud, and the data needs to be acquired, processed and stored, and various data services such as query, analysis and the like are finally provided.

In the existing service system for big log data analysis, after being preprocessed, logs are transmitted to a data pipeline through a network for caching, and data are processed in the corresponding pipeline in a form of sub-service sub-queues.

Disclosure of Invention

In order to solve the problems in the related art, embodiments of the present disclosure provide a data management and processing method, apparatus, computer system, and readable storage medium.

In a first aspect, an embodiment of the present disclosure provides a data management method.

Specifically, the data management method includes:

acquiring logical topologies of a plurality of data acquisition devices, wherein the logical topologies are determined according to grouping information of the plurality of data acquisition devices;

acquiring a resource-node corresponding relation between a computing resource and a node in the logical topological structure;

determining a designated computing resource for processing data acquired by a designated data acquisition device of the plurality of data acquisition devices based on the logical topology and the resource-node correspondence.

With reference to the first aspect, in a first implementation manner of the first aspect, the data acquisition apparatus includes an acquisition entity; and/or

The data acquisition device comprises an acquisition agent and one or more acquisition entities; and/or

The grouping information includes physical grouping information and/or logical grouping information of the data acquisition apparatus.

With reference to the first aspect, in a second implementation manner of the first aspect, the logical topology includes a tree structure; and/or

The computing resources include data pipes.

With reference to the first aspect, in a third implementation manner of the first aspect, the determining, according to the logical topology and the resource-node correspondence, a designated computing resource for processing the data acquired by the designated data acquiring apparatus includes:

and traversing from the node of the logic topological structure corresponding to the specified data acquisition device to the top along the logic topological structure, determining a first specified node with corresponding computing resources for processing the corresponding service data of the specified data acquisition device, and taking the computing resources corresponding to the specified node as the specified computing resources for processing the data acquired by the specified data acquisition device.

With reference to the first aspect, in a fourth implementation manner of the first aspect, the present disclosure further includes:

acquiring state information of the computing resource;

determining one or more downstream nodes of the nodes corresponding to the computing resources under the condition that the state information of the computing resources meets a first preset condition;

establishing a correspondence between the downstream node and another computing resource and updating the resource-node correspondence.

With reference to the fourth implementation manner of the first aspect, in a fifth implementation manner of the first aspect, the state information of the computing resource meets a first preset condition, where the condition includes that a load of the computing resource reaches or exceeds a preset threshold; and/or

The other computing resource is different from the computing resources corresponding to the other nodes in the logical topological structure, or the other computing resource is a computing resource which meets a second preset condition in the computing resources corresponding to the other nodes in the logical topological structure.

With reference to the fifth implementation manner of the first aspect, in a sixth implementation manner of the first aspect, the second preset condition is that a load of the computing resource corresponding to the other node is less than a predetermined threshold.

With reference to the fourth implementation manner of the first aspect, in a seventh implementation manner of the first aspect, the determining, according to the logical topology and the resource-node correspondence, a designated computing resource for processing data acquired by a designated data acquisition device of the plurality of data acquisition devices includes:

and determining the designated computing resources for processing the data acquired by the designated data acquisition device according to the logical topology and the updated resource-node correspondence.

In a second aspect, a data processing method is provided in an embodiment of the present disclosure.

Specifically, the data processing method includes: acquiring data through a specified data acquisition device;

determining a designated computing resource for processing the data, the designated computing resource determined from a logical topology of a plurality of data acquisition devices and resource-node correspondence of computing resources to nodes in the logical topology;

transmitting the data to the designated computing resource for processing,

wherein the logical topology is determined from grouping information of the plurality of data acquisition devices.

With reference to the second aspect, in a first implementation manner of the second aspect, the data acquisition apparatus includes an acquisition entity; and/or

With reference to the second aspect, in a second implementation manner of the second aspect, the logical topology includes a tree structure; and/or

The computing resources include data pipes.

With reference to the second aspect, in a third implementation manner of the second aspect, the determining the designated computing resource according to a logical topology of a plurality of data acquisition devices and a resource-node correspondence between the computing resource and a node in the logical topology includes:

In a third aspect, an embodiment of the present disclosure provides a data management apparatus.

Specifically, the data management apparatus includes:

a logical topology acquisition module, configured to acquire logical topologies of a plurality of data acquisition devices, where the logical topologies are determined according to grouping information of the plurality of data acquisition devices;

a correspondence obtaining module for obtaining a resource-node correspondence between a computing resource and a node in the logical topology;

a first computing resource determining module, configured to determine, according to the logical topology and the resource-node correspondence, a designated computing resource for processing data acquired by a designated data acquisition device of the plurality of data acquisition devices.

With reference to the third aspect, in a first implementation manner of the third aspect, the data acquisition apparatus includes an acquisition entity; and/or

With reference to the third aspect, the present disclosure provides in a second implementation manner of the third aspect, wherein the logical topology includes a tree structure; and/or

The computing resources include data pipes.

With reference to the third aspect, in a third implementation manner of the third aspect, the determining, according to the logical topology and the resource-node correspondence, a designated computing resource for processing the data acquired by the designated data acquiring apparatus includes:

With reference to the third aspect, in a fourth implementation manner of the third aspect, the present disclosure further includes:

the computing resource state information acquisition module is used for acquiring the state information of the computing resource;

the node determining module is used for determining one or more downstream nodes of the node corresponding to the computing resource under the condition that the state information of the computing resource meets a first preset condition;

and the corresponding relation updating module is used for establishing the corresponding relation between the downstream node and another computing resource and updating the resource-node corresponding relation.

With reference to the fourth implementation manner of the third aspect, in a fifth implementation manner of the third aspect, the status information of the computing resource meets a first preset condition, where the first preset condition includes that a load of the computing resource reaches or exceeds a preset threshold; and/or

With reference to the fifth implementation manner of the third aspect, in a sixth implementation manner of the third aspect, the second preset condition is that a load of the computing resource corresponding to the other node is less than a predetermined threshold.

With reference to the fourth implementation manner of the third aspect, in a seventh implementation manner of the third aspect, the determining, according to the logical topology and the resource-node correspondence, a designated computing resource for processing data acquired by a designated data acquisition device of the plurality of data acquisition devices includes:

In a fourth aspect, a data processing apparatus is provided in an embodiment of the present disclosure.

Specifically, the data processing apparatus includes:

the data acquisition module is used for acquiring data through the specified data acquisition device;

a second computing resource determination module to determine a designated computing resource for processing the data, the designated computing resource determined from a logical topology of a plurality of data acquisition devices and a resource-node correspondence of computing resources to nodes in the logical topology;

a data transmission module for transmitting the data to the designated computing resource for processing,

With reference to the fourth aspect, in a first implementation manner of the fourth aspect, the data acquisition apparatus includes an acquisition entity; and/or

With reference to the fourth aspect, the present disclosure provides in a second implementation manner of the fourth aspect, the logical topology includes a tree structure; and/or

The computing resources include data pipes.

With reference to the fourth aspect, in a third implementation manner of the fourth aspect, the determining the designated computing resource according to a logical topology of a plurality of data acquisition devices and a resource-node correspondence between the computing resource and a node in the logical topology includes:

In a fifth aspect, a computer system is provided in embodiments of the present disclosure. Specifically, the computer system includes: a processor; a memory storing executable instructions that, when executed by the processor, perform the method steps of:

With reference to the fifth aspect, in a first implementation manner of the fifth aspect, the data acquisition apparatus includes an acquisition entity; and/or

With reference to the fifth aspect, in a second implementation form of the fifth aspect, the logical topology includes a tree structure; and/or

The computing resources include data pipes.

With reference to the fifth aspect, in a third implementation manner of the fifth aspect, the determining, according to the logical topology and the resource-node correspondence, a specific computing resource for processing the data acquired by the specific data acquisition apparatus includes:

With reference to the fifth aspect, in a fourth implementation manner of the fifth aspect, when the executable instructions are executed by the processor, the following method steps are further implemented:

acquiring state information of the computing resource;

With reference to the fourth implementation manner of the fifth aspect, in a fifth implementation manner of the fifth aspect, the status information of the computing resource meets a first preset condition, which includes that the load of the computing resource reaches or exceeds a preset threshold; and/or

With reference to the fifth implementation manner of the fifth aspect, in a sixth implementation manner of the fifth aspect, the second preset condition is that a load of the computing resource corresponding to the other node is less than a predetermined threshold.

With reference to the fourth implementation manner of the fifth aspect, in a seventh implementation manner of the fifth aspect, the determining, according to the logical topology and the resource-node correspondence, a designated computing resource for processing data acquired by a designated data acquisition apparatus of the plurality of data acquisition apparatuses includes:

In a sixth aspect, a computer system is provided in embodiments of the present disclosure. Specifically, the computer system includes: a processor; a memory storing executable instructions that, when executed by the processor, perform the method steps of:

acquiring data through a specified data acquisition device;

transmitting the data to the designated computing resource for processing,

With reference to the sixth aspect, in a first implementation manner of the sixth aspect, the data acquisition apparatus includes an acquisition entity; and/or

With reference to the sixth aspect, the present disclosure provides in a second implementation form of the sixth aspect, the logical topology includes a tree structure; and/or

The computing resources include data pipes.

With reference to the sixth aspect, in a third implementation manner of the sixth aspect, the determining the specified computing resources according to a logical topology of a plurality of data acquisition devices and resource-node correspondence between the computing resources and nodes in the logical topology includes:

In a seventh aspect, a computer-readable storage medium is provided in the disclosed embodiments.

In particular, the computer-readable storage medium stores executable instructions that, when executed by a processor, implement the method according to any one of the first aspect, the first implementation manner of the first aspect to the seventh implementation manner of the first aspect, the second aspect, and the first implementation manner of the second aspect to the third implementation manner of the second aspect.

Drawings

Other objects and advantages of the present disclosure will become more apparent from the following detailed description of non-limiting embodiments thereof, when taken in conjunction with the accompanying drawings. In the drawings:

FIG. 1 shows a schematic diagram of an application scenario according to an embodiment of the present disclosure;

FIG. 2A shows a flow diagram of a data management method according to an embodiment of the present disclosure;

FIG. 2B shows a schematic diagram of a logical topology of a plurality of data acquisition devices according to an embodiment of the present disclosure;

FIG. 3A shows a flow diagram of a data management method according to an embodiment of the present disclosure;

FIG. 3B shows a schematic diagram of mounting a new pipe according to an embodiment of the present disclosure;

FIG. 4 shows a flow diagram of a data processing method according to an embodiment of the present disclosure;

FIG. 5 shows a block diagram of a data management device according to an embodiment of the present disclosure;

FIG. 6 shows a block diagram of a data management device according to an embodiment of the present disclosure;

FIG. 7 shows a block diagram of a data processing apparatus according to an embodiment of the present disclosure;

FIG. 8 shows a block diagram of a computer system according to an embodiment of the present disclosure;

FIG. 9 illustrates a block diagram of a computer architecture suitable for implementing methods for data management and/or data processing, according to embodiments of the present disclosure.

Detailed Description

Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily implement them. Also, for the sake of clarity, parts not relevant to the description of the exemplary embodiments are omitted in the drawings.

In the present disclosure, it is to be understood that terms such as "including" or "having," etc., are intended to indicate the presence of the disclosed features, numbers, steps, behaviors, components, parts, or combinations thereof, and are not intended to preclude the possibility that one or more other features, numbers, steps, behaviors, components, parts, or combinations thereof may be present or added.

It should be further noted that the embodiments and features of the embodiments in the present disclosure may be combined with each other without conflict. The present disclosure will be described in detail below with reference to the accompanying drawings in conjunction with embodiments.

In the process of providing the present disclosure, the inventor finds that, in the existing service system for log big data analysis, data is processed in a corresponding pipeline in a form of sub-service sub-queues, the management granularity is coarse, and the increase of the service data volume easily causes full pipeline. Moreover, the existing service system for analyzing big log data does not have the capacity expansion and splitting capabilities of pipelines, and the expansibility is limited. The system only has task management and scheduling capability and lacks task monitoring capability. The present disclosure is made to at least partially solve the above problems.

Fig. 1 shows a schematic diagram of an application scenario according to an embodiment of the present disclosure. It is to be understood that the application scenario illustrated in fig. 1 is merely for illustrating the concepts and principles of the present disclosure, and is not meant to imply that the present disclosure is only applicable to such application scenario.

As shown in fig. 1, an exemplary application scenario includes a configuration center 10, an acquisition end 20, a pipeline management system 30, a data service system 40, a message channel 50, and a coordinator 60.

The configuration center 10 is used for maintaining meta-information such as pipeline division and data routing, and providing centralized configuration services for the acquisition end 20, the pipeline management and control system 30, the data service system 40, and the like.

The acquisition end 20 is used for acquiring data.

The pipeline management and control system 30 is used for providing data pipelines, realizing creation, updating, splitting, destruction and the like of the data pipelines, and monitoring the pipeline state.

The data service system 40 is used for providing services such as data query and analysis.

The message channel 50 is used for uniformly collecting status information of the acquisition end 20 and the pipeline management and control system 30 and sending the status information to the coordinator 60, so as to decouple the acquisition end 20, the pipeline management and control system 30 and the coordinator 60.

The coordinator 60 is used to collect the states of the collection terminal 20 and the pipeline management system 30 and to make appropriate actions against errors and abnormal situations.

In fig. 1, the solid link is a data link, and is used for data acquisition, storage, and later acquisition and analysis; the dotted link is a control link to realize the control function of the system.

In step S101, the configuration center 10 provides centralized configuration services to the acquisition end 20, the pipeline management system 30, and the data service system 40.

In step S102, the acquisition end 20 sends status information to the message channel 50;

in step S103, the pipeline management system 30 sends status information to the message channel 50;

in step S104, the coordinator 60 acquires status information from the message channel 50;

in step S105, the coordinator 60 issues a control command to the acquisition terminal 20 and the pipeline management system 30 based on the state information.

Fig. 2A illustrates a flow diagram of a data management method according to an embodiment of the present disclosure.

In step S201, a logical topology of a plurality of data acquisition devices is acquired, the logical topology being determined based on grouping information of the plurality of data acquisition devices.

In step S202, a resource-node correspondence of computing resources and nodes in the logical topology is obtained.

In step S203, a designated computing resource for processing the data acquired by a designated data acquisition device of the plurality of data acquisition devices is determined according to the logical topology and the resource-node correspondence.

According to the embodiment of the present disclosure, the data management method may be implemented by, for example, the configuration center 10 shown in fig. 1, the data acquisition device may include the acquisition end shown in fig. 1, and the computing resource may be any resource capable of processing data, for example, a data pipeline provided by the pipeline management and control system 30 shown in fig. 1. Alternatively, the computing resources may also include, for example, any one or more of computing devices, processors, controllers, allocated computing power in a cloud computing platform.

According to the embodiment of the disclosure, the data acquisition device acquires data, such as performance, log and other information from the running database host. The computing resources are used for processing the data acquired by the data acquisition device. The data acquisition devices are grouped according to preset rules to form a specific logic topological structure. The computing resources are distributed in the logical topology and mounted on the nodes of the logical topology. According to the logical topology structure of the data acquisition device and the corresponding relation between the computing resource and the mounting node, the designated computing resource for processing the data acquired by the designated data acquisition device is determined, and further, the routing information from the designated data acquisition device to the designated computing resource can be determined.

According to an embodiment of the present disclosure, the logical topology includes a tree structure; and/or the computing resource comprises a data pipe. For example, the logical topology includes a tree structure, but may also include other topologies such as a graph. The computing resources include data pipes, and may also include other data processing means.

FIG. 2B shows a schematic diagram of a logical topology of a plurality of data acquisition devices according to an embodiment of the present disclosure.

According to an embodiment of the present disclosure, the data acquisition device includes an acquisition entity; and/or the data acquisition device comprises an acquisition agent and one or more acquisition entities; and/or the grouping information comprises physical grouping information and/or logical grouping information of the data acquisition device.

For example, as shown in fig. 2B, the data acquisition apparatus may include acquisition entities, such as acquisition entities a1, a2, B1, B2, C1, C2. One acquisition entity or a plurality of acquisition entities can be included on the same database server and used for acquiring data of the same or different types.

Alternatively, the data acquisition device may also include an acquisition agent and one or more acquisition entities. There may be one collection agent on the same database server for data summarization of all collection entities on the database server. For example, the collection entities a1, a2 and the collection agent a may constitute a data acquisition apparatus, the collection entities B1, B2 and the collection agent B may constitute a data acquisition apparatus, and the collection entities C1, C2 and the collection agent C may constitute a data acquisition apparatus.

According to the embodiment of the present disclosure, the logical topology of the plurality of data acquisition devices may be determined according to grouping information of the plurality of data acquisition devices. The grouping information of the data acquisition device includes physical grouping information and/or logical grouping information. For example, data acquisition devices located on the same physical host may be grouped into one physical grouping and data acquisition devices located on the same physical host may be grouped into the same or different logical groupings.

The logical topology may include a plurality of nodes determined by grouping information of the plurality of data acquisition devices.

As shown in fig. 2B, if the collection entity is used as the data acquisition device, the grouping information may include physical grouping information and logical grouping information. For example, the collection entity a1 and collection entity a2 located on host a belong to physical group 1, the collection entity B1 and collection entity B2 located on host B belong to physical group 2, and the collection entity C1 and collection entity C2 located on host C belong to physical group 3. The logical grouping may be a grouping of physical and/or data acquisition devices, as shown in fig. 2B,

clusters

1, 2, 3, 4 are logical groupings, where cluster 1 may include physical grouping 1, cluster 2 may include physical grouping 2, cluster 3 may include collection entity C1, and cluster 4 may include collection entity C2.

Data subdomains

1, 2 are upper layer logical groupings of

clusters

1, 2, 3, 4, data subdomain 1 may include cluster 1 and cluster 2, and data subdomain 2 may include cluster 3 and cluster 4. The logical grouping of the upper layers of the data sub-fields is, in turn, available area, region and cell. The available region 1 may include a data subfield 1, and the available region 2 may include a data subfield 2. Area 1 may include available area 1 and area 2 may include available area 2. The cell may include region 1 and region 2. Accordingly, the logical topology shown in fig. 2B includes, from top to bottom, nodes: unit, region 1, region 2, available region 1, available region 2, data subdomain 1, data subdomain 2, cluster 1, cluster 2, cluster 3, cluster 4, physical grouping 1, physical grouping 2, acquisition entity C1, acquisition entity C2.

Alternatively, the grouping may be performed not physically for the data acquisition apparatuses but logically only for the data acquisition apparatuses. For example, as shown in fig. 2B, cluster 1 may include acquisition entities a1, a2, cluster 2 may include acquisition entities B1, B2, cluster 3 may include acquisition entity C1, and cluster 4 may include acquisition entity C2.

Data subdomains

1, 2 are upper layer logical groupings of

clusters

1, 2, 3, 4, data subdomain 1 may include cluster 1 and cluster 2, and data subdomain 2 may include cluster 3 and cluster 4. The logical grouping of the upper layers of the data sub-fields is, in turn, available area, region and cell. The available region 1 may include a data subfield 1, and the available region 2 may include a data subfield 2. Area 1 may include available area 1 and area 2 may include available area 2. The cell may include region 1 and region 2. Accordingly, the logical topology shown in fig. 2B includes, from top to bottom, nodes: a unit, a region 1, a region 2, an available region 1, an available region 2, a data subfield 1, a data subfield 2, a cluster 1, a cluster 2, a cluster 3, a cluster 4, an acquisition entity a1, an acquisition entity a2, an acquisition entity B1, an acquisition entity B2, an acquisition entity C1, and an acquisition entity C2.

As shown in fig. 2B, if the data acquisition device includes an acquisition agent and an acquisition entity located on the same host, the grouping information may include logical grouping information. For example, as shown in fig. 2B,

clusters

1, 2, 3, and 4 are logical groups, where cluster 1 may include a logical acquisition apparatus composed of acquisition entities a1, a2 and acquisition agent a, cluster 2 may include a logical acquisition apparatus composed of acquisition entities B1, B2 and acquisition agent B, cluster 3 may include acquisition entity C1, and cluster 4 may include acquisition entity C2.

Data subdomains

1, 2 are upper layer logical groupings of

clusters

1, 2, 3, 4, data subdomain 1 may include cluster 1 and cluster 2, and data subdomain 2 may include cluster 3 and cluster 4. The logical grouping of the upper layers of the data sub-fields is, in turn, available area, region and cell. The available region 1 may include a data subfield 1, and the available region 2 may include a data subfield 2. Area 1 may include available area 1 and area 2 may include available area 2. The cell may include region 1 and region 2. Accordingly, the logical topology shown in fig. 2B includes, from top to bottom, nodes: the system comprises units, an area 1, an area 2, an available area 1, an available area 2, a data subdomain 1, a data subdomain 2, a cluster 1, a cluster 2, a cluster 3, a cluster 4, a data acquisition device composed of acquisition entities A1, A2 and an acquisition agent A, a data acquisition device composed of acquisition entities B1, B2 and an acquisition agent B, an acquisition entity C1 and an acquisition entity C2.

It is noted that in case the data acquisition arrangement comprises an acquisition agent and an acquisition entity, the data acquisition arrangement may comprise one or more acquisition entities. The data acquisition device may include other components besides the acquisition agent and the acquisition entity, in addition to the acquisition entity or the acquisition agent and the acquisition entity. Alternatively, the data acquisition device may take other forms as long as the data acquisition device is capable of performing the data acquisition function.

According to embodiments of the present disclosure, a computing resource (e.g., a data pipe) may have a correspondence with a node in a logical topology. For example, the performance data pipe 1 and the log data pipe 1 are mounted on the area 1, and the performance data pipe 2 and the log data pipe 2 are mounted on the area 2.

According to an embodiment of the present disclosure, determining, according to the logical topology and the resource-node correspondence, a designated computing resource for processing the data acquired by the designated data acquisition device includes: and traversing from the node of the logic topological structure corresponding to the specified data acquisition device to the top along the logic topological structure, determining a first specified node with corresponding computing resources for processing the corresponding service data of the specified data acquisition device, and taking the computing resources corresponding to the specified node as the specified computing resources for processing the data acquired by the specified data acquisition device. For example, in the tree structure, the specified data acquisition means is located on a specified leaf node of the lowermost layer. In order to determine which computing resource the data acquired by the specified data acquisition device is stored in, the configuration center may traverse along the tree structure from bottom to top starting from the specified leaf node in the tree structure, find a first node on which a data pipe for processing a corresponding service of the specified data acquisition device is mounted, and use the pipe as the specified computing resource for processing the data acquired by the specified data acquisition device. For example, in fig. 2B, for log data obtained by a specified data obtaining device (e.g., the collecting entity a1), the tree structure may be traversed upward from the leaf node where the collecting entity a1 is located, and the log data pipeline 1 mounted in the found area 1 may process the log data, and then the log data pipeline 1 is used as a specified computing resource for processing the log data obtained by the collecting entity 1.

According to the embodiment of the disclosure, after the node of the data pipe for processing the corresponding service of the specified data acquisition device is determined, the routing information for transmitting the data to the pipe can be determined accordingly.

Different from the prior art, the method and the device do not only divide the pipelines according to the services, but define a more general data processing scheme based on the logical topological structure of the data acquisition device, and are more convenient for the expansion, organization and management of computing resources.

FIG. 3A illustrates a flow diagram of a data management method according to an embodiment of the disclosure.

Fig. 3A adds steps S204 to S206 in addition to steps S201 to S203 which are the same as those of fig. 2.

In step S204, state information of the computing resource is acquired;

in step S205, when the state information of the computing resource satisfies a first preset condition, determining one or more downstream nodes of the node corresponding to the computing resource;

in step S206, a correspondence between the downstream node and another computing resource is established and the resource-node correspondence is updated.

According to the embodiment of the disclosure, state information of the computing resource is acquired, one or more downstream nodes of a node corresponding to the computing resource are determined under the condition that the state information of the computing resource meets a first preset condition, a corresponding relationship between the downstream node and another computing resource is established, and the resource-node corresponding relationship is updated. For example, the coordinator may acquire status information of the data pipe through the message channel. When the state information meets a first preset condition, one or more downstream nodes of the nodes mounted by the data pipeline are determined, another data pipeline is mounted on the downstream nodes, and the resource-node corresponding relation is updated.

According to the embodiment of the disclosure, the state information of the computing resource meets a first preset condition, including that the load of the computing resource reaches or exceeds a preset threshold; and/or the another computing resource is different from the computing resources corresponding to the other nodes in the logical topology structure, or the another computing resource is a computing resource which meets a second preset condition in the computing resources corresponding to the other nodes in the logical topology structure. For example, the first preset condition may be that the data pipe is overloaded, or may be another preset condition. At this time, another data pipe is mounted on one or more downstream nodes of the node on which the overloaded data pipe is mounted. For example, assuming that the load of the node N on which the overloaded data pipe is mounted is W, another data pipe may be mounted on the downstream node M whose load of the node N is greater than or equal to W/2 to reduce the load of the node N.

The other data pipe mounted to the downstream node may be different from the data pipes mounted to other nodes in the tree topology, that is, a newly built data pipe. Alternatively, the another data pipe may be a data pipe mounted on another node of the tree topology, and the second preset condition is satisfied. Alternatively, the other data pipe may be a data pipe satisfying other conditions.

According to the embodiment of the present disclosure, the second preset condition is that the load of the computing resource corresponding to the other node is less than a predetermined threshold. For example, the second preset condition may be that the load of the data pipe mounted by other nodes is less than a predetermined threshold, i.e. a light-load data pipe already exists. At this time, the light-load data pipeline can be mounted to the downstream node, the load on the overload node is shunted, and the resource-node corresponding relation is updated correspondingly.

FIG. 3B shows a schematic diagram of mounting a new pipe according to an embodiment of the present disclosure.

In fig. 3B, it is assumed that the status of the log data pipe 2 is overloaded, and the cluster 4 is found as a hot spot node with a large data traffic at the lower layer of the overloaded node. A new log data pipe 3 may be mounted on the cluster 4 and the data in the collecting entity C2 is stored in the log data pipe 3, i.e. the routing information from the collecting entity C2 to the log data pipe 2 is updated to: from the collecting entity C2 to the log data pipe 3.

According to an embodiment of the present disclosure, the determining, according to the logical topology and the resource-node correspondence, a designated computing resource for processing data acquired by a designated data acquisition device of the plurality of data acquisition devices includes: and determining the designated computing resources for processing the data acquired by the designated data acquisition device according to the logical topology and the updated resource-node correspondence. As the resource-node correspondence changes, the specified computing resources for processing the data acquired by the specified data acquisition device may be determined based on the logical topology and the updated resource-node correspondence, and corresponding routing information determined.

According to the embodiment of the invention, the state information of the computing resources is monitored, and new computing resources are correspondingly set or the existing computing resources are redistributed and utilized, so that the timely and convenient computing resource expansion is realized, the problem of limited expansibility is solved, and the reliability of data processing and the system stability are ensured.

Fig. 4 shows a flow chart of a data processing method according to an embodiment of the present disclosure.

In step S401, data is acquired by specifying a data acquisition means;

in step S402, determining specified computing resources for processing the data, the specified computing resources being determined from a logical topology of a plurality of data acquisition devices and resource-node correspondence of computing resources to nodes in the logical topology;

in step S403, the data is transmitted to the specified computing resource for processing.

For example, the specified data acquisition device acquires performance or log data on the database host. The computing resource side determines a designated data pipe for processing performance or log data through configuration information of the configuration center. The designated data pipe is determined based on the logical topology and resource-node correspondence of the plurality of data acquisition devices. The performance or log data transfer is then specified for processing in the data pipe.

According to an embodiment of the present disclosure, the logical topology includes a tree structure; and/or the computing resource comprises a data pipe. For example, the logical topology includes a tree structure, but may be other topologies such as a graph. The computing resources include data pipes, and may also include other data processing means.

According to an embodiment of the present disclosure, determining the designated computing resource according to a logical topology and a resource-node correspondence of computing resources of a plurality of data acquisition devices to nodes in the logical topology includes: and traversing from the node of the logic topological structure corresponding to the specified data acquisition device to the top along the logic topological structure, determining a first specified node with corresponding computing resources for processing the corresponding service data of the specified data acquisition device, and taking the computing resources corresponding to the specified node as the specified computing resources for processing the data acquired by the specified data acquisition device.

For example, in the tree structure, the specified data acquisition means is located at a specified leaf node of the lowermost layer of the tree. In order to determine which computing resource the data acquired by the specified data acquisition device is stored in, the configuration center may traverse along the tree structure from bottom to top starting from the specified leaf node in the tree structure, find a first node on which a data pipe for processing a corresponding service of the specified data acquisition device is mounted, and use the pipe as the specified computing resource for processing the data acquired by the specified data acquisition device.

According to the embodiment of the disclosure, in the tree structure-based pipelining scheme, the configuration center maintains and issues pipeline division information and routing information to the acquisition end and the pipeline management and control system, and the convenient expansion of the pipeline is realized by combining the coordinator, so that the error processing and exception recovery capabilities are provided, and the expandability and reliability of the whole data link are greatly improved.

Fig. 5 illustrates a structure diagram of a data management apparatus according to an embodiment of the present disclosure.

In fig. 5, the data management apparatus 500 includes:

a logical topology obtaining module 501, configured to obtain logical topologies of a plurality of data obtaining apparatuses, where the logical topologies are determined according to grouping information of the plurality of data obtaining apparatuses;

a correspondence obtaining module 502, configured to obtain a resource-node correspondence between a computing resource and a node in the logical topology;

a first computing resource determining module 503, configured to determine, according to the logical topology and the resource-node correspondence, a specified computing resource for processing data acquired by a specified data acquisition apparatus of the plurality of data acquisition apparatuses.

According to an embodiment of the present disclosure, the logical topology includes a tree structure; and/or the computing resource comprises a data pipe.

According to an embodiment of the present disclosure, determining, according to the logical topology and the resource-node correspondence, a designated computing resource for processing the data acquired by the designated data acquisition device includes: and traversing from the node of the logic topological structure corresponding to the specified data acquisition device to the top along the logic topological structure, determining a first specified node with corresponding computing resources for processing the corresponding service data of the specified data acquisition device, and taking the computing resources corresponding to the specified node as the specified computing resources for processing the data acquired by the specified data acquisition device.

Fig. 6 illustrates a structure diagram of a data management apparatus according to an embodiment of the present disclosure.

In fig. 6, the data management apparatus 600 includes, in addition to the same logical topology obtaining module 501, the correspondence obtaining module 502, and the first computing resource determining module 503 as those in fig. 5:

a computing resource status information obtaining module 504, configured to obtain status information of the computing resource;

a node determining module 505, configured to determine one or more downstream nodes of a node corresponding to the computing resource when the state information of the computing resource satisfies a first preset condition;

a correspondence update module 506, configured to establish a correspondence between the downstream node and another computing resource and update the resource-node correspondence.

According to the embodiment of the disclosure, the state information of the computing resource meets a first preset condition, including that the load of the computing resource reaches or exceeds a preset threshold; and/or the another computing resource is different from the computing resources corresponding to the other nodes in the logical topology structure, or the another computing resource is a computing resource which meets a second preset condition in the computing resources corresponding to the other nodes in the logical topology structure.

According to the embodiment of the present disclosure, the second preset condition is that the load of the computing resource corresponding to the other node is less than a predetermined threshold.

According to an embodiment of the present disclosure, the determining, according to the logical topology and the resource-node correspondence, a designated computing resource for processing data acquired by a designated data acquisition device of the plurality of data acquisition devices includes: and determining the designated computing resources for processing the data acquired by the designated data acquisition device according to the logical topology and the updated resource-node correspondence.

Fig. 7 shows a block diagram of a data processing apparatus according to an embodiment of the present disclosure.

In fig. 7, a data processing apparatus 700 includes:

a data acquisition module 701, configured to acquire data through a specified data acquisition device;

a second computing resource determination module 702 for determining specified computing resources for processing the data, the specified computing resources being determined from a logical topology of a plurality of data acquisition devices and resource-node correspondence of computing resources to nodes in the logical topology;

a data transmission module 703, configured to transmit the data to the specified computing resource for processing.

FIG. 8 shows a block diagram of a computer system, according to an embodiment of the present disclosure.

As shown in fig. 8, the computer system 800 may include one or more processors 801 and one or more memories 802. The one or more memories 802 are used to store one or more executable instructions that, when executed by the one or more processors 801, may perform the following steps:

a method for managing data, comprising: acquiring logical topologies of a plurality of data acquisition devices, wherein the logical topologies are determined according to grouping information of the plurality of data acquisition devices; acquiring a resource-node corresponding relation between a computing resource and a node in the logical topological structure; determining a designated computing resource for processing data acquired by a designated data acquisition device of the plurality of data acquisition devices based on the logical topology and the resource-node correspondence.

According to an embodiment of the disclosure, when the executable instructions are executed by a processor, the following steps are also implemented: acquiring state information of the computing resource; determining one or more downstream nodes of the nodes corresponding to the computing resources under the condition that the state information of the computing resources meets a first preset condition; establishing a correspondence between the downstream node and another computing resource and updating the resource-node correspondence.

The one or more memories 802 are configured to store one or more executable instructions that, when executed by the one or more processors 801, may perform the following steps, according to embodiments of the disclosure:

a data processing method, comprising: acquiring data through a specified data acquisition device; determining a designated computing resource for processing the data, the designated computing resource determined from a logical topology of a plurality of data acquisition devices and resource-node correspondence of computing resources to nodes in the logical topology; transmitting the data to the designated computing resource for processing, wherein: the logical topology is determined from grouping information of the plurality of data acquisition devices.

According to an embodiment of the present disclosure, the data management and/or processing apparatus may be implemented in a distributed database system. The distributed database system may be implemented using multiple computers.

As shown in fig. 9, the computer system 900 includes a processor (CPU)901, which can execute the above-described method according to a program stored in a Read Only Memory (ROM)902 or a program loaded from a storage section 908 into a Random Access Memory (RAM) 903. In the RAM 903, various programs and data necessary for the operation of the system 900 are also stored. The CPU 901, ROM 902, and RAM 903 are connected to each other via a bus 904. An input/output (I/O) interface 905 is also connected to bus 904.

The following components are connected to the I/O interface 905: an input portion 906 including a keyboard, a mouse, and the like; an output section 907 including components such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage portion 908 including a hard disk and the like; and a communication section 909 including a network interface card such as a LAN card, a modem, or the like. The communication section 909 performs communication processing via a network such as the internet. The drive 910 is also connected to the I/O interface 905 as necessary. A removable medium 911 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 910 as necessary, so that a computer program read out therefrom is mounted into the storage section 908 as necessary.

In addition, the methods described above may be implemented as computer software programs, in accordance with embodiments of the present disclosure. For example, embodiments of the present disclosure include a computer program product comprising a computer program tangibly embodied on a medium readable thereby, the computer program comprising program code for performing the above-described method. In such an embodiment, the computer program may be downloaded and installed from a network via the communication section 909, and/or installed from the removable medium 911.

According to the embodiment of the present disclosure, the method according to the embodiment of the present disclosure may be implemented by using one computer architecture as described above, or may be implemented by using a plurality of computer architectures as described above in cooperation with each other.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The units or modules described in the embodiments of the present disclosure may be implemented by software or by programmable hardware. The units or modules described may also be provided in a processor, and the names of the units or modules do not in some cases constitute a limitation of the units or modules themselves.

As another aspect, the present disclosure also provides a computer-readable storage medium, which may be the computer-readable storage medium included in the apparatus in the above-described embodiment; or it may be a separate computer readable storage medium not incorporated into the device. The computer readable storage medium stores one or more programs for use by one or more processors in performing the methods described in the present disclosure.

The foregoing description is only exemplary of the preferred embodiments of the disclosure and is illustrative of the principles of the technology employed. It will be appreciated by those skilled in the art that the scope of the invention in the present disclosure is not limited to the specific combination of the above-mentioned features, but also encompasses other embodiments in which any combination of the above-mentioned features or their equivalents is possible without departing from the inventive concept. For example, the above features and (but not limited to) the features disclosed in this disclosure having similar functions are replaced with each other to form the technical solution.

Claims

1. A method for managing data, comprising:

2. The method of claim 1, wherein:

the data acquisition device comprises an acquisition entity; and/or

3. The method of claim 1, wherein:

the logical topology comprises a tree structure; and/or

The computing resources include data pipes.

4. The method of claim 1, wherein determining the designated computing resources for processing the data obtained by the designated data obtaining device based on the logical topology and the resource-node correspondence comprises:

5. The method of claim 1, further comprising:

acquiring state information of the computing resource;

6. The method of claim 5, wherein:

the state information of the computing resource meets a first preset condition, including that the load of the computing resource reaches or exceeds a preset threshold value; and/or

7. The method according to claim 6, wherein the second preset condition is that the load of the computing resources corresponding to the other nodes is less than a predetermined threshold.

8. The method of claim 5, wherein determining the designated computing resources for processing the data obtained by the designated one of the plurality of data obtaining devices based on the logical topology and the resource-node correspondence comprises:

9. A data processing method, comprising:

acquiring data through a specified data acquisition device;

transmitting the data to the designated computing resource for processing,

10. The method of claim 9, wherein:

the data acquisition device comprises an acquisition entity; and/or

11. The method of claim 9, wherein:

the logical topology comprises a tree structure; and/or

The computing resources include data pipes.

12. The method of claim 9, wherein determining the specified computing resources based on a logical topology of a plurality of data acquisition devices and resource-node correspondence of computing resources to nodes in the logical topology comprises:

13. A data management apparatus, comprising:

14. The data management device according to claim 13, further comprising:

15. A data processing apparatus, comprising:

16. A computer system, comprising:

a processor;

a memory storing executable instructions which, when executed by the processor, implement the transaction processing method of any one of claims 1 to 12.

17. A computer readable storage medium storing executable instructions which, when executed by a processor, implement a transaction processing method according to any one of claims 1 to 12.