CN116112499B

CN116112499B - Construction method of data acquisition system and data acquisition method

Info

Publication number: CN116112499B
Application number: CN202310390402.6A
Authority: CN
Inventors: 夏全军; 姚宁坡; 孙敬松
Original assignee: Sprixin Technology Co ltd
Current assignee: Sprixin Technology Co ltd
Priority date: 2023-04-13
Filing date: 2023-04-13
Publication date: 2023-06-20
Anticipated expiration: 2043-04-13
Also published as: CN116112499A

Abstract

The invention provides a construction method of a data acquisition system and a data acquisition method, and relates to the technical field of data processing, wherein the construction method of the data acquisition system comprises the following steps: determining node capacity of each server node for data acquisition, and creating at least one management fragment, wherein the fragment attribute of the management fragment comprises fragment capacity and fragment copy number; according to the capacity of the fragments, each channel is respectively allocated to the management fragments, the allocated management fragments are obtained, and the channels represent physical links for data interaction; and according to the node capacity and the number of the fragment copies, deploying each allocated management fragment on each server node to obtain a data acquisition system. The invention can flexibly cope with data acquisition systems of various scales, solves the unlimited horizontal expansion of the acquisition scale while controlling the hardware cost, ensures the high-efficiency availability of channels through a slicing mechanism, and is also applicable to embedded devices; the compatibility and the flexibility of the data acquisition system are improved.

Description

Construction method of data acquisition system and data acquisition method

Technical Field

The present invention relates to the field of data processing technologies, and in particular, to a method for constructing a data acquisition system and a data acquisition method.

Background

With the development of power systems, the need for monitoring energy storage power stations is increasingly prominent. The traditional energy storage power station mainly relies on a plurality of secondary monitoring systems in the station to monitor and manage the whole operation state of the power station.

However, since a plurality of secondary monitoring systems are used for monitoring the power station, the problems of non-uniform management standard, non-communication of data among the stations, large daily management workload and the like exist, and the requirement of fine management cannot be met.

Therefore, a novel power station management scheme capable of efficiently and stably accessing operation data of multiple sets of monitoring systems is needed, unified data management, unified operation monitoring and unified return management of a power station are achieved, and management and control capability of operation of the power station is improved.

Disclosure of Invention

Aiming at the problems existing in the prior art, the embodiment of the invention provides a construction method of a data acquisition system and a data acquisition method.

The invention provides a construction method of a data acquisition system, which comprises the following steps:

determining node capacity of each server node for data acquisition, and creating at least one management fragment, wherein the fragment attribute of the management fragment comprises fragment capacity and fragment copy number;

according to the capacity of the fragments, each channel is respectively allocated to the management fragments to obtain allocated management fragments, and the channels represent physical links of data interaction;

And according to the node capacity and the number of the partitioned copies, deploying each distributed management partition on each server node to obtain a data acquisition system.

According to the method for constructing the data acquisition system provided by the invention, each allocated management fragment is deployed on each server node according to the node capacity and the fragment copy number to obtain the data acquisition system, and the method comprises the following steps:

determining the deployment number of the distributed management fragments according to the number of the distributed management fragments copies of any distributed management fragments;

and respectively deploying the distributed management fragments on the target server nodes with the number of fragments deployed to obtain a data acquisition system, wherein the target server nodes are server nodes with the number of deployed fragments smaller than the capacity of the nodes.

According to the method for constructing the data acquisition system provided by the invention, each channel is respectively allocated to the management fragment according to the fragment capacity to obtain the allocated management fragment, and the method comprises the following steps:

for any channel, distributing the channel to the target management fragment when the target management fragment exists in the at least one management fragment, wherein the target management fragment is a management fragment with the number of distributed channels smaller than the capacity of the fragment;

In case that the target management slice does not exist in the at least one management slice, a new management slice is created and the channel is allocated to the new management slice.

According to the construction method of the data acquisition system provided by the invention, the slicing attribute further comprises a slicing identifier;

accordingly, the allocating the channel to the target management slice in the case that the target management slice exists in the at least one management slice includes:

and in the case that the target management fragment exists in the at least one management fragment, allocating the channel to the target management fragment with the minimum fragment identification.

The construction method of the data acquisition system provided by the invention further comprises the following steps:

and under the condition that a node quantity adjusting instruction is received, carrying out quantity adjustment on the server nodes in the data acquisition system.

The invention also provides a data acquisition method, which is applied to the data acquisition system constructed by the construction method of the data acquisition system, and comprises the following steps:

responding to a data acquisition instruction, and starting each management fragment deployed on each server node in the data acquisition system, wherein the fragment attribute of each management fragment comprises a fragment name;

Selecting a main management fragment from at least one designated management fragment with the same fragment name through a distributed coordination unit;

and starting the channels on each main management partition to acquire data.

According to the data acquisition method provided by the invention, the main management fragment is selected from the at least one designated management fragment through the distributed coordination unit, and the method comprises the following steps:

determining the request time of each distributed lock request received by the distributed coordination unit, wherein the distributed lock request is sent to the distributed coordination unit after the specified management fragment is started;

and determining the designated management fragments corresponding to the distributed lock request with the earliest request time as a master management fragment, and determining other management fragments except the master management fragment in the at least one designated management fragment as standby management fragments.

The data acquisition method provided by the invention further comprises the following steps:

and under the condition that the abnormality of the main management fragments is detected, determining a new main management fragment from the standby management fragments according to each standby distributed lock request received by the distributed coordination unit, wherein the standby distributed lock request is periodically sent to the distributed coordination unit for the standby management fragments.

The invention also provides a construction device of the data acquisition system, which comprises:

a first determining module configured to determine node capacities of respective server nodes for data acquisition and create at least one management slice, a slice attribute of the management slice including a slice capacity and a slice copy number;

the distribution module is configured to distribute each channel to the management fragments according to the fragment capacity to obtain distributed management fragments, and the channels represent physical links of data interaction;

the deployment module is configured to deploy each allocated management fragment on each server node according to the node capacity and the fragment copy number to obtain a data acquisition system.

The invention also provides a data acquisition device, which is applied to the data acquisition system constructed by the construction method of the data acquisition system, and comprises the following steps:

the first starting module is configured to respond to a data acquisition instruction and start each management fragment deployed on each server node in the data acquisition system, wherein the fragment attribute of each management fragment comprises a fragment name;

the selecting module is configured to select a main management fragment from at least one appointed management fragment with the same fragment name through the distributed coordination unit;

And the second starting module is configured to start the channels on each main management partition to acquire data.

The invention also provides an electronic device comprising a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein the processor realizes the construction method or the data acquisition method of any one of the data acquisition systems when executing the program.

The invention also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements a method of constructing a data acquisition system or a method of data acquisition as described in any of the above.

The invention also provides a computer program product comprising a computer program which, when executed by a processor, implements a method of constructing or a method of data acquisition as described in any of the above.

According to the construction method and the data acquisition method of the data acquisition system, the channels are distributed to the management fragments, and the management fragments are deployed on the server nodes, so that the relation between the channels and the server nodes is decoupled, the management units of the server nodes are changed into the management fragments, the server nodes are ensured to only need to maintain the management fragments deployed on the server nodes, and each server node only maintains and acquires the data of the channels in the management fragments on the server nodes; because the main and standby and the expansion of the channel are both in the unit of management fragmentation, the robustness of data acquisition adapting to different data acquisition system scales is very good, the system can be distributed by a single machine or multiple machines, the system changing flow of server node expansion and contraction is simplified, and the lightweight and horizontal expansion of the data acquisition system is ensured.

In addition, the data acquisition system constructed by the construction method of the data acquisition system provided by the invention realizes management fragmentation of a small data acquisition scene, can fully utilize the hardware performance of a server node, makes up the bottleneck of single-machine single-acquisition process performance to a great extent, and improves the overall availability of the data acquisition system and the reliability of acquired data by managing multiple preparations of the fragmentation in a large data acquisition scene. The construction method of the data acquisition system can flexibly cope with data acquisition systems of various scales, not only can meet the scene of single machine multi-acquisition process, but also can meet the expansion of distributed multi-machine acquisition, and solve the unlimited horizontal expansion of the acquisition scale while controlling the hardware cost. And the efficient availability of the channel is ensured through a slicing mechanism. Due to the lightweight of the method, the method is also applicable to the embedded device, and the compatibility and the flexibility of a data acquisition system are improved.

Drawings

In order to more clearly illustrate the invention or the technical solutions of the prior art, the following description will briefly explain the drawings used in the embodiments or the description of the prior art, and it is obvious that the drawings in the following description are some embodiments of the invention, and other drawings can be obtained according to the drawings without inventive effort for a person skilled in the art.

FIG. 1 is a schematic flow chart of a method for constructing a data acquisition system according to the present invention;

FIG. 2 is one of the structural intents of the data acquisition system provided by the present invention;

FIG. 3 is a second flow chart of a method for constructing a data acquisition system according to the present invention;

FIG. 4 is a schematic diagram of a second embodiment of a data acquisition system according to the present invention;

FIG. 5 is a third view of the structural intent of the data acquisition system provided by the present invention;

FIG. 6 is a schematic flow chart of a data acquisition method according to the present invention;

FIG. 7 is a second flow chart of the data collection method according to the present invention;

FIG. 8 is a schematic diagram of a construction device of the data acquisition system according to the present invention;

FIG. 9 is a schematic diagram of a data acquisition device according to the present invention;

fig. 10 is a schematic structural diagram of an electronic device provided by the present invention.

Detailed Description

For the purpose of making the objects, technical solutions and advantages of the present invention more apparent, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is apparent that the described embodiments are some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

In order to facilitate a clearer understanding of various embodiments of the present invention, some relevant background knowledge is first presented as follows.

With the development of power systems, the need for monitoring energy storage power stations is increasingly prominent. The traditional energy storage power station mainly relies on a plurality of secondary monitoring systems in the station to monitor and manage the whole operation state of the power station. Because a plurality of secondary monitoring systems are used for monitoring the power station, the problems of non-uniform management standard, non-communication of data among the stations, large daily management workload and the like exist, and the requirement of fine management cannot be met. Therefore, a novel power station management scheme capable of efficiently and stably accessing operation data of multiple sets of monitoring systems is needed, unified data management, unified operation monitoring and unified return management of a power station are achieved, and management and control capability of operation of the power station is improved.

In order to achieve the goal of accessing multiple monitoring systems, there are two common methods: the first way is that a higher-configuration server is adopted to improve the processing capacity of the system, and a single server deploys a plurality of sets of acquisition programs to achieve the purpose. The second way is to increase the number of acquisition servers and share the acquisition channels so as to achieve the purpose of improving the overall access capability.

However, when the first method is adopted for data acquisition, the cost is increased in a series with the hardware (server) reaching a certain level, and the problem of single-point fault of the system is more and more obvious due to centralized acquisition, the problem investigation is complicated, the abnormality of one channel can cause the breakdown of the whole acquisition system, and the annual availability index of the system is reduced. The second method for collecting multiple services requires that the capacity of the node is increased conveniently, the probability of system faults is increased along with the increase of network nodes, and a high-availability scheme is required to be provided, so that the method is complex in design and heavy in weight, and the robustness to small-scale items is poor. Therefore, the invention provides a construction method of a data acquisition system and a data acquisition method to solve the problems.

The method of constructing the data acquisition system of the present invention is described below in conjunction with fig. 1-5.

Fig. 1 is a schematic flow chart of a method for constructing a data acquisition system according to the present invention, and referring to fig. 1, the method includes steps 101 to 103, where:

step 101: the node capacity of each server node for data acquisition is determined, and at least one management shard is created, wherein the shard attribute of the management shard comprises the shard capacity and the shard copy number.

It should be noted that the execution body of the present invention may be any electronic device capable of constructing a data acquisition system, for example, any one of a smart phone, a smart watch, a desktop computer, a laptop computer, and the like.

Specifically, the management slice, that is, the slice is a set group of at least one channel, and is used for managing channels inside the management slice, and is a basic unit of channel management and migration in the server node. The server nodes are carriers for managing the slicing operation and are independent individuals for data acquisition, and the number of the server nodes is at least one, namely greater than or equal to 1. The node attributes of the server node include: node identification (node unique identification) and node capacity, wherein the node identification may be a name, code number, flag, etc. of the server. Node capacity refers to the maximum number of management slices that can run on the server node. The shard capacity refers to managing the maximum number of channels that can be included in a shard. The number of the slice copies may be the number of copy versions (copies) corresponding to the current management slice, for example, the number of slice copies of slice 1 is 3, then slice 1 may be copied 3 times to obtain 3 copy versions (copies) of slice 1, or may be the total number of versions corresponding to the current management slice, for example, the number of slice copies of slice 2 is 3, then slice 1 may be copied 2 times to obtain 3 slices 1.

In practical application, all server nodes which can be used for data acquisition are defined, and node identification and node capacity are set for each server node, namely, the node capacity of each server node used for data acquisition is determined. Meanwhile, at least one management fragment for channel management is defined, and a fragment attribute is set for each management fragment, wherein the fragment attribute mainly comprises fragment capacity and fragment copy number.

Step 102: and respectively distributing each channel to the management fragments according to the fragment capacity to obtain distributed management fragments, wherein the channels represent physical links for data interaction.

In particular, a channel is an abstraction of one physical link, and specific communication protocols, communication media, are all contained within the channel.

In practical applications, a channel may be created for each process or physical link of data interaction, and specific communication protocols and communication media are included in the channel. After the channel is established, the channel is automatically distributed into the established management fragments: distributing channels for each management fragment according to the fragment capacity of each management fragment to obtain distributed management fragments; or automatically calculating and distributing the management assignment to be distributed to each channel to obtain the distributed management fragments.

The management fragment after allocation may be a management fragment in which the number of allocated channels reaches the capacity of the fragment, or may be a management fragment in which the number of allocated channels does not reach the capacity of the fragment after allocation of the channels is completed. The invention is not limited in this regard.

Step 103: and according to the node capacity and the number of the partitioned copies, deploying each distributed management partition on each server node to obtain a data acquisition system.

In practical application, after all channels are allocated, the allocated management fragments can be deployed on each server node according to the node capacity of each server node and the number of the fragment copies of each allocated management fragment, so as to obtain at least one deployed server node, and each deployed server node forms a data acquisition system, namely a data acquisition cluster.

Referring to fig. 2, fig. 2 is one of the structural intents of the data acquisition system provided by the present invention, and the constructed data acquisition system includes three server nodes: node 1, node 2, and node 3. Wherein, 3 management slices are deployed on node 1: segment 1, segment 2, and segment 3; 3 management slices are deployed on node 2: segment 3, segment 2, and segment 4; 3 management slices are deployed on node 3: segment 1, segment 3, and segment 4. The same management shard is backed up by multiple copies among shards, namely, the shard 1 in the node 3 can be obtained by backing up the shard 1 in the node 1, the shard 2 in the node 2 can be obtained by backing up the shard 2 in the node 1, the shard 3 in the node 2 and the shard 3 in the node 3 can be obtained by backing up the shard 3 in the node 1, and the shard 4 in the node 3 can be obtained by backing up the shard 4 in the node 2.

According to the construction method of the data acquisition system, the channels are distributed to the management fragments, the management fragments are deployed on the server nodes, the relation between the channels and the server nodes is decoupled, so that the management units of the server nodes are changed into the management fragments, the server nodes are ensured to only need to maintain the management fragments deployed on the server nodes, and each server node only maintains and acquires the data of the channels in the management fragments on the server nodes; because the main and standby and the expansion of the channel are both in the unit of management fragmentation, the robustness of data acquisition adapting to different data acquisition system scales is very good, the system can be distributed by a single machine or multiple machines, the system changing flow of server node expansion and contraction is simplified, and the lightweight and horizontal expansion of the data acquisition system is ensured.

In one or more alternative embodiments of the present invention, the number of shard deployments for each management shard may be determined first, then the target server may be determined based on the number of shard deployments and the node capacity, and the shard deployments may be performed on the target server node. Namely, according to the node capacity and the number of the partitioned copies, deploying each distributed management partition on each server node to obtain a data acquisition system, wherein the specific implementation process can be as follows:

Specifically, the number of sharded deployments refers to the number of server nodes that are to deploy the management shard. The number of deployed shards refers to the number of management shards that have been deployed on the server node.

In practical application, for any allocated management fragment, the number of the partitioned management fragment copies is obtained, where the number of the partitioned copies is the total number of versions corresponding to the current management fragment, the number of the partitioned copies can be directly used as the number of the partitioned deployments of the allocated management fragment, and where the number of the partitioned copies is the number of the replicated versions corresponding to the current management fragment, the number of the partitioned copies is added to one to obtain the number of the partitioned deployments of the allocated management fragment. To reduce the data throughput, the number of shard copies is preferably directly used as the number of shard deployments of the allocated management shards.

And screening out target server nodes with the number of the partitioned copies being smaller than the capacity of the nodes according to set screening rules from the server nodes. Further, the assigned management shards are deployed at the number of target server nodes of the shard copy, respectively. Traversing all the distributed management fragments, and sequentially deploying the distributed management fragments.

For example, if the number of sharded copies of the management shard a is 3, the number of sharded deployments of the management shard a is 4. And 4 target server nodes are screened out from the server nodes, and management fragments A are deployed on the 4 target server nodes respectively.

It should be noted that, when the number of the target server nodes is greater than or equal to the number of the deployment fragments, the allocated management fragments may be deployed on the number of the target servers of the deployment fragments; and under the condition that the number of the target servers is smaller than the deployment number of the fragments, the distributed management fragments are deployed on all the target servers.

In addition, the setting of the screening rule may be random screening, or may be screening for a difference between the node capacity and the number of deployed slices, for example, selecting a number of target server nodes in the number of deployed slices with the smallest difference between the node capacity and the number of deployed slices, or selecting a number of target server nodes in the number of deployed slices with the largest difference between the node capacity and the number of deployed slices.

Therefore, the distributed management fragments are deployed on each target server node according to the partition copy number, the distributed control fragments can be reasonably deployed, the deployment efficiency is improved, and the reliability of the built acquisition server is improved. Through the deployment step, under the condition of adding the server node, the new management fragment can be added to the new server node rapidly, so that the management fragment change caused by capacity expansion to the original server node is avoided, and the construction efficiency and reliability are improved.

In one or more optional embodiments of the present invention, according to the node capacity and the number of the shard copies, each allocated management shard is deployed on each server node to obtain a data acquisition system, or according to the number of the shard copies, the number of shard deployments of each allocated management shard may be respectively determined; for any server node, according to the node capacity of the server node, deploying a plurality of specific management fragments with multi-node capacity on the server node, wherein the specific management fragments are the management fragments with the allocated quantity smaller than the deployment quantity of the fragments.

For example, if the node capacity of the server node S is 5, a maximum of 5 specific management slices are deployed on the server node S.

It should be noted that, for any server node, when the number of specific management slices is greater than or equal to the node capacity of the server node, the node capacity of the specific management slices is deployed on the server node, and when the number of specific management slices is less than the node capacity of the server node, all the specific management slices are deployed on the server node. Therefore, the server node can be deployed as full as possible, and resource waste is avoided, so that management fragments can be deployed on the server node as much as possible. Thereby improving the efficiency of the acquisition server.

Preferably, in a case where the number of specific management slices is greater than or equal to the node capacity of the server node, the node capacity with the largest number of the slice copies is preferentially deployed on the server node.

Optionally, according to the partition capacity, each channel is allocated to the management partition, so as to obtain an allocated management partition, and the specific implementation process may be as follows:

Specifically, the allocated channel number refers to the channel number that has been allocated to the management slice.

In practical application, starting from the first channel, allocating the channel to the management fragments with the number of allocated channels smaller than the fragment capacity, namely, the target management fragments; if the target management fragment does not exist, a new management fragment is created, and the channel is allocated to the newly created management fragment. And so on until the last channel.

Therefore, the management is distributed to the management fragments with the quantity smaller than the fragment capacity, so that the problem that the management fragments cannot normally operate due to too many channels in the management fragments can be avoided, the reliable normal operation of the management fragments is ensured, and the reliability of the acquisition management system is improved. By creating a new management fragment, all channels can be distributed, complete data can be acquired, and the reliability of the acquisition management system is further improved.

In an embodiment of the present invention, for any channel, in the case that there is a target management slice in the at least one management slice, the channel may be randomly allocated to any one target management slice.

In another embodiment of the present invention, for any channel, in the case that a target management slice exists in the at least one management slice, the channel may be further allocated to the target management slice according to a set allocation rule. Namely, the slicing attribute further comprises a slicing identifier; correspondingly, in the case that the target management slice exists in the at least one management slice, the channel is allocated to the target management slice, and the specific implementation process may be as follows:

Specifically, the tile identifier characterizes the ordering of the management tiles, which may be the names or the order of the management tiles, such as the first tile, the second tile, and so on. The slicing attribute can also comprise slicing keys, slicing heartbeat time and the like on the basis of comprising slicing capacity, slicing copy number and slicing identification.

In practical application, for any channel, when a target management slice exists in the at least one management slice, the channel is preferentially allocated to the target management slice with the minimum slice identifier.

Therefore, the management fragments can be distributed as full as possible, so that channels can be distributed on the management fragments as much as possible, the number of the management fragments is reduced, and resource waste is avoided. And the distributed locks are used for the main and the auxiliary of the management fragments, so that the complexity of the main and the auxiliary of the management fragments by heartbeat is avoided, the method can be suitable for the determination of the main and the auxiliary of the management fragments with different numbers, and the application range is wider.

Illustratively, starting from the first channel, sequentially finding management fragments with less than full fragment capacity according to the fragment identification from small to large, namely, target management fragments, placing the channel in the first target management fragments, if no target management fragments with less than full fragment capacity exist, creating a new management fragment, and placing the channel in the new management fragments until the last channel.

It should be noted that, after all the channels are allocated, the allocated management fragments may be deployed. Management slices with full slice capacity (the number of the allocation channels is equal to the slice capacity) can be deployed on a server node (a target server node) capable of running the management slices according to the number of the slice copies in the process of allocating the channels. In addition, for the management fragments with full fragment capacity, when the corresponding target server node is selected, the corresponding target server node can be selected manually, namely, the target server node is selected according to the received deployment instruction (the deployment instruction initiated by the staff), so that when the server node expands, a new management fragment can be quickly added to the new server node, and the management fragment change caused by the expansion of the original server node is avoided.

In addition, the constructed data acquisition system supports the addition and deletion of server nodes, namely, the number of the server nodes in the data acquisition system is adjusted under the condition that a node number adjusting instruction is received.

Specifically, the node number adjustment instruction includes two types, i.e., a node number increase instruction and a node number decrease instruction, that is, a capacity expansion instruction and a capacity contraction instruction.

In practical application, under the condition that an instruction for increasing the number of nodes is received, adding server nodes in a data acquisition system; and reducing the server nodes in the data acquisition system under the condition that the instruction for reducing the number of the nodes is received.

Further, the node number adjustment instructions may include an adjustment number, such as increasing a number of server nodes, decreasing a number of server nodes. For the node number reduction instruction, the node identification of the reduced server node may also be included to explicitly need to delete which server nodes.

Referring to fig. 3, fig. 3 is a second flow chart of a method for constructing a data acquisition system according to the present invention: firstly defining nodes, and determining the node capacity of each server node for data acquisition; then creating a partition, namely creating a management partition; then, channel allocation is performed, and starting from the first channel, it is determined whether the created management slice has full slice capacity. If not, the channel is allocated for the segment, and then the allocation of the next channel is continued. If yes, the management fragments with full fragment capacity are deployed in a fragment mode, and new management fragments are created, namely the fragments are created. And so on.

The method of constructing the data acquisition system of the present invention is further described below with reference to fig. 4 and 5.

Referring to fig. 4, fig. 4 is a schematic diagram of a second embodiment of a data acquisition system according to the present invention: in a single-machine multi-process scene of a new energy power station, only one server node is provided, and node 1 is provided; there are two management slices, slice 1 and slice 2. The created data channels are shown in table 1. Segment 1 has a segment capacity of 2, and segment 1 contains only EMS channels and PMU channels because of the large data traffic of these two channels. Other channels are added to slice 2. Since there is only one server node, shard 1 and shard 2 can only be deployed into node 1.

Table 1 creates a channel

On the basis of fig. 4, referring to fig. 5, fig. 5 is a third structural intention of the data acquisition system provided by the present invention: under the distributed mutual backup scene of the new energy power station, as the single machine multi-process scene of the new energy power station is that backup is not needed among the single machine environment fragments, the problem of high availability exists, the entry improvement is improved, the deployment from single machine to distributed type can be completed by only two simple steps, the convenience is high, and the robustness of the deployment of the method is also verified. Only the extension procedure from single machine to distributed is described below: the first step is to add a server node, namely node 2; in the second step, the number of the fragment copies can be 2 for the node 1 during the fragment deployment, and the fragment 2 can be automatically deployed on the node 2, so that the purpose of operating the node 1 on the node 2 is achieved.

In addition, the number of shard copies to shard 2 may be set to 2 or more based on demand, where more server node support is required.

The data acquisition system constructed by the invention not only can be applied to a novel power station data acquisition scene, but also can be applied to any general data acquisition and monitoring control (Supervisory Control And Data Acquisition, SCADA) scene.

The data acquisition method of the present invention is described below with reference to fig. 6 and 7.

Fig. 6 is a schematic flow chart of a data acquisition method provided by the present invention, referring to fig. 6, where the data acquisition method is applied to a data acquisition system constructed by the method for constructing a data acquisition system described above, and includes steps 601-603, where:

step 601: and responding to the data acquisition instruction, and starting each management fragment deployed on each server node in the data acquisition system, wherein the fragment attribute of the management fragment comprises a fragment name.

In particular, the data acquisition instruction may be an instruction triggered by the initiation of a data acquisition operation. Specifically, the slice name refers to the name that identifies the management slice, and may be the slice identifier of the management slice.

In practical application, a worker can start the data acquisition system, or the worker sends the data acquisition instruction to the data acquisition instruction through the control end, and accordingly, the data acquisition system receives the data acquisition instruction. Further, the data acquisition system responds to the data acquisition instruction, each server node in the data acquisition system is started, each management partition deployed on the server node is started, and in addition, channels in each management partition can be loaded.

Step 602: and selecting a main management fragment from at least one designated management fragment with the same fragment name through a distributed coordination unit.

In particular, the distributed coordination unit may be a distributed key-value system etcd, which is a highly consistent distributed key-value store that provides a reliable way to store data that needs to be accessed by a distributed system or cluster of machines. The method can also be a distributed coordination service zookeeper, which is a distributed application coordination service of open source codes. At least one designated management slice refers to a collection of management slices having the same slice name.

In practical application, because the same management fragments may be deployed on different server nodes, in order to avoid resource waste caused by data collection of the same plurality of management fragments at the same time, the main management fragment may be selected from the same plurality of management fragments to perform data collection, that is, the distributed coordination unit is called, and the main management fragment is selected from at least one designated management fragment with the same fragment name.

Because the main and standby of the channel are realized through multiple copies of the management fragments, each management fragment can have multiple copies in the data acquisition system, namely at least one designated management fragment with the fragment name exists, the main management fragment can be selected by using a distributed coordination unit based on strong consistency among the same designated management fragments, and only one main management fragment exists at the same time in each management fragment in the data acquisition system.

If there are a plurality of management fragments (designated management fragments) corresponding to any fragment name, selecting one management fragment from the plurality of management fragments by the distributed coordination unit; if there are only a plurality of management slices (designated management slices) corresponding to the slice names, the management slice is the master management slice.

Step 603: and starting the channels on each main management partition to acquire data.

In practical application, after each main management partition is determined, a channel in each main management partition is started to collect data.

According to the data acquisition method provided by the invention, each server node only maintains and acquires the data of the channel in the management partition on the server node; because the main and standby and the expansion of the channel are both in the unit of management fragmentation, the robustness of data acquisition adapting to different data acquisition system scales is very good, the system can be distributed by a single machine or multiple machines, the system changing flow of server node expansion and contraction is simplified, and the lightweight and horizontal expansion of the data acquisition system is ensured. The management segmentation of the small data acquisition scene is realized, the hardware performance of the server node can be fully utilized, the bottleneck of the single-machine single acquisition process performance is made up to a great extent, and on the other hand, the overall availability of the data acquisition system and the reliability of acquired data are improved through the multiple preparations of the management segmentation in the large data acquisition scene.

In one or more alternative embodiments of the invention, the master management fragment may be selected from the at least one designated management fragment by a distributed lock of the distributed coordination unit. That is, the selecting, by the distributed coordination unit, the master management slice from the at least one designated management slice may be implemented as follows:

Specifically, the distributed lock is used for controlling the distributed system to operate on the shared resource orderly, and consistency is kept through mutual exclusion, namely, only one management partition in the same management partition is ensured to perform or participate in data acquisition.

In practical application, since the same management fragments (designated management fragments) can be deployed in different server nodes, a plurality of the same management fragments can be controlled by the distributed lock to perform data collection. A distributed lock name, such as a shard key, is defined for each designated management shard of the same shard name. When each management fragment is started, each designated management fragment applies for a distributed lock to the distributed coordination unit respectively, namely, sends a distributed lock request to the distributed coordination unit. The distributed coordination unit determines the request time of the distributed lock request sent by each appointed management fragment; and determining the designated management fragment corresponding to the distributed lock request with the earliest request time as a master management fragment, determining other designated management fragments as standby management fragments, namely feeding back the distributed lock to the designated management fragment corresponding to the distributed lock request with the earliest request time, wherein the designated management fragment received by the distributed lock is the master management fragment, and the designated management fragment not received by the distributed lock is the standby management fragment.

For example, a distributed lock name is defined for each management partition under the name of the partition. For example, the lock name of slice 1 is rxfep1. Then, when data collection starts, each management slice tries to apply for a distributed lock to etcd, and for the management slice that acquires the lock successfully, the master flag (master_flag) of the management slice becomes true, that is, the management slice is the master management slice. And then the main management partition starts an io acquisition thread, and reads, writes and acquires data through the channel.

In one or more optional embodiments of the present invention, during the data collection process, there may be an abnormality in the main management partition, where the main management partition cannot perform data collection normally, and in order to ensure the integrity and accuracy of data collection, the main management partition needs to be reselected from the standby management partition, that is, the method further includes:

In practical application, the standby management fragments apply for the distributed locks to the distributed coordination units periodically according to the set time interval, that is, send standby distributed lock requests to the distributed coordination units. In the case that the abnormality of the main management partition is detected, a new main management partition can be determined from the standby management partitions according to each standby distributed lock request periodically sent by each standby management.

For example, for each standby management slice corresponding to the abnormal main management slice, the standby management slice which sends the standby distributed lock request and is closest to the current moment is determined to be the new main management slice, so that the new main management slice can be ensured to normally run. For another example, for each standby management fragment corresponding to the main management fragment with the exception, the standby management fragment which sends the standby distributed lock request first in the sending period is determined to be the new main management fragment, so that the process of selecting the main management fragment for the first time is kept consistent, and the selection procedure is simplified.

Illustratively, a distributed lock name is defined for each management partition under the name of the partition. When data acquisition is started, each management fragment tries to apply for a distributed lock to the etcd, and for the standby management fragment with failed lock acquisition, the management fragment is indicated to have a main management fragment on other server nodes, and the management fragment is in a standby state, namely the standby management fragment. And then dormancy (sleep) is performed for a period of time (set time interval) before the application of the distributed lock, if the main partition management fails, the standby management partition applies for the distributed lock successfully, the standby management partition becomes a new main partition, otherwise, the acquisition of the distributed lock is always attempted.

Therefore, the distributed lock is the same flow for selecting the main management fragment from the plurality of management fragments and selecting the main management fragment from the two management fragments, is simple and easy to operate, is clear as a whole, is quite lightweight, and effectively avoids the complexity and the coupling of selecting the main management fragment (heartbeat selecting main) based on the heartbeat packet among the applicable multiple machines.

The fragmentation attribute of the management fragmentation includes a fragmentation heartbeat time, that is, the management fragmentation sends a heartbeat packet to the server node or the distributed coordination unit every time the fragmentation heartbeat time, and if the heartbeat packet sent by the main management fragmentation is not received under the condition that the time interval is longer than the fragmentation heartbeat time, it is determined that the main management fragmentation is abnormal. Or monitoring the data collected by the main management fragments, and if the collected data has abnormality such as messy codes, blank and the like, determining that the main management fragments have abnormality. Or the main management fragment automatically detects the abnormality and sends information for explaining the abnormal condition to the server node, and the main management fragment is determined to have the abnormality.

Referring to fig. 7, fig. 7 is a second flowchart of a data collection method according to the present invention: when a data acquisition instruction is received, each management partition deployed on each server node in the data acquisition system is started, namely the partition is started, and channels on each management partition, namely the loading channels, are loaded. Then, screening the main management fragments through a distributed coordination unit, namely, each appointed management fragment acquires a distributed lock, the appointed management fragment which is successfully acquired is the main management fragment, and the main management fragment starts a channel to acquire data, namely, the channel is acquired; the failed designated management slice is a standby management slice, and the standby management slice periodically tries to acquire the distributed lock.

The construction device of the data acquisition system provided by the invention is described below, and the construction device of the data acquisition system described below and the construction method of the data acquisition system described above can be referred to correspondingly.

Fig. 8 is a schematic structural diagram of a construction device of a data acquisition system according to the present invention, and as shown in fig. 8, a construction device 800 of the data acquisition system includes: a first determining module 801, an allocating module 802, a deploying module 803, wherein:

a first determining module 801 configured to determine a node capacity of each server node for data acquisition and create at least one management slice, the slice attribute of the management slice including a slice capacity and a slice copy number;

the allocation module 802 is configured to allocate each channel to the management slice according to the slice capacity, so as to obtain an allocated management slice, where the channels represent physical links of data interaction;

and the deployment module 803 is configured to deploy each allocated management fragment on each server node according to the node capacity and the fragment copy number to obtain a data acquisition system.

According to the construction device of the data acquisition system, the channels are distributed to the management fragments, the management fragments are deployed on the server nodes, the relation between the channels and the server nodes is decoupled, so that the management units of the server nodes are changed into the management fragments, the server nodes are ensured to only need to maintain the management fragments deployed on the server nodes, and each server node only maintains and acquires the data of the channels in the management fragments on the server nodes; because the main and standby and the expansion of the channel are both in the unit of management fragmentation, the robustness of data acquisition adapting to different data acquisition system scales is very good, the system can be distributed by a single machine or multiple machines, the system changing flow of server node expansion and contraction is simplified, and the lightweight and horizontal expansion of the data acquisition system is ensured.

In addition, the data acquisition system constructed by the construction device of the data acquisition system realizes management fragmentation of a small data acquisition scene, can fully utilize the hardware performance of a server node, makes up the bottleneck of single-machine single acquisition process performance to a great extent, and improves the overall availability of the data acquisition system and the reliability of acquired data by managing multiple preparations of the fragmentation in a large data acquisition scene. The construction method of the data acquisition system can flexibly cope with data acquisition systems of various scales, not only can meet the scene of single machine multi-acquisition process, but also can meet the expansion of distributed multi-machine acquisition, and solve the unlimited horizontal expansion of the acquisition scale while controlling the hardware cost. And the efficient availability of the channel is ensured through a slicing mechanism. Due to the lightweight of the method, the method is also applicable to the embedded device, and the compatibility and the flexibility of a data acquisition system are improved.

Optionally, the deployment module 803 is further configured to:

Optionally, the allocation module 802 is further configured to:

Optionally, the slice attribute further includes a slice identifier;

accordingly, the allocation module 802 is further configured to:

Optionally, the apparatus further comprises an adjustment module configured to:

The data acquisition device provided by the invention is described below, and the data acquisition device described below and the data acquisition method described above can be referred to correspondingly.

Fig. 9 is a schematic structural diagram of a data acquisition device provided by the present invention, as shown in fig. 9, the data acquisition device 900 is applied to a data acquisition system constructed by the method for constructing a data acquisition system described above, and includes: a first starting module 901, a selecting module 902, and a second starting module 903, wherein:

a first starting module 901, configured to respond to a data acquisition instruction, and start each management slice deployed on each server node in the data acquisition system, where a slice attribute of each management slice includes a slice name;

a selection module 902 configured to select, for at least one designated management slice having the same slice name, a master management slice from the at least one designated management slice through the distributed coordination unit;

a second starting module 903 is configured to start the channels on each of the main management slices to perform data collection.

Optionally, the selecting module 902 is further configured to:

Optionally, the apparatus further comprises a second determining module configured to:

Fig. 10 illustrates a physical structure diagram of an electronic device, as shown in fig. 10, which may include: a processor 1010, a communication interface (Communications Interface) 1020, a memory 1030, and a communication bus 1040, wherein the processor 1010, the communication interface 1020, and the memory 1030 communicate with each other via the communication bus 1040. Processor 1010 may invoke logic instructions in memory 1030 to perform a method of constructing a data acquisition system, the method comprising: determining node capacity of each server node for data acquisition, and creating at least one management fragment, wherein the fragment attribute of the management fragment comprises fragment capacity and fragment copy number; according to the capacity of the fragments, each channel is respectively allocated to the management fragments to obtain allocated management fragments, and the channels represent physical links of data interaction; and according to the node capacity and the number of the partitioned copies, deploying each distributed management partition on each server node to obtain a data acquisition system. Or processor 1010 may invoke logic instructions in memory 1030 to perform a data acquisition method comprising: responding to a data acquisition instruction, and starting each management fragment deployed on each server node in the data acquisition system, wherein the fragment attribute of each management fragment comprises a fragment name; selecting a main management fragment from at least one designated management fragment with the same fragment name through a distributed coordination unit; and starting the channels on each main management partition to acquire data.

Further, the logic instructions in the memory 1030 described above may be implemented in the form of software functional units and stored in a computer readable storage medium when sold or used as a stand alone product. Based on this understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.

In another aspect, the present invention also provides a computer program product, where the computer program product includes a computer program, where the computer program can be stored on a non-transitory computer readable storage medium, where the computer program when executed by a processor can perform a method for constructing a data acquisition system provided by the above methods, where the method includes: determining node capacity of each server node for data acquisition, and creating at least one management fragment, wherein the fragment attribute of the management fragment comprises fragment capacity and fragment copy number; according to the capacity of the fragments, each channel is respectively allocated to the management fragments to obtain allocated management fragments, and the channels represent physical links of data interaction; and according to the node capacity and the number of the partitioned copies, deploying each distributed management partition on each server node to obtain a data acquisition system. Or the computer can execute the data acquisition method provided by the methods, and the method comprises the following steps: responding to a data acquisition instruction, and starting each management fragment deployed on each server node in the data acquisition system, wherein the fragment attribute of each management fragment comprises a fragment name; selecting a main management fragment from at least one designated management fragment with the same fragment name through a distributed coordination unit; and starting the channels on each main management partition to acquire data.

In yet another aspect, the present invention further provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, is implemented to perform a method of constructing a data acquisition system provided by the above methods, the method comprising: determining node capacity of each server node for data acquisition, and creating at least one management fragment, wherein the fragment attribute of the management fragment comprises fragment capacity and fragment copy number; according to the capacity of the fragments, each channel is respectively allocated to the management fragments to obtain allocated management fragments, and the channels represent physical links of data interaction; and according to the node capacity and the number of the partitioned copies, deploying each distributed management partition on each server node to obtain a data acquisition system. Or the computer program is implemented when executed by a processor to perform the data acquisition method provided by the methods above, the method comprising: responding to a data acquisition instruction, and starting each management fragment deployed on each server node in the data acquisition system, wherein the fragment attribute of each management fragment comprises a fragment name; selecting a main management fragment from at least one designated management fragment with the same fragment name through a distributed coordination unit; and starting the channels on each main management partition to acquire data.

The apparatus embodiments described above are merely illustrative, wherein the elements illustrated as separate elements may or may not be physically separate, and the elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art will understand and implement the present invention without undue burden.

From the above description of the embodiments, it will be apparent to those skilled in the art that the embodiments may be implemented by means of software plus necessary general hardware platforms, or of course may be implemented by means of hardware. Based on this understanding, the foregoing technical solution may be embodied essentially or in a part contributing to the prior art in the form of a software product, which may be stored in a computer readable storage medium, such as ROM/RAM, a magnetic disk, an optical disk, etc., including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method described in the respective embodiments or some parts of the embodiments.

Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present invention, and are not limiting; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method of constructing a data acquisition system, comprising:

determining node capacity of each server node for data acquisition, and creating at least one management fragment, wherein the fragment attribute of the management fragment comprises fragment capacity and fragment copy number, the node capacity refers to the maximum number of the management fragments which can be operated on the server node, the management fragment is a basic unit for channel management and migration in the server node, and the fragment capacity refers to the maximum number of channels which can be contained in the management fragment;

According to the node capacity and the number of the partitioned copies, deploying each distributed management partition on each server node to obtain a data acquisition system;

and allocating each channel to the management fragment according to the fragment capacity to obtain an allocated management fragment, wherein the method comprises the following steps:

creating a new management slice and distributing the channel to the new management slice under the condition that the target management slice does not exist in the at least one management slice;

and deploying each allocated management fragment on each server node according to the node capacity and the fragment copy number to obtain a data acquisition system, wherein the data acquisition system comprises:

2. The method for constructing a data acquisition system according to claim 1, wherein the fragmentation attribute further comprises a fragmentation identifier;

3. The method for constructing a data acquisition system according to claim 1 or 2, further comprising:

4. A data acquisition method, characterized by being applied to a data acquisition system constructed by the construction method of a data acquisition system according to any one of claims 1 to 3, comprising:

And starting the channels on each main management partition to acquire data.

5. The data collection method according to claim 4, wherein the selecting, by the distributed coordination unit, a master management slice from the at least one designated management slice comprises:

6. The data acquisition method of claim 5, further comprising:

7. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the method of constructing a data acquisition system according to any one of claims 1 to 3 or implements the method of data acquisition according to any one of claims 4 to 6 when the program is executed by the processor.

8. A non-transitory computer readable storage medium having stored thereon a computer program, characterized in that the computer program, when executed by a processor, implements a method of constructing a data acquisition system according to any one of claims 1 to 3 or implements a data acquisition method according to any one of claims 4 to 6.