CN117194010A - Optimization system, method, electronic equipment and storage medium for data acquisition cluster processing - Google Patents

Abstract

The invention discloses an optimization system, method, electronic equipment and storage medium for data acquisition cluster processing, wherein the system comprises: the system comprises a plurality of gateways, a message queue supporting clusters and multithreading and a plurality of processors, wherein the message queue comprises a message queue manager and a queue group set formed by a plurality of queue groups, and each queue group comprises a plurality of message queues; the message queue manager is used for managing each queue group, calculating the data quantity threshold value of each queue and distributing proper queues and processors to the data transmitted by the gateway; and the cluster processing of the gateway, the message queue and the processor is realized through the scheduling of the message queue manager. The invention improves the data quantity of the data which can be acquired by the acquisition end, ensures the stability of large data quantity acquisition, ensures that important data is not lost, satisfies unbalanced equipment data acquisition, and simultaneously supports multi-thread and batch data persistence operation.

Description

Optimization system, method, electronic equipment and storage medium for data acquisition cluster processing

Technical Field

The invention relates to an electronic information technology, in particular to an optimization system, method, electronic equipment and storage medium for data acquisition cluster processing, which can be applied to electronic semiconductors and new energy industries.

Background

The device data collection has a time sequence requirement. The existing equipment data acquisition technology is that after the gateway generates data, the generated data is packaged into a data packet, then the data packet is directly sent to a message queue, the message queue continuously sends the sent data packet to a Processor (processing) end, and finally the Processor (processing) end performs relevant processing. This is a single-node, single-thread message queue, and even if there are multiple gateways, data is sent to the same message queue. Meanwhile, only one Processor (processing) end is provided, and all data is collected and processed through the Processor (processing) end, as shown in fig. 1.

According to the data acquisition technology, when the data volume from the equipment end becomes large and the acquired frequency requirement is frequent, a large number of data packets are accumulated in the message queue, so that the real-time property of the acquired data of the acquisition end is poor, and when too many data packets exist in the queue, packet loss occurs and some data cannot be acquired. This phenomenon is caused by that there is only one message queue between the gateway and the Processor, and the cluster is not supported, and when the message queue reaches the performance bottleneck, data accumulation is caused.

Disclosure of Invention

The invention aims to: the invention aims to provide an optimization method for data acquisition cluster processing, which aims to improve the data quantity of data which can be acquired by an acquisition end, ensure the stability and time sequence of large data quantity acquisition, ensure that important data is not lost, meet unbalanced equipment data acquisition and simultaneously support multi-thread and batch data persistence operation.

It is another object of the present invention to provide an optimization system for data acquisition cluster processing.

The technical scheme is as follows: the invention discloses an optimization system for data acquisition cluster processing, which comprises the following components: the system comprises a plurality of gateways, a message Queue supporting clusters and multithreading and a plurality of Processor processors, wherein the message Queue comprises a message Queue manager GroupQueue Manager and a Queue Group set formed by a plurality of Queue Group Queue, and each Queue Group Queue comprises a plurality of message Queue; the message Queue manager Group Queue Manager is configured to manage the Queue groups Group Queue, calculate a data amount threshold of each Queue, and allocate a proper Queue and a Processor to data transmitted from the gateway; the message queue manager Group Queue Manager performs cluster processing of the gateway, message queue and Processor, and ensures the time sequence of data.

Preferably, the registration management mechanism of Group Queue Manager is:

the Group Queue is successfully deployed, namely automatically registered to Group Queue Manager, and the Group Queue stores the serial number i of each Queue and the corresponding data volume threshold T in the Group Queue _i ；

Registering Group Queue Manager when the gateway is started, maintaining registration information of the gateway, registering subscribed point location information, and returning a proper Queue according to the data load of the gateway Group Queue Manager; if the data load of the gateway is larger than the residual processing threshold value of the currently recommended Queue, according to the size of the data load of the gateway and in combination with the data processing threshold value of other idle Queue, returning recommended gateway subscription point information splitting suggestions comprising subscription point information of the recommended splitting and the sequence of resending the subscription point information to Group Queue Manager, splitting the previous subscription point information after the gateway receives the suggestions, and sending the split point subscription information to GroupQueue Manager one by one to be bound with the proper Queue in sequence; otherwise, the data load of the gateway is directly bound with a proper Queue;

the Processor is started, registration is required to be carried out to Group Queue Manager, group Queue Manager maintains registration information of the Processor, and a Queue to be processed is allocated to the Processor;

through Group Queue Manager, a proper relation is established among the gateway, the message Queue and the Processor, so that the data size sent to the Queue per second is ensured not to exceed the data size threshold T of the message Queue.

Preferably, a data payload in a gateway must be in a message Queue, and a Processor allocates a message Queue.

Based on the same given inventive concept, the optimization method of the data acquisition cluster processing comprises the following steps:

the message Queue manager Group Queue Manager calculates a data amount threshold T of each Queue in the Group Queue;

the registration management mechanism of Group Queue Manager is: the Group Queue is successfully deployed, namely automatically registered to GroupQueue Manager, and the Group Queue stores the serial number i of each Queue and the corresponding data volume threshold T in the Group Queue _i The method comprises the steps of carrying out a first treatment on the surface of the Registering Group Queue Manager when the gateway is started, maintaining registration information of the gateway, registering subscribed point location information, and returning a proper Queue according to the data load of the gateway Group Queue Manager; if the data load of the gateway is larger than the residual processing threshold value of the currently recommended Queue, according to the size of the data load of the gateway and in combination with the data processing threshold value of other idle Queue, returning recommended gateway subscription point information splitting suggestions comprising subscription point information of the recommended splitting and the sequence of resending the subscription point information to Group Queue Manager, splitting the previous subscription point information after the gateway receives the suggestions, and sending the split point subscription information to GroupQueue Manager one by one to be bound with the proper Queue in sequence; otherwise, the data load of the gateway is directly bound with a proper Queue; the Processor is started, registration is required to be carried out to Group Queue Manager, group Queue Manager maintains registration information of the Processor, and a Queue to be processed is allocated to the Processor;

the gateway assembles the real-time value of the point location into a data packet according to the actual subscribed point location information, and pushes the data packet to the corresponding Queue, and Group Queue Manager monitors the data change in the Queue;

the Processor acquires data from the queue, firstly analyzes the data packet, and then processes each point bit in the data packet.

Further, the calculation method of the data quantity threshold value T of the Queue is as follows:

the simulation client sends a point position value simulation data packet with the size of y, the data structure is consistent with that of a real data packet, and the throughput Tps of the Queue at the moment is recorded as x;

adjusting the size y of the data packet, continuing to record the throughput x of the Queue, and reciprocating the data packet until x <1;

obtaining a fitting function y=f (x) by a least square method according to the data [ x, y ];

throughput x according to actual configuration _real Value, calculate threshold t=x for single Queue _real *f(x _real )。

Further, the Group Queue starts from the sequence number 1, wherein the Queue also starts from the sequence number 1 and automatically increases when the data volume threshold T is reached; when the sequence number exceeds the number k of the message queues in the Queue Group, the next Group Queue number is automatically switched to the Group Queue number +1, and if the next Group Queue is not present, namely, the Group Queue number is larger than the registered Group Queue number h, a warning prompt is given.

Further, the gateway registration specifically includes:

when the gateway registers, the gateway sends subscribed equipment monitoring point location information to Group QueueManager group by group according to subscribed equipment monitoring point location information in the packets, group Queue Manager records the number p of the group of equipment monitoring point locations, and the acquisition frequency F, the point location data size Q and the importance degree L of the point locations of each point location;

calculating a data load W:

wherein p is the number of the monitoring points of the group of equipment, fj is the acquisition frequency of the monitoring point of the jth equipment, qj is the point data size of the monitoring point of the jth equipment, lj is the point importance degree of the monitoring point of the jth equipment, fj×Qj represents the data quantity of the monitoring point of the jth equipment cached in the message queue in unit time, and 2Lj takes the values [1,2] as coefficients;

returning the appropriate Queue sequence number according to the following logic:

group Queue is taken from Group Queue 1 registered in Group Queue Manager and denoted as g (a), a=1; the Queue is taken from the Queue 1 in the Group Queue 1, and is marked as g_q (i), i=1, and the data volume threshold of the ith Queue in the Group Queue is marked as T _i The size of the load posted in the Queue is noted as TOccupy _i ：

If TOccupy _i +W≤T _i Returning g_q (i) in g (a);

if TOccupy _i +W>T _i And carrying out the process of splitting and reassigning the subscription point information:

(1) Finding the maximum number of bits q that can be accommodated in g_q (i), i.e. at this time And->Recording subscribed device monitoring point location information of j=1 to j=q and an execution sequence s;

(2) Continuously calculating the monitoring point position information which can be contained in the g_q (i+1) from the rest equipment monitoring point position information in the group: recording j=q+1 at this time, and judgingWhether or not it is greater than T _i+1 If->Less than or equal to T _i+1 Device that notes subscriptions for j=q+1 to j=pMonitoring point location information and an execution sequence s+1; otherwise, the maximum number of points qmax still to be accommodated in g_q (i+1), i.e. at this point +.>And is also provided withNoting subscribed device monitoring point location information of j=q+1 to j=qmax, and execution order s+1; setting q=qmax, and continuing to circularly execute the step (2); during this period, if i+1>k, k is the number of queues, and g_q (i) in g (a+1) is returned, i=1; if a+1>h, h is the number of Group Queue, and an alarm is sent out;

(3) Combining the subscribed equipment monitoring point location information recorded in the step with the execution sequence to form recommended gateway subscription point location information splitting suggestion, and returning and sending the recommended gateway subscription point location information splitting suggestion to a gateway;

(4) The gateway splits the subscribed equipment monitoring point location information according to the recommended gateway subscription point location information splitting suggestion, and sends the split group to Group Queue Manager again according to the returned sequence; then re-executing the allocation operation;

if i+1>k, g_q (i) in g (a+1), i=1 is returned;

and if a is plus 1>h, an alarm is sent out.

Further, the processing of each point in the data packet by the Processor includes: storing real time values, storing the time sequence data base into historical values and interfacing with other processing logic.

Based on the same inventive concept, the electronic device of the present invention comprises a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of the optimization method of data acquisition cluster processing as described above when executing the program.

Based on the same inventive concept, a computer-readable storage medium of the present invention has stored thereon a computer program which, when executed by a processor, implements the steps of the optimization method of data acquisition cluster processing as described above.

The beneficial effects are that: compared with the prior art, the invention has the remarkable technical effects that:

(1) Compared with the prior art, the method can calculate the data quantity processing threshold of each queue through the calculation algorithm of the queue data threshold only by setting the number of the queues of a single node; compared with the original single queue, the data volume is not limited, so that data backlog and data loss cannot be caused, and the stability of data transmission is ensured;

(2) The invention adopts an allocation algorithm between the gateway and the Queue and Processor, and the corresponding relation between the gateway and the Queue and Processor is obtained in Group Queue Manager through the algorithm; compared with the prior art, a plurality of nodes and a plurality of queues process data from different gateways, and the queues perform their own roles, so that the data acquisition is more real-time and stable;

(3) Under the conditions of large data volume and high acquisition frequency, the real-time property and stability of the acquired data can be ensured, the acquired data can be used for more precise calculation, and the accuracy of upper-layer application calculation can be ensured;

firstly, the single queue in the prior art is changed into multi-queue parallel, so that the processing rate of the data packet is improved; for each queue, when the queue is started initially, the bottleneck of the queue for processing data is calculated through an algorithm, and the processing capacity of the queue for the data is ensured; for a plurality of devices, the invention also utilizes an allocation algorithm to allocate the data of the devices to the corresponding queues for processing; finally, the invention also supports clustering for more devices or processing larger amounts of data, which is theoretically infinitely extensive, of course depending on the actual resources.

Drawings

FIG. 1 is a schematic diagram of a prior art data acquisition system;

FIG. 2 is a schematic diagram of the system architecture of the present invention;

FIG. 3 is a flow chart of the method of the present invention;

FIG. 4 is a schematic diagram of a persistence structure for a Processor to obtain data from a Queue.

Detailed Description

The present invention will now be described in detail with reference to the drawings and specific embodiments thereof, wherein the embodiments described are some, but not all, of the embodiments of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

The invention aims to solve the problems that when the data of equipment is acquired, the acquired data volume is large, the acquisition frequency is high, the acquired data needs to ensure the time sequence, and the data of an upper acquisition end cannot be acquired in real time or even the data is lost under the condition that the sizes and the frequencies of the data generated by various equipment are different. The technology belongs to the electronic information technology, and can be applied to the electronic semiconductor and new energy industries.

Example 1

As shown in fig. 2, the optimization system for data acquisition cluster processing of the invention comprises a plurality of gateways, a message queue supporting clusters and multithreading, and a plurality of Processor processors, wherein the message queue supporting clusters and multithreading solves the problems of large data volume, data backlog and even packet loss under high frequency conditions in the prior art. The message Queue comprises a message Queue manager Group Queue Manager and a Queue Group set formed by a plurality of Queue Group Queue, wherein each Queue Group Queue comprises a plurality of Queue; the message Queue manager Group Queue Manager is used to manage each Queue group, calculate the data volume threshold for each Queue and assign appropriate Queue and Processor to the data transmitted from the gateway.

Example two

As shown in fig. 3, the optimization method of the data acquisition cluster processing of the present invention adopts the above system, and implements the cluster processing of the gateway, the message queue and the Processor through the scheduling of the message queue manager Group Queue Manager. The method specifically comprises the following steps:

s1, a message Queue manager Group Queue Manager calculates a data quantity threshold T of each Queue in each Queue Group Queue;

the Queue Group Queue is single-node software of the message Queue. Group Queue Manager supports registration of an acceptance Queue Group Queue, each of which may contain multiple message Queue queues. The number of Group Queue is the number actually registered to Group Queue Manager. Each queue group defaults to 4 message queues Q1-Q4 (the number of message queues can be specifically configured according to actual requirements).

A single message Queue may set a data amount threshold T (Byte/S), and the system may automatically set T as follows:

(1) The simulation client sends a point value data packet with the size of y (Byte) (the data packet is not real equipment monitoring point position information, is a simulation data packet, but is consistent with the format of the real equipment monitoring point position information); recording the Tps (throughput) of the Queue at the moment as x;

(2) Adjusting the size y of the analog data packet, continuing to record the Tps (throughput) x of the Queue, and reciprocating until x <1 (when the throughput is smaller than 1, the content of one analog data packet cannot be completed within 1 second, so that the instantaneity of the data is affected);

(3) Obtaining a fitting function y=f (x) by a least square method according to the two-step data [ x, y ] (wherein x and y are obtained by the analog data packet test);

(4) X according to actual configuration _real (TPS) value, calculating the threshold t=x for a single Queue _real *f(x _real )

Compared with the prior art, the method can calculate the data quantity processing threshold of each queue through the calculation algorithm of the queue data threshold only by setting the number of the queues of a single node. Compared with the prior single queue with unrestricted data volume, the method can not cause data backlog and data loss, and ensures the stability of data transmission.

S2, a registration management mechanism of Group Queue Manager; the method comprises the following steps:

s21, managing an internal message Queue by a Group Queue;

the Group Queue deployment is successful, i.e. automatically registered with Group Queue Manager. The Group Queue stores the serial number i of each Queue and the corresponding data volume threshold T. Group Queue starts from sequence number 1, wherein Queue also starts from sequence number 1 and automatically increases when data volume threshold T is reached; when the sequence number exceeds the number k of the message queues in the Queue Group, the next Group Queue is automatically switched to the Group Queue sequence number +1, and if the next Group Queue does not exist (the Group Queue sequence number is larger than the registered Group Queue number h), a warning prompt is sent.

S22, gateway registration;

registering Group Queue Manager when the gateway is started, maintaining registration information of the gateway, registering subscribed point location information, and returning a proper Queue according to the data load of the gateway Group Queue Manager; if the data load of the gateway is larger than the residual processing threshold value of the currently recommended Queue, according to the size of the data load of the gateway and in combination with the data processing threshold value of other idle Queue, returning recommended gateway subscription point information splitting suggestions comprising subscription point information of the recommended splitting and the sequence of resending the subscription point information to Group Queue Manager, splitting the previous subscription point information after the gateway receives the suggestions, and sending the split point subscription information to GroupQueue Manager one by one to be bound with the proper Queue in sequence; otherwise, the data load of the gateway is directly bound with the proper Queue:

(1) When the gateway registers, the gateway sends subscribed equipment monitoring point location information to GroupQueue Manager group by group according to the subscribed equipment monitoring point location information in the group, group Queue Manager records the number p of the group of equipment monitoring point locations, and the acquisition frequency F, the point location data size Q and the importance degree L (L E [50%,100% ]) of the point locations;

(2) Calculating a data load W:

where p is the number of monitoring points of the group of devices and Fj is the jth deviceThe acquisition frequency of the monitoring point location, qj is the point location data size (Q _j The size is the sum of the size of the equipment monitoring point name, the data type length of the equipment monitoring point and the time stamp length), L _j Is the point location importance degree of the j-th equipment monitoring point location, F _j ×Q _j Representing the data amount of the j-th device monitoring point in the message queue in unit time, 2L _j Take the value of [1,2]As coefficients.

(3) Returning a proper Queue sequence number according to the following logic;

group Queue is taken from Group Queue 1 registered in Group Queue Manager and denoted as g (a), a=1; the Queue is taken from the Queue 1 in the Group Queue 1, and is marked as g_q (i), i=1, and the data volume threshold of the ith Queue in the Group Queue is marked as T _i The size of the load posted in the Queue is noted as TOccupy _i ：

If TOccupy _i +W≤T _i Returning g_q (i) in g (a);

if TOccupy _i +W>T _i And carrying out the process of splitting and reassigning the subscription point information:

(1) finding the maximum number of bits q that can be accommodated in g_q (i), i.e. at this timeAnd->Noting subscribed device monitoring point location information of j=1 to j=q, and execution order s (1 at this time);

(2) continuously calculating the monitoring point position information which can be contained in the g_q (i+1) from the rest equipment monitoring point position information in the group: recording j=q+1 at this time, and judgingWhether or not it is greater than T _i+1 If->Less than or equal to T _i+1 Recording subscribed device monitoring point location information of j=q+1 to j=p and an execution sequence s+1; otherwise, the maximum number of points qmax still to be accommodated in g_q (i+1), i.e. at this point +.>And is also provided withRecording subscribed device monitoring point location information of j=q+1 to j=qmax and an execution sequence s+1; setting q=qmax, and continuing to circularly execute the step (2); during this period, if i+1>k (number of queues), g_q (i), i=1, in g (a+1) is returned; if a+1>h (Group Queue number), sending an alarm;

(3) combining the subscribed equipment monitoring point location information recorded in the step with the execution sequence to form recommended gateway subscription point location information splitting suggestion, and returning and sending the recommended gateway subscription point location information splitting suggestion to a gateway;

(4) the gateway splits the subscribed equipment monitoring point location information according to the recommended gateway subscription point location information splitting suggestion, and sends the split group to Group Queue Manager again according to the returned sequence. Then re-executing the allocation operation;

if i+1>k (number of queues), g_q (i), i=1, in g (a+1) is returned;

if a+1>h (Group Queue number), an alarm is issued.

S23, registering a Processor;

processor startup requires registration with Group Queue Manager. Group Queue Manager maintains registration information for processors and assigns to the processors the requests that need to be processed (default one Processor assigns one).

Through Group Queue Manager, a proper relation is established among the gateway, the Queue and the Processor, so that the data volume sent to the Queue per second is ensured not to exceed the data volume threshold T of the Queue, and the sent data is ensured to be processed by the Processor.

In summary, the present invention employs an allocation algorithm between a gateway, queue and Processor. By this algorithm, the correspondence of the gateway (device) to the Queue and Processor will be obtained in Group Queue Manager. Compared with the prior art, a plurality of nodes and a plurality of queues process data from different gateways, and the queues perform their own roles, so that data acquisition is more real-time and stable.

S3, the gateway sends monitoring point location data to the Queue;

the gateway assembles the real point position real-time values (including the names, the current values and the current time stamps of all subscribed device monitoring points) into data packets according to the actual subscribed device monitoring point position information (because not all points in the gateway need to be acquired, and the monitoring points need to be acquired are called as 'subscribed' points), pushes the data packets to the corresponding Queue (the gateway sends monitoring point data to the Queue allocated to the gateway), and Group Queue Manager monitors the data change in the Queue.

S4, the Processor acquires data from the Queue and persists the data;

the Processor acquires data from the queue, firstly analyzes the data packet, and then respectively processes each point bit in the data packet, comprising: storing the real time value and the time sequence database into a historical value and interfacing with other processing logic; as shown in fig. 4.

Deploying 2 nodes, 4 Queue per node, 4 gateway, 4 Processor cases:

(1) Group Queue Manager and calculation of data volume threshold T for each Queue: deployment Group Queue Manager (message queue group manager). And (3) deploying Queue Group nodes on the two servers respectively, configuring the number of the queues in the Queue Group to be 4 respectively, and automatically registering in Group Queue Manager. Group Queue Manager simulates the transmission of data to the queues in the queue group and processes the data, so as to obtain a plurality of discrete points of x (queue throughput), y (data packet size)), a fitting function y=f (x) is obtained through a least square method, and a data volume processing threshold t=xf (x) of the queues is calculated according to the actually configured x (queue throughput).

(2) Registration management of Group Queue Manager:

(2.1) when the Group Queue registration is successful, group Queue Manager assigns a sequence number to it, starting with 1, which will be Group Queue 1 and Group Queue 2.

(2.2) four gateways are respectively registered, and when the gateway is registered, a group of equipment monitoring point location information containing 2000 equipment monitoring point locations is provided, wherein the information contains the point location name of each point location, the acquisition frequency F (times/s), the point location data size Q (Byte) and the importance degree L (L E [50%, 100%)]). The sum then yields the set of data packets for the gateway asReturning the appropriate Queue sequence number according to the following logic:

group Queue is taken from Group Queue 1 registered in Group Queue Manager and denoted as g (a), a=1; the Queue is taken from the Queue 1 in the Group Queue 1, and is marked as g_q (i), i=1, and the data volume threshold of the ith Queue in the Group Queue is marked as T _i The size of the load recorded in the queue is recorded as TOccupy _i ：

If TOccupy _i +W≤T _i Returning g_q (i) in g (a);

if TOccupy _i +W>T _i And carrying out the process of splitting and reassigning the subscription point information:

(1) finding the maximum number of bits q that can be accommodated in g_q (i), i.e. at this timeAnd->Recording subscribed device monitoring point location information of j=1 to j=q and an execution sequence s (1 at the moment);

(2) continuously calculating the monitoring point position information which can be contained in the g_q (i+1) from the rest equipment monitoring point position information in the group: recording j=q+1 at this time, and judgingWhether or not it is greater thanT _i+1 If->Less than or equal to T _i+1 Recording subscribed device monitoring point location information of j=q+1 to j=p and an execution sequence s+1; otherwise, the maximum number of points qmax still to be accommodated in g_q (i+1), i.e. at this point +.>And is also provided withRecording subscribed device monitoring point location information of j=q+1 to j=qmax and an execution sequence s+1; setting q=qmax, and continuing to circularly execute the step (2); during this period, if i+1>4 (number of queues), g_q (i), i=1, in g (a+1) is returned; if a+1>2 (Group Queue number), then alarm is sent out;

(3) combining the subscribed equipment monitoring point location information recorded in the step with the execution sequence to form recommended gateway subscription point location information splitting suggestion, and returning and sending the recommended gateway subscription point location information splitting suggestion to a gateway;

(4) the gateway splits the subscribed equipment monitoring point location information according to the recommended gateway subscription point location information splitting suggestion, and sends the split group to Group Queue Manager again according to the returned sequence. Then re-executing the allocation operation;

if i+1>4 (number of queues), g_q (1) in g (2) is returned;

if a+1>2 (Group Queue number), an alarm is issued.

(2.3) four processors register respectively:

processor startup requires registration with Group Queue Manager. Group Queue Manager maintains registration information for processors and assigns processors a Queue to be processed (one Processor assigns one Queue at this time).

(3) And pushing gateway data. The point device monitoring bit data (including the name, current value, current timestamp of each point) of the actual subscription is pushed in the gateway, and the data is sent to Queue of the specified relationship in Group Queue Manager.

(4) Processor data processing. The Processor receives the data pushed in the designated Queue, parses and further processes: store real time values, store history values, dock other logic.

By the method, the data quantity of the data which can be acquired by the acquisition end is improved, the stability of large data quantity acquisition is ensured, important data is not lost, unbalanced equipment data acquisition is met, and multi-thread and batch data persistence operation is supported.

Example III

An electronic device of the present embodiment includes a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor implements the steps of the optimization method for device data acquisition cluster processing as described above when the program is executed by the processor.

The processor and the memory communicate with each other via the communication interface and the communication bus, and the processor may call logic instructions in the memory to perform the method in the second embodiment.

Further, the logic instructions in the memory described above may be implemented in the form of software functional units and stored in a computer-readable storage medium when sold or used as a stand-alone product. Based on this understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.

Example IV

A computer readable storage medium of the present embodiment has stored thereon a computer program which, when executed by a processor, implements the steps of the optimization method of device data acquisition clustering as described above.

The computer storage media of embodiments of the invention may take the form of any combination of one or more computer-readable media. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

The computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, either in baseband or as part of a carrier wave. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

Computer program code for carrying out operations of the present invention may be written in one or more programming languages, including an object oriented programming language such as Java, smalltalk, C ++ and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (for example, through the Internet using an Internet service provider).

Of course, the storage medium containing the computer executable instructions provided in the embodiments of the present invention is not limited to the above method operations, but may also perform the related operations in the method provided in any embodiment of the present invention.

Example five

The present invention also provides a computer program product comprising a computer program stored on a computer readable storage medium, the computer program comprising program instructions which, when executed by a computer, enable the computer to perform the method of embodiment two.

The system embodiments described above are merely illustrative, in which the modules illustrated as separate components may or may not be physically separate, and the components shown as modules may or may not be physical, i.e., may be located in one place, or may be distributed over multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art will understand and implement the present invention without undue burden.

From the above detailed description of the embodiments, it will be apparent to those skilled in the art that the embodiments may be implemented by means of software plus necessary general hardware platforms, or of course by means of hardware. Based on such understanding, the foregoing technical solutions may be embodied essentially or in part in the form of a software product that may be stored in a computer-readable storage medium including Read-Only Memory (ROM), random-access Memory (Random Access Memory, RAM), programmable Read-Only Memory (Programmable Read-Only Memory, PROM), erasable programmable Read-Only Memory (Erasable Programmable Read Only Memory, EPROM), one-time programmable Read-Only Memory (OTPROM), electrically erasable programmable Read-Only Memory (EEPROM), compact disc Read-Only Memory (Compact Disc Read-Only Memory, CD-ROM) or other optical disc Memory, magnetic disc Memory, tape Memory, or any other medium that can be used for computer-readable carrying or storing data.

Claims (10)

1. An optimization system for data acquisition cluster processing, comprising: the system comprises a plurality of gateways, a message Queue supporting clusters and multithreading and a plurality of Processor processors, wherein the message Queue comprises a message Queue manager Group Queue Manager and a Queue Group set formed by a plurality of Queue Group Queue, and each Queue Group Queue comprises a plurality of message Queue; the message Queue manager Group Queue Manager is configured to manage the Queue groups Group Queue, calculate a data amount threshold of each Queue, and allocate a proper Queue and a Processor to data transmitted from the gateway; the message queue manager Group Queue Manager performs cluster processing of the gateway, message queue and Processor, and ensures the time sequence of data.

2. The optimization system of claim 1, wherein the registration management mechanism of Group Queue Manager is:

successful Group Queue deployment, i.e., automatically proceed to Group Queue ManagerRegistering, wherein the Group Queue stores the serial number i of each Queue and the corresponding data volume threshold T _i ；

Registering Group Queue Manager when the gateway is started, maintaining registration information of the gateway, registering subscribed point location information, and returning a proper Queue according to the data load of the gateway Group Queue Manager; if the data load of the gateway is larger than the residual processing threshold value of the currently recommended Queue, according to the size of the data load of the gateway and in combination with the data processing threshold value of other idle Queue, returning recommended gateway subscription point information splitting suggestions comprising subscription point information of the recommended splitting and the sequence of resending the subscription point information to Group Queue Manager, splitting the previous subscription point information after the gateway receives the suggestions, and sending the split point subscription information to Group Queue Manager one by one to be bound with the proper Queue in sequence; otherwise, the data load of the gateway is directly bound with a proper Queue;

the Processor is started, registration is required to be carried out to Group Queue Manager, group Queue Manager maintains registration information of the Processor, and a Queue to be processed is allocated to the Processor;

through Group Queue Manager, a proper relation is established among the gateway, the message Queue and the Processor, so that the data size sent to the Queue per second is ensured not to exceed the data size threshold T of the message Queue.

3. The optimization system of claim 1 wherein a data payload in a gateway is defined in a message Queue and a Processor allocates a message Queue.

4. The optimization method for the data acquisition cluster processing is characterized by comprising the following steps of:

the message Queue manager Group Queue Manager calculates a data amount threshold T of each Queue in the Group Queue;

the registration management mechanism of Group Queue Manager is: successful deployment of Group Queue, i.e. automatic deployment to Group QuThe ue Manager registers, and the Group Queue stores the serial number i of each Queue and the corresponding data volume threshold T _i The method comprises the steps of carrying out a first treatment on the surface of the Registering Group Queue Manager when the gateway is started, maintaining registration information of the gateway, registering subscribed point location information, and returning a proper Queue according to the data load of the gateway Group Queue Manager; if the data load of the gateway is larger than the residual processing threshold value of the currently recommended Queue, according to the size of the data load of the gateway and in combination with the data processing threshold value of other idle Queue, returning recommended gateway subscription point information splitting suggestions comprising subscription point information of the recommended splitting and the sequence of resending the subscription point information to Group Queue Manager, splitting the previous subscription point information after the gateway receives the suggestions, and sending the split point subscription information to Group Queue Manager one by one to be bound with the proper Queue in sequence; otherwise, the data load of the gateway is directly bound with a proper Queue; the Processor is started, registration is required to be carried out to Group Queue Manager, group Queue Manager maintains registration information of the Processor, and a Queue to be processed is allocated to the Processor;

the gateway assembles the real-time value of the point location into a data packet according to the actual subscribed point location information, and pushes the data packet to the corresponding Queue, and Group Queue Manager monitors the data change in the Queue;

the Processor acquires data from the queue, firstly analyzes the data packet, and then processes each point bit in the data packet.

5. The optimization method for data acquisition cluster processing according to claim 4, wherein the calculation method for the data amount threshold T of the Queue is as follows:

the simulation client sends a point position value simulation data packet with the size of y, the data structure is consistent with that of a real data packet, and the throughput Tps of the Queue at the moment is recorded as x;

adjusting the size y of the data packet, continuing to record the throughput x of the Queue, and reciprocating the data packet until x <1;

obtaining a fitting function y=f (x) by a least square method according to the data [ x, y ];

throughput according to actual configurationx _real Value, calculate threshold t=x for single Queue _real *f(x _real )。

6. The method of claim 4, wherein the Group Queue starts from sequence number 1, wherein the Queue also starts from sequence number 1 and automatically increases when the data amount threshold T is reached; when the sequence number exceeds the number k of the message queues in the Queue Group, the next Group Queue number is automatically switched to the Group Queue number +1, and if the next Group Queue is not present, namely, the Group Queue number is larger than the registered Group Queue number h, a warning prompt is given.

7. The optimization method for data acquisition cluster processing according to claim 4, wherein the gateway registration specifically comprises:

when the gateway registers, the gateway sends subscribed equipment monitoring point location information to Group Queue Manager group by group according to subscribed equipment monitoring point location information in the packets, group Queue Manager records the number p of the group of equipment monitoring point locations, and the acquisition frequency F, the point location data size Q and the importance degree L of the point locations of each point location;

calculating a data load W:

wherein p is the number of the monitoring points of the group of equipment, F _j Is the collection frequency of the monitoring point position of the j-th equipment, Q _j Is the point data size of the monitoring point of the j-th equipment, L _j Is the point location importance degree of the j-th equipment monitoring point location, F _j ×Q _j Representing the data amount of the j-th device monitoring point in the message queue in unit time, 2L _j Take the value of [1,2]As coefficients;

returning the appropriate Queue sequence number according to the following logic:

group Queue is taken from Group Queue 1 registered in Group Queue Manager and denoted as g (a), a=1; the Queue is taken from Queue 1 in Group Queue 1 and denoted as g_q(i) I=1, and the data amount threshold of the i-th Queue in the group Queue is denoted as T _i The size of the load posted in the Queue is noted as TOccupy _i ：

If TOccupy _i +W≤T _i Returning g_q (i) in g (a);

if TOccupy _i +W>T _i And carrying out the process of splitting and reassigning the subscription point information:

(1) Finding the maximum number of bits q that can be accommodated in g_q (i), i.e. at this time And->Recording subscribed device monitoring point location information of j=1 to j=q and an execution sequence s;

(2) Continuously calculating the monitoring point position information which can be contained in the g_q (i+1) from the rest equipment monitoring point position information in the group: recording j=q+1 at this time, and judgingWhether or not it is greater than T _i+1 If->Less than or equal to T _i+1 Recording subscribed device monitoring point location information of j=q+1 to j=p and an execution sequence s+1; otherwise, the maximum number of points qmax still to be accommodated in g_q (i+1), i.e. at this point +.>And is also provided withDevice supervision with notes of subscriptions for j=q+1 to j=qmaxControl point location information and execution sequence s+1; setting q=qmax, and continuing to circularly execute the step (2); during this period, if i+1>k, k is the number of queues, and g_q (i) in g (a+1) is returned, i=1; if a+1>h, h is the number of Group Queue, and an alarm is sent out;

(3) Combining the subscribed equipment monitoring point location information recorded in the step with the execution sequence to form recommended gateway subscription point location information splitting suggestion, and returning and sending the recommended gateway subscription point location information splitting suggestion to a gateway;

(4) The gateway splits the subscribed equipment monitoring point location information according to the recommended gateway subscription point location information splitting suggestion, and sends the split group to Group Queue Manager again according to the returned sequence; then re-executing the allocation operation;

if i+1>k, g_q (i) in g (a+1), i=1 is returned;

and if a is plus 1>h, an alarm is sent out.

8. The optimization method for data acquisition cluster processing according to claim 4, wherein the processing of each point in the data packet by the Processor comprises: storing real time values, storing the time sequence data base into historical values and interfacing with other processing logic.

9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the steps of the optimization method of data acquisition cluster processing according to any one of claims 4-8 when the program is executed by the processor.

10. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, carries out the steps of the optimization method of data acquisition cluster processing according to any one of claims 4-8.