CN110417620A - A kind of data-optimized statistical system of parallel type and method - Google Patents
A kind of data-optimized statistical system of parallel type and method Download PDFInfo
- Publication number
- CN110417620A CN110417620A CN201910669749.8A CN201910669749A CN110417620A CN 110417620 A CN110417620 A CN 110417620A CN 201910669749 A CN201910669749 A CN 201910669749A CN 110417620 A CN110417620 A CN 110417620A
- Authority
- CN
- China
- Prior art keywords
- data
- load
- statistics
- collection
- statistical
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/04—Processing captured monitoring data, e.g. for logfile generation
- H04L43/045—Processing captured monitoring data, e.g. for logfile generation for graphical visualisation of monitoring data
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
- H04L43/0805—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters by checking availability
- H04L43/0817—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters by checking availability by checking functioning
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/02—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
- H04L67/1004—Server selection for load balancing
- H04L67/1008—Server selection for load balancing based on parameters of servers, e.g. available memory or workload
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Data Mining & Analysis (AREA)
- Environmental & Geological Engineering (AREA)
- Computer Hardware Design (AREA)
- General Engineering & Computer Science (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention discloses a kind of data-optimized statistical system of parallel type and methods, the system includes data collection system, front-collection load system, data statistics load system and database, the front-collection load system includes data receiver load blocks, data transfer server and front-end system, the front-end system includes message subscribing module and packet parsing module, and the data statistics load system includes data statistics load blocks and data statistics system.The present invention achievees the effect that greatly reduce data volume when statisticalling analyze with statistics parallel type processing by being acquired after reception data in the case where distortionless compression section time hop counts evidence as far as possible, so that system reaches stable and High Availabitity.
Description
Technical field
The present invention relates to the acquisition data mart modelings in internet of things field, and in particular to a kind of data-optimized department of statistic of parallel type
System and method.
Background technique
At present when the operation conditions to electrical equipment carries out data acquisition, it is for statistical analysis often to have logarithm strong point
The case where, as soon as and traditional statistical analysis such as calculate day in data trend curve hourly, can only be by the number in one day
According to taking-up, packeting average is done, calculation amount is too concentrated, it is easy to cause the high pressure that calculates to which mentioning for fast and stable cannot be reached
For statistical data.
Summary of the invention
To solve the above-mentioned problems, the invention proposes a kind of data-optimized statistical system of parallel type and method, remembering
Part simple statistics are just carried out when recording data and have reached compression section time hop counts evidence distortionless as far as possible, are had reached and are being counted
Data volume is reduced when analysis.The specific technical solution of the present invention is as follows:
A kind of data-optimized statistical system of parallel type, including data collection system, front-collection load system, data statistics
Load system and database, in which: the front-collection load system includes data receiver load blocks, data transfer server
And front-end system, the data receiver load blocks are used to be data transfer server configuration load；The front-end system includes
Message subscribing module and packet parsing module, the message subscribing module are used to subscribe to the related subject of data transfer server,
The packet parsing module sends out data collection system to the related subject of data transfer server according to specified message format
The data parsing sent, and the data after parsing are imported in database；The data statistics load system includes that data statistics is negative
Module and data statistics system are carried, the data statistics load blocks are used to be data statistics system configuration load, the data
The history that statistical system combines the data after the newest parsing obtained from front-collection load system and obtains from database
Statistical data carries out accrual accounting, and statistical result is imported in database in the form of statistical data.
Furthermore, the data transfer server is MQTT server, and MQTT server and front-end system use
Node.js is built.
Furthermore, the database is NoSQL memory type database.
Furthermore, the data receiver load blocks use the Stream functional module of Nginx.
Furthermore, the data statistics load blocks use the basic module of Nginx.
A kind of data-optimized statistical method of parallel type, comprising the following steps:
Data collection system sends message to front-collection load system；
The data transfer server of front-collection load system receives message；
The message subscribing module of the front-end system of front-collection load system is transmitted according to the related subject of subscription from data
Server receives message；
Number is written in data after parsing by the packet parsing module analytic message of the front-end system of front-collection load system
According to library；
Front-collection load system calls the interface of data statistics load system, and data statistics load system is based on from data
The historical statistical data obtained in library carries out accrual accounting to the data after newest parsing, obtains statistical data；
Data statistics load system updates statistical data into database.
Furthermore, further includes:
Before data transfer server receives message, data receiver load blocks are server configuration load；
After front-collection load system calls the interface of data statistics load system, data statistics load blocks are data system
Meter systems configuration load；
Furthermore, the mode of the configuration load is reasonably to be weighed according to network condition and server performance setting
Value.
Compared with prior art, the invention has the following advantages:
The present invention is by data receiver and data statistics two operations while carrying out, and counts the data in the current statistic period
Number, maximum value, minimum value, average value etc., the calculation amount of system when greatly reducing each calculating.
The present invention devises two sets of loads, so that front-end system and data statistics system when receiving mass data, calculate
Amount can be shared, and the redundancy and failure of system are greatly reduced, so that system reaches stable and High Availabitity.
Detailed description of the invention
Fig. 1 is system structure diagram.
Specific embodiment
Embodiment 1:
Present embodiment discloses a kind of data-optimized statistical systems of parallel type, including communication processor, front-collection to load
System, data statistics load system and database.Communication processor is responsible for acquiring the data message of electrical equipment, constitutes a number
According to acquisition system.Front-collection load system is mainly used for receiving the data that communication processor sends over, including data receiver
Three load blocks, MQTT server and front-end system component parts.Data statistics load system is mainly used for received to its
Data carry out the accrual accounting of data, including two data statistics load blocks, data statistics system component parts.
Data receiver load blocks are used to be MQTT server configuration load.MQTT (message queue telemetering transmission) is ISO
Messaging protocol based on publish/subscribe normal form under standard (ISO/IEC PRF 20922).It works on TCP/IP protocol suite,
For publish/subscribe type messaging protocol, the present embodiment realizes that data are transmitted using MQTT server.The effect of front-end system is pair
The data received from MQTT server carry out dissection process, including message subscribing module and packet parsing module, wherein message
Subscribing module is used to subscribe to the related subject of MQTT server, and packet parsing module is according to specified message format by the number of acquisition
It is imported data in database according to parsing, and finally.
Using Nginx, (Nginx:engine x is a high performance HTTP and reverse proxy service and one
IMAP/POP3/SMTP service) Stream module be used as data receiver load blocks, for more MQTT server configuration loads,
And it is closed according to network condition or server performance (such as server bandwidth, telecom operators, process performance, the indexs such as memory) setting
The weight of reason.Weight is the server poll weight in Nginx, which is setting when technical staff builds system.The power
Value effect is carries out load distribution when carrying out load balancing, and when distribution is preferentially distributed according to weight, if than 2 services
Device weight be respectively 1:4 then, the 1st server receives 20% data, the 2nd receive 80% data.
MQTT server is built using Node.js (Node.js is a Javascript running environment) and is directed to the clothes
Business device carries out the front-end system of message subscription, to after realizing service starting while open MQTT server and front-end system.
Front-end system is after subscribing to MQTT server designated key, preceding if there is client (communication processor) to send data to the theme
The system of setting can respond the data of transmission, and then packet parsing module parses data according to specified message format,
Finally import data in database.Here it can be referred to as " initial data " by the data that front-end system is stored to database.
Data statistics load blocks are used to be data statistics system configuration load, are built using the basic module of Nginx.Number
Load blocks build load distribution to multiple data statistics systems according to statistics, are configured according to network condition and server performance
Reasonable weight, because the data that receive in front-end system include multiple equipment, which can will be more
A equipment mitigates the pressure that single server largely calculates with weight polling mode.
Data statistics system mainly copes with the accrual accounting of data and the inquiry of data and update, is built using java.Work
When making, using the multi-thread concurrent of java, the corresponding data acquisition system of equipment is retrieved, and increment system is carried out to data
Meter, the corresponding data acquisition system of final updating.The accrual accounting of the data include record the current statistic period in data amount check,
Maximum value, minimum value, average value etc..The result of statistics can be passed in database in the form of statistical data.And increment is united every time
Timing, the data source of data statistics system had both included being called by interface from front-collection to bear by data statistics load system
The latest data that loading system obtains, also includes the historical statistical data from database.
Database is NoSQL memory type database, NoSQL memory type database can as the repository after data acquisition
The characteristics of to play high-speed read-write and huge storage capacity, such as have the ability MongoDB database of high-speed read-write.
This system use process is as follows:
Communication processor sends message to the data-optimized statistical system of parallel type；
Data receiver load blocks are MQTT server configuration load；
MQTT server receives message；
The message subscribing module of front-end system receives message from server according to the related subject of subscription；
The packet parsing module analytic message of front-end system is initial data, and database is written in initial data；
Front-collection load system calls the interface of data statistics load system, and data statistics load blocks are data statistics
System configuration load, data statistics load system increase initial data based on the historical statistical data obtained from database
Amount statistics, obtains statistical data；
Data statistics load system updates statistical data into database.
Wherein, the mode of configuration load is that reasonable weight is arranged according to network condition and server performance.
This system have already been mades part simple statistics when recording data, in the compression section time distortionless as far as possible
In the case where segment data, achieve the effect that greatly reduce data volume when statisticalling analyze.
Claims (8)
1. a kind of data-optimized statistical system of parallel type, which is characterized in that including data collection system, front-collection load system
System, data statistics load system and database, in which:
The front-collection load system includes data receiver load blocks, data transfer server and front-end system, the number
It is used to be data transfer server configuration load according to load blocks are received；The front-end system includes message subscribing module and message
Parsing module, the message subscribing module are used to subscribe to the related subject of data transfer server, and the packet parsing module is pressed
It is parsed according to the data that specified message format sends data collection system to the related subject of data transfer server, and will solution
Data after analysis import in database；
The data statistics load system includes data statistics load blocks and data statistics system, and the data statistics loads mould
Block is used to be data statistics system configuration load, and the data statistics system is newest in conjunction with obtaining from front-collection load system
Parsing after data and the historical statistical data that is obtained from database carry out accrual accounting, and by statistical result with statistical number
According to form import database in.
2. the data-optimized statistical system of parallel type according to claim 1, which is characterized in that the data transfer server
For MQTT server, MQTT server and front-end system are built using node.js.
3. the data-optimized statistical system of parallel type according to claim 1, which is characterized in that the database is NoSQL
Memory type database.
4. the data-optimized statistical system of parallel type according to claim 1, which is characterized in that the data receiver loads mould
Block uses the Stream functional module of Nginx.
5. the data-optimized statistical system of parallel type according to claim 1, which is characterized in that the data statistics loads mould
Block uses the basic module of Nginx.
6. a kind of data-optimized statistical method of parallel type, which comprises the following steps:
Data collection system sends message to front-collection load system；
The data transfer server of front-collection load system receives message；
The message subscribing module of the front-end system of front-collection load system is according to the related subject of subscription from data transport service
Device receives message；
Data are written in data after parsing by the packet parsing module analytic message of the front-end system of front-collection load system
Library；
Front-collection load system calls the interface of data statistics load system, and data statistics load system is based on from database
The historical statistical data of acquisition carries out accrual accounting to the data after newest parsing, obtains statistical data；
Data statistics load system updates statistical data into database.
7. the data-optimized statistical method of parallel type according to claim 6, which is characterized in that further include:
Before data transfer server receives message, data receiver load blocks are server configuration load；
After front-collection load system calls the interface of data statistics load system, data statistics load blocks are data statistics system
System configuration load.
8. the data-optimized statistical method of parallel type according to claim 7, which is characterized in that the mode of the configuration load
For reasonable weight is arranged according to network condition and server performance.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910669749.8A CN110417620A (en) | 2019-07-24 | 2019-07-24 | A kind of data-optimized statistical system of parallel type and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910669749.8A CN110417620A (en) | 2019-07-24 | 2019-07-24 | A kind of data-optimized statistical system of parallel type and method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110417620A true CN110417620A (en) | 2019-11-05 |
Family
ID=68362822
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910669749.8A Pending CN110417620A (en) | 2019-07-24 | 2019-07-24 | A kind of data-optimized statistical system of parallel type and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110417620A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112822171A (en) * | 2020-12-30 | 2021-05-18 | 南京南瑞继保电气有限公司 | Preposed acquisition system and method based on Internet of things protocol |
| WO2022236809A1 (en) * | 2021-05-14 | 2022-11-17 | 华北电力大学扬中智能电气研究中心 | Data collection system and method, electronic device, and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101557316A (en) * | 2009-05-14 | 2009-10-14 | 阿里巴巴集团控股有限公司 | Method and system for updating statistical data |
| CN104618255A (en) * | 2014-12-29 | 2015-05-13 | 国家电网公司 | Front acquiring service system and data processing method |
| CN104639625A (en) * | 2015-01-27 | 2015-05-20 | 华南理工大学 | Data concentrator acquisition control method based on MQTT (Message Queuing Telemetry Transport), data concentrator acquisition control device based on MQTT and data concentrator acquisition control system based on MQTT |
| CN107613017A (en) * | 2017-10-13 | 2018-01-19 | 天津科技大学 | Dangerous matter sources monitoring system and its implementation based on Internet of Things |
| CN109739919A (en) * | 2019-01-04 | 2019-05-10 | 广东电网有限责任公司 | A kind of front end processor and acquisition system for electric system |
-
2019
- 2019-07-24 CN CN201910669749.8A patent/CN110417620A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101557316A (en) * | 2009-05-14 | 2009-10-14 | 阿里巴巴集团控股有限公司 | Method and system for updating statistical data |
| CN104618255A (en) * | 2014-12-29 | 2015-05-13 | 国家电网公司 | Front acquiring service system and data processing method |
| CN104639625A (en) * | 2015-01-27 | 2015-05-20 | 华南理工大学 | Data concentrator acquisition control method based on MQTT (Message Queuing Telemetry Transport), data concentrator acquisition control device based on MQTT and data concentrator acquisition control system based on MQTT |
| CN107613017A (en) * | 2017-10-13 | 2018-01-19 | 天津科技大学 | Dangerous matter sources monitoring system and its implementation based on Internet of Things |
| CN109739919A (en) * | 2019-01-04 | 2019-05-10 | 广东电网有限责任公司 | A kind of front end processor and acquisition system for electric system |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112822171A (en) * | 2020-12-30 | 2021-05-18 | 南京南瑞继保电气有限公司 | Preposed acquisition system and method based on Internet of things protocol |
| WO2022236809A1 (en) * | 2021-05-14 | 2022-11-17 | 华北电力大学扬中智能电气研究中心 | Data collection system and method, electronic device, and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US12075106B2 (en) | Message sending method and device, readable medium and electronic device | |
| CN108769017B (en) | Data communication method and device | |
| CN110417620A (en) | A kind of data-optimized statistical system of parallel type and method | |
| US20160294569A1 (en) | Quota control policy | |
| CN108228625B (en) | Push message processing method and device | |
| CN103548315A (en) | Method and apparatus for high performance low latency real time notification delivery | |
| US20100146112A1 (en) | Efficient communication techniques | |
| CN112422684A (en) | Target message processing method and device, storage medium and electronic device | |
| CN109257335B (en) | Method for maintaining back source link, back source method, related device and storage medium | |
| US20210119889A1 (en) | Processing local area network diagnostic data | |
| CN113438129A (en) | Data acquisition method and device | |
| CN113395671B (en) | Message pushing rate adjusting method and device and server | |
| CN111479161B (en) | Live broadcast quality data reporting method and device | |
| US20160088501A1 (en) | Processing customer experience events from a plurality of source systems | |
| CN109040286B (en) | Client online state maintenance method based on memory database | |
| US9130827B2 (en) | Sampling from distributed streams of data | |
| CN111698677A (en) | Method for reporting and receiving user plane statistical information and network equipment | |
| CN106850153B (en) | Data retransmission method and system | |
| EP3002910A1 (en) | Connecting computer management systems via cellular digital telecommunication networks | |
| CN112769741B (en) | Message communication method and electronic equipment | |
| CN115002209A (en) | Data processing method, device and system | |
| CN115883639A (en) | Web real-time message pushing method and device, equipment and storage medium | |
| US20160295028A1 (en) | Tracking usage window for quota control policy | |
| CN111585686B (en) | Data transmission method and device, electronic equipment and computer-readable storage medium | |
| CN116388854B (en) | Method, apparatus and storage medium for transmitting data information by adjusting virtual channel |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20191105 |