CN111258758A - Streaming data processing system - Google Patents
Streaming data processing system Download PDFInfo
- Publication number
- CN111258758A CN111258758A CN202010033117.5A CN202010033117A CN111258758A CN 111258758 A CN111258758 A CN 111258758A CN 202010033117 A CN202010033117 A CN 202010033117A CN 111258758 A CN111258758 A CN 111258758A
- Authority
- CN
- China
- Prior art keywords
- data
- processing
- unit
- storage
- statistical
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 claims abstract description 34
- 238000007781 pre-processing Methods 0.000 claims abstract description 6
- 230000002452 interceptive effect Effects 0.000 claims abstract description 4
- 230000002159 abnormal effect Effects 0.000 claims description 3
- 238000011064 split stream procedure Methods 0.000 claims 3
- 238000013500 data storage Methods 0.000 abstract description 4
- 238000010801 machine learning Methods 0.000 abstract description 4
- 238000007405 data analysis Methods 0.000 abstract description 3
- 238000000034 method Methods 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 230000009172 bursting Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000007774 longterm Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5083—Techniques for rebalancing the load in a distributed system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2462—Approximate or statistical queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/248—Presentation of query results
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Probability & Statistics with Applications (AREA)
- Mathematical Physics (AREA)
- Fuzzy Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a streaming data processing system which comprises a plurality of acquisition terminals, a background terminal and a plurality of interactive terminals, wherein the background terminal further comprises a streaming unit, a first storage unit, a second storage unit, a third storage unit, a judgment unit, a first processing unit and a second processing unit. By the technical scheme, the storage and processing pressure of a single database can be reduced by preprocessing and shunting the original data when a large amount of complicated water service data are faced, so that the practical requirements of water service enterprises on data storage are met; meanwhile, the data processing speed of the water affair data is increased, efficient and safe big data analysis is realized, and machine learning, load balancing and redundancy backup are supported.
Description
Technical Field
The invention relates to the field of water affair data processing, in particular to a streaming data processing system.
Background
At present, data storage and processing in the industrial field basically adopt a single database structure, which includes: relational databases (Oracle, MySQL, SQL Server, DB2, Postgres), non-relational databases (GP, Redis, MongoDB, Memcache, Cassandra), and industrial real-time databases (Pspace, iHistorian, Historian), but none of the above single databases can meet the processing requirements of water business enterprises for massive water business data: taking 100 measuring points of one set of equipment, and taking 1000 sets of equipment as an example, the number of the measuring points reaches 10 thousands, wherein if the measuring points upload collected data every 2 seconds, 216000 times of data are needed to be uploaded for 21.6 hundred million data in one day, and one 4 byte (4B) of data is needed, the data volume reaches 86.4 hundred million B in one day, about 8GB, and the data volume reaches 2.8T in one year.
Meanwhile, when the data volume of the existing single database is TB-level data, the reading speed will be slow, frequent downtime is easy, machine learning autonomous analysis is not supported, distributed deployment and load balancing are not supported, the bursting capability is poor, the actual requirement of water data processing cannot be met, and the improvement matched with the requirement needs to be carried out on a data processing structure.
Disclosure of Invention
In view of the above problems in the prior art, a streaming data processing system is provided, and the specific technical solution is as follows:
a kind of stream data processing system, include specifically:
each acquisition end corresponds to an external water service device and is used for acquiring the working condition data of each water service device and outputting the working condition data as original data;
the backstage terminal is connected with each acquisition end respectively and is used for receiving, processing and storing the original data, and the backstage terminal further comprises:
the shunting unit is used for receiving and classifying and preprocessing the original data, and outputting a real-time operation result, statistical data and redundant data;
the first storage unit is connected with the shunting unit and used for receiving and storing a real-time operation result;
the second storage unit is connected with the shunting unit and used for receiving and storing the statistical data;
the third storage unit is connected with the shunting unit and used for receiving and storing the redundant data;
the judging unit is connected with the first storage unit and used for judging whether the corresponding water service equipment is in an abnormal working condition or not according to the real-time operation result and outputting a corresponding judgment result;
the first processing unit is connected with the second storage unit and used for extracting corresponding statistical data according to an external first user instruction, performing statistical processing and outputting a statistical report;
the second processing unit is connected with the third storage unit and used for extracting corresponding redundant data according to an external second user instruction, performing big data statistical processing and outputting a big data statistical result;
and each interactive terminal is connected with the background terminal and is used for receiving and sending a first user instruction and a second user instruction to the background terminal, and simultaneously receiving and displaying original data, a real-time operation result, a judgment result, a statistical report and a big data statistical result in real time.
Preferably, the streaming data processing system, wherein the raw data comprises real-time switching value data, real-time analog value data and basic parameter data of the water service equipment.
Preferably, the streaming data processing system, wherein the redundant data includes all original data except the traceback-type switching value data.
Preferably, in the streaming data processing system, each acquisition end acquires the working condition data of the corresponding water service equipment according to a preset frequency and transmits the original data to the background terminal in real time;
the shunting unit classifies and preprocesses the received original data according to a preset frequency.
Preferably, in the hierarchical data processing system, the statistical report includes a second-level statistical report, a hierarchical statistical report, a time-level statistical report, a daily statistical report, and a monthly statistical report.
Preferably, in the streaming data processing system, the third storage unit includes a plurality of sub storage modules, and each sub storage module is connected to the streaming data processing unit;
one of the plurality of sub-storage modules is set as a main storage module, the other sub-storage modules are all used as slave storage modules, and the main storage module monitors the storage states of all the slave storage modules in real time and performs load balancing control on the third storage unit according to the storage states.
Preferably, the streaming data processing system, wherein the second processing unit comprises a self-learning module;
the self-learning module is used for improving the efficiency of big data statistical processing.
Preferably, the streaming data processing system, wherein the second processing unit comprises a plurality of sub-processing modules, and each sub-processing module is connected to the third storage unit;
one of the plurality of sub-processing modules is set as a main processing module, the other sub-processing modules are all used as slave processing modules, the main processing module monitors the processing working conditions of all the slave processing modules in real time, and load balancing control is carried out on the second processing unit according to the processing working conditions.
By the technical scheme, when a large amount of complicated water affair data are faced, the storage and processing pressure of a single database is reduced by preprocessing and shunting the original water affair data, so that the practical requirement of a water affair enterprise on data storage is met; meanwhile, the data processing speed of the water affair data is increased, efficient and safe big data analysis is realized, and machine learning, load balancing and redundancy backup are supported.
Drawings
Fig. 1 is a schematic structural diagram of a streaming data processing system according to the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that the embodiments and features of the embodiments may be combined with each other without conflict.
The invention is further described with reference to the following drawings and specific examples, which are not intended to be limiting.
Aiming at the problems in the prior art, a streaming data processing system is provided, which is applied to the field of water service data processing, and the specific technical scheme is as follows:
a system for processing streaming data, as shown in fig. 1, specifically includes:
each acquisition end 1 corresponds to an external water service device and is used for acquiring the working condition data of each water service device and outputting the working condition data as original data;
the background terminal 2 is respectively connected with each acquisition end 1 and used for receiving, processing and storing the original data, and the background terminal 2 further comprises:
the shunting unit 21 is used for receiving and classifying and preprocessing the original data, and outputting a real-time operation result, statistical data and redundant data;
the first storage unit 22 is connected with the shunting unit 21 and used for receiving and storing a real-time operation result;
the second storage unit 23 is connected with the shunting unit 21 and used for receiving and storing the statistical data;
the third storage unit 24 is connected with the shunting unit 21 and used for receiving and storing redundant data;
the judging unit 25 is connected with the first storage unit 22 and is used for judging whether the corresponding water service equipment is in an abnormal working condition or not according to the real-time operation result and outputting a corresponding judgment result;
the first processing unit 26 is connected to the second storage unit 23, and is configured to extract corresponding statistical data according to an external first user instruction, perform statistical processing on the statistical data, and output a statistical report;
the second processing unit 27 is connected to the third storage unit 24, and is configured to extract corresponding redundant data according to an external second user instruction, perform big data statistics processing, and output a big data statistics result;
each interactive terminal 3 is connected with the background terminal 2 and is used for receiving and sending a first user instruction and a second user instruction to the background terminal 2, and simultaneously receiving and displaying original data, a real-time operation result, a judgment result, a statistical report and a big data statistical result in real time.
In a preferred embodiment, the raw data includes real-time switching value data, real-time analog value data and basic parameter data of the water service equipment.
In another preferred embodiment of the present invention, the raw data in the technical solution is further defined and explained, the raw data represents the operation condition of each water service equipment,
in a preferred embodiment, the data processing system includes a data processing unit, wherein the redundant data includes all original data except non-traceback switching value data.
In another preferred embodiment of the present invention, the redundant data is defined as all original data except the non-traceback type switching value data, which ensures that all vertical data can be saved as much as possible. It should be noted here that, in the actual operation monitoring process, the overall scale of the switching value data of the water service equipment is relatively large and most of the switching value data does not have a traceable characteristic, so that the direct rejection and discarding process is given in the data processing process; for data requiring long-term trace back, such as the switching value of the access control system of the water service equipment room, the data is divided into trace back switching value data and stored in the third storage unit 24 because of its storage value and trace back nature in the later period.
In the preferable embodiment, each acquisition end 1 acquires the working condition data of the corresponding water service equipment according to a preset frequency and transmits the original data to the background terminal 2 in real time;
the shunting unit 21 classifies and preprocesses the received original data according to a preset frequency.
In a preferred embodiment, the statistical report includes a second-level statistical report, a hierarchical statistical report, a time-level statistical report, a day-level statistical report, and a month-level statistical report.
As a preferred embodiment, the streaming data processing system, wherein the third storage unit 24 includes a plurality of sub storage modules, each of which is connected to the streaming unit 21;
one of the plurality of sub-storage modules is set as a main storage module, the other sub-storage modules are all used as slave storage modules, and the main storage module monitors the storage states of all the slave storage modules in real time and performs load balancing control on the third storage unit according to the storage states.
In a preferred embodiment, the second processing unit 27 comprises a self-learning module;
the self-learning module is used for improving the efficiency of big data statistical processing.
As a preferred embodiment, the streaming data processing system, wherein the second processing unit 27 includes a plurality of sub-processing modules, each of which is connected to the third storage unit 24;
one of the plurality of sub-processing modules is set as a main processing module, the other sub-processing modules are all used as slave processing modules, the main processing module monitors the processing working conditions of all the slave processing modules in real time, and load balancing control is carried out on the second processing unit according to the processing working conditions.
In summary, according to the technical scheme, when a large amount of complicated water affair data are faced, the storage and processing pressure of a single database can be reduced by preprocessing and shunting the original water affair data, so that the real requirement of a water affair enterprise on data storage is met; meanwhile, the data processing speed of the water affair data is increased, efficient and safe big data analysis is realized, and machine learning, load balancing and redundancy backup are supported.
While the invention has been described with reference to a preferred embodiment, it will be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the invention.
Claims (8)
1. A streaming data processing system, comprising:
each acquisition end corresponds to an external water service device and is used for acquiring the working condition data of each water service device and outputting the working condition data as original data;
the background terminal is respectively connected with each acquisition end and used for receiving, processing and storing the original data, and the background terminal further comprises:
the shunting unit is used for receiving and classifying and preprocessing the original data, and outputting a real-time operation result, statistical data and redundant data;
the first storage unit is connected with the shunting unit and used for receiving and storing the real-time operation result;
the second storage unit is connected with the shunting unit and used for receiving and storing the statistical data;
the third storage unit is connected with the shunting unit and used for receiving and storing the redundant data;
the judging unit is connected with the first storage unit and used for judging whether the corresponding water service equipment is in an abnormal working condition or not according to the real-time operation result and outputting a corresponding judgment result;
the first processing unit is connected with the second storage unit and used for extracting corresponding statistical data according to an external first user instruction, performing statistical processing and outputting a statistical report;
the second processing unit is connected with the third storage unit and used for extracting corresponding redundant data according to an external second user instruction, performing big data statistical processing and outputting a big data statistical result;
and each interactive terminal is connected with the background terminal and is used for receiving and sending the first user instruction and the second user instruction to the background terminal, and simultaneously receiving and displaying the original data, the real-time operation result, the judgment result, the statistical form and the big data statistical result in real time.
2. The split stream data processing system of claim 1, wherein the raw data comprises real-time on-off data, real-time analog data, and base parameter data for the water service equipment.
3. The split stream data processing system of claim 1, wherein the redundancy data comprises all of the original data except traceback switch quantum data.
4. The system according to claim 1, wherein each of the collection terminals collects the operating condition data of the corresponding water service equipment at a preset frequency and transmits the original data to the background terminal in real time;
and the shunting unit classifies and preprocesses the received original data according to the preset frequency.
5. The split stream data processing system of claim 1, wherein the statistics reports comprise second level statistics reports, hierarchical statistics reports, time level statistics reports, day level statistics reports, and monthly statistics reports.
6. The offload data processing system of claim 1, wherein the third storage unit comprises a plurality of sub-storage modules, each of the sub-storage modules being coupled to the offload unit;
one of the plurality of sub-storage modules is set as a main storage module, the rest of the sub-storage modules are all used as auxiliary storage modules, and the main storage module monitors the storage states of all the auxiliary storage modules in real time and performs load balancing control on the third storage unit according to the storage states.
7. The offload data processing system of claim 1, wherein the second processing unit comprises a self-learning module;
the self-learning module is used for improving the efficiency of the big data statistical processing.
8. The offload data processing system of claim 1, wherein the second processing unit comprises a plurality of sub-processing modules, each of the sub-processing modules being coupled to the third storage unit;
and one of the plurality of sub-processing modules is set as a main processing module, the other sub-processing modules are all set as slave processing modules, and the main processing module monitors the processing working conditions of all the slave processing modules in real time and performs load balancing control on the second processing unit according to the processing working conditions.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010033117.5A CN111258758A (en) | 2020-01-13 | 2020-01-13 | Streaming data processing system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010033117.5A CN111258758A (en) | 2020-01-13 | 2020-01-13 | Streaming data processing system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111258758A true CN111258758A (en) | 2020-06-09 |
Family
ID=70946903
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010033117.5A Pending CN111258758A (en) | 2020-01-13 | 2020-01-13 | Streaming data processing system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111258758A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111766815A (en) * | 2020-07-14 | 2020-10-13 | 上海威派格智慧水务股份有限公司 | Operation monitoring system of water affair equipment |
| CN112732846A (en) * | 2021-01-27 | 2021-04-30 | 深圳市科荣软件股份有限公司 | Water affair operation analysis system, method, electronic equipment and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105892364A (en) * | 2016-05-05 | 2016-08-24 | 中科智水（北京）科技有限公司 | System for centralized monitoring and management of water-related affairs |
| CN105955178A (en) * | 2016-05-05 | 2016-09-21 | 中科智水（北京）科技有限公司 | TCP series water data acquisition control intelligent terminal and water concentrated monitoring system |
| CN106097161A (en) * | 2016-08-15 | 2016-11-09 | 成都九鼎瑞信科技股份有限公司 | Water affairs management system and data processing method thereof |
| US20180299842A1 (en) * | 2017-04-12 | 2018-10-18 | Bayer Ag | Value Added Pest Control System with Smart Learning |
| CN109740941A (en) * | 2019-01-04 | 2019-05-10 | 北京环境特性研究所 | Military products data processing system |
-
2020
- 2020-01-13 CN CN202010033117.5A patent/CN111258758A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105892364A (en) * | 2016-05-05 | 2016-08-24 | 中科智水（北京）科技有限公司 | System for centralized monitoring and management of water-related affairs |
| CN105955178A (en) * | 2016-05-05 | 2016-09-21 | 中科智水（北京）科技有限公司 | TCP series water data acquisition control intelligent terminal and water concentrated monitoring system |
| CN106097161A (en) * | 2016-08-15 | 2016-11-09 | 成都九鼎瑞信科技股份有限公司 | Water affairs management system and data processing method thereof |
| US20180299842A1 (en) * | 2017-04-12 | 2018-10-18 | Bayer Ag | Value Added Pest Control System with Smart Learning |
| CN109740941A (en) * | 2019-01-04 | 2019-05-10 | 北京环境特性研究所 | Military products data processing system |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111766815A (en) * | 2020-07-14 | 2020-10-13 | 上海威派格智慧水务股份有限公司 | Operation monitoring system of water affair equipment |
| CN112732846A (en) * | 2021-01-27 | 2021-04-30 | 深圳市科荣软件股份有限公司 | Water affair operation analysis system, method, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107256219B (en) | Big data fusion analysis method applied to mass logs of automatic train control system | |
| CN204347600U (en) | A kind of production line Intelligent management control system | |
| CN110674189B (en) | Method for monitoring secondary state and positioning fault of intelligent substation | |
| CN102361458B (en) | Method for realizing high efficient data compression in rapid data management system | |
| CN109669406A (en) | A kind of remote online monitoring system and its workflow of industrial equipment | |
| CN111258758A (en) | Streaming data processing system | |
| CN111177276B (en) | Spark computing framework-based kinetic energy data processing system and method | |
| CN102750350A (en) | Monitoring system and method | |
| CN111581196A (en) | Supply and distribution power grid intelligent data acquisition and arrangement system based on intelligent factory framework | |
| CN103678688A (en) | Method for rapidly storing and retrieving messages in whole process of electric power automation communication | |
| CN111007299A (en) | Power grid data acquisition, analysis and processing system | |
| CN104601374A (en) | Network fault processing method and device for numerical control machine tool | |
| CN104778017A (en) | Method for acquiring state data of secondary device of intelligent substation | |
| CN110555583A (en) | method for uniformly processing wide-area operation data of intelligent power grid dispatching control system | |
| CN111983939A (en) | Monitoring data general acquisition and storage system and method based on cloud platform | |
| CN101158859B (en) | Defect data real time collecting device and method thereof, and production line real-time data capturing system | |
| CN112947364B (en) | System and method for early warning of equipment faults of power distribution station based on big data | |
| CN104978837A (en) | Alarm system orienting user-side transformer station and realization method thereof | |
| CN110348517B (en) | Clean energy heating system alarm classification prediction method based on big data | |
| CN112422663A (en) | Data centralization system of industrial Internet of things and data aggregation, processing and storage method | |
| CN111461915A (en) | Photovoltaic power plant operation real-time information management system and management method | |
| CN103268353A (en) | Power grid alarming automatic response system and power grid alarming automatic response method | |
| KR102453852B1 (en) | Sensor data processing system | |
| CN114003774A (en) | A big data information collection system of electric power for wisdom city | |
| CN111210205A (en) | Data processing system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200609 |
|
| RJ01 | Rejection of invention patent application after publication |