CN111210205A - Data processing system - Google Patents
Data processing system Download PDFInfo
- Publication number
- CN111210205A CN111210205A CN202010033900.1A CN202010033900A CN111210205A CN 111210205 A CN111210205 A CN 111210205A CN 202010033900 A CN202010033900 A CN 202010033900A CN 111210205 A CN111210205 A CN 111210205A
- Authority
- CN
- China
- Prior art keywords
- data
- storage
- real
- statistical
- unit
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 claims abstract description 80
- 238000007405 data analysis Methods 0.000 claims abstract description 34
- 230000003993 interaction Effects 0.000 claims abstract description 14
- 238000007619 statistical method Methods 0.000 claims description 41
- 230000002452 interceptive effect Effects 0.000 claims description 9
- 230000005540 biological transmission Effects 0.000 claims description 3
- 238000013500 data storage Methods 0.000 abstract description 5
- 238000004364 calculation method Methods 0.000 description 5
- 238000004458 analytical method Methods 0.000 description 4
- 238000010801 machine learning Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 230000009172 bursting Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000006835 compression Effects 0.000 description 1
- 238000007906 compression Methods 0.000 description 1
- 230000006837 decompression Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000007774 longterm Effects 0.000 description 1
- 238000000034 method Methods 0.000 description 1
- 230000000737 periodic effect Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/10—Office automation; Time management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2462—Approximate or statistical queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2477—Temporal data queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
- G06Q50/06—Energy or water supply
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Business, Economics & Management (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Resources & Organizations (AREA)
- Strategic Management (AREA)
- Data Mining & Analysis (AREA)
- Probability & Statistics with Applications (AREA)
- Economics (AREA)
- Tourism & Hospitality (AREA)
- Software Systems (AREA)
- Entrepreneurship & Innovation (AREA)
- General Business, Economics & Management (AREA)
- General Engineering & Computer Science (AREA)
- Health & Medical Sciences (AREA)
- Fuzzy Systems (AREA)
- Mathematical Physics (AREA)
- Marketing (AREA)
- Databases & Information Systems (AREA)
- Computational Linguistics (AREA)
- Water Supply & Treatment (AREA)
- General Health & Medical Sciences (AREA)
- Primary Health Care (AREA)
- Public Health (AREA)
- Operations Research (AREA)
- Quality & Reliability (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention discloses a data processing system, which is applied to the field of water treatment and processing and specifically comprises a plurality of acquisition ends, a processing end, a first storage end, a second storage end and an interaction end. By the technical scheme, when a large amount of complicated water affair data are faced, the requirement of data storage of a water affair enterprise is met by setting the systematized multilayer database, the work of repeatedly purchasing the database is reduced, and the storage cost is reduced; meanwhile, the data processing speed of the water service data is increased, efficient and safe big data analysis is realized, and distributed deployment, load balancing and redundant backup are supported.
Description
Technical Field
The invention relates to the field of water affair data processing, in particular to a data processing system.
Background
At present, data storage and processing in the industrial field basically adopt a single database structure, which includes: relational databases (Oracle, MySQL, SQL Server, DB2, Postgres), non-relational databases (GP, Redis, MongoDB, Memcache, Cassandra), and industrial real-time databases (Pspace, iHistorian, Historian), but none of the above single databases can meet the processing requirements of water business enterprises for massive water business data: taking 100 measuring points of one set of equipment, and taking 1000 sets of equipment as an example, the number of the measuring points reaches 10 thousands, wherein if the measuring points upload collected data every 2 seconds, 216000 times of data are needed to be uploaded for 21.6 hundred million data in one day, and one 4 byte (4B) of data is needed, the data volume reaches 86.4 hundred million B in one day, about 8GB, and the data volume reaches 2.8T in one year.
Meanwhile, when the data volume of the existing single database is TB-level data, the reading speed is slow, frequent downtime is easy, machine learning autonomous analysis is not supported, distributed deployment and load balancing are not supported, and accordingly the bursting capability is poor.
Disclosure of Invention
To solve the above problems in the prior art, a data processing system is provided, and the specific technical solution is as follows:
a data processing system is applied to water affair data processing and specifically comprises:
the plurality of acquisition ends are used for acquiring and outputting water affair data in real time;
the first storage end is respectively connected with each acquisition end and used for temporarily storing the water affair data acquired in real time and outputting a value connected with the control end of the first storage end;
the control end feeds back a real-time operation result to the first storage end after performing real-time operation according to the water affair data so as to be temporarily stored by the first storage end;
the second storage end is respectively connected with the first storage end and the control end and is used for receiving the water affair data and the real-time operation result output by the first storage end and storing the water affair data and the real-time operation result as statistical data according to a preset storage period;
the second storage end outputs the statistical data to the control end for statistical analysis, and the control end feeds back a corresponding statistical analysis result to the second storage end so that the second storage end can store the statistical analysis result according to a preset storage period;
when the storage time of the statistical data and the corresponding statistical analysis results in the second storage end exceeds the storage period, the second storage end outputs the statistical data and the corresponding statistical analysis results as scale operation data and deletes the data in the second storage end;
the third storage end is respectively connected with the first storage end, the second storage end and the control end and is used for receiving the water affair data, the real-time operation result and the scale operation data and storing the water affair data, the real-time operation result and the scale operation data as historical data for a long time;
and the third storage end simultaneously sends the scale operation data to the control end for big data analysis, and the control end feeds back a corresponding big data analysis result to the third storage end.
Preferably, the data processing system further comprises an interactive end;
the interaction end is connected with the processing end and used for providing a human-computer interaction platform, and a user can send a user instruction to the processing end through the human-computer interaction platform and check water affair data, a real-time operation result, a statistical analysis result and a big data analysis result through the human-computer interaction platform.
Preferably, in the data processing system, each acquisition end acquires the water service data according to a preset frequency;
the first storage end temporarily stores the water affair data and the real-time operation result according to the preset frequency.
Preferably, the data processing system, wherein the storage period has a value in the range of [7 days, 30 days ].
Preferably, the data processing system, wherein the second storage further comprises:
the storage unit is used for storing statistical data and statistical analysis results;
the timing unit is connected with the storage unit and used for recording and outputting the storage time of the statistical data and the statistical analysis result;
a judging unit connected with the timing unit for judging whether the storage time is longer than the storage period and outputting the judgment result
And the transmission unit is respectively connected with the storage unit and the judgment unit and used for outputting the statistical data and the statistical analysis result as scale operation data and deleting the data in the storage unit according to the judgment result when the storage time is longer than the storage period.
Preferably, the data processing system, wherein the statistical analysis result includes a second-level statistical report, a hierarchical statistical report, a time-level statistical report, a day-level statistical report and a month-level statistical report.
Preferably, the data processing system, wherein the control end further comprises:
the first processing unit is used for performing real-time operation according to the water affair data and outputting a real-time operation result;
the second processing unit is used for carrying out statistical analysis according to the statistical data and an external first user instruction and outputting a statistical analysis result;
and the third processing unit is used for performing big data analysis according to the scale operation data and an external second user instruction and outputting a big data analysis result.
Preferably, the data processing system, wherein the control end further comprises a self-learning unit;
the self-learning unit is connected with the third processing unit and used for improving the efficiency of big data analysis.
Preferably, in the data processing system, the third storage end includes a plurality of sub-storage units, and each sub-storage unit is respectively connected to the second storage end and the control end;
one of the plurality of sub-storage units is set as a main storage unit, the other sub-storage units are all used as slave storage units, and the main storage unit monitors the storage states of all the slave storage units in real time and performs load balancing control on the third storage end according to the storage states.
Preferably, in the data processing system, the third processing unit includes a plurality of sub-processing modules, and each sub-processing module is connected to the third storage end;
and a seat main processing module is arranged in the plurality of sub-processing modules, the rest sub-processing modules are all used as slave processing modules, the main processing module monitors the processing states of all the slave processing modules in real time, and the main processing module performs load balancing control on the big data analysis of the third processing unit according to the processing states.
This technical scheme has following advantage or beneficial effect:
by the technical scheme, when a large amount of complicated water affair data are faced, the requirement of data storage of a water affair enterprise is met by setting the systematized multilayer database, the work of repeatedly purchasing the database is reduced, and the storage cost is reduced; meanwhile, the data processing speed of the water service data is increased, efficient and safe big data analysis is realized, and distributed deployment, load balancing and redundant backup are supported.
Drawings
FIG. 1 is a block diagram of a data processing system according to the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that the embodiments and features of the embodiments may be combined with each other without conflict.
The invention is further described with reference to the following drawings and specific examples, which are not intended to be limiting.
Aiming at the problems in the prior art, a data processing system is provided, which is applied to the field of water affair data processing, and the specific technical scheme is as follows:
a data processing system, applied to water affair data processing, as shown in fig. 1, specifically includes:
the acquisition terminals 1 are used for acquiring and outputting water affair data in real time;
the first storage end 2 is respectively connected with each acquisition end 1 and is used for temporarily storing the water affair data acquired in real time and outputting a control value connected with the first storage end 5;
the control end 5 feeds back a real-time operation result to the first storage end 2 after performing real-time operation according to the water affair data so as to be temporarily stored in the first storage end 2;
the second storage end 3 is respectively connected with the first storage end 2 and the control end 5, and is used for receiving the water affair data and the real-time operation result output by the first storage end 2 and storing the water affair data and the real-time operation result as statistical data according to a preset storage period;
the second storage end 3 outputs the statistical data to the control end for statistical analysis, and the control end 5 feeds back a corresponding statistical analysis result to the second storage end 3 so that the second storage end 3 can store the statistical analysis result according to a preset storage period;
when the storage time of the statistical data and the corresponding statistical analysis result in the second storage terminal 3 exceeds the storage period, the second storage terminal 3 outputs the statistical data and the corresponding statistical analysis result as scale operation data and deletes the data in the second storage terminal 3;
the third storage end 4 is respectively connected with the first storage end 2, the second storage end 3 and the control end 5 and is used for receiving the water affair data, the real-time operation result and the scale operation data and storing the water affair data, the real-time operation result and the scale operation data as historical data for a long time;
the third storage end 4 simultaneously sends the scale operation data to the control end for big data analysis, and the control end 5 feeds back a corresponding big data analysis result to the third storage end 4.
In a preferred embodiment, the data processing system further comprises an interactive terminal 6;
the interaction end 6 is connected with the processing end 5 and used for providing a human-computer interaction platform, a user can send a user instruction to the processing end 5 through the human-computer interaction platform, and water affair data, a real-time operation result, a statistical analysis result and a big data analysis result are checked through the human-computer interaction platform.
In a preferred embodiment of the present invention, each of the acquisition terminals 1 is connected to an external water service device, collects working condition data of the water service device according to a predetermined frequency and transmits the working condition data to the first storage terminal 2 in real time, the first storage terminal 2 temporarily stores the water service data and performs real-time operation on the received water service data through the control terminal 5, and the obtained real-time operation result is presented through the interactive terminal 5.
In the above preferred embodiment, since the amount of the water traffic data per unit time is too large, the first storage end 2 only temporarily stores the water traffic data, i.e. only stores the water traffic data at each acquisition interval, and when the newly acquired water traffic data is transmitted to the first storage end 2
In the above preferred embodiment, the second storage terminal 3 is used for storing statistical data, the statistical data includes the real-time operation data and the working condition data, and the statistical analysis result is counted and output by the control terminal 5 according to the user instruction sent by the user through the interactive terminal 6.
In the above preferred embodiment, since the statistical data stored in the second storage terminal 3 often needs to be called and processed by the client, in order to keep the extraction of the data, i.e. the processing speed, the storage capacity of the second storage terminal 3 is not set to be too large, so that the storage capacity is not enough to support the storage for a long time, when the statistical data stored in the second storage terminal 3 exceeds a preset storage period, the part of data is stored as the scale calculation data into the third storage terminal 4 to implement the redundant backup; meanwhile, the third storage terminal 4 can also perform big data analysis on the scale operation data according to a user instruction sent by the user through the interactive terminal 6 through the control terminal 5 and output a corresponding big data analysis result.
In the above preferred embodiment, the big data analysis means statistical analysis of the whole water consumption condition based on the historical data, and the result is not only a simple form of a statistical report, but also a water consumption change condition in a certain area, or a water consumption habit analysis result of a user, and can be flexibly set according to the user's needs.
In particular, in the above preferred embodiment, in the practical process, the control terminal 5 may not be represented as an independent structure, but be combined with the first storage terminal, the second storage terminal and the third storage terminal respectively, so as to implement its own function in the form of a database.
In a preferred embodiment, each acquisition end of the data processing system acquires water service data according to a preset frequency;
the first storage end temporarily stores the water affair data and the real-time operation result according to the preset frequency.
In another preferred embodiment of the present invention, the collection mode of the collection end 1 is further defined: the acquisition end 1 acquires water affair data according to a preset frequency, wherein the preset frequency can be 1 minute/time or 2 minutes/time; correspondingly, the first storage end 2 also synchronously receives the water service data according to the preset frequency and temporarily stores the water service data.
In the above preferred embodiment, the temporary storage means that the first storage terminal 2 keeps storing the previous acquired water affair data during the acquisition interval of the acquisition terminal 1, and when new water affair data is acquired and transmitted to the first storage terminal 2, the first storage terminal 2 transmits the previous water affair data and the real-time operation result to the second storage terminal 3 and clears the data in the first storage terminal 2, so as to leave enough space for temporary storage of the subsequent incoming water affair data.
In a preferred embodiment, the data processing system, wherein the storage period has a value range of [7 days, 30 days ].
In another preferred embodiment of the present invention, the value range of the storage period of the statistical data in the second storage terminal 3 is limited to [7 days, 30 days ], and the specific value of the storage period can be flexibly set by a user according to the storable data amount of the second storage terminal 3 and the statistical data amount to be stored daily, while it is required to ensure that there is no risk of data overflow.
As a preferred embodiment, the data processing system, wherein the second storage terminal 3 further includes:
the storage unit is used for storing statistical data and statistical analysis results;
the timing unit is connected with the storage unit and used for recording and outputting the storage time of the statistical data and the statistical analysis result;
a judging unit connected with the timing unit for judging whether the storage time is longer than the storage period and outputting the judgment result
And the transmission unit is respectively connected with the storage unit and the judgment unit and used for outputting the statistical data and the statistical analysis result as scale operation data and deleting the data in the storage unit according to the judgment result when the storage time is longer than the storage period.
In another preferred embodiment of the present invention, the recording of the statistical data and the storage time of the statistical analysis result is accomplished by a timing unit.
In a preferred embodiment, the data processing system, wherein the statistical analysis result includes a second-level statistical report, a hierarchical statistical report, a time-level statistical report, a day-level statistical report, and a month-level statistical report.
As a preferred embodiment, the data processing system, wherein the control end 5 further comprises:
the first processing unit is used for performing real-time operation according to the water affair data and outputting a real-time operation result;
the second processing unit is used for carrying out statistical analysis according to the statistical data and an external first user instruction and outputting a statistical analysis result;
and the third processing unit is used for performing big data analysis according to the scale operation data and an external second user instruction and outputting a big data analysis result.
In another preferred embodiment of the present invention, the control terminal 5 comprises three independent processing units, respectively for
As a preferred embodiment, the data processing system, wherein the control end further comprises a self-learning unit;
the self-learning unit is connected with the third processing unit and used for improving the efficiency of big data analysis.
In another preferred embodiment of the present invention, a self-learning unit is disposed in the control end 5, and connected to the third processing unit, for optimizing and improving the efficiency of big data analysis. The self-learning unit can perform machine learning by observing user instructions given by a user and data analysis conditions of the third processing unit for a long time, and can optimize decompression and instruction analysis steps, so that the efficiency of big data processing and analysis is improved.
In a preferred embodiment, the data processing system, wherein the third storage terminal 4 includes a plurality of sub-storage units, each sub-storage unit is respectively connected to the second storage terminal and the control terminal;
one of the plurality of sub-storage units is set as a main storage unit, the other sub-storage units are all used as slave storage units, and the main storage unit monitors the storage states of all the slave storage units in real time and performs load balancing control on the third storage end according to the storage states.
As a preferred implementation manner, the data processing system, wherein the third processing unit includes a plurality of sub-processing modules, each of which is connected to the third storage terminal 4;
and a seat main processing module is arranged in the plurality of sub-processing modules, the rest sub-processing modules are all used as slave processing modules, the main processing module monitors the processing states of all the slave processing modules in real time, and the main processing module performs load balancing control on the big data analysis of the third processing unit according to the processing states.
A specific example is now provided to further explain and explain the present technical solution:
in the specific embodiment of the invention, firstly, a plurality of acquisition terminals 1 respectively acquire relevant water service data presented when each water service device works at a set frequency, each acquisition terminal 1 corresponds to one water service device, the acquired water service data is transmitted to a first storage terminal 2 in a wired/wireless connection mode for temporary storage, the water service data comprises switching value, analog value and basic parameter data (pressure parameter, flow parameter and the like) of the water service device, the first storage terminal 2 carries out real-time calculation on the temporary stored water service data through a control terminal 5, and the real-time calculation result is displayed to a user in real time through an interaction terminal 6.
In the above embodiment, the second storage end 3 receives the real-time water data and the real-time calculation result output by the first storage end 2 and records the real-time water data and the real-time calculation result as the statistical data for periodic storage.
In the above specific embodiment, the third storage end 4 is configured to implement redundant backup of data, support a compression ratio of 20:1 and PB-level data storage, receive the super storage period data released by the second storage end 3, and perform long-term compressed storage as history data.
In the above embodiment, the control end 5 is configured to implement processing on the water service data, and includes a first processing unit, a second processing unit, and a third processing unit, where the first processing unit is configured to perform real-time operation according to the water service data and output a real-time operation result; the second processing unit is used for carrying out statistical analysis according to the statistical data and a first user instruction input by a user through the interactive terminal 6 and outputting a statistical analysis result; the third processing unit is used for performing big data analysis according to the scale operation data and a second user instruction input by the user through the interactive terminal 6, and outputting a big data analysis result.
In the above embodiment, the first processing unit performs real-time operation on the water service data according to a preset real-time operation rule, and the second processing unit and the third processing unit perform processing according to a control instruction sent by the client through the interactive terminal 6, so that the user can select the data type and the time span covered by the statistical form by himself, and the user experience can be further optimized.
In the above specific embodiment, the third processing units of the third storage end 4 and the control end 5 both support distributed deployment and load balancing, and the operation speed and processing efficiency of big data analysis can be further improved.
In particular, in the above embodiment, the technical solution is implemented in practical practice by setting three layers of databases, wherein the first processing units of the first storage terminal 2 and the control terminal 5 are integrated into one memory database; the second processing units of the second storage terminal 3 and the control terminal 5 are integrated into at least one relational database; the third storage end 4 and the third processing unit of the control end 5 become a plurality of distributed deployed non-relational databases, and various requirements of the technical scheme can be perfectly realized.
In conclusion, by the technical scheme, when a large amount of complicated water affair data are faced, the requirement of data storage of a water affair enterprise can be met by setting the systematized multilayer databases, the work of repeatedly purchasing the databases is reduced, and the storage cost is reduced; meanwhile, the data processing speed of the water service data is increased, efficient and safe big data analysis is realized, and distributed deployment, load balancing and redundant backup are supported.
While the invention has been described with reference to a preferred embodiment, it will be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the invention.
Claims (10)
1. A data processing system for water affair data processing, comprising:
the plurality of acquisition ends are used for acquiring and outputting the water affair data in real time;
the first storage end is respectively connected with each acquisition end and used for temporarily storing the water affair data acquired in real time and outputting a value connected with the control end of the first storage end;
the control end feeds back a real-time operation result to the first storage end after performing real-time operation according to the water affair data so as to be temporarily stored by the first storage end;
the second storage end is respectively connected with the first storage end and the control end and is used for receiving the water affair data and the real-time operation result output by the first storage end and storing the water affair data and the real-time operation result as statistical data according to a preset storage period;
the second storage end outputs the statistical data to the control end for statistical analysis, and the control end feeds back a corresponding statistical analysis result to the second storage end so that the second storage end can store the statistical analysis result according to a preset storage period;
when the storage time of the statistical data and the corresponding statistical analysis result in the second storage end exceeds the storage period, the second storage end outputs the statistical data and the corresponding statistical analysis result as scale operation data and deletes the data in the second storage end;
the third storage end is respectively connected with the first storage end, the second storage end and the control end and is used for receiving the water affair data, the real-time operation result and the scale operation data and storing the water affair data, the real-time operation result and the scale operation data as historical data for a long time;
and the third storage end simultaneously sends the scale operation data to the control end for big data analysis, and the control end feeds back a corresponding big data analysis result to the third storage end.
2. The data processing system of claim 1, wherein the data processing system further comprises an interactive port;
the interaction end is connected with the processing end and used for providing a human-computer interaction platform, and a user can send a user instruction to the processing end through the human-computer interaction platform and check the water affair data, the real-time operation result, the statistical analysis result and the big data analysis result through the human-computer interaction platform.
3. The data processing system of claim 1, wherein each of the collection terminals collects the water service data at a predetermined frequency;
and the first storage end temporarily stores the water affair data and the real-time operation result according to the preset frequency.
4. The data processing system of claim 1, wherein the storage period has a value in a range of [7 days, 30 days ].
5. The data processing system of claim 1, wherein the second storage further comprises:
the storage unit is used for storing the statistical data and the statistical analysis result;
the timing unit is connected with the storage unit and is used for recording and outputting the storage time of the statistical data and the statistical analysis result;
the judging unit is connected with the timing unit, judges whether the storage time is greater than the storage period or not and outputs a judgment result
And the transmission unit is respectively connected with the storage unit and the judgment unit and used for outputting the statistical data and the statistical analysis result as scale operation data and deleting the data in the storage unit according to the judgment result when the storage time is longer than the storage period.
6. The data processing system of claim 1, wherein the statistical analysis results include a second-level statistics report, a hierarchical statistics report, a time-level statistics report, a day-level statistics report, and a monthly statistics report.
7. The data processing system of claim 1, wherein the control terminal further comprises:
the first processing unit is used for performing real-time operation according to the water affair data and outputting a real-time operation result;
the second processing unit is used for carrying out statistical analysis according to the statistical data and an external first user instruction and outputting a statistical analysis result;
and the third processing unit is used for carrying out big data analysis according to the scale operation data and an external second user instruction and outputting a big data analysis result.
8. The data processing system of claim 7, wherein the control end further comprises a self-learning unit;
the self-learning unit is connected with the third processing unit and used for improving the efficiency of big data analysis.
9. The data processing system of claim 1, wherein said third storage comprises a plurality of sub-storage units, each of said sub-storage units being coupled to said second storage and said control respectively;
one of the plurality of sub storage units is set as a main storage unit, the rest of the sub storage units are all used as auxiliary storage units, the main storage unit monitors the storage states of all the auxiliary storage units in real time, and load balancing control is carried out on the third storage end according to the storage states.
10. The data processing system of claim 7, wherein the third processing unit comprises a plurality of sub-processing modules, each of the sub-processing modules being coupled to the third storage;
and arranging a seat main processing module in the plurality of sub-processing modules, wherein the rest sub-processing modules are used as slave processing modules, and the main processing module monitors the processing states of all the slave processing modules in real time and performs load balancing control on the big data analysis of the third processing unit according to the processing states.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010033900.1A CN111210205A (en) | 2020-01-13 | 2020-01-13 | Data processing system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010033900.1A CN111210205A (en) | 2020-01-13 | 2020-01-13 | Data processing system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111210205A true CN111210205A (en) | 2020-05-29 |
Family
ID=70789079
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010033900.1A Pending CN111210205A (en) | 2020-01-13 | 2020-01-13 | Data processing system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111210205A (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103108159A (en) * | 2013-01-17 | 2013-05-15 | 新疆电力公司乌鲁木齐电业局 | Electric power intelligent video analyzing and monitoring system and method |
| CN103218680A (en) * | 2013-01-31 | 2013-07-24 | 广东省自动化研究所 | Data analysis and decision making system of water enterprise |
| CN105892364A (en) * | 2016-05-05 | 2016-08-24 | 中科智水（北京）科技有限公司 | System for centralized monitoring and management of water-related affairs |
| CN106060476A (en) * | 2016-06-29 | 2016-10-26 | 国家电网公司 | Ultra-high voltage transmission line monitoring system |
| CN106095796A (en) * | 2016-05-30 | 2016-11-09 | 中国邮政储蓄银行股份有限公司 | Distributed data storage method, Apparatus and system |
| CN109740941A (en) * | 2019-01-04 | 2019-05-10 | 北京环境特性研究所 | Military products data processing system |
-
2020
- 2020-01-13 CN CN202010033900.1A patent/CN111210205A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103108159A (en) * | 2013-01-17 | 2013-05-15 | 新疆电力公司乌鲁木齐电业局 | Electric power intelligent video analyzing and monitoring system and method |
| CN103218680A (en) * | 2013-01-31 | 2013-07-24 | 广东省自动化研究所 | Data analysis and decision making system of water enterprise |
| CN105892364A (en) * | 2016-05-05 | 2016-08-24 | 中科智水（北京）科技有限公司 | System for centralized monitoring and management of water-related affairs |
| CN106095796A (en) * | 2016-05-30 | 2016-11-09 | 中国邮政储蓄银行股份有限公司 | Distributed data storage method, Apparatus and system |
| CN106060476A (en) * | 2016-06-29 | 2016-10-26 | 国家电网公司 | Ultra-high voltage transmission line monitoring system |
| CN109740941A (en) * | 2019-01-04 | 2019-05-10 | 北京环境特性研究所 | Military products data processing system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110022226B (en) | Object-oriented data acquisition system and acquisition method | |
| CN110674189B (en) | Method for monitoring secondary state and positioning fault of intelligent substation | |
| US20170177646A1 (en) | Processing time series data from multiple sensors | |
| CN107729366A (en) | A kind of pervasive multi-source heterogeneous large-scale data synchronization system | |
| CN102361458B (en) | Method for realizing high efficient data compression in rapid data management system | |
| CN109507924B (en) | Remote monitoring system for oil field operation equipment | |
| CN103971500B (en) | A kind of remote equipment failure information acquisition system data compression transmission method | |
| CN111258758A (en) | Streaming data processing system | |
| CN102750350A (en) | Monitoring system and method | |
| CN104601374B (en) | Network failure processing method and device for Digit Control Machine Tool | |
| CN105743595A (en) | Fault early warning method and device for medium and short wave transmitter | |
| CN113141410B (en) | Dynamically adjusted QPS control method, system, device and storage medium | |
| CN107844320A (en) | State's net software version information management system and implementation method | |
| JP5591869B2 (en) | Sensor terminal device | |
| CN104391891B (en) | A kind of database isomery clone method | |
| CN111210205A (en) | Data processing system | |
| CN105338107A (en) | Stronghold operation synchronous management system and stronghold operation synchronous management method | |
| CN105553700A (en) | Intelligent equipment fault identification detection and solution providing system | |
| CN108829051B (en) | Method and device for simulating DCS avalanche working condition | |
| CN107703888A (en) | The data acquisition analysis method and Internet of Things communication equipment of Internet of Things communication equipment | |
| CN103227644A (en) | Compression method of automobile body small-format data | |
| CN106790474A (en) | A kind of energy data real-time acquisition system and acquisition method | |
| CN106254540A (en) | A kind of node serve monitoring system being applicable to distributed system and implementation method | |
| CN117609315B (en) | Data processing method, device, equipment and readable storage medium | |
| CN111858589A (en) | Method, device and system for acquiring and transmitting time sequence data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |