CN111061768A - Data processing method and system - Google Patents
Data processing method and system Download PDFInfo
- Publication number
- CN111061768A CN111061768A CN201911339999.1A CN201911339999A CN111061768A CN 111061768 A CN111061768 A CN 111061768A CN 201911339999 A CN201911339999 A CN 201911339999A CN 111061768 A CN111061768 A CN 111061768A
- Authority
- CN
- China
- Prior art keywords
- data
- module
- database
- caching
- analyzed
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2462—Approximate or statistical queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2457—Query processing with adaptation to user needs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/248—Presentation of query results
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- General Engineering & Computer Science (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- Probability & Statistics with Applications (AREA)
- Software Systems (AREA)
- Mathematical Physics (AREA)
- Fuzzy Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Memory System Of A Hierarchy Structure (AREA)
Abstract
The invention discloses a data processing method and a system, wherein the method comprises the following steps: the method comprises the steps of storing data analyzed from a data source into a database, caching the analyzed data into a data caching module, providing the data to a front-end page in real time for displaying, taking out the data from the data caching module at regular time according to preset timing duration for data statistics, and writing statistical results into the database. By adopting the technical scheme of the invention, the real-time performance of the front-end page display data can be improved.
Description
Technical Field
The present invention relates to the field of databases, and in particular, to a data processing method and system.
Background
With the rapid development of various internet services, the service volume is continuously increased, the storage capacity of a database based on data storage is increased, and the huge data volume has certain performance bottleneck on the operation of the database, for example, firstly, ten data are taken out from ten thousand data, and then, ten data are taken out from millions or even ten million data at present, and then, the extracted data are processed and applied to a service system, so that the original higher data processing performance of the database is.
The general data processing flow is as follows: processing source data provided by a data source, and then storing the data in a database; when the data is viewed, the data is extracted from the database. A series of processes need to pass through a plurality of data processing procedures, so that the data processing process is complex, a user who needs to check statistical data on an interactive platform needs to wait for a long time, and the requirement of the user for checking the current data in real time cannot be met.
Disclosure of Invention
The invention aims to provide an efficient data processing method and system aiming at the technical problem that the data processing process in the prior art cannot meet the requirement of a user for checking current data in real time.
In an embodiment of the present invention, a data processing method is provided, which includes:
the method comprises the steps of storing data analyzed from a data source into a database, caching the analyzed data into a data caching module, providing the data to a front-end page for display by the data caching module in real time, then taking out the data from the data caching module at regular time through preset timing duration for data statistics, and writing statistical results into the database.
In this embodiment of the present invention, the data processing method further includes:
and receiving the original data from the data source, and analyzing the original data.
In this embodiment of the present invention, the data processing method further includes:
and processing the analyzed data to generate data which can be stored in a database.
In this embodiment of the present invention, the data processing method further includes:
and when the data cache module fails to read the data, the front-end page acquires the data from the database for displaying.
In this embodiment of the present invention, the data processing method further includes:
and when the data cache module fails to read the data, acquiring the data from the database at regular time for data statistics, and writing the statistical result into the database.
The embodiment of the invention also provides a data processing system which comprises a database, a data cache module, a display interaction module and a timing statistic module,
the database is used for storing data to be analyzed from a data source;
the display interaction module is used for generating a front-end page for a user to view data;
the data caching module is used for caching the data analyzed from the data source and providing the data to the front-end page generated by the display interaction module in real time for display;
and the timing counting module is used for taking out data from the data caching module at regular time according to preset timing duration to carry out data counting, and writing a counting result into the database.
In an embodiment of the present invention, the data processing system further includes:
and the data analysis module is used for receiving the original data from the data source and analyzing the original data.
In an embodiment of the present invention, the data processing system further includes:
and the data processing module is used for processing the data analyzed by the data analysis module to generate data storable in the database.
In the embodiment of the invention, when the data cache module fails to read the data, the front-end page display interaction module acquires the data from the database for displaying.
In the embodiment of the invention, when the data cache module fails to read data, the timing counting module acquires the data from the database at regular time to carry out data counting, and writes the counting result into the database.
Compared with the prior art, in the data statistical method and the system, the data analyzed from the data source is stored in the database, the data is cached in the data caching module and provided to the front-end page in real time for display, then the data is regularly taken out from the data caching module for data statistics through the preset timing duration, the statistical result is written into the database, the real-time caching and timing statistics mode is adopted, the database reading and writing performance pressure brought by the traditional data processing mode is reduced, meanwhile, the operation experience of a user is improved through real-time cache data pushing, and the requirement of the user for checking the current data in real time is met; in addition, when the data in the data cache module is lost, the data can be taken out from the database for displaying and counting, so that the problem that data abnormity or follow-up operation cannot be executed due to the fault of the data cache module is solved.
Drawings
FIG. 1 is a block diagram of a data processing system according to an embodiment of the present invention.
FIG. 2 is a flow chart of a data statistics method according to an embodiment of the present invention.
Detailed Description
As shown in fig. 1, in the embodiment of the present invention, a data processing system is provided, which includes a data parsing module 1, a data processing module 2, a database 3, a display interaction module 4, a data caching module 5, and a timing statistics module 6.
The data analysis module 1 is configured to receive original data from a data source, and perform analysis processing on the original data. And the data processing module 2 is used for processing the data analyzed by the data analysis module 1 to generate data storable in a database. And the database 3 is used for storing the data processed by the data processing module 2. And the display interaction module 4 is used for providing a front-end page for a user to view data. The data caching module 5 is configured to cache the data processed by the data processing module 2 and provide the data to the front-end page generated by the display interaction module 4 in real time for display. The timing statistic module 6 is configured to periodically take out data from the data cache module 5 for data statistics according to a preset timing duration, and write a statistical result into the database 3.
When it needs to be explained, the original data acquired from the data source is analyzed by the data analyzing module 1 and processed by the data processing module to obtain the data storable in the database, and the data is sent to the database for storage and is also sent to the data caching module 5 for caching, so that the display interaction module 4 can directly acquire the data from the data caching module 5 for display without acquiring the data from the database for display, thereby improving the timeliness of the display interaction module 4 for displaying the data.
As shown in fig. 2, in the embodiment of the present invention, the data processing procedure of the data processing system includes a flow ① -a flow ⑧, which are described below.
A flow ①, namely a data analysis module → a data processing module, which sends the data analyzed from the data source by the data analysis module 1 to the data processing module 2 for processing, so as to generate the data storable in the database 3.
The process ② includes data processing module → database, and the data processed by the data processing module 2 is directly stored in the database 3.
In the process ③, the data processing module → the data caching module is synchronized with the process ①, the data processed by the data processing module 2 is cached in the data caching module 5, and the data in the data caching module 5 is stored in the memory, so that the data caching module has better performance when being used for real-time data operation.
In the conventional manner, the data is generally actively read from the database through the display interaction module 4 through an interface program, so that timeliness is poor, and reading efficiency depends on the performance of the database and the interface program, whereas in the embodiment, the data is directly pushed to the display interaction module 4 through the data cache module 5, so that timeliness of the data displayed on the front page can be improved.
In the flow ⑤, the data cache module → the timing statistic module, the timing statistic module 6 presets time length, and takes out the analytic data from the data cache module 5 at regular time for a series of data statistic processing, in the flow, the timing time length in the timing statistic module 6 is designed as a variable parameter, and how long time can be selected for data statistics for one time through self-defined time length.
The process ⑥, timing statistics module → database, writes the statistical data obtained by the timing statistics module into the database 3.
A flow ⑦ database → display interactive module, which is used to fetch the required data from the relational database for display when the data cache module 5 fails to read the data.
A flow ⑧, database → timing statistic module, which is used to take out the needed data from the relational database for statistics when the data cache module 5 fails to read the data.
Through the data processing manner in the flow ⑦ and the flow ⑧, the problem that data exception or subsequent operation cannot be executed due to a data cache module failure can be avoided.
In summary, in the data processing method and system of the present invention, while data analyzed from a data source is stored in a database, the data is cached in a data caching module and provided to a front-end page in real time for display, then the data is periodically fetched from the data caching module for data statistics through a preset timing duration, and a statistical result is written into the database, and a real-time caching and timing statistics manner is adopted to reduce the pressure on the read-write performance of the database caused by a conventional data processing manner, and meanwhile, the operation experience of a user is improved by pushing the real-time cached data, thereby satisfying the requirement of the user for checking the current data in real time; in addition, when the data in the data cache module is lost, the data can be taken out from the database for displaying and counting, so that the problem that data abnormity or follow-up operation cannot be executed due to the fault of the data cache module is solved.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents and improvements made within the spirit and principle of the present invention are intended to be included within the scope of the present invention.
Claims (10)
1. A data processing method, comprising:
the method comprises the steps of storing data analyzed from a data source into a database, caching the analyzed data into a data caching module, providing the data to a front-end page for display by the data caching module in real time, then taking out the data from the data caching module at regular time through preset timing duration for data statistics, and writing statistical results into the database.
2. The data processing method of claim 1, further comprising:
and receiving the original data from the data source, and analyzing the original data.
3. The data processing method of claim 2, further comprising:
and processing the analyzed data to generate data which can be stored in a database.
4. The data processing method of claim 1, further comprising:
and when the data cache module fails to read the data, the front-end page acquires the data from the database for displaying.
5. The data processing method of claim 1, further comprising:
and when the data cache module fails to read the data, acquiring the data from the database at regular time for data statistics, and writing the statistical result into the database.
6. A data processing system is characterized by comprising a database, a data cache module, a display interaction module and a timing statistic module,
the database is used for storing data to be analyzed from a data source;
the display interaction module is used for generating a front-end page for a user to view data;
the data caching module is used for caching the data analyzed from the data source and providing the data to the front-end page generated by the display interaction module in real time for display;
and the timing counting module is used for taking out data from the data caching module at regular time according to preset timing duration to carry out data counting, and writing a counting result into the database.
7. The data processing system of claim 6, further comprising:
and the data analysis module is used for receiving the original data from the data source and analyzing the original data.
8. The data processing system of claim 7, further comprising:
and the data processing module is used for processing the data analyzed by the data analysis module to generate data storable in the database.
9. The data processing system of claim 6, wherein the front-end page display interaction module retrieves data from the database for display when the data cannot be read due to a failure of the data cache module.
10. The data processing system of claim 6, wherein when the data cache module fails to read the data, the timing statistics module periodically obtains the data from the database for data statistics and writes the statistical result into the database.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911339999.1A CN111061768A (en) | 2019-12-23 | 2019-12-23 | Data processing method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911339999.1A CN111061768A (en) | 2019-12-23 | 2019-12-23 | Data processing method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111061768A true CN111061768A (en) | 2020-04-24 |
Family
ID=70300885
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911339999.1A Pending CN111061768A (en) | 2019-12-23 | 2019-12-23 | Data processing method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111061768A (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108282522A (en) * | 2018-01-15 | 2018-07-13 | 吉浦斯信息咨询（深圳）有限公司 | Data storage access method based on dynamic routing and system |
| US20180341693A1 (en) * | 2016-01-18 | 2018-11-29 | Alibaba Group Holding Limited | Data synchronization method, apparatus, and system |
| CN110445856A (en) * | 2019-07-31 | 2019-11-12 | 南京联迪信息系统股份有限公司 | A kind of Internet of Things communication part managing and control system and method for supporting multi-protocols access |
-
2019
- 2019-12-23 CN CN201911339999.1A patent/CN111061768A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180341693A1 (en) * | 2016-01-18 | 2018-11-29 | Alibaba Group Holding Limited | Data synchronization method, apparatus, and system |
| CN108282522A (en) * | 2018-01-15 | 2018-07-13 | 吉浦斯信息咨询（深圳）有限公司 | Data storage access method based on dynamic routing and system |
| CN110445856A (en) * | 2019-07-31 | 2019-11-12 | 南京联迪信息系统股份有限公司 | A kind of Internet of Things communication part managing and control system and method for supporting multi-protocols access |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2022126984A1 (en) | Cache data detection method and apparatus, computer device and storage medium | |
| US8688621B2 (en) | Systems and methods for information compression | |
| CN109947924B (en) | Dialogue system training data construction method and device, electronic equipment and storage medium | |
| US20090307257A1 (en) | Techniques for creating reports using cached data | |
| WO2002098822A3 (en) | System and method for generating multi-lingual reports | |
| CN103902718A (en) | Report control and application thereof | |
| CN114547274B (en) | Multi-turn question and answer method, device and equipment | |
| CN112231407B (en) | DDL synchronization method, device, equipment and medium of PostgreSQL database | |
| CN111382329A (en) | Data mining method and system for big data analysis | |
| CN111382182A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN110995273A (en) | Data compression method, device, equipment and medium for power database | |
| CN110222046B (en) | List data processing method, device, server and storage medium | |
| CN105303430B (en) | Transaction index acquisition method and device | |
| CN111061768A (en) | Data processing method and system | |
| CN114185885A (en) | Streaming data processing method and system based on column storage database | |
| CN104346378B (en) | A kind of method, apparatus and system for realizing complex data processing | |
| CN111753015B (en) | Data query method and device of payment clearing system | |
| CN113468196A (en) | Method, apparatus, system, server and medium for processing data | |
| CN112749639A (en) | Model training method and device, computer equipment and storage medium | |
| CN115827922B (en) | Visual analysis processing method, system and computer equipment based on wind power data | |
| CN117076803A (en) | Log preview optimization method and system under large-data-volume scene | |
| CN106909623A (en) | A kind of data set and date storage method of supporting efficient mass data to analyze and retrieve | |
| CN115866331A (en) | Video frame extraction analysis method, device, equipment and storage medium | |
| CN111767703B (en) | Form data acquisition method, device and system | |
| CN114564354A (en) | Database performance monitoring method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200424 |
|
| RJ01 | Rejection of invention patent application after publication |