CN113220716B - Device and method for improving real-time database data processing performance - Google Patents
Device and method for improving real-time database data processing performance Download PDFInfo
- Publication number
- CN113220716B CN113220716B CN202110498192.3A CN202110498192A CN113220716B CN 113220716 B CN113220716 B CN 113220716B CN 202110498192 A CN202110498192 A CN 202110498192A CN 113220716 B CN113220716 B CN 113220716B
- Authority
- CN
- China
- Prior art keywords
- data
- real
- database
- data reading
- time
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2453—Query optimisation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Quality & Reliability (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Debugging And Monitoring (AREA)
Abstract
A device and a processing method for improving the data processing performance of a real-time database comprise the following steps: the real-time database is used for storing production real-time data collected from each power plant; the data reading node module is internally provided with a data reading node program, is arranged in the computing server, is provided with a plurality of data reading nodes, reads the production real-time data in the real-time database in parallel and stores the read data into the shared memory database; and the data reading node monitoring module is internally provided with a data reading node monitoring program and is used for starting the data reading node program and distributing the measuring point table of the real-time database to each data reading node module. The invention writes the mass data read from the real-time database into the shared memory by simultaneously operating a plurality of data reading node programs, the data processing program takes out and processes the data from the shared memory, and then writes the processed result into the real-time database.
Description
Technical Field
The invention relates to the technical field of development of computer server sides, in particular to a device and a processing method for improving the data processing performance of a real-time database.
Background
In developing a business system based on real-time data, a real-time database is required as a storage medium for the data. The common databases on the market are PI, rython, etc., and all these databases provide SDKs with data access to the service system. Given the access pressure on databases, none of these SDKs generally support multi-threaded parallel access. Therefore, in some cases where the calculation needs milliseconds and the amount of data is large, the performance of real-time data access cannot meet the services in the conventional manner.
Disclosure of Invention
In order to solve the problem of low performance of real-time data access in a conventional manner, the invention aims to provide a device and a processing method for improving the data processing performance of a real-time database.
In order to achieve the purpose, the technical scheme adopted by the invention and the beneficial effects of the invention are as follows:
an apparatus for improving real-time database data processing performance, comprising:
a real-time database 1 for storing production real-time data collected from each power plant;
the data reading node module 2 is internally provided with a data reading node program, is arranged in the computing server, is provided with a plurality of data reading nodes, reads the production real-time data in the real-time database 1 in parallel, and stores the read data into the shared memory database 4;
and the data reading node monitoring module 3 is internally provided with a data reading node monitoring program and is used for starting the data reading node program and distributing the measuring point table of the real-time database 1 to each data reading node module 2.
The data reading node module 2 consists of four parts of functions, including measuring point table and configuration acquisition, real-time data reading, cache queue and memory database writing;
the measuring point table and the configuration acquire the measuring point table of the real-time database 1 required to be read from the node monitoring module 3 through the web service, and the configuration of the reading interval and the number of the write-in threads of the memory database;
the real-time data reading is configured by the obtained measuring point table and the reading interval, and the data in the real-time database 1 is periodically read by the measuring point name;
the buffer queue passes the test roll name; a value; writing the read data into a cache queue in a timestamp mode;
and the memory database writing creates a plurality of threads according to the obtained number of the memory database writing threads, and each thread takes out data from the cache queue and writes the data into the memory database 4.
The shared memory database 4 is used for storing the real-time data to be processed read by the data reading node module 2 and providing a high-concurrency and high-performance data reading interface for the data processing module 5;
the data processing module 5 is provided with a data processing program inside, reads data from the shared memory database 4 for processing, and writes the processing result into the real-time database 1 again.
And the data processing module 5 reads data from the shared memory database 4 for data cleaning or calculation.
A processing method for improving the data processing performance of a real-time database comprises the following steps;
step 1:
deploying all programs except the real-time database 1 on one server;
step 2:
starting a shared memory database 4;
and step 3:
configuring the number of nodes needing to be started in a data reading node monitoring module 3, and configuring a measuring point table of a real-time database 1 needing to be read by each node;
and 4, step 4:
the data reading node monitoring program module 3 starts all the data reading node program modules 2 and distributes the measuring point table of the real-time database 1 to different nodes;
and 5:
the data reading node program module 2 writes the read data into the shared memory database 4;
and 6:
and the data processing program module 5, wherein the data processing program 5 writes the data processing result into the real-time database 1.
The invention has the beneficial effects that:
the invention reads the data in the real-time database 1 in parallel by deploying a plurality of data reading node program modules 2. The read data is then stored in the shared memory database 4. The data processing program 5 reads data from the in-memory database and performs data cleaning or calculation. In the data reading process, parallel access is simulated in a multi-node mode, and the data reading performance is improved. In the data processing process, the overhead caused by network access and file access is reduced by using the shared memory database 4. And finally, the calculation function of millisecond level required by the business system is supported.
Drawings
Fig. 1 is a schematic deployment view of the present apparatus.
FIG. 2 is a schematic diagram of a data processing flow according to the present invention.
FIG. 3 is a data reading node module program architecture diagram.
Fig. 4 is a schematic diagram of a conventional data processing flow.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings.
The specific process of a device and a processing method for improving the data processing performance of a real-time database is introduced as follows:
the SDK for data access of the real-time database has the characteristics that only one access instance can exist in each data reading node program, and even if the multithreading technology is used, the access instances are finally restrained in a form of locks and are guaranteed to be used by threads in a queuing mode in a concurrent state. However, if a plurality of data reading node programs are started simultaneously, each data reading node program can have its own access instance, so that a plurality of access instances can read data in parallel, and the data reading efficiency is improved.
The data of each data reading node program is isolated by using a process. For security reasons, the data processing program cannot directly read the data of the clients. The data of these clients needs to be written into a common database. Here to improve read write performance; network and file access expenses are reduced, and the memory database is selected to be used for storing the data.
The data processing program reads data from the shared memory database 4 in a multithreading mode, and performs processing such as cleaning and calculation on the data. The results are then written to the real-time database 1.
As shown in fig. 3: the data reading node module 2 consists of four parts of functions, including measuring point table and configuration acquisition, real-time data reading, cache queue and memory database writing;
the measuring point table and the configuration obtain the measuring point table of the real-time database 1 required to be read from the node monitoring module 3 through the web service, and the configuration of the reading interval and the number of the writing threads of the memory database;
the real-time data reading is configured by the obtained measuring point table and the reading interval, and the data in the real-time database 1 is periodically read by the measuring point name;
the buffer queue passes the test roll name; a value; writing the read data into a cache queue in a timestamp mode;
and the memory database writing creates a plurality of threads according to the obtained number of the memory database writing threads, and each thread takes out data from the cache queue and writes the data into the memory database 4.
In summary, the specific implementation steps of the present invention are as follows:
step 1:
according to fig. 1, all programs except the real-time database 1 are deployed on one server;
step 2:
starting a shared memory database 4;
and step 3:
configuring the number of nodes needing to be started in a data reading node monitoring program 3, and configuring a measuring point table of a real-time database 1 needing to be read by each node;
and 4, step 4:
the data reading node monitoring program module 3 starts all the data reading node program modules 2 and distributes the measuring point tables of the database 1 to different nodes;
and 5:
the data reading node program module 2 writes the read data into the shared memory database 4;
and 5:
and the data processing program module 5 is used for writing the data processing result into the real-time database 1 by the data processing program 5.
Compared with the conventional data processing flow (figure 4), the invention (figure 2) improves the processing performance of the real-time database data. Support is provided for the millisecond-level computing function required by the business system.
Claims (5)
1. An apparatus for improving real-time database data processing performance, comprising:
a real-time database (1) for storing production real-time data collected from each power plant;
the data reading node module (2) is internally provided with a data reading node program, is arranged in the computing server, is provided with a plurality of data reading nodes, reads the production real-time data in the real-time database (1) in parallel, and stores the read data into the shared memory database (4);
the data reading node monitoring module (3) is internally provided with a data reading node monitoring program and is used for starting the data reading node program and distributing the measuring point table of the real-time database (1) to each data reading node module (2);
the data reading node module (2) consists of four parts of functions, including measuring point table and configuration acquisition, real-time data reading, cache queue and memory database writing;
the measuring point table and the configuration acquire the measuring point table of the real-time database (1) required to be read from the node monitoring module (3) through the web service, and the configuration of the number of the read intervals and the write threads of the memory database;
the real-time data reading is configured by the acquired measuring point table and the reading interval, and the data in the real-time database (1) is periodically read by the name of the measuring point;
the buffer queue passes the test roll name; a value; writing the read data into a cache queue in a timestamp mode;
and the memory database writing creates a plurality of threads according to the obtained number of the memory database writing threads, and each thread takes out data from the cache queue and writes the data into the memory database (4).
2. An apparatus for improving data processing performance of a real-time database according to claim 1, wherein the shared memory database (4) is configured to store the real-time data to be processed read by the data reading node module (2), and provide a high-concurrency, high-performance data reading interface to the data processing module (5).
3. The device for improving the data processing performance of the real-time database according to claim 2, wherein the data processing module (5) is internally provided with a data processing program, reads data from the shared memory database (4) for processing, and rewrites the processing result into the real-time database (1).
4. An arrangement for improving the performance of real-time database data processing according to claim 2, characterized in that the data processing module (5) reads data from the shared memory database (4) for data cleansing or calculation.
5. The processing method of the device for improving the data processing performance of the real-time database is characterized by comprising the following steps;
step 1:
deploying all programs except the real-time database (1) on one server;
step 2:
starting a shared memory database (4);
and step 3:
configuring the number of nodes needing to be started in a data reading node monitoring module (3), and configuring a measuring point table of a real-time database (1) needing to be read by each node;
and 4, step 4:
the data reading node monitoring program module (3) starts all the data reading node program modules (2) and distributes the measuring point tables of the real-time database (1) to different nodes;
and 5:
the data reading node program module (2) writes the read data into the shared memory database (4);
step 6:
and the data processing program module (5) is used for writing the data processing result into the real-time database (1) by the data processing program (5).
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110498192.3A CN113220716B (en) | 2021-05-08 | 2021-05-08 | Device and method for improving real-time database data processing performance |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110498192.3A CN113220716B (en) | 2021-05-08 | 2021-05-08 | Device and method for improving real-time database data processing performance |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113220716A CN113220716A (en) | 2021-08-06 |
| CN113220716B true CN113220716B (en) | 2023-03-03 |
Family
ID=77091751
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110498192.3A Active CN113220716B (en) | 2021-05-08 | 2021-05-08 | Device and method for improving real-time database data processing performance |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113220716B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114547018B (en) * | 2022-04-24 | 2022-08-16 | 西安热工研究院有限公司 | Method and system for automatically cleaning waste points of SIS real-time database |
| CN114579408A (en) * | 2022-05-05 | 2022-06-03 | 西安热工研究院有限公司 | System and method for analyzing real-time equation of real-time database |
| CN115794900A (en) * | 2022-11-10 | 2023-03-14 | 南京捷崎信息科技有限公司 | Data processing method and system |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106599036A (en) * | 2016-11-07 | 2017-04-26 | 济中节能技术（苏州）有限公司 | Server cluster-based parallel real-time database |
| CN106897458A (en) * | 2017-03-10 | 2017-06-27 | 广州白云电器设备股份有限公司 | A kind of storage and search method towards electromechanical equipment data |
| CN109669929A (en) * | 2018-12-14 | 2019-04-23 | 江苏瑞中数据股份有限公司 | Method for storing real-time data and system based on distributed parallel database |
| CN109947810A (en) * | 2017-11-16 | 2019-06-28 | 上海明匠智能系统有限公司 | Industrial equipment data real-time collecting system |
| CN112269835A (en) * | 2020-11-10 | 2021-01-26 | 浪潮云信息技术股份公司 | Method for asynchronously reading and processing batch data by distributed database |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8229945B2 (en) * | 2008-03-20 | 2012-07-24 | Schooner Information Technology, Inc. | Scalable database management software on a cluster of nodes using a shared-distributed flash memory |
| US8788465B2 (en) * | 2010-12-01 | 2014-07-22 | International Business Machines Corporation | Notification of configuration updates in a cluster system |
| CN103514261B (en) * | 2013-08-13 | 2017-03-15 | 北京华电天益信息科技有限公司 | A kind of asynchronous storage of data for being applied to industrial control system and access method |
| US11074265B2 (en) * | 2018-04-02 | 2021-07-27 | International Business Machines Corporation | Expression data structure dataset processing |
-
2021
- 2021-05-08 CN CN202110498192.3A patent/CN113220716B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106599036A (en) * | 2016-11-07 | 2017-04-26 | 济中节能技术（苏州）有限公司 | Server cluster-based parallel real-time database |
| CN106897458A (en) * | 2017-03-10 | 2017-06-27 | 广州白云电器设备股份有限公司 | A kind of storage and search method towards electromechanical equipment data |
| CN109947810A (en) * | 2017-11-16 | 2019-06-28 | 上海明匠智能系统有限公司 | Industrial equipment data real-time collecting system |
| CN109669929A (en) * | 2018-12-14 | 2019-04-23 | 江苏瑞中数据股份有限公司 | Method for storing real-time data and system based on distributed parallel database |
| CN112269835A (en) * | 2020-11-10 | 2021-01-26 | 浪潮云信息技术股份公司 | Method for asynchronously reading and processing batch data by distributed database |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113220716A (en) | 2021-08-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113220716B (en) | Device and method for improving real-time database data processing performance | |
| Zhang et al. | GraphP: Reducing communication for PIM-based graph processing with efficient data partition | |
| US9619430B2 (en) | Active non-volatile memory post-processing | |
| CN111077870A (en) | Intelligent OPC data real-time acquisition and monitoring system and method based on stream calculation | |
| Qian et al. | Timestream: Reliable stream computation in the cloud | |
| Rabl et al. | Solving big data challenges for enterprise application performance management | |
| CN104391903A (en) | Distributed storage and parallel calculation-based power grid data quality detection method | |
| KR20100070968A (en) | Cluster data management system and method for data recovery using parallel processing in cluster data management system | |
| CN110032558B (en) | Data synchronization method, device and system and storage medium | |
| CN108595605A (en) | A kind of construction method of car networking platform database | |
| CN103793442A (en) | Spatial data processing method and system | |
| CN112800058A (en) | Method for realizing HBase secondary index | |
| Lu et al. | TridentKV: A read-Optimized LSM-tree based KV store via adaptive indexing and space-efficient partitioning | |
| Andreolini et al. | A software architecture for the analysis of large sets of data streams in cloud infrastructures | |
| US10558391B2 (en) | Data processing system and data processing method | |
| CN112328569A (en) | Construction method based on Flume distributed data collection architecture | |
| CN108628897A (en) | Operation management method based on fast data and big data Technical Architecture | |
| CN103236938A (en) | Method and system for user action collection based on cache memory and asynchronous processing technology | |
| Ding et al. | Commapreduce: An improvement of mapreduce with lightweight communication mechanisms | |
| CN111666144A (en) | Batch processing task execution method and system and machine room deployment system | |
| Sang et al. | A fast implementation of parallel discrete-event simulation on GPGPU | |
| CN116185298A (en) | Method for distributed storage of logs | |
| Song | Performance and energy optimization on TeraSort algorithm by task self-resizing | |
| CN113641654A (en) | Marketing handling rule engine method based on real-time event | |
| CN112783620A (en) | Optimization method for timing task scheduling, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |