CN113495903A - Electric power time sequence database caching method, system, equipment and readable storage medium - Google Patents
Electric power time sequence database caching method, system, equipment and readable storage medium Download PDFInfo
- Publication number
- CN113495903A CN113495903A CN202111051011.9A CN202111051011A CN113495903A CN 113495903 A CN113495903 A CN 113495903A CN 202111051011 A CN202111051011 A CN 202111051011A CN 113495903 A CN113495903 A CN 113495903A
- Authority
- CN
- China
- Prior art keywords
- time sequence
- sequence data
- file
- data file
- database
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G05—CONTROLLING; REGULATING
- G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
- G05B19/00—Programme-control systems
- G05B19/02—Programme-control systems electric
- G05B19/418—Total factory control, i.e. centrally controlling a plurality of machines, e.g. direct or distributed numerical control [DNC], flexible manufacturing systems [FMS], integrated manufacturing systems [IMS], computer integrated manufacturing [CIM]
- G05B19/4185—Total factory control, i.e. centrally controlling a plurality of machines, e.g. direct or distributed numerical control [DNC], flexible manufacturing systems [FMS], integrated manufacturing systems [IMS], computer integrated manufacturing [CIM] characterised by the network communication
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2474—Sequence data queries, e.g. querying versioned data
-
- G—PHYSICS
- G05—CONTROLLING; REGULATING
- G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
- G05B2219/00—Program-control systems
- G05B2219/30—Nc systems
- G05B2219/33—Director till display
- G05B2219/33273—DCS distributed, decentralised controlsystem, multiprocessor
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/02—Total factory control, e.g. smart factories, flexible manufacturing systems [FMS] or integrated manufacturing systems [IMS]
Abstract
The invention discloses a method, a system and equipment for caching a power time sequence database and a readable storage medium, wherein the method comprises the steps of acquiring the connection state of a data acquisition program and the time sequence database; if the connection is disconnected, the acquired time sequence data is written into a preset cache pool; the preset cache pool is a circular cache queue; when the residual writable capacity of the preset cache pool is smaller than the preset capacity, compressing a part of time sequence data files written into the preset cache pool, and storing the compressed time sequence data files into a local file of the system; deleting the partial time sequence data file from a preset cache pool; judging whether the data acquisition program is connected with the time sequence database again, if so, reading a compressed time sequence data file in a local file of the system and time sequence data stored in a preset cache pool, and writing the time sequence data into the time sequence database; the method avoids the loss of the time sequence data after the single historical station time sequence database fails, and ensures the completeness and accuracy of the historical data retrieval and query.
Description
Technical Field
The invention belongs to the technical field of distributed control systems, relates to processing of data transmitted by a distributed control system, and particularly relates to a power time sequence database caching method, system, equipment and a readable storage medium.
Background
In the field of power industry production, a large amount of time sequence data is generated in the operation process of field equipment; particularly in the era of rapid network development, the data volume shows a explosive growth trend; the time sequence data has the characteristics of high generation frequency, serious dependence on acquisition time and large amount of information of multiple measuring points, and the operation condition of the power plant can be obtained by analyzing and processing the historical time sequence data, so that the time sequence data plays a vital role in the production of the power plant.
A time sequence database of an upper computer of a Distributed Control System (DCS) stores time sequence data of all measuring points of a lower computer; the number of the measuring points of the lower computer in the distributed control system is large, time sequence data of the measuring points are mostly stored by taking time as a unit, and the data volume of the time sequence database of the upper computer is large due to the short sampling interval; at present, the storage of time sequence data of power enterprises mostly adopts a single history station operation mode; deploying a time sequence database server on a single historical station to complete operations such as receiving, compressing, storing and retrieving time sequence data; however, if the time series database server fails, the time series data cannot be put in a database, the time series data is lost, and the historical data cannot be retrieved and queried.
Disclosure of Invention
Aiming at the technical problems in the prior art, the invention provides a power time sequence database caching method, a system, equipment and a readable storage medium, and aims to solve the technical problems that time sequence data cannot be put in a warehouse after a service end of the conventional time sequence database fails, so that the time sequence data is lost, and further, historical data cannot be retrieved and queried.
In order to achieve the purpose, the invention adopts the technical scheme that:
the invention provides a power time sequence database caching method, which comprises the following steps:
acquiring a connection state of a data acquisition program and a time sequence database; if the connection is disconnected, the acquired time sequence data is written into a preset cache pool; the preset cache pool is a circular cache queue; when the residual writable capacity of the preset cache pool is smaller than the preset capacity, compressing part of the time sequence data files written into the preset cache pool to obtain compressed time sequence data files; storing the compressed time sequence data file into a local file of a system; deleting the partial time sequence data file from a preset cache pool; when the compressed time sequence data file is stored into a local file of a system, establishing and storing a timestamp index, a file index number and an MD5 check value of the compressed time sequence data file;
judging whether the data acquisition program is connected with the time sequence database again, if so, reading a compressed time sequence data file in a local file of the system and time sequence data stored in a preset cache pool, and writing the time sequence data into the time sequence database; the method comprises the following specific steps:
reading a compressed time sequence data file in a local file of the system, decompressing to obtain a decompressed time sequence data file, and sending the decompressed time sequence data file to a time sequence database;
reading a time sequence data file in a preset cache pool, and sending the time sequence data file to a time sequence database;
before reading a compressed time sequence data file in a local file of the system, calculating an MD5 check value of the compressed time sequence data file in the local file of the system, and comparing the MD5 check value with an MD5 check value of a pre-saved compressed time sequence data file; if the two are consistent, reading and decompressing; and if the two are not consistent, the system sends an alarm.
Further, the connection state between the data acquisition program and the time sequence database is obtained as follows:
and sending a data storage request to the time sequence database through the data acquisition program, wherein if the data acquisition program does not receive a storage request response of the time sequence database in a continuous preset period, the time sequence database is abnormal, namely the data acquisition program and the time sequence database are in a disconnected state.
Furthermore, a preset data space is opened up in the memory by the preset cache pool, and the preset data space is mapped to a file under a preset path.
Further, the file under the preset path is a file under a/user/mapdata path of the system physical storage device.
Further, the timestamp index, the file index number and the MD5 check value of the compressed time series data file are stored in the memory.
Further, whether the data acquisition program is connected with the time sequence database is judged, specifically as follows:
after the data acquisition program is disconnected with the time sequence database, periodically sending a key connection request to the time sequence database through the data acquisition program, and if the key connection is successful, restoring the connection between the data acquisition program and the time sequence database; if the key connection is unsuccessful, the data acquisition program is disconnected from the time sequence database.
Further, the process of reading the compressed time sequence data file in the local file of the system and the time sequence data stored in the preset cache pool and writing the time sequence data into the time sequence database specifically comprises the following steps:
acquiring a recovery request, and sending the recovery request to a plurality of idle threads in a thread pool to form a plurality of recovery threads; and restoring the compressed time sequence data file in the local file of the system and the time sequence data file in the preset cache pool to the time sequence database by utilizing a plurality of restoring threads.
The invention also provides a power time sequence database caching system, which comprises:
the cache module is used for acquiring the connection state of the data acquisition program and the time sequence database; if the connection is disconnected, the acquired time sequence data is written into a preset cache pool; the preset cache pool is a circular cache queue; when the residual writable capacity of the preset cache pool is smaller than the preset capacity, compressing part of the time sequence data files written into the preset cache pool to obtain compressed time sequence data files; storing the compressed time sequence data file into a local file of a system; deleting the partial time sequence data file from a preset cache pool; when the compressed time sequence data file is stored into a local file of a system, establishing and storing a timestamp index, a file index number and an MD5 check value of the compressed time sequence data file;
the recovery module is used for judging whether the data acquisition program is recovered and connected with the time sequence database, if so, reading a compressed time sequence data file in a local file of the system and time sequence data stored in a preset cache pool, and writing the compressed time sequence data file and the time sequence data into the time sequence database; the method comprises the following specific steps:
reading a compressed time sequence data file in a local file of the system, decompressing to obtain a decompressed time sequence data file, and sending the decompressed time sequence data file to a time sequence database;
reading a time sequence data file in a preset cache pool, and sending the time sequence data file to a time sequence database;
before reading a compressed time sequence data file in a local file of the system, calculating an MD5 check value of the compressed time sequence data file in the local file of the system, and comparing the MD5 check value with an MD5 check value of a pre-saved compressed time sequence data file; if the two are consistent, reading and decompressing; and if the two are not consistent, the system sends an alarm.
The invention also provides a power time sequence database caching device, which comprises:
a memory for storing a computer program;
and the processor is used for realizing the steps of the power time sequence database caching method when executing the computer program.
The invention also provides a computer-readable storage medium storing a computer program which, when executed by a processor, implements the steps of the power timing database caching method.
Compared with the prior art, the invention has the beneficial effects that:
the invention provides a power time sequence database caching method, which is characterized in that a preset cache pool is configured, and time sequence data of a time sequence database failure time period are stored by utilizing the preset cache pool; after the time sequence database is recovered, the time sequence data stored in the preset cache pool is recovered to the time sequence database server side, so that the time sequence data is prevented from being lost after the single historical station time sequence database fails, and the completeness and accuracy of historical data retrieval and query are ensured.
Furthermore, the process of recovering the time sequence data file in the preset cache pool by adopting a plurality of recovery threads effectively improves the efficiency of recovering the time sequence data to the time sequence database.
Furthermore, before reading the compressed time sequence data file in the local file of the system, comparing the MD5 check value of the compressed time sequence data file in the local file of the system with the MD5 check value of the compressed time sequence data file pre-stored in the memory, so that data in the time sequence database is prevented from being mistaken due to tampering of the time sequence data file, and the correctness of the time sequence database is ensured.
Drawings
FIG. 1 is a flow chart of a time series database caching method according to the present invention;
FIG. 2 is a block diagram of a timing database caching system according to the present invention;
fig. 3 is a block diagram of a time-series database caching apparatus according to the present invention.
Detailed Description
In order to make the technical problems, technical solutions and advantageous effects of the present invention more apparent, the following embodiments further describe the present invention in detail. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
As shown in fig. 1, the present invention provides a power timing database caching method, including:
acquiring a connection state of a data acquisition program and a time sequence database; if the connection is disconnected, the acquired time sequence data is written into a preset cache pool; the preset cache pool is a circular cache queue; when the residual writable capacity of the preset cache pool is smaller than the preset capacity, compressing part of the time sequence data files written into the preset cache pool to obtain compressed time sequence data files; storing the compressed time sequence data file into a local file of a system; and deleting the partial time sequence data file from a preset cache pool.
And judging whether the data acquisition program is connected with the time sequence database again, if so, reading a compressed time sequence data file in a local file of the system and time sequence data stored in a preset cache pool, and writing the time sequence data into the time sequence database.
The method specifically comprises the following steps:
step 1, configuring a preset cache pool in a memory of a system; the method comprises the steps that a preset cache pool opens up a preset data space in an internal memory, and the preset data space is mapped to a file under a preset path; preferably, the file under the preset path is a file under a user/mapdata path of the system physical storage device, and the system physical storage device is an existing data disk or a newly added disk of the system; in the invention, the preset buffer pool is a circular buffer queue.
Step 2, acquiring the connection state of the data acquisition program and the time sequence database; if the connection state is normal, ending the method; if the connection is disconnected, executing the step 3; the process of acquiring the connection state of the data acquisition program and the time sequence database specifically comprises the following steps:
and sending a data storage request to the time sequence database through the data acquisition program, wherein if the data acquisition program does not receive a storage request response of the time sequence database in a continuous preset period, the time sequence database is abnormal, namely the data acquisition program and the time sequence database are in a disconnected state.
Step 3, writing the acquired time sequence data into a preset cache pool; when the residual writable capacity of the preset cache pool is smaller than the preset capacity, compressing part of the time sequence data files written into the preset cache pool to obtain compressed time sequence data files; storing the compressed time sequence data file into a local file of a system; and deleting the partial time sequence data file from a preset cache pool to release the storage space of the preset cache pool for storing subsequent time sequence data.
In the invention, when the compressed time sequence data file is stored in a local file of a system, the time stamp index, the file index number and the MD5 check value of the compressed time sequence data file are established, and the time stamp index, the file index number and the MD5 check value of the compressed time sequence data file are stored in a memory and are used for checking the time sequence data file in the reading process.
Step 4, judging whether the data acquisition program is connected with the time sequence database again; if the data acquisition program is disconnected with the time sequence database, returning to the step 3, and continuously writing the time sequence data acquired by the data acquisition program into the preset cache pool; and if the connection between the data acquisition program and the time sequence database is recovered, executing the step 5.
In the invention, the process of judging whether the connection between the data acquisition program and the time sequence database is recovered is specifically as follows:
after the data acquisition program is disconnected with the time sequence database, periodically sending a key connection request to the time sequence database through the data acquisition program, and if the key connection is successful, restoring the connection between the data acquisition program and the time sequence database; if the key connection is unsuccessful, the data acquisition program is disconnected from the time sequence database.
Step 5, reading a compressed time sequence data file in a local file of the system and time sequence data stored in a preset cache pool, and writing the time sequence data into a time sequence database; specifically, a recovery request is obtained and sent to a plurality of idle threads in a thread pool to form a plurality of recovery threads; and restoring the compressed time sequence data file in the local file of the system and the time sequence data file in the preset cache pool to the time sequence database by utilizing a plurality of restoring threads.
The recovery process is as follows:
reading a compressed time sequence data file in a local file of the system, decompressing to obtain a decompressed time sequence data file, and sending the decompressed time sequence data file to a time sequence database;
reading a time sequence data file in a preset cache pool, and sending the time sequence data file to a time sequence database;
before reading the compressed time sequence data file in the local file of the system, calculating an MD5 check value of the compressed time sequence data file in the local file of the system, and comparing the MD5 check value with an MD5 check value of the compressed time sequence data file prestored in a memory; if the two are consistent, reading and decompressing; and if the two are not consistent, the system sends an alarm.
As shown in fig. 2, the present invention further provides a power timing database caching system, which includes a caching module and a recovery module;
the cache module is used for acquiring the connection state of the data acquisition program and the time sequence database; if the connection is disconnected, the acquired time sequence data is written into a preset cache pool; the preset cache pool is a circular cache queue; when the residual writable capacity of the preset cache pool is smaller than the preset capacity, compressing part of the time sequence data files written into the preset cache pool to obtain compressed time sequence data files; storing the compressed time sequence data file into a local file of a system; deleting the partial time sequence data file from a preset cache pool; when the compressed time sequence data file is stored into a local file of a system, establishing and storing a timestamp index, a file index number and an MD5 check value of the compressed time sequence data file; the recovery module is used for judging whether the data acquisition program is recovered and connected with the time sequence database, if so, reading a compressed time sequence data file in a local file of the system and time sequence data stored in a preset cache pool, and writing the compressed time sequence data file and the time sequence data into the time sequence database; the method comprises the following specific steps: reading a compressed time sequence data file in a local file of the system, decompressing to obtain a decompressed time sequence data file, and sending the decompressed time sequence data file to a time sequence database;
reading a time sequence data file in a preset cache pool, and sending the time sequence data file to a time sequence database; before reading a compressed time sequence data file in a local file of the system, calculating an MD5 check value of the compressed time sequence data file in the local file of the system, and comparing the MD5 check value with an MD5 check value of a pre-saved compressed time sequence data file; if the two are consistent, reading and decompressing; and if the two are not consistent, the system sends an alarm.
As shown in fig. 3, the present invention further provides a power timing database caching device, including: a memory for storing a computer program; and the processor is used for realizing the steps of the power time sequence database caching method when executing the computer program. The communication interface shown in fig. 3 is used to access an external device to obtain data.
When the processor executes the computer program, the steps of the power time sequence database caching method are implemented, for example: acquiring a connection state of a data acquisition program and a time sequence database; if the connection is disconnected, the acquired time sequence data is written into a preset cache pool; the preset cache pool is a circular cache queue; when the residual writable capacity of the preset cache pool is smaller than the preset capacity, compressing part of the time sequence data files written into the preset cache pool to obtain compressed time sequence data files; storing the compressed time sequence data file into a local file of a system; deleting the partial time sequence data file from a preset cache pool; when the compressed time sequence data file is stored into a local file of a system, establishing and storing a timestamp index, a file index number and an MD5 check value of the compressed time sequence data file; judging whether the data acquisition program is connected with the time sequence database again, if so, reading a compressed time sequence data file in a local file of the system and time sequence data stored in a preset cache pool, and writing the time sequence data into the time sequence database; the method comprises the following specific steps: reading a compressed time sequence data file in a local file of the system, decompressing to obtain a decompressed time sequence data file, and sending the decompressed time sequence data file to a time sequence database; reading a time sequence data file in a preset cache pool, and sending the time sequence data file to a time sequence database; before reading a compressed time sequence data file in a local file of the system, calculating an MD5 check value of the compressed time sequence data file in the local file of the system, and comparing the MD5 check value with an MD5 check value of a pre-saved compressed time sequence data file; if the two are consistent, reading and decompressing; and if the two are not consistent, the system sends an alarm.
Alternatively, the processor implements the functions of the modules in the system when executing the computer program, for example: the cache module is used for acquiring the connection state of the data acquisition program and the time sequence database; if the connection is disconnected, the acquired time sequence data is written into a preset cache pool; the preset cache pool is a circular cache queue; when the residual writable capacity of the preset cache pool is smaller than the preset capacity, compressing part of the time sequence data files written into the preset cache pool to obtain compressed time sequence data files; storing the compressed time sequence data file into a local file of a system; deleting the partial time sequence data file from a preset cache pool; when the compressed time sequence data file is stored into a local file of a system, establishing and storing a timestamp index, a file index number and an MD5 check value of the compressed time sequence data file; the recovery module is used for judging whether the data acquisition program is recovered and connected with the time sequence database, if so, reading a compressed time sequence data file in a local file of the system and time sequence data stored in a preset cache pool, and writing the compressed time sequence data file and the time sequence data into the time sequence database; the method comprises the following specific steps: reading a compressed time sequence data file in a local file of the system, decompressing to obtain a decompressed time sequence data file, and sending the decompressed time sequence data file to a time sequence database; reading a time sequence data file in a preset cache pool, and sending the time sequence data file to a time sequence database; before reading a compressed time sequence data file in a local file of the system, calculating an MD5 check value of the compressed time sequence data file in the local file of the system, and comparing the MD5 check value with an MD5 check value of a pre-saved compressed time sequence data file; if the two are consistent, reading and decompressing; and if the two are not consistent, the system sends an alarm.
Illustratively, the computer program may be partitioned into one or more modules/units that are stored in the memory and executed by the processor to implement the invention. The one or more modules/units may be a series of computer program instruction segments capable of performing specific functions, the instruction segments describing the execution process of the computer program in the power timing database cache device. For example, the computer program may be divided into a cache module and a recovery module, and the specific functions of each module are as follows: the cache module is used for acquiring the connection state of the data acquisition program and the time sequence database; if the connection is disconnected, the acquired time sequence data is written into a preset cache pool; the preset cache pool is a circular cache queue; when the residual writable capacity of the preset cache pool is smaller than the preset capacity, compressing part of the time sequence data files written into the preset cache pool to obtain compressed time sequence data files; storing the compressed time sequence data file into a local file of a system; deleting the partial time sequence data file from a preset cache pool; when the compressed time sequence data file is stored into a local file of a system, establishing and storing a timestamp index, a file index number and an MD5 check value of the compressed time sequence data file; the recovery module is used for judging whether the data acquisition program is recovered and connected with the time sequence database, if so, reading a compressed time sequence data file in a local file of the system and time sequence data stored in a preset cache pool, and writing the compressed time sequence data file and the time sequence data into the time sequence database; the method comprises the following specific steps: reading a compressed time sequence data file in a local file of the system, decompressing to obtain a decompressed time sequence data file, and sending the decompressed time sequence data file to a time sequence database; reading a time sequence data file in a preset cache pool, and sending the time sequence data file to a time sequence database; before reading a compressed time sequence data file in a local file of the system, calculating an MD5 check value of the compressed time sequence data file in the local file of the system, and comparing the MD5 check value with an MD5 check value of a pre-saved compressed time sequence data file; if the two are consistent, reading and decompressing; and if the two are not consistent, the system sends an alarm.
The power time sequence database cache device can be computing devices such as a desktop computer, a notebook computer, a palm computer and a cloud server. The power timing database caching apparatus may include, but is not limited to, a processor, a memory. Those skilled in the art will appreciate that fig. 3 is merely an example of a power timing database caching device, and does not constitute a limitation of the power timing database caching device, and may include more components than those shown, or combine some components, or different components, for example, the power timing database caching device may further include an input/output device, a network access device, a bus, and the like.
The processor may be a Central Processing Unit (CPU), other general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, etc. The general purpose processor may be a microprocessor or the processor may be any conventional processor or the like, the processor is a control center of the power timing database cache device, and various interfaces and lines are used to connect various parts of the whole power timing database cache device.
The memory may be used to store the computer programs and/or modules, and the processor may implement various functions of the power timing database caching apparatus by running or executing the computer programs and/or modules stored in the memory and invoking data stored in the memory.
The memory may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required by at least one function (such as a sound playing function, an image playing function, etc.), and the like; the storage data area may store data (such as audio data, a phonebook, etc.) created according to the use of the cellular phone, and the like. In addition, the memory may include high speed random access memory, and may also include non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash memory card (FlashCard), at least one disk storage device, a flash memory device, or other volatile solid state storage device.
The invention also provides a computer-readable storage medium storing a computer program which, when executed by a processor, implements the steps of the power timing database caching method.
The integrated module/unit of the power time sequence database caching device can be stored in a computer readable storage medium if the module/unit is realized in the form of a software functional unit and sold or used as an independent product.
Based on such understanding, all or part of the processes in the above method can be implemented by the present invention, and the implementation can also be accomplished by instructing relevant hardware through a computer program, which can be stored in a computer readable storage medium, and the computer program can implement the steps of the above power time sequence database caching method when being executed by a processor. Wherein the computer program comprises computer program code, which may be in source code form, object code form, executable file or preset intermediate form, etc.
The computer-readable storage medium may include: any entity or device capable of carrying the computer program code, recording medium, usb disk, removable hard disk, magnetic disk, optical disk, computer memory, Read-only memory (ROM), Random Access Memory (RAM), electrical carrier wave signals, telecommunications signals, software distribution medium, etc.
It should be noted that the computer readable storage medium may contain content that is subject to appropriate increase or decrease as required by legislation and patent practice in jurisdictions, for example, in some jurisdictions, computer readable storage media that does not include electrical carrier signals and telecommunications signals in accordance with legislation and patent practice.
Examples
In this embodiment, taking a distributed control system of a power plant as an example, the distributed control system of the power plant configures a single historical station to operate, and deploys a time sequence database on the single historical station; the embodiment provides a power time sequence database caching method, which specifically comprises the following steps:
step 1, configuring a preset cache pool in a memory of a system for caching time sequence data; the method comprises the steps that a preset cache pool opens up a preset data space in a memory, and the preset data space is mapped to a file under a preset path; the file under the preset path is the file under the user/mapdata path of the system physical storage device, and the system physical storage device is an existing data disk or a newly added disk of the system; in the invention, a preset buffer pool is a circular buffer queue; the length of the circular buffer queue is determined according to the size of the storage space of the existing data disk or the newly added disk of the historical station and the maximum collection point number of the data collection program.
In the embodiment, the length of the circular buffer queue is set to be the size capable of storing the 30-day time sequence data file of the power plant, a data space of 1000 moments is opened up in the memory of the historical station, and the preset data space is mapped to a newly added disk of the historical station, wherein the file under the preset path is located on the newly added disk of the historical station; wherein, the newly added disk adopts three hard disk groups RAID5, a file is newly built under a/user/mapdata path, and the file map is stored in the memory; the time sequence data file is written into the memory firstly and is written into the disk at regular time, so that the low speed of directly writing into the system disk is avoided, and the requirement on the read-write capability of the disk is low; the format of the file under the preset path is the total number of points + the number of uses + the storage space of each point data, and the specific format is as shown in table 1 below:
table 1 table of file formats mapped from cache pool to preset path
Step 2, acquiring the connection state of the data acquisition program and the time sequence database; if the connection state is normal, ending the method; if the connection is disconnected, executing the step 3; the process of acquiring the connection state of the data acquisition program and the time sequence database specifically comprises the following steps:
and sending a data storage request to the time sequence database through the data acquisition program, wherein if the data acquisition program does not receive a storage request response of the time sequence database in a continuous preset period, the time sequence database is abnormal, namely the data acquisition program and the time sequence database are in a disconnected state.
Step 3, writing the acquired time sequence data into a preset cache pool; when the residual writable capacity of the preset cache pool is smaller than the preset capacity, compressing part of the time sequence data files written into the preset cache pool to obtain compressed time sequence data files; storing the compressed time sequence data file into a local file of a system; deleting the partial time sequence data files from the preset cache pool to release the storage space of the preset cache pool for storing subsequent time sequence data
When the compressed time sequence data file is stored into a local file of a system, establishing a timestamp index, a file index number and an MD5 check value of the compressed time sequence data file, and storing the timestamp index, the file index number and the MD5 check value of the compressed time sequence data file in an internal memory for checking the time sequence data file in a reading process; the MD5 check value of the compressed time sequence data file is obtained by calculating the check value of the compressed time sequence data file through an MD5SUM command; the file index number is encoded from 0, the file index number of the first generated compressed time series data file is 0, and the file index numbers are +1 in sequence; and the permission of compressing the time sequence data file is read only, and the format of the compression time sequence data file is time + index.
Step 4, judging whether the data acquisition program is connected with the time sequence database again; if the data acquisition program is disconnected with the time sequence database, returning to the step 3, and continuously writing the time sequence data acquired by the data acquisition program into the preset cache pool; and if the connection between the data acquisition program and the time sequence database is recovered, executing the step 5.
The process of judging whether the connection between the data acquisition program and the time sequence database is recovered is as follows:
after the data acquisition program is disconnected with the time sequence database, periodically sending a key connection request to the time sequence database through the data acquisition program, and if the key connection is successful, restoring the connection between the data acquisition program and the time sequence database; if the key connection is unsuccessful, the data acquisition program is disconnected from the time sequence database.
Step 5, reading a compressed time sequence data file in a local file of the system and time sequence data stored in a preset cache pool, and writing the time sequence data into a time sequence database; specifically, a recovery request is obtained and sent to a plurality of idle threads in a thread pool to form a plurality of recovery threads, and an operation record is sent to prompt operation and maintenance personnel to start recovering data; recovering a compressed time sequence data file in a local file of the system and a time sequence data file in a preset cache pool to a time sequence database by utilizing a plurality of recovery threads; after the recovery is finished, the system sends a finishing alarm, and the content of finishing the alarm is the starting time + the ending time + 'the recovery of the time sequence data is finished'.
The recovery process is as follows:
reading a compressed time sequence data file in a local file of the system, decompressing to obtain a decompressed time sequence data file, and sending the decompressed time sequence data file to a time sequence database;
reading a time sequence data file in a preset cache pool, and sending the time sequence data file to a time sequence database;
before reading the compressed time sequence data file in the local file of the system, calculating an MD5 check value of the compressed time sequence data file in the local file of the system, and comparing the MD5 check value with an MD5 check value of the compressed time sequence data file prestored in a memory; if the two are consistent, reading and decompressing; and if the two are not consistent, the system sends an alarm.
In the embodiment, when the data recovery is performed by adopting a plurality of recovery threads, each recovery thread is adopted to recover the time sequence data file every 10min, and the recovery speed of the data and the recovery speed of the service can be improved by utilizing the plurality of recovery threads; meanwhile, the data volume of each time sequence data file is fixed, and the data distributed by each recovery thread is ensured to be uniform.
For a description of a relevant part in the power time sequence database caching system, the device, and the computer-readable storage medium provided in this embodiment, reference may be made to a detailed description of a corresponding part in the power time sequence database caching method described in this embodiment, and details are not repeated here.
The invention stores the time sequence data of the time sequence database failure time period by configuring a preset cache pool and utilizing the preset cache pool; after the time sequence database is recovered, the time sequence data stored in the preset cache pool is recovered to the time sequence database server side, so that the time sequence data is prevented from being lost after the single historical station time sequence database fails, and the completeness and accuracy of historical data retrieval and query are ensured.
The above-described embodiment is only one of the embodiments that can implement the technical solution of the present invention, and the scope of the present invention is not limited by the embodiment, but includes any variations, substitutions and other embodiments that can be easily conceived by those skilled in the art within the technical scope of the present invention disclosed.
Claims (10)
1. A power time sequence database caching method is characterized by comprising the following steps:
acquiring a connection state of a data acquisition program and a time sequence database; if the connection is disconnected, the acquired time sequence data is written into a preset cache pool; the preset cache pool is a circular cache queue; when the residual writable capacity of the preset cache pool is smaller than the preset capacity, compressing part of the time sequence data files written into the preset cache pool to obtain compressed time sequence data files; storing the compressed time sequence data file into a local file of a system; deleting the partial time sequence data file from a preset cache pool; when the compressed time sequence data file is stored into a local file of a system, establishing and storing a timestamp index, a file index number and an MD5 check value of the compressed time sequence data file;
judging whether the data acquisition program is connected with the time sequence database again, if so, reading a compressed time sequence data file in a local file of the system and time sequence data stored in a preset cache pool, and writing the time sequence data into the time sequence database; the method comprises the following specific steps:
reading a compressed time sequence data file in a local file of the system, decompressing to obtain a decompressed time sequence data file, and sending the decompressed time sequence data file to a time sequence database;
reading a time sequence data file in a preset cache pool, and sending the time sequence data file to a time sequence database;
before reading a compressed time sequence data file in a local file of the system, calculating an MD5 check value of the compressed time sequence data file in the local file of the system, and comparing the MD5 check value with an MD5 check value of a pre-saved compressed time sequence data file; if the two are consistent, reading and decompressing; and if the two are not consistent, the system sends an alarm.
2. The power time sequence database caching method according to claim 1, wherein a connection state between the data acquisition program and the time sequence database is obtained as follows:
and sending a data storage request to the time sequence database through the data acquisition program, wherein if the data acquisition program does not receive a storage request response of the time sequence database in a continuous preset period, the time sequence database is abnormal, namely the data acquisition program and the time sequence database are in a disconnected state.
3. The power time sequence database caching method according to claim 1, wherein a preset caching pool is used for opening up a preset data space in a memory, and the preset data space is mapped to a file under a preset path.
4. The method as claimed in claim 3, wherein the file in the predetermined path is a file in a/user/mapdata path of the system physical storage device.
5. The method as claimed in claim 1, wherein the timestamp index, the file index number and the MD5 check value of the compressed time series data file are stored in the memory.
6. The method according to claim 1, wherein whether the data acquisition program is connected to the time sequence database is determined as follows:
after the data acquisition program is disconnected with the time sequence database, periodically sending a key connection request to the time sequence database through the data acquisition program, and if the key connection is successful, restoring the connection between the data acquisition program and the time sequence database; if the key connection is unsuccessful, the data acquisition program is disconnected from the time sequence database.
7. The electric power time sequence database caching method according to claim 1, wherein a process of reading a compressed time sequence data file in a local file of a system and time sequence data stored in a preset cache pool and writing the time sequence data into the time sequence database specifically comprises:
acquiring a recovery request, and sending the recovery request to a plurality of idle threads in a thread pool to form a plurality of recovery threads; and restoring the compressed time sequence data file in the local file of the system and the time sequence data file in the preset cache pool to the time sequence database by utilizing a plurality of restoring threads.
8. A power timing database caching system, comprising:
the cache module is used for acquiring the connection state of the data acquisition program and the time sequence database; if the connection is disconnected, the acquired time sequence data is written into a preset cache pool; the preset cache pool is a circular cache queue; when the residual writable capacity of the preset cache pool is smaller than the preset capacity, compressing part of the time sequence data files written into the preset cache pool to obtain compressed time sequence data files; storing the compressed time sequence data file into a local file of a system; deleting the partial time sequence data file from a preset cache pool; when the compressed time sequence data file is stored into a local file of a system, establishing and storing a timestamp index, a file index number and an MD5 check value of the compressed time sequence data file;
the recovery module is used for judging whether the data acquisition program is recovered and connected with the time sequence database, if so, reading a compressed time sequence data file in a local file of the system and time sequence data stored in a preset cache pool, and writing the compressed time sequence data file and the time sequence data into the time sequence database; the method comprises the following specific steps:
reading a compressed time sequence data file in a local file of the system, decompressing to obtain a decompressed time sequence data file, and sending the decompressed time sequence data file to a time sequence database;
reading a time sequence data file in a preset cache pool, and sending the time sequence data file to a time sequence database;
before reading a compressed time sequence data file in a local file of the system, calculating an MD5 check value of the compressed time sequence data file in the local file of the system, and comparing the MD5 check value with an MD5 check value of a pre-saved compressed time sequence data file; if the two are consistent, reading and decompressing; and if the two are not consistent, the system sends an alarm.
9. A power timing database caching apparatus, comprising:
a memory for storing a computer program;
a processor for implementing the steps of a power timing database caching method according to any one of claims 1 to 7 when executing said computer program.
10. A computer-readable storage medium, in which a computer program is stored, which, when being executed by a processor, carries out the steps of a power timing database caching method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111051011.9A CN113495903B (en) | 2021-09-08 | 2021-09-08 | Electric power time sequence database caching method, system, equipment and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111051011.9A CN113495903B (en) | 2021-09-08 | 2021-09-08 | Electric power time sequence database caching method, system, equipment and readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113495903A true CN113495903A (en) | 2021-10-12 |
| CN113495903B CN113495903B (en) | 2021-12-17 |
Family
ID=77997139
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111051011.9A Active CN113495903B (en) | 2021-09-08 | 2021-09-08 | Electric power time sequence database caching method, system, equipment and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113495903B (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114205372A (en) * | 2021-12-08 | 2022-03-18 | 南方电网深圳数字电网研究院有限公司 | Data storage method and device of Internet of things |
| CN114218245A (en) * | 2022-02-22 | 2022-03-22 | 国网北京市电力公司 | Data processing method and device and electronic equipment |
| CN114791823A (en) * | 2022-05-10 | 2022-07-26 | 广州慧思软件科技有限公司 | Software engineering database maintenance and early warning method and system based on artificial intelligence |
| CN116257493A (en) * | 2022-12-29 | 2023-06-13 | 北京京桥热电有限责任公司 | OPC (optical clear control) network gate penetrating interface based on caching mechanism |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1407458A (en) * | 2001-08-30 | 2003-04-02 | 华为技术有限公司 | Asynchronous FIFO data caching method |
| CN103744627A (en) * | 2014-01-26 | 2014-04-23 | 武汉英泰斯特电子技术有限公司 | Method and system for compressing and storing data collected in real time |
| CN104077380A (en) * | 2014-06-26 | 2014-10-01 | 深圳信息职业技术学院 | Method and device for deleting duplicated data and system |
| CN106354774A (en) * | 2016-08-22 | 2017-01-25 | 东北大学 | Real-time industrial process big data compression and storage system and method |
| CN107871022A (en) * | 2017-12-20 | 2018-04-03 | 清华大学 | A kind of storage of time series data column, querying method and system |
| CN107995148A (en) * | 2016-10-27 | 2018-05-04 | 中国电信股份有限公司 | The anti-tamper method of file, system, terminal and credible cloud platform |
| CN108399263A (en) * | 2018-03-15 | 2018-08-14 | 北京大众益康科技有限公司 | The storage of time series data and querying method and storage and processing platform |
| CN109428899A (en) * | 2017-08-21 | 2019-03-05 | 中国石油化工股份有限公司 | File security transfer management method and system |
| CN112650807A (en) * | 2021-01-04 | 2021-04-13 | 成都知道创宇信息技术有限公司 | Data storage management method and device, electronic equipment and readable storage medium |
| CN112968751A (en) * | 2021-01-27 | 2021-06-15 | 伊之密机器人自动化科技(苏州)有限公司 | Industrial time sequence data compression method and system |
-
2021
- 2021-09-08 CN CN202111051011.9A patent/CN113495903B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1407458A (en) * | 2001-08-30 | 2003-04-02 | 华为技术有限公司 | Asynchronous FIFO data caching method |
| CN103744627A (en) * | 2014-01-26 | 2014-04-23 | 武汉英泰斯特电子技术有限公司 | Method and system for compressing and storing data collected in real time |
| CN104077380A (en) * | 2014-06-26 | 2014-10-01 | 深圳信息职业技术学院 | Method and device for deleting duplicated data and system |
| CN106354774A (en) * | 2016-08-22 | 2017-01-25 | 东北大学 | Real-time industrial process big data compression and storage system and method |
| CN107995148A (en) * | 2016-10-27 | 2018-05-04 | 中国电信股份有限公司 | The anti-tamper method of file, system, terminal and credible cloud platform |
| CN109428899A (en) * | 2017-08-21 | 2019-03-05 | 中国石油化工股份有限公司 | File security transfer management method and system |
| CN107871022A (en) * | 2017-12-20 | 2018-04-03 | 清华大学 | A kind of storage of time series data column, querying method and system |
| CN108399263A (en) * | 2018-03-15 | 2018-08-14 | 北京大众益康科技有限公司 | The storage of time series data and querying method and storage and processing platform |
| CN112650807A (en) * | 2021-01-04 | 2021-04-13 | 成都知道创宇信息技术有限公司 | Data storage management method and device, electronic equipment and readable storage medium |
| CN112968751A (en) * | 2021-01-27 | 2021-06-15 | 伊之密机器人自动化科技(苏州)有限公司 | Industrial time sequence data compression method and system |
Non-Patent Citations (3)
| Title |
|---|
| LAIYU LI，等: "Research on Compression Method of Cement Production Time Series Data Based on Feature Preservation", 《2019 IEEE 3RD ADVANCED INFORMATION MANAGEMENT, COMMUNICATES, ELECTRONIC AND AUTOMATION CONTROL CONFERENCE (IMCEC)》 * |
| 李正欣，等: "多元时间序列相似性搜索研究综述", 《控制与决策》 * |
| 陈启安，等: "《网络空间安全技术基础》", 31 December 2017 * |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114205372A (en) * | 2021-12-08 | 2022-03-18 | 南方电网深圳数字电网研究院有限公司 | Data storage method and device of Internet of things |
| CN114218245A (en) * | 2022-02-22 | 2022-03-22 | 国网北京市电力公司 | Data processing method and device and electronic equipment |
| CN114791823A (en) * | 2022-05-10 | 2022-07-26 | 广州慧思软件科技有限公司 | Software engineering database maintenance and early warning method and system based on artificial intelligence |
| CN114791823B (en) * | 2022-05-10 | 2023-03-24 | 广州慧思软件科技有限公司 | Software engineering database maintenance and early warning method and system based on artificial intelligence |
| CN116257493A (en) * | 2022-12-29 | 2023-06-13 | 北京京桥热电有限责任公司 | OPC (optical clear control) network gate penetrating interface based on caching mechanism |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113495903B (en) | 2021-12-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113495903B (en) | Electric power time sequence database caching method, system, equipment and readable storage medium | |
| CN106874348B (en) | File storage and index method and device and file reading method | |
| WO2021091489A1 (en) | Method and apparatus for storing time series data, and server and storage medium thereof | |
| CN109496292B (en) | Disk management method, disk management device and electronic equipment | |
| US10248598B2 (en) | Intelligent storage device signal transmission method for backing up data on intelligent storage module based on system type of electronic device | |
| CN113806301A (en) | Data synchronization method, device, server and storage medium | |
| CN110851474A (en) | Data query method, database middleware, data query device and storage medium | |
| CN111414362A (en) | Data reading method, device, equipment and storage medium | |
| CN111008183B (en) | Storage method and system for business wind control log data | |
| CN104317850A (en) | Data processing method and device | |
| CN115203159A (en) | Data storage method and device, computer equipment and storage medium | |
| CN110019347B (en) | Data processing method and device of block chain and terminal equipment | |
| CN115794756A (en) | Simulation file compression method and device, equipment and storage medium | |
| CN115080563A (en) | Data capturing method, device, system, electronic equipment and storage medium | |
| CN113434489A (en) | Real-time database online capacity expansion method, system, equipment and storage medium | |
| CN111538702A (en) | Hadoop-based massive small file processing method and device | |
| CN112650733A (en) | Intelligent contract state data processing method, system and device | |
| CN111597098A (en) | Data processing method and equipment | |
| CN111142804A (en) | Management system for computer storage equipment | |
| CN112035471B (en) | Transaction processing method and computer equipment | |
| CN113821492A (en) | Redis-based fan data read-write method, device, system and medium | |
| CN111625186B (en) | Data processing method, device, electronic equipment and storage medium | |
| CN112040248B (en) | Video compression method, system, terminal device and storage medium | |
| CN111935237B (en) | Log processing method and system, electronic device and storage medium | |
| CN114443713A (en) | Data writing method, device, storage medium and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |