CN112380260B - Broken line caching method based on different acquisition scenes - Google Patents
Broken line caching method based on different acquisition scenes Download PDFInfo
- Publication number
- CN112380260B CN112380260B CN202110057265.5A CN202110057265A CN112380260B CN 112380260 B CN112380260 B CN 112380260B CN 202110057265 A CN202110057265 A CN 202110057265A CN 112380260 B CN112380260 B CN 112380260B
- Authority
- CN
- China
- Prior art keywords
- data
- data acquisition
- abnormal
- storage database
- acquisition module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2308—Concurrency control
- G06F16/2315—Optimistic concurrency control
- G06F16/2322—Optimistic concurrency control using timestamps
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2365—Ensuring data consistency and integrity
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Security & Cryptography (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
A broken line caching method based on different acquisition scenes is characterized in that different processing is performed according to different data acquisition source types and abnormal types; when the data acquisition source is a database, recording the abnormal starting time when the data acquisition source is abnormal, and trying to reconnect until the connection is recovered, and continuing to acquire data; when the data acquisition source is of a protocol push type and the abnormal type is abnormal at the data acquisition source side, outputting a data loss report; when the data acquisition source is of a protocol push type and the abnormal type is of a storage database side abnormality, the data acquisition module acquires data normally according to configuration requirements, stores the data into a specified temporary storage, and sends a connection request to the storage database according to a preset period until the connection with the storage database is recovered to be normal. The invention can adopt different line break caching mechanisms according to different configuration information; when data are collected from the time sequence database, the integrity of the data can be automatically ensured in a database caching mode.
Description
Technical Field
The invention belongs to the technical field of data acquisition, and relates to a broken line caching method for different acquisition scenes.
Background
With the development of science and technology, data information becomes more and more important, which not only can reflect the operation state of the service, but also can assist the improvement of production efficiency and operation decision. Data acquisition plays an important role in the whole power plant information system, and the stability of the working state of the system is important to the influence of main equipment. Along with the complexity and diversity of service scenes, when various databases and data acquisition protocols are simultaneously applied, the problem of timely data complementary acquisition when data acquisition is abnormal is an important problem at present. The research on the data acquisition abnormity problem is focused on a specific application scene as a target, different acquisition abnormity problems exist in the application scenes generally, the importance of abnormity is described, and a data cache mechanism capable of flexibly processing different scenes is designed, so that the important problem is solved.
Various methods for processing data acquisition abnormity problems exist in the academic world and the industrial world, most abnormity researches based on data acquisition pay attention to a single scene, and certain defects exist in data complementary acquisition capacity. The method is based on a broken line cache mechanism combining multiple scenes, creatively provides a new idea and a new method for solving the problem of abnormal data acquisition, and can effectively ensure the integrity of data by adopting different broken line cache mechanisms according to different changing scenes among a data acquisition source, a data acquisition module and a database.
Disclosure of Invention
In order to overcome the defects in the prior art, the invention provides a broken line caching method based on different scenes.
The invention specifically adopts the following technical scheme:
a line break caching method based on different acquisition scenes is characterized by comprising the following steps:
step 1: the data acquisition module configures a data acquisition source address, a data acquisition type, a storage database type and a database address, connects the data acquisition source with the storage database and starts to acquire data, wherein the data acquisition source type comprises a database type or a protocol pushing type;
step 2: judging the type of the current data acquisition source according to the data acquisition type configured in the step 1; when the type of the current data acquisition source is a database, executing the step 3; when the type of the current data acquisition source is a protocol pushing type, executing the step 7;
and step 3: judging whether an abnormality occurs in the data acquisition process, if no abnormality occurs, continuing acquisition, otherwise, judging the type of the abnormality, wherein the type of the abnormality comprises the abnormality at the data acquisition source side and the abnormality at the storage database side; when the abnormal type data acquisition source side is abnormal, executing the step 4; when the exception type is storage database side exception, executing step 5;
and 4, step 4: the data acquisition module acquires that the current time is the abnormal starting time, sends a connection request to the data acquisition source according to a preset period until the connection with the data acquisition source is recovered to be normal, and executes the step 6;
and 5: the data acquisition module acquires that the current time is the abnormal starting time, sends a connection request to the storage database according to a preset period until the connection with the storage database is recovered to be normal, and executes the step 6;
step 6: the data acquisition module records the abnormal ending time, simultaneously acquires real-time data and historical data and writes the acquired real-time data and the historical data into a storage database together, wherein the historical data refers to data in an abnormal time period;
and 7: judging whether an abnormality occurs in the data acquisition process, if no abnormality occurs, continuing acquisition, otherwise, judging the type of the abnormality, wherein the type of the abnormality comprises the abnormality at the data acquisition source side and the abnormality at the storage database side; when the exception type is that the data acquisition source side is abnormal, the output data is lost; if the storage database side is abnormal, executing the step 8;
and 8: the data acquisition module normally acquires data according to configuration requirements, stores the data into a designated temporary memory of the data acquisition module, and sends a connection request to a storage database according to a preset period until the connection with the storage database is recovered to be normal.
The invention further adopts the following preferred technical scheme:
in the step 3, the data acquisition source side abnormality refers to an abnormality occurring in a network between the data acquisition module and the acquisition source or an abnormality occurring in the data acquisition source;
the storage database side exception refers to the exception of the data acquisition module and the storage database network or the exception of the storage database.
If the data on the data acquisition source side is normal in step 3, the protocol sent by the data acquisition module does not receive the response of the data acquisition source side for n times continuously, and the data acquisition module and the acquisition source network are judged to be abnormal;
if the data on the data acquisition source side is abnormal, or the connection of the server on the data source side is normal but the protocol data acquisition source side sent by the data acquisition module has no response, judging that the data acquisition source side is abnormal;
the data exception of the data acquisition source side means that the data of the data acquisition source side is null or the data timestamp is not updated, otherwise, the data of the data acquisition source side is normal;
if the data on the data acquisition module side is normal, the data is not accessed into the storage database, and the protocol sent by the data acquisition module continuously receives no response of the storage database for n times, the data acquisition module and the storage database are judged to be abnormal; the data on the data acquisition module side is normal, namely the data acquisition module can check the data and the refreshing condition of the timestamp corresponding to the data;
if the storage database can not be operated normally, or the network of the storage server is normal but the protocol storage database side sent by the data acquisition module has no response, judging that the storage database is abnormal; the storage database cannot normally operate, namely the storage database cannot see the data and the update condition of the time stamp corresponding to the data.
The step 4 comprises the following steps:
step 401: the data acquisition module sends a connection request for the mth time, when the connection is successful, the recorded abnormal starting time is cleared, and the data acquisition is restarted, otherwise, the step 402 is executed, wherein m is less than or equal to 3;
step 402: when the connection fails for n times, the data acquisition module records the current time as the abnormal starting time, and continuously sends a connection request to the data acquisition source according to a preset period until the connection with the data acquisition source is recovered to be normal, wherein n is more than or equal to 3.
The step 5 comprises the following steps:
step 501: the data acquisition module sends a connection request for the mth time, when the connection is successful, the recorded abnormal starting time is cleared, and the data transmission is started again, otherwise, the step 502 is executed, wherein m is less than or equal to 3;
step 502: when the connection fails for n times, stopping the acquisition operation, wherein n is more than or equal to 3;
step 503: and the data acquisition module records the first abnormal time as the abnormal starting time, and sends a connection request to the storage database according to a preset period until the connection with the storage database is recovered to be normal.
The step 6 comprises the following steps:
step 601: after reconnection with a data acquisition source or a storage database, recording the moment when reconnection is performed as abnormal ending time by a data acquisition module;
step 602: the data acquisition module simultaneously starts a real-time data acquisition mode and a historical data acquisition mode, the real-time data acquisition mode acquires the latest written real-time data from the data acquisition source, and the historical data acquisition mode acquires the data of the historical time period from the data acquisition source according to the abnormal starting time and the abnormal ending time and writes the data into the storage database.
In the step 7, the process is carried out,
if the data on the data acquisition source side is normal, the protocol sent by the data acquisition module does not receive the response of the data acquisition source side for n times continuously, and the data acquisition module and the acquisition source network are judged to be abnormal;
if the data on the data acquisition source side is abnormal, or the connection of the server on the data source side is normal but the protocol data acquisition source side sent by the data acquisition module has no response, judging that the data acquisition source side is abnormal;
the data exception of the data acquisition source side means that the data of the data acquisition source side is null or the data timestamp is not updated, otherwise, the data of the data acquisition source side is normal;
if the data on the data acquisition module side is normal, the data is not accessed into the storage database, and the protocol sent by the data acquisition module continuously receives no response of the storage database for n times, the data acquisition module and the storage database are judged to be abnormal; the data on the data acquisition module side is normal, namely the data acquisition module can check the data and the refreshing condition of the timestamp corresponding to the data;
if the storage database can not be operated normally, or the network of the storage server is normal but the protocol storage database side sent by the data acquisition module has no response, judging that the storage database is abnormal; the storage database cannot normally operate, namely the storage database cannot see the data and the update condition of the time stamp corresponding to the data.
The step 8 comprises the following steps:
step 801: the data acquisition module normally acquires data according to the configuration requirement in the step 1 and temporarily stores the data in a temporary storage designated by the data acquisition module;
step 802: the data acquisition module sends a connection request to the storage database according to a preset period until the connection with the storage database is recovered to be normal;
step 803: and the data acquisition module simultaneously starts a real-time data acquisition mode and a historical data acquisition mode, the real-time data acquisition mode acquires the latest written real-time data from the data acquisition source, and the historical data acquisition mode reads the temporarily stored historical data from the temporary storage and writes the historical data into a storage database according to the time sequence.
In step 801, the data collection module temporarily stores the data in a file queue, and each file has a predetermined size.
In the step 803, the historical data acquisition mode sequentially reads the data in the designated directory according to the sequence of the temporary storage time and uploads the data to the storage database, and after a complete data file is successfully written into the storage database, the data file is deleted from the designated directory until all cached data files are successfully uploaded.
The beneficial effect that this application reached:
the invention can adopt different line break caching mechanisms according to different configuration information; when data are collected from the time sequence database, the integrity of the data can be automatically ensured in a database caching mode.
The method provides a new idea for the data acquisition broken line cache method, can effectively handle the problem of data acquisition abnormity aiming at different acquisition scenes, and has better effect compared with the traditional broken line cache mechanism.
Drawings
FIG. 1 is a flow chart of a method for caching broken lines based on different acquisition scenarios according to the present invention;
FIG. 2 is a line break cache method of a data acquisition module when the data acquisition source is a system database and the data acquisition module is in an abnormal condition with the system database network or the system database;
FIG. 3 is a line break cache method of a data acquisition module when the data acquisition source is a system database and the data acquisition module and a storage database network are abnormal or the storage database is abnormal;
fig. 4 shows a method for caching a data acquisition module in a broken line when a protocol is pushed by a data acquisition source and the data acquisition module is abnormal in a network of the data acquisition module and the database or the database is abnormal.
Detailed Description
The present application is further described with reference to the following drawings and specific embodiments. The following examples are only for illustrating the technical solutions of the present invention more clearly, and the protection scope of the present application is not limited thereby.
In one embodiment of the present invention, the VeStore database is used as the database in the acquisition source type and the VeStore-TSDB database is used as the storage database.
As shown in fig. 1, the method for caching broken lines based on different acquisition scenarios of the present invention specifically includes the following steps:
step 1: starting a data acquisition module, configuring addresses and data acquisition types of a VeStore database to be acquired, connecting and confirming that the connection is successful; configuring a VeStore-TSDB database type and a VeStore-TSDB database address for storing data, connecting, and confirming that the connection is successful; the data collection module operates normally to begin collecting data from the VeStore database for storage in the VeStore-TSDB database.
Starting a data acquisition module, configuring a data acquisition protocol to be acquired, connecting and confirming that the connection is successful; configuring a VeStore-TSDB database type and a database address for storing data, connecting and confirming that the connection is successful; the data acquisition module operates normally and begins to acquire data from the source acquisition protocol for storage in the VeStore-TSDB database.
Step 2: judging the type of the current data acquisition source according to the data acquisition type configured in the step 1; when the type of the current data acquisition source is the VeStore database, executing the step 3; and 7, when the type of the current data acquisition source is the protocol push type, executing the step 7.
And step 3: and judging whether the data is abnormal or not in the data acquisition process, if not, continuing to acquire the data, and otherwise, judging the type of the abnormality.
The exception types comprise a Vestroe database side exception and a VeStore-TSDB database side exception. Specifically, the Vestroe database refers to the condition that the network between the data acquisition module and the Vestroe database is abnormal or the Vestroe database is abnormal; the VeStore-TSDB database side exception refers to the fact that an exception occurs in a network between a data acquisition module and a VeStore-TSDB database or the VeStore-TSDB database exception occurs.
And when the exception type is a Vestroe database side exception, executing the step 4.
And when the exception type is an eStore-TSDB database side exception, executing the step 5.
Specifically, if the data of the Vestroe database side is normal, the protocol sent by the data acquisition module does not receive the response of the Vestroe database side, and if the response is continued for 3 times, it is determined that the data acquisition module and the acquisition source network are abnormal.
And if the data of the Vestroe database side is abnormal, or the data of the protocol Vestroe database side sent by the data acquisition module is not responded when the data of the Vestroe database side is connected normally by the data source side server, judging that the Vestroe database is abnormal.
The abnormal data of the Vestroe database side means that the data of the Vestroe database side is null or the data timestamp is not updated, and otherwise, the data of the Vestroe database is normal. If the data on the data acquisition module side is normal, the VeStore-TSDB database does not access data, the protocol sent by the data acquisition module does not receive the response of the VeStore-TSDB database, and if the response is not received for 3 times continuously, the data acquisition module and the VeStore-TSDB database network are judged to be abnormal.
The normal data at the data acquisition module side means that the data acquisition module can check the data and the refreshing condition of the timestamp corresponding to the data.
And if the VeStore-TSDB database cannot operate normally, or the storage server network is normal but the protocol VeStore-TSDB database side sent by the data acquisition module has no response, judging that the VeStore-TSDB database is abnormal.
The fact that the VeStore-TSDB database cannot normally operate means that the VeStore-TSDB database cannot check data and the updating condition of the time stamp corresponding to the data.
And 4, step 4: and the data acquisition module acquires that the current time is the abnormal starting time, sends a connection request to the Vestroe database according to a preset period until the connection with the Vestroe database is recovered to be normal, and executes the step 6.
As shown in fig. 2, step 4 specifically includes the following steps: step 401: the data acquisition module sends a connection request for the mth time, when the connection is successful, the recorded abnormal starting time is cleared, and the data acquisition is restarted, otherwise, the step 402 is executed, wherein m is less than or equal to 3;
step 402: when the connection fails for n times, the data acquisition module records the current time as the abnormal starting time, continuously sends a connection request to the Vestroe database according to a preset period until the connection with the Vestroe database is recovered to be normal, wherein n is more than or equal to 3, and executes the step 6.
The period for the data acquisition module to send the reconnection request to the Vestroe database is preferably 1 second.
And 5: and the data acquisition module acquires that the current time is the abnormal starting time, sends a connection request to the VeStore-TSDB database according to a preset period until the connection with the Vestroe database is recovered to be normal, and executes the step 6.
As shown in fig. 3, step 5 specifically includes the following steps:
step 501: and when the data acquisition module is successfully connected, clearing the recorded abnormal starting time and restarting data acquisition, otherwise, executing step 502, wherein m is less than or equal to 3.
Step 502: when the connection fails for n times, stopping data acquisition operation, wherein n is more than 3;
step 503: the data acquisition module records the first abnormal time as an abnormal starting time, sends a connection request to the VeStore-TSDB database according to a preset period until the connection with the VeStore-TSDB database is recovered to be normal, and executes the step 6.
Wherein the period for the data collection module to send the reconnection request to the VeStore-TSDB database is preferably 1 second.
Step 6: the data acquisition module records the abnormal ending time, simultaneously acquires real-time data and historical data and writes the acquired real-time data and the historical data into the VeStore-TSDB database together, wherein the historical data refers to data in the abnormal time period.
Specifically, step 6 includes the steps of:
step 601: and after the data acquisition module is reconnected with the Vestroe database or the VeStore-TSDB database, the moment when the data acquisition module is reconnected is the abnormal ending time.
Step 602: the data acquisition module simultaneously starts a real-time data acquisition mode and a historical data acquisition mode, the real-time data acquisition mode acquires the latest written real-time data from the Vestroe database, and the historical data acquisition mode acquires data of a historical time period from the Vestroe database according to the abnormal starting time and ending time and writes the data into the VeStore-TSDB database together.
And 7: and judging whether the data is abnormal or not in the data acquisition process, if not, continuing to acquire the data, and otherwise, judging the type of the abnormality.
The exception types comprise a Vestroe database side exception and a VeStore-TSDB database side exception. Specifically, the Vestroe database refers to the condition that the network between the data acquisition module and the Vestroe database is abnormal or the Vestroe database is abnormal; the VeStore-TSDB database side exception refers to the fact that an exception occurs in a network between a data acquisition module and a VeStore-TSDB database or the VeStore-TSDB database exception occurs.
And when the exception type is a Vestroe database side exception, outputting a data loss report.
And when the exception type is a VeStore-TSDB database side exception, executing the step 8.
If the data of the Vestroe database side is normal, the protocol sent by the data acquisition module does not receive the response of the Vestroe database side for 3 times continuously, and it is judged that the network between the data acquisition module and the Vestroe database is abnormal.
And if the data of the Vestroe database side is abnormal, or the connection of the Vestroe database side server is normal but the protocol Vestroe database side sent by the data acquisition module has no response, judging that the data acquisition source is abnormal.
The abnormal data of the Vestroe database side means that the data of the Vestroe database side is null or the data timestamp is not updated, and otherwise, the data of the data acquisition source side is normal.
And if the data on the data acquisition module side is normal, the VeStore-TSDB database does not access the data, and the protocol sent by the data acquisition module continuously receives no response of the VeStore-TSDB database for 3 times, judging that the network between the data acquisition module and the VeStore-TSDB database is abnormal.
The normal data at the data acquisition module side means that the data acquisition module can check the data and the refreshing condition of the timestamp corresponding to the data.
And if the VeStore-TSDB database cannot operate normally, or the network of the VeStore-TSDB database server is normal but the protocol VeStore-TSDB database side sent by the data acquisition module has no response, judging that the VeStore-TSDB database is abnormal.
The storage database cannot normally operate, namely the storage database cannot see the data and the update condition of the time stamp corresponding to the data.
And 8: and the data acquisition module acquires data normally according to configuration requirements, stores the data into a temporary memory appointed by the data acquisition module, and sends a connection request to the VeStore-TSDB database according to a preset period until the connection with the data acquisition source is recovered to be normal.
Specifically, as shown in fig. 4, step 8 includes the steps of:
step 801: the data acquisition module normally acquires data according to configuration requirements, stores the data in a temporary storage of an acquisition server in a file queue mode, and sets each file to be of a fixed size, preferably 1 GB.
Step 802: and the data acquisition module performs reconnection operation at 1 second time intervals until the connection between the VeStore-TSDB databases is recovered to be normal.
Step 803: the data acquisition module starts a real-time data acquisition mode and a historical data acquisition mode.
The real-time data acquisition mode acquires the latest written real-time data and uploads the latest written real-time data to the VeStore-TSDB database.
And the historical data acquisition mode sequentially reads the data of the historical cache files according to the sequence of the storage time and uploads the data to the historical database, and after one file is completely and successfully written into the VeStore-TSDB database, the file is deleted until all the cache files are successfully uploaded.
The invention can adopt different line break caching methods according to different configuration information; when data are collected from the time sequence database, the integrity of the data can be automatically ensured in a database caching mode.
The method and the device provide a new idea for a data acquisition broken line cache mechanism, can effectively handle the problem of data acquisition abnormity aiming at different acquisition scenes, and have better effect compared with the traditional broken line cache method.
The present applicant has described and illustrated embodiments of the present invention in detail with reference to the accompanying drawings, but it should be understood by those skilled in the art that the above embodiments are merely preferred embodiments of the present invention, and the detailed description is only for the purpose of helping the reader to better understand the spirit of the present invention, and not for limiting the scope of the present invention, and on the contrary, any improvement or modification made based on the spirit of the present invention should fall within the scope of the present invention.
Claims (9)
1. A line break caching method based on different acquisition scenes is characterized by comprising the following steps:
step 1: the data acquisition module configures a data acquisition source address, a data acquisition type, a storage database type and a database address, connects the data acquisition source with the storage database and starts to acquire data, wherein the data acquisition source type comprises a database type or a protocol pushing type;
step 2: judging the type of the current data acquisition source according to the data acquisition type configured in the step 1; when the type of the current data acquisition source is a database, executing the step 3; when the type of the current data acquisition source is a protocol pushing type, executing the step 7;
and step 3: judging whether an abnormality occurs in the data acquisition process, if no abnormality occurs, continuing acquisition, otherwise, judging the type of the abnormality, wherein the type of the abnormality comprises the abnormality at the data acquisition source side and the abnormality at the storage database side; when the abnormal type data acquisition source side is abnormal, executing the step 4; when the exception type is storage database side exception, executing step 5; the abnormal data acquisition source side refers to the abnormal network between the data acquisition module and the acquisition source or the abnormal data acquisition source; the storage database side abnormity refers to the abnormity of the data acquisition module and the storage database network or the abnormity of the storage database;
and 4, step 4: the data acquisition module acquires that the current time is the abnormal starting time, sends a connection request to the data acquisition source according to a preset period until the connection with the data acquisition source is recovered to be normal, and executes the step 6;
and 5: the data acquisition module acquires that the current time is the abnormal starting time, sends a connection request to the storage database according to a preset period until the connection with the storage database is recovered to be normal, and executes the step 6;
step 6: the data acquisition module records the abnormal ending time, simultaneously acquires real-time data and historical data and writes the acquired real-time data and the historical data into a storage database together, wherein the historical data refers to data in an abnormal time period;
and 7: judging whether an abnormality occurs in the data acquisition process, if no abnormality occurs, continuing acquisition, otherwise, judging the type of the abnormality, wherein the type of the abnormality comprises the abnormality at the data acquisition source side and the abnormality at the storage database side; when the exception type is that the data acquisition source side is abnormal, the output data is lost; if the storage database side is abnormal, executing the step 8;
and 8: the data acquisition module normally acquires data according to configuration requirements, stores the data into a designated temporary memory of the data acquisition module, and sends a connection request to a storage database according to a preset period until the connection with the storage database is recovered to be normal.
2. The line break caching method based on different acquisition scenarios according to claim 1, wherein:
if the data on the data acquisition source side is normal in step 3, the protocol sent by the data acquisition module does not receive the response of the data acquisition source side for n times continuously, and the data acquisition module and the acquisition source network are judged to be abnormal;
if the data on the data acquisition source side is abnormal, or the connection of the server on the data source side is normal but the protocol data acquisition source side sent by the data acquisition module has no response, judging that the data acquisition source side is abnormal;
the data exception of the data acquisition source side means that the data of the data acquisition source side is null or the data timestamp is not updated, otherwise, the data of the data acquisition source side is normal;
if the data on the data acquisition module side is normal, the data is not accessed into the storage database, and the protocol sent by the data acquisition module continuously receives no response of the storage database for n times, the data acquisition module and the storage database are judged to be abnormal; the data on the data acquisition module side is normal, namely the data acquisition module can check the data and the refreshing condition of the timestamp corresponding to the data;
if the storage database can not be operated normally, or the network of the storage server is normal but the protocol storage database side sent by the data acquisition module has no response, judging that the storage database is abnormal; the storage database cannot normally operate, namely the storage database cannot see the data and the update condition of the time stamp corresponding to the data.
3. The line break caching method based on different acquisition scenarios according to claim 1 or 2, wherein:
the step 4 comprises the following steps:
step 401: the data acquisition module sends a connection request for the mth time, when the connection is successful, the recorded abnormal starting time is cleared, and the data acquisition is restarted, otherwise, the step 402 is executed, wherein m is less than or equal to 3;
step 402: when the connection fails for n times, the data acquisition module records the current time as the abnormal starting time, and continuously sends a connection request to the data acquisition source according to a preset period until the connection with the data acquisition source is recovered to be normal, wherein n is more than or equal to 3.
4. The line break caching method based on different acquisition scenarios according to claim 1 or 2, wherein:
the step 5 comprises the following steps:
step 501: the data acquisition module sends a connection request for the mth time, when the connection is successful, the recorded abnormal starting time is cleared, and the data transmission is started again, otherwise, the step 502 is executed, wherein m is less than or equal to 3;
step 502: when the connection fails for n times, stopping the acquisition operation, wherein n is more than or equal to 3;
step 503: and the data acquisition module records the first abnormal time as the abnormal starting time, and sends a connection request to the storage database according to a preset period until the connection with the storage database is recovered to be normal.
5. The line break caching method based on different acquisition scenarios according to claim 1 or 2, wherein:
the step 6 comprises the following steps:
step 601: after reconnection with a data acquisition source or a storage database, recording the moment when reconnection is performed as abnormal ending time by a data acquisition module;
step 602: the data acquisition module simultaneously starts a real-time data acquisition mode and a historical data acquisition mode, the real-time data acquisition mode acquires the latest written real-time data from the data acquisition source, and the historical data acquisition mode acquires the data of the historical time period from the data acquisition source according to the abnormal starting time and the abnormal ending time and writes the data into the storage database.
6. The line break caching method based on different acquisition scenarios according to claim 1 or 2, wherein:
in the step 7, the process is carried out,
if the data on the data acquisition source side is normal, the protocol sent by the data acquisition module does not receive the response of the data acquisition source side for n times continuously, and the data acquisition module and the acquisition source network are judged to be abnormal;
if the data on the data acquisition source side is abnormal, or the connection of the server on the data source side is normal but the protocol data acquisition source side sent by the data acquisition module has no response, judging that the data acquisition source side is abnormal;
the data exception of the data acquisition source side means that the data of the data acquisition source side is null or the data timestamp is not updated, otherwise, the data of the data acquisition source side is normal;
if the data on the data acquisition module side is normal, the data is not accessed into the storage database, and the protocol sent by the data acquisition module continuously receives no response of the storage database for n times, the data acquisition module and the storage database are judged to be abnormal; the data on the data acquisition module side is normal, namely the data acquisition module can check the data and the refreshing condition of the timestamp corresponding to the data;
if the storage database can not be operated normally, or the network of the storage server is normal but the protocol storage database side sent by the data acquisition module has no response, judging that the storage database is abnormal; the storage database cannot normally operate, namely the storage database cannot see the data and the update condition of the time stamp corresponding to the data.
7. The line break caching method based on different acquisition scenarios according to claim 1 or 2, wherein:
the step 8 comprises the following steps:
step 801: the data acquisition module normally acquires data according to the configuration requirement in the step 1 and temporarily stores the data in a temporary storage designated by the data acquisition module;
step 802: the data acquisition module sends a connection request to the storage database according to a preset period until the connection with the storage database is recovered to be normal;
step 803: and the data acquisition module simultaneously starts a real-time data acquisition mode and a historical data acquisition mode, the real-time data acquisition mode acquires the latest written real-time data from the data acquisition source, and the historical data acquisition mode reads the temporarily stored historical data from the temporary storage and writes the historical data into a storage database according to the time sequence.
8. The line break caching method based on different acquisition scenarios according to claim 7, wherein:
in step 801, the data collection module temporarily stores the data in a file queue, and each file has a predetermined size.
9. The line break caching method based on different acquisition scenarios according to claim 7, wherein:
in the step 803, the historical data acquisition mode sequentially reads the data of the temporary storage memory according to the sequence of the temporary storage time and uploads the data to the storage database, and after a complete data file is successfully written into the storage database, the data file is deleted from the temporary storage memory until all the cached data files are successfully uploaded.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110057265.5A CN112380260B (en) | 2021-01-15 | 2021-01-15 | Broken line caching method based on different acquisition scenes |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110057265.5A CN112380260B (en) | 2021-01-15 | 2021-01-15 | Broken line caching method based on different acquisition scenes |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112380260A CN112380260A (en) | 2021-02-19 |
| CN112380260B true CN112380260B (en) | 2021-04-13 |
Family
ID=74581865
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110057265.5A Active CN112380260B (en) | 2021-01-15 | 2021-01-15 | Broken line caching method based on different acquisition scenes |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112380260B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115277723B (en) * | 2022-07-19 | 2024-06-18 | 国能信控互联技术有限公司 | Breakpoint continuous transmission method and system of edge collection history module based on buffer event |
| CN117575482B (en) * | 2023-11-23 | 2024-08-20 | 广州方舟信息科技有限公司 | Drug data warehousing method, device, equipment and storage medium |
| CN117555699A (en) * | 2024-01-11 | 2024-02-13 | 杭州剑齿虎信息技术有限公司 | LCK real-time acquisition system |
Family Cites Families (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103365286B (en) * | 2013-07-30 | 2015-09-16 | 国家电网公司 | A kind of intelligent building integrated communication control system |
| US20170337213A1 (en) * | 2016-05-20 | 2017-11-23 | Hewlett Packard Enterprise Development Lp | Metadata regeneration |
| CN106850292A (en) * | 2017-01-23 | 2017-06-13 | 北京思特奇信息技术股份有限公司 | A kind of method of the client monitors node of distributed type assemblies |
| CN107018363A (en) * | 2017-03-30 | 2017-08-04 | 安徽森度科技有限公司 | A kind of multichannel image acquisition processing system |
| CN109614535B (en) * | 2018-11-29 | 2021-10-08 | 中电万维信息技术有限责任公司 | Method and device for acquiring network data based on Scapy framework |
| CN109800498A (en) * | 2019-01-16 | 2019-05-24 | 国能日新科技股份有限公司 | A kind of photovoltaic plant data diagnosis system |
| CN110191025B (en) * | 2019-06-06 | 2021-06-08 | 国家工业信息安全发展研究中心 | Intelligent gateway data acquisition and test method and device |
| CN111212136B (en) * | 2019-12-31 | 2022-10-18 | 北京力控华康科技有限公司 | Modbus protocol-based data caching method, device and system and storage medium |
-
2021
- 2021-01-15 CN CN202110057265.5A patent/CN112380260B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN112380260A (en) | 2021-02-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112380260B (en) | Broken line caching method based on different acquisition scenes | |
| CN108255641B (en) | CDP disaster recovery method based on cloud platform | |
| CN108933798B (en) | Data storage method, storage server and system | |
| CN106709043A (en) | Data synchronous loading method based on database log | |
| CN110609813B (en) | Data storage system and method | |
| CN101291342A (en) | Data synchronizing method and system in distributed system | |
| CN113542263B (en) | Firewall policy migration method and device | |
| US6243830B1 (en) | State information managing method and communication system | |
| CN113377815B (en) | Method for continuously transmitting write snapshot breakpoint facing real-time database | |
| CN111901566A (en) | Monitoring data storage system and storage method | |
| CN112968815B (en) | Method for realizing continuous transmission in broken network | |
| CN101257661B (en) | Method and system for considerable user service information load | |
| CN112925759A (en) | Data file processing method and device, storage medium and electronic device | |
| CN103019626B (en) | Storage system, method and device for controlling cluster metadata | |
| CN115208841A (en) | Industrial internet identification flow caching processing method based on SDN | |
| CN101692249B (en) | Method for performing data access on power demand side system | |
| CN113381907A (en) | Log collection method and device, electronic equipment and storage medium | |
| US20160037119A1 (en) | Video recording apparatus, and video recording method when temporary network failure occurs | |
| CN116980475B (en) | Data pushing system based on binlog and double annular buffer areas | |
| EP4174675A1 (en) | On-board data storage method and system | |
| CN113076298A (en) | Distributed small file storage system | |
| CN109040286B (en) | Client online state maintenance method based on memory database | |
| CN102780570A (en) | Achieving method and system for management of cloud computing devices | |
| CN104639880A (en) | Method and system for video monitoring and centralized recording in GPRS network environment | |
| CN116204573A (en) | Configurable multi-protocol data acquisition device and method based on real-time database |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |