CN113836120B - Breakpoint resume method and system based on data acquisition engine to data application - Google Patents
Breakpoint resume method and system based on data acquisition engine to data application Download PDFInfo
- Publication number
- CN113836120B CN113836120B CN202111427236.XA CN202111427236A CN113836120B CN 113836120 B CN113836120 B CN 113836120B CN 202111427236 A CN202111427236 A CN 202111427236A CN 113836120 B CN113836120 B CN 113836120B
- Authority
- CN
- China
- Prior art keywords
- data
- link
- database
- application
- data application
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2358—Change logging, detection, and notification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The system is set by a link monitoring module, regularly starts a link monitoring task according to a data application data receiving period, judges whether a data link is abnormal or not, directly writes data into a data application database according to a monitoring KAFKA theme when the data application database can be normally connected, directly loads data when the data application is recovered, saves the time for continuously transmitting the interrupted data from the data acquisition engine into the data application database from the breakpoint starting time, recovers the data more timely, relieves the pressure for intensively writing the data into a network and the data application database, and can automatically position problems, record the breakpoint starting time and adopt different continuous transmission schemes aiming at data interruption caused by different reasons, the data integrity of the data application and the timeliness of data continuous transmission are guaranteed.
Description
Technical Field
The application relates to the technical field of big data transmission and storage, in particular to a breakpoint resume method and a breakpoint resume system based on data acquisition engine to data application.
Background
The technical idea of breakpoint resume is widely implemented and applied in the field of data acquisition and transmission, and the breakpoint resume mainly refers to that when data transmission is performed, a task is divided into a plurality of parts, each part is performed by using one thread, and if a network fault is met, unfinished parts can be resumed from the transmitted parts without uploading or downloading from the beginning.
In an industrial scenario, the data transmission architecture is divided into three layers, including an edge layer, a platform layer, and an application layer. The edge layer is field industrial data, and is collected by edge data collectors dispersed in an industrial plant area in each gateway machine; the platform layer is a data acquisition engine which is connected with each edge data acquisition device and acquires data in the edge data acquisition devices, and the data are formatted uniformly, converged and forwarded to each data application of the application layer; the application layer is each data application system, such as large screen and billboard, and the data engine transfers data to each data application of the application layer through kafka (apache kafka).
The breakpoint resuming technology is common in the edge layer acquisition at present, and can be regarded as an important technical characteristic of an edge data collector, the breakpoint resuming logic of the network shutdown machine of the edge layer is simple, when a link fault occurs, data is stored locally, then the data is uploaded after route recovery, and because the data volume aimed by the network shutdown machine is not very large and the local storage space is limited, a large amount of IO (Input/Output) consumption cannot be caused.
However, with the industrial digital innovation, a large amount of data is required to be on the cloud and is displayed through various application systems, unlike data applications of other industries, data may be refreshed once a day or once a week, data interaction between a data acquisition engine and the data applications in an industrial scene is uninterrupted for 24 hours, the data applications are directly oriented to users, user experience is quite critical, and data transmission links are complex. At present, various problems such as service stop of a data application, failure of a server where the data application is located, or failure of a database used by the data application exist, so that the data acquisition engine cannot continuously and stably push data to the data application. Under certain conditions, when the service of the data application is stopped or the server where the data application is located fails, the data application database can still run normally, if the data application is executed according to the breakpoint resume logic of the internet shutdown machine, when the data application is recovered, a large amount of data is recovered and transmitted to the data application from the data acquisition engine, so that the IO (input/output) is too high, the running of other systems is influenced, and the data experience of enterprises is poor.
Disclosure of Invention
The application provides a breakpoint resume method and a breakpoint resume system based on a data acquisition engine to a data application, so as to improve the stability of a data transmission link between the existing data acquisition engine and the data application.
In a first aspect, the present application provides a breakpoint continuous transmission system based on a data acquisition engine to a data application, including a plurality of edge data collectors, a data acquisition engine, a plurality of data applications, and a plurality of link monitoring modules, where the link monitoring modules are arranged on a data link connecting the data acquisition engine and each data application in an upper layer, and are used to execute a link monitoring task of the data link where the link monitoring module is located;
the link monitoring module comprises scheduling period configuration, database configuration and monitoring KAFKA subject configuration;
the scheduling period configuration is used for configuring the scheduling period of the link monitoring task according to the data receiving period of the data application;
the listening KAFKA theme is configured to replace a listening task of a data application performing data transfer;
the database configuration comprises a database connecting unit and a writing logic unit, and is used for connecting the data application database to check the latest data timestamp through the database connecting unit when a link monitoring task is started so as to judge whether the data link is abnormal, receiving data of the data acquisition engine according to the monitored KAFKA theme when the data link is abnormal but the data application database is normally operated, and directly writing the data into the data application database through the database connecting unit and the writing logic unit so as to directly load the data when the data application is recovered.
Optionally, the data acquisition engine-to-data application-based breakpoint resuming system further includes a redis database;
the redis database is used for storing state numbers, problem numbers and circuit breaking starting time corresponding to each data link;
the state numbers comprise a first state number and a second state number, the first state number represents that the data link is normal, and the second state number represents that the data link is abnormal;
the problem numbers comprise a first problem number, a second problem number, a third problem number and a fourth problem number, the first problem number represents that the data application database normally operates, the second problem number represents that the data application database is in failure, the third problem number represents that a server where the data application is located suspends service, and the fourth problem number represents that the service of the data application is suspended.
Optionally, the initial value of the state number corresponding to each data link is defaulted to be the first state number by the redis database.
In a second aspect, the present application further provides a breakpoint resuming method based on a data acquisition engine to a data application, which specifically includes the following steps:
acquiring a data receiving period of a data application, and setting a scheduling period of a link monitoring task corresponding to the data application according to the data receiving period;
according to the scheduling period, a link monitoring task is started regularly, a latest data timestamp in a data application database is obtained, and if the current timestamp minus the latest data timestamp is greater than a set threshold value, the data link is judged to be abnormal;
if the data link is abnormal, testing the connection of the data application database, and when the connection of the data application database passes, the link monitoring task is configured according to the monitoring KAFKA theme, replaces the data application to execute the monitoring task, receives the data of the data acquisition engine, and writes the data into the data application database.
Optionally, the breakpoint resuming method based on data acquisition engine to data application further includes:
after the data link is judged to be abnormal, updating the state number of the data link to be a second state number;
and if the data link is abnormal, testing the connection of the data application database, updating the problem number to be the first problem number after the connection of the data application database passes, and recording the link monitoring task in a redis database to replace the data application to execute data monitoring and warehousing operation.
Optionally, the breakpoint resuming method based on data acquisition engine to data application further includes:
if the data link is abnormal and the data application database fails to be connected, executing open circuit problem positioning operation, wherein the open circuit problem positioning operation comprises the following steps:
executing the IP of the server where the Ping data application is located, and if the Ping operation fails, updating the problem number to be a third problem number;
if the Ping operation passes, the server IP and the port where the Telnet data are applied are used;
if the Telnet operation fails, updating the question number to be a fourth question number;
and if the Telnet operation passes, updating the question number to be a second question number.
Optionally, the breakpoint resuming method based on data acquisition engine to data application further includes:
if the data link is abnormal and the connection of the data application database fails, inquiring whether the disconnection starting time of the data link is stored in a redis database or not, and if the disconnection starting time of the data link is not stored in the redis database, writing the current timestamp as the disconnection starting time.
Optionally, the breakpoint resuming method based on data acquisition engine to data application further includes:
after the link monitoring task is started regularly according to the scheduling period and the latest data timestamp in the data application database is acquired, if the current timestamp minus the latest data timestamp is less than or equal to a set threshold value, the data link is judged to be normal;
if the data link is normal, inquiring the state number of the data link, and if the state number of the data link is the second state number, inquiring the problem number of the data link;
and if the problem number is the second problem number, the third problem number or the fourth problem number, writing data after the breakpoint start time into the data application from the data acquisition engine according to the breakpoint start time recorded in the redis database, deleting the breakpoint start time, and updating the state number of the data link to be the first state number.
Optionally, the breakpoint resuming method based on data acquisition engine to data application further includes:
and if the problem number is the first problem number, checking whether the server and the data application are recovered, if both the server and the data application are recovered, stopping the data monitoring and warehousing operation of the link monitoring task, updating the state number of the data link to be the first state number, recovering the data application monitoring KAFKA, and finishing the data writing into the data application database.
According to the technical scheme, the breakpoint continuous transmission method and system from the data acquisition engine to the data application provided by the application can regularly start the link monitoring task according to the data receiving period of the data application through the setting of the link monitoring module, monitor the data link condition from the data acquisition engine to the data application in real time, judge whether the data link is abnormal or not through checking the latest data timestamp of the data application database by the link monitoring task, and when the data link is abnormal but the data application database can be normally connected, receive the data of the data acquisition engine according to the monitored KAFKA theme, directly connect the data application database according to the database connection configuration, write the data into the data application database, and when the data application is recovered, directly load the data, so as to avoid the time for continuously transmitting the interrupted data from the data acquisition engine to the data application database from the breakpoint starting time, the data can be recovered more timely, and the pressure of data centralized writing in a network and a data application database can be relieved, so that the stability of the system is maintained.
Drawings
In order to more clearly explain the technical solution of the present application, the drawings needed to be used in the embodiments will be briefly described below, and it is obvious to those skilled in the art that other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a structural diagram of a breakpoint resuming system based on a data acquisition engine to a data application according to an embodiment of the present application.
Detailed Description
In an industrial scene, the concept of breakpoint continuous transmission is well applied to the shutdown of an edge layer network, but a good solution is not provided between a data engine and data application, and generally, a user finds that data time is not correct, so that an operation and maintenance worker can find a problem only by checking a data transmission link, and the problem of data loss is caused. However, if the data transmission between the data engine and the data application is executed according to the breakpoint resuming logic of the internet shutdown, when a link failure occurs, the data is stored locally, and then the data is uploaded after the data transmission link is recovered, so that the IO is too high when a large amount of data is recovered and transmitted, thereby affecting the operation of the system. In the prior art, the data acquisition engine cannot continuously and stably push data to the data application due to various problems such as service stoppage of the data application, failure of a server where the data application is located, failure of a data application database and the like, however, under certain conditions, when the data application or the server where the data application is located suspends the service, the data application database can still normally operate, so that the embodiment of the application provides a breakpoint continuous transmission system based on the data acquisition engine to the data application.
Referring to fig. 1, a breakpoint continuous transmission system based on a data acquisition engine to a data application provided in an embodiment of the present application includes a plurality of edge data collectors, a data acquisition engine, a plurality of data applications, and a plurality of link monitoring modules, where a link monitoring module is disposed on a data link where the data acquisition engine is connected to each data application in an upper layer, and is used to execute a link monitoring task of the data link where the link monitoring module is located, and the link monitoring module includes scheduling cycle configuration, database configuration, and KAFKA theme configuration monitoring, where the link monitoring module may be managed by a distributed task scheduling service, and includes operations such as starting, retrying, and exiting of the link monitoring task.
The scheduling period configuration is used for configuring the scheduling period of the link monitoring task according to the data receiving period of the data application, and the scheduling period configuration of the link monitoring task can be the same as the data receiving period of the data application, so that the real-time monitoring of the data transmission link is achieved, for example, the data application receives one wave of data every 1000ms, and then the scheduling period of the link monitoring task is also 1000ms, that is, the link monitoring task is started at regular intervals of 1000 ms.
The snoop KAFKA theme is configured to perform a snoop task of data transfer, i.e., data snooping, in place of a data application. In the breakpoint resume system of the embodiment of the application, a link monitoring module is configured with a monitoring KAFKA theme, when a link monitoring task is started and executed, if a data link is abnormal, the data application or a server suspends service, but the data application database normally runs, the link monitoring task receives data according to the monitoring KAFKA theme, the data application database is directly connected and configured according to database connection, the data is written into the data application database, when the data application resumes, the data can be directly loaded, the time for resuming the interrupted data from the data acquisition engine to the data application database from the breakpoint time is saved, the data is more timely resumed, and the pressure for writing the data into the network and the data application database in the data set can be relieved, thereby maintaining the stability of the system.
The database configuration comprises a database connection unit and a database writing logic unit, the database connection unit is used for connecting the data application database, and the database writing logic unit is used for writing data into the data application database. When a link monitoring task is started, the database connecting unit is connected with the data application database to check the latest data timestamp so as to judge whether the data link is abnormal, and when the data link is abnormal but the data application database is normally operated, the data of the data acquisition engine is received according to the monitored KAFKA theme, and the database connecting unit and the database writing logic unit are directly connected with the data application database so as to write the data into the data application database.
When a data link fails, in order to recover the link in the fastest time and ensure the stability of data, the problem can be positioned in real time, which block has the problem is automatically detected, the problem is recorded, and the breakpoint of the link is positioned. The breakpoint resume system based on data acquisition engine to data application provided by the embodiment of the application further includes a Remote directory Server (redis) database, where the redis database is used to store state numbers, problem numbers, and disconnection start times corresponding to each data link.
The state number, that is, whether the data link is abnormal or not, includes a first state number and a second state number, the first state number indicates that the data link is normal, and the second state number indicates that the data link is abnormal. The initial value of the state number corresponding to each data link is defaulted to be the first state number by the redis database, namely the data link is normal.
The problem number is the problem location when the data link is abnormal and comprises a first problem number, a second problem number, a third problem number and a fourth problem number, wherein the first problem number represents that the data application database normally operates, the second problem number represents that the data application database fails, the third problem number represents that a server where the data application is located suspends service and the data application database cannot be connected, and the fourth problem number represents that the service of the data application is suspended and the data application database cannot be connected.
In order to more clearly illustrate the automatic execution process and the obtained beneficial effects of the breakpoint resuming system based on the data acquisition engine to the data application provided by the embodiment of the present application, the embodiment of the present application further provides a breakpoint resuming method based on the data acquisition engine to the data application, and the method is executed by the breakpoint resuming system based on the data acquisition engine to the data application provided by the embodiment, and includes the following steps:
and S1, acquiring a data receiving period of the data application, and setting a scheduling period of the link monitoring task corresponding to the data application according to the data receiving period.
According to the data receiving period of the data application, the scheduling period of the link monitoring task corresponding to the data application is set, the link monitoring task is automatically scheduled and started periodically, the scheduling period configuration of the link monitoring task can be the same as the data receiving period of the data application, and therefore real-time monitoring of the data transmission link is achieved, for example, the data application receives wave data every 1000ms, the scheduling period of the link monitoring task is also 1000ms, and the link monitoring task is started regularly every 1000 ms.
And S2, periodically starting a link monitoring task according to the scheduling period, acquiring the latest data time stamp in the data application database, and judging that the data link is abnormal if the current time stamp minus the latest data time stamp is greater than a set threshold value.
Each time the link monitoring task is scheduled, the data application database is firstly connected through the database configuration, the latest data timestamp in the data application database is checked, if the current timestamp minus the latest data timestamp is greater than a set threshold value, the data link is judged to be abnormal, and the current timestamp minus the latest data timestamp can be set to be greater than 5 scheduling periods, the data link is considered to be abnormal, for example, the data application receiving data period is 1000ms, the scheduling period of the link monitoring task is also 1000ms, the current timestamp is 1632219569400, the database latest data timestamp is 1632219563400, 1632219569400-16322163400 =6000,
if the data link is abnormal, the next link monitoring task is performed.
And S3, if the data link is abnormal, testing the connection of the data application database, and when the connection of the data application database passes, the link monitoring task is configured according to the monitoring KAFKA theme, replaces the data application to execute the monitoring task, receives the data of the data acquisition engine, and writes the data into the data application database.
If the data link is judged to be abnormal, the link is continuously checked, and the connection of the data application database is tested, because the data application monitors the KAFKA theme under normal conditions, when the data application suspends the service, the data link is interrupted, but the data application suspends, and the data application database does not necessarily suspend the service, the data can be directly written into the data application database, and when the data application is recovered, the data can be directly loaded, so that the time for uploading and recovering the data from the breakpoint time is saved. Therefore, when the data link is abnormal, the data application database connection is firstly tested, when the data application database connection passes, the link monitoring task replaces the data application to execute the monitoring task according to the configured KAFKA theme, the data of the data acquisition engine is received, the data is written into the data application database, the operation that the interrupted data is continuously transmitted into the data application database when the data application is recovered is avoided, the data is recovered more timely, the pressure of data concentrated writing into a network and the data application database can be relieved, and the stability of the system is maintained.
When a data link fails, in order to recover the link in the fastest time and ensure the stability of data, the problem can be positioned in real time, the problem of which block is detected is automatically detected, and the data link breakpoint is positioned. In another embodiment, after determining that the data link is abnormal, updating the data link state number to be the second state number into the redis database, testing the data application database connection if the data link is abnormal, updating the problem number to be the first problem number in the redis database after the data application database connection passes, and simultaneously, the link monitoring task replaces the data application to perform data monitoring and warehousing operations.
If the data link is abnormal and the connection of the data application database fails, the open circuit problem positioning operation is continuously executed, firstly, the server IP where the Ping (Packet Internet Groper) data application is located is executed, if the Ping operation fails, the problem number is updated to be the third problem number, the server where the data application is located suspends service, if the Ping operation passes, the server IP and the port where the Telnet (Telecommunications Network, remote login) data application is located, if the Telnet operation fails, the problem number is updated to be the fourth problem number, the service of the data application is suspended, if the Telnet operation passes, the problem number is updated to be the second problem number, and the data link abnormality is caused by the data application database fault.
When the data link is abnormal, the data application database cannot be connected, the data application database cannot be directly connected to execute the substitute writing warehousing operation, and the data after the self-disconnection starting time can be rewritten into the data application database only after the data link is recovered. And inquiring whether the disconnection starting time of the data link is stored in the redis database, and if the disconnection starting time of the data link is not stored in the redis database, writing the current timestamp as the disconnection starting time. If the disconnection starting time of the data link is stored in the redis database, the data link is not recovered all the time, and no other operation is performed. When the data link is recovered, the data after the starting time of the disconnection is inquired from the data acquisition engine and written into the database.
In another embodiment, the link monitoring task is started, and after the timestamp of the latest data in the data application database is obtained, if the current timestamp minus the timestamp of the latest data is less than or equal to a set threshold, the data link is determined to be normal.
If the data link is normal, inquiring the state number of the data link, and if the state number of the data link is the first state number, judging that the data link is abnormal last time and is judged to be a normal data link, and judging to be finished; and if the state number of the data link is the second state number, which indicates that the data link is judged to be an abnormal data link last time, inquiring the problem number of the data link.
And if the problem number is a second problem number, a third problem number or a fourth problem number, which indicates that the last link check is a data application database fault, a server suspends service or a data application suspends service, and the data application database is not connected, but data is normally written, which indicates that the current data link is recovered to be normal, writing data after the breakpoint starting time into the data application from the data acquisition engine according to the breakpoint starting time recorded in the redis database, deleting the breakpoint starting time, and updating the state number of the data link to be the first state number.
If the problem number is the first problem number, the data application or the server is indicated to be in fault but the data application database is normally connected, at the moment, data is directly written into the database, whether the server and the data application are recovered or not is checked, if the server and the data application are both recovered, the data monitoring and warehousing operation of the link monitoring task is stopped, the state number of the data link is updated to be the first state number, the data application monitoring KAFKA is recovered, and the next data writing into the data application database is completed. And if the server and the data application are not recovered, continuing to execute the operation of the substitute library of the link monitoring task.
The embodiment of the application provides a breakpoint continuous transmission method and system from a data acquisition engine to a data application, through the setting of a link monitoring module, a link monitoring task is started regularly according to a data receiving period of the data application, the data link condition from the data acquisition engine to the data application is monitored in real time, the link monitoring task can judge whether a data link is abnormal or not by checking a latest data timestamp of a data application database, and when the data link is abnormal but the data application database can be normally connected, the data of the data acquisition engine is received according to monitoring a KAFKA theme, the data application database is directly connected according to database connection configuration, the data is written into the data application database, when the data application is recovered, the data can be directly loaded, the time for continuously transmitting the interrupted data from the data acquisition engine to the data application database from the breakpoint starting time is saved, the data can be recovered more timely, and the pressure of data centralized writing in a network and a data application database can be relieved, so that the stability of the system is maintained.
Furthermore, the breakpoint resuming system and method based on data acquisition engine to data application provided in the embodiment of the present application solve the problem that data link positioning is started after data interruption in data application is manually found at present, so that pain points of data loss after data interruption are caused, and realize that data link is scheduled and monitored at regular time according to a data receiving period of the data application, and different resuming schemes are adopted for automatically finding problems, recording problems, positioning link breakpoint starting time and data interruption caused by different reasons, so as to achieve transmission monitoring, problem positioning, matching schemes and automatic resuming of the data link, achieve stable operation of the data transmission link, and ensure data integrity of the data application and timeliness of data resuming.
The above-described embodiments of the present application do not limit the scope of the present application.
Claims (9)
1. A breakpoint continuous transmission system based on a data acquisition engine to a data application comprises a plurality of edge data acquisition devices, the data acquisition engine and a plurality of data applications, and is characterized by further comprising a plurality of link monitoring modules, wherein the link monitoring modules are arranged on data links connected between the data acquisition engine and the data applications on the upper layer and used for executing link monitoring tasks of the data links;
the link monitoring module comprises scheduling period configuration, database configuration and monitoring KAFKA subject configuration;
the scheduling period configuration is used for configuring the scheduling period of the link monitoring task according to the data receiving period of the data application;
the monitoring KAFKA theme is configured to perform a data transmission monitoring task when the data application suspends service but the data application database is operating normally, the data transmission monitoring task being performed by the data application when the data application is operating normally;
the database configuration comprises a database connecting unit and a writing logic unit, and is used for connecting the data application database to check the latest data timestamp through the database connecting unit when a link monitoring task is started so as to judge whether the data link is abnormal, receiving data of the data acquisition engine according to the monitored KAFKA theme when the data link is abnormal but the data application database is normally operated, and directly writing the data into the data application database through the database connecting unit and the writing logic unit so as to directly load the data when the data application is recovered.
2. The breakpoint resume system based on data mining engine to data application according to claim 1, further comprising a redis database;
the redis database is used for storing state numbers, problem numbers and circuit breaking starting time corresponding to each data link;
the state numbers comprise a first state number and a second state number, the first state number represents that the data link is normal, and the second state number represents that the data link is abnormal;
the problem numbers comprise a first problem number, a second problem number, a third problem number and a fourth problem number, the first problem number represents that the data application database normally operates, the second problem number represents that the data application database is in failure, the third problem number represents that a server where the data application is located suspends service, and the fourth problem number represents that the service of the data application is suspended.
3. The breakpoint resume system from data mining engine to data application according to claim 2, wherein the redis database defaults that the initial value of the state number corresponding to each data link is the first state number.
4. A breakpoint resume method from a data acquisition engine to a data application is characterized by comprising the following steps:
acquiring a data receiving period of a data application, and setting a scheduling period of a link monitoring task corresponding to the data application according to the data receiving period;
according to the scheduling period, a link monitoring task is started regularly, a latest data timestamp in a data application database is obtained, and if the current timestamp minus the latest data timestamp is greater than a set threshold value, the data link is judged to be abnormal;
if the data link is abnormal, the data application database connection is tested, when the data application database connection passes, the link monitoring task is configured according to a monitoring KAFKA theme, the monitoring task is executed instead of the data application, the data of the data acquisition engine is received, the data is written into the data application database, and the monitoring task of data transmission is executed by the data application when the data application is normally served.
5. The breakpoint resuming method based on data acquisition engine to data application according to claim 4, further comprising:
after the data link is judged to be abnormal, updating the state number of the data link to be a second state number;
and if the data link is abnormal, testing the connection of the data application database, updating the problem number to be the first problem number after the connection of the data application database passes, and recording the link monitoring task in a redis database to replace the data application to execute data monitoring and warehousing operation.
6. The breakpoint resuming method based on data acquisition engine to data application according to claim 5, further comprising:
if the data link is abnormal and the data application database fails to be connected, executing open circuit problem positioning operation, wherein the open circuit problem positioning operation comprises the following steps:
executing the IP of the server where the Ping data application is located, and if the Ping operation fails, updating the problem number to be a third problem number;
if the Ping operation passes, the server IP and the port where the Telnet data are applied are used;
if the Telnet operation fails, updating the question number to be a fourth question number;
and if the Telnet operation passes, updating the question number to be a second question number.
7. The breakpoint resume method for data application based on data acquisition engine as claimed in claim 6, further comprising:
if the data link is abnormal and the connection of the data application database fails, inquiring whether the disconnection starting time of the data link is stored in a redis database or not, and if the disconnection starting time of the data link is not stored in the redis database, writing the current timestamp as the disconnection starting time.
8. The breakpoint resuming method based on data acquisition engine to data application according to claim 7, further comprising:
after the link monitoring task is started regularly according to the scheduling period and the latest data timestamp in the data application database is acquired, if the current timestamp minus the latest data timestamp is less than or equal to a set threshold value, the data link is judged to be normal;
if the data link is normal, inquiring the state number of the data link, wherein the inquired state number represents the state of the data link in the last scheduling period;
if the state number of the data link is a second state number, which indicates that the data link is judged to be an abnormal data link in the last scheduling period, continuing to inquire the problem number of the data link;
and if the problem number is the second problem number, the third problem number or the fourth problem number, writing data after the breakpoint start time into the data application from the data acquisition engine according to the breakpoint start time recorded in the redis database, deleting the breakpoint start time, and updating the state number of the data link to be the first state number.
9. The breakpoint resuming method based on data acquisition engine to data application according to claim 8, further comprising:
and if the problem number is the first problem number, checking whether the server and the data application are recovered, if both the server and the data application are recovered, stopping the data monitoring and warehousing operation of the link monitoring task, updating the state number of the data link to be the first state number, recovering the data application monitoring KAFKA, and finishing the data writing into the data application database.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111427236.XA CN113836120B (en) | 2021-11-29 | 2021-11-29 | Breakpoint resume method and system based on data acquisition engine to data application |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111427236.XA CN113836120B (en) | 2021-11-29 | 2021-11-29 | Breakpoint resume method and system based on data acquisition engine to data application |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113836120A CN113836120A (en) | 2021-12-24 |
| CN113836120B true CN113836120B (en) | 2022-03-11 |
Family
ID=78971672
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111427236.XA Active CN113836120B (en) | 2021-11-29 | 2021-11-29 | Breakpoint resume method and system based on data acquisition engine to data application |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113836120B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115842860B (en) * | 2023-02-28 | 2023-06-13 | 江苏金恒信息科技股份有限公司 | Monitoring method, device and system for data link |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110413701A (en) * | 2019-08-08 | 2019-11-05 | 江苏满运软件科技有限公司 | Distributed data base storage method, system, equipment and storage medium |
| CN111143158A (en) * | 2019-12-04 | 2020-05-12 | 武汉光谷信息技术股份有限公司 | Monitoring data real-time storage method and system, electronic equipment and storage medium |
-
2021
- 2021-11-29 CN CN202111427236.XA patent/CN113836120B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110413701A (en) * | 2019-08-08 | 2019-11-05 | 江苏满运软件科技有限公司 | Distributed data base storage method, system, equipment and storage medium |
| CN111143158A (en) * | 2019-12-04 | 2020-05-12 | 武汉光谷信息技术股份有限公司 | Monitoring data real-time storage method and system, electronic equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113836120A (en) | 2021-12-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN101197621B (en) | Method and system for remote diagnosing and locating failure of network management system | |
| CN105430327A (en) | NVR cluster backup method and device | |
| CN109286529B (en) | Method and system for recovering RabbitMQ network partition | |
| CN101753357A (en) | Network server centralized monitoring system and method | |
| US20130205017A1 (en) | Computer failure monitoring method and device | |
| CN113836120B (en) | Breakpoint resume method and system based on data acquisition engine to data application | |
| CN102394791A (en) | Downtime recovery method and system | |
| CN111464612B (en) | Method for providing stable computing service in severe environment | |
| CN103532753A (en) | Double-computer hot standby method based on memory page replacement synchronization | |
| CN110895488B (en) | Task scheduling method and device | |
| CN109144789A (en) | A kind of method, apparatus and system for restarting OSD | |
| CN102404386A (en) | Method for ensuring high-reliability service of metadata server group of distributed file system | |
| CN104601668A (en) | State management based data push method, device and system | |
| CN107395771A (en) | Full redundancy balancing the load production process data acquisition system | |
| CN112968815A (en) | Method for realizing continuous transmission in broken network | |
| JP6418377B2 (en) | Management target device, management device, and network management system | |
| CN104679710A (en) | Software fault quick recovery method for semiconductor production line transportation system | |
| CN115017235B (en) | Data synchronization method, electronic device and storage medium | |
| JP2001034304A (en) | Monitor and control supporting device | |
| CN113127435A (en) | Intelligent synchronization method and system for files of main and standby systems | |
| CN103124367A (en) | Remote online debugging method of video distribution network node servers | |
| CN103825752B (en) | Device and method for supervisory control system running state | |
| CN207251674U (en) | Full redundancy balancing the load production process data acquisition system | |
| CN113504740A (en) | Data acquisition, remote monitoring and remote control method based on web technology | |
| CN105550094A (en) | Automatic state monitoring method of high-availability system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |