CN110442770B - Data grabbing and storing method and device, computer equipment and storage medium - Google Patents

Data grabbing and storing method and device, computer equipment and storage medium Download PDF

Info

Publication number
CN110442770B
CN110442770B CN201910730486.7A CN201910730486A CN110442770B CN 110442770 B CN110442770 B CN 110442770B CN 201910730486 A CN201910730486 A CN 201910730486A CN 110442770 B CN110442770 B CN 110442770B
Authority
CN
China
Prior art keywords
data
time
grabbing
capturing
difference
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201910730486.7A
Other languages
Chinese (zh)
Other versions
CN110442770A (en
Inventor
邵健锋
崔巍
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
New Trend International Logis Tech Co ltd
Original Assignee
New Trend International Logis Tech Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by New Trend International Logis Tech Co ltd filed Critical New Trend International Logis Tech Co ltd
Priority to CN201910730486.7A priority Critical patent/CN110442770B/en
Publication of CN110442770A publication Critical patent/CN110442770A/en
Application granted granted Critical
Publication of CN110442770B publication Critical patent/CN110442770B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types
    • G06F16/95—Retrieval from the web
    • G06F16/951—Indexing; Web crawling techniques
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
    • Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
    • Y02P90/02—Total factory control, e.g. smart factories, flexible manufacturing systems [FMS] or integrated manufacturing systems [IMS]

Landscapes

  • Engineering & Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The invention discloses a data grabbing and storing method, a device, computer equipment and a storage medium, wherein the data grabbing and storing method comprises the steps of grabbing data in a data source when a plurality of data grabbing devices trigger a data grabbing condition; wherein, at least one of the data sources of the plurality of data grabbing devices grabbing data is the same data source; synchronizing the grabbing time of grabbing data by the plurality of data grabbing devices; storing the data captured by the plurality of data capturing devices and the corresponding capturing time into a database in a record form; and responding to the received data request instruction, returning the record corresponding to the appointed time, or acquiring the record corresponding to the grabbing time closest to the appointed time and returning the record, and/or acquiring the estimated data of the appointed time and returning the record. The method can avoid the situation of data loss, and realize continuous and stable data grabbing and storing and data alignment.

Description

Data grabbing and storing method and device, computer equipment and storage medium
Technical Field
The present invention relates to the field of data capturing and saving technologies, and in particular, to a data capturing and saving method and apparatus, a computer readable storage medium, and a computer device.
Background
Currently, most data capture software captures data for display, i.e. the captured data is displayed on a user interface for a user to view, in which case short-term data loss has little effect and is affordable. However, when the purpose of capturing data is mathematical modeling and mathematical analysis, the data to be captured is used for mathematical modeling and mathematical analysis, in which case the continuity of the data will have an important effect on the construction of the model, as the loss of data may seriously affect the stability and accuracy of the modeling and analysis results.
In the prior art, the data grabbing end works on a computer and is limited by software and hardware of a computer system, so that long-term continuous and stable operation of the data grabbing end cannot be ensured. Therefore, how to avoid data loss, and optimizing the data capturing and saving process to achieve continuous and stable capturing and saving of data are technical problems that a person skilled in the art needs to solve at present.
Disclosure of Invention
The embodiment of the invention provides a data grabbing and storing method, a data grabbing and storing device, a computer readable storage medium and computer equipment, which can avoid data loss and optimize the data grabbing and storing process to realize continuous and stable grabbing and storing of data.
In a first aspect, an embodiment of the present invention provides a data capturing and saving method, including:
capturing data in a data source when a plurality of data capturing devices trigger data capturing conditions; wherein, at least one of the data sources of the plurality of data grabbing devices grabbing data is the same data source;
synchronizing the grabbing time of grabbing data by the plurality of data grabbing devices;
storing the data captured by the plurality of data capturing devices and the corresponding capturing time into a database in a record form;
responding to the received data request instruction, and acquiring information corresponding to the appointed time;
judging whether corresponding records can be searched from the database according to the appointed time; if yes, directly returning the record corresponding to the appointed time, otherwise, acquiring the record corresponding to the grabbing time closest to the appointed time and returning the record, and/or acquiring the estimated data of the appointed time and returning the estimated data.
In a second aspect, an embodiment of the present invention provides a data capturing and saving device, including:
the data grabbing module is used for grabbing data in the data source when the plurality of data grabbing devices trigger the data grabbing conditions; wherein, at least one of the data sources of the plurality of data grabbing devices grabbing data is the same data source;
The time synchronization module is used for synchronizing the grabbing time of the plurality of data grabbing devices for grabbing data;
the storage module is used for storing the data captured by the plurality of data capturing devices and the corresponding capturing time into a database in a recorded form;
the request response module is used for responding to the received data request instruction and acquiring information corresponding to the appointed time;
the data alignment module is used for judging whether corresponding records can be searched from the database according to the appointed time; if yes, directly returning the record corresponding to the appointed time, otherwise, acquiring the record corresponding to the grabbing time closest to the appointed time and returning the record, and/or acquiring the estimated data of the appointed time and returning the estimated data.
In a third aspect, an embodiment of the present invention further provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and capable of running on the processor, where the processor implements the data grabbing and saving method described in the first aspect when executing the computer program.
In a fourth aspect, an embodiment of the present invention further provides a computer readable storage medium, where the computer readable storage medium stores a computer program, where the computer program when executed by a processor implements the data grabbing and saving method described in the first aspect.
The embodiment of the invention provides a data grabbing and storing method, which comprises the steps that a plurality of data grabbing devices grab data in a data source, at least one of the data sources grabbed by the plurality of data grabbing devices is the same data source, then the grabbing time is synchronized, the data are stored and aligned, and the method can allow one data grabbing device to maintain or allow other data grabbing devices to continue grabbing the data under the condition that a plurality of data grabbing devices maintain at the same time; while also allowing maintenance operations on the database. The method optimizes the data grabbing and storing process, can avoid the condition of data loss in the data grabbing and storing process, realizes continuous and stable grabbing and storing of data, simultaneously synchronizes the grabbing time, avoids the problem of data jitter, aligns the data and ensures that the data with accurate time can be provided. The embodiment of the invention also provides a data grabbing and storing device, a computer readable storage medium and a computer device, which have the beneficial effects and are not repeated herein.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flowchart of a data grabbing and saving method according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a data capturing and saving method according to an embodiment of the present invention;
fig. 3 is a schematic structural diagram of a data capturing and saving device according to an embodiment of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are some, but not all embodiments of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
It should be understood that the terms "comprises" and "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
It is also to be understood that the terminology used in the description of the invention herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in this specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.
It should be further understood that the term "and/or" as used in the present specification and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes such combinations.
Data crawling depends on the hardware and software environment of the data crawling device and the availability and stability of the written database, and the data crawling process is subject to data loss, which can lead to data crawling window periods, which are catastrophic for mathematical modeling and mathematical analysis, as such conditions can have serious impact on the results of the mathematical modeling and mathematical analysis. The inventor discovers that the data grabbing and storing methods in the prior art have defects through creative labor. As in the prior art, there is a data grabbing mode: for example, local data service software conforming to SCADA (Supervisory ControlAnd DataAcquisition, data acquisition and monitoring control) standard, such as OPC Server (Open Platform Communications Server, open platform communication Server), is used as a data source Server conforming to OPC standard for mirroring data from software and hardware conforming to OPC protocol into computer memory), as a data source, the service software is installed on a computer using data and acts to pull the data, although such software maps actual data into the local memory, when a link pulling the data fails, the use of upper software is not affected, but at this time the data is only old data before the failure, the upper software may not detect the failure in time, and due to the limitation of the working principle, the data is delayed by 1-2 seconds even in normal use. These limitations result in such schemes being essentially unusable for grabbing in the mathematical modeling sense. Another data grabbing mode exists in the prior art: OPC Server is still used, but this software is deployed separately from the upper layer presentation software. However, when the data link fails in the prior art, the upper layer software can detect the existence of the fault in most cases, but still lacks the recovery capability. Additionally, data delays of 1-2 seconds still exist, resulting in such schemes being unsuitable for grabbing in the mathematical modeling sense. Another data grabbing mode exists in the prior art: the single machine is directly connected with the equipment to be grabbed, finished software such as OPC Server is not used, and the single machine is directly connected with the equipment to be grabbed by using an industry standard bottom network protocol. However, in the prior art, when the computer is not available due to software and hardware faults or maintenance, the data still has a blank window period, so that the computer can only be applied to a mathematical model with low requirements on data continuity. Another data grabbing mode exists in the prior art: the problem in the prior art is that the failover exists in detection time (time required for detecting a computer failure, typically tens of seconds to 1 minute) and switching time (operation time required for transferring a load to other computers working normally after confirming the computer failure, typically several seconds), and during this time, data still has a window period, so that the system can only be applied to mathematical models with low requirements on data continuity. Another data grabbing mode exists in the prior art: software is deployed using fault tolerant servers (e.g., stratus ftServer), in which case system hardware failures do not affect system execution and detection and switchover times are negligible. However, the prior art still cannot cope with the empty window caused by software errors and system maintenance, and the fault tolerant server is expensive and has high cost.
Aiming at the defects in the prior art, the invention provides a novel data grabbing and storing method through the following embodiments, which can solve the problems in the prior art, achieve the effects of avoiding the data loss, optimizing the data grabbing and storing process to realize continuous and stable grabbing of data, synchronizing the grabbing time and aligning the grabbed data, and solve a series of novel problems in the multi-point acquisition process.
Referring to fig. 1, fig. 1 is a flowchart of a data capturing and saving method according to an embodiment of the present invention.
The specific steps may include:
s101: capturing data in a data source when a plurality of data capturing devices trigger data capturing conditions;
wherein, at least one of the data sources of the plurality of data grabbing devices grabbing data is the same data source.
The method can be applied to the data grabbing and storing process of the data grabbing device, wherein the data grabbing device is used for grabbing various types of data. The data grabbing device can refer to software devices, hardware devices and soft and hard combination devices. For example, the data capture device is data capture software (a software device, which can be run on a computer or other devices), or the data capture device is a sensing device (a hardware device), or the data capture device is a computer (a hard-soft combination device) in which the data capture software is installed. In a specific application scenario, the data capturing device is a capturing machine, and the capturing machine can be any one of the three devices according to different types and requirements of captured data.
In this step, the data grasped by the data grasping apparatus may be of various types, and for example, may be specifically various data of temperature, pressure, voltage, current, sound, and the like. The data source refers to a captured data source, in a specific scene, the data source depends on specific equipment, for example, the data source of temperature depends on temperature equipment, the data source of pressure depends on pressure equipment, and the specific equipment can be the same equipment or different equipment. In a specific scenario, the data source refers to a sensing device capable of collecting data on a specific device, such as a temperature sensor, a pressure sensor, etc., i.e. the temperature sensor may collect temperature data of the temperature device, and the pressure sensor may collect pressure data of the pressure device. The sensing devices can continuously collect data and provide the collected data as a data source to the data grabbing device for grabbing. In addition, the sensing devices can be hardware devices, software devices or software and hardware devices according to different data types and requirements.
It should be noted that, at least one of the data sources of the plurality of data grabbing devices grabs data is the same data source, and the purpose of the embodiment of the invention can be achieved only in this application scenario, that is, the data of the same data source is grabbed by the plurality of data grabbing devices, so that even if one or a plurality of data grabbing devices fail or need maintenance, the rest of data grabbing devices can continue to grab the data of the data source, and the situation that the empty window period and the data are lost in the data grabbing process is ensured. In other words, the embodiment of the invention can not only cope with software and hardware faults and offline time of maintenance, but also cope with the problem of grabbing interruption caused by accidental faults of a software body in a multi-machine grabbing (also called multi-point total activity) mode. The embodiment adopts a mode of multi-point full activity and no backup machine, that is, all data grabbing devices normally perform grabbing work, and all data grabbing devices normally store the grabbed data. The data grabbing equipment does not need to be monitored and switched in real time, so that the following problems are solved: 1. instability by monitoring: monitoring itself cannot guarantee reliability from time to time, for example, the monitoring host has a possibility of failure, the monitoring dependent lines and logic have a possibility of instability, and these may lead to the possibility of temporary failure of the monitoring, or a false alarm may be generated. 2. Monitoring the delay brought by: monitoring does not immediately ascertain at the moment of failure, but necessarily there is some monitoring delay. 3. Delay from reassigning tasks: and detecting that tasks need to be redistributed, wherein a time blank period exists between the stop work of the old data grabbing device and the start work of the new data grabbing device.
If a data grabbing device grabs only data of one data source, if a plurality of data sources exist, in order to ensure that the situation of data loss does not occur in the process of grabbing the data of the data sources, the data of each data source needs to be grabbed by the plurality of data grabbing devices, which can lead to a large increase in the number of the data grabbing devices and increase the grabbing cost. In the actual deployment process, each data grabbing device can grab the data of a plurality of data sources at the same time, but the grabbing of the data of each data source in the system is not limited, and only the grabbing of a target data source is ensured to be carried out by simultaneously providing a plurality of data grabbing devices, wherein the target data source is the data source which needs to continuously and stably provide the data, and the continuity and the stability of the data sources are very important. The embodiment of the invention aims at the data sources which need to continuously and stably provide data, namely, at least one of the data sources which are used for capturing the data by the plurality of data capturing devices and are referred to as the same data source in the embodiment of the invention. In addition, the same data source in the embodiment of the invention can be one or a plurality of data sources.
For example, as shown in fig. 2, a total of A, B data grabbing devices are deployed in a specific application scenario, where a data source 1 and a data source 2 are to be grabbed, and both the data source 1 and the data source 2 are target data sources, i.e. data sources that need to continuously and stably provide data. The actual grasping pattern may be: the data grabbing device a grabs data of the data source 1 and the data source 2, and the data grabbing device B grabs data of the data source 1 and the data source 2. In the above capturing mode, the same data sources from which the data capturing device a and the data capturing device B capture data are the data source 1 and the data source 2, so that the data of the data source 1 and the data source 2 are captured by 2 data capturing devices, and continuous and stable capturing of the data source 1 and the data source 2 can be ensured.
For example, in another specific application scenario, a total of A, B data grabbing devices are deployed, and 1, 2, and 3 data sources are simultaneously available for grabbing, where the data source 1, the data source 2, and the data source 3 are all target data sources, i.e. data sources that need to continuously and stably provide data. The actual grasping pattern may be: the data grabbing device a grabs data of the data source 1, the data source 2 and the data source 3, and the data grabbing device B grabs data of the data source 1, the data source 2 and the data source 3. In the above capturing mode, the same data sources from which the data capturing device a and the data capturing device B capture data are the data source 1, the data source 2 and the data source 3, so that the data of each data source is captured by 2 data capturing devices, and continuous and stable capturing of the data can be ensured. In the above application scenario, if only the data source 1 and the data source 2 are target data sources, the data sources need to continuously and stably provide data. The actual grasping pattern may be: the data grabbing device A grabs data of the data sources 1 and 2, and the data grabbing device B grabs data of the data sources 1 and 2 and 3. In the above capturing mode, the same data sources from which the data capturing device a and the data capturing device B capture data are the data source 1 and the data source 2, so that only the data of the data source 1 and the data source 2 are captured by 2 data capturing devices, and continuous and stable capturing of the data source 1 and the data source 2 can be ensured.
For example, in another specific application scenario, a total of A, B, C data grabbing devices are deployed, and 1, 2, 3, and 4 data sources are simultaneously available to be grabbed, where all of the data source 1, the data source 2, the data source 3, and the data source 4 are target data sources, i.e. data sources that need to continuously and stably provide data. The data capturing device a captures data of the data source 1, the data source 2 and the data source 4, the data capturing device B captures data of the data source 1, the data source 3 and the data source 4, the data capturing device C captures data of the data source 2 and the data source 3, in the specific application scene, the data of the data source 1 is captured by the data capturing device a and the data capturing device B at the same time, the data of the data source 2 is captured by the data capturing device a and the data capturing device C at the same time, the data of the data source 3 is captured by the data capturing device B and the data capturing device C at the same time, so that the data of the data source 1, the data source 2, the data source 3 and the data source 4 are captured by the data capturing device 2, and continuous and stable capturing of the data source 1, the data source 2, the data source 3 and the data source 4 can be ensured. It should be noted that, in the embodiment of the present invention, although the data capturing device A, B, C does not have the same data source, it does not affect the continuous and stable capturing of the data of the target data source in the embodiment of the present invention. This is because in the embodiment of the present invention, the plurality of data capturing devices are determined based on whether the same data source exists, that is, the plurality of data capturing devices in the embodiment of the present invention do not refer to all deployed data capturing devices, but are determined by the data capturing device having the same data source, that is, for the data source 1, the plurality of data capturing devices refer to the data capturing device a and the data capturing device B, the two data capturing devices have the same data source 1, for the data source 2, the plurality of data capturing devices refer to the data capturing device a and the data capturing device C, the two data capturing devices have the same data source 2, and for the data source 3, the plurality of data capturing devices refer to the data capturing device B and the data capturing device C, and the two data capturing devices have the same data source 3, which is not limited to only one of course; for example, for the data source 1 and the data source 4, the plurality of data capture devices refer to a data capture device a and a data capture device B, and the two data capture devices have the same data source 1 and data source 4. That is, in all deployed data capturing devices, as long as at least one of the data sources in which a plurality of data capturing devices capture data is the same data source, it is within the scope of the embodiments of the present invention. In other words, from the perspective of data sources, it is within the scope of the embodiments of the present invention if there is one data source with multiple data grabbing devices grabbing data at the same time.
The triggering of the data grabbing condition may be performed in various manners, for example, the data grabbing device itself is triggered according to a preset data grabbing condition, such as a timing trigger. As a preferred way of triggering the data grabbing condition, the master control machine may be used to issue a data grabbing instruction to the data grabbing device, so as to trigger the data grabbing condition. Specifically, when the data grabbing condition is triggered, grabbing the data in the data source includes: step one, triggering the data grabbing condition when responding to a data grabbing instruction issued by a master control machine; and step two, capturing data in a data source according to the data capturing instruction. The master control machine is in communication connection with the data grabbing equipment, can send data grabbing instructions to the data grabbing equipment, and can be used for coordinating the work of all components, and the master control machine does not directly participate in data grabbing, so that the offline maintenance of the master control machine in a short time does not influence the normal operation of the system. The master control may also issue instructions to the databases to confirm whether the databases need to be replaced, e.g., if a database is full, the databases need to be replaced, and in particular, a new database may be created for data writing. In order to enhance availability, the master control machine may be deployed in a failover cluster, which is a high availability infrastructure layer, and is composed of multiple computers, each computer being equivalent to a redundant node, and the entire cluster system allows a portion of the nodes to be dropped, failed or damaged without affecting the normal operation of the entire system. The process of one server taking over a failed server is often referred to as "failover," and if one server becomes unavailable, the other server automatically takes over the failed server and continues to process tasks. Each server in the cluster determines that at least one other server in the cluster is its backup server.
S102: synchronizing the grabbing time of grabbing data by the plurality of data grabbing devices;
the data capture device is typically running on a computer or is itself a computer, and the time systems of different computers may not be completely identical. In this embodiment, a mode of multiple points with full activity and no backup machine is adopted, but the capturing time of data is based on the time system of the data capturing device, and the difference of the time systems of different data capturing devices causes the problem that in the final recording, the data of the same data source is dithered due to the asynchronous capturing time. For example, in the process that the degree of one temperature sensor is stably increased, due to different grabbing time of two data grabbing devices, the phenomenon that the degree is repeatedly and linearly saw-toothed after final data records (including temperature and grabbing time) are ordered according to time can occur. That is, it is necessary to ensure that the time system according to which the data captured by the different data capturing devices is identical, otherwise many of the above-described data jitter problems may occur. In order to solve the problem of unification of the time system of the data grabbing process, the purpose of this step is to provide an accurate grabbing time and corresponding to the grabbed data, wherein the grabbing time is not different according to the time of the data grabbing device.
Further, synchronizing the capturing time of capturing data by the plurality of data capturing devices includes:
acquiring equipment time of a data source when capturing data, taking the equipment time as first capturing time, and corresponding the first capturing time to the captured data; and/or after timing with the time service server deployed on the master control machine, acquiring the equipment time of the data grabbing equipment when grabbing data, and taking the equipment time as the second grabbing time and corresponding to the grabbed data.
As an alternative implementation manner, this step may obtain the device time of the data source when capturing the data, and take the device time as the first capturing time, and simultaneously correspond the first capturing time to the captured data. The device time is not the device time of the data capturing device itself, but the device time of the device corresponding to the data source, for example, the data source depends on the temperature sensing device, and then the device time is the device time of the temperature sensing device, that is, the data capturing device captures the temperature data of the temperature sensing device and also captures the device time of the temperature sensing device at the time. Therefore, even if the time systems of different data grabbing devices are inconsistent, the time systems according to which the data grabbing process is performed can be ensured to be uniform, namely, the time systems of the devices corresponding to the data sources are adopted.
As another optional implementation manner, after timing with the timing server deployed on the central control unit, the step may acquire the device time of the data capturing device when capturing the data, and use the device time as the second capturing time and correspond to the captured data. The time service server can be deployed on the master control machine in advance, and when needed, the plurality of data grabbing devices can be used for timing with the time service server deployed on the master control machine, so that the time of each data grabbing device is kept consistent, and when the data grabbing devices grab data, the time of each data grabbing device can be directly used as grabbing time. The implementation can be performed when the first capturing time cannot be acquired, so that the time systems of different data capturing devices are ensured to be kept uniform.
Specifically, the time service server undergoes the following processes when performing single time service: (1) A client (specifically, data grabbing equipment in the embodiment of the invention) initiates a time service request; (2) The time service server obtains the time service request, (3) the time service server processes the time service request, (4) the time service server returns a result, and (5) the client obtains the result.
However, the foregoing time service process still has a problem of time service delay, which is mainly because the foregoing steps all require communication time or/and processing time, so that the client cannot directly use the obtained result.
In the single time service process, the client and the time service server cannot ascertain the time required by the steps. However, in general, the time consumed for each of the above steps is almost fixed during the multiple calls, and the time required for step (3) can be reduced to negligible by simplifying the algorithm logic. Therefore, in order to improve the time service accuracy, the time service method is optimized according to the embodiment of the invention, and the method is specifically as follows:
1. the client sends the current time to the time service server, and the time service server returns the difference value (first time difference) between the current time and the current time of the time service server to the client;
2. the client uses the first time difference to correct the own time;
3. the client sends the corrected current time to the time service server, and the time service server returns the difference value (second time difference) between the time and the current time of the time service server to the client;
4. after the client receives the second time difference, taking half of the second time difference, correcting the own time again by using the half of the second time difference, and sending the corrected current time to the time service server again, wherein the time service server still only returns the difference value (third time difference) between the time and the current time of the time service server to the client; the principle of taking half the second time difference is that the time consumed by the aforementioned step (1) +step (2) =the time consumed by the step (4) +step (5) is defaulted.
5. After the client receives the third time difference, continuing to take half of the third time difference, correcting the time of the client again by half of the third time difference, and sending the corrected current time to the time service server, wherein the time service server still only returns the difference value (fourth time difference) between the time and the current time of the time service server to the client;
the second time difference, the third time difference, the fourth time difference, and the like described above are considered to be caused by the difference between the time consumed in step (1) +step (2) and the time consumed in step (4) +step (5). The above difference can be used to correct the client time.
After repeating steps 4 and 5, excluding the abnormally delayed data, the time difference overall converges to 0. After the number of repetitions reaches a predetermined number (e.g., steps 4 and 5 are repeated up to 10 times) and the time difference converges to an acceptable range (a predetermined time difference range, e.g., within 1 ms), the time service operation is completed. The above process is an integral time service process, namely a single time service process comprising a plurality of times.
To prevent the time overhead associated with the network establishing a connection, the process is typically run again before the actual time service process, but the data is discarded. This procedure is used to offset the overhead of the first network connection setup.
The overall time service process may be repeated after a period of time (e.g., minutes), but steps 1-2 need not be performed when repeated. However, if the time difference cannot be converged, the whole process (including step 1-2) needs to be reworked, and the reasons for the incapacitation are usually caused by excessive temporary load of the server, unstable network burst, changed computer time and the like.
After the time service operation is finished, the client side does not set the time of the client side in the time modification process, but stores the difference value between the time service server and the local time, and calculates the time service server time, namely the standard time, by a method of adding the difference value between the local time and the local time when the time service server is used.
In the embodiment of the invention, the time service server is arranged in the master control machine. In the embodiment of the invention, the master control machine is preferably dual-machine hot standby deployment, so that the stability of the whole framework is ensured, and the fault tolerance is higher. The time service server needs to keep time stable as much as possible and can bear offline for a period of time, so when the master controllers are deployed by adopting the dual hot standby, one master controller is designated to be fixed as the time service server and does not follow the switching of the dual hot standby mechanism.
As another alternative embodiment, the two embodiments are combined, that is, the first capturing time and the second capturing time are acquired simultaneously, and the first capturing time and the second capturing time are corresponding to the captured data, that is, two types of time are provided for the user, and the user determines what type of time system is adopted according to the needs, or adopts two types of time systems, that is, either one of the capturing times can be adopted, or both types of capturing times can be adopted simultaneously.
S103: storing the data captured by the plurality of data capturing devices and the corresponding capturing time into a database in a record form;
the step is to write the obtained corresponding relation between the first capturing time and the captured data and/or the obtained corresponding relation between the second capturing time and the captured data into the database when the data is written into the database, namely, store the information into the database in a record form. This information is used to provide mathematical modeling and mathematical analysis to the user, and since both the first capture time and the second capture time are based on a consistent time system, the problem of jitter due to time inconsistencies is avoided.
S104: responding to the received data request instruction, and acquiring information corresponding to the appointed time;
the purpose of this step is to obtain information of a specified time according to the received data request instruction. Specifically, when a data request instruction is received, the data request instruction is parsed, so that information of specified time contained in the data request instruction is obtained. The designated time refers to the time corresponding to the data required by the data user.
The data request instruction may be issued by a data user to the database storing data by using the device or directly to the database storing data. The data request instruction can comprise a plurality of appointed times so as to simultaneously acquire a plurality of corresponding data, thereby improving the data request efficiency. The specified time may be a certain time point or a certain period of time. The data request instruction may further include information for specifying the data capturing device, that is, the requested data is the data captured by the specified data capturing device, so as to improve the data request efficiency on the premise of ensuring the accuracy, the continuity and the stability of the data.
S105: judging whether corresponding records can be searched from the database according to the appointed time; if yes, directly returning the record corresponding to the appointed time, otherwise, acquiring the record corresponding to the grabbing time closest to the appointed time and returning the record, and/or acquiring the estimated data of the appointed time and returning the estimated data.
The above steps are to judge whether the data corresponding to the specified time can be found in the database, if so, the data can be directly returned, that is, the record corresponding to the specified time is directly obtained and returned if the record corresponding to the specified time is just stored in the database. If not, the manner of returning the record needs to be selected according to the actual situation, for example, a record of the grabbing time closest to the designated time can be returned, or an estimated data return can be directly obtained.
Further, the acquiring the record corresponding to the capture time closest to the designated time and returning includes:
and acquiring the grabbing time which is not later than the designated time and is closest to the designated time from a database, taking the record corresponding to the grabbing time as the record corresponding to the designated time, and returning.
The step is to acquire corresponding data according to the acquired information of the designated time. Because of the higher capture density of the data capture device, the lifecycle of each data item is relatively short, and most data items can be addressed by providing earlier data. And searching the grabbing time which is not later than the appointed time and is closest to the appointed time, taking the record corresponding to the grabbing time as the record corresponding to the appointed time, and returning. Since the data are stored in the database, the time density of each item of data capture is not necessarily the same, so that only a record which is no later than the specified time and is closest to the specified time needs to be found for each item of data, for example, when the specified time is requested by a data user to be 1 point 23 minutes 45 seconds 678 milliseconds on the same day, the record which is no later than the specified time in each item of data can be searched in the database, and the record closest to the specified time is returned as the record corresponding to the specified time. As a preferred embodiment, the capture time (first capture time and/or second capture time) of the data is provided and returned at the same time when the record is returned, so that the data user can screen the validity of the data.
The acquiring the estimated data of the specified time and returning includes:
and acquiring the estimated data of the specified time according to a continuity rule, and returning the estimated data and the specified time in a recorded form.
This step is to acquire the estimated data at a specified time based on the rule of continuity of the data, so that it is required that the data itself should have continuity, i.e., the requested data should be continuously changed with time, otherwise the accuracy of the acquired estimated data will be affected.
In a specific application scenario, the obtaining the estimated data of the specified time according to the continuity rule includes: step one, acquiring data of a preset time period before and/or after the appointed time; and step two, obtaining estimated data of the specified time according to the continuous change relation of the data in a preset time period. In step one, data specifying a predetermined period of time associated with the time is acquired first, for example, data specifying a predetermined period of time before the time, or data specifying a predetermined period of time after the time, or data specifying predetermined periods of time before and after the time. The data of the predetermined time period includes data of a time period, which implies a change rule of the data, namely, a continuity rule, so that the estimated data is obtained based on the data. In the second step, according to the continuous change relation of the data in the preset time period, namely, the continuous law, the estimated data of the appointed time is obtained, namely, the data of the appointed time is predicted based on the continuous change relation of the data in the preset time period, the predicted data is taken as the estimated data of the appointed time, and the estimated data can be taken as the data corresponding to the appointed time and returned.
It should be noted that, in the foregoing specific application scenario, the data-based continuity rule is a continuity rule of a single data item, that is, the single data item, so the foregoing application scenario acquires the estimated data based on the single data-based continuity rule. If the data of different data items have an association relationship, that is, the data of different data items also have a certain rule of continuity, the estimated data of the specified data items can be obtained based on the data of a plurality of data items (which may or may not contain the specified data items), and the data items are jointly used, so that complex mathematical logic or business logic can be combined and implemented. For example, the data items of the setting input are: the alarm state is true, the offset time is 60 seconds, other input data items comprise a certain group of motor state data, and the corresponding output data items are motor state data during fault. When the system is executed, the system can provide the motor state and the alarm state 60 seconds after the current processing time, and when the alarm state is true, the system copies the output motor state data and outputs the motor state data by the name of 'motor state data when fault'; and if the alarm state is false, the alarm state is not output. The output data can be selectively stored in a database or can be recorded in a memory only for other processors in the same group to continuously work in series as a condition. At the final export, the "alarm state" and the "motor state data at fault" are selected for export, but the original motor state data is not selected for export.
In addition, the implementation is performed on the condition that the application scenarios are all based on the continuity rule of the data, and obviously, the continuity rule in the embodiment of the invention is generalized and also comprises other rules, such as mutation rules, and the like, and the method in the embodiment of the invention can be adopted to acquire the data corresponding to the appointed time according to the change rules of the data.
The data of different data items also have different continuity rules, namely, the continuous change models are different, so the mode of acquiring the estimated data is also different. For example, a data item whose continuously changing relationship is linearly changing over a predetermined period of time; for another data item, the continuously changing relationship thereof over a predetermined period of time is parabolic in form; for yet another data item, its continuously changing relationship over a predetermined period of time is changed in a pre-curved form. The manner in which the estimated data is obtained is different for data of different continuously varying relationships.
Taking the example that the continuous change relation of the data in the preset time period is linear change, a linear formula of the data and the grabbing time in the preset data period can be obtained: f (t) =at+b, where t is time, a is slope, b is linear relationship offset value, and f (t) is a data value. According to the linear formula, the estimated data of the specified time can be calculated.
Specifically, for most data, it is generally applied to a continuous variation model, and data can be approximately understood as continuous linear variation in a short time, so that the value of a specified time can be estimated by f (t) =at+b (where a is a slope, t is a time offset, b is a linear relationship offset value, and f (t) is a data value). The method of determining a and b requires an average of the reference data over a short period of time.
Firstly, for each data item, after the time is unified, each data item may contain data from different data grabbing devices at the same time, so that mathematical average is performed on the data from different data grabbing devices at each time to obtain a "data time pair" at each time (one record includes data and corresponding grabbing time), that is, mathematical average is performed on the data from all the data grabbing devices in the same data item at the same time; the above linear formula is then obtained from these "data time pairs".
When the linear formula is obtained, the average value of f (t) and the average value of t are required to be obtained to determine a and b, and according to different data properties, calculation of mathematical average values can be simply performed to obtain the two average values (the two average values can be generally used for processing data without time accumulation or data with very average sampling time) so as to save calculation force, namely; the two average values (which can be generally used for data with insignificant change period or large sampling time interval dynamic range) can be calculated by generating a line graph according to adjacent data, integrating the area and dividing the area by the time so as to improve the accuracy of data calculation.
In statistics, the acquisition of the above linear formula is a regression analysis that models the relationship between an independent and dependent variable using a least squares function. This function is a linear combination of model parameters called regression coefficients.
Specifically, the formula for solving a and b is as follows:
Figure BDA0002160412220000151
wherein, the liquid crystal display device comprises a liquid crystal display device,
Figure BDA0002160412220000152
is the average value representing t,/->
Figure BDA0002160412220000153
Represents the average value of f (t), x i Indicating the time of the ith moment, y i The f (t) value at the i-th time is shown.
In order for the data user to make the data source more clear, when the data corresponding to the specified time is returned, if the data is the estimated data, the presentation data is the estimated data and not the actual capture value. In addition, the data source for acquiring the estimated data, that is, the data of the specified time period acquired in the first step may be simultaneously provided when the data corresponding to the specified time is returned, so that the data user can understand the basis for estimating the estimated data, determine whether the estimated data is accurate, and the like. Furthermore, the continuous change relation type according to the data corresponding to the designated time can be provided at the same time when the data corresponding to the designated time is returned, so that a data user can know whether the mode of acquiring the estimated data is accurate or not. The data source and/or the continuously changing relationship type returned in the above steps may be provided according to the request of the data requester, or may be provided automatically. In addition, the determination of the continuous change relation type of the data in the preset time period can be determined according to the automatic determination of the system, and also can be determined according to the type preset by a data user. The combined intelligent algorithm can be, for example, a currently popular deep learning algorithm is used for constructing an analysis model, the analysis model is trained by adopting data within a preset time period or outside the preset time period, and then a specified time is input, so that data corresponding to the specified time is output.
In this embodiment, even if there is no data of a specified time requested by the data user in the database, it is also possible to obtain the estimated data of the specified time by way of estimation and as the data corresponding to the specified time, since the estimated data is determined based on the continuous change relation of the data over a predetermined period of time, the accuracy of the estimated data is high, and there is strong reliability in use for mathematical modeling and mathematical analysis, that is, this embodiment provides the data alignment service, facilitating the data user to pull the data with the specified time as a standard.
Further, the storing the data captured by the plurality of data capturing devices and the corresponding capturing time in a database in a record form includes:
judging whether the database is in a storable state or not;
the purpose of this step is to determine if the captured data can be saved to a database. If yes, directly storing, otherwise, temporarily storing. For how to judge whether the captured data can be stored in the database, the method can be determined by judging whether the database can be written normally, if the database can be written normally, the captured data can be stored in the database, and if the database cannot be written normally, the captured data cannot be stored in the database. The reasons for the abnormal writing of the database are various, for example, abnormal writing of the database is caused, normal writing of the database is caused due to the fact that the database is maintained, or normal writing of the database is caused due to the fact that the database is disconnected from the data grabbing device. However, no matter what kind of reasons the database cannot be written normally, the continuous capturing of the data in the embodiment of the invention is not affected, because the database and the data capturing device are separately arranged, and the data capturing device can still continuously capture the data in the embodiment of the invention. The embodiment of the invention also adopts corresponding means to process the situation that the database cannot be written normally.
If the database is in a storable state, the captured data and the corresponding capturing time are stored in the database in a record form; the step is to write the captured data into the database normally under the condition that the database can be written normally, so as to realize the data storage function.
And if the database is not in a storable state, temporarily storing the captured data to the local of the corresponding data capturing equipment, so as to restore and store the temporarily stored data and the corresponding capturing time in the database in a record form when needed. Namely, for the situation that the database cannot be written normally, the processing mode of the embodiment of the invention is to temporarily store the captured data to the local of the data capturing device, then restore and store the temporarily stored data and the corresponding capturing time to the database in a record form, and the premise of restoring and storing is that the database restores the normal writing function. By temporarily storing locally and then writing the data into the database, the situation that the data is lost when the database cannot be normally written can be avoided, that is, the embodiment of the invention can not only cope with the situation that one or a plurality of data grabbing devices are abnormal or are maintained, ensure that the data grabbing process is continuously and stably carried out for a long time, but also cope with the situation that the database is abnormal or is maintained, so that the embodiment of the invention can also ensure that the data storing process is continuously and stably carried out, avoid the situation that the data is empty, and avoid the situation that the data is lost.
In a specific application scenario, the temporarily storing the captured data to the local of the corresponding data capturing device, so as to restore and save the temporarily stored data to the database when needed includes: step one, temporarily storing the captured data to the local of corresponding data capturing equipment; and secondly, restoring and storing the temporarily stored data and the corresponding grabbing time into the database in a record mode when the normal writing function of the database is restored. When the normal writing function of the database is restored, the data can be written normally, so that the data temporarily stored locally can be restored and stored in the database in time at the moment. Because the local part of the data grabbing device generally does not have a large storage space, the embodiment of the invention can only generally cope with the situation that the database cannot be written normally in a short time, if the database cannot be written normally for a long time, the local part of the data grabbing device cannot temporarily store a large amount of data, so that the data cannot be grabbed and stored normally, and when the normal writing function is required to be recovered, the temporarily stored data is recovered and stored to the database in time, so that the situation of data loss is avoided. As an alternative way, a temporary storage condition can be set for temporary storage data, so that the data capture device cannot store the data without limitation, and the normal capture function of the data capture device is affected, for example, the temporary storage condition is that the data with the predetermined time length is at most temporarily stored, or the data with the predetermined size is temporarily stored at most, the predetermined time length is customized by a deployment personnel according to the specific requirement of database maintenance, the predetermined size can be generally set to be 5-10 minutes, the predetermined size can be set according to a similar principle, in general, the temporary storage data in the data capture device is not too much, otherwise, the data capture device occupies the internal memory of the data capture device, and delay is too much, so that the data written in the database is the data earlier than the current time, and the stability and accuracy of mathematical modeling and analysis are affected. The function realized by the steps is essentially a delayed writing function, namely, the data and the corresponding grabbing time are delayed to be written into the database in a record form, and the more important advantage of the method is that the database in the embodiment of the invention can be allowed to be a single-point database, wherein the single-point database is only deployed, and under the condition that only one database is deployed, the condition that the database cannot be normally written is still allowed to appear, so that the application scene of the embodiment of the invention can be the scene of the single-point database, and the deployment cost of the database is effectively reduced under the condition that the writing stability is ensured. Of course, the application scenario of the embodiment of the present invention may also be a scenario of a clustered database, that is, the database is a clustered database, and the clustered database may improve writing stability without considering cost, because the clustered database may allow any one or several databases to be maintained for a long time or abnormal, without causing a situation of empty window period and data loss.
In this embodiment, the database is directly written in a state that the database can be saved, and only when a temporary problem (such as offline) occurs in the database, the database is locally cached as an emergency solution. This solves the following problems: (1) Since the reliability of the database is higher than that of a general file system, the probability of possible data write errors can be reduced. (2) The robustness of the database server itself is generally far superior to servers carrying general applications, reducing the likelihood of data errors and loss.
Referring to fig. 3, fig. 3 is a schematic structural diagram of a data capturing and saving device according to an embodiment of the present invention;
the apparatus may include:
the data grabbing module 301 is configured to grab data in a data source when a plurality of data grabbing devices trigger a data grabbing condition; wherein, at least one of the data sources of the plurality of data grabbing devices grabbing data is the same data source;
the time synchronization module 302 is configured to synchronize the capturing time of the plurality of data capturing devices for capturing data;
a saving module 303, configured to save the data captured by the plurality of data capturing devices and the corresponding capturing time to a database in a record form;
A request response module 304, configured to obtain information corresponding to the specified time in response to the received data request instruction;
a data alignment module 305, configured to determine whether a corresponding record can be found from the database according to the specified time; if yes, directly returning the record corresponding to the appointed time, otherwise, acquiring the record corresponding to the grabbing time closest to the appointed time and returning the record, and/or acquiring the estimated data of the appointed time and returning the estimated data.
In this embodiment, the data in the data sources are captured by the plurality of data capturing devices, at least one of the data sources captured by the plurality of data capturing devices is the same data source, and then capturing time is synchronized, and the data are stored and aligned. The embodiment can allow other data grabbing devices to continue grabbing data under the condition that one data grabbing device is maintained or a plurality of data grabbing devices are maintained at the same time; while also allowing maintenance operations on the database. The device optimizes the data grabbing and storing process, can avoid the condition of data loss in the data grabbing and storing process, realizes continuous and stable grabbing of data, simultaneously synchronizes the grabbing time, avoids the problem of data jitter, aligns the data and ensures that the data with accurate time can be provided.
Further, the data grabbing module 301 includes:
the triggering unit is used for triggering the data grabbing condition when responding to the data grabbing instruction issued by the master control machine;
and the grabbing unit is used for grabbing data in the data source according to the data grabbing instruction.
Further, the saving module 303 includes:
the judging unit is used for judging whether the database is in a storable state or not;
the storage unit is used for storing the captured data and the corresponding capture time into the database in a record form when the database is in a storable state;
and the temporary storage unit is used for temporarily storing the captured data to the local of the corresponding data capture equipment when the database is not in a storable state, so that the temporarily stored data and the corresponding capture time can be restored to be stored in the database in a record mode when the data is required.
Further, the time synchronization module 302 includes:
the time data corresponding unit is used for acquiring the equipment time of the data source when the data are captured, taking the equipment time as first capturing time and corresponding the first capturing time to the captured data; and/or after timing with the time service server deployed on the master control machine, acquiring the equipment time of the data grabbing equipment when grabbing data, and taking the equipment time as the second grabbing time and corresponding to the grabbed data.
Further, the data alignment module 305 includes:
the first data return unit is used for acquiring the grabbing time which is not later than the appointed time and is closest to the appointed time from the database, and taking the record corresponding to the grabbing time as the record corresponding to the appointed time and returning the record.
Further, the data alignment module 305 includes:
and the second data return unit is used for acquiring the estimated data of the specified time according to the continuity rule and returning the estimated data and the specified time in a recorded form.
Further, the second data return unit includes:
a predetermined time period data acquisition subunit configured to acquire data of a predetermined time period before and/or after the specified time;
and the data estimation subunit is used for acquiring the estimated data of the specified time according to the continuous change relation of the data in the preset time period.
Since the embodiments of the apparatus portion and the embodiments of the method portion correspond to each other, the embodiments of the apparatus portion are referred to the description of the embodiments of the method portion, and are not repeated herein.
The present invention also provides a computer-readable storage medium having stored thereon a computer program which, when executed, implements the method provided by the above embodiments. The storage medium may include: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RandomAccess Memory, RAM), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
The invention also provides a computer device, which can comprise a memory and a processor, wherein the memory stores a computer program, and the processor can realize the method provided by the embodiment when calling the computer program in the memory. Of course the computer device may also include various network interfaces, power supplies, and the like.
In the description, each embodiment is described in a progressive manner, and each embodiment is mainly described by the differences from other embodiments, so that the same similar parts among the embodiments are mutually referred. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant points refer to the description of the method section. It should be noted that it will be apparent to those skilled in the art that various modifications and adaptations of the invention can be made without departing from the principles of the invention and these modifications and adaptations are intended to be within the scope of the invention as defined in the following claims.
It should also be noted that in this specification, relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.

Claims (8)

1. The data grabbing and storing method is characterized by comprising the following steps:
capturing data in a data source when a plurality of data capturing devices trigger data capturing conditions; wherein, at least one of the data sources of the plurality of data grabbing devices grabbing data is the same data source;
synchronizing the grabbing time of grabbing data by the plurality of data grabbing devices;
storing the data captured by the plurality of data capturing devices and the corresponding capturing time into a database in a record form;
responding to the received data request instruction, and acquiring information corresponding to the appointed time;
judging whether corresponding records can be searched from the database according to the appointed time; if yes, directly returning the record corresponding to the appointed time, otherwise, acquiring the record corresponding to the grabbing time closest to the appointed time and returning the record, and/or acquiring the estimated data of the appointed time and returning the record;
when the plurality of data grabbing devices trigger the data grabbing conditions, grabbing the data in the data source comprises the following steps:
triggering the data grabbing condition when responding to a data grabbing instruction issued by the master control machine;
capturing data in a data source according to the data capturing instruction;
The synchronizing the capturing time of capturing data by the plurality of data capturing devices includes:
acquiring equipment time of a data source when capturing data, taking the equipment time as first capturing time, and corresponding the first capturing time to the captured data; when the first grabbing time cannot be acquired, acquiring equipment time of data grabbing equipment when grabbing data after timing with a timing server arranged on a master control machine, and taking the equipment time as second grabbing time and corresponding to the grabbed data;
the time service process comprises the following steps: (1) The client sends the current time to the time service server, and the time service server returns the difference value between the current time and the current time of the time service server to the client, wherein the difference value is a first time difference; (2) The client uses the first time difference to correct the self time; (3) The client sends the corrected current time to the time service server, and the time service server returns the difference between the time and the current time of the time service server to the client, wherein the difference is a second time difference; (4) After the client receives the second time difference, taking half of the second time difference, correcting the own time again by using the half of the second time difference, and sending the corrected current time to the time service server again, wherein the time service server still only returns the difference value between the time and the current time of the time service server to the client, and the difference value is a third time difference; (5) After the client receives the third time difference, continuing to take half of the third time difference, correcting the time of the client again by half of the third time difference, and sending the corrected current time to the time service server, wherein the time service server still only returns the difference value between the time and the current time of the time service server to the client, and the difference value is a fourth time difference; repeating the steps (4) and (5), and completing the time service operation after the repetition number reaches a preset number and the time difference is converged to a preset time difference range.
2. The method for capturing and storing data according to claim 1, wherein storing the data captured by the plurality of data capturing devices and the corresponding capturing time in a database in a record form includes:
judging whether the database is in a storable state or not;
if yes, the captured data and the corresponding capturing time are stored in a database in a record mode;
if not, the captured data is temporarily stored in the local of the corresponding data capturing device, so that the temporarily stored data and the corresponding capturing time are restored and stored in the database in a record mode when needed.
3. The method for capturing and saving data according to claim 1, wherein the acquiring and returning the record corresponding to the capturing time closest to the specified time includes:
and acquiring the grabbing time which is not later than the designated time and is closest to the designated time from a database, taking the record corresponding to the grabbing time as the record corresponding to the designated time, and returning.
4. The data grabbing and saving method of claim 1, wherein said obtaining the estimated data for the specified time and returning comprises:
And acquiring the estimated data of the specified time according to a continuity rule, and returning the estimated data and the specified time in a recorded form.
5. The method for capturing and storing data according to claim 4, wherein the obtaining the estimated data of the specified time according to the rule of continuity comprises:
acquiring data of a preset time period before and/or after the appointed time;
and obtaining the estimated data of the specified time according to the continuous change relation of the data in the preset time period.
6. A data grabbing and storing device, comprising:
the data grabbing module is used for grabbing data in the data source when the plurality of data grabbing devices trigger the data grabbing conditions; wherein, at least one of the data sources of the plurality of data grabbing devices grabbing data is the same data source;
the time synchronization module is used for synchronizing the grabbing time of the plurality of data grabbing devices for grabbing data;
the storage module is used for storing the data captured by the plurality of data capturing devices and the corresponding capturing time into a database in a recorded form;
the request response module is used for responding to the received data request instruction and acquiring information corresponding to the appointed time;
The data alignment module is used for judging whether corresponding records can be searched from the database according to the appointed time; if yes, directly returning the record corresponding to the appointed time, otherwise, acquiring the record corresponding to the grabbing time closest to the appointed time and returning the record, and/or acquiring the estimated data of the appointed time and returning the record;
the data grabbing module comprises:
the triggering unit is used for triggering the data grabbing condition when responding to the data grabbing instruction issued by the master control machine;
the grabbing unit is used for grabbing data in a data source according to the data grabbing instruction;
the time synchronization module includes:
the time data corresponding unit is used for acquiring the equipment time of the data source when the data are captured, taking the equipment time as first capturing time and corresponding the first capturing time to the captured data; when the first grabbing time cannot be acquired, acquiring equipment time of data grabbing equipment when grabbing data after timing with a timing server arranged on a master control machine, and taking the equipment time as second grabbing time and corresponding to the grabbed data;
the time service process comprises the following steps: (1) The client sends the current time to the time service server, and the time service server returns the difference value between the current time and the current time of the time service server to the client, wherein the difference value is a first time difference; (2) The client uses the first time difference to correct the self time; (3) The client sends the corrected current time to the time service server, and the time service server returns the difference between the time and the current time of the time service server to the client, wherein the difference is a second time difference; (4) After the client receives the second time difference, taking half of the second time difference, correcting the own time again by using the half of the second time difference, and sending the corrected current time to the time service server again, wherein the time service server still only returns the difference value between the time and the current time of the time service server to the client, and the difference value is a third time difference; (5) After the client receives the third time difference, continuing to take half of the third time difference, correcting the time of the client again by half of the third time difference, and sending the corrected current time to the time service server, wherein the time service server still only returns the difference value between the time and the current time of the time service server to the client, and the difference value is a fourth time difference; repeating the steps (4) and (5), and completing the time service operation after the repetition number reaches a preset number and the time difference is converged to a preset time difference range.
7. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the data grabbing and saving method of any one of claims 1 to 5 when the computer program is executed by the processor.
8. A computer readable storage medium, characterized in that the computer readable storage medium has stored thereon a computer program which, when executed by a processor, implements the data grabbing and saving method of any one of claims 1 to 5.
CN201910730486.7A 2019-08-08 2019-08-08 Data grabbing and storing method and device, computer equipment and storage medium Active CN110442770B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201910730486.7A CN110442770B (en) 2019-08-08 2019-08-08 Data grabbing and storing method and device, computer equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201910730486.7A CN110442770B (en) 2019-08-08 2019-08-08 Data grabbing and storing method and device, computer equipment and storage medium

Publications (2)

Publication Number Publication Date
CN110442770A CN110442770A (en) 2019-11-12
CN110442770B true CN110442770B (en) 2023-06-20

Family

ID=68434014

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201910730486.7A Active CN110442770B (en) 2019-08-08 2019-08-08 Data grabbing and storing method and device, computer equipment and storage medium

Country Status (1)

Country Link
CN (1) CN110442770B (en)

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2009069892A (en) * 2007-09-10 2009-04-02 Ricoh Co Ltd Log time adjustment method, server and information processing system
CN102761456A (en) * 2011-04-29 2012-10-31 腾讯科技（深圳）有限公司 Method, device and system for acquiring performance parameters of browser of mobile terminal
CN102916935A (en) * 2011-08-04 2013-02-06 深圳华强电子交易网络有限公司 Site content pickup-preventing method

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP3613550B2 (en) * 2000-03-29 2005-01-26 インターナショナル・ビジネス・マシーンズ・コーポレーション Web page acquisition service system, server, web page acquisition method, storage medium, and program transmission device
US7707203B2 (en) * 2005-03-11 2010-04-27 Yahoo! Inc. Job seeking system and method for managing job listings
CN106339385B (en) * 2015-07-08 2020-06-16 阿里巴巴集团控股有限公司 System for capturing webpage, method for distributing webpage capturing nodes and method for capturing webpage
CN107436904B (en) * 2016-05-27 2020-06-30 北京京东尚科信息技术有限公司 Data acquisition method, data acquisition device, and computer-readable storage medium
CN107729139B (en) * 2017-09-18 2021-02-26 北京京东尚科信息技术有限公司 Method and device for concurrently acquiring resources
CN109101521A (en) * 2018-06-12 2018-12-28 江苏开拓信息与系统有限公司 The automatic extraction system of data based on big data

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2009069892A (en) * 2007-09-10 2009-04-02 Ricoh Co Ltd Log time adjustment method, server and information processing system
CN102761456A (en) * 2011-04-29 2012-10-31 腾讯科技（深圳）有限公司 Method, device and system for acquiring performance parameters of browser of mobile terminal
CN102916935A (en) * 2011-08-04 2013-02-06 深圳华强电子交易网络有限公司 Site content pickup-preventing method

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
冲击波测试系统中GPS同步技术研究;王晓楠;王代华;苏尚恩;高阳;;现代雷达(第01期);全文 *
基于分布式算法的时间同步系统的研究和实现;张卓;陈春玲;;电力系统通信(第05期);全文 *

Also Published As

Publication number Publication date
CN110442770A (en) 2019-11-12

Similar Documents

Publication Publication Date Title
CN107291787B (en) Main and standby database switching method and device
CN106202075B (en) Method and device for switching between main database and standby database
CN105471622B (en) A kind of high availability method and system of the control node active-standby switch based on Galera
EP1623325B1 (en) Managing tasks in a data processing environment
CN109032849B (en) Hot backup system, hot backup method and computer equipment
US9189348B2 (en) High availability database management system and database management method using same
TWI677797B (en) Management method, system and equipment of master and backup database
CN106339278A (en) Data backup and recovery method of network file system
CN112118130B (en) Self-adaptive distributed cache active-standby state information switching method and device
US20180074925A1 (en) Systems and methods for resynchronizing mirroring partners in a storage system
US7669080B2 (en) Reducing likelihood of data loss during failovers in high-availability systems
JP2010033467A (en) Information management system
CN110011853B (en) Cross fault troubleshooting method and device for multiple platforms and clusters
CN112069018B (en) Database high availability method and system
CN116781488A (en) Database high availability implementation method, device, database architecture, equipment and product
US20050234919A1 (en) Cluster system and an error recovery method thereof
CN110442770B (en) Data grabbing and storing method and device, computer equipment and storage medium
WO2015196692A1 (en) Cloud computing system and processing method and apparatus for cloud computing system
CN116185697B (en) Container cluster management method, device and system, electronic equipment and storage medium
CN117271227A (en) Database cluster master node switching method, system and management and control platform
CN110677316A (en) Distributed storage server network card detection method and system
Yang et al. Multi-Active Multi-Datacenter Distributed Database Architecture Design based-on Secondary Development Zookeeper
CN110675614A (en) Transmission method of power monitoring data
CN113472891A (en) SDN controller cluster data processing method, device and medium
CN115344008B (en) High-reliability time keeping method for cooperative application of multiple controllers

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant