CN111090662B - Method, equipment and medium for acquiring data from real-time database to relational database - Google Patents
Method, equipment and medium for acquiring data from real-time database to relational database Download PDFInfo
- Publication number
- CN111090662B CN111090662B CN201911318305.6A CN201911318305A CN111090662B CN 111090662 B CN111090662 B CN 111090662B CN 201911318305 A CN201911318305 A CN 201911318305A CN 111090662 B CN111090662 B CN 111090662B
- Authority
- CN
- China
- Prior art keywords
- data
- difference
- time
- dead zone
- acquiring
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 47
- 238000007405 data analysis Methods 0.000 abstract description 5
- 230000008569 process Effects 0.000 description 13
- 230000006870 function Effects 0.000 description 11
- 238000012545 processing Methods 0.000 description 11
- 238000003860 storage Methods 0.000 description 11
- 238000004458 analytical method Methods 0.000 description 10
- 238000010586 diagram Methods 0.000 description 8
- 238000004590 computer program Methods 0.000 description 7
- 230000008859 change Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 239000007788 liquid Substances 0.000 description 2
- 238000005259 measurement Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000005856 abnormality Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000010924 continuous production Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 230000007717 exclusion Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000005025 nuclear technology Methods 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 210000000697 sensory organ Anatomy 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000000153 supplemental effect Effects 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2308—Concurrency control
- G06F16/2315—Optimistic concurrency control
- G06F16/2322—Optimistic concurrency control using timestamps
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/258—Data format conversion from or to a database
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a method, equipment and medium for acquiring data from a real-time database to a relational database. The method comprises the following steps: acquiring first data in a real-time database and acquiring historical data corresponding to the first data; acquiring a difference dead zone corresponding to the first data, wherein the difference dead zone comprises a difference proportion dead zone and/or a difference absolute value dead zone; determining an absolute difference between the first data and the historical data, and determining a reference difference between the first data and the historical data according to a difference dead zone; if the absolute difference and the reference difference meet the preset conditions, the first data are converted into corresponding second data and then written into the relation coefficient database. According to the method and the device, a part of invalid data can be removed from the data of the real-time database according to the difference dead zone, and the valid data are converted into the relational data, so that the efficiency and the reliability of subsequent data analysis can be improved, and the data in the real-time database can be utilized more effectively.
Description
Technical Field
The present disclosure relates to the field of computer technologies, and in particular, to a method, an apparatus, and a medium for acquiring data from a real-time database to a relational database.
Background
At present, the development of the nuclear technology in China is important, how to ensure the safe operation of service equipment (including nuclear power equipment) becomes a main topic, the operation key data of the nuclear power equipment comprises temperature, pressure, flow and other switching value data and the like, the data are obtained through corresponding sensors and are transferred into a real-time database through sensor interfaces, the data are huge, the real-time database can only be used for storage because of the huge data, and the common real-time database comprises a Plant Information System (PI) database, so that how to analyze the data, thereby realizing the judgment of the safety, becoming a main research direction and mainly embodying the following aspects:
1. artifacts in real-time databases
The data stored in the real-time database by the sensor has a large amount of false signals, and the existence of the false signals only causes disturbance to data analysis and influences on analysis speed;
2. analysis of appearance influence of a large number of repeated data
The method comprises the steps that a large amount of repeated data exist in signals in a real-time database, the repeated data are generated due to the fact that process data differences of service equipment are extremely small under stable operation and good state, trend analysis of the data is not great, therefore the difference data are required to be analyzed, a difference dead zone of the difference data is calculated according to a specific algorithm, whether the difference value falls in the difference dead zone or not is determined to be the repeated data or not, the repeated data can be removed, and only the difference fatigue data are reserved for analysis;
3. The presence of special data
Some switching value signal data and special information data do not need to participate in analysis, and the signal data can be removed;
4. data fault caused by artificial objective factors
For some special reasons, the data in the real-time database is not complete, for example, the service equipment can be powered off and stopped for debugging or other reasons, and the equipment fatigue data can not be generated; for another example, real-time data is not stored from the sensors into the real-time database for some special reasons; both of these conditions can lead to data faults, whereas trend analysis algorithms require fatigue data to be continuous, thus requiring supplemental means for interrupted data.
Based on this, a scheme capable of more effectively utilizing data in a real-time database is required.
Disclosure of Invention
The embodiment of the application provides a method, equipment and medium for acquiring data from a real-time database to a relational database, which are used for solving the following technical problems in the prior art: there is a need for a scheme that enables more efficient use of data in a real-time database.
The embodiment of the application adopts the following technical scheme:
a method of retrieving data from a real-time database to a relational database, comprising:
Acquiring first data in the real-time database and acquiring historical data corresponding to the first data;
acquiring a difference dead zone corresponding to the first data, wherein the difference dead zone comprises a difference proportion dead zone and/or a difference absolute value dead zone;
determining an absolute difference between the first data and the historical data, and determining a reference difference between the first data and the historical data according to the difference dead zone;
and if the absolute difference and the reference difference meet preset conditions, converting the first data into corresponding second data and writing the second data into the relational database.
Optionally, if the difference dead zone includes the difference absolute value dead zone, acquiring the difference dead zone corresponding to the first data includes:
acquiring a corresponding difference absolute value dead zone preset for the first data; or,
and determining a difference absolute value dead zone corresponding to the first data according to the historical data.
Optionally, if the difference dead zone includes the difference proportional dead zone and the difference absolute value dead zone, determining a reference difference of the first data compared to the historical data according to the difference dead zone includes:
Determining a proportion difference according to the difference proportion dead zone and the historical data;
and determining a reference difference of the first data compared with the historical data according to the proportion difference and the difference absolute value dead zone.
Optionally, the method further comprises:
acquiring a first time corresponding to the first data, wherein the first time is the recording time of the first data;
determining a second time that is not earlier than the first time;
acquiring third data in the real-time database during a preset deadline from the second time to later than the second time;
and writing data into the relational database according to the third data.
Optionally, after the first data is acquired, before the third data is acquired, the first data is the data with the latest recording time which is acquired from the real-time database currently.
Optionally, acquiring third data in the real-time database during the second time to a preset deadline later than the second time includes:
determining the second time as a starting point, and attempting to acquire third data in the real-time database when the recording time is within a set duration after the starting point;
If the acquisition is successful, the starting point is redetermined according to the recording time of the latest recorded data acquired at this time, then the acquisition of the third data is continuously attempted according to the redetermined starting point, and the step is repeatedly executed until the set duration contains the preset deadline, wherein the redetermined starting point is not earlier than the recording time.
Optionally, after determining the second time as a starting point and attempting to acquire the third data written in the real-time database within a set period of time after the starting point, the method further includes:
if the acquisition fails, the set time length is increased, and the acquisition of the third data written in the real-time database is retried in the increased set time length from the starting point.
Optionally, the preset deadline is a current time of a server of the relational database.
An apparatus for retrieving data from a real-time database to a relational database, comprising:
at least one processor; the method comprises the steps of,
a memory communicatively coupled to the at least one processor; wherein,
the memory stores instructions executable by the at least one processor to enable the at least one processor to:
Acquiring first data in the real-time database and acquiring historical data corresponding to the first data;
acquiring a difference proportion dead zone and a difference absolute value dead zone corresponding to the first data;
determining an absolute difference between the first data and the historical data, and determining a reference difference between the first data and the historical data based on the difference proportional dead zone and the difference absolute value dead zone;
and if the absolute difference and the reference difference meet preset conditions, converting the first data into corresponding second data and writing the second data into the relational database.
A medium for retrieving data from a real-time database to a relational database, the medium comprising a non-volatile computer storage medium having stored thereon computer executable instructions configured to:
acquiring first data in the real-time database and acquiring historical data corresponding to the first data;
acquiring a difference proportion dead zone and a difference absolute value dead zone corresponding to the first data;
determining an absolute difference between the first data and the historical data, and determining a reference difference between the first data and the historical data based on the difference proportional dead zone and the difference absolute value dead zone;
And if the absolute difference and the reference difference meet preset conditions, converting the first data into corresponding second data and writing the second data into the relational database.
The above-mentioned at least one technical scheme that this application embodiment adopted can reach following beneficial effect: according to the method, a part of invalid data can be removed from the data of the real-time database according to the difference dead zone, and the valid data are converted into the relational data, so that the efficiency and the reliability of subsequent data analysis can be improved, and the data in the real-time database can be utilized more effectively.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the application and do not constitute an undue limitation to the application. In the drawings:
FIG. 1 is a flow chart of a method for retrieving data from a real-time database to a relational database provided in some embodiments of the present application;
fig. 2 is a detailed flowchart of the method of fig. 1 in an application scenario provided in some embodiments of the present application;
fig. 3 is a detailed flow chart of a continuous data reading scheme in an application scenario provided in some embodiments of the present application;
Fig. 4 is a schematic diagram of the scheme of fig. 3 in an application scenario provided in some embodiments of the present application.
Detailed Description
For the purposes, technical solutions and advantages of the present application, the technical solutions of the present application will be clearly and completely described below with reference to specific embodiments of the present application and corresponding drawings. It will be apparent that the described embodiments are only some, but not all, of the embodiments of the present application. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are within the scope of the present disclosure.
The application discloses a scheme for acquiring data from a real-time database to a relational database. The scheme mainly has the following characteristics: converting real-time data in the real-time database into relational data, and writing the relational data into the relational database (the process can be called updating the real-time data into the relational database); when the real-time data is transferred from the real-time database to the relational database, invalid data (which can comprise data with little effect on subsequent data analysis) is removed according to some algorithms, wherein the invalid data comprises false signals, repeated data, switching value data and other special types of data; real-time data can be acquired from service equipment (such as temperature, pressure, flow and the like of nuclear power equipment), and analysis and prediction of equipment data are realized through preprocessing and analysis of the real-time data.
By analyzing and predicting the equipment data, the fatigue degree of the internal facilities such as business equipment can be judged instead of the mode of experience identification and sense organs. In particular to nuclear power equipment, the nuclear power equipment has certain danger, and through analysis and prediction of nuclear power equipment data, the state of the nuclear power equipment can be accurately judged, and the safety of the nuclear power equipment and the safety of operators can be better ensured.
In one application scenario, data is obtained from a real-time database to a relational database by obtaining real-time data from the real-time database and calling application program interface (Application Programming Interface, API) functions to obtain by using a real-time interface. However, through actual measurement operation, numerous problems exist in developing an API interface by a database vendor, including: the API function may transfer a large amount of meaningless duplicate data that is normal in the real-time database, but the API function may automatically generate messy duplicate data when it obtains the data; the API function cannot realize the functions of removing the false signal, eliminating the repeated signal, and eliminating the switching value data; the API function cannot distinguish between time intervals of different types of signals, and cannot continuously acquire signal data. The solution of the present application can at least partially solve these specific problems, and is described below.
FIG. 1 is a flow chart of a method for retrieving data from a real-time database to a relational database according to some embodiments of the present application.
The method of fig. 1 comprises the steps of:
s100: acquiring first data in the real-time database, and acquiring historical data corresponding to the first data.
In some embodiments of the present application, assuming that the first data is a value of a certain temperature parameter, the historical data, such as a last recorded value of the temperature parameter or an average value of last recorded values of the temperature parameter, may be obtained from a real-time database, or may be obtained from a relational database if the historical data has been updated into the relational database.
The purpose of obtaining the history data is to determine whether the first data needs to be updated in the relational database, and if the first data is changed too little compared with the history data or is predetermined other invalid data (for example, switching value data, etc.), the first data may be ignored and not updated in the relational database.
S102: and acquiring a difference dead zone corresponding to the first data, wherein the difference dead zone comprises a difference proportion dead zone and/or a difference absolute value dead zone.
In some embodiments of the present application, the difference dead zone can be used to determine whether the first data is changing too little compared to the historical data, e.g., when the difference between the first data and the historical data falls within the difference dead zone, it can be determined that the first data is changing too little compared to the historical data.
The difference, dead zone of difference may be measured by a ratio, which may refer to the absolute difference between the first data and the historical data as compared to the ratio of the historical data, which may be more intuitively, and in particular may be a percentage. If the first data and the history data are numerical data, the absolute difference is, for example, the difference between the first data and the history data or the absolute value of the difference.
S104: determining an absolute difference between the first data and the historical data, and determining a reference difference between the first data and the historical data according to the difference dead zone.
In some embodiments of the present application, the reference difference may be at the critical point inside and outside the difference dead zone. When the difference dead zone includes only one of the difference proportional dead zone and the difference absolute value dead zone, the reference difference may be determined directly using the absolute difference. If the difference dead zone comprises a difference proportion dead zone and a difference absolute value dead zone at the same time, the difference proportion dead zone or the difference absolute value dead zone can be selected according to a certain strategy, and then the reference difference is determined according to the selected difference dead zone; alternatively, another synthetically considered dead zone may be derived by comprehensively considering the differential proportional dead zone and the differential absolute dead zone (e.g., taking the average of the two, etc.) for calculating the reference differential.
S106: and if the absolute difference and the reference difference meet preset conditions, converting the first data into corresponding second data and writing the second data into the relational database.
In some embodiments of the present application, if the absolute difference is greater than the reference difference, it may be considered that the absolute difference falls outside of the difference dead zone, and the first data needs to be updated into the relational database. The conversion in step S106 is mainly to convert the format of the real-time data into the format of the relational data so as to be able to be successfully written into the relational database without affecting the substantial content of the data, in which case the substantial content of the first data and the second data are identical.
One of the preset conditions is listed in the previous paragraph by way of example only, and the present application is not limited to this only preset condition in step S106.
By the method of fig. 1, a part of invalid data can be removed from the data of the real-time database according to the difference dead zone, and the valid data is converted into the relational data, so that the efficiency and the reliability of the subsequent data analysis can be improved, and the data in the real-time database can be utilized more effectively.
In some embodiments of the present application, if the difference dead zone includes a difference absolute value dead zone, obtaining the difference dead zone corresponding to the first data may include: acquiring a corresponding difference absolute value dead zone preset for the first data; or determining a difference absolute value dead zone corresponding to the first data according to the historical data. The former scheme has the advantages that the dead zone of the difference absolute value is relatively fixed, so that processing resources are reduced conveniently, and the latter scheme has the advantages that the dead zone of the difference can be flexibly adjusted according to the actual change trend of data, and has good adaptability.
In some embodiments of the present application, if the difference dead zone includes both a difference proportion dead zone and a difference absolute value dead zone, the proportion difference may be determined according to the difference proportion dead zone and the historical data, and then the reference difference of the first data compared to the historical data may be determined according to the proportion difference and the difference absolute value dead zone. Note that, the ratio difference here refers to an absolute value difference calculated as possible reference according to a ratio specified in advance, but since an absolute value difference calculated as possible reference according to a dead zone of a difference absolute value can also be calculated, which absolute value difference is ultimately used as the reference difference may depend on a predetermined policy, for example, a smaller one of the two may be specified as the reference difference in advance.
In view of the foregoing, some embodiments of the present application provide a detailed flow schematic of the method of fig. 1 in an application scenario, as shown by way of example in fig. 2, where the data collected in the real-time database may be from a nuclear power plant, for example, assuming temperature, pressure, flow, level switches.
The flow of fig. 2 may involve the steps of:
The following difference judgment parameters are set for the measuring points of the real-time data except the liquid level switch: differential proportional dead zone, differential absolute dead zone for temperature, differential absolute dead zone for pressure, differential absolute dead zone for flow, differential absolute dead zone for NULL. In addition, an upper and lower limit comparison mode is set for determining whether the difference proportion dead zone is used as a reference or the difference absolute value dead zone is used as a reference, wherein the difference proportion dead zone is used as a logic value, the difference proportion dead zone is used as a maximum value, the difference absolute value dead zone is used as a minimum value, and the difference proportion dead zone is used as a default.
Based on the above setting, whether the measuring point is a liquid level switch is determined, if not, a difference absolute value dead zone is calculated, if yes, whether a new value (for example, the above first data) is equal to an old value (for example, the above history data of the first data) is compared, if not, the new value is updated to the relational database, and if equal, the new value is not updated.
Calculating a difference absolute value dead zone, comprising:
according to the current measuring point type (temperature, pressure or flow), determining a corresponding difference absolute value dead zone set for the current measuring point type, and if the set difference absolute value dead zone is not empty, then:
directly making [ difference absolute value dead zone ] = [ set difference absolute value dead zone ];
If the dead zone of the set difference absolute value is empty, the formula is modified as follows:
[ difference absolute value dead zone ] = [ old value ] ×1/100 (the percentage values here are exemplary, and other percentage values may be set as well); the following steps are continued.
Calculating the proportion difference, wherein the formula is as follows:
[ proportional difference ] = [ old value ] × [ differential proportional dead zone ];
determining a reference difference according to an upper limit and lower limit comparison mode:
if [ upper and lower limit comparison mode ] is equal to "yes", taking the maximum value of the [ difference absolute value dead zone ] and the [ proportion difference ], and the formula is:
[ reference difference ] = |max ([ difference absolute value dead zone ], [ proportion difference ])|;
if the [ upper and lower limit comparison mode ] is equal to 'no', taking the minimum value of the [ difference absolute value dead zone ] and the [ proportion difference ], wherein the formula is as follows:
[ reference difference ] = |min ([ difference absolute value dead zone ], [ proportion difference ])|;
calculating an absolute difference:
absolute difference = | [ new value ] - [ old value ] |;
judging whether the new value needs to be updated into the relational database or not:
if the absolute difference is larger than the difference, the new value is considered to be valid, and updating is needed, otherwise, the new value can be ignored.
In practical application, converting real-time data into relational data is a continuous process, and the process is continued as long as the business equipment is running, but in consideration of various abnormal situations, the situation that the converting process is interrupted may occur, when the converting process continues, the converting process cannot be acquired from the beginning, so when the abnormality occurs, a time breakpoint needs to be recorded, the next acquisition should be continued from the breakpoint position, and the breakpoint may be the server time of the real-time database.
Because the server time of the real-time database may not be consistent with the server time of the relational database, the present application may use the server time of the real-time database from which the data was last obtained as a breakpoint, rather than using the server time of the relational database.
The adoption of the server time of the real-time database as the breakpoint time also causes a problem that the real-time data actually records the service equipment time, and the time may not be consistent with the server time of the real-time database, so that the server time of the real-time database cannot be taken as the breakpoint, and the time of the service equipment measuring point cannot be taken as the breakpoint, however, each service equipment has own clock, and there may be a time difference or inconsistent with the server time of the real-time database, and the API function cannot determine the time of the service equipment.
To solve this problem, the solution of the present application performs related processing, including: the breakpoint of each measuring point is determined as the time of acquiring the data last time (the time can be based on the recording time of the data), or the time is later, so that when the process is interrupted and restarted, the starting time of acquiring the real-time data can be taken as the recording time of the real-time data acquired last time, and the deadline of the time interval can be taken as the current time of a server of the relational database; if the process is always in progress, the last time the cut-off time is taken as the starting time, so that the server time and the service equipment time of the real-time database can be bypassed, and meanwhile, the acquired data is ensured to be continuous and the data cannot be lost.
Based on the above analysis, the description is continued based on fig. 1,
in some embodiments of the present application, with respect to the flow of fig. 1, it may further perform: acquiring a first time corresponding to the first data, wherein the first time is the recording time of the first data; determining a second time not earlier than the first time; acquiring third data in the real-time database during the period from the second time to a preset deadline later than the second time; and writing data into the relational database according to the third data. The preset deadline is, for example, the current time of the server of the relational database.
Here, it is assumed that after updating the first data (the first data is the data having the latest recording time that has been acquired from the real-time database before the third data is acquired), a breakpoint occurs, and when the breakpoint resumes to acquire data, the data may be continuously acquired backward with reference to the first data, and if the second time is equal to the first time, it is still possible to repeatedly acquire data, and the second time may be slightly later than the first time, for example, 1 second later than the first time. The third data represents real-time data having a recording time later than the first time.
In some embodiments of the present application, if the breakpoint is longer than the current time of the server of the relational database, the time segmentation may attempt to obtain the third data for management, based on which, obtaining the third data in the real-time database during the period from the second time to a preset deadline later than the second time may include: determining the second time as a starting point, and attempting to acquire third data in the real-time database within a set time length after the starting point; if the acquisition is successful, the starting point is redetermined according to the recording time of the latest recorded data acquired at this time, then the acquisition of the third data is continuously attempted according to the redetermined starting point, and the steps are repeatedly executed until the set duration comprises a preset deadline, wherein the redetermined starting point is not earlier than the recording time.
Further, after determining the second time as the starting point and attempting to acquire the third data written in the real-time database within a set period of time after the starting point, the method may further include: if the acquisition fails (possibly due to no data or the occurrence of a new breakpoint), the set time period is increased, and the acquisition of the third data written in the real-time database is retried within the increased set time period from the starting point. Thus, on the one hand, it helps to prevent missing data, and on the other hand, it is also possible to efficiently continue to acquire data backward.
In view of the foregoing, more intuitively, some embodiments of the present application provide a detailed flow chart of a continuous data reading scheme in an application scenario, where, as shown in fig. 3, a recording time of last acquired data (assuming that the first data is the first acquired data) is denoted as t, a second time is denoted as a, a deadline of a current time segment is denoted as B, and an initial set duration is 1 hour.
The flow in fig. 3 includes the following steps:
and acquiring the last data and the recording time t thereof from the real-time database, if the acquisition fails, setting t as a preset default value (2016/12/01 is taken as an example in the figure), if the acquisition is successful, setting A as t+1 seconds, setting B as A+1 hours, taking A as a starting point, and attempting to acquire the data with the recording time in the real-time database in the time interval from A to B.
If the acquisition attempt is successful, the acquired data is stored by an array, difference judgment is carried out on the array circulation to determine whether the data in the data need to be updated into a relational database, and after the array traversing processing is finished, the data acquisition attempt is continued.
If the acquisition attempt fails, the A is inconvenient, the B is increased for 1 hour, the acquisition attempt is continued, and if the current time interval exceeds the current time of the server of the relational database, the B is directly set as the current time of the server of the relational database.
Fig. 4 is a schematic diagram of the scheme of fig. 3 in an application scenario provided in some embodiments of the present application.
In fig. 4, the recording time of the first data (the last value in the time interval) is 0:37, the next time (assuming that the last value in the time interval is recorded as the 2 nd time) the time interval for attempting to acquire the data can be taken (0:37, 1:37), the left end point can take (2:10, 4:10) for example, because the time interval is increased by one hour after the 2 nd time of acquisition failure without the value, the time interval is taken (0:37, 2:37), the latest recording time of the acquired data is 2:10 (the last value in the time interval is marked), the time interval for the 3 rd time of acquisition can be taken (2:10, 3:10), the time interval can be taken (2:10, 4:10) for example, but because the current time of the server of the relational database is 3:40 and is earlier than 4:10, the time interval is taken (2:10, 3:40), that is, the current time of the server of the relational database is cut off to the current time of the server of the relational database.
In some embodiments of the present application, data is obtained from a real-time database to a relational database, and because real-time data tends to be huge in data volume, the efficiency of data processing needs to be considered. In order to solve the efficiency problem, the processing procedures such as reading and updating of data can be realized by adopting multiple threads.
For example, the use of three threads each performs a significant increase in algorithm efficiency, and is described below in conjunction with exemplary codes, which are incorporated by reference only and not limiting of the present application.
Thread one: the basic information of the measuring point for collecting the real-time data of the service equipment is read, and the measuring point can be changed at any time, such as adding one service equipment or removing one service equipment or suspending one service equipment due to maintenance, so that the measuring point needs to be refreshed regularly, available measuring point information is obtained from a real-time database, and real-time data is obtained according to the measuring point change.
The measuring point information list is realized by a data list:
list < PIPoint > lstpoint=new List < PIPoint > (); a list of/(and/or) points;
note that lstppoint is data read and written by multiple threads, and needs to be noted about the problem of process mutual exclusion, and in order to make data conflict, when refreshing a list, the lstppoint can be locked in advance, and the code is shown as follows:
Thread two: and circularly reading real-time data from the real-time database for the measurement point list read by the first thread, processing the data according to the scheme provided before, and temporarily storing the processed data in a data dictionary. In the data dictionary, the values are stored in a queue mode, so that real-time data are inserted from the head of the queue, and are read from the tail of the queue when stored in a relational database, and the code is shown as follows:
dictionary < PIPoint, queue > pidatacollection=newdictionary < PIPoint >, queue > (); a/store the acquired real-time data set;
the algorithm also needs to pay attention to the problem of thread conflict, for lstPIPoint, in order to avoid the conflict, locking is needed, then a new temporary list is established, the later operation is ensured not to generate the conflict, and the code is shown as follows:
thread three: updating real-time data read by line Cheng Er to relational database
And storing the processed data of the second thread in a queue, and taking out the data from the tail of the queue by the third thread and updating the data to the relational database.
This also prevents the occurrence of mutually exclusive conflicts, as indicated below for piDataCollection:
through multithreading, real-time data processing requirements of more than 1000 measuring points can be realized.
Based on the same thought, some embodiments of the present application further provide a device and a medium corresponding to the above system.
Some embodiments of the present application provide an apparatus for retrieving data from a real-time database to a relational database, comprising:
at least one processor; the method comprises the steps of,
a memory communicatively coupled to the at least one processor; wherein,
the memory stores instructions executable by the at least one processor to enable the at least one processor to:
acquiring first data in the real-time database and acquiring historical data corresponding to the first data;
acquiring a difference proportion dead zone and a difference absolute value dead zone corresponding to the first data;
determining an absolute difference between the first data and the historical data, and determining a reference difference between the first data and the historical data based on the difference proportional dead zone and the difference absolute value dead zone;
and if the absolute difference and the reference difference meet preset conditions, converting the first data into corresponding second data and writing the second data into the relational database.
Some embodiments of the present application provide a medium for retrieving data from a real-time database to a relational database, the medium comprising a non-volatile computer storage medium having stored thereon computer-executable instructions configured to:
acquiring first data in the real-time database and acquiring historical data corresponding to the first data;
acquiring a difference proportion dead zone and a difference absolute value dead zone corresponding to the first data;
determining an absolute difference between the first data and the historical data, and determining a reference difference between the first data and the historical data based on the difference proportional dead zone and the difference absolute value dead zone;
and if the absolute difference and the reference difference meet preset conditions, converting the first data into corresponding second data and writing the second data into the relational database.
All embodiments in the application are described in a progressive manner, and identical and similar parts of all embodiments are mutually referred, so that each embodiment mainly describes differences from other embodiments. In particular, for method, apparatus and medium embodiments, the description is relatively simple, as it is substantially similar to system embodiments, with reference to portions of the system embodiments being relevant.
The method, the medium and the system provided by the embodiment of the application are corresponding, so that the method, the device and the medium also have similar beneficial technical effects as the corresponding system.
It will be appreciated by those skilled in the art that embodiments of the invention may be provided as a system, apparatus, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In one typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include volatile memory in a computer-readable medium, random Access Memory (RAM) and/or nonvolatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of computer-readable media.
Computer readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. Computer-readable media, as defined herein, does not include transitory computer-readable media (transmission media), such as modulated data signals and carrier waves.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article or apparatus that comprises the element.
The foregoing is merely exemplary of the present application and is not intended to limit the present application. Various modifications and changes may be made to the present application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. which are within the spirit and principles of the present application are intended to be included within the scope of the claims of the present application.
Claims (5)
1. A method of retrieving data from a real-time database to a relational database, comprising:
acquiring first data in the real-time database and acquiring historical data corresponding to the first data;
acquiring a difference dead zone corresponding to the first data, wherein the difference dead zone comprises a difference proportion dead zone and a difference absolute value dead zone;
if the difference dead zone includes the difference absolute value dead zone, acquiring a difference dead zone corresponding to the first data, including:
determining a difference absolute value dead zone corresponding to the first data according to the historical data;
determining an absolute difference between the first data and the historical data, and determining a reference difference between the first data and the historical data according to the difference dead zone;
if the difference dead zone includes the difference proportional dead zone and the difference absolute value dead zone, determining a reference difference of the first data compared to the historical data according to the difference dead zone, including:
Determining a proportion difference according to the difference proportion dead zone and the historical data;
determining a reference difference of the first data compared with the historical data according to the proportion difference and the difference absolute value dead zone;
if the absolute difference and the reference difference meet preset conditions, converting the first data into corresponding second data and then writing the second data into the relational database;
when a breakpoint occurs after the first data is written into the relational database, and the breakpoint resumes acquiring data, acquiring a first time corresponding to the first data, wherein the first time is the recording time of acquiring the first data last time;
determining a second time that is not earlier than the first time;
acquiring third data in the real-time database during a preset deadline from the second time to later than the second time, wherein the third data comprises the following steps:
determining the second time as a starting point, attempting to acquire third data in the real-time database within a set duration after the starting point, wherein the preset deadline is the current time of a server of the relational database;
if the acquisition is successful, the starting point is redetermined according to the recording time of the latest recorded data acquired at this time, then the acquisition of the third data is continuously attempted according to the redetermined starting point, and the step is repeatedly executed until the set duration contains the preset deadline, wherein the redetermined starting point is not earlier than the recording time;
If the acquisition fails, the set time length is increased, and the acquisition of the third data written in the real-time database is retried in the increased set time length from the starting point;
and writing data into the relational database according to the third data.
2. The method of claim 1, wherein if the difference dead zone comprises the difference absolute value dead zone, obtaining a difference dead zone corresponding to the first data comprises:
and acquiring a corresponding difference absolute value dead zone preset for the first data.
3. The method of claim 1, wherein after the first data is acquired, the first data is data having a latest recording time that has been currently acquired from the real-time database before the third data is acquired.
4. An apparatus for retrieving data from a real-time database to a relational database, comprising:
at least one processor; the method comprises the steps of,
a memory communicatively coupled to the at least one processor; wherein,
the memory stores instructions executable by the at least one processor to enable the at least one processor to:
Acquiring first data in the real-time database and acquiring historical data corresponding to the first data;
acquiring a difference proportion dead zone and a difference absolute value dead zone corresponding to the first data;
if the difference dead zone includes the difference absolute value dead zone, acquiring a difference dead zone corresponding to the first data, including:
determining a difference absolute value dead zone corresponding to the first data according to the historical data;
determining an absolute difference between the first data and the historical data, and determining a reference difference between the first data and the historical data based on the difference proportional dead zone and the difference absolute value dead zone;
if the difference dead zone includes the difference proportional dead zone and the difference absolute value dead zone, determining a reference difference of the first data compared to the historical data according to the difference dead zone, including:
determining a proportion difference according to the difference proportion dead zone and the historical data;
determining a reference difference of the first data compared with the historical data according to the proportion difference and the difference absolute value dead zone;
if the absolute difference and the reference difference meet preset conditions, converting the first data into corresponding second data and then writing the second data into the relational database;
When a breakpoint occurs after the first data is written into the relational database, and the breakpoint resumes acquiring data, acquiring a first time corresponding to the first data, wherein the first time is the recording time of acquiring the first data last time;
determining a second time that is not earlier than the first time;
acquiring third data in the real-time database during a preset deadline from the second time to later than the second time, wherein the third data comprises the following steps:
determining the second time as a starting point, attempting to acquire third data in the real-time database within a set duration after the starting point, wherein the preset deadline is the current time of a server of the relational database;
if the acquisition is successful, the starting point is redetermined according to the recording time of the latest recorded data acquired at this time, then the acquisition of the third data is continuously attempted according to the redetermined starting point, and the step is repeatedly executed until the set duration contains the preset deadline, wherein the redetermined starting point is not earlier than the recording time;
if the acquisition fails, the set time length is increased, and the acquisition of the third data written in the real-time database is retried in the increased set time length from the starting point;
And writing data into the relational database according to the third data.
5. A medium for retrieving data from a real-time database to a relational database, storing computer-executable instructions, the computer-executable instructions configured to:
acquiring first data in the real-time database and acquiring historical data corresponding to the first data;
acquiring a difference proportion dead zone and a difference absolute value dead zone corresponding to the first data;
if the difference dead zone includes the difference absolute value dead zone, acquiring a difference dead zone corresponding to the first data, including:
determining a difference absolute value dead zone corresponding to the first data according to the historical data;
determining an absolute difference between the first data and the historical data, and determining a reference difference between the first data and the historical data based on the difference proportional dead zone and the difference absolute value dead zone;
if the difference dead zone includes the difference proportional dead zone and the difference absolute value dead zone, determining a reference difference of the first data compared to the historical data according to the difference dead zone, including:
Determining a proportion difference according to the difference proportion dead zone and the historical data;
determining a reference difference of the first data compared with the historical data according to the proportion difference and the difference absolute value dead zone;
if the absolute difference and the reference difference meet preset conditions, converting the first data into corresponding second data and then writing the second data into the relational database;
when a breakpoint occurs after the first data is written into the relational database, and the breakpoint resumes acquiring data, acquiring a first time corresponding to the first data, wherein the first time is the recording time of acquiring the first data last time;
determining a second time that is not earlier than the first time;
acquiring third data in the real-time database during a preset deadline from the second time to later than the second time, wherein the third data comprises the following steps:
determining the second time as a starting point, attempting to acquire third data in the real-time database within a set duration after the starting point, wherein the preset deadline is the current time of a server of the relational database;
if the acquisition is successful, the starting point is redetermined according to the recording time of the latest recorded data acquired at this time, then the acquisition of the third data is continuously attempted according to the redetermined starting point, and the step is repeatedly executed until the set duration contains the preset deadline, wherein the redetermined starting point is not earlier than the recording time;
If the acquisition fails, the set time length is increased, and the acquisition of the third data written in the real-time database is retried in the increased set time length from the starting point;
and writing data into the relational database according to the third data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911318305.6A CN111090662B (en) | 2019-12-19 | 2019-12-19 | Method, equipment and medium for acquiring data from real-time database to relational database |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911318305.6A CN111090662B (en) | 2019-12-19 | 2019-12-19 | Method, equipment and medium for acquiring data from real-time database to relational database |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111090662A CN111090662A (en) | 2020-05-01 |
| CN111090662B true CN111090662B (en) | 2024-01-30 |
Family
ID=70396015
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911318305.6A Active CN111090662B (en) | 2019-12-19 | 2019-12-19 | Method, equipment and medium for acquiring data from real-time database to relational database |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111090662B (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104331495A (en) * | 2014-11-19 | 2015-02-04 | 北京国电软通江苏科技有限公司 | Data compressing method |
| CN105425683A (en) * | 2015-12-28 | 2016-03-23 | 北京四方继保自动化股份有限公司 | Analog quantity acquisition method based on stepwise dynamic dead zone |
| CN106886244A (en) * | 2015-12-15 | 2017-06-23 | 上海汽车集团股份有限公司 | Method and gear shifting actuating mechanism for obtaining zero delivery control electric current |
| CN108320480A (en) * | 2018-04-03 | 2018-07-24 | 山东科技大学 | A kind of parameter determination method of dead zone alarm |
| CN108320069A (en) * | 2017-12-20 | 2018-07-24 | 甘肃银光化学工业集团有限公司 | A kind of manufacturing execution system |
| CN108694241A (en) * | 2018-05-14 | 2018-10-23 | 平安科技（深圳）有限公司 | A kind of storage method and equipment of data |
| CN109614393A (en) * | 2018-10-26 | 2019-04-12 | 阿里巴巴集团控股有限公司 | Verification of data method and device |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9740760B2 (en) * | 2014-04-11 | 2017-08-22 | Bank Of America Corporation | Method for tracking differences between production and development databases to perform an automatic synchronization |
-
2019
- 2019-12-19 CN CN201911318305.6A patent/CN111090662B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104331495A (en) * | 2014-11-19 | 2015-02-04 | 北京国电软通江苏科技有限公司 | Data compressing method |
| CN106886244A (en) * | 2015-12-15 | 2017-06-23 | 上海汽车集团股份有限公司 | Method and gear shifting actuating mechanism for obtaining zero delivery control electric current |
| CN105425683A (en) * | 2015-12-28 | 2016-03-23 | 北京四方继保自动化股份有限公司 | Analog quantity acquisition method based on stepwise dynamic dead zone |
| CN108320069A (en) * | 2017-12-20 | 2018-07-24 | 甘肃银光化学工业集团有限公司 | A kind of manufacturing execution system |
| CN108320480A (en) * | 2018-04-03 | 2018-07-24 | 山东科技大学 | A kind of parameter determination method of dead zone alarm |
| CN108694241A (en) * | 2018-05-14 | 2018-10-23 | 平安科技（深圳）有限公司 | A kind of storage method and equipment of data |
| CN109614393A (en) * | 2018-10-26 | 2019-04-12 | 阿里巴巴集团控股有限公司 | Verification of data method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111090662A (en) | 2020-05-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110661659B (en) | Alarm method, device and system and electronic equipment | |
| CN112631957B (en) | Data acquisition method and device, electronic equipment and storage medium | |
| CN113641526B (en) | Alarm root cause positioning method and device, electronic equipment and computer storage medium | |
| CN112926636A (en) | Method and device for detecting abnormal temperature of traction converter cabinet body | |
| CN111090662B (en) | Method, equipment and medium for acquiring data from real-time database to relational database | |
| CN106294470B (en) | Method for reading real-time incremental log information based on split log | |
| CN113806155B (en) | Industrial equipment management method and system based on Internet of things | |
| CN114064402A (en) | Server system monitoring method | |
| CN114429249A (en) | Method, system, equipment and storage medium for predicting service life of steel pipe bundle production equipment | |
| CN112583944A (en) | Processing method and device for updating domain name certificate | |
| CN109284193A (en) | A kind of distributed data processing method and server based on multithreading | |
| CN111160874B (en) | Method and device for judging abnormal well entry and exit based on coal mine personnel networking system | |
| CN113672167B (en) | Data consistency verification method, device and equipment for distributed storage system | |
| CN115114275A (en) | Data acquisition method, device and medium | |
| Huang et al. | Research on accident inversion and analysis method of the oil and gas pipeline SCADA system | |
| CN112184487A (en) | Method and device for predicting power supply index | |
| CN112035322A (en) | JVM monitoring method and device | |
| CN111858075A (en) | Deadlock detection method, system, equipment and readable storage medium | |
| US20200110642A1 (en) | Funnel locking for normal rcu grace period requests | |
| US8370405B2 (en) | Variable-length record, corruption recovery apparatus, system, and method | |
| CN117743265A (en) | Data reading method and device, storage medium and electronic equipment | |
| CN116069526B (en) | Data access method and computer equipment based on lock-free message pool | |
| CN116258403B (en) | Big data-based water quality monitoring and early warning method and system | |
| CN109490788B (en) | Method and device for predicting capacity of storage battery pack | |
| CN114256977A (en) | Intelligent inspection method and system for intelligent substation relay protection device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |