CN116541388A - Data cleaning method and system for ultrasonic metering instrument - Google Patents
Data cleaning method and system for ultrasonic metering instrument Download PDFInfo
- Publication number
- CN116541388A CN116541388A CN202310592408.1A CN202310592408A CN116541388A CN 116541388 A CN116541388 A CN 116541388A CN 202310592408 A CN202310592408 A CN 202310592408A CN 116541388 A CN116541388 A CN 116541388A
- Authority
- CN
- China
- Prior art keywords
- data
- flashback
- target
- database
- request
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 68
- 238000004140 cleaning Methods 0.000 title abstract description 4
- 238000003860 storage Methods 0.000 claims description 98
- 230000007246 mechanism Effects 0.000 claims description 13
- 238000002360 preparation method Methods 0.000 claims description 13
- 238000004590 computer program Methods 0.000 claims description 12
- 238000002955 isolation Methods 0.000 claims description 11
- 230000003993 interaction Effects 0.000 abstract description 7
- 230000002159 abnormal effect Effects 0.000 abstract 6
- 206010016754 Flashback Diseases 0.000 description 225
- 230000004048 modification Effects 0.000 description 14
- 238000012986 modification Methods 0.000 description 14
- 230000037430 deletion Effects 0.000 description 10
- 238000012217 deletion Methods 0.000 description 10
- 238000012545 processing Methods 0.000 description 10
- 230000008569 process Effects 0.000 description 9
- 238000004891 communication Methods 0.000 description 5
- 230000006870 function Effects 0.000 description 5
- 238000011084 recovery Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 238000007781 pre-processing Methods 0.000 description 4
- 230000008859 change Effects 0.000 description 3
- 230000002085 persistent effect Effects 0.000 description 3
- 230000001960 triggered effect Effects 0.000 description 3
- 238000012512 characterization method Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000012805 post-processing Methods 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000005520 cutting process Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000004064 recycling Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 230000000717 retained effect Effects 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01F—MEASURING VOLUME, VOLUME FLOW, MASS FLOW OR LIQUID LEVEL; METERING BY VOLUME
- G01F1/00—Measuring the volume flow or mass flow of fluid or fluent solid material wherein the fluid passes through a meter in a continuous flow
- G01F1/66—Measuring the volume flow or mass flow of fluid or fluent solid material wherein the fluid passes through a meter in a continuous flow by measuring frequency, phase shift or propagation time of electromagnetic or other waves, e.g. using ultrasonic flowmeters
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/10—Pre-processing; Data cleansing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/243—Classification techniques relating to the number of classes
- G06F18/2433—Single-class perspective, e.g. one-against-all classification; Novelty detection; Outlier detection
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Quality & Reliability (AREA)
- Fluid Mechanics (AREA)
- Electromagnetism (AREA)
- Measurement Of Mechanical Vibrations Or Ultrasonic Waves (AREA)
Abstract
According to the data cleaning method and system for the ultrasonic metering instrument, based on the data node description attribute of the target data segment and the data node description attribute of the abnormal range in the target data segment, the data node description attribute of the abnormal range in the determined target data segment is information described by the data positioning of the data node, so that the data positioning of the abnormal range can be determined very accurately, and ultrasonic interaction data obtained by cleaning the data node description attribute of the abnormal range is more accurate and reliable; in addition, because the description attribute of the data node without the abnormal range in the target data segment is determined firstly, and then the description attribute of the data node without the abnormal range is cleaned, more accurate and reliable ultrasonic data can be obtained, and the accuracy and reliability of the subsequent ultrasonic metering instrument metering are improved.
Description
Technical Field
The present disclosure relates to the field of computer technologies, and in particular, to a data flashback method, apparatus, electronic device, and storage medium.
Background
In the field of databases, MVCC (Multi-version concurrency control ) can be generally used to manage the version of stored data, when a version of a certain stored data is updated (for example, changed, deleted data, etc.), no real data deletion occurs, only one new version of data can be added, and historical data can be retained, so that the historical data in the database can be read.
However, in the related art, the MVCC cannot be used to modify the data that is subjected to the error modification or deletion operation in the database. Under the condition that the user carries out error modification or deletion operation on the data in the database, the data which is subjected to error modification or deletion in the database can be modified only in a backup recovery or manual modification mode, so that the modification difficulty and modification cost of the data which is subjected to error modification or deletion are increased, and the recovery difficulty and recovery cost of historical data are increased.
Disclosure of Invention
The embodiment of the application provides a data flashing method, which aims to solve the problems of high difficulty and high cost of modifying data which is mistakenly modified or deleted in a database and high recovery difficulty and recovery cost of historical data in the database in the related technology.
Correspondingly, the embodiment of the application also provides a data flashing device, electronic equipment and a storage medium, which are used for guaranteeing the implementation and application of the method.
In one aspect, an embodiment of the present application provides a data flashback method, including:
receiving a data flashback request; the data flashback request comprises a flashback timestamp;
determining target data which is pre-locked in a database and corresponds to the data flashback request according to the data flashback request; determining whether reference data corresponding to the flashback timestamp exists in the database;
under the condition that the reference data exists in the database, performing first coverage operation on the target data based on a comparison result of the target data and the reference data;
the database comprises a first pre-written structured query statement which executes a first overlay operation.
In another aspect, an embodiment of the present application provides a data flashback apparatus, including:
the data flashback request receiving module is used for receiving a data flashback request; the data flashback request comprises a flashback timestamp;
the flashback operation data determining module is used for determining target data which is locked in advance in the database and corresponds to the data flashback request according to the data flashback request; determining whether reference data corresponding to the flashback time stamp exists in the database;
The flashback operation execution module is used for carrying out a first covering operation on the target data based on the comparison result of the target data and the reference data under the condition that the reference data exist in the database;
the database comprises a first pre-written structured query statement which executes a first overlay operation.
In another aspect, an embodiment of the present application provides an electronic device, including a processor and a memory, where the processor and the memory are connected to each other;
the memory is used for storing a computer program;
the processor is configured to execute the data flashback method provided by the embodiment of the application when the computer program is called.
In another aspect, embodiments of the present application provide a computer readable storage medium storing a computer program that is executed by a processor to implement the data flashback method provided by embodiments of the present application.
In the embodiment of the application, through receiving a data flashback request, determining target data which is locked in advance in a database and corresponds to the data flashback request according to the data flashback request; determining whether reference data corresponding to the flashback timestamp exists in the database according to the flashback timestamp in the data flashback request; and under the condition that the reference data exists in the database, referring to a first structured query statement which is written in advance and used for executing the first overlay operation in the database, and carrying out the first overlay operation on the target data based on the comparison result of the target data and the reference data. The method and the device can modify the target data pre-locked in the database by executing the first structured query statement, have simple operation, and reduce the difficulty and cost of modifying the data with error modification or error deletion operation in the database.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a schematic flow chart of a data flashback method according to an embodiment of the present application;
fig. 2 illustrates an application scenario schematic diagram of a data flashback method provided in an embodiment of the present application;
FIG. 3 is a schematic flow chart of a data flashback method according to an embodiment of the present application;
fig. 4 shows a schematic structural diagram of a data flashback device according to an embodiment of the present application;
fig. 5 shows a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
The following description of the embodiments of the present application will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are only some, but not all, of the embodiments of the present application. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are within the scope of the present disclosure.
As described above, in the database field, when data that has been erroneously modified or deleted in the database is modified by means of backup restoration or manual modification, the difficulty and cost of modifying the data that has been erroneously modified or deleted are increased.
And in the related art, when performing data flashback operation on data in an Oracle database, the implementation is usually based on the undo data, and additional opening of supplementary log is required. In performing data flashback operations on data in databases in the Polar db, snapshot backup and redox log implementations based on Polar FS (file system) are common. Specifically, the polar db recreates a new cluster through the snapshot and restores the data to the designated point in time according to the redox log. As can be seen, each data flashback operation is relatively complex and the operation difficulty is relatively high.
In view of this, the embodiment of the application provides a data flashback method, which can be applied to a data flashback device, and the embodiment of the application does not specifically limit the data flashback device. The data flashing device can be a server or a terminal independent of a database, and can also take a component with corresponding function as the data flashing device based on the function of each component in the database service.
The data flashback method provided by the embodiment of the application can be applied to databases including but not limited to TiDB databases and the like. In a specific execution process, the data flashback method provided by the embodiment of the application can be implemented by combining a processing engine (i.e. a component for performing data calculation operations), a storage engine (i.e. a component for performing data storage), a scheduling engine (i.e. a component for performing storage meta-information and scheduling load balancing) and other components in a database. In particular, the processing engine may be configured as the data flashback device described above.
Specifically, the data flashback method provided by the embodiment of the application can be applied to application scenes such as error modification, error deletion and the like of current data.
As shown in fig. 1, the data flashback method includes:
step S110: receiving a data flashback request; the data flashback request includes a flashback timestamp therein.
Data flashback: the method is mainly based on historical data and is used for modifying existing data (namely current data) in a database.
In this implementation, a data flashback request may be used to trigger a request to perform the data flashback method, and by responding to the data flashback request, a data flashback device may be triggered to perform a data flashback operation.
Step S120: determining target data which is pre-locked in a database and corresponds to the data flashback request according to the data flashback request; and determining whether reference data corresponding to the flashback timestamp exists in the database.
The target data may be real-time data in a database in response to a data flashback request. Or real-time data in the database when SQL is written in the database. The real-time data in the database can also be used for specifically performing the data flashback operation. It may be determined according to actual situations, and this is not particularly limited in the embodiment of the present application.
In the embodiment of the application, in order to avoid influencing the target data in the execution process of the data flashback operation, the target data needs to be locked.
The reference data may be real-time data stored in a database corresponding to the flashback timestamp. Specifically, when the reference data is determined based on the flashback time stamp, if the real-time data exists in the database at the flashback time stamp, the reference data is determined to exist, and the real-time data is used as the reference data; and if the real-time data does not exist in the database at the flashback time stamp, determining that the reference data does not exist.
Wherein, the timestamp corresponding to the target data is later than the timestamp corresponding to the reference time. Step S130: and performing a first overlay operation on the target data based on a comparison result of the target data and the reference data in the case that the reference data is determined to exist in the database.
The database comprises a first pre-written structured query statement which executes a first overlay operation.
In embodiments of the present application, operations that perform the entire data flashback method may be referred to as data flashback operations, and data required to perform the data flashback operations may be referred to as flashback operation data (e.g., flashback operation data may include reference data and target data). When the data flashback method is executed in a structured query statement manner, the corresponding structured query statement may be referred to as SQL that performs the data flashback operation.
It is to be appreciated that the data flashback operation can include, but is not limited to, a first override operation. Naturally, SQL that performs data flashback operations may include a first structured query statement.
Wherein a flashback timestamp may be included in the first structured query statement.
It can be understood that, in the actual application process, the data flashback device can be triggered to execute the data flashback method provided by the embodiment of the application by writing the SQL for executing the data flashback operation in the database. The SQL for executing the data flashback operation can also be written in the database in advance, and the data flashback device is triggered to execute the data flashback method provided by the embodiment of the application through the update operation of the flashback timestamp set in the SQL for executing the data flashback operation. The embodiment of the application does not limit a specific trigger mechanism.
It should be noted that, before the triggering data flashback device executes the data flashback method provided in the embodiment of the application based on the SQL for executing the data flashback operation, it may also be determined whether the data flashback device satisfies an execution condition of the SQL for executing the data flashback operation. Specifically, the user authority written in the SQL for executing the data flashback operation can be authenticated, whether the flashback timestamp is in a time period corresponding to a preset time interval corresponding to a preset garbage collection mechanism or not is determined, and the like.
The user list capable of being written into the SQL for executing the data flashback operation can be preset, and after the written SQL for executing the data flashback operation is received, whether the current user written into the SQL for executing the data flashback operation is in the preset user list is determined. Under the condition that the current user is in a preset user list and the flashback timestamp is in a time period corresponding to a preset time interval corresponding to a preset garbage collection mechanism, determining that the written SQL for executing the data flashback operation can be executed, namely, satisfying the execution condition of the SQL for executing the data flashback operation.
Wherein, the garbage recycling mechanism is Garbage Collection, which is called GC for short. By starting a garbage collection mechanism for the database, the data with longer interval time between the corresponding time stamp and the current time stamp in the database can be thoroughly cleared. For example, the preset time interval corresponding to the garbage collection mechanism may be set to 30 minutes, that is, in the case that the garbage collection mechanism is set to retain the data of the past 30 minutes, the data of the database with the interval time between the corresponding time stamp and the current time stamp greater than 30 minutes may be deleted. And a time period corresponding to the preset time interval, namely, the current time stamp is taken as a time end point, the preset time interval is traced forward, and after the time start point is determined, the time period from the time start point to the time end point is reserved.
When the coverage operation is performed on the B data based on the A data, the A data can be used for covering the B data in the storage space corresponding to the B data, but the B data is not deleted, so that the effect of hiding the B data and exposing the A data in the storage space corresponding to the B data is achieved.
Specifically, when the first overlay operation is performed on the target data based on the comparison result of the target data and the reference data, the first overlay operation on the target data may be implemented by overlaying the target data with the reference data, replacing the target data with the reference data, modifying the target data based on the reference data, and the like. The current data can be modified through the historical data.
Wherein, in the second coverage operation on the target data based on the comparison result of the target data and the reference data, the following manner may be specifically implemented:
under the condition that the target data and the reference data are the same, the target data are reserved;
in the case where the target data and the reference data are different, a first overlay operation is performed on the target data based on the reference data.
And under the condition that the target data and the reference data are the same, the storage space where the characterization target data are located has no change of the storage data between the flashback time stamp and the target time stamp corresponding to the target data. By reserving the target data, that is, not processing the data in the storage space where the target data is located (that is, the target data), the available storage space of the storage space where the target data is located can be further saved.
And under the condition that the target data and the reference data are different, the storage space in which the characterization target data are located has the change of the storage data between the flashback time stamp and the target time stamp corresponding to the target data. And performing a first covering operation on the target data based on the reference data, namely covering the target data by adopting the reference data in a storage space where the target data is located. The modification operation of the target data is realized, namely the data flashback processing of the target data is effectively realized.
In the embodiment of the application, through receiving a data flashback request, determining target data which is locked in advance in a database and corresponds to the data flashback request according to the data flashback request; determining whether reference data corresponding to the flashback timestamp exists in the database according to the flashback timestamp in the data flashback request; and under the condition that the reference data exists in the database, referring to a first structured query statement which is written in advance and used for executing the first overlay operation in the database, and carrying out the first overlay operation on the target data based on the comparison result of the target data and the reference data. The method and the device can modify the target data pre-locked in the database by executing the first structured query statement, have simple operation, and reduce the difficulty and cost of modifying the data with error modification or error deletion operation in the database.
Optionally, the method may further include:
performing a second overlay operation on the target data based on the flush instruction if it is determined that the reference data does not exist in the database;
the database may further include a pre-written second structured query statement that performs a second overlay operation.
It should be noted that, the above data flashback operation may further include a second override operation. Naturally, the SQL used to perform the data flashback operation may include a first structured query statement and a second structured query statement.
The flush instruction may be a DELETE instruction. The flush instruction may be preset prior to performing the data flashback method.
When the second overlay operation is performed on the target data based on the flush instruction, the flush instruction may be written in the storage space where the target data is located, so that the storage space where the target data is located is "empty", but in fact "hidden" target data is also stored in the storage space where the target data is located.
In the database, different storage areas can be set, the different storage areas are associated with the server, and each storage area is used for storing interaction data between the corresponding server and the client served by the server. Specifically, each storage area may be managed by a storage engine in a database.
Each server may be associated with an actual physical area, and one or more servers corresponding to one physical area may be set. For example, five sub-areas are included in one area, each sub-area may correspond to one server, each server corresponds to one storage area, that is, one storage area may store interaction data between the server corresponding to the storage area and each client in the sub-area corresponding to the server. As shown in fig. 2, the storage area a may store interaction data between the server a and the clients A1', A2', and A3 '.
It is contemplated that continued servicing of the corresponding client or external component by the database during execution of the data flashback operation may cause the database to generate new data, which may affect the processing of the data flashback operation. In the embodiment of the present application, the following optional implementations are also provided:
the method may further include, prior to receiving the data flashback request:
receiving a data flashback preparation request, and performing data isolation operation on the database; and
And locking the target storage area in the database.
The data flashback operation may also include the data isolation operation and an operation to lock a target storage area in the database.
In the embodiment of the application, after receiving the SQL for executing the data flashback operation, the data flashback operation may be implemented in stages by executing the SQL for executing the data flashback operation, for example, a data flashback preparation operation stage (hereinafter referred to as a first stage) and a data flashback execution operation stage (hereinafter referred to as a second stage) may be set, where the data flashback execution operation may be, for example, the above-mentioned operation of determining the reference data, the operation of determining the target data, the first overlay operation, and the second overlay operation.
Specifically, the data flashback preparation operation may be performed by designating a target storage area associated with the data flashback operation in the SQL for performing the data flashback operation, and by executing the SQL for performing the data flashback operation, generating a different data processing request based on each operation phase, for example, when the data flashback preparation operation phase is performed, to perform the data flashback preparation operation based on the data flashback preparation request; in executing the data flashback execution operation phase, a data flashback execution request (i.e., the data flashback request described above) is generated to execute the data flashback execution operation based on the data flashback preparation request.
In the stage of data flashback preparation operation, a data flashback preparation request can be sent to the storage engine, so that the storage engine executes the data flashback preparation operation on the target storage area; in the data flashback execution operation stage, a data flashback request is sent to the storage engine, so that the storage engine executes the data flashback execution operation on the target storage area.
The consistency of the data flashback transaction can be ensured by realizing the data flashback operation in stages.
When the data isolation operation is performed on the database, all the data interaction relations with the database can be cut off.
When the target storage area is locked, the data in the target storage area can be reserved, and new data is prevented from being stored in the target storage area.
In the embodiment of the application, the data isolation operation is performed on the database, and the target storage area in the database is locked, so that the occurrence of interference to the execution process of the data flashback operation due to the fact that new data are generated by the database in the execution process of the data flashback operation due to an external component can be avoided.
Optionally, the data isolation operation on the database may include at least one of the following:
Closing a garbage collection mechanism for the database;
setting read-write permission for the database as read-only permission;
and closing the connection relation between the database and any external component.
In the implementation manner, by closing the garbage collection mechanism for the database, erroneous cleaning of the flashback operation data can be avoided under the condition that the data flashback operation time is longer than the preset time interval corresponding to the garbage collection mechanism.
The read-write permission of the database may be set by a system variable specific to the database, where the read-write permission may include read-only permission, read-write permission (i.e., both read and write). For example, when the database is a TiDB database, in the case where the attribute value of the system variable tidb_super_read_only is set to "YES", the read-write authority of the TiDB database may be set to read-only authority; in the case where the attribute value of the system variable tidb_super_read_only is set to "NO", the read-write authority of the TiDB database may be set to the read-write authority.
When the connection relation between the database and any external component is closed, the scheduling engine can specifically acquire a time stamp, and take the acquired time stamp as a startTS (start time stamp) of the data flashing operation, and based on the startTS, writing a related connection relation cutting instruction into the database to cut off the connection relation between the database and any external component.
It will be appreciated that after the data flashback operation is completed, the data isolation operations may also be resumed to restore the relevant services of the database. Specifically, the garbage collection mechanism for the database may be restarted; setting the read-write permission for the database as the read-write permission; and reconnecting the connection relation between the database and any external component.
In the embodiment of the application, the data isolation operation is performed on the database, so that the interference of external data on the data flashback operation in the data flashback operation process can be avoided.
Optionally, the locking the target storage area in the database may include:
determining a target line lock corresponding to the target storage area, and scanning and rollback operation the target line lock;
and determining the user key with the longest time span between the storage time stamp and the flashback time stamp in the target storage area, and locking the determined user key.
The target row Lock may be a row Lock in an MVCC-based implemented data table, i.e., MVCC Lock.
When rollback operation is performed on the scanned target line lock, when the version of the target line lock changes, the information of the last version of the target line lock can be recorded into a log corresponding to the target line lock, namely the current version and the information of the last version of each target line lock are obtained.
During the rollback operation on the target line lock, data information associated with the flashback timestamp in the target storage area may be further determined.
It is to be appreciated that embodiments herein may also scan all MVCC-based locks and perform rollback operations.
In the target storage area, the interaction data of the server and the client can be stored in a key-value storage mode. The key may correspond to each client identifier, and the key value is interaction data corresponding to the corresponding client.
The user key with the longest time span between the storage time stamp and the flashback time stamp in the target storage area, i.e., the user key newly stored in the target storage area.
When locking the determined user key, a storage area locking instruction may be written in the target storage area based on startTS, and the determined user key may be locked.
By locking the determined user key, writing of updated data in the target storage area can be avoided.
By scanning and rollback operations on the target row lock and locking the determined user key, the target storage area can be locked from two aspects of a data table and user data, operations such as reading, writing, scheduling and the like on the database are blocked on the granularity of the storage area, and interference of external data on the data flashback operation is avoided in the data flashback operation process.
It can be appreciated that after the data flashback operation is completed, the target storage area that can be locked is also subjected to an unlocking operation.
Optionally, determining target data, which is pre-locked in the database and corresponds to the data flashback request, may include:
determining a target user key with a changed key value in a target storage area in a time period from a flashback time stamp to a target time stamp; the target timestamp indicates a timestamp for locking the determined user key;
determining a key value of the target user key corresponding to the flashback timestamp as target data;
the determining whether the reference data corresponding to the flashback timestamp exists in the database may include:
determining a key value of the target user key corresponding to the target timestamp as reference data; .
And determining whether each key value in the target storage area changes in the time period from the flashback time stamp to the target time stamp, and determining the user key with the changed key value as the target user key. And further determining the key value of the target user key corresponding to the target timestamp as reference data, and determining the key value of the target user key corresponding to the flashback timestamp as target data, so that flashback operation data corresponding to the data flashback request can be accurately determined.
Optionally, after performing the data flashback operation on the target data, the method further includes:
updating a data table corresponding to the target storage area;
and generating data table information under the condition that version information corresponding to the updated data table is detected.
As described above, when the target storage area is locked, the data table and the user data associated with the target storage area may be specifically locked.
When updating the data table (schema) corresponding to the target storage area, the version (schema version) of the data table may be specifically updated.
By further detecting version information of the updated data table, the data table information is generated under the condition that the version information corresponding to the updated data table is detected, whether the data flashback operation is successful or not can be further verified, and synchronous updating of the data table is achieved.
In order to more clearly illustrate the data flashback method provided in the embodiments of the present application, the following takes a database as a TiDB database as an example, and the data flashback method is described in detail with reference to the above embodiments and fig. 3:
first, description is made of the TiDB database:
the TiDB database is an open source distributed HTAP database. TiDB database services (i.e., tiDB clusters) can be formed by 3 basic components of a processing engine TiDB Server, a storage engine TiKV Server, and a scheduling engine PD (i.e., placement Driver, PD for short).
Wherein the TiDB Server may be used to perform computing operations associated with a TiDB database, which may also be referred to as a TiDB computing node. The PD Server may be used to store meta information of the TiDB cluster and schedule load balancing, which may also be referred to as a TiDB management node. TiKV Server may be used to store data, and may also be referred to as a TIDB storage node.
In combination with the specific functions of each node in the TiDB database service, in the embodiment of the application, when data flashback operation is performed on data in the TiDB database, the TiDB Server can be used as a data flashback device.
The specific implementation data flashback method comprises the following steps: step 1: the TiDB Server receives the flashback SQL and determines whether the execution condition of the flashback SQL is met.
Step 2: when the TiDB Server determines that the execution condition of the flash back SQL is satisfied, the GC is closed by executing the flash back SQL, and the TiDB Server is set to a read-only mode by setting the attribute value of the system variable tidb_super_read_only to "YES".
Step 3: the TiDB Server acquires the startTS through the PD, and persists the startTS, and disconnects the TiDB database from the external component.
Step 4: the TiDB Server sends flashback prepare requests (data flashback preparation requests) based on RPC implementations to the TiKV Server. After the TiKV Server receives the flashback prepare request, read, write and dispatch operations to the designated regions (i.e., the target storage areas) are blocked with region granularity, so that the designated regions enter a flashback state. Specifically:
a. Scanning all MVCC locks corresponding to TiKV, and performing rollback;
TiKV Server selects the latest user key in the designated regions, and locks the latest user key based on the persistent startTS.
Step 5: the TiDB Server acquires the commit TS (namely, the timestamp after the locking operation is completed) through the PD, persists the commit TS, and sends a flash back request (data flashback request) based on RPC implementation to the TiKV Server. After receiving the flash back request, the TiKV Server scans each key value pair in the designated regions, determines the target user key with version change in each key value pair, determines the target data of the target user key corresponding to the persistent commit, determines whether reference data corresponding to the flash time stamp exists in each target user key, and executes data flash operation based on the target data and the reference data.
Wherein, in the case that the reference data corresponding to the flashback timestamp does not exist in the target user key, writing the DELETE instruction into the key value corresponding to the target user key;
in the case that the reference data corresponding to the flashback timestamp exists in the target user key, and the target data corresponding to the persistent commit is different from the reference data, the target data is covered based on the commit and the reference data corresponding to the target user key;
In the case that the reference data corresponding to the flashback timestamp exists in the target user key, and the target data corresponding to the commit is the same as the reference data, the target data corresponding to the target user key is not processed.
Step 6: the TiDB Server updates the schema version, and regenerates the schema information after the TiDB Server detects the updated schema version, thereby completing the flash back of the schema.
Step 7: the TiDB Server re-opens the GC and sets the TiDB Server to read/write mode by setting the attribute value of the system variable tidb_super_read_only to "NO".
Step 8: the data flashback operation is ended and a prompt result is returned to the client, such as "TiDB database can be re-enabled".
By the data flashback method, flashback time stamp can be directly appointed by an SQL statement in the TiDB database, flashback operation of data and data tables is realized, components outside the TiDB database such as logs and backup data are not relied on, learning cost is low, and processing speed is high.
In addition, in the data flashing method, although the data flashing operation can be realized in a two-stage submitting mode, the data flashing operation is specifically performed in a data coverage mode, historical data in a target storage area is not destroyed, only a new piece of data is written in the target storage area, the time stamps of the data writing are consistent, when multiple flashbacks are needed, the data flashing operation can be performed again based on the time stamps corresponding to the data in each storage area, the multiple flashbacks cannot affect each other, the data can be repeatedly flashed on a time axis, the consistency of transactions is ensured, and the ecological tool is easier to dock.
Based on the same principle as the data flashback method provided by the embodiment of the application, the embodiment of the application also provides a data flashback device. As shown in fig. 4, the apparatus 40 includes:
a data flashback request receiving module 401, configured to receive a data flashback request; the data flashback request comprises a flashback timestamp;
a flashback operation data determining module 402, configured to determine, according to the data flashback request, target data locked in advance in the database and corresponding to the data flashback request; determining whether reference data corresponding to the flashback time stamp exists in the database;
a flashback operation execution module 403, configured to, in a case where it is determined that the reference data exists in the database, perform a first overlay operation on the target data based on a comparison result of the target data and the reference data;
the database comprises a first pre-written structured query statement which executes a first overlay operation.
Optionally, the flashback operation execution module 403 may be further configured to:
performing a second overlay operation on the target data based on the flush instruction if it is determined that the reference data does not exist in the database;
and the database also comprises a pre-written second structured query statement for executing the second overlay operation.
Optionally, the apparatus may further include a preprocessing module, where the preprocessing module may be configured to:
before receiving the data flashback request, receiving a data flashback preparation request, and performing data isolation operation on the database; and
and locking the target storage area in the database.
Optionally, the preprocessing module may be configured to, when locking a target storage area in the database:
determining a target line lock corresponding to the target storage area, and scanning and rollback operation the target line lock;
and determining the user key with the longest time span between the storage time stamp and the flashback time stamp in the target storage area, and locking the determined user key.
Optionally, the flashback operation data determining module 402, when determining target data, which is pre-locked in the database and corresponds to the data flashback request, may be configured to:
determining a target user key with a changed key value in a target storage area in a time period from a flashback time stamp to a target time stamp; the target timestamp indicates a timestamp for locking the determined user key;
determining a key value of the target user key corresponding to the flashback timestamp as target data;
The flashback operation data determining module 402, when determining whether the reference data corresponding to the flashback timestamp exists in the database, may be configured to:
the key value of the target user key corresponding to the target timestamp is determined as the reference data.
Optionally, the preprocessing module may be configured to perform at least one of the following operations when performing a data isolation operation on a database:
closing a garbage collection mechanism for the database;
setting read-write permission for the database as read-only permission;
and closing the connection relation between the database and any external component.
Optionally, the apparatus may further include a post-processing module, where the post-processing module may be configured to:
after executing data flashback operation on the target data, updating a data table corresponding to the target storage area;
and generating data table information under the condition that version information corresponding to the updated data table is detected.
The apparatus of the embodiments of the present application may perform the method provided by the embodiments of the present application, and implementation principles of the method are similar, and actions performed by each module in the apparatus of each embodiment of the present application correspond to steps in the method of each embodiment of the present application, and detailed functional descriptions of each module of the apparatus may be referred to in the corresponding method shown in the foregoing, which is not repeated herein.
In the embodiment of the application, through receiving a data flashback request, determining target data which is locked in advance in a database and corresponds to the data flashback request according to the data flashback request; determining whether reference data corresponding to the flashback timestamp exists in the database according to the flashback timestamp in the data flashback request; and under the condition that the reference data exists in the database, referring to a first structured query statement which is written in advance and used for executing the first overlay operation in the database, and carrying out the first overlay operation on the target data based on the comparison result of the target data and the reference data. The method and the device can modify the target data pre-locked in the database by executing the first structured query statement, have simple operation, and reduce the difficulty and cost of modifying the data with error modification or error deletion operation in the database.
Based on the same principle as the data flashing method and apparatus provided in the embodiments of the present application, an electronic device (such as a server) is also provided in the embodiments of the present application, where the electronic device may include a memory, a processor, and a computer program stored on the memory, where the processor executes the computer program to implement:
Receiving a data flashback request; the data flashback request comprises a flashback timestamp;
determining target data which is pre-locked in a database and corresponds to the data flashback request according to the data flashback request; determining whether reference data corresponding to the flashback timestamp exists in the database;
under the condition that the reference data exists in the database, performing first coverage operation on the target data based on a comparison result of the target data and the reference data;
the database comprises a first pre-written structured query statement which executes a first overlay operation.
In the embodiment of the application, through receiving a data flashback request, determining target data which is locked in advance in a database and corresponds to the data flashback request according to the data flashback request; determining whether reference data corresponding to the flashback timestamp exists in the database according to the flashback timestamp in the data flashback request; and under the condition that the reference data exists in the database, referring to a first structured query statement which is written in advance and used for executing the first overlay operation in the database, and carrying out the first overlay operation on the target data based on the comparison result of the target data and the reference data. The method and the device can modify the target data pre-locked in the database by executing the first structured query statement, have simple operation, and reduce the difficulty and cost of modifying the data with error modification or error deletion operation in the database.
Referring to fig. 5, fig. 5 shows a schematic structural diagram of an electronic device according to an embodiment of the present application. As shown in fig. 5, the electronic device 500 in the present embodiment may include: processor 501, network interface 504 and memory 505, and in addition, electronic device 500 described above may further include: an object interface 503, and at least one communication bus 502. Wherein a communication bus 502 is used to enable connected communications between these components. The object interface 503 may include a Display screen (Display) and a Keyboard (Keyboard), and the optional object interface 503 may further include a standard wired interface and a standard wireless interface. The network interface 504 may optionally include a standard wired interface, a wireless interface (e.g., WI-FI interface). The memory 505 may be a high-speed RAM memory or a non-volatile memory (NVM), such as at least one disk memory. The memory 505 may also optionally be at least one storage device located remotely from the processor 501. As shown in fig. 5, an operating system, a network communication module, an object interface module, and a device control application may be included in the memory 505, which is one type of computer-readable storage medium.
In the electronic device 500 shown in fig. 5, the network interface 504 may provide network communication functions; while object interface 503 is primarily an interface for providing input to an object; and the processor 501 may be configured to invoke a device control application stored in the memory 505 to implement:
In some possible embodiments, the processor 501 is configured to:
it should be appreciated that in some possible embodiments, the above processor 501 may be a central processing unit (central processing unit, CPU), which may also be other general purpose processors, digital signal processors (digital signal processor, DSP), application specific integrated circuits (application specific integrated circuit, ASIC), off-the-shelf programmable gate arrays (field-programmable gate array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The memory may include read only memory and random access memory and provide instructions and data to the processor. A portion of the memory may also include non-volatile random access memory. For example, the memory may also store information of the device type.
In a specific implementation, the electronic device 500 may execute, through each functional module built in the electronic device, an implementation manner provided by each step in fig. 1, and specifically, the implementation manner provided by each step may be referred to, which is not described herein again.
The embodiments of the present application further provide a computer readable storage medium, where a computer program is stored and executed by a processor to implement the method provided by each step in fig. 1, and specifically refer to the implementation manner provided by each step, which is not described herein.
The computer readable storage medium may be the data flashback apparatus provided in any of the foregoing embodiments or an internal storage unit of an electronic device, such as a hard disk or a memory of the electronic device. The computer readable storage medium may also be an external storage device of the electronic device, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash card (flash card) or the like, which are provided on the electronic device. The computer readable storage medium may also include a magnetic disk, an optical disk, a read-only memory (ROM), a random access memory (random access memory, RAM), or the like. Further, the computer-readable storage medium may also include both an internal storage unit and an external storage device of the electronic device. The computer-readable storage medium is used to store the computer program and other programs and data required by the electronic device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.
Embodiments of the present application provide a computer program product comprising a computer program for executing the method provided by the steps of fig. 1 by a processor.
The terms "first," "second," and the like in the claims and specification and drawings of this application are used for distinguishing between different objects and not for describing a particular sequential order.
Furthermore, as used herein, the singular forms "a," "an," "the," and "the" are intended to include the plural forms as well, unless expressly stated otherwise. The terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion. For example, a process, method, system, article, or electronic device that comprises a list of steps or elements is not limited to the list of steps or elements but may, alternatively, include other steps or elements not listed or inherent to such process, method, article, or electronic device.
Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the present application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Those of skill in the art will explicitly and implicitly appreciate that the embodiments described herein may be combined with other embodiments. The term "and/or" as used in this specification and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes such combinations.
Those of ordinary skill in the art will appreciate that the elements and algorithm steps described in connection with the embodiments disclosed herein may be embodied in electronic hardware, in computer software, or in a combination of the two, and that the elements and steps of the examples have been generally described in terms of function in the foregoing description to clearly illustrate the interchangeability of hardware and software. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
The foregoing disclosure is only illustrative of the preferred embodiments of the present application and is not to be construed as limiting the scope of the claims, and therefore, equivalent variations in terms of the claims are intended to be included herein.
Claims (10)
1. A method of data flashback, comprising:
receiving a data flashback request; the data flashback request comprises a flashback time stamp;
determining target data which is pre-locked in a database and corresponds to the data flashback request according to the data flashback request; determining whether reference data corresponding to the flashback timestamp exists in the database;
Performing a first overlay operation on the target data based on a comparison result of the target data and the reference data under the condition that the reference data exists in the database;
the database comprises a first pre-written structured query statement for executing the first overlay operation.
2. The method according to claim 1, wherein the method further comprises:
performing a second overlay operation on the target data based on a flush instruction if it is determined that reference data does not exist in the database;
and the database further comprises a pre-written second structured query statement for executing the second overlay operation.
3. The method of claim 1, wherein prior to receiving the data flashback request, the method further comprises:
receiving a data flashback preparation request, and performing data isolation operation on the database; and
and locking the target storage area in the database.
4. A method according to claim 3, wherein said locking a target storage area in said database comprises:
determining a target row lock corresponding to the target storage area, and scanning and rollback the target row lock;
And determining the user key with the longest time span between the storage time stamp and the flashback time stamp in the target storage area, and locking the determined user key.
5. The method of claim 4, wherein the determining the flashback operation data corresponding to the data flashback request comprises:
determining a target user key with a changed key value in the target storage area in a time period from the flashback time stamp to a target time stamp; the target timestamp indicates a timestamp of locking the determined user key;
determining a key value of the target user key corresponding to the target timestamp as the reference data; and determining a key value of the target user key corresponding to the flashback timestamp as the target data.
6. A method according to claim 3, wherein said data isolation of said database comprises at least one of:
closing a garbage collection mechanism for the database;
setting read-write permission for the database as read-only permission;
and closing the connection relation between the database and any external component.
7. The method of claim 6, wherein after performing a data flashback operation on the target data, the method further comprises:
Updating a data table corresponding to the target storage area;
and generating data table information under the condition that version information corresponding to the updated data table is detected.
8. A data flashback apparatus, the apparatus comprising:
the data flashback request receiving module is used for receiving a data flashback request; the data flashback request comprises a flashback time stamp;
the flashback operation data determining module is used for determining target data which is locked in advance in a database and corresponds to the data flashback request according to the data flashback request; determining whether reference data corresponding to the flashback timestamp exists in the database;
a flashback operation execution module, configured to, if it is determined that the reference data exists in the database, perform a first overlay operation on the target data based on the target data and the reference data;
the database comprises a pre-written structured query statement for executing the first overlay operation.
9. An electronic device comprising a processor and a memory, the processor and the memory being interconnected;
the memory is used for storing a computer program;
The processor is configured to perform the method of any of claims 1 to 7 when the computer program is invoked.
10. A computer readable storage medium, characterized in that the computer readable storage medium stores a computer program which, when executed by a processor, implements the method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310592408.1A CN116541388A (en) | 2023-05-24 | 2023-05-24 | Data cleaning method and system for ultrasonic metering instrument |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310592408.1A CN116541388A (en) | 2023-05-24 | 2023-05-24 | Data cleaning method and system for ultrasonic metering instrument |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116541388A true CN116541388A (en) | 2023-08-04 |
Family
ID=87454137
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310592408.1A Pending CN116541388A (en) | 2023-05-24 | 2023-05-24 | Data cleaning method and system for ultrasonic metering instrument |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116541388A (en) |
-
2023
- 2023-05-24 CN CN202310592408.1A patent/CN116541388A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106415538B (en) | File service using shared file access-REST interface | |
| JP4598821B2 (en) | System and method for snapshot queries during database recovery | |
| CN107070919B (en) | Idempotency for database transactions | |
| US10013447B2 (en) | Transaction execution commitment without updating of data row transaction status | |
| WO2021018020A1 (en) | Data processing method and apparatus, and electronic device and computer storage medium | |
| US20030200212A1 (en) | Method, system, and program product for transaction management in a distributed content management application | |
| US20030028682A1 (en) | System and method for object persistence life-cycle and object caching integration | |
| US9747288B1 (en) | Scalable transaction-based data repository service | |
| US11657035B2 (en) | Methods, devices and systems for real-time checking of data consistency in a distributed heterogenous storage system | |
| CA2764510A1 (en) | Method and system for auditing transaction data from database operations | |
| EP4189914B1 (en) | Using multiple blockchains for applying transactions to a set of persistent data objects in persistent storage systems | |
| CN116541387A (en) | Data flashback method and device, electronic equipment and storage medium | |
| EP4276651A1 (en) | Log execution method and apparatus, and computer device and storage medium | |
| WO2022242372A1 (en) | Object processing method and apparatus, computer device, and storage medium | |
| US11061889B2 (en) | Systems and methods of managing manifest refresh in a database | |
| CN111857939A (en) | Method, system, electronic device and storage medium for deleting and pushing mirror image | |
| CN116541388A (en) | Data cleaning method and system for ultrasonic metering instrument | |
| WO2023111910A1 (en) | Rolling back database transaction | |
| CN116089359A (en) | Database snapshot generation method and device, electronic equipment and medium | |
| CN112860740B (en) | Method and device for realizing trigger | |
| CN114371870A (en) | Code scanning and submitting method, code scanning server, client and server | |
| CN117874145B (en) | Strong agreement method, device, equipment and storage medium for master-slave database | |
| CN116150126A (en) | Data processing method, device, electronic equipment and storage medium | |
| CN115794648A (en) | SQL (structured query language) auditing method and device | |
| CN115956232A (en) | Method and system for implementing multi-level real-time collaboration in a game engine supporting heterogeneous project states |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |