CN117076457A - Data processing method, device, equipment and storage medium - Google Patents
Data processing method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN117076457A CN117076457A CN202311076939.1A CN202311076939A CN117076457A CN 117076457 A CN117076457 A CN 117076457A CN 202311076939 A CN202311076939 A CN 202311076939A CN 117076457 A CN117076457 A CN 117076457A
- Authority
- CN
- China
- Prior art keywords
- snapshot
- data
- abstract
- target
- digest
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 27
- 238000012545 processing Methods 0.000 claims description 18
- 238000000034 method Methods 0.000 claims description 10
- 238000004364 calculation method Methods 0.000 claims description 4
- 230000008859 change Effects 0.000 abstract description 15
- 238000010586 diagram Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 239000013307 optical fiber Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 230000004075 alteration Effects 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000013479 data entry Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2457—Query processing with adaptation to user needs
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application provides a data processing method, a device, equipment and a storage medium, wherein data to be snapshot is obtained from a third party system, a first abstract is calculated for the data to be snapshot, and a second abstract is calculated for the data in a current snapshot data table; comparing the first abstract with the second abstract to obtain a target abstract, wherein the target abstract is the first abstract different from the second abstract; and carrying out snapshot operation on the data to be snapshot corresponding to the target abstract and storing the snapshot operation in a current snapshot data table. By using the embodiment, through calculating the abstract and comparing the abstract information, the data which is not changed in the third party system is eliminated, the purposes that the whole data in the third party system is not subjected to one-to-one snapshot, but only the data which have differences compared with the snapshot data in the current snapshot list are subjected to snapshot are achieved, so that the change trend of the data can be traced, meanwhile, the rapid expansion of the snapshot list data is avoided as much as possible, and the system redundancy is reduced.
Description
Technical Field
The present application relates to the field of computer technologies, and in particular, to a data processing method, apparatus, device, and storage medium.
Background
In the related art, a report system is used to record various resources and states thereof, but in general, the system records the current states of the resources, and cannot track the change trend of the related resources under a certain state, and cannot analyze or mine the historical change trend of the resource data.
In order to meet the above requirements, related technologies adopt snapshot (snapshot) technology to record a snapshot of the full data, the snapshot data cannot be modified, and data consistency in the process of backing up the data can be ensured. However, this will result in a fast expansion of the snapshot data table and the presence of a large amount of duplicate data, resulting in system redundancy.
It should be noted that the information disclosed in the foregoing background section is only for enhancement of understanding of the background of the application and thus may include information that does not form the prior art that is already known to those of ordinary skill in the art.
Disclosure of Invention
Aiming at the problems in the prior art, the application aims to provide a data processing method, a device, equipment and a storage medium, which overcome the difficulties in the prior art and can reduce the redundancy of a system.
The embodiment of the disclosure provides a data processing method, which comprises the following steps:
obtaining data to be snapshot from a third party system and calculating a first abstract of the data to be snapshot;
calculating a second abstract for snapshot data in the current snapshot data table;
comparing the first abstract with the second abstract to obtain a target abstract, wherein the target abstract is the first abstract different from the second abstract;
and carrying out snapshot operation on the data to be snapshot corresponding to the target abstract and storing the snapshot operation in a current snapshot data table.
In some embodiments, computing the second digest for the data in the current snapshot data table includes:
grouping the data in the current snapshot data table by using a resource identifier, and acquiring the latest state of each resource;
and taking out the snapshot data corresponding to the latest state to obtain the full snapshot data, and calculating a second abstract of each piece of snapshot data.
In some embodiments, comparing the first digest with the second digest to obtain a target digest includes:
and comparing the second abstract of each piece of snapshot data with the first abstract based on the resource identification to obtain a target abstract.
In some embodiments, obtaining the latest state for each resource includes:
the tag field under each resource identifier is identified and the latest state of the snapshot data under the resource identifier is determined by the tag field.
In some embodiments, in the case of performing a snapshot operation on data to be snapshot corresponding to the target abstract and storing the snapshot data in the current snapshot data table, the data processing method further includes:
in the current snapshot data table, a marking field is added under the resource identifier of the new snapshot data, and the marking field marks the latest state of the new snapshot data.
In some embodiments, the first digest that is different from the second digest comprises at least one of the following:
a first digest that is different from the second digest under the same resource identification;
and a first abstract corresponding to the newly added resource identifier.
In some embodiments, computing the second digest for the snapshot data in the current snapshot data table includes:
a second digest is computed for a portion of the target fields of the snapshot data in the current snapshot data table.
In some embodiments, performing snapshot operation on data to be snapshot corresponding to the target abstract and storing the snapshot data in a current snapshot data table, including:
performing snapshot operation on data to be snapshot corresponding to the target abstract to obtain a data snapshot;
and adding a data snapshot in the current snapshot data table.
The embodiment of the present disclosure also provides a data processing apparatus, including:
the first computing module is used for acquiring data to be snapshot from the third party system and computing a first abstract of the data to be snapshot;
the second calculation module calculates a second abstract of the data in the current snapshot data table;
the comparison module is used for comparing the first abstract with the second abstract to obtain a target abstract, wherein the target abstract is the first abstract different from the second abstract;
and the snapshot module is used for carrying out snapshot operation on the data to be snapshot corresponding to the target abstract in the third-party system and storing the snapshot data in the current snapshot data table.
The embodiment of the disclosure also provides an electronic device, which includes:
a processor;
a memory having stored therein executable instructions of a processor;
wherein the processor is configured to perform the steps of the data processing method of any of the embodiments described above via execution of executable instructions.
The present disclosure also provides a computer-readable storage medium storing a program which, when executed, implements the steps of the data processing method of any of the above embodiments.
According to the data processing method, the device, the equipment and the storage medium, the data to be snapshot is obtained from the third-party system, the first abstract is calculated for the data to be snapshot, and the second abstract is calculated for the data in the current snapshot data table; comparing the first abstract with the second abstract to obtain a target abstract, wherein the target abstract is the first abstract different from the second abstract; and carrying out snapshot operation on the data to be snapshot corresponding to the target abstract and storing the snapshot operation in a current snapshot data table. By using the embodiment, through calculating the abstract and comparing the abstract information, the data which is not changed in the third party system is eliminated, the purposes that the whole data in the third party system is not subjected to one-to-one snapshot, but only the data which have differences compared with the snapshot data in the current snapshot list are subjected to snapshot are achieved, so that the change trend of the data can be traced, meanwhile, the rapid expansion of the snapshot list data is avoided as much as possible, and the system redundancy is reduced.
Drawings
Other features, objects and advantages of the present application will become more apparent upon reading of the detailed description of non-limiting embodiments, made with reference to the following drawings.
Fig. 1 is a flow chart of a data processing method of an embodiment of the present disclosure.
Fig. 2 shows a schematic structural diagram of a data processing apparatus of an embodiment of the present disclosure.
Fig. 3 is a schematic structural diagram of an electronic device according to an embodiment of the present disclosure. And
fig. 4 is a schematic structural view of a computer-readable storage medium according to an embodiment of the present application.
Detailed Description
Other advantages and effects of the present application will be readily apparent to those skilled in the art from the following disclosure, which describes the embodiments of the present application by way of specific examples. The application may be practiced or carried out in other embodiments and with various details, and various modifications and alterations may be made to the details of the application from various points of view and applications without departing from the spirit of the application. It should be noted that, without conflict, the embodiments of the present application and features of the embodiments may be combined with each other.
The embodiments of the present application will be described in detail below with reference to the attached drawings so that those skilled in the art to which the present application pertains can easily implement the present application. This application may be embodied in many different forms and is not limited to the embodiments described herein.
Fig. 1 is a flowchart of a data processing method according to an embodiment of the present disclosure, and as shown in fig. 1, the embodiment of the present disclosure provides a data processing method, including the following steps:
step 110: obtaining data to be snapshot from a third party system and calculating a first abstract of the data to be snapshot;
step 120: calculating a second abstract for the data in the current snapshot data table;
step 130: comparing the first abstract with the second abstract to obtain a target abstract, wherein the target abstract is the first abstract different from the second abstract;
step 140: and carrying out snapshot operation on the data to be snapshot corresponding to the target abstract and storing the snapshot operation in a current snapshot data table.
By using the embodiment, through calculating the abstract and comparing the abstract information, the data which is not changed in the third party system is eliminated, the purposes that the whole data in the third party system is not subjected to one-to-one snapshot, but only the data which have differences compared with the snapshot data in the current snapshot list are subjected to snapshot are achieved, so that the change trend of the data can be traced, meanwhile, the rapid expansion of the snapshot list data is avoided as much as possible, and the system redundancy is reduced.
In the embodiment of the present disclosure, the above data processing method may be executed periodically, so as to store the data in the third party system in a snapshot periodically.
In the embodiment of the disclosure, a data summarization algorithm may be adopted to calculate a first summary for the snapshot data and a second summary for the snapshot data in the current snapshot data table. Data summarization algorithms such as MD5 algorithm.
In the embodiment of the disclosure, the data to be snapshot acquired from the third party system may be the total amount of data to be snapshot of the third party system, and the first digest is calculated for each and every data to be snapshot. Accordingly, the second digest is also computed one-to-one for the full amount of snapshot data in the snapshot data table. Furthermore, comparison errors caused by the change of certain field definitions in the snapshot data table can be avoided by calculating the abstract in real time and comparing the snapshots.
In an embodiment of the present disclosure, calculating a second digest for data in a current snapshot data table includes:
grouping the data in the current snapshot data table by using a resource identifier, and acquiring the latest state of each resource;
and taking out the snapshot data corresponding to the latest state to obtain the full snapshot data, and calculating a second abstract of each piece of snapshot data.
In this embodiment, on the one hand, each snapshot data corresponds to a unique resource identifier, the snapshot data in the snapshot data table is grouped through the resource identifier, and then summary comparison can be performed based on the data identifier, so that data processing efficiency is improved. On the other hand, the snapshot data record under each resource identifier may be updated, and the second abstract is calculated by taking the snapshot data in the latest state under the resource identifier each time, so that the cost of full data retrieval can be reduced, the data processing efficiency is further improved, and the system redundancy is reduced.
In an embodiment of the present disclosure, calculating the second digest for the snapshot data in the current snapshot data table may include:
a second digest is computed for a portion of the target fields of the snapshot data in the current snapshot data table.
In this embodiment, in some snapshot data, if a portion of the target field changes, the snapshot data is considered to change as a whole, and then the second digest may be calculated only for the data of the portion of the target field. The calculation time of the second abstract and the subsequent comparison and snapshot storage time can be reduced, and the data processing efficiency is improved.
In the embodiment of the present disclosure, comparing the first abstract with the second abstract to obtain a target abstract includes:
and comparing the second abstract of each piece of snapshot data with the first abstract based on the resource identification to obtain a target abstract.
In the embodiment, the second abstract is orderly extracted based on the resource identifier and the abstract information is compared, so that the feasibility of the scheme can be improved, and omission is avoided.
In an embodiment of the present disclosure, obtaining the latest state of each resource includes:
the tag field under each resource identifier is identified and the latest state of the snapshot data under the resource identifier is determined by the tag field.
In this embodiment, a tag field is added to the data entry of each resource identifier in advance, and the snapshot data of the latest state is tagged with the tag field, so that the latest state of the snapshot data can be quickly identified.
In an embodiment of the present disclosure, the first digest that is different from the second digest includes at least one of the following digests:
a first digest that is different from the second digest under the same resource identification;
and a first abstract corresponding to the newly added resource identifier.
In this embodiment, when the data to be snapshot in the third party system is updated compared to the snapshot data in the current snapshot data table, the corresponding first summary and second summary are changed. When the snapshot data to be snapshot in the third party system is newly added compared with the snapshot data in the current snapshot data table, the corresponding first abstract does not have the corresponding second abstract. These cases are considered to be the case where the first digest is different from the second digest.
In the embodiment of the present disclosure, in the case of performing a snapshot operation on data to be snapshot corresponding to the target abstract and storing the snapshot data in the current snapshot data table, the data processing method further includes:
in the current snapshot data table, a marking field is added under the resource identifier of the new snapshot data, and the marking field marks the latest state of the new snapshot data.
In this embodiment, the current new snapshot data belongs to the newly added data in the current snapshot data table, and has the latest state, and at this time, the snapshot data of the latest state can be identified by generating a flag field for the next snapshot.
In the embodiment of the present disclosure, performing snapshot operation on data to be snapshot corresponding to a target abstract and storing the snapshot data in a current snapshot data table, including:
performing snapshot operation on data to be snapshot corresponding to the target abstract to obtain a data snapshot;
and adding a data snapshot in the current snapshot data table.
In this embodiment, the data change trend can be recorded by newly adding means to data snapshot.
In an embodiment of the present disclosure, a method for processing a web,
the method is a general database table snapshot updating method, and can flexibly customize a target field to achieve a strategy of flexibly modifying incremental data updating as long as each piece of data has a unique resource identifier. The change trend of the data set can be comprehensively recorded, only the changed data is recorded, and the rapid expansion of the snapshot table caused by a large amount of redundant data is avoided.
Fig. 2 is a schematic diagram of the structure of the data processing apparatus of the present application. As shown in fig. 2, the data processing apparatus 200 of the present application includes:
the first computing module 210 obtains data to be snapshot from the third party system and computes a first abstract of the data to be snapshot;
a second calculation module 220 that calculates a second digest of the data in the current snapshot data table;
the comparison module 230 compares the first abstract with the second abstract to obtain a target abstract, wherein the target abstract is the first abstract different from the second abstract;
and the snapshot module 240 performs snapshot operation on the data to be snapshot corresponding to the target abstract in the third party system and stores the snapshot data in the current snapshot data table.
In an alternative embodiment, the second computing module 220 is specifically configured to:
grouping the data in the current snapshot data table by using a resource identifier, and acquiring the latest state of each resource;
and taking out the snapshot data corresponding to the latest state to obtain the full snapshot data, and calculating a second abstract of each piece of snapshot data.
In an alternative embodiment, the comparison module 230 is specifically configured to:
and comparing the second abstract of each piece of snapshot data with the first abstract based on the resource identification to obtain a target abstract.
In an alternative embodiment, the second computing module 220 is specifically further configured to:
the tag field under each resource identifier is identified and the latest state of the snapshot data under the resource identifier is determined by the tag field.
In an alternative embodiment, snapshot module 240 is specifically further configured to:
in the current snapshot data table, a marking field is added under the resource identifier of the new snapshot data, and the marking field marks the latest state of the new snapshot data.
In an alternative embodiment, the first digest that is different from the second digest includes at least one of the following digests:
a first digest that is different from the second digest under the same resource identification;
and a first abstract corresponding to the newly added resource identifier.
In an alternative embodiment, the second computing module 220 is specifically further configured to:
a second digest is computed for a portion of the target fields of the snapshot data in the current snapshot data table.
In an alternative embodiment, snapshot module 240:
performing snapshot operation on data to be snapshot corresponding to the target abstract to obtain a data snapshot;
and adding a data snapshot in the current snapshot data table.
By using the data processing device of the embodiment, through calculating the abstract and comparing the abstract information, the data which does not change in the third party system is eliminated, the purposes that all data in the third party system are not subjected to one-to-one snapshot, but only the data which have differences compared with the snapshot data in the current snapshot table are subjected to snapshot are achieved, so that the change trend of the data can be traced, meanwhile, the rapid expansion of the snapshot table data is avoided as much as possible, and the system redundancy is reduced.
The embodiment of the application also provides electronic equipment which comprises a processor. A memory having stored therein executable instructions of a processor. Wherein the processor is configured to perform the steps of the data processing method via execution of the executable instructions.
As described above, the electronic device of the application can remove the data which does not change in the third party system by calculating the abstract and comparing the abstract information, and can realize that the whole data in the third party system is not subjected to one-to-one snapshot, but only the data which has difference compared with the snapshot data in the current snapshot table is subjected to snapshot, thereby tracing the change trend of the data, avoiding the rapid expansion of the snapshot table data as much as possible, and reducing the redundancy of the system.
Those skilled in the art will appreciate that the various aspects of the application may be implemented as a system, method, or program product. Accordingly, aspects of the application may be embodied in the following forms, namely: an entirely hardware embodiment, an entirely software embodiment (including firmware, micro-code, etc.) or an embodiment combining hardware and software aspects may be referred to herein as a "circuit," module "or" platform.
Fig. 3 is a schematic structural view of the electronic device of the present application. An electronic device 300 according to this embodiment of the application is described below with reference to 36. The electronic device 300 shown in fig. 3 is merely an example and should not be construed as limiting the functionality and scope of use of embodiments of the present application.
As shown in fig. 3, the electronic device 300 is embodied in the form of a general purpose computing device. Components of electronic device 300 may include, but are not limited to: at least one processing unit 310, at least one storage unit 320, a bus 330 connecting the different platform components (including the storage unit 320 and the processing unit 310), a display unit 340, and the like.
In which a memory unit stores program code that may be executed by the processing unit 310, such that the processing unit 310 performs the steps according to various exemplary embodiments of the application described in the data processing method section of this description. For example, the processing unit 310 may perform the steps as shown in fig. 1.
The storage unit 320 may include a readable medium in the form of a volatile storage unit, such as a Random Access Memory (RAM) 321 and/or a cache memory 322, and may further include a Read Only Memory (ROM) 323.
The storage unit 320 may also include a program/utility 324 having a set (at least one) of program modules 325, such program modules 325 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of which may include an implementation of a network environment.
Bus 330 may be one or more of several types of bus structures including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.
The electronic device 300 may also communicate with one or more external devices 30 (e.g., keyboard, pointing device, bluetooth device, etc.), one or more devices that enable a user to interact with the electronic device 300, and/or any device (e.g., router, modem, etc.) that enables the electronic device 300 to communicate with one or more other computing devices. Such communication may occur through an input/output (I/O) interface 350. Also, electronic device 300 may communicate with one or more networks such as a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network, such as the Internet, through network adapter 360. The network adapter 360 may communicate with other modules of the electronic device 300 via the bus 330. It should be appreciated that although not shown, other hardware and/or software modules may be used in connection with electronic device 300, including, but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, data backup storage platforms, and the like.
The embodiment of the application also provides a computer readable storage medium for storing a program, and the steps of the data processing method implemented when the program is executed. In some possible embodiments, the various aspects of the application may also be implemented in the form of a program product comprising program code for causing a terminal device to carry out the steps according to the various exemplary embodiments of the application as described in the data processing method section of this specification, when the program product is run on the terminal device.
As described above, the program of the computer-readable storage medium of this embodiment, when executed, is capable of acquiring data to be snapshot from a third party system and calculating a first digest for the data to be snapshot, and calculating a second digest for the data in the current snapshot data table; comparing the first abstract with the second abstract to obtain a target abstract, wherein the target abstract is the first abstract different from the second abstract; and carrying out snapshot operation on the data to be snapshot corresponding to the target abstract and storing the snapshot operation in a current snapshot data table. By using the embodiment, through calculating the abstract and comparing the abstract information, the data which is not changed in the third party system is eliminated, the purposes that the whole data in the third party system is not subjected to one-to-one snapshot, but only the data which have differences compared with the snapshot data in the current snapshot list are subjected to snapshot are achieved, so that the change trend of the data can be traced, meanwhile, the rapid expansion of the snapshot list data is avoided as much as possible, and the system redundancy is reduced.
Fig. 4 is a schematic structural view of a computer-readable storage medium of the present application. Referring to fig. 4, a program product 400 for implementing the above-described method according to an embodiment of the present application is described, which may employ a portable compact disc read only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present application is not limited thereto, and in this document, a readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium would include the following: an electrical connection having one or more wires, a portable disk, a hard disk, random Access Memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
The computer readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, with readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A readable storage medium may also be any readable medium that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Program code for carrying out operations of the present application may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device, partly on a remote computing device, or entirely on the remote computing device or server. In the case of remote computing devices, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., connected via the Internet using an Internet service provider).
In summary, the data processing method, the device, the equipment and the storage medium of the application calculate the second abstract for the data in the current snapshot data table by acquiring the data to be snapshot from the third party system and calculating the first abstract for the data to be snapshot; comparing the first abstract with the second abstract to obtain a target abstract, wherein the target abstract is the first abstract different from the second abstract; and carrying out snapshot operation on the data to be snapshot corresponding to the target abstract and storing the snapshot operation in a current snapshot data table. By using the embodiment, through calculating the abstract and comparing the abstract information, the data which is not changed in the third party system is eliminated, the purposes that the whole data in the third party system is not subjected to one-to-one snapshot, but only the data which have differences compared with the snapshot data in the current snapshot list are subjected to snapshot are achieved, so that the change trend of the data can be traced, meanwhile, the rapid expansion of the snapshot list data is avoided as much as possible, and the system redundancy is reduced.
The foregoing is a further detailed description of the application in connection with the preferred embodiments, and it is not intended that the application be limited to the specific embodiments described. It will be apparent to those skilled in the art that several simple deductions or substitutions may be made without departing from the spirit of the application, and these should be considered to be within the scope of the application.
Claims (11)
1. A method of data processing, comprising:
obtaining data to be snapshot from a third party system and calculating a first abstract of the data to be snapshot;
calculating a second abstract for snapshot data in the current snapshot data table;
comparing the first abstract with the second abstract to obtain a target abstract, wherein the target abstract is a first abstract different from the second abstract;
and carrying out snapshot operation on the data to be snapshot corresponding to the target abstract and storing the snapshot operation in the current snapshot data table.
2. The method of claim 1, wherein said computing a second digest of the data in the current snapshot data table comprises:
grouping the data in the current snapshot data table by using a resource identifier to acquire the latest state of each resource;
and taking out the snapshot data corresponding to the latest state to obtain the full snapshot data, and calculating a second abstract of each piece of snapshot data.
3. The method of claim 2, wherein comparing the first digest with the second digest to obtain a target digest comprises:
and comparing the second abstract of each piece of snapshot data with the first abstract based on the resource identification to obtain a target abstract.
4. The method of claim 2, wherein the obtaining the latest state of each resource comprises:
a tag field under each resource identifier is identified and a latest state of snapshot data under the resource identifier is determined by the tag field.
5. The data processing method according to claim 2, wherein in the case of performing a snapshot operation on data to be snapshot corresponding to the target digest and storing in the current snapshot data table, the data processing method further comprises:
and adding a marking field in the current snapshot data table under the resource identifier of the new snapshot data, wherein the marking field marks the latest state of the new snapshot data.
6. The data processing method of claim 1, wherein the first digest that is different from the second digest comprises at least one of:
a first digest that is different from the second digest under the same resource identification;
and a first abstract corresponding to the newly added resource identifier.
7. The data processing method of claim 1, wherein said computing a second digest of snapshot data in the current snapshot data table comprises:
and calculating the second abstract for partial target fields of snapshot data in the current snapshot data table.
8. The data processing method according to claim 1, wherein the snapshot operation on the data to be snapshot corresponding to the target abstract is stored in the current snapshot data table, and includes:
performing snapshot operation on the data to be snapshot corresponding to the target abstract to obtain a data snapshot;
and newly adding the data snapshot in the current snapshot data table.
9. A data processing apparatus, comprising:
the first computing module is used for acquiring data to be snapshot from a third party system and computing a first abstract of the data to be snapshot;
the second calculation module calculates a second abstract of the data in the current snapshot data table;
the comparison module is used for comparing the first abstract with the second abstract to obtain a target abstract, wherein the target abstract is a first abstract different from the second abstract;
and the snapshot module performs snapshot operation on the data to be snapshot corresponding to the target abstract in the third party system and stores the snapshot data in the current snapshot data table.
10. An electronic device, comprising:
a processor;
a memory having stored therein executable instructions of the processor;
wherein the processor is configured to perform the steps of the data processing method of any of claims 1 to 8 via execution of the executable instructions.
11. A computer-readable storage medium storing a program, characterized in that the program when executed implements the steps of the data processing method of any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311076939.1A CN117076457A (en) | 2023-08-24 | 2023-08-24 | Data processing method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311076939.1A CN117076457A (en) | 2023-08-24 | 2023-08-24 | Data processing method, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117076457A true CN117076457A (en) | 2023-11-17 |
Family
ID=88705771
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311076939.1A Pending CN117076457A (en) | 2023-08-24 | 2023-08-24 | Data processing method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117076457A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117312175A (en) * | 2023-11-29 | 2023-12-29 | 盛业信息科技服务(深圳)有限公司 | Data processing method, device, computer equipment and storage medium |
-
2023
- 2023-08-24 CN CN202311076939.1A patent/CN117076457A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117312175A (en) * | 2023-11-29 | 2023-12-29 | 盛业信息科技服务(深圳)有限公司 | Data processing method, device, computer equipment and storage medium |
| CN117312175B (en) * | 2023-11-29 | 2024-03-22 | 盛业信息科技服务(深圳)有限公司 | Data processing method, device, computer equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104008064B (en) | The method and system compressed for multi-level store | |
| US8732666B2 (en) | Automatic identification of subroutines from test scripts | |
| CN111240689A (en) | Application program construction method, device, equipment and storage medium | |
| CN110555150B (en) | Data monitoring method, device, equipment and storage medium | |
| CN111241203A (en) | Hive data warehouse synchronization method, system, equipment and storage medium | |
| CN117076457A (en) | Data processing method, device, equipment and storage medium | |
| CN112559140B (en) | Transaction control method, system, equipment and storage medium for data consistency | |
| CN112947986B (en) | Multi-version code sign-on control method, device, client and storage medium | |
| CN111818145B (en) | File transmission method, device, system, equipment and storage medium | |
| US10001925B2 (en) | Method to improve page out mechanism with compressed memory pools | |
| CN110990346A (en) | File data processing method, device, equipment and storage medium based on block chain | |
| CN110716804A (en) | Method and device for automatically deleting useless resources, storage medium and electronic equipment | |
| CN110865829A (en) | Database upgrading method, system, device and storage medium | |
| CN111104259B (en) | Database recovery method and device and storage medium | |
| US11216427B2 (en) | Method, electronic device and computer-readable medium for managing metadata | |
| CN109542860B (en) | Service data management method based on HDFS and terminal equipment | |
| CN110795259A (en) | Method and device for analyzing application collapse | |
| CN112286909B (en) | Execution state determining method, device, server and storage medium | |
| US20200341648A1 (en) | Method, device and computer program product for storage management | |
| CN113467941A (en) | Method and device for sharing information | |
| CN113297028A (en) | Script execution method and device, computer equipment and storage medium | |
| CN112364268A (en) | Resource acquisition method and device, electronic equipment and storage medium | |
| CN111897568B (en) | System verification method, device, equipment and storage medium | |
| US20220342600A1 (en) | Method, electronic device, and computer program product for restoring data | |
| US20120233224A1 (en) | Data processing |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |