CN110175182B - Data checking method and device - Google Patents
Data checking method and device Download PDFInfo
- Publication number
- CN110175182B CN110175182B CN201910465753.2A CN201910465753A CN110175182B CN 110175182 B CN110175182 B CN 110175182B CN 201910465753 A CN201910465753 A CN 201910465753A CN 110175182 B CN110175182 B CN 110175182B
- Authority
- CN
- China
- Prior art keywords
- data
- data change
- change message
- target
- determining
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2358—Change logging, detection, and notification
Abstract
The invention discloses a data checking method and device. The method comprises the following steps: determining a plurality of data sources corresponding to the collation rules, and for each of the plurality of data sources, generating a data change message corresponding to a target data change when the target data change occurs in the data source; subscribing to data change messages of a plurality of data sources; storing the data change message in a target storage space; and performing data check on the data in the target storage space by using the check rule. The scheme can store the check data in the target storage space in time, thereby avoiding the defect of large check delay of data check in the prior art; and data checking is carried out by using the data in the target storage space, so that the service processing pressure of the service system can be further reduced, and the service processing efficiency of the service system is improved.
Description
Technical Field
The invention relates to the technical field of computers, in particular to a data checking method and device.
Background
With the rapid development of the internet technology, the processing efficiency of various services is continuously improved, thereby greatly facilitating the work and life of people. In the process of business processing, the circulation of business data is usually involved. At present, in order to avoid user resource loss caused by abnormal service data flow, data checking operation is usually performed in the data flow process. For example, in a bank, a third-party payment platform, or other various platforms relating to article transactions, it is generally necessary to perform a checking operation for business processing, such as account checking, account certificate checking, inter-system checking, and checking of each node between systems, so as to ensure the accuracy of business flow or the like.
At present, in the process of checking data, there are two checking methods (for example, chinese patent application with publication number CN 103136276A): firstly, acquiring check data according to a preset period, and further realizing the check of the data; and secondly, searching the corresponding on-line database in real time to further realize the data check.
However, the inventor finds the following problems in the prior art in the implementation process: when the check of the data is realized by acquiring the check data according to the preset period, the data cannot be checked in time, so that larger check delay is easily caused; and the corresponding on-line database is searched in real time to realize the data check, so that the pressure of the service system is increased, and the service processing efficiency of the service system is further reduced.
Disclosure of Invention
In view of the above, the present invention has been made to provide a data collating method and apparatus that overcomes or at least partially solves the above problems.
According to an aspect of the present invention, there is provided a data collation method including:
determining a plurality of data sources corresponding to the collation rules, and generating a data change message corresponding to a target data change when the target data change occurs in each of the plurality of data sources;
subscribing to data change messages of the plurality of data sources;
storing the data change message in a target storage space;
and performing data check on the data in the target storage space by using the check rule.
Optionally, when a target data change occurs in the data source, generating a data change message corresponding to the target data change further includes:
when the target data in the data source is changed, generating a data change message corresponding to the target data change according to the snapshot information of the changed target data.
Optionally, the storing the data change message in the target storage space further includes:
determining a unique key corresponding to the data change message;
judging whether the target storage space has storage data corresponding to the unique key or not;
if so, storing the data change message in a target storage space in a mode of covering the data change message with the stored data corresponding to the unique key;
if not, directly storing the data change message in the target storage space.
Optionally, the determining the unique key corresponding to the data change message further includes:
determining a collation rule corresponding to the data change message;
determining a unique key determination mode corresponding to the data change message according to the check rule corresponding to the data change message;
and determining the unique key corresponding to the data change message by adopting the unique key determination mode.
Optionally, the determining the unique key corresponding to the data change message further includes:
and determining the unique key corresponding to the data change message according to the service table identifier, the service main key and/or the service ID in the data change message.
Optionally, the method further includes:
and determining the data dimension corresponding to the target data according to the checking rule.
Optionally, the data dimension corresponding to the target data includes at least one of the following data dimensions: a data table dimension, a single data dimension, and/or a database dimension.
According to another aspect of the present invention, there is provided a data collation apparatus comprising:
a determining module adapted to determine a plurality of data sources corresponding to the collation rules, for each of the plurality of data sources, when a target data change occurs in the data source, generating a data change message corresponding to the target data change;
a subscription module adapted to subscribe to data change messages of the plurality of data sources;
a storage module adapted to store the data change message in a target storage space;
and the checking module is suitable for performing data checking on the data in the target storage space by using the checking rule.
Optionally, the determining module is further adapted to:
when the target data in the data source is changed, generating a data change message corresponding to the target data change according to the snapshot information of the changed target data.
Optionally, the storage module is further adapted to: determining a unique key corresponding to the data change message;
judging whether the target storage space has storage data corresponding to the unique key or not;
if so, storing the data change message in a target storage space in a mode of covering the data change message with the stored data corresponding to the unique key;
if not, directly storing the data change message in the target storage space.
Optionally, the storage module is further adapted to: determining a collation rule corresponding to the data change message;
determining a unique key determination mode corresponding to the data change message according to the check rule corresponding to the data change message;
and determining the unique key corresponding to the data change message by adopting the unique key determination mode.
Optionally, the storage module is further adapted to: and determining the unique key corresponding to the data change message according to the service table identifier, the service main key and/or the service ID in the data change message.
Optionally, the apparatus further comprises:
and the dimension determining module is suitable for determining the data dimension corresponding to the target data according to the checking rule.
Optionally, the data dimension corresponding to the target data includes at least one of the following data dimensions: a data table dimension, a single data dimension, and/or a database dimension.
According to yet another aspect of the present invention, there is provided a computing device comprising: the system comprises a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface complete mutual communication through the communication bus;
the memory is used for storing at least one executable instruction, and the executable instruction enables the processor to execute the operation corresponding to the data checking method.
According to still another aspect of the present invention, there is provided a computer storage medium having at least one executable instruction stored therein, the executable instruction causing a processor to perform operations corresponding to the data collation method.
According to the data checking method and the device provided by the invention, a plurality of data sources corresponding to the checking rule are determined, and for each data source in the plurality of data sources, when target data change occurs in the data source, a data change message corresponding to the target data change is generated; further subscribing to data change messages of a plurality of data sources; storing the data change message in a target storage space; and finally, carrying out data check on the data in the target storage space by using a check rule. The scheme can store the check data in the target storage space in time, thereby avoiding the defect of large check delay of data check in the prior art; and data checking is carried out by using the data in the target storage space, so that the service processing pressure of the service system is further reduced, and the service processing efficiency of the service system is improved.
The foregoing description is only an overview of the technical solutions of the present invention, and the embodiments of the present invention are described below in order to make the technical means of the present invention more clearly understood and to make the above and other objects, features, and advantages of the present invention more clearly understandable.
Drawings
Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the invention. Also, like reference numerals are used to refer to like parts throughout the drawings. In the drawings:
FIG. 1 is a flow chart illustrating a data collation method according to an embodiment of the present invention;
FIG. 2 is a flow chart illustrating a data collation method according to another embodiment of the present invention;
FIG. 3 is a functional block diagram of a data collation apparatus according to an embodiment of the present invention;
fig. 4 is a schematic structural diagram of a computing device according to an embodiment of the present invention.
Detailed Description
Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
Fig. 1 is a flow chart illustrating a data collation method according to an embodiment of the present invention. The data checking method provided by the embodiment is not executed by the service system, but executed by a preset data checking system.
As shown in fig. 1, the method includes:
step S110 is to identify a plurality of data sources corresponding to the collation rule, and for each of the plurality of data sources, generate a data change message corresponding to a target data change when the target data change occurs in the data source.
Unlike the prior art that the database interface is periodically called to obtain the verification data, in the embodiment, when the target data is changed in the data source corresponding to the verification rule, a data change message corresponding to the target data change is generated, and the verification data is obtained in real time according to the subscription operation of the data change message in the subsequent step S120, so as to avoid a large verification delay in data verification. In the specific implementation process:
first, a plurality of data sources corresponding to the collation rules are determined: in the prior art, a service system generally receives each data check request, and determines data required for the data check by analyzing check information in the data check request, so as to invoke a corresponding database interface to obtain corresponding check-pair data for data check. The present embodiment is different from the operation manner in the prior art, and the present embodiment specifically obtains the collation rule first, and the collation rule is configured for data collation. The obtained check rule may be one or multiple, and this embodiment does not limit this. Further, after the collation rule is acquired, a plurality of data sources corresponding to the collation rule are determined in accordance with the collation rule. The checking rule includes data source information for checking in the checking process and a specific checking mode, so that a plurality of data sources corresponding to the checking rule can be automatically determined according to a mode of field analysis and the like in a specific implementation process.
Further, for each of the plurality of data sources, when a target data change occurs in the data source, a data change message corresponding to the target data change is generated. Specifically, after a plurality of data sources corresponding to the collation rule are determined, for each data source, target data in the data source is determined, wherein the target data is determined by the collation rule. And when the target data is changed, generating a data change message corresponding to the target data change.
Step S120, subscribe to data change messages of multiple data sources.
In this embodiment, the preset data verification system subscribes to data change messages of a plurality of data sources, so that changes of verification data corresponding to the verification rule can be acquired in time. The present embodiment does not limit the specific subscription method.
Step S130, storing the data change message in the target storage space.
After the preset data checking system subscribes the data change messages of the plurality of data sources, the check data can be obtained in time, and the subscribed data change messages are stored in the target storage space, so that the data in the target storage space can be directly used for data checking in the subsequent steps, the processing pressure of an online business system is reduced, and the processing efficiency of the business system is improved. In this embodiment, a specific target storage space is not limited, for example, the target storage space may be hbase or the like.
In step S140, data collation is performed on the data in the target storage space using the collation rule.
The step can acquire corresponding check data from the target storage space according to the specific check time set by the check rule, and the data check is realized by adopting the check mode in the check rule.
As can be seen from this, in the present embodiment, when a target data change occurs in a data source corresponding to a collation rule, a data change message corresponding to the target data change is generated, a preset data collation system subscribes to the data change messages of a plurality of data sources, and the data change message is further stored in a target storage space and stored in the target storage space. Therefore, the defect that the data checking has larger checking delay in the prior art is avoided; and data checking is carried out by using the data in the target storage space, so that the service processing pressure of the service system is further reduced, and the service processing efficiency of the service system is improved.
Fig. 2 is a schematic flow chart illustrating a data collation method according to another embodiment of the present invention. The data verification method provided in this embodiment is specifically a further optimization of the embodiment of the method shown in fig. 1. As shown in fig. 2, the method includes:
in step S210, a plurality of data sources corresponding to the collation rules are determined.
In this embodiment, the collation rule may be one or more collation rules configured in a preset data collation system. In a specific implementation process, a plurality of data sources corresponding to the collation rules may be automatically determined according to field parsing and the like.
Optionally, in order to adjust the determined multiple data sources in time, the present embodiment may monitor a change operation of the checking rule in the checking system for the preset data. Wherein the change operation includes an add operation, a delete operation, and/or a modify operation, etc. And then according to the change operation aiming at the checking rule, dynamically adjusting the plurality of determined data sources, thereby avoiding the occurrence of invalid data and improving the accuracy of data checking.
Further optionally, the determined multiple data sources may be updated according to valid verification time limits corresponding to the verification rules. In an actual implementation process, a plurality of check rules usually include corresponding valid check time limits, for example, a check rule for a certain promotional activity, where the check rule is usually valid only during or after the promotional activity, in this embodiment, the valid check time limit corresponding to the check rule may be obtained by parsing the valid check time limit field of the check rule, and whether to reject the check rule is determined by comparing the valid check time limit with the current time. When the check rule is determined to be eliminated, the subscription of the data source corresponding to the check rule is removed, so that the overall processing efficiency of the data check system is improved, and the waste of system resources is avoided.
Step S220 is executed to generate, for each of the plurality of data sources, a data change message corresponding to a change in the target data according to the snapshot information of the changed target data when the target data in the data source is changed.
For each of a plurality of data sources corresponding to the collation rules, target data in the data source is determined. In a specific implementation process, the data dimension corresponding to the target data can be determined according to the checking rule. The data dimension corresponding to the target data comprises at least one of the following data dimensions: a data table dimension, a single data dimension, and/or a database dimension, among others. That is, in this embodiment, for each collation rule, the data dimension matched with the collation rule is automatically analyzed through automatic analysis of the data source field of the collation rule, so that the efficiency of subsequent data collation is improved.
Further, when the target data in the data source is changed, a data change message corresponding to the target data change is generated according to the snapshot information of the changed target data. After determining the target data in the multiple data sources corresponding to the collation rule, whenever the target data in the data source is changed, acquiring snapshot information of the changed target data, where a specific snapshot information acquisition manner is not limited in this embodiment, and a person skilled in the art may select a corresponding snapshot information generation manner according to an actual situation. Further, a data change message corresponding to the target data change is generated based on the snapshot information of the target data after the change. Namely, the generated data change message is the latest data state of the current target data, thereby ensuring the accuracy of subsequent data verification and reducing the data verification noise.
Optionally, in order to avoid more redundant data in the subsequent target storage space, the data change message generated in this step further includes a change time of the target data corresponding to the data change message.
Step S230, subscribe to data change messages of multiple data sources.
In this embodiment, the preset data verification system subscribes to data change messages of a plurality of data sources, so that changes of verification data corresponding to the verification rule can be acquired in time. The present embodiment does not limit the specific subscription method.
Step S240, determining a unique key corresponding to the data change message, and storing the data change message in the target storage space according to the unique key.
The data stored in the target storage space can be guaranteed to be the latest data, and more data redundancy of the target storage space can be avoided. In this embodiment, a corresponding unique key is specifically set for the data change message, that is, the data stored in the target storage space also has a corresponding unique key, so that the data change message is stored in the target storage space according to the unique key.
In a specific implementation, a unique key corresponding to the data change message is first determined. When the unique key corresponding to the data change message is determined, a check rule corresponding to the data change message can be determined, a unique key determination mode corresponding to the data change message is determined according to the check rule corresponding to the data change message, and finally the unique key determination mode is adopted to determine the unique key corresponding to the data change message. By the unique key determining method, the unique key determining method can be dynamically adjusted or determined in real time according to the change of the check rule, so that the unique key corresponding to the data change message is dynamically adjusted, the strong relation between the stored data and the check rule is realized, the data check accuracy is further improved, and the data check noise is avoided; in addition, required verification data can be accurately and quickly acquired according to the verification rule in the data verification process, and the data verification efficiency is further improved. In addition, the unique key corresponding to the data change message can also be determined according to the service table identifier, the service primary key and/or the service ID in the data change message. For example, the result obtained after the hash operation is performed on the service table identifier, the service primary key, and/or the concatenation field of the service ID is determined as the unique key corresponding to the data change message.
Further, whether the storage data corresponding to the unique key exists in the target storage space is judged. In a specific implementation process, a unique key corresponding to the storage data in the target storage space needs to be determined. The unique key corresponding to the data stored in the target storage space can be determined according to the above-mentioned manner of determining the unique key corresponding to the data change message. And judging whether the storage data corresponding to the unique key exists in the target storage space or not by judging whether the unique key corresponding to the storage data in the target storage space is matched with the unique key corresponding to the data change message or not. If the unique key for storing certain storage data in the target storage space is matched with the unique key corresponding to the data change message, determining that the storage data corresponding to the unique key exists in the target storage space; otherwise, determining that the storage data corresponding to the unique key does not exist in the target storage space.
And if the target storage space has the storage data corresponding to the unique key corresponding to the data change message, storing the data change message in the target storage space in a mode of covering the storage data corresponding to the unique key with the data change message. Namely deleting the stored data corresponding to the unique key, and storing the data change message at the position corresponding to the stored data, thereby ensuring that the data stored in the target storage space is the latest data.
And if the target storage space does not have the storage data corresponding to the unique key corresponding to the data change message, directly storing the data change message in the target storage space.
In an optional implementation manner, centralized management can be performed on the stored data corresponding to the same checking rule, so that the storage efficiency of the data change message in the target storage space is improved, and the overall improvement of the checking efficiency is facilitated; when the collation rule is changed, batch management of the stored data corresponding to the collation rule can be realized. Alternatively, the target storage space may be hbase or the like.
Step S250, using the checking rule, performs data checking on the data in the target storage space.
In the step, corresponding verification data can be obtained from the target storage space according to the specific verification time set by the verification rule, so that data verification is realized.
Therefore, in the embodiment, when the target data change occurs in the data source corresponding to the collation rule, the data change message corresponding to the target data change is generated, the preset data collation system subscribes the data change messages of the multiple data sources, the data change messages are further stored in the target storage space, and the data change messages are stored in the target storage space, so that the defect that the data collation in the prior art has large collation delay is avoided; data checking is carried out by using data in the target storage space, so that the service processing pressure of the service system is further reduced, and the service processing efficiency of the service system is improved; in addition, the data change message is stored in the target storage space according to the unique key corresponding to the data change message, so that more data redundancy in the target storage space is avoided; in addition, the determination mode of the unique key is determined through the check rule, so that strong connection between the stored data and the check rule is realized, the accuracy of data check is further improved, and the occurrence of data check noise is avoided.
Fig. 3 is a schematic diagram illustrating a functional structure of a data collating apparatus according to an embodiment of the present invention. As shown in fig. 3, the apparatus includes: a determination module 31, a subscription module 32, a storage module 33 and a verification module 34.
Wherein the determining module 31 is adapted to determine a plurality of data sources corresponding to the collation rule, and for each of the plurality of data sources, when a target data change occurs in the data source, generate a data change message corresponding to the target data change;
a subscription module 32 adapted to subscribe to data change messages of the plurality of data sources;
a storage module 33 adapted to store the data change message in a target storage space;
and the checking module 34 is suitable for performing data checking on the data in the target storage space by using the checking rule.
Optionally, the determining module 31 is further adapted to:
when the target data in the data source is changed, generating a data change message corresponding to the target data change according to the snapshot information of the changed target data.
Optionally, the storage module 33 is further adapted to: determining a unique key corresponding to the data change message;
judging whether the target storage space has storage data corresponding to the unique key or not;
if so, storing the data change message in a target storage space in a mode of covering the data change message with the stored data corresponding to the unique key;
if not, directly storing the data change message in the target storage space.
Optionally, the storage module 33 is further adapted to: determining a collation rule corresponding to the data change message;
determining a unique key determination mode corresponding to the data change message according to the check rule corresponding to the data change message;
and determining the unique key corresponding to the data change message by adopting the unique key determination mode.
Optionally, the storage module 33 is further adapted to: and determining the unique key corresponding to the data change message according to the service table identifier, the service main key and/or the service ID in the data change message.
Optionally, the apparatus further comprises: and the dimension determining module (not shown in the figure) is suitable for determining the data dimension corresponding to the target data according to the checking rule.
Optionally, the data dimension corresponding to the target data includes at least one of the following data dimensions: a data table dimension, a single data dimension, and/or a database dimension.
The specific implementation process of each module in the apparatus of this embodiment may refer to the description of the corresponding part in the method embodiment shown in fig. 1 and/or fig. 2, and this embodiment is not limited herein.
As can be seen from this, in the present embodiment, when a target data change occurs in a data source corresponding to a collation rule, a data change message corresponding to the target data change is generated, a preset data collation system subscribes to the data change messages of a plurality of data sources, and the data change message is further stored in a target storage space and stored in the target storage space. Therefore, the phenomenon of large checking delay of data checking in the prior art is avoided; and data checking is carried out by using the data in the target storage space, so that the service processing pressure of the service system is further reduced, and the service processing efficiency of the service system is improved.
According to an embodiment of the present invention, a non-volatile computer storage medium is provided, the computer storage medium storing at least one executable instruction, the computer executable instruction being capable of executing the data checking method in any of the above method embodiments.
Fig. 4 is a schematic structural diagram of a computing device according to an embodiment of the present invention, and the specific embodiment of the present invention does not limit the specific implementation of the computing device.
As shown in fig. 4, the computing device may include: a processor (processor)402, a Communications Interface 404, a memory 406, and a Communications bus 408.
Wherein:
the processor 402, communication interface 404, and memory 406 communicate with each other via a communication bus 408.
A communication interface 404 for communicating with network elements of other devices, such as clients or other servers.
The processor 402 is configured to execute the program 410, and may specifically perform the relevant steps in the above-described data checking method embodiment.
In particular, program 410 may include program code comprising computer operating instructions.
The processor 402 may be a central processing unit CPU or an application Specific Integrated circuit asic or one or more Integrated circuits configured to implement embodiments of the present invention. The computing device includes one or more processors, which may be the same type of processor, such as one or more CPUs; or may be different types of processors such as one or more CPUs and one or more ASICs.
And a memory 406 for storing a program 410. Memory 406 may comprise high-speed RAM memory, and may also include non-volatile memory (non-volatile memory), such as at least one disk memory.
The program 410 may specifically be configured to cause the processor 402 to perform the following operations:
determining a plurality of data sources corresponding to the collation rules, and generating a data change message corresponding to a target data change when the target data change occurs in each of the plurality of data sources;
subscribing to data change messages of the plurality of data sources;
storing the data change message in a target storage space;
and performing data check on the data in the target storage space by using the check rule.
In an alternative embodiment, the program 410 may be specifically configured to cause the processor 402 to perform the following operations:
when the target data in the data source is changed, generating a data change message corresponding to the target data change according to the snapshot information of the changed target data.
In an alternative embodiment, the program 410 may be specifically configured to cause the processor 402 to perform the following operations:
determining a unique key corresponding to the data change message;
judging whether the target storage space has storage data corresponding to the unique key or not;
if so, storing the data change message in a target storage space in a mode of covering the data change message with the stored data corresponding to the unique key;
if not, directly storing the data change message in the target storage space.
In an alternative embodiment, the program 410 may be specifically configured to cause the processor 402 to perform the following operations:
determining a collation rule corresponding to the data change message;
determining a unique key determination mode corresponding to the data change message according to the check rule corresponding to the data change message;
and determining the unique key corresponding to the data change message by adopting the unique key determination mode.
In an alternative embodiment, the program 410 may be specifically configured to cause the processor 402 to perform the following operations:
and determining the unique key corresponding to the data change message according to the service table identifier, the service main key and/or the service ID in the data change message.
In an alternative embodiment, the program 410 may be specifically configured to cause the processor 402 to perform the following operations:
and determining the data dimension corresponding to the target data according to the checking rule.
In an optional implementation, the data dimension corresponding to the target data includes at least one of the following data dimensions: a data table dimension, a single data dimension, and/or a database dimension.
The algorithms and displays presented herein are not inherently related to any particular computer, virtual machine, or other apparatus. Various general purpose systems may also be used with the teachings herein. The required structure for constructing such a system will be apparent from the description above. Moreover, the present invention is not directed to any particular programming language. It is appreciated that a variety of programming languages may be used to implement the teachings of the present invention as described herein, and any descriptions of specific languages are provided above to disclose the best mode of the invention.
In the description provided herein, numerous specific details are set forth. It is understood, however, that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.
Similarly, it should be appreciated that in the foregoing description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. However, the disclosed method should not be interpreted as reflecting an intention that: that the invention as claimed requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of this invention.
Those skilled in the art will appreciate that the modules in the device in an embodiment may be adaptively changed and disposed in one or more devices different from the embodiment. The modules or units or components of the embodiments may be combined into one module or unit or component, and furthermore they may be divided into a plurality of sub-modules or sub-units or sub-components. All of the features disclosed in this specification (including any accompanying claims, abstract and drawings), and all of the processes or elements of any method or apparatus so disclosed, may be combined in any combination, except combinations where at least some of such features and/or processes or elements are mutually exclusive. Each feature disclosed in this specification (including any accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.
Furthermore, those skilled in the art will appreciate that while some embodiments described herein include some features included in other embodiments, rather than other features, combinations of features of different embodiments are meant to be within the scope of the invention and form different embodiments. For example, in the claims, any of the claimed embodiments may be used in any combination.
The various component embodiments of the invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It will be appreciated by those skilled in the art that a microprocessor or Digital Signal Processor (DSP) may be used in practice to implement some or all of the functions of some or all of the components of a data collating apparatus according to an embodiment of the present invention. The present invention may also be embodied as apparatus or device programs (e.g., computer programs and computer program products) for performing a portion or all of the methods described herein. Such programs implementing the present invention may be stored on computer-readable media or may be in the form of one or more signals. Such a signal may be downloaded from an internet website or provided on a carrier signal or in any other form.
It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by one and the same item of hardware. The usage of the words first, second and third, etcetera do not indicate any ordering. These words may be interpreted as names.
Claims (14)
1. A data collation method comprising:
acquiring a check rule, determining a plurality of data sources corresponding to the check rule, and generating a data change message corresponding to target data change when the target data change occurs in each data source in the plurality of data sources;
subscribing to data change messages of the plurality of data sources;
determining a check rule corresponding to the data change message, determining a unique key determination mode corresponding to the data change message according to the check rule corresponding to the data change message, determining a unique key corresponding to the data change message by adopting the unique key determination mode, and storing the data change message in a target storage space according to the unique key;
and performing data check on the data in the target storage space by using the check rule.
2. The method of claim 1, wherein generating a data change message corresponding to a target data change when the target data change occurs in the data source further comprises:
when the target data in the data source is changed, generating a data change message corresponding to the target data change according to the snapshot information of the changed target data.
3. The method of claim 1, wherein said storing the data change message in a target storage space further comprises:
determining a unique key corresponding to the data change message;
judging whether the target storage space has storage data corresponding to the unique key or not;
if so, storing the data change message in a target storage space in a mode of covering the data change message with the stored data corresponding to the unique key;
if not, directly storing the data change message in the target storage space.
4. The method of claim 3, wherein the determining a unique key corresponding to the data change message further comprises:
and determining the unique key corresponding to the data change message according to the service table identifier, the service main key and/or the service ID in the data change message.
5. The method according to any one of claims 1-3, wherein the method further comprises:
and determining the data dimension corresponding to the target data according to the checking rule.
6. The method of claim 5, wherein the data dimension corresponding to the target data comprises at least one of the following data dimensions: a data table dimension, a single data dimension, and/or a database dimension.
7. A data collating apparatus comprising:
the determining module is suitable for acquiring a check rule firstly, determining a plurality of data sources corresponding to the check rule, and generating a data change message corresponding to target data change when the target data change occurs in each data source in the plurality of data sources;
a subscription module adapted to subscribe to data change messages of the plurality of data sources;
the storage module is suitable for determining a check rule corresponding to the data change message, determining a unique key determining mode corresponding to the data change message according to the check rule corresponding to the data change message, determining a unique key corresponding to the data change message by adopting the unique key determining mode, and storing the data change message in a target storage space according to the unique key;
and the checking module is suitable for performing data checking on the data in the target storage space by using the checking rule.
8. The apparatus of claim 7, wherein the determination module is further adapted to:
when the target data in the data source is changed, generating a data change message corresponding to the target data change according to the snapshot information of the changed target data.
9. The apparatus of claim 7, wherein the storage module is further adapted to: determining a unique key corresponding to the data change message;
judging whether the target storage space has storage data corresponding to the unique key or not;
if so, storing the data change message in a target storage space in a mode of covering the data change message with the stored data corresponding to the unique key;
if not, directly storing the data change message in the target storage space.
10. The apparatus of claim 9, wherein the storage module is further adapted to: and determining the unique key corresponding to the data change message according to the service table identifier, the service main key and/or the service ID in the data change message.
11. The apparatus of any one of claims 7-10, wherein the apparatus further comprises:
and the dimension determining module is suitable for determining the data dimension corresponding to the target data according to the checking rule.
12. The apparatus of claim 11, wherein the data dimension to which the target data corresponds comprises at least one of the following data dimensions: a data table dimension, a single data dimension, and/or a database dimension.
13. A computing device, comprising: the system comprises a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface complete mutual communication through the communication bus;
the memory is used for storing at least one executable instruction, and the executable instruction causes the processor to execute the operation corresponding to the data checking method according to any one of claims 1-6.
14. A computer storage medium having stored therein at least one executable instruction for causing a processor to perform operations corresponding to the data collation method according to any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910465753.2A CN110175182B (en) | 2019-05-30 | 2019-05-30 | Data checking method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910465753.2A CN110175182B (en) | 2019-05-30 | 2019-05-30 | Data checking method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110175182A CN110175182A (en) | 2019-08-27 |
| CN110175182B true CN110175182B (en) | 2021-07-02 |
Family
ID=67696872
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910465753.2A Active CN110175182B (en) | 2019-05-30 | 2019-05-30 | Data checking method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110175182B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111309331B (en) * | 2020-02-11 | 2022-11-18 | 支付宝(杭州)信息技术有限公司 | Business processing method and device, electronic equipment and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101216832A (en) * | 2007-12-28 | 2008-07-09 | 腾讯科技（深圳）有限公司 | Data synchronization process and device |
| CN105740465A (en) * | 2016-03-04 | 2016-07-06 | 浪潮软件集团有限公司 | Flexible custom comparison method |
| CN107015991A (en) * | 2016-01-28 | 2017-08-04 | 阿里巴巴集团控股有限公司 | Self checking method, device, system and the service apparatus of data consistency |
| CN108228618A (en) * | 2016-12-14 | 2018-06-29 | 平安科技（深圳）有限公司 | The method and apparatus of document verification of data |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101968793B (en) * | 2010-08-25 | 2012-09-05 | 大唐软件技术股份有限公司 | Method and system for checking on basis of disparate data source data |
| CN106326219B (en) * | 2015-06-16 | 2020-01-24 | 阿里巴巴集团控股有限公司 | Method, device and system for checking business system data |
| CN106161138A (en) * | 2016-06-17 | 2016-11-23 | 贵州电网有限责任公司贵阳供电局 | A kind of intelligence automatic gauge method and device |
| CN109597828B (en) * | 2018-09-29 | 2023-04-25 | 蚂蚁金服（杭州）网络技术有限公司 | Offline data checking method, device and server |
| CN109299222A (en) * | 2018-09-29 | 2019-02-01 | 阿里巴巴集团控股有限公司 | Verification of data method and device |
-
2019
- 2019-05-30 CN CN201910465753.2A patent/CN110175182B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101216832A (en) * | 2007-12-28 | 2008-07-09 | 腾讯科技（深圳）有限公司 | Data synchronization process and device |
| CN107015991A (en) * | 2016-01-28 | 2017-08-04 | 阿里巴巴集团控股有限公司 | Self checking method, device, system and the service apparatus of data consistency |
| CN105740465A (en) * | 2016-03-04 | 2016-07-06 | 浪潮软件集团有限公司 | Flexible custom comparison method |
| CN108228618A (en) * | 2016-12-14 | 2018-06-29 | 平安科技（深圳）有限公司 | The method and apparatus of document verification of data |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110175182A (en) | 2019-08-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109104336B (en) | Service request processing method and device, computer equipment and storage medium | |
| CN109657434B (en) | Application access method and device | |
| CN110968478A (en) | Log collection method, server and computer storage medium | |
| CN106341280A (en) | Service processing method and device | |
| CN104424352A (en) | System and method for providing agent service to user terminal | |
| CN110175182B (en) | Data checking method and device | |
| CN111475532A (en) | Data processing optimization method and device, storage medium and terminal | |
| CN108897858B (en) | Distributed cluster index fragmentation evaluation method and device and electronic equipment | |
| CN110955390A (en) | Data processing method and device and electronic equipment | |
| CN108712486B (en) | Workload proving method and device | |
| CN110889073A (en) | Page request response method, server and computer storage medium | |
| CN110209512B (en) | Data checking method and device based on multiple data sources | |
| CN110769020B (en) | Resource request processing method, device, equipment and system | |
| CN108848183B (en) | Login method and device for simulation user | |
| CN114520808A (en) | Request processing method and device, electronic equipment and computer readable storage medium | |
| CN107832124B (en) | Method and device for processing task in file processing system | |
| CN111222783A (en) | Method for processing workflow and electronic equipment | |
| US20240095116A1 (en) | Early detection and avoidance of mainframe job errors | |
| CN114884974B (en) | Data multiplexing method, system and computing device | |
| CN111444057A (en) | Page performance data acquisition method and device and computing equipment | |
| CN112905602B (en) | Data comparison method, computing device and computer storage medium | |
| CN111209292B (en) | Bin data management method, device and system | |
| CN113656549B (en) | Content searching method of electronic book, electronic device and computer storage medium | |
| CN117331600A (en) | Data interaction self-adaption method for heterogeneous software system | |
| CN112468541B (en) | Data processing method, device and system based on Internet of things |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |