CN114138786A - Method, device, medium, product and equipment for duplicate removal of online transaction message - Google Patents
Method, device, medium, product and equipment for duplicate removal of online transaction message Download PDFInfo
- Publication number
- CN114138786A CN114138786A CN202111452832.3A CN202111452832A CN114138786A CN 114138786 A CN114138786 A CN 114138786A CN 202111452832 A CN202111452832 A CN 202111452832A CN 114138786 A CN114138786 A CN 114138786A
- Authority
- CN
- China
- Prior art keywords
- message
- vector table
- online transaction
- value
- hash value
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
- G06F16/2255—Hash tables
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
- G06F16/2237—Vectors, bitmaps or matrices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/252—Integrating or interfacing systems involving database management systems between a Database Management System and a front-end application
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q40/00—Finance; Insurance; Tax strategies; Processing of corporate or income taxes
- G06Q40/04—Trading; Exchange, e.g. stocks, commodities, derivatives or currency exchange
Abstract
The present disclosure relates to an online transaction message deduplication method, apparatus, medium, product, and device. And caching the message in a key-value pair database to reconfigure the list, and storing the hash value corresponding to the primary key value designated by the received online transaction message by utilizing a first bit vector table in a first vector table filter. And when the online transaction message is newly received, obtaining a message de-reconfiguration list from the key value pair database, and determining a corresponding message de-reconfiguration index main key. And determining whether the first hash value corresponding to the primary key value is stored, and if so, indicating not to process the new online transaction message to realize message deduplication. Because the key value pair database can provide the message de-reconfiguration list with lower delay and faster response speed, and the vector table filter can realize the fast lookup of the bit vector table, the timeliness of the message de-reconfiguration can be effectively ensured.
Description
Technical Field
The present disclosure relates to the field of data processing technologies, and in particular, to a method, an apparatus, a medium, a product, and a device for removing duplicate of an online transaction message.
Background
This section is intended to provide a background or context to the embodiments of the disclosure recited in the claims. The description herein is not admitted to be prior art by inclusion in this section.
With the development of information systems, the message interaction mode between online transaction systems is more and more multi-sourced. With the intervention of technologies such as block chains, Kafka distributed message publishing and the like, great convenience is provided for the establishment of the online trading system, but the problem of message repetition is also brought. For example, assuming that there are A, B two nodes in the blockchain, a message for an online transaction is sent to the blockchain a node, the blockchain synchronizes the message to the B node through a consensus mechanism, and both A, B nodes send the message downstream in the cluster mode, thereby causing a problem of message duplication.
For online transaction systems, repeating messages can introduce unnecessary misunderstandings. Particularly for the financial transaction, repeated messages may cause a decrease in transaction success rate, even a wrong account, and cause great troubles.
The current message deduplication mainly comprises the following two schemes:
the first scheme is as follows:
a database unique constraint technique is used. The system stores the received information of each online transaction in a database, and establishes a main key of the information through one attribute or a plurality of attributes of the information. Abnormal errors may occur if messages are repeatedly inserted into the database.
By using the unique constraint technology of the database, if the repeated judgment logic is not added in the system, error logs at the database system level are thrown out, and if the abnormal error report is not processed in the system, the error is directly returned to the user, so that the user experience is unfriendly. If a repeated judgment logic is added into the system, different online transaction main keys in the same system may be different, so that on one hand, the code amount is greatly increased, and the workload of a developer is increased, and on the other hand, the maintainability of the system is also influenced due to the dispersed codes. And meanwhile, the online transaction message is frequently interacted with the database, and the timeliness of online transaction message duplicate removal cannot be guaranteed.
The second scheme is as follows:
adding an interceptor to all the requested Uniform Resource Locators (URLs) in the system, storing the requested URLs into Session control (Session) each time, verifying whether the data of the URLs are the same at the next request, and if so, denying access.
However, adding interceptors based on Session has certain limitations, the server may have several or even dozens of services at the same time, and if the access volume is large, the server has a high requirement on the performance of the server, and the timeliness of removing duplicate of the online transaction message cannot be guaranteed. Meanwhile, as the memory of the server cannot be shared among a plurality of servers, the expansibility of the scheme is influenced.
Therefore, it is desirable to provide an online transaction message deduplication mechanism, which can ensure timeliness when performing message deduplication.
Disclosure of Invention
The embodiment of the disclosure provides an online transaction message deduplication method, an online transaction message deduplication device, an online transaction message deduplication medium, an online transaction message deduplication product and online transaction message deduplication equipment, which are used for solving the problem of poor timeliness of online transaction message deduplication.
In a first aspect, the present disclosure provides an online transaction message deduplication method, the method comprising:
if an online transaction message is received, acquiring a message de-reconfiguration list cached by a key value pair database, wherein the message de-reconfiguration list comprises at least one online transaction type number which is configured in advance and at least one message de-duplication index main key which corresponds to each online transaction type number respectively;
searching at least one message duplication removal index main key corresponding to the corresponding online transaction type number from the message duplication removal list according to the online transaction type corresponding to the online transaction message;
according to at least one found message duplicate removal index primary key, each primary key value corresponding to the online transaction message is obtained;
determining a first hash value according to each obtained primary key value, searching a first bit vector table from a first vector table filter, and determining whether a bit corresponding to the first hash value is a designated value;
and if the bit corresponding to the first hash value is determined to be the designated value, determining the online transaction message to be a repeated message, and indicating not to process the online transaction message.
Optionally, the method further includes:
if the bit corresponding to the first hash value is determined not to be the designated value, determining a second hash value according to each obtained primary key value, searching a second bit vector table from a second vector table filter, and determining whether the bit corresponding to the second hash value is the designated value;
and if the bit corresponding to the second hash value is determined to be the designated value, determining that the online transaction message is a repeated message, and indicating not to process the online transaction message.
Optionally, the method further includes:
if the bit corresponding to the second hash value is determined not to be the designated value, determining that the online transaction message is not a repeated message, and indicating to process the online transaction message; and the number of the first and second electrodes,
updating the bits corresponding to the first hash value in the first bit vector table to a specified value, and updating the bits corresponding to the second hash value in the second bit vector table to a specified value.
Optionally, the first vector table filter and the second vector table filter perform timing asynchronous reconstruction according to a set time length;
looking up a first bit vector table from a first vector table filter, comprising:
searching a first bit vector table from a first vector table filter corresponding to the current moment;
looking up a second bit vector table from a second vector table filter, comprising:
and searching a second bit vector table from a second vector table filter corresponding to the current moment.
Optionally, determining the first hash value according to each obtained primary key value includes:
carrying out encryption compression on each obtained primary key value through an information abstract algorithm to obtain a first character string;
and carrying out Hash operation on the first character string to obtain a first Hash value.
Optionally, determining the second hash value according to each obtained primary key value includes:
multiplying and summing the obtained appointed code of each character of each primary key value and an appointed prime number to obtain a second character string;
and carrying out Hash operation on the second character string to obtain a second Hash value.
In a second aspect, the present disclosure also provides an online transaction message deduplication apparatus, the apparatus comprising:
the acquisition module is used for acquiring a message de-reconfiguration list cached by a key value pair database if an online transaction message is received, wherein the message de-reconfiguration list comprises at least one online transaction type number which is configured in advance and at least one message de-duplication index main key which corresponds to each online transaction type number respectively;
the main key searching module is used for searching at least one message duplication removal index main key corresponding to the corresponding online transaction type number from the message duplication removal list according to the online transaction type corresponding to the online transaction message;
a primary key value obtaining module, configured to obtain each primary key value corresponding to the online transaction message according to the found at least one message duplicate removal index primary key;
the vector table searching module is used for determining a first hash value according to each obtained primary key value, searching a first bit vector table from a first vector table filter, and determining whether a bit corresponding to the first hash value is a designated value;
and the processing module is used for determining that the online transaction message is a repeated message and indicating that the online transaction message is not processed if the bit corresponding to the first hash value is determined to be the designated value.
Optionally, the vector table lookup module is further configured to determine a second hash value according to each obtained primary key value if it is determined that the bit corresponding to the first hash value is not the designated value, search a second bit vector table from a second vector table filter, and determine whether the bit corresponding to the second hash value is the designated value;
and the processing module is further configured to determine that the online transaction message is a repeated message if it is determined that the bit corresponding to the second hash value is the designated value, and indicate that the online transaction message is not to be processed.
Optionally, the processing module is further configured to determine that the online transaction message is not a duplicate message and instruct to process the online transaction message if it is determined that the bit corresponding to the second hash value is not the specified value; and updating the bits corresponding to the first hash value in the first bit vector table to a specified value, and updating the bits corresponding to the second hash value in the second bit vector table to a specified value.
Optionally, the first vector table filter and the second vector table filter perform timing asynchronous reconstruction according to a set time length;
the vector table lookup module, which looks up the first bit vector table from the first vector table filter, includes: searching a first bit vector table from a first vector table filter corresponding to the current moment; looking up a second bit vector table from a second vector table filter, comprising: and searching a second bit vector table from a second vector table filter corresponding to the current moment.
Optionally, the determining, by the vector table lookup module, a first hash value according to each obtained primary key value includes: carrying out encryption compression on each obtained primary key value through an information abstract algorithm to obtain a first character string; and carrying out Hash operation on the first character string to obtain a first Hash value.
Optionally, the determining, by the vector table lookup module, a second hash value according to each obtained primary key value includes: multiplying and summing the obtained appointed code of each character of each primary key value and an appointed prime number to obtain a second character string; and carrying out Hash operation on the second character string to obtain a second Hash value.
In a third aspect, the present disclosure also provides a computer program product comprising an executable program which is executed by a processor to implement the method as described above.
In a fourth aspect, the present disclosure also provides a non-volatile computer storage medium storing an executable program that is executed by a processor to implement the method as described above.
In a fifth aspect, the present disclosure further provides an online transaction message deduplication device, including a processor, a communication interface, a memory, and a communication bus, where the processor, the communication interface, and the memory complete communication with each other through the communication bus;
the memory is used for storing a computer program;
the processor, when executing the program stored in the memory, is configured to implement the method steps as described above.
According to the scheme provided by the embodiment of the disclosure, the configurable message reconfiguration list can be cached in the key-value pair database, and the hash value corresponding to the primary key value designated by the received online transaction message can be stored by using the first bit vector table in the first vector table filter. Thus, when the online transaction message is received, the message de-reconfiguration list can be obtained from the key value pair database, and the corresponding message de-reconfiguration index main key is determined according to the type of the online transaction message. And then, according to the first hash value corresponding to the primary key value, searching a first bit vector table in the first vector table filter to determine whether the first hash value has been stored. If the first hash value is determined to be stored, the online transaction message received this time can be determined to be a repeated message, and the online transaction message can be instructed not to be processed, so that the deduplication of the online transaction message is realized. Because the key value pair database can provide the message de-reconfiguration list with lower delay and faster response speed, and the vector table filter can realize the fast lookup of the bit vector table, the timeliness of the online transaction message de-reconfiguration can be effectively ensured.
Additional features and advantages of the disclosure will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by the practice of the disclosure. The objectives and other advantages of the disclosure may be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.
Drawings
In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present disclosure, and other drawings can be obtained by those skilled in the art without creative efforts.
Fig. 1 is a schematic flow chart illustrating an online transaction message deduplication method according to an embodiment of the disclosure;
FIG. 2 is a schematic diagram illustrating an exemplary embodiment of an apparatus for removing duplicate online transaction messages;
fig. 3 is a schematic structural diagram of an online transaction message deduplication device according to an embodiment of the disclosure.
Detailed Description
To make the objects, technical solutions and advantages of the present disclosure clearer, the present disclosure will be described in further detail with reference to the accompanying drawings, and it is apparent that the described embodiments are only a part of the embodiments of the present disclosure, not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments disclosed herein without making any creative effort, shall fall within the protection scope of the present disclosure.
It should be noted that, the "plurality" or "a plurality" mentioned herein means two or more. "and/or" describes the association relationship of the associated objects, meaning that there may be three relationships, e.g., a and/or B, which may mean: a exists alone, A and B exist simultaneously, and B exists alone. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship.
The terms "first," "second," and the like in the description and in the claims of the present disclosure and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the disclosure described herein are capable of operation in sequences other than those illustrated or otherwise described herein.
Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
According to the technical scheme, the data acquisition, storage, use, processing and the like meet relevant regulations of national laws and regulations.
In order to ensure timeliness of online transaction message deduplication, the application proposes that message deduplication can be realized by using a key-value pair database and a vector table filter together.
Reference herein to a key-value pair database may be understood as a database storing data as a set of key-value pairs, with keys as unique identifiers. The key-value pair database mentioned in the present application may be, but is not limited to, a remote dictionary service (redis) database or a memcache database, etc.
The vector table filter mentioned in the present application may be understood as a filter that implements data saving by setting corresponding bits in the bit vector table to a specified value (e.g., 1) according to the hash value of data, and may perform filtering of saved data according to the bit vector table. The vector table filter mentioned in the present application may be, but is not limited to, a bloom filter or a cuckoo filter, etc.
Based on the above description, the embodiments of the present disclosure provide an online transaction message deduplication method, where the flow of the steps of the method may be as shown in fig. 1, and the method includes:
step 101, receiving an online transaction message.
In this step, an online transaction message may be received, and if so, step 102 may be continued.
And 102, acquiring a message cached by the key value pair database to reconfigure the list.
In view of the advantages of low latency and fast response speed of a key-value pair database, such as a redis database, in this embodiment, the message reconfiguration list may be stored in the key-value pair database, so as to ensure high availability of the system. In this step, the cached message de-reconfiguration list may be obtained from the key-value pair database. The key value pair message reconfiguration list in the database can be configured according to the requirement.
In view of the fact that there may be multiple online transaction types in a system, such as a pay bank payment transaction, a wechat payment transaction, a bank card payment transaction, and the like, in this embodiment, at least one online transaction type number may be configured in the message reconfiguration list, and for each online transaction type number, a corresponding at least one message deduplication index main key may be configured.
It can be understood that, in this embodiment, a message de-duplication list may be used to support de-duplication of messages of multiple types of online transactions, so as to implement reusable, configurable, efficient and stable message de-duplication.
And 103, searching at least one message duplication removal index main key corresponding to the corresponding online transaction type number from the message duplication removal list according to the online transaction type corresponding to the online transaction message.
In this step, the received online transaction message may be analyzed, and according to the corresponding online transaction type, the corresponding message deduplication index key may be searched from the obtained message deduplication reconfiguration list.
And step 104, according to the searched at least one message duplicate removal index primary key, obtaining each primary key value corresponding to the online transaction message.
In this step, the primary key value corresponding to the found primary key may be obtained according to the analysis of the received online transaction message. Therefore, the uniqueness and the repeatability of the online transaction can be judged subsequently according to the obtained primary key values, and the system disorder caused by the repeated transaction is avoided.
And 105, determining a first hash value according to each obtained primary key value.
Considering that when a large amount of data is stored in the database by using the key value pair, the requirement on the memory space is linearly increased, the cost is higher, the data storage of the vector table filter saves space, the complete data does not need to be stored, only the hash value of the data needs to be determined, and then the corresponding bit in the bit vector table is set to be a specified value (such as 1) to realize the data storage. And the data search speed of the vector table filter is high, and the data search can be realized only by determining the hash value of the data to be searched and then determining whether the corresponding bit in the bit vector table is set to be a specified value (such as 1). Therefore, in the present embodiment, the bit vector table of the vector table filter may be used to store the hash value corresponding to the primary key value specified by the received online transaction message.
In this step, a first hash value may be determined according to each obtained primary key value for a newly received online transaction message.
The first hash value may be determined in any manner as long as it is determined in the same manner as the hash value stored in the bit vector table of the vector table filter.
Step 106, searching the first bit vector table from the first vector table filter, and determining whether the bit corresponding to the first hash value is a specified value.
In this step, a first bit vector table of the first vector table filter for storing the hash value corresponding to the primary key value specified by the received online transaction message may be searched, and it may be determined whether the bit corresponding to the first hash value determined in step 105 is a specified value, for example, 1.
If the determination is a specified value, the online transaction message received this time can be determined to be a repeat message, and step 107 is executed, and after step 107, the process is ended. Otherwise, it may be determined that the received online transaction message is not a duplicate message, step 108 may be performed, and after step 108, the process is ended.
Step 107, indicating that the online transaction message is not to be processed.
In this step, it may be indicated that the received online transaction message is not processed, so as to implement message deduplication.
And 108, indicating to process the online transaction message, and updating the bit corresponding to the first hash value in the first bit vector table to a specified value.
In this step, the processing of the currently received online transaction message may be instructed, and the first bit vector table may be updated by using the first hash value corresponding to the currently received online transaction message, so that the updated first bit vector table may be subsequently used to deduplicate the same message as the currently received online transaction message.
In a possible implementation manner, it is considered that the vector table filter has a certain probability of erroneous determination, and the characteristic of the erroneous determination is that the existing filter has a certain existence, and the nonexistence is not necessarily absent, that is, if it is determined that the bit corresponding to the first hash value is not a specified value, the result may be a result of the erroneous determination. Therefore, in this embodiment, when it is determined that the bit corresponding to the first hash value is not the specified value, after step 106, step 108' may be continued to perform further message deduplication using the second vector table filter:
and 108', determining a second hash value according to each obtained primary key value.
The second hash value may be determined in any manner as long as it is determined in the same manner as the hash value stored in the second bit vector table of the second vector table filter.
In a possible implementation manner, in order to further improve the accuracy of message deduplication, the second bit vector table of the second vector table filter may store a hash value in a different manner than the hash value stored in the second bit vector table of the first vector table filter, and in this case, the second hash value may also be determined in a different manner than the first hash value.
For example, in one possible implementation, the first hash value may be obtained, but is not limited to, by:
performing encryption compression on each obtained primary key value through an information digest algorithm (for example, an MD5 algorithm) to obtain a first character string; and carrying out Hash operation on the first character string to obtain a first Hash value.
The second hash value may be obtained by, but is not limited to:
multiplying and summing the obtained appointed code of each character of each primary key value and an appointed prime number to obtain a second character string; and carrying out Hash operation on the second character string to obtain a second Hash value.
Step 109', look up the second bit vector table from the second vector table filter, and determine whether the bit corresponding to the second hash value is a specified value.
Similarly to step 106, in this step, a second bit vector table of the second vector table filter for storing the hash value corresponding to the primary key value specified by the received online transaction message may be searched, and it is determined whether the bit corresponding to the second hash value determined in step 108' is a specified value, for example, 1.
If the determination is a designated value, the online transaction message received this time can be determined to be a repeat message, and step 110 'is executed, and after step 110', the process is ended. Otherwise, it may be determined that the received online transaction message is not a duplicate message, step 111 'may be executed, and after step 111', the process is ended.
Step 110' indicates that the online transaction message is not to be processed.
In this step, it may be indicated that the received online transaction message is not processed, so as to implement message deduplication. That is, by setting two vector table filters, when the first vector table filter does not determine that there is duplication of a message, it can be determined that there is duplication of a message by the second vector table filter.
And step 111', indicating to process the online transaction message, updating the bit corresponding to the first hash value in the first bit vector table to a specified value, and updating the bit corresponding to the second hash value in the second bit vector table to the specified value.
In this embodiment, in order to ensure timeliness of message deduplication, two vector table filters are set for description, but of course, more vector table filters may be set for message deduplication, and this embodiment will not be further described.
If the existence of message repetition is not judged according to the two set vector table filters, in the step, the online transaction message received this time can be indicated to be processed, the first bit vector table is updated by using the first hash value corresponding to the online transaction message received this time, and the second bit vector table is updated by using the second hash value, so that the same message as the online transaction message received this time can be deduplicated subsequently by further using the updated first bit vector table and second bit vector table.
In addition, considering that the misjudgment probability of the vector table filter is related to the amount of data stored therein, the misjudgment probability is increased when the amount of data stored therein approaches the estimated maximum capacity. Therefore, the vector table filter, such as the bloom filter, can be subjected to timing asynchronous reconstruction, and the latest vector table filter is used for replacing the original vector table filter, so that the data volume in the vector table filter is prevented from being too large.
In one possible implementation, the first vector filter and the second vector filter may both perform timed asynchronous reconstruction according to a set duration. At this time, looking up the first bit vector table from the first vector table filter may be understood as including: and searching the first bit vector table from the first vector table filter corresponding to the current moment. Looking up the first bit vector table from the first vector table filter may be understood as including: and searching a second bit vector table from a second vector table filter corresponding to the current moment. Therefore, the accuracy of message deduplication can be improved, and message deduplication can be performed within a specified time range.
Furthermore, considering that the misjudgment probability is small when the capacity of the vector table filter is a specified multiple of the stored data amount, for example, 5 to 10 times, in this embodiment, the capacity of the vector table filter may be determined according to the set time length for asynchronous reconstruction of the vector table filter. That is, the capacity of the vector table filter may be determined based on the online transaction traffic for a timed duration (e.g., one day or one week) during which the vector table filter performs asynchronous rebuilds. Therefore, the capacity of the vector table filter can meet the specified multiple of the stored data quantity as much as possible within the timing duration, so that the misjudgment probability of the vector table filter is further reduced, and the accuracy of information duplicate removal is improved.
It should be noted that if two vector table filters are set to perform message deduplication, by controlling the misjudgment probabilities of the two set vector table filters, for example, if the misjudgment probabilities of the first vector table filter and the second vector table filter can be both controlled to be 0.01, the accuracy of the whole message deduplication process can be greatly improved.
According to the scheme provided by the invention, when various types of online transactions exist in the same system, reusable, configurable, efficient and stable online transaction message deduplication can be provided, and an application logic layer does not need to be modified. The high performance of the key-value pair database can ensure the high availability of the system, and the key-value pair database can be realized by adopting a cluster so as to further improve the reliability of the system. The flexibility of judging the uniqueness of the multi-type online transaction messages of the system is enhanced through configuration, the functions of supporting a multi-message mode and judging the uniqueness of the multi-online transaction messages are effectively integrated into the system, the expandability and the maintainability are strong, and when the online transaction messages of more sources are added later, the message duplication can be removed only through configuration, and the uniqueness of the messages is kept.
Through the uniqueness and the repeatability judgment of the online transaction message, the service confusion caused by repeated transactions can be avoided, and the message deduplication scheme with quick response and short time consumption can be provided based on the characteristic that the online transaction service needs higher real-time performance. Moreover, with the increase of the online transaction volume, the requirement of larger traffic volume increase can be met only by increasing less hardware cost. By controlling the misjudgment rate of the message duplicate removal process, the stability of the online transaction system can be effectively improved.
Corresponding to the provided method, the following device is further provided.
The embodiment of the present disclosure provides an online transaction message duplication eliminating apparatus, which may be configured as shown in fig. 2, and includes:
the obtaining module 11 is configured to obtain a message de-reconfiguration list cached by the key-value pair database if an online transaction message is received, where the message de-reconfiguration list includes at least one online transaction type number configured in advance, and at least one message de-duplication index main key corresponding to each online transaction type number;
the primary key searching module 12 is configured to search, according to the online transaction type corresponding to the online transaction message, at least one message duplication removal index primary key corresponding to the corresponding online transaction type number from the message duplication removal configuration list;
the primary key value obtaining module 13 is configured to obtain each primary key value corresponding to the online transaction message according to the found at least one message duplicate removal index primary key;
the vector table lookup module 14 is configured to determine a first hash value according to each obtained primary key value, lookup a first bit vector table from a first vector table filter, and determine whether a bit corresponding to the first hash value is a specified value;
the processing module 15 is configured to determine that the online transaction message is a duplicate message if it is determined that the bit corresponding to the first hash value is a specified value, and indicate not to process the online transaction message.
Optionally, the vector table lookup module 14 is further configured to determine a second hash value according to each obtained primary key value if it is determined that the bit corresponding to the first hash value is not the designated value, search a second bit vector table from a second vector table filter, and determine whether the bit corresponding to the second hash value is the designated value;
the processing module 15 is further configured to determine that the online transaction message is a duplicate message if it is determined that the bit corresponding to the second hash value is a specified value, and indicate not to process the online transaction message.
Optionally, the processing module 15 is further configured to determine that the online transaction message is not a duplicate message and instruct to process the online transaction message if it is determined that the bit corresponding to the second hash value is not the specified value; and updating the bits corresponding to the first hash value in the first bit vector table to a specified value, and updating the bits corresponding to the second hash value in the second bit vector table to a specified value.
Optionally, the first vector table filter and the second vector table filter perform timing asynchronous reconstruction according to a set time length;
the vector table lookup module 14 looks up a first bit vector table from the first vector table filter, including: searching a first bit vector table from a first vector table filter corresponding to the current moment; looking up a second bit vector table from a second vector table filter, comprising: and searching a second bit vector table from a second vector table filter corresponding to the current moment.
Optionally, the determining, by the vector table lookup module 14, a first hash value according to each obtained primary key value includes: carrying out encryption compression on each obtained primary key value through an information abstract algorithm to obtain a first character string; and carrying out Hash operation on the first character string to obtain a first Hash value.
Optionally, the determining, by the vector table lookup module 14, a second hash value according to each obtained primary key value includes: multiplying and summing the obtained appointed code of each character of each primary key value and an appointed prime number to obtain a second character string; and carrying out Hash operation on the second character string to obtain a second Hash value.
The functions of the functional units of each device provided in the above embodiments of the present disclosure may be implemented by the steps of the corresponding methods, and therefore, possible working processes and beneficial effects of the functional units in each device provided in the embodiments of the present disclosure are not repeated herein.
Based on the same inventive concept, the embodiments of the present disclosure provide the following apparatus and medium.
The disclosed embodiment provides an online transaction message deduplication device, which may have a structure as shown in fig. 3, and includes a processor 21, a communication interface 22, a memory 23, and a communication bus 24, where the processor 21, the communication interface 22, and the memory 23 complete mutual communication through the communication bus 24;
the memory 23 is used for storing computer programs;
the processor 21 is configured to implement the steps of the above method embodiments of the present disclosure when executing the program stored in the memory.
Alternatively, the processor 21 may include a Central Processing Unit (CPU), an Application Specific Integrated Circuit (ASIC), one or more Integrated circuits for controlling program execution, a hardware Circuit developed by using a Field Programmable Gate Array (FPGA), and a baseband processor.
Optionally, the processor 21 may include at least one processing core.
Alternatively, the Memory 23 may include a Read-Only Memory (ROM), a Random Access Memory (RAM), and a disk Memory. The memory 23 is used for storing data required by the at least one processor 21 during operation. The number of the memory 23 may be one or more.
The embodiment of the present disclosure also provides a non-volatile computer storage medium, which stores an executable program, and when the executable program is executed by a processor, the method provided by the above method embodiment of the present disclosure is implemented.
In a possible implementation, the computer storage medium may include: various storage media capable of storing program codes, such as a Universal Serial Bus Flash Drive (USB), a mobile hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
The embodiment of the present disclosure further provides a computer program product, where the computer program product includes an executable program, and the executable program is executed by a processor to implement the method provided by the above method embodiment of the present disclosure.
In the disclosed embodiments, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the above-described embodiments of the apparatus are merely illustrative, and for example, the described unit or division of units is only one division of logical functions, and there may be other divisions when actually implemented, for example, a plurality of units or components may be combined or may be integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical or other form.
The functional units in the embodiments of the present disclosure may be integrated into one processing unit, or each unit may be an independent physical module.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, all or part of the technical solutions of the embodiments of the present disclosure may be embodied in the form of a software product, which is stored in a storage medium and includes several instructions to enable a computer device, such as a personal computer, a server, or a network device, or a processor (processor), to execute all or part of the steps of the methods described in the embodiments of the present disclosure. And the aforementioned storage medium includes: various media capable of storing program codes, such as a Universal Serial Bus Flash Drive (usb Flash Drive), a removable hard disk, a ROM, a RAM, a magnetic disk, or an optical disk.
As will be appreciated by one skilled in the art, embodiments of the present disclosure may be provided as a method, system, or computer program product. Accordingly, the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present disclosure may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and so forth) having computer-usable program code embodied therein.
The present disclosure is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the disclosure. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
While preferred embodiments of the present disclosure have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, it is intended that the appended claims be interpreted as including preferred embodiments and all alterations and modifications as fall within the scope of the disclosure.
It will be apparent to those skilled in the art that various changes and modifications can be made in the present disclosure without departing from the spirit and scope of the disclosure. Thus, if such modifications and variations of the present disclosure fall within the scope of the claims of the present disclosure and their equivalents, the present disclosure is intended to include such modifications and variations as well.
Claims (15)
1. A method for deduplication of online transaction messages, the method comprising:
if an online transaction message is received, acquiring a message de-reconfiguration list cached by a key value pair database, wherein the message de-reconfiguration list comprises at least one online transaction type number which is configured in advance and at least one message de-duplication index main key which corresponds to each online transaction type number respectively;
searching at least one message duplication removal index main key corresponding to the corresponding online transaction type number from the message duplication removal list according to the online transaction type corresponding to the online transaction message;
according to at least one found message duplicate removal index primary key, each primary key value corresponding to the online transaction message is obtained;
determining a first hash value according to each obtained primary key value, searching a first bit vector table from a first vector table filter, and determining whether a bit corresponding to the first hash value is a designated value;
and if the bit corresponding to the first hash value is determined to be the designated value, determining the online transaction message to be a repeated message, and indicating not to process the online transaction message.
2. The method of claim 1, wherein the method further comprises:
if the bit corresponding to the first hash value is determined not to be the designated value, determining a second hash value according to each obtained primary key value, searching a second bit vector table from a second vector table filter, and determining whether the bit corresponding to the second hash value is the designated value;
and if the bit corresponding to the second hash value is determined to be the designated value, determining that the online transaction message is a repeated message, and indicating not to process the online transaction message.
3. The method of claim 2, wherein the method further comprises:
if the bit corresponding to the second hash value is determined not to be the designated value, determining that the online transaction message is not a repeated message, and indicating to process the online transaction message; and the number of the first and second electrodes,
updating the bits corresponding to the first hash value in the first bit vector table to a specified value, and updating the bits corresponding to the second hash value in the second bit vector table to a specified value.
4. The method of claim 2, wherein the first vector filter and the second vector filter are periodically asynchronously reconstructed for a set duration;
looking up a first bit vector table from a first vector table filter, comprising:
searching a first bit vector table from a first vector table filter corresponding to the current moment;
looking up a second bit vector table from a second vector table filter, comprising:
and searching a second bit vector table from a second vector table filter corresponding to the current moment.
5. The method of any one of claims 1 to 4, wherein determining the first hash value according to each obtained primary key value comprises:
carrying out encryption compression on each obtained primary key value through an information abstract algorithm to obtain a first character string;
and carrying out Hash operation on the first character string to obtain a first Hash value.
6. The method of claim 5, wherein determining the second hash value from each obtained primary key value comprises:
multiplying and summing the obtained appointed code of each character of each primary key value and an appointed prime number to obtain a second character string;
and carrying out Hash operation on the second character string to obtain a second Hash value.
7. An online transaction message deduplication apparatus, the apparatus comprising:
the acquisition module is used for acquiring a message de-reconfiguration list cached by a key value pair database if an online transaction message is received, wherein the message de-reconfiguration list comprises at least one online transaction type number which is configured in advance and at least one message de-duplication index main key which corresponds to each online transaction type number respectively;
the main key searching module is used for searching at least one message duplication removal index main key corresponding to the corresponding online transaction type number from the message duplication removal list according to the online transaction type corresponding to the online transaction message;
a primary key value obtaining module, configured to obtain each primary key value corresponding to the online transaction message according to the found at least one message duplicate removal index primary key;
the vector table searching module is used for determining a first hash value according to each obtained primary key value, searching a first bit vector table from a first vector table filter, and determining whether a bit corresponding to the first hash value is a designated value;
and the processing module is used for determining that the online transaction message is a repeated message and indicating that the online transaction message is not processed if the bit corresponding to the first hash value is determined to be the designated value.
8. The apparatus of claim 7, wherein the vector table lookup module is further configured to determine a second hash value according to each obtained primary key value if it is determined that the bit corresponding to the first hash value is not the designated value, lookup a second bit vector table from a second vector table filter, and determine whether the bit corresponding to the second hash value is the designated value;
and the processing module is further configured to determine that the online transaction message is a repeated message if it is determined that the bit corresponding to the second hash value is the designated value, and indicate that the online transaction message is not to be processed.
9. The apparatus of claim 8, wherein the processing module is further configured to determine that the online transaction message is not a duplicate message indicating processing of the online transaction message if it is determined that the bit corresponding to the second hash value is not a specified value; and updating the bits corresponding to the first hash value in the first bit vector table to a specified value, and updating the bits corresponding to the second hash value in the second bit vector table to a specified value.
10. The apparatus of claim 9, wherein the first vector filter and the second vector filter are timed asynchronously reconstructed at a set duration;
the vector table lookup module, which looks up the first bit vector table from the first vector table filter, includes: searching a first bit vector table from a first vector table filter corresponding to the current moment; looking up a second bit vector table from a second vector table filter, comprising: and searching a second bit vector table from a second vector table filter corresponding to the current moment.
11. The apparatus according to any one of claims 7 to 10, wherein the vector table lookup module determines the first hash value according to each obtained primary key value, including: carrying out encryption compression on each obtained primary key value through an information abstract algorithm to obtain a first character string; and carrying out Hash operation on the first character string to obtain a first Hash value.
12. The apparatus of claim 11, wherein the vector table lookup module to determine a second hash value based on each primary key value obtained comprises: multiplying and summing the obtained appointed code of each character of each primary key value and an appointed prime number to obtain a second character string; and carrying out Hash operation on the second character string to obtain a second Hash value.
13. A non-transitory computer storage medium storing an executable program for execution by a processor to perform the method of any one of claims 1 to 6.
14. A computer program product, characterized in that it comprises an executable program which is executed by a processor to implement the method of any one of claims 1 to 6.
15. An online transaction message deduplication device, wherein the device comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete mutual communication through the communication bus;
the memory is used for storing a computer program;
the processor, when executing the program stored in the memory, implementing the method steps of any of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111452832.3A CN114138786A (en) | 2021-12-01 | 2021-12-01 | Method, device, medium, product and equipment for duplicate removal of online transaction message |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111452832.3A CN114138786A (en) | 2021-12-01 | 2021-12-01 | Method, device, medium, product and equipment for duplicate removal of online transaction message |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114138786A true CN114138786A (en) | 2022-03-04 |
Family
ID=80387064
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111452832.3A Pending CN114138786A (en) | 2021-12-01 | 2021-12-01 | Method, device, medium, product and equipment for duplicate removal of online transaction message |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114138786A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115617809A (en) * | 2022-11-08 | 2023-01-17 | 广州睿帆科技有限公司 | Database uniqueness constraint processing method, device, equipment and medium |
-
2021
- 2021-12-01 CN CN202111452832.3A patent/CN114138786A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115617809A (en) * | 2022-11-08 | 2023-01-17 | 广州睿帆科技有限公司 | Database uniqueness constraint processing method, device, equipment and medium |
| CN115617809B (en) * | 2022-11-08 | 2023-03-21 | 广州睿帆科技有限公司 | Database uniqueness constraint processing method, device, equipment and medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107832406B (en) | Method, device, equipment and storage medium for removing duplicate entries of mass log data | |
| CN110321387B (en) | Data synchronization method, equipment and terminal equipment | |
| US11296940B2 (en) | Centralized configuration data in a distributed file system | |
| CN108108127B (en) | File reading method and system | |
| CN108255647B (en) | High-speed data backup method under samba server cluster | |
| CN106815254B (en) | Data processing method and device | |
| CN105049287A (en) | Log processing method and log processing devices | |
| CN110018996B (en) | Snapshot rollback method and related device of distributed storage system | |
| EP3125501A1 (en) | File synchronization method, server, and terminal | |
| CN112714359B (en) | Video recommendation method and device, computer equipment and storage medium | |
| CN110347651A (en) | Method of data synchronization, device, equipment and storage medium based on cloud storage | |
| CN106909595B (en) | Data migration method and device | |
| CN109981715B (en) | Session management method and device | |
| CN104079623A (en) | Method and system for controlling multilevel cloud storage synchrony | |
| CN114138786A (en) | Method, device, medium, product and equipment for duplicate removal of online transaction message | |
| CN114398520A (en) | Data retrieval method, system, device, electronic equipment and storage medium | |
| CN111163118A (en) | Message transmission method and device in Kafka cluster | |
| CN112433757A (en) | Method and device for determining interface calling relationship | |
| CN110442439B (en) | Task process processing method and device and computer equipment | |
| CN111427917A (en) | Search data processing method and related product | |
| CN111061719B (en) | Data collection method, device, equipment and storage medium | |
| CN110677497B (en) | Network medium distribution method and device | |
| CN111258821B (en) | Cloud computing-based backup data rapid extraction method | |
| CN109739883B (en) | Method and device for improving data query performance and electronic equipment | |
| CN110569231B (en) | Data migration method, device, equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |