CN108449231B - Transaction data filtering method and device and implementation device - Google Patents
Transaction data filtering method and device and implementation device Download PDFInfo
- Publication number
- CN108449231B CN108449231B CN201810216684.7A CN201810216684A CN108449231B CN 108449231 B CN108449231 B CN 108449231B CN 201810216684 A CN201810216684 A CN 201810216684A CN 108449231 B CN108449231 B CN 108449231B
- Authority
- CN
- China
- Prior art keywords
- transaction
- filtering rule
- transaction data
- matching
- flow
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/02—Capturing of monitoring data
- H04L43/028—Capturing of monitoring data by filtering
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/02—Capturing of monitoring data
- H04L43/026—Capturing of monitoring data using flow identification
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Financial Or Insurance-Related Operations Such As Payment And Settlement (AREA)
- Image Processing (AREA)
Abstract
The invention provides a method and a device for filtering transaction data and an implementation device; wherein, the method comprises the following steps: merging the transaction characteristics with the same characteristic value in the multiple filtering rule flows; the combined transaction characteristics share the same matching identification; matching the transaction data with a plurality of filter rule streams one by one; when the first filtering rule flow is matched, each transaction characteristic is matched, and a corresponding matching identifier is added; when the matching is carried out with other filtering rule flows, the transaction characteristics without matching identification are matched, and the corresponding matching identification is added; and if the transaction data are successfully matched with each transaction characteristic in the current filtering rule flow, carrying the identity of the current filtering rule flow in the transaction data. The invention reduces the matching times between the transaction characteristics and the rules, reduces the calculation complexity of filtering the transaction data, reduces the calculation time of the CPU, and improves the processing efficiency and performance of transaction data filtering.
Description
Technical Field
The invention relates to the technical field of computers, in particular to a method and a device for filtering transaction data and an implementation device.
Background
By decoding the transaction data captured on the network, the transaction characteristics of the transaction, such as an IP (Internet Protocol) address, a port, a transaction type, a transaction return code, and the like, can be extracted; the characteristic values of the transaction characteristics are the attributes of the transactions, so that two transactions with all the same characteristic values do not exist, but the transactions can have partially same characteristic values, such as the same source IP address and the same transaction type. One or more transactions can be filtered out through one or more transaction characteristics, and the filtered transactions can be used for calculating statistical characteristics such as transaction amount and success rate.
Generally, a rule of a characteristic value of a set of transaction characteristics is defined as a transaction flow, and the transaction flow comprises one or more characteristic values of transaction data; because of the large statistical data requirements, multiple transaction streams are often used simultaneously to filter transaction data; in the filtering process, the transaction characteristics in each transaction data need to be matched with the corresponding characteristic values in each transaction flow one by one, so that to complete the filtering task, the CPU needs to repeat a large number of matching operations, and along with the increase of the transaction data volume or the transaction flow volume, the operation amount of the CPU is multiplied, which results in lower processing performance of the filtered data.
Disclosure of Invention
In view of this, the present invention provides a method and an apparatus for filtering transaction data, and an implementation apparatus thereof, so as to reduce the computational complexity of filtering transaction data, reduce the computation time of a CPU, and improve the processing efficiency and the processing performance of filtering transaction data.
In a first aspect, an embodiment of the present invention provides a method for filtering transaction data, where the method includes: acquiring a plurality of filter rule flows; each filtering rule flow comprises one or more transaction characteristics and characteristic values corresponding to the transaction characteristics; merging the transaction characteristics with the same characteristic value in the multiple filtering rule flows; the combined transaction characteristics share the same matching identification; acquiring transaction data to be matched; matching the transaction data with a plurality of filter rule streams one by one; when the transaction data is matched with the first filtering rule flow, each transaction characteristic in the first filtering rule flow is matched, and a corresponding matching identifier is added; when the transaction data is matched with the filtering rule flows except the first filtering rule flow, the transaction characteristics without the matching identification in the filtering rule flows are matched, and the corresponding matching identification is added; and if the transaction data are successfully matched with each transaction characteristic in the current filtering rule flow, carrying the identity of the current filtering rule in the transaction data.
With reference to the first aspect, an embodiment of the present invention provides a first possible implementation manner of the first aspect, where the step of matching the transaction data with the multiple filter rule flows one by one includes: decoding the transaction data to obtain transaction characteristics in the transaction data and characteristic values corresponding to the transaction characteristics; the transaction characteristics at least comprise various IP addresses, port addresses, transaction types and transaction return codes of the transaction data; and matching the transaction characteristics and the characteristic values corresponding to the transaction characteristics with a plurality of filtering rule flows one by one.
With reference to the first possible implementation manner of the first aspect, an embodiment of the present invention provides a second possible implementation manner of the first aspect, where the step of matching the transaction characteristics and the characteristic values corresponding to the transaction characteristics with the multiple filter rule flows one by one includes: when the transaction characteristics and the characteristic values corresponding to the transaction characteristics are matched with the current filtering rule flow, the transaction characteristics and the characteristic values in the current filtering rule flow are obtained; comparing the transaction characteristics and the characteristic values in the transaction data according to the transaction characteristics and the characteristic values in the current filtering rule flow; and if the characteristic value corresponding to the transaction data is the same as the characteristic value corresponding to the current filtering rule flow in the same transaction characteristic, the transaction characteristic is successfully matched.
With reference to the second possible implementation manner of the first aspect, an embodiment of the present invention provides a third possible implementation manner of the first aspect, where when the transaction characteristics are successfully matched, the matching identifier is "1"; when the transaction feature matching fails, the match is identified as "0".
With reference to the first aspect, an embodiment of the present invention provides a fourth possible implementation manner of the first aspect, where the method further includes: pre-allocating a bit for each transaction feature in the plurality of filter rule streams; and filling the added matching identification into corresponding bits in the matching process.
With reference to the fourth possible implementation manner of the first aspect, an embodiment of the present invention provides a fifth possible implementation manner of the first aspect, where the step of merging transaction characteristics with the same characteristic value in multiple filtering rule flows includes: and combining the bits distributed by the transaction characteristics with the same characteristic value in the multiple filtering rule flows.
With reference to the first aspect, an embodiment of the present invention provides a sixth possible implementation manner of the first aspect, where the step of carrying the identity of the current filtering rule in the transaction data if the transaction data is successfully matched with each transaction feature in the current filtering rule flow includes: adding a field in the transaction data; the identity of the current filter rule flow is filled into the field.
In a second aspect, an embodiment of the present invention further provides a device for filtering transaction data, including: a filtering rule flow obtaining module for obtaining a plurality of filtering rule flows; each filtering rule flow comprises one or more transaction characteristics and characteristic values corresponding to the transaction characteristics; the combined transaction characteristic module is used for combining the transaction characteristics with the same characteristic value in the multiple filtering rule flows; the combined transaction characteristics share the same matching identification; the transaction data acquisition module is used for acquiring transaction data to be matched; the matching module is used for matching the transaction data with the multiple filtering rule flows one by one; when the transaction data is matched with the first filtering rule flow, each transaction characteristic in the first filtering rule flow is matched, and a corresponding matching identifier is added; when the transaction data is matched with the filtering rule flows except the first filtering rule flow, the transaction characteristics without the matching identification in the filtering rule flows are matched, and the corresponding matching identification is added; and the identification carrying module is used for carrying the identification of the current filtering rule in the transaction data if the transaction data is successfully matched with each transaction characteristic in the current filtering rule flow.
With reference to the second aspect, an embodiment of the present invention provides a first possible implementation manner of the second aspect, where the matching module is further configured to: decoding the transaction data to obtain transaction characteristics in the transaction data and characteristic values corresponding to the transaction characteristics; the transaction characteristics at least comprise various IP addresses, port addresses, transaction types and transaction return codes of the transaction data; and matching the transaction characteristics and the characteristic values corresponding to the transaction characteristics with a plurality of filtering rule flows one by one.
In a third aspect, an embodiment of the present invention further provides a device for implementing transaction data filtering, including a memory and a processor, where the memory is used to store one or more computer instructions, and the one or more computer instructions are executed by the processor, in the method for implementing transaction data filtering.
The embodiment of the invention has the following beneficial effects:
the embodiment of the invention provides a method, a device and an implementation device for filtering transaction data, which are used for merging transaction characteristics with the same characteristic value in a plurality of filtering rule flows; the combined transaction characteristics share the same matching identification; then, the transaction data are matched with a plurality of filtering rule flows one by one; when the transaction data is matched with the first filtering rule flow, each transaction characteristic in the first filtering rule flow is matched, and a corresponding matching identifier is added according to a matching result; when the transaction data is matched with the filtering rule flows except the first filtering rule flow, the transaction data is matched with the transaction characteristics without the matching identification in the filtering rule flow, and the corresponding matching identification is added; if the transaction data is successfully matched with each transaction feature in the current filtering rule flow, carrying the identity of the current filtering rule in the transaction data; the method reduces the matching times between the transaction characteristics and the rules, reduces the calculation complexity of filtering the transaction data, reduces the calculation time of a CPU (central processing unit), and improves the processing efficiency and performance of transaction data filtering.
Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by the practice of the invention as set forth above.
In order to make the aforementioned and other objects, features and advantages of the present invention comprehensible, preferred embodiments accompanied with figures are described in detail below.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.
FIG. 1 is a flow chart of a method for filtering transaction data according to an embodiment of the present invention;
FIG. 2 is a flow chart of another method for filtering transaction data according to an embodiment of the invention;
FIG. 3 is a schematic structural diagram of a transaction data filtering apparatus according to an embodiment of the present invention;
fig. 4 is a schematic structural diagram of a device for filtering transaction data according to an embodiment of the present invention.
Detailed Description
To make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is apparent that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
A set of rules for the feature values of the transaction features is defined as a transaction flow. When each feature value in the flow rule completely matches with the respective attribute of the real transaction data, the transaction data is said to belong to the current transaction flow. Each transaction flow has a unique flow ID (identity), and the transaction data is filtered by setting the rules of the flow. When the transaction data matches the flow rule, the transaction is tagged with a flow ID tag. As defined above, a flow may filter out one or more transactions, and a transaction may belong to one or more transaction flows.
Multiple matching rules are contained in one transaction stream and the same matching rules may exist in different streams. When transaction data are filtered, each transaction characteristic rule in each transaction flow is matched with the true value of a certain attribute field in each transaction one by one, background transaction data are acquired during each matching, and the time complexity is O (n)2). Considering that the actual transaction data volume is huge, the transaction rules are gradually increased along with the deep and more detailed transaction monitoring requirements. When the amount of data exceeds a certain range, the amount of calculation becomes too large, which causes a significant problem in processing performance.
If 2000 transaction flows are defined in the system, and a transaction flow contains 5 filtering rules, a total of 2000 × 5 ═ 10000 matches are required. Therefore, a relatively simple filtering task needs to perform ten thousand transaction comparison queries in the background; if the transaction flow rule of the filtering task is complex, an ingenious-concept transaction flow definition can be caused by excessive calculation amount, and the transaction monitoring tool designed and realized based on the definition is difficult to be popularized and applied in a large-data-volume scene.
According to the experience in practical application, there are a lot of identical matching rules among transaction flows, such as identical IP ports or identical transaction types. The original matching algorithm is used, data of the same field needs to be obtained from transaction data for multiple times, the calculation time of a CPU is wasted, and the working efficiency is reduced. Based on this, the method, the device and the implementation device for filtering transaction data provided by the embodiment of the invention can be applied to the processing of transaction data and the processing process of data statistics.
For the convenience of understanding the embodiment, a detailed description will be given to a method for filtering transaction data disclosed in the embodiment of the present invention.
Referring to fig. 1, a flow chart of a method for filtering transaction data is shown, the method comprising:
step S100, acquiring a plurality of filter rule flows; each filtering rule flow comprises one or more transaction characteristics and characteristic values corresponding to the transaction characteristics;
the transaction data of each transaction has respective transaction characteristics such as an IP (Internet Protocol) address, a port, a transaction type, a transaction return code, and the like. According to the filtering rule flow, a set of related transactions may be filtered to enable statistics of certain transaction parameters. For example, the filtering rule stream a includes three transaction characteristics, namely, an IP address (corresponding to a feature value X), a transaction type (corresponding to a feature value Y), and a transaction type (corresponding to a feature value Z); and filtering the transaction data in the set range according to the filtering rule flow to obtain one or more transaction data with an IP address of X, a transaction type of Y and a transaction type of Z, wherein the transaction data all meet the filtering rule flow, and the set transaction parameters can be calculated according to the transaction data.
In actual implementation, statistics of multiple parameters may be performed on the same range of transaction data, so that filtering processing may be performed on the transaction data according to multiple filtering rule flows, where the filtering rule flows correspond to the transaction flows.
Step S102, merging the transaction characteristics with the same characteristic value in a plurality of filtering rule flows; the combined transaction characteristics share the same matching identification;
when filtering the transaction data, after adding the matching identifier for the transaction feature of a specific feature value in a certain filtering rule flow, the transaction features of the same feature value in other filtering rule flows do not need to be matched with the transaction data again, and the matching identifier can be shared. For example, the transaction characteristics in the multiple filtering rule flows may be arranged in the form of a bitmap or a list, and when the matching between the current transaction data and the transaction characteristics m in a certain filtering rule flow is completed, a matching identifier is added; after the filtering rule stream a and the filtering rule stream B are merged in the above manner, if both the filtering rule stream a and the filtering rule stream B have the transaction characteristics m and the characteristic values are the same, after the same transaction data is matched with the transaction characteristics m in the filtering rule stream a, due to the matching identification and the addition, the transaction data is not matched with the transaction characteristics m in the filtering rule stream B.
Transaction characteristics with the same characteristic value in a plurality of filtering rule streams are combined, so that repeated matching of the transaction characteristics with the same characteristic value in the filtering process is avoided; and the combined transaction characteristics share the same matching identifier, so that the storage space of the matching identifier is saved.
Step S104, acquiring transaction data to be matched;
step S106, the transaction data is matched with a plurality of filter rule flows one by one; when the transaction data is matched with the first filtering rule flow, each transaction characteristic in the first filtering rule flow is matched, and a corresponding matching identifier is added; when the transaction data is matched with the filtering rule flows except the first filtering rule flow, the transaction characteristics without the matching identification in the filtering rule flows are matched, and the corresponding matching identification is added;
and step S108, if the transaction data is successfully matched with each transaction characteristic in the current filtering rule flow, carrying the identity of the current filtering rule in the transaction data.
The embodiment of the invention provides a transaction data filtering method, which is used for merging transaction characteristics with the same characteristic value in a plurality of filtering rule flows; the combined transaction characteristics share the same matching identification; then, the transaction data are matched with a plurality of filtering rule flows one by one; when the transaction data is matched with the first filtering rule flow, each transaction characteristic in the first filtering rule flow is matched, and a corresponding matching identifier is added according to a matching result; when the transaction data is matched with the filtering rule flows except the first filtering rule flow, the transaction data is matched with the transaction characteristics without the matching identification in the filtering rule flow, and the corresponding matching identification is added; if the transaction data is successfully matched with each transaction feature in the current filtering rule flow, carrying the identity of the current filtering rule in the transaction data; the method reduces the matching times between the transaction characteristics and the rules, reduces the calculation complexity of filtering the transaction data, reduces the calculation time of a CPU (central processing unit), and improves the processing efficiency and performance of transaction data filtering.
Referring to FIG. 2, another flow chart of a method for filtering transaction data is shown; when all the rules are completely matched, an array type field streams (data stream) is added to the transaction, and the corresponding value is the identification ID of each matched filtering rule stream.
The method specifically comprises the following steps:
step S200, acquiring a plurality of filtering rule flows and pre-distributing a bit for each transaction characteristic in the filtering rule flows;
in this embodiment, the transaction data is processed by a bitmap technique. The bitmap is a data structure, namely binary bits are used for describing a certain state, and the bitmap is suitable for processing data with larger scale under the condition that the existing state is known.
The above-mentioned filtering rule flow is also called transaction flow; each transaction stream contains one or more transaction characteristics. Taking stream1 and stream2 as examples, these are referred to hereinafter simply as stream1 and stream 2.
Flow 1 has the following rules: IP-A, Port-B, Trantype-C, Retcode-D;
flow 2 has the following rules: IP-A, Port-B, Trantype-E, Retcode-F;
wherein, IP is network protocol address, Port is Port, Tractypee is transaction type, Retcode is transaction return code, the above rules can be shown in Table 1:
TABLE 1
| Stream ID | IP | Port | Transtype | Retcode |
| stream1 | A | B | C | D |
| stream2 | A | B | E | F |
After bits are allocated to each transaction feature of stream1 and stream2, the bitmaps corresponding to stream1 and stream2 are shown in table 2:
TABLE 2
| Stream ID | IP | Port | Transtype | Retcode |
| stream1 | 0 | 0 | 0 | 0 |
| stream2 | 0 | 0 | 0 | 0 |
Step S202, merging bits distributed by the transaction characteristics with the same characteristic value in a plurality of filtering rule flows;
step 202, described above, is also referred to as merging identical flow rules present in different transaction flows. The merging results of merging transaction signatures having the same signature values of the stream1 and the stream2 are shown in table 3:
TABLE 3
After the corresponding bits are merged, the bitmaps for stream1 and stream2 are shown in table 4:
TABLE 4
Therefore, when matching the feature value of a transaction with stream1 and stream2, the matching of the two attributes of IP and Port only needs to be performed once, and the matching of the second time is not needed to be performed again because the corresponding bit is marked.
Step S204, decoding the transaction data to obtain transaction characteristics in the transaction data and characteristic values corresponding to the transaction characteristics; the transaction characteristics at least comprise various IP addresses, port addresses, transaction types and transaction return codes of the transaction data;
for example, after decoding the transaction data m, the transaction characteristics obtained are IP A, Port B, Trantype D C, Retcode D, see table 5:
TABLE 5
| IP | Port | Transtype | Retcode |
| A | B | C | D |
Step S206, matching the transaction characteristics and the characteristic values corresponding to the transaction characteristics with a plurality of filter rule flows one by one;
the step 206 may be specifically implemented by the following steps:
(1) when the transaction characteristics and the characteristic values corresponding to the transaction characteristics are matched with the current filtering rule flow, the transaction characteristics and the characteristic values in the current filtering rule flow are obtained;
(2) comparing the transaction characteristics and the characteristic values in the transaction data according to the transaction characteristics and the characteristic values in the current filtering rule flow; and if the characteristic value corresponding to the transaction data is the same as the characteristic value corresponding to the current filtering rule flow in the same transaction characteristic, the transaction characteristic is successfully matched.
Specifically, in the matching process, the added matching identifier is filled into the corresponding bit, and when the transaction characteristics are successfully matched, the matching identifier is "1"; when the transaction feature matching fails, the match is identified as "0".
The process of filling the matching identification in the corresponding bit is also called bitmap marking. The bitmap obtained after matching the transaction with stream1 and stream2 is shown in table 6:
TABLE 6
And step 208, if the transaction data is successfully matched with each transaction feature in the current filtering rule flow, adding a field in the transaction data, and filling the identity of the current filtering rule flow into the field.
The step 208 is specifically implemented as follows:
(1) comparing each bit of the filtering rule flow;
this step is also called result comparison, that is, comparing the results of the bitmaps corresponding to the transaction flow, and if each bit in the bitmap of the flow rule is 1, it indicates that all transaction characteristics of the current transaction satisfy the filtering condition of the flow rule, that is, it is determined that the transaction belongs to the current flow. From the matched bitmap, the transaction with IP A, Port B, Trantype C, Retcode D belongs to flow 1 but not to flow 2.
(2) If all the bits of the filtering rule flow are 1, adding a field in the transaction data, and filling the identity of the current filtering rule flow into the field.
This step is also called flow marking, i.e. the flow that will match successfully is marked on the transaction. This is done by adding a streams field to the transaction and adding the value of the stream ID as the value of this field. streams is an array type, and when stream rules of a plurality of transaction streams are matched with the transaction characteristics at the same time, labels of all streams can be marked on the current transaction. Therefore, transaction characteristics are flexibly configured through transaction flow, concerned transactions are filtered out, and then the transactions meeting the conditions are monitored or statistically analyzed.
Therefore, the label of flow 1 is added to the field of the transaction matching flow 1 successfully as shown in table 7:
TABLE 7
| IP | Port | Transtype | Retcode | streams |
| A | B | C | D | stream1 |
In the example mentioned in the above step, one transaction matches two flow rules, which together need to be compared 6 times. If the original matching scheme is adopted, 8 times of matching are needed, the number of matching is reduced by 2 times, if 10 transactions are reduced by 20 times, 100 transactions are reduced by 200 times, and the like. A simple filtering scenario improves the processing performance by 25%.
When the algorithm is used for dealing with the condition that the same filtering rule exists in a large data volume, multi-field and multi-transaction flow, for example, an extremely scene: if there are 2000 transaction flows in total, each flow has 5 rules and all flow rules are the same; when transaction characteristics of the same characteristic value are not combined, the bitmap of each transaction stream is shown in table 8:
TABLE 8
| Stream ID | IP | Port | Transtype | Retcode | Channel |
| stream1 | 0 | 0 | 0 | 0 | 0 |
| stream2 | 0 | 0 | 0 | 0 | 0 |
| … | … | … | … | … | … |
| stream2000 | 0 | 0 | 0 | 0 | 0 |
After merging the transaction features of the same feature value, the bitmap of all streams is shown in table 9. As can be seen from the above, the matching of one transaction according to the original method requires 10000 times of inquiry and comparison. And in the bitmap mode, only 5 comparisons are needed.
TABLE 9
In this case, the performance of transaction matching is greatly improved.
The method takes each filtering rule of the transaction flow as one bit of the bitmap through a bitmap marking technology. When the flow rule is matched, the last transaction is matched. The rule is mapped to the bitmap position as 1. And if the bit corresponding to the flow rule is 1 in the next matching, which indicates that the matching is successful, the field matching is not performed any more. The method saves the times of inquiring and comparing all transactions for multiple times by the same rule, and the performance is remarkably improved under the condition that the same filtering rule exists in a large amount of transaction flows.
Of course, for the special case that each flow rule is completely different, the method cannot effectively improve the performance, but does not increase the processing load in comparison with the original method.
The real transaction monitoring scenario is neither very idealized nor very exceptional. By combining with the real monitoring requirement, the calculation processing resource can be saved by 70-80% through theoretical analysis and practical application verification. This means that if a filtering rule of 2000 flows could be handled originally, then nearly 10000 flow rules could now be handled. If the original situation is close to the processing performance bottleneck, the normal operation of the system can be completely ensured by applying the bitmap-based matching method, and meanwhile, the configuration is allowed to be expanded continuously, so that the actual situation that the transaction types are increased increasingly and the monitoring requirements are refined increasingly is effectively coped with. Compared with the original method in the past, the performance is improved by times, and the transaction matching of the whole system is improved and optimized slightly.
In addition, the transaction data is filtered by the method, so that the processing time is reduced, the resource consumption is reduced, the expansibility of the transaction flow in practical application is ensured and a reliable solution is provided for the matching of the transaction flow rules serving as the basis of transaction monitoring when large-scale data volume and more repeated rules in different flow rules are dealt with.
Corresponding to the above method embodiment, referring to fig. 3, a schematic structural diagram of a transaction data filtering apparatus is shown, the apparatus includes: a filtering rule flow obtaining module 300, configured to obtain multiple filtering rule flows; each filtering rule flow comprises one or more transaction characteristics and characteristic values corresponding to the transaction characteristics; a merge transaction feature module 302, configured to merge transaction features with the same feature value in the multiple filter rule streams; the combined transaction characteristics share the same matching identification; a transaction data obtaining module 304, configured to obtain transaction data to be matched; the matching module is used for matching the transaction data with the multiple filtering rule flows one by one; when the transaction data is matched with the first filtering rule flow, each transaction characteristic in the first filtering rule flow is matched, and a corresponding matching identifier is added; when the transaction data is matched with the filtering rule flows except the first filtering rule flow, the transaction characteristics without the matching identification in the filtering rule flows are matched, and the corresponding matching identification is added; and an identifier carrying module 306, configured to carry the identifier of the current filtering rule in the transaction data if the transaction data is successfully matched with each transaction feature in the current filtering rule flow.
Further, the matching module is further configured to: decoding the transaction data to obtain transaction characteristics in the transaction data and characteristic values corresponding to the transaction characteristics; the transaction characteristics at least comprise various IP addresses, port addresses, transaction types and transaction return codes of the transaction data; and matching the transaction characteristics and the characteristic values corresponding to the transaction characteristics with a plurality of filtering rule flows one by one.
The embodiment provides a device for filtering and realizing transaction data, which corresponds to the method embodiment. FIG. 4 is a schematic diagram of a transaction data filtering apparatus, which may be disposed at a monitoring node in a distributed storage system; as shown in fig. 4, the apparatus includes a memory 100 and a processor 101; the memory 100 is used to store one or more computer instructions, which are executed by the processor to implement the above-described method for filtering transaction data, which may include one or more of the above methods.
Further, the device for filtering transaction data shown in fig. 4 further includes a bus 102 and a communication interface 103, and the processor 101, the communication interface 103 and the memory 100 are connected through the bus 102.
The Memory 100 may include a high-speed Random Access Memory (RAM) and may further include a non-volatile Memory (non-volatile Memory), such as at least one disk Memory. The communication connection between the network element of the system and at least one other network element is realized through at least one communication interface 103 (which may be wired or wireless), and the internet, a wide area network, a local network, a metropolitan area network, and the like can be used. The bus 102 may be an ISA bus, PCI bus, EISA bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one double-headed arrow is shown in FIG. 4, but that does not indicate only one bus or one type of bus.
The processor 101 may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware or instructions in the form of software in the processor 101. The Processor 101 may be a general-purpose Processor, and includes a Central Processing Unit (CPU), a Network Processor (NP), and the like; the device can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA) or other Programmable logic device, a discrete Gate or transistor logic device, or a discrete hardware component. The various methods, steps, and logic blocks disclosed in the embodiments of the present disclosure may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in connection with the embodiments of the present disclosure may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module may be located in ram, flash memory, rom, prom, or eprom, registers, etc. storage media as is well known in the art. The storage medium is located in the memory 100, and the processor 101 reads the information in the memory 100, and completes the steps of the method of the foregoing embodiment in combination with the hardware thereof.
Embodiments of the present invention further provide a machine-readable storage medium, where the machine-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are called and executed by a processor, the machine-executable instructions cause the processor to implement the above method for filtering transaction data, and specific implementation may refer to method embodiments, and will not be described herein again.
The method and the device for filtering transaction data and the computer program product for implementing the device provided by the embodiments of the present invention include a computer-readable storage medium storing a program code, where instructions included in the program code may be used to execute the method described in the foregoing method embodiments, and specific implementation may refer to the method embodiments, and will not be described herein again.
In addition, in the description of the embodiments of the present invention, unless otherwise explicitly specified or limited, the terms "mounted," "connected," and "connected" are to be construed broadly, e.g., as meaning either a fixed connection, a removable connection, or an integral connection; can be mechanically or electrically connected; they may be connected directly or indirectly through intervening media, or they may be interconnected between two elements. The specific meanings of the above terms in the present invention can be understood in specific cases to those skilled in the art.
The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicate orientations or positional relationships based on the orientations or positional relationships shown in the drawings, and are only for convenience of description and simplicity of description, but do not indicate or imply that the device or element being referred to must have a particular orientation, be constructed and operated in a particular orientation, and thus, should not be construed as limiting the present invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance.
Finally, it should be noted that: the above-mentioned embodiments are only specific embodiments of the present invention, which are used for illustrating the technical solutions of the present invention and not for limiting the same, and the protection scope of the present invention is not limited thereto, although the present invention is described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: any person skilled in the art can modify or easily conceive the technical solutions described in the foregoing embodiments or equivalent substitutes for some technical features within the technical scope of the present disclosure; such modifications, changes or substitutions do not depart from the spirit and scope of the embodiments of the present invention, and they should be construed as being included therein. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (10)
1. A method of filtering transaction data, comprising:
acquiring a plurality of filter rule flows; each filtering rule flow comprises one or more transaction characteristics and characteristic values corresponding to the transaction characteristics;
merging the transaction characteristics with the same characteristic value in the multiple filtering rule flows; the merged transaction characteristics share the same matching identification;
acquiring transaction data to be matched;
matching the transaction data with a plurality of filtering rule flows one by one;
when the transaction data is matched with a first filtering rule flow, matching each transaction characteristic in the first filtering rule flow, and adding a corresponding matching identifier; when the transaction data is matched with the filtering rule flows except the first filtering rule flow, the transaction characteristics without the matching identification in the filtering rule flows are matched, and the corresponding matching identification is added;
and if the transaction data is successfully matched with each transaction characteristic in the current filtering rule flow, carrying the identity of the current filtering rule flow in the transaction data.
2. The method of claim 1, wherein said step of matching said transaction data to said plurality of filter rule flows one by one comprises:
decoding the transaction data to obtain transaction characteristics in the transaction data and characteristic values corresponding to the transaction characteristics; the transaction characteristics at least comprise various IP addresses, port addresses, transaction types and transaction return codes of the transaction data;
and matching the transaction characteristics and the characteristic values corresponding to the transaction characteristics with the plurality of filtering rule flows one by one.
3. The method according to claim 2, wherein the step of matching the transaction characteristics and the characteristic values corresponding to the transaction characteristics one by one with the plurality of filter rule flows comprises:
when the transaction characteristics and the characteristic values corresponding to the transaction characteristics are matched with a current filtering rule flow, acquiring the transaction characteristics and the characteristic values in the current filtering rule flow;
comparing the transaction characteristics and the characteristic values in the transaction data according to the transaction characteristics and the characteristic values in the current filtering rule flow;
and if the feature value corresponding to the transaction data is the same as the feature value corresponding to the current filtering rule flow in the same transaction feature, the transaction feature is successfully matched.
4. The method of claim 3, wherein when the transaction characteristics match successfully, the match is identified as a "1"; when the transaction feature match fails, the match is identified as "0".
5. The method of claim 1, further comprising: pre-allocating a bit for each transaction feature in a plurality of said filter rule streams;
and filling the added matching identification into the corresponding bit in the matching process.
6. The method according to claim 5, wherein the step of combining transaction characteristics with the same characteristic value in the plurality of filtering rule flows comprises:
combining the bits distributed by the transaction characteristics with the same characteristic value in a plurality of filtering rule flows.
7. The method of claim 1, wherein the step of carrying the identity of the current filter rule in the transaction data if the transaction data matches each transaction feature in the current filter rule stream successfully comprises:
adding a field in the transaction data;
and filling the identity of the current filtering rule flow into the field.
8. A transaction data filtering device, comprising:
a filtering rule flow obtaining module for obtaining a plurality of filtering rule flows; each filtering rule flow comprises one or more transaction characteristics and characteristic values corresponding to the transaction characteristics;
the combined transaction characteristic module is used for combining the transaction characteristics with the same characteristic value in the filtering rule flows; the merged transaction characteristics share the same matching identification;
the transaction data acquisition module is used for acquiring transaction data to be matched;
the matching module is used for matching the transaction data with the plurality of filtering rule flows one by one; when the transaction data is matched with a first filtering rule flow, matching each transaction characteristic in the first filtering rule flow, and adding a corresponding matching identifier; when the transaction data is matched with the filtering rule flows except the first filtering rule flow, the transaction characteristics without the matching identification in the filtering rule flows are matched, and the corresponding matching identification is added;
and the identification carrying module is used for carrying the identity identification of the current filtering rule flow in the transaction data if the transaction data is successfully matched with each transaction characteristic in the current filtering rule flow.
9. The apparatus of claim 8, wherein the matching module is further configured to:
decoding the transaction data to obtain transaction characteristics in the transaction data and characteristic values corresponding to the transaction characteristics; the transaction characteristics at least comprise various IP addresses, port addresses, transaction types and transaction return codes of the transaction data;
and matching the transaction characteristics and the characteristic values corresponding to the transaction characteristics with the plurality of filtering rule flows one by one.
10. An apparatus for implementing filtering of transaction data, comprising a memory and a processor, wherein the memory is configured to store one or more computer instructions, which are executed by the processor to implement the method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810216684.7A CN108449231B (en) | 2018-03-15 | 2018-03-15 | Transaction data filtering method and device and implementation device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810216684.7A CN108449231B (en) | 2018-03-15 | 2018-03-15 | Transaction data filtering method and device and implementation device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108449231A CN108449231A (en) | 2018-08-24 |
| CN108449231B true CN108449231B (en) | 2020-07-07 |
Family
ID=63194733
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810216684.7A Active CN108449231B (en) | 2018-03-15 | 2018-03-15 | Transaction data filtering method and device and implementation device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108449231B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110851687A (en) * | 2019-11-11 | 2020-02-28 | 厦门市美亚柏科信息股份有限公司 | Data identification method, terminal equipment and storage medium |
| CN116993505B (en) * | 2023-09-25 | 2024-01-16 | 腾讯科技（深圳）有限公司 | Transaction processing method, device, electronic equipment and storage medium |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1669038A (en) * | 2002-05-17 | 2005-09-14 | 科罗尼尔第一州立投资有限公司 | Transaction management system |
| CN101655857A (en) * | 2009-09-18 | 2010-02-24 | 西安建筑科技大学 | Method for mining data in construction regulation field based on associative regulation mining technology |
| CN102012918A (en) * | 2010-11-26 | 2011-04-13 | 中金金融认证中心有限公司 | System and method for excavating and executing rule |
| CN102043789A (en) * | 2009-10-21 | 2011-05-04 | 阿里巴巴集团控股有限公司 | Method and device for updating data table |
| US8880539B2 (en) * | 2005-10-26 | 2014-11-04 | Cortica, Ltd. | System and method for generation of signatures for multimedia data elements |
| CN105426365A (en) * | 2014-08-01 | 2016-03-23 | 阿里巴巴集团控股有限公司 | Method and apparatus for distinguishing interactive behavior |
| CN106202389A (en) * | 2016-07-08 | 2016-12-07 | 中国银联股份有限公司 | A kind of method for monitoring abnormality based on transaction data and device |
| CN106991145A (en) * | 2017-03-23 | 2017-07-28 | 中国银联股份有限公司 | A kind of method and device of Monitoring Data |
| CN107256479A (en) * | 2017-05-19 | 2017-10-17 | 深圳市威富通科技有限公司 | The classification of trade mode performs method and device |
-
2018
- 2018-03-15 CN CN201810216684.7A patent/CN108449231B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1669038A (en) * | 2002-05-17 | 2005-09-14 | 科罗尼尔第一州立投资有限公司 | Transaction management system |
| US8880539B2 (en) * | 2005-10-26 | 2014-11-04 | Cortica, Ltd. | System and method for generation of signatures for multimedia data elements |
| CN101655857A (en) * | 2009-09-18 | 2010-02-24 | 西安建筑科技大学 | Method for mining data in construction regulation field based on associative regulation mining technology |
| CN102043789A (en) * | 2009-10-21 | 2011-05-04 | 阿里巴巴集团控股有限公司 | Method and device for updating data table |
| CN102012918A (en) * | 2010-11-26 | 2011-04-13 | 中金金融认证中心有限公司 | System and method for excavating and executing rule |
| CN105426365A (en) * | 2014-08-01 | 2016-03-23 | 阿里巴巴集团控股有限公司 | Method and apparatus for distinguishing interactive behavior |
| CN106202389A (en) * | 2016-07-08 | 2016-12-07 | 中国银联股份有限公司 | A kind of method for monitoring abnormality based on transaction data and device |
| CN106991145A (en) * | 2017-03-23 | 2017-07-28 | 中国银联股份有限公司 | A kind of method and device of Monitoring Data |
| CN107256479A (en) * | 2017-05-19 | 2017-10-17 | 深圳市威富通科技有限公司 | The classification of trade mode performs method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108449231A (en) | 2018-08-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106302104B (en) | User relationship identification method and device | |
| CN108449231B (en) | Transaction data filtering method and device and implementation device | |
| CN113485792B (en) | Pod scheduling method in kubernetes cluster, terminal equipment and storage medium | |
| CN104283891A (en) | Method and device for access of service function node to service link network | |
| CN111800430A (en) | Attack group identification method, device, equipment and medium | |
| CN112182007A (en) | Cloud computing data processing method based on artificial intelligence and artificial intelligence platform | |
| CN111159577B (en) | Community dividing method and device, storage medium and electronic device | |
| CN115409490A (en) | Project management system and method based on intelligent park | |
| US7337230B2 (en) | Method and system for eliminating redundant rules from a rule set | |
| CN106657128B (en) | Data packet filtering method and device based on wildcard mask rule | |
| CN115952398B (en) | Traditional calculation method, system and storage medium based on data of Internet of things | |
| CN110807036A (en) | Associated data network construction method and device | |
| EP3264716B1 (en) | State transition compression mechanism to efficiently compress dfa based regular expression signatures | |
| CN112861004B (en) | Method and device for determining rich media | |
| CN109918277A (en) | Electronic device, the evaluation method of system log cluster analysis result and storage medium | |
| CN111339373B (en) | Atlas feature extraction method, atlas feature extraction system, computer equipment and storage medium | |
| CN111510940B (en) | Signaling analysis method and device | |
| CN110191462B (en) | Method, device, medium and equipment for determining mobile terminal | |
| CN113392131A (en) | Data processing method and device and computer equipment | |
| CN112016466A (en) | Face recognition method, face recognition system, electronic device and computer storage medium | |
| CN115499338B (en) | Data processing method, device, medium and cloud network observation system | |
| CN111431929B (en) | Method and system for constructing multi-protocol distinguishing flow table of software defined network | |
| CN113472654B (en) | Network traffic data forwarding method, device, equipment and medium | |
| CN115865740B (en) | Key link identification method and device based on network structure | |
| CN113225308B (en) | Network access control method, node equipment and server |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information | ||
| CB02 | Change of applicant information |
Address after: Room 802, 8th Floor, Shining Building, 35 College Road, Haidian District, Beijing Applicant after: FUSIONSKYE (BEIJING) SOFTWARE Co.,Ltd. Address before: 100020 room 1005, Jin Ji Ye building, No. 2, Sheng Gu Road, anzhen bridge, Chaoyang District, Beijing Applicant before: FUSIONSKYE (BEIJING) TECHNOLOGY Co.,Ltd. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |