CN114615018B - Abnormality detection method for financial transaction all-link log - Google Patents
Abnormality detection method for financial transaction all-link log Download PDFInfo
- Publication number
- CN114615018B CN114615018B CN202210138369.3A CN202210138369A CN114615018B CN 114615018 B CN114615018 B CN 114615018B CN 202210138369 A CN202210138369 A CN 202210138369A CN 114615018 B CN114615018 B CN 114615018B
- Authority
- CN
- China
- Prior art keywords
- sequence
- module
- abnormality
- transaction
- link
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
- H04L63/1408—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic by monitoring network traffic
- H04L63/1425—Traffic logging, e.g. anomaly detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q40/00—Finance; Insurance; Tax strategies; Processing of corporate or income taxes
- G06Q40/06—Asset management; Financial planning or analysis
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Business, Economics & Management (AREA)
- Finance (AREA)
- Software Systems (AREA)
- Computer Security & Cryptography (AREA)
- Development Economics (AREA)
- Accounting & Taxation (AREA)
- Computing Systems (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- Computer Networks & Wireless Communication (AREA)
- Data Mining & Analysis (AREA)
- Mathematical Physics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Computer Hardware Design (AREA)
- Artificial Intelligence (AREA)
- Signal Processing (AREA)
- Entrepreneurship & Innovation (AREA)
- Medical Informatics (AREA)
- Game Theory and Decision Science (AREA)
- Human Resources & Organizations (AREA)
- Operations Research (AREA)
- Economics (AREA)
- Marketing (AREA)
- Strategic Management (AREA)
- Technology Law (AREA)
- General Business, Economics & Management (AREA)
- Financial Or Insurance-Related Operations Such As Payment And Settlement (AREA)
Abstract
The invention provides an anomaly detection method for a financial transaction all-link log, which solves the problem that statistics of error return codes of transaction links can only be manually set to a threshold value by adopting the technical scheme. The invention models the characteristics of various transaction sequences of the transaction link in a machine learning mode, learns the distribution probability of transactions in different time periods throughout the day, and can identify the abnormality of newly added sequences, the abnormality of sequence distribution sudden increase and sudden drop and the abnormality of sequence interruption. A potential malfunction of the transaction may be detected and alerted at a first time. Meanwhile, the problem that the traditional log abnormality detection mode can lose the prior sequence knowledge of a large number of links is solved, the sequence abnormality of the transaction links can be better identified, and the abnormality occurs for many times in the actual production environment of a plurality of financial institutions, so that serious faults are caused, and the tracing and positioning are difficult.
Description
Technical Field
The invention relates to the field of anomaly detection methods and systems, in particular to an anomaly detection method for a financial transaction all-link log.
Background
The prior art scheme has the following defects:
1. the manually carded link topology cannot be used for managing the interrupted transaction sequence and the newly added transaction nodes, and cannot be identified when the distribution situation of the transaction is abnormal.
2. The traditional log anomaly detection mode loses the prior knowledge of the link topology constructed manually/automatically for sequence anomalies of the transaction link, and the accuracy and the interpretability of the result are not good enough.
Therefore, an anomaly detection method for financial transaction all-link logs becomes a problem to be solved in the whole society.
Disclosure of Invention
In order to solve the technical problems, the technical scheme provided by the invention is as follows: an anomaly detection method for financial transaction all-link logs comprises assuming a constructed link topology,
link 1: module a= (module b= (module C);
link 2: module a= (module d= (module e= (module F);
link 3: module a= (module D);
(1) Firstly, caching all transaction sequences into a memory database;
(2) Learning the distribution of transaction sequences over a historically fixed time slice, e.g., calculating 15 pm every weekday: 00-15: 05, removing noise by using a 3 sigma principle, wherein the probability of a link 1 is 20% -40%; the distribution probability of the link 2 is 40% -50%; the distribution probability of the link 3 is 20% -30%;
(3) Learning historically the transition probability case between modules within a fixed time slice, e.g. calculating 15 pm every weekday: 00-15: and 05, removing noise by using a 3 sigma principle, and obtaining the transition probability of the module A to the module D of 60-80% and the transition probability of the module A to the module B of 20-40%.
Further, after a new transaction sequence is finished or overtime, the transaction sequence is compared with the existing transaction sequence in the memory data:
(1) If the sequence does not appear in the history and is not a subsequence of any sequence, recording the abnormality, giving an alarm when the same abnormality occurring in a certain fixed time slice (for example, 1 minute or 5 minutes) exceeds one experience parameter, and marking the abnormality as an abnormality of a history newly added sequence;
(2) If the sequence is a subsequence of a sequence that has been historically shown (with sequence length greater than a certain parameter), it is indicated that the sequence is an interrupted sequence, the anomaly is recorded, and an alarm is given when the same anomaly that occurred within a certain fixed time slice (e.g., 1 minute or 5 minutes) exceeds an empirical parameter. If the sequence belongs to a plurality of subsequences of the sequences, the sequence with the highest probability is selected according to the transition probability sequence, and the sequence is marked as sequence interruption abnormality;
(3) For all sequences in the current time slice, calculating the statistical proportion of each sequence, comparing with the historical probability distribution, giving an alarm if the upper limit and the lower limit are exceeded, marking as abnormal sequence distribution, and setting experience parameters, wherein the alarm is given when sudden increase or sudden drop occurs.
Compared with the prior art, the invention has the advantages that the invention adopts the technical scheme and has the following advantages:
(1) The problem that statistics of error return codes of a transaction link can only be manually set to a threshold value is solved. The invention models the characteristics of various transaction sequences of the transaction link in a machine learning mode, learns the distribution probability of transactions in different time periods throughout the day, and can identify the abnormality of newly added sequences, the abnormality of sequence distribution sudden increase and sudden drop and the abnormality of sequence interruption. A potential malfunction of the transaction may be detected and alerted at a first time.
(2) The method solves the problem that the traditional log anomaly detection mode can lose the prior sequence knowledge of a large number of links, can better identify the sequence anomalies of the transaction links, and causes serious faults due to the fact that the anomalies occur for a plurality of times in the actual production environments of a plurality of financial institutions, and is difficult to track and locate.
Drawings
FIG. 1 is a schematic block diagram of a method for anomaly detection of a financial transaction full-link log according to the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings.
The present invention will be described in detail with reference to the accompanying drawings.
The invention provides an anomaly detection method for financial trade transaction all-link logs, which comprises the steps of assuming constructed link topology,
link 1: module a= (module b= (module C);
link 2: module a= (module d= (module e= (module F);
link 3: module a= (module D);
(1) Firstly, caching all transaction sequences into a memory database;
(2) Learning the distribution of transaction sequences over a historically fixed time slice, e.g., calculating 15 pm every weekday: 00-15: 05, removing noise by using a 3 sigma principle, wherein the probability of a link 1 is 20% -40%; the distribution probability of the link 2 is 40% -50%; the distribution probability of the link 3 is 20% -30%;
(3) Learning historically the transition probability case between modules within a fixed time slice, e.g. calculating 15 pm every weekday: 00-15: and 05, removing noise by using a 3 sigma principle, and obtaining the transition probability of the module A to the module D of 60-80% and the transition probability of the module A to the module B of 20-40%.
2. The anomaly detection method for the financial transaction all-link log according to claim 1, wherein the anomaly detection method comprises the following steps: after a new transaction sequence is finished or overtime, the new transaction sequence is compared with the existing transaction sequence in the memory data:
(1) If the sequence does not appear in the history and is not a subsequence of any sequence, recording the abnormality, giving an alarm when the same abnormality occurring in a certain fixed time slice (for example, 1 minute or 5 minutes) exceeds one experience parameter, and marking the abnormality as an abnormality of a history newly added sequence;
(2) If the sequence is a subsequence of a sequence that has been historically shown (with sequence length greater than a certain parameter), it is indicated that the sequence is an interrupted sequence, the anomaly is recorded, and an alarm is given when the same anomaly that occurred within a certain fixed time slice (e.g., 1 minute or 5 minutes) exceeds an empirical parameter. If the sequence belongs to a plurality of subsequences of the sequences, the sequence with the highest probability is selected according to the transition probability sequence, and the sequence is marked as sequence interruption abnormality;
(3) For all sequences in the current time slice, calculating the statistical proportion of each sequence, comparing with the historical probability distribution, giving an alarm if the upper limit and the lower limit are exceeded, marking as abnormal sequence distribution, and setting experience parameters, wherein the alarm is given when sudden increase or sudden drop occurs.
As a further illustration of the present invention, in the cloud-native scenario, the modules listed above may also be micro services, when there are a large number of micro services, it is recorded that the transition probabilities between all micro services have performance problems during training and calculation, and we greatly reduce the calculation amount by limiting the length of the calculation nodes by using HMM (hidden markov model).
As a further illustration of the present invention, in some scenarios, a certain subsequence of sequences may loop multiple times, reducing the total number of sequences by eliminating loops, reducing storage difficulty and implementation complexity.
The invention and its embodiments have been described above with no limitation, and the actual construction is not limited to the embodiments of the invention as shown in the drawings. In summary, if one of ordinary skill in the art is informed by this disclosure, a structural manner and an embodiment similar to the technical solution should not be creatively devised without departing from the gist of the present invention.
Claims (1)
1. A method for detecting abnormality of financial transaction all-link log is characterized by comprising constructed link topology,
link 1: module a= (module b= (module C);
link 2: module a= (module d= (module e= (module F);
link 3: module a= (module D);
(1) Firstly, caching all transaction sequences into a memory database; after a new transaction sequence is finished or overtime, the new transaction sequence is compared with the existing transaction sequence in the memory data:
(a) If the sequence does not appear in the history and is not a subsequence of any sequence, recording the abnormality, giving an alarm when the same abnormality occurring within 1 minute or 5 minutes exceeds one experience parameter, and marking the abnormality as an abnormality of a history added sequence;
(b) If the sequence is a subsequence of a certain sequence which appears in history and the sequence length is larger than a certain parameter, the sequence is indicated to be an interrupted sequence, the abnormality is recorded, an alarm is sent after the same abnormality which occurs within 1 minute or 5 minutes exceeds an experience parameter, if the sequence belongs to the subsequence of a plurality of sequences, the sequence with the highest probability is sequenced according to the transition probability, and the sequence is identified as the sequence interruption abnormality;
(c) For all sequences in the current time slice, calculating the statistical proportion of each sequence, comparing with the historical probability distribution, giving an alarm if the upper limit and the lower limit are exceeded, marking as abnormal sequence distribution, and setting experience parameters, wherein the alarm is given when sudden increase or sudden drop occurs;
(2) Learning the distribution situation of the transaction sequence in the historical fixed time slice, and calculating 15 pm per working day: 00-15: 05, removing noise by using a 3 sigma principle, wherein the probability of the link 1 is 20% -40%; the distribution probability of the link 2 is 40% -50%; the distribution probability of the link 3 is 20% -30%;
(3) The transition probability situation among modules in the fixed time slices in the history is learned, and 15 pm in each working day is calculated: 00-15: and 05, removing noise by using a 3 sigma principle, and obtaining the transition probability of the module A to the module D of 60-80% and the transition probability of the module A to the module B of 20-40%.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210138369.3A CN114615018B (en) | 2022-02-15 | 2022-02-15 | Abnormality detection method for financial transaction all-link log |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210138369.3A CN114615018B (en) | 2022-02-15 | 2022-02-15 | Abnormality detection method for financial transaction all-link log |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114615018A CN114615018A (en) | 2022-06-10 |
| CN114615018B true CN114615018B (en) | 2023-10-03 |
Family
ID=81859038
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210138369.3A Active CN114615018B (en) | 2022-02-15 | 2022-02-15 | Abnormality detection method for financial transaction all-link log |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114615018B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115883346B (en) * | 2023-02-23 | 2023-05-23 | 广州嘉为科技有限公司 | Abnormality detection method and device based on FDEP log and storage medium |
| CN117112371B (en) * | 2023-10-25 | 2024-01-26 | 杭银消费金融股份有限公司 | Observable full-link log tracking method and system |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109889538A (en) * | 2019-03-20 | 2019-06-14 | 中国工商银行股份有限公司 | User's anomaly detection method and system |
| CN111796957A (en) * | 2020-06-30 | 2020-10-20 | 中国工商银行股份有限公司 | Transaction abnormal root cause analysis method and system based on application log |
| CN113064873A (en) * | 2021-04-15 | 2021-07-02 | 上海浦东发展银行股份有限公司 | Log anomaly detection method with high recall rate |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20050222929A1 (en) * | 2004-04-06 | 2005-10-06 | Pricewaterhousecoopers Llp | Systems and methods for investigation of financial reporting information |
| FR2979447B1 (en) * | 2011-08-29 | 2015-09-25 | Commissariat Energie Atomique | METHOD FOR CONFIGURING SENSOR DETECTION DEVICE, COMPUTER PROGRAM, AND CORRESPONDING ADAPTIVE DEVICE |
-
2022
- 2022-02-15 CN CN202210138369.3A patent/CN114615018B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109889538A (en) * | 2019-03-20 | 2019-06-14 | 中国工商银行股份有限公司 | User's anomaly detection method and system |
| CN111796957A (en) * | 2020-06-30 | 2020-10-20 | 中国工商银行股份有限公司 | Transaction abnormal root cause analysis method and system based on application log |
| CN113064873A (en) * | 2021-04-15 | 2021-07-02 | 上海浦东发展银行股份有限公司 | Log anomaly detection method with high recall rate |
Non-Patent Citations (1)
| Title |
|---|
| 基于Makov链状态转移概率矩阵的网络入侵检测;韩红光;周改云;;控制工程(03);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114615018A (en) | 2022-06-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114615018B (en) | Abnormality detection method for financial transaction all-link log | |
| US10402511B2 (en) | System for maintenance recommendation based on performance degradation modeling and monitoring | |
| Lu et al. | Predictive condition‐based maintenance for continuously deteriorating systems | |
| CN113282635B (en) | Method and device for positioning fault root cause of micro-service system | |
| WO2014145977A1 (en) | System and methods for automated plant asset failure detection | |
| CA2938766A1 (en) | Method of identifying anomalies | |
| Knorn et al. | Adaptive kalman filtering for anomaly detection in software appliances | |
| CN111737095A (en) | Batch processing task time monitoring method and device, electronic equipment and storage medium | |
| EP3026518A1 (en) | Method for Root analysis of an alarm flood sequence | |
| EP3430767B1 (en) | Method and device for real-time network event processing | |
| CN113360722B (en) | Fault root cause positioning method and system based on multidimensional data map | |
| CN115514619B (en) | Alarm convergence method and system | |
| CN114095965A (en) | Index detection model obtaining and fault positioning method, device, equipment and storage medium | |
| US11887465B2 (en) | Methods, systems, and computer programs for alarm handling | |
| CN110659147B (en) | Self-repairing method and system based on module self-checking behavior | |
| CN107911762A (en) | A kind of ONU method for diagnosing faults based on decision tree | |
| CN109885978B (en) | Remote sensing ground station fault diagnosis system and method | |
| CN117596119A (en) | Equipment data acquisition and monitoring method and system based on SNMP (simple network management protocol) | |
| CN110399278B (en) | Alarm fusion system and method based on data center anomaly monitoring | |
| CN108521346B (en) | Method for positioning abnormal nodes of telecommunication bearer network based on terminal data | |
| CN117519006A (en) | Production line data processing method, device, computer equipment and storage medium | |
| CN115118580B (en) | Alarm analysis method and device | |
| Li et al. | Meteorological radar fault diagnosis based on deep learning | |
| KR101982235B1 (en) | Method and device for similar equipment clustering using feature extraction method based on survival function | |
| CN115080286A (en) | Method and device for discovering log exception of network equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |