KR20220168420A - Real-time abnormal symptoms detection system and method by in-memory - Google Patents

Abstract

An analysis method for real-time analysis of abnormal symptoms in an RTA analysis processor unit in an in-memory-based abnormal symptom real-time analysis system comprises: a first comparison field processing step for performing a process of comparing whether data corresponds to a special condition among second input data transmitted from a second Kafka unit; a second profiled data comparing step for performing a process of comparing the data requiring analysis with past profiled data based on a specific reference value when the data corresponds to the special condition in the first comparison field processing step; and a third aggregation data comparing step of performing a process of extracting and setting an abnormal symptom class aggregation for a corresponding customer from memory and comparing the second input data with a given set abnormal symptom class aggregation. When the second data corresponds to the abnormal symptom class aggregation set in the third aggregation data comparing step, the RT analysis processor unit outputs a case as an abnormal symptom. Accordingly, real-time analysis of abnormal symptoms is possible.

Description

Real-time abnormal symptoms detection system and method by in-memory}

The present invention relates to an in-memory based anomaly analysis method in real time.

With the development of communication networks including the Internet and the development of automated devices, related financial fraud using the Internet is rapidly increasing worldwide.

For example, the methods of financial fraud are getting more sophisticated day by day with various methods, such as the method of stealing an individual ID to perform account transfer to the Daepo bankbook and then making withdrawals, and the method of withdrawing after inducing deposits into the Daepo bankbook through voice phishing. are becoming intelligent.

The damage caused by such financial fraud is becoming a serious social problem, causing time and money losses not only to individuals but also to banks and electronic financial service providers.

As a related technology to prevent financial fraud, a technology that blocks Internet banking service when accessing to an IP reported as being used for financial fraud is used.

In addition, research on a rule-based abnormal financial transaction detection system that collects various information of a payer to create a pattern, catches an abnormal payment that is different from the pattern, and blocks the payment route is being continuously conducted.

with this In relation to this, Korean Patent Publication No. 10-1153968 discloses a financial fraud prevention system. Patent No. 10-1153968 discloses a financial fraud pattern database unit that stores and manages data on financial fraud cases collected through multiple channels by type, and stores and Financial fraud that blocks the user's financial transaction by determining whether the user's financial transaction through the communication network corresponds to a financial fraud according to the managed user pattern database unit and the user's usual financial transaction data and financial fraud case data A financial fraud prevention system including a detection unit and a prevention method using the same are provided. Accordingly, since financial fraud cases are collected and utilized through multiple channels, there is an advantage in flexibly coping with continuously evolving financial fraud techniques.

However, real-time analysis is difficult with a relational database and IP tracking technology based on a relational data model used in the prior art.

Recently, an in-memory database technology has been proposed and introduced. The in-memory based system is a database management system that is installed and operated in the main memory of data storage, and has the advantage of faster processing speed than the method installed on a disk.

Republic of Korea Patent Registration No. 10-1153968 (Financial fraud prevention system and method)

An object of the present invention is to provide an analysis system and method capable of analyzing in-memory-based anomalies in real time.

According to one aspect of the present invention, an in-memory based anomaly symptom real-time analysis system includes: a first Kafka unit formed of a plurality of kafka modules and receiving input data for analysis; a parser unit parsing the input data transmitted from the first Kapuka unit and converting it into second input data that is normalized data; a second kafka unit that distributes and transmits the second input data converted by the parser unit to a plurality of kafka modules; an RTA analysis processor unit performing real-time analysis according to a given process on the second input data transmitted from the second KaPuka unit; a profiler unit for profiling the second input data analyzed by the RTA analysis processor unit 120 according to a condition process set based on a specific reference value and converting the profiled data into a key value method; and a Redis unit that stores the data converted by the profiler unit in a key value method in a memory. It is characterized by including.

In addition, the first Kafka unit and the second Kafka unit are characterized in that 2 to 10 Kafka modules are formed in parallel.

In addition, the RTA analysis processor unit is characterized in that a plurality of RTA analysis processors are formed, and each processing quantity is divided and analyzed in parallel.

In addition, in the profiler unit, the specific reference value includes a customer number or ID, and the set condition process is performed based on the specific reference value, and the terminal device, country, account number, transaction channel, and average transaction amount are any one or more. It is characterized in that it includes.

In addition, the analysis system is characterized in that the input data is received through a customer interface and an account system of a financial server.

According to another aspect of the present invention, in the analysis system, the analysis method of analyzing the anomaly in the RTA analysis processor unit in real time determines whether the second input data transmitted from the second KaPuka unit corresponds to a specific condition. a first comparison field processing step in which a comparison process is performed; In the first comparison field processing step, when the specific condition is met, a second profiled data comparison step in which a process of comparing the data required for analysis with data profiled in the past based on a specific reference value is performed. ; and in the second profiled data comparison step, if the specific reference value is not satisfied, an anomaly symptom class aggregation for the corresponding customer is extracted from the memory and set, and the second input data is set with the anomaly symptom class aggregation given. a third aggregation data comparison step in which a comparison process is performed; and outputting the RTA analysis processor unit as an abnormal symptom transaction when it corresponds to the abnormal symptom class aggregation set in the third aggregation data comparison step.

In addition, if it does not correspond to the anomaly symptom class aggregation set in the third aggregation data comparison step, the second input data is profiled by the profiler and stored in the Redis unit.

In addition, the specific condition of the first comparison field processing step is “transaction made while the accessing terminal is mobile and the accessing country is not Korea” or, as the terminal information, the terminal identification key (MAC, HDD, IMEI, UUID, androidID . / Including any one or more of the classification of receipt of withdrawal, customer number, ID, terminal information, account number, counterpart account number, counterpart bank code, transaction amount, balance, account holder name, counterpart account holder name, corporate account status, change record code It is characterized by doing.

In addition, the specific reference value of the second profiled data comparison step is based on customer standards, if there has been overseas use in the past, Characterized in that it includes any one or more of whether the terminal was used in the past on a customer basis and whether the account was transferred in the past on a customer basis.

In addition, the abnormal symptom class aggregation set in the third aggregation data comparison step is, when the same ID accesses over a predetermined threshold value during the reference time, the accumulated amount processed during the reference time is greater than the threshold value, the transfer customer, the duration ( It is characterized in that the count for the reference value during duration) includes any one or more of when the threshold value is exceeded.

According to an embodiment of the present invention, an algorithm is installed and operated in the main memory of data storage, and processing time is reduced by a simple architecture, thereby enabling real-time analysis of anomalies.

1 is a schematic diagram of a structure for explaining an in-memory based anomaly symptom real-time analysis system according to an embodiment of the present invention.
FIG. 2 illustrates a real-time analysis method for abnormal symptoms in the RTA analysis processor unit 120 of the in-memory based real-time abnormal symptom analysis system according to an embodiment of the present invention.
3 illustrates an example of applying the in-memory based real-time anomaly analysis system according to an embodiment of the present invention to a financial system.

Terms used in this application are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise.

In this application, when a certain component is said to "include", it means that it may further include other components without excluding other components unless otherwise stated.

In addition, terms such as "...unit", "...unit", "module", and "device" described in the specification mean a unit that processes at least one function or operation, which is hardware or software, or a combination of hardware and software. can be implemented as

Also, terms such as first and second may be used in describing components of an embodiment of the present invention. These terms are only used to distinguish the component from other components, and the nature, order, or order of the corresponding component is not limited by the term. When an element is described as being 'connected', 'coupled' or 'connected' to another element, the element may be directly connected, coupled or connected to the other element, but not between the element and the other element. It should be understood that another component may be 'connected', 'coupled' or 'connected' between elements.

Hereinafter, an in-memory based anomaly symptom real-time analysis system and method according to an implementation of the present invention will be described in detail.

An in-memory based data processing system is a database that holds all data in main memory rather than on disk. The biggest advantage is that data access is much faster than disk search. Accordingly, it is possible to solve the problem of slow database response due to the rapid increase in the amount of data, and to analyze anomalies in real time.

A typical disk method performs a query on data stored on disk, but an in-memory method puts an index in memory and can quickly retrieve all necessary information through the in-memory index.

1 is a schematic diagram of a structure for explaining an in-memory based anomaly symptom real-time analysis system according to an embodiment of the present invention.

Referring to FIG. 1, in the in-memory based anomaly symptom real-time analysis system 100 according to an embodiment of the present invention, raw data for analysis is input, and the input raw data is distributed and processed to parser 160 ), the first Kafka module 150 formed in parallel with a plurality of Kafka modules to process the transmission, parsing the unstructured raw data received from the first Kafka module 150, and parsing the normalized data, the second A parser unit (160) that converts into input data, and a plurality of kafka modules are formed in parallel to distribute and process the second input data converted by the parser unit (160) and transmit it to the RTA unit The second KaPuka unit 170, the RTA analysis processor unit 120 performing a function of analyzing data in real time according to a given process for the second input data transmitted from the second KaPuka unit 170, the RTA analysis processor The profiler unit 130 performs profiling according to the conditional process set for the second input data analyzed in the unit 120 and converts the profiled data into a key=value format, and the profiler unit 130 key=value It includes a Redis unit 110 that stores data converted in this way in memory.

In one embodiment of the present invention, the first KaPuka unit 150 and the second KaPuka unit 170 are distributed data queues (temporary storage) for distributed processing between the front and rear ends of the parser unit 160 for data processing. ) characterized in that it is used as.

According to an embodiment of the present invention, a kafka module refers to a module used as a distributed data queue to temporarily store additional data in a buffer while a connected process receiving data from a socket reads them.

In one embodiment of the present invention, the first Kafka unit 150 is distributed and processed with 2 to 10 kafka modules according to the throughput of the parser unit 160 to increase efficient data transmission and data transmission speed. there is.

In FIG. 1 , when the amount of input data is greater than the amount parsed in real time by the parser unit 160, the first Kafka unit 150 distributes it to 2 to 10 kafka modules and processes them sequentially. In addition, when the parser data processing performance is higher than the RTA data performance during the process in which the parser unit 160 normalizes data and the RTA analysis processor unit 120 analyzes the data, the second Kapuka unit 170 analyzes 2 to 10 data. By distributing to the Kafka module and sequentially transmitting, the parser unit 160 can continue parsing the input data signal without waiting for processing by the RTA analysis processor unit 120.

Therefore, it has an effect of enabling real-time analysis by increasing the speed of the overall data flow according to an embodiment of the present invention.

According to an embodiment of the present invention, the parser 160 performs a function of parsing unstructured raw data and converting it into normalized data.

Parser 160 refers to a program that determines if a sequence of given termination symbols is generated from a particular grammar, and finds the sequence of production rules that reach that sequence from the initial symbol. Among objects with a structure, a statement written in a natural language or a program written in an artificial language is parsed and converted according to grammar rules.

For example, an example of input unstructured data is as follows.

- Unstructured data example 1: "KR", "abc", "1.1.1.1.'...

- Unstructured data example 2: con="JP", id="efg", ip="2.2.2.2", ...

The parser 160 converts the unstructured data into normalized second input data as shown in Table 1 below in order to efficiently perform analysis work in the RTA analysis processor unit 120.

Table 1 shows an example of normalized second input data converted from input unstructured data.

country ID IP KR abc 1.1.1.1 JP efg 2.2.2.2

The RTA analysis processor unit 120 is an analysis processor that analyzes the input second input data in real time with the data profiled in the Redis unit 110 according to a given process. In one embodiment of the present invention, for real-time processing speed It is formed of a plurality of real-time analyzer (RTA) analysis processors in parallel and is characterized in that each processing volume is divided and analyzed and processed in parallel.

For example, if there are 100 scenarios to be analyzed in real time, if two RTA analysis processors are driven, 50 scenarios are assigned to each and analyzed in real time.

The profiler unit 130 according to an embodiment of the present invention is a module that continuously profiles the second input data based on a specific reference value.

Profiling is the operation of calculating and accumulating data for a certain period of time based on a specific reference value (e.g. customer number or ID). For example, the terminal device, country, account number, transaction channel, and average transaction amount are accumulated based on the customer number.

In order to apply this to the financial system to detect anomalies, an embodiment of the present invention includes the step of continuously profiling the past history of each customer and accumulating data, as in the above example.

For example, when there is a scenario in which a transfer of more than 50 million won is regarded as an abnormal behavior, if the transfer is made to an account where there was a transaction in the past even though the current transfer is more than 50 million won, it can be regarded as normal.

In one embodiment of the present invention, the relative account number that has been continuously transferred for each customer is profiled and continuously stored in memory, and the RTA analysis processor unit 120 compares it with the current data and detects anomalies step by step according to conditions. judge

Characterized in that the data profiled by the profiler unit 130 is converted into non-relational data of a key value method, and the Redis unit 110 stores the converted key value method data in a memory.

Compared to general DBMS, which creates several tables, can form relationships between tables, and manages data by creating several fields in each table, the Redis unit (110) according to an embodiment of the present invention ) is stored in a simple key=value method, so searching for the value for the key is very fast, and as a result, it has the effect of real-time searching.

The key=value method stores data as a set of key-value pairs using the key as a unique identifier. Keys and values can be anything from simple objects to complex aggregates. The key=value approach allows for partitioning and horizontal scaling to an extent not possible with other types of databases. This key=value method can be performed using the open source “DynamoDB”.

FIG. 2 illustrates a real-time analysis method for abnormal symptoms in the RTA analysis processor unit 120 of the in-memory based real-time abnormal symptom analysis system according to an embodiment of the present invention.

The in-memory based anomaly symptom real-time analysis system according to an embodiment of the present invention can be applied to a financial system to analyze anomaly symptoms for financial transactions and provide information as an output in case of an anomaly.

3 illustrates an example of applying the in-memory based real-time anomaly analysis system according to an embodiment of the present invention to a financial system.

Referring to FIG. 3, a customer using a financial system accesses a financial server 200 through a network using a customer terminal 500 to process financial affairs.

The customer terminal includes a smart phone, telebanking, PC, ATM, and the like.

The in-memory based anomaly symptom real-time analysis system 100 according to an embodiment of the present invention provides customers' financial processing data in real time through the multi-channel integration (MCI) 210 and the account system 250 of the financial server 200. receive input

The in-memory based anomaly symptom real-time analysis system 100 according to an embodiment of the present invention is a financial server 200 through a multi channel integration (MCI) 210 customer interface. Receive data about work. In addition, by accessing the account system 250 of the financial server 200, the customer receives data on financial business processed through other financial institutions in real time.

Referring to FIG. 2, in the real-time analysis method for anomalies in the RTA analysis processor unit 120 of the in-memory-based real-time anomaly analysis system according to an embodiment of the present invention, first comparison field processing (1 ^st comparsion filed operation) step 1210 is performed.

In the first comparison field processing (1st comparison field operation) step 1210, data corresponding to the specific condition required from among the data required for normalized analysis transmitted from the second ^Kabuka unit 170 is extracted and compared. process is carried out

In step ¹²¹⁰ of a first comparison field operation according to an embodiment of the present invention, operation processing of a condition field and a data field is performed.

For example,

- Operations on fields and values: =, !=, like, not like, in, not in, >, <, >=, <=, etc. are supported.

- Ex> When there are variables of A, B, C, D

(A=1 and B <= 3 and A != B) or (C like 'man*' and D in ('t', 's')) …

In addition, when the in-memory-based real-time analysis system for anomalies according to an embodiment of the present invention is applied to a method for analyzing anomalies in financial processing applied to a financial system, a specific condition may be set as a condition corresponding to financial fraud.

For example, in the case of an anomaly analysis method for financial processing in which the in-memory-based real-time analysis system for anomaly analysis according to an embodiment of the present invention is applied to a financial system, the first comparison field processing (1st ^comparison filed operation) In step 1210, the specific condition may be set to “transfer transaction while the accessing country is not Korea and the accessing terminal is mobile”.

The conditions corresponding to such financial fraud are terminal information such as terminal identification key (MAC, HDD, IMEI, UUID, androidID, widevineID), access IP, access country code, whether to use VPN, whether to use VM, whether to use PROXY, WIFI It may include usage status, terminal OS, terminal language, and the like. In addition, as transaction information, transaction classification (preliminary transaction/main transaction), transaction channel, reception classification (deposit/withdrawal), customer number, ID, terminal information, account number, counterparty account number, counterparty bank code, transaction amount, balance , the name of the account holder, the name of the other account holder, whether or not a corporate account, and the change record code can be set.

In the 1st comparison field operation step (step 1210), if it does not correspond to the specific case (NO), it ends, and the input data is profiled in the profiler 130 and the Redis unit 110 is stored in

In the first comparison field operation step (step 1210), if a specific condition is met (YES), the data required for analysis is compared with data profiled in the past based on a specific reference value. A 2 ^nd comparison profiled data step (step 1220) is performed.

In the case of analysis of anomalies in financial processing, examples of specific reference values in the second comparison profiled data comparison step 1220 may include the following.

- If there has been overseas use in the past on a customer basis,

- In the case of a terminal used in the past on a customer basis,

- Whether the account has been transferred in the past on a customer basis

When the specific reference value is satisfied (YES), it ends, and the input data is profiled in the profiler 130, converted into a key=value value, and stored in the Redis unit 110.

On the other hand, when the customer has no overseas transaction history in the past, used a terminal that has not been used in the past, and accounts with no transaction history in the past, that is, if the above specific reference value is not satisfied (NO), the aggregation step (step 1231) After going through, the anomaly symptom class aggregation for the customer is extracted (1231) from the memory 1232 and set, and the 3 ^rd comparison aggregation data step (step 1235) is performed for the given setting do.

In one embodiment of the present invention, the aggregation step (step 1231) includes counting the number of transactions based on customers who have no overseas transaction history, have used a terminal that has not been used in the past, and have traded with accounts without past transaction history. do.

In addition, in the aggregation data by pivot value step (step 1232), the operation calculated in step 1231 is temporarily stored in memory for a critical time, and the 3 ^rd comparison aggregation data step (step 1235) It may be performed by including a threshold comparison operation in

For example, "when the customer has no overseas transaction history in the past, uses a terminal that has not been used in the past, and trades two or more accounts with no transaction history in the past for 1 hour", in one embodiment of the present invention, the aggregation Step 1231 includes counting the number of transactions for 1 hour based on customers who have no overseas transaction history, have used a terminal that has not been used in the past, and have made transactions with accounts without past transaction history.

In addition, in the aggregation data by pivot value step (step 1232), the operation calculated in the aggregation step (step 1231) is temporarily stored in memory and the 3 ^rd comparison aggregation data step (step 1235) If more than 2 cases occur in , it is performed including detecting as an anomaly.

In the case of analysis of anomalies in financial processing, for example of the aggregation data data and comparative analysis set in the 3 ^rd comparison aggregation data step (step 1235), the following You may include more such as

- Configuration values: duration, reference value, target value, operator (count, sum), threshold value

- When the count for the reference value exceeds the threshold value during the reference time or duration

(Example> When the same ID (standard value) accesses more than 10 (threshold value) times in 1 hour)

- When the SUM threshold of the target value for the reference value is exceeded during the reference time or duration

(Example> Customers who transferred more than KRW 10 million (threshold value) of the cumulative amount (target value) for 30 minutes (standard value))

- When the unique COUNT threshold of the target value for the reference value is exceeded during the duration

(Example> When there are 10 (threshold) or more IDs (target value) accessed from the same terminal (standard value) for 1 hour)

- Keep the above operations in memory only for the duration and delete the past data from memory (why)

According to an embodiment of the present invention, anomaly symptom class aggregation for a corresponding customer is continuously accumulated in a memory for expected financial accidents.

For example, “If a customer with no transaction history abroad in the past transfers over 10 million won overseas to an account that has never been transferred to an account that has never been transferred to a mobile device that has not been used in the past, the set aggregation data ) The data is judged to be satisfied (YES), and the output is output as an anomaly transaction, and the input data required for analysis are profiled in the profiler 130 and stored in the Redis unit 110 do.

Alternatively, if it does not correspond to the set aggregation data (NO), the input data is profiled in the profiler 130 and stored in the Redis unit 110.

100: analysis system
110: Redis part
120: RTA analysis processor
130: profiler unit
150: 1st Kapukabu
160: parser
170: second kapukabu
200: financial server
500: customer terminal

Claims (10)

In the in-memory based anomaly real-time analysis system,
The analysis system,
a first Kafka unit formed of a plurality of kafka modules and receiving input data for analysis;
a parser unit parsing the input data transmitted from the first Kapuka unit and converting it into second input data that is normalized data;
a second kafka unit that distributes and transmits the second input data converted by the parser unit to a plurality of kafka modules;
an RTA analysis processor unit performing real-time analysis according to a given process on the second input data transmitted from the second KaPuka unit;
a profiler unit for profiling the second input data analyzed by the RTA analysis processor unit 120 according to a condition process set based on a specific reference value and converting the profiled data into a key value method; and
a Redis unit that stores the data converted in the key value method by the profiler unit in a memory; Analysis system comprising a. According to claim 1,
Wherein the first Kafka unit and the second Kafka unit are characterized in that 2 to 10 Kafka modules are formed in parallel. According to claim 1,
The RTA analysis processor unit is characterized in that a plurality of RTA analysis processors are formed, each processing volume is divided and analyzed and processed in parallel. According to claim 1,
In the profiler unit, the specific reference value includes a customer number or ID, and the set condition process includes any one or more of the terminal device, country, account number, transaction channel, and average transaction amount that have been transacted based on the specific reference value. An analysis system characterized in that for doing. According to claim 1,
The analysis system, characterized in that for receiving the input data through the customer interface and account system of the financial server.
In the analysis system of claim 1, the analysis method of real-time analysis of anomalies in the RTA analysis processor unit,
a first comparison field processing step of performing a process of comparing whether or not data corresponding to a specific condition among the second input data transmitted from the second KaPuka unit is performed;
In the first comparison field processing step, when the specific condition is met, a second profiled data comparison step in which a process of comparing the data required for analysis with data profiled in the past based on a specific reference value is performed. ; and
In the second profiled data comparison step, when the specific reference value is not satisfied, an anomaly symptom class aggregation for the corresponding customer is extracted from memory and set, and the second input data is compared with the given abnormal symptom class aggregation. A third aggregation data comparison step in which the process of performing is performed; Including,
Characterized in that the RTA analysis processor unit outputs the abnormal symptom transaction when it corresponds to the abnormal symptom class aggregation set in the third aggregation data comparison step. According to claim 6,
And if it does not correspond to the anomaly symptom class aggregation set in the third aggregation data comparison step, the second input data is profiled in the profiler and stored in the Redis unit. According to claim 6,
The specific condition of the first comparison field processing step is,
”The access country is not Korea and the access terminal is mobile and transfer transaction” or, as terminal information, terminal identification key, access IP, access country code, whether to use VPN, whether to use VM, whether to use PROXY, whether to use WIFI, terminal Including any one or more of OS and terminal language,
Or, as transaction information, transaction channel, deposit/withdrawal receipt classification, customer number, ID, terminal information, account number, counterparty account number, counterparty bank code, transaction amount, balance, account holder name, counterpart account holder name, corporate account status , Analysis method characterized in that it includes any one or more of the change record code. According to claim 6,
The specific reference value of the second profiled data comparison step,
Analysis method characterized in that it includes any one or more of whether there has been overseas use in the past on a customer basis, whether the terminal has been used in the past on a customer basis, and whether it is an account that has been transferred in the past on a customer basis. According to claim 6,
The abnormal symptom class aggregation set in the third aggregation data comparison step,
When the same ID accesses more than a set threshold during the reference time, the accumulated amount processed during the reference time includes more than one of the transfer customers, and the count for the reference value during the duration exceeds the threshold. Analysis method by

Images