CN117971819B - Management method and system for automatically collecting flow data - Google Patents

Abstract

A management method for automatically collecting running data includes: obtaining multiple pieces of running data; identifying the type name and type information of each piece of running data, and generating an associated mapping value based on the type information; judging whether its associated running data sets are all in the cache based on the mapping value of the running data, and if not, storing the running data in the cache; if so, judging the accuracy of the recognition based on the running data and its associated running data sets, and storing them in a relational database. The present invention adopts a new method for automatically collecting running data, which is different from the traditional method of only using regular expressions to identify running data, or using natural language processing for identification. This method has a very high recognition accuracy rate.

Description

Management method and system for automatically collecting flow data

Technical Field

The present invention belongs to the technical field of data analysis, and more specifically, relates to a management method and system for automatically collecting flow data.

Background technique

In modern economic activities, flow data, such as bank transaction records, electronic payment information, and communication bills, play a vital role. These data reflect the financial status, consumption habits, and communication patterns of individuals and businesses. However, since this information usually appears in the form of unstructured text, the integration, analysis, and management of data becomes extremely complex and time-consuming. In the current context, businesses and individuals need to collect and integrate flow data from various sources to support key business activities such as decision making, financial management, and customer service.

Currently, the technology for automatically aggregating transaction data mainly relies on methods such as regular expressions, natural language processing (NLP), data mining, and machine learning. By implementing advanced entity recognition and pattern matching algorithms, the system is able to identify key information in the text, such as dates, amounts, and parties involved. In addition, text classification and sentiment analysis techniques are used to further analyze the data to provide insights to users. The integration of these technologies has formed a system that can automatically process and aggregate transaction data, providing users with a clearer and more organized financial picture.

Despite significant progress, current technologies for automatically aggregating streaming data still have some shortcomings. First, the processing of complex and non-standardized texts with standard regular expressions remains a challenge. Second, natural language processing does not yet have a deep enough understanding of the implicit meaning and contextual information in the text, which limits the accuracy of information extraction and the scope of application of the system.

Summary of the invention

In order to solve the deficiencies in the prior art, the purpose of the present invention is to solve the above-mentioned defects and further propose a management method and system for automatically collecting flow data.

The present invention adopts the following technical solution.

The first aspect of the present invention discloses a management method for automatically collecting flow data, comprising steps 1 to 3;

Step 1, obtain multiple copies of flow data;

Step 2: Identify the type name and type information of each piece of pipeline data, and generate an associated mapping value based on the type information;

Step 3, based on the mapping value of the flow data, determine whether its associated flow data sets are all in the cache. If not, store the flow data in the cache; if so, determine the accuracy of the recognition based on the flow data and its associated flow data sets, and store them in the associative database.

Furthermore, before obtaining the multiple pieces of flow data, the multiple pieces of flow data are pre-processed, including data cleaning and standardization.

Furthermore, generating the associated mapping value based on the type information specifically includes: generating the associated mapping value based on the transaction counterparty information or the transaction platform information in the type information.

Furthermore, in step 2, regular expressions are used to identify and segment the pipeline data.

Furthermore, based on the type information, the associated mapping values are generated specifically as follows: based on the transaction time in the type information, a first mapping value is generated, and based on multiple scarce keywords in the transaction counterparty information or the transaction platform information in the type information, multiple second mapping values are generated; wherein a scarce keyword refers to a Chinese character or a word.

Further, in step 3, based on the mapping value of the pipeline data, determining whether the associated pipeline data sets are all in the cache specifically includes steps 3.1 to 3.3;

Step 3.1, based on the first mapping value, obtaining a first subsequent mapping value;

Step 3.2, based on the first mapping value of the pipeline data, obtaining all pipeline data in the first mapping value and the first subsequent mapping value as the pipeline data set to be compared;

Step 3.3, compare the second mapping value of the pipeline data with the second mapping value of each pipeline data in the pipeline data set to be compared in turn, if the similarity between the two is greater than or equal to a preset similarity threshold, then determine that each pipeline data is associated pipeline data.

Furthermore, the process of selecting scarce keywords may specifically include steps S101 to S103;

Step S101, obtaining each keyword in the transaction counterparty information or the transaction platform information in the type information in the transaction data;

Step S102, calculating the mapping value corresponding to each keyword according to Huffman coding;

Step S103, sorting the mapping values from high to low, selecting n mapping values with the largest mapping values as multiple second mapping values, where n is the number of second mapping values.

The second aspect of the present invention discloses a management system for automatically collecting flow data, which is applied to the method described in the first aspect, and comprises: a data acquisition module, a logic judgment module, a first data storage module and a second data storage module;

The data acquisition module is used to obtain multiple copies of flow data;

The logic judgment module is used to identify the type name and type information of each piece of pipeline data, and generate an associated mapping value based on the type information; and based on the mapping value of the pipeline data, determine whether its associated pipeline data set is in the cache; and based on the pipeline data and its associated pipeline data set, determine the accuracy of the recognition;

The first data storage module is used to store the running data in the cache;

The second data storage module is used to store the synthesis into a relational database.

The third aspect of the present invention discloses a terminal, including a processor and a storage medium; the characteristics are:

The storage medium is used to store instructions;

The processor is used to operate according to the instructions to execute the steps of the method described in the first aspect.

The fourth aspect of the present invention discloses a computer-readable storage medium having a computer program stored thereon, characterized in that when the program is executed by a processor, the steps of the method described in the first aspect are implemented.

The beneficial effects of the present invention are that, compared with the prior art, the present invention has the following advantages:

The present invention adopts a new method for automatically collecting running data, which is different from the traditional method of only using regular expression format to identify running data, or using natural language processing to identify. The present method has a very high recognition accuracy rate.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flow chart of a management method for automatically collecting flow data according to an embodiment of the present invention.

Detailed ways

The present application is further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present application.

The present invention discloses a management method for automatically collecting flow data, as shown in FIG1 , which may include steps 1 to 3.

Step 1: Obtain multiple copies of transaction data.

The transaction data can be bank account, electronic payment platform (such as Alipay, WeChat, etc.), or transaction data of other communication service providers. The transaction data is text data with various formats and structures.

In some embodiments, before obtaining multiple pieces of transaction data, the multiple pieces of transaction data are pre-processed, including data cleaning and standardization. Data cleaning is used to remove irrelevant information and reduce the amount of transaction data. Standardization is used to uniformly format all transaction data and identify key information therein, such as transaction date, amount, transaction party, transaction description, etc., for later storage.

Step 2: Identify the type name and type information of each piece of flow data, and generate an associated mapping value based on the type information.

Among them, entity information may include transaction time, transaction amount, transaction type, transaction party information, transaction party information, transaction platform information, order number, etc.

In the first embodiment of the present invention, the associated mapping value can be generated based on the transaction counterparty information or transaction platform information in the type information. Assuming that a piece of transaction data X can contain the following information: transaction party information: "x0", transaction counterparty information: "x1", then its associated mapping value is only the mapping value corresponding to "x1"; assuming that a piece of transaction data Y can contain the following information: transaction party information: "y0", transaction counterparty information: "y1", transaction platform information: "y2", then its associated mapping value is the 2 mapping values corresponding to "y1" and "y2" respectively.

In the first embodiment, the mapping value may be generated by a hash function, and its specific formula will not be described in detail.

Step 3, based on the mapping value of the flow data, determine whether its associated flow data sets are all in the cache. If not, store the flow data in the cache; if so, determine the accuracy of the recognition based on the flow data and its associated flow data sets, and store them in the associative database.

In some embodiments, the cache may select a redis database.

It can be understood that, assuming that the running data is running data X, the number of running data in its associated running data set is at least 1; assuming that the running data is running data Y, the number of running data in its associated running data set is at least 2. In step 3, the comprehensive storage in the associated database means not only storing the running data in the associated database, but also transferring its associated running data set from the cache to the associated database.

In step 3, taking transaction data Y as an example, at least two transaction data S and T respectively associated with its two corresponding mapping values should be the same transaction data as transaction data Y, and only the information of the transaction parties and the transaction platform information has changed.

In modern data processing practice, automatically identifying and aggregating flow data is an important and challenging task. Standardized mechanical identification of the entity information of each flow data, such as quickly identifying the entity information in the flow data through a preset regular expression, is a common method, but this method has its inherent limitations. In particular, the order and type of entity types contained in the flow data may not be the same, affecting the consistency and accuracy of data processing. In addition, problems in the data acquisition process, such as page number conversion errors, may cause flow records to be incorrectly merged or split, or mixed with non-related information, further reducing the accuracy of automated processing. Even if the success rate of automatically aggregating flow data is as high as 99%, the subsequent checking and filling of gaps may still result in high costs, thereby offsetting the efficiency advantages brought by automation.

In other embodiments, the introduction of natural language processing (NLP)-based technology can improve the recognition rate of entities in texts, however, its recognition accuracy is not even as high as regular expressions, and natural language processing cannot solve the problem of data definition. In other words, it is difficult for existing technologies to accurately define which content belongs to a specific log record, or does not belong to any log data at all.

In the embodiment of the present invention, since step 3 determines the accuracy of recognition by cross-validation (i.e., based on the running data and its associated running data set), the cross-validation method can easily solve the problems of recognition accuracy and data definition. Therefore, in step 2, it is entirely possible to quickly identify and segment the running data by regular expressions. However, there are still defects in the above steps. First and foremost, the purpose of the mapping value of the present invention is to accurately classify and identify the running data, and the mapping value itself is uniquely determined by the type information in the running data (for example, transaction counterparty information or transaction platform information). Therefore, once the classification and recognition by regular expression is wrong, the mapping value is wrong, which essentially forms a chicken or egg problem. Secondly, since the running data is usually processed in batches (usually the transaction party information is the same), it is not difficult to understand that the first batch of running data must be stored in the cache, and the cache is usually the memory, which may not be able to adapt to the huge amount of data in terms of space.

Based on this, in the second embodiment of the present invention, based on the type information, the associated mapping values are generated specifically as follows: based on the transaction time in the type information, a first mapping value is generated, and based on multiple scarce keywords in the transaction counterparty information or the transaction platform information in the type information, multiple second mapping values are generated; wherein the scarce keyword (and the keyword hereinafter) refers to a Chinese character or a word.

The advantage of using transaction time to generate associated mapping values is that the format of the transaction time itself is fixed, and the order of the transaction data is usually sorted according to the transaction time. The upper and lower transaction data can be associated for cross-validation to obtain the accurate transaction time.

More specifically, considering that the running data may have a time difference with its associated running data Therefore, the first mapping value should cover the entire mapping interval (that is, [/> ]). Therefore, in the second embodiment, the first mapping value/> A timestamp can be associated with the transaction time as shown below:

in, is the remainder operator, /> The timestamp of the transaction time, /> is greater than/> An integer.

It can be understood that the second embodiment essentially utilizes the idea of bucket sorting, namely: first generate a first mapping value (essentially a mapping interval, each mapping interval is regarded as a barrel) according to the transaction time to roughly determine in which barrel the transaction data is; and then utilize scarce keywords, such as through a hash table, to determine the specific location of the transaction data.

Therefore, in step 3, based on the mapping value of the pipeline data, determining whether its associated pipeline data sets are all in the cache specifically includes steps 3.1 to 3.3.

Step 3.1, based on the first mapping value, obtain a first subsequent mapping value.

The first successor mapping value refers to the next mapping value of the first mapping value. .

Step 3.2: Based on the first mapping value of the pipeline data, obtain all pipeline data in the first mapping value and the first subsequent mapping value as the pipeline data set to be compared.

Step 3.3, compare the second mapping value of the pipeline data with the second mapping value of each pipeline data in the pipeline data set to be compared in turn, if the similarity between the two is greater than or equal to a preset similarity threshold, then determine that each pipeline data is associated pipeline data.

It is understandable that in the cache, the multiple second mapping values of the associated pipeline data should also be sorted from high to low to form a vector. Generally, the total number n of second mapping values should be at least greater than or equal to 5, and the preset similarity threshold can be set to 80%. The similarity between the two refers to the ratio of the number of repeated data in the second mapping values of the two pipeline data to the total number.

It should be noted that in step 3.3, if the similarity between the two is greater than or equal to the preset similarity threshold, it is usually necessary to further compare the two pieces of pipeline data information to further determine whether each of the pipeline data is associated pipeline data. However, this determination method is nothing more than continuing to expand the length of the number of second mapping values for comparison to prevent the previous n from being too small, resulting in inaccurate judgment. Accordingly, when the number of second mapping values expands, the similarity threshold should also be increased. The specific process will not be repeated.

In order to prevent the scarce keywords from being common keywords in the flow data, in the embodiment of the present invention, the process of selecting the scarce keywords may specifically include steps S101 to S103.

Step S101, obtaining each keyword in the transaction counterparty information in the type information or the transaction platform information in the transaction data.

Step S102: Calculate the mapping value corresponding to each keyword according to Huffman coding.

Step S103, sorting the mapping values from high to low, selecting n mapping values with the largest mapping values as multiple second mapping values, where n is the number of second mapping values.

It can be understood that the keywords corresponding to the largest n second mapping values are scarce keywords.

It should be noted that the use of Huffman Coding in step S102 is not for compressing data, but because the core idea of Huffman Coding is to assign bit sequences of unequal lengths, i.e., codes, according to the frequency or probability of occurrence of each keyword. In the Huffman coding process, the keyword with the highest frequency of occurrence is assigned the shortest mapping code, while the keyword with the lowest frequency of occurrence is assigned a longer mapping code. Therefore, the mapping value corresponding to each keyword in step S102 is the mapping code in the Huffman coding corresponding to the keyword.

Correspondingly, the present invention also discloses a management system for automatically collecting flow data, including: a data acquisition module, a logic judgment module, a first data storage module and a second data storage module;

The data acquisition module is used to obtain multiple copies of flow data;

The logic judgment module is used to identify the type name and type information of each piece of pipeline data, and generate an associated mapping value based on the type information; and based on the mapping value of the pipeline data, determine whether its associated pipeline data set is in the cache; and based on the pipeline data and its associated pipeline data set, determine the accuracy of the recognition;

The first data storage module is used to store the running data in the cache;

The second data storage module is used to store the synthesis into a relational database.

Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention may include non-volatile and/or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory.

By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims (6)

1. A management method for automatically collecting flow data, characterized in that the method comprises steps 1 to 3; Step 1, obtain multiple copies of flow data; Step 2, identifying the type name and type information of each piece of flow data, and generating an associated mapping value based on the type information; generating the associated mapping value based on the type information specifically includes: generating a first mapping value based on the transaction time in the type information, and generating multiple second mapping values based on multiple scarce keywords in the transaction counterparty information or the transaction platform information in the type information; wherein the scarce keyword refers to a Chinese character or a word; The process of selecting scarce keywords specifically includes steps S101 to S103; Step S101, obtaining each keyword in the transaction counterparty information or the transaction platform information in the type information in the transaction data; Step S102, calculating the mapping value corresponding to each keyword according to Huffman coding; Step S103, sorting the mapping values from high to low, selecting n mapping values with the largest mapping values as a plurality of second mapping values, where n is the number of second mapping values; Step 3, based on the mapping value of the water flow data, determine whether the water flow data sets associated with it are all in the cache, if not, store the water flow data in the cache; if so, determine the accuracy of the recognition based on the water flow data and its associated water flow data sets, and store them in the association database; In step 3, based on the mapping value of the pipeline data, determining whether the associated pipeline data sets are all in the cache specifically includes steps 3.1 to 3.3; Step 3.1, based on the first mapping value, obtaining a first subsequent mapping value; Step 3.2, based on the first mapping value of the pipeline data, obtaining all pipeline data in the first mapping value and the first subsequent mapping value as the pipeline data set to be compared; Step 3.3, compare the second mapping value of the pipeline data with the second mapping value of each pipeline data in the pipeline data set to be compared in turn, if the similarity between the two is greater than or equal to a preset similarity threshold, then determine that each pipeline data is associated pipeline data. 2. According to a management method for automatically collecting flow data as described in claim 1, it is characterized in that before obtaining multiple pieces of flow data, it also includes pre-processing the multiple pieces of flow data, including: data cleaning and standardization. 3. A management method for automatically collecting flow data according to claim 1, characterized in that in step 2, the flow data is identified and segmented by means of regular expressions. 4. A management system for automatically collecting flow data, applied to the method according to any one of claims 1 to 3, characterized in that the system comprises: a data acquisition module, a logic judgment module, a first data storage module and a second data storage module; The data acquisition module is used to obtain multiple copies of flow data; The logic judgment module is used to identify the type name and type information of each piece of pipeline data, and generate an associated mapping value based on the type information; and based on the mapping value of the pipeline data, determine whether its associated pipeline data set is in the cache; and based on the pipeline data and its associated pipeline data set, determine the accuracy of the recognition; The first data storage module is used to store the running data in the cache; The second data storage module is used to store the synthesis into a relational database. 5. A terminal comprising a processor and a storage medium; characterized in that: The storage medium is used to store instructions; The processor is configured to operate according to the instructions to execute the steps of the method according to any one of claims 1-3. 6. A computer-readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the steps of the method according to any one of claims 1 to 3.