CN116737682A - Log processing method, device, equipment and storage medium based on transaction data - Google Patents
Log processing method, device, equipment and storage medium based on transaction data Download PDFInfo
- Publication number
- CN116737682A CN116737682A CN202310702171.8A CN202310702171A CN116737682A CN 116737682 A CN116737682 A CN 116737682A CN 202310702171 A CN202310702171 A CN 202310702171A CN 116737682 A CN116737682 A CN 116737682A
- Authority
- CN
- China
- Prior art keywords
- data
- log
- transaction
- file
- log file
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 22
- 238000012545 processing Methods 0.000 claims abstract description 97
- 238000000034 method Methods 0.000 claims abstract description 40
- 238000013500 data storage Methods 0.000 claims description 6
- 230000000875 corresponding effect Effects 0.000 description 89
- 238000004891 communication Methods 0.000 description 10
- 238000010586 diagram Methods 0.000 description 8
- 238000005516 engineering process Methods 0.000 description 6
- 238000013075 data extraction Methods 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 230000005236 sound signal Effects 0.000 description 4
- 230000009286 beneficial effect Effects 0.000 description 3
- 230000001133 acceleration Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000004590 computer program Methods 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 238000007726 management method Methods 0.000 description 2
- 230000009471 action Effects 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000013475 authorization Methods 0.000 description 1
- 230000002596 correlated effect Effects 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 230000000977 initiatory effect Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000005192 partition Methods 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 230000008685 targeting Effects 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q40/00—Finance; Insurance; Tax strategies; Processing of corporate or income taxes
- G06Q40/04—Trading; Exchange, e.g. stocks, commodities, derivatives or currency exchange
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Business, Economics & Management (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- General Engineering & Computer Science (AREA)
- Accounting & Taxation (AREA)
- Finance (AREA)
- Development Economics (AREA)
- Economics (AREA)
- Marketing (AREA)
- Strategic Management (AREA)
- Technology Law (AREA)
- General Business, Economics & Management (AREA)
- Fuzzy Systems (AREA)
- Mathematical Physics (AREA)
- Probability & Statistics with Applications (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Debugging And Monitoring (AREA)
Abstract
The application provides a log processing method, device and equipment based on transaction data and a storage medium, which can be used in the technical field of big data. The method comprises the following steps: acquiring a transaction log and determining target data in the transaction log; the transaction log comprises transaction data of a transaction, and the target data is used for representing transaction data of a first data category preset in the transaction log; generating a blank log file corresponding to the target data, and distributing a file identifier for the blank log file; storing the transaction data in a blank log file; storing the file identification association corresponding to the target data and the log file containing the transaction data in a preset database, and storing the log file containing the transaction data and the corresponding file identification association in a preset backup log server. According to the application, the target data is determined from the transaction log, so that the flexibility and pertinence of transaction data processing are improved.
Description
Technical Field
The present application relates to the field of big data technologies, and in particular, to a method, an apparatus, a device, and a storage medium for processing logs based on transaction data.
Background
The transaction log may be used to record information about the entire process of a transaction from initiation to termination, e.g., customer operation information, background access records, application resource conditions, etc. may be recorded in the transaction log. Transaction logs are used for a variety of purposes including problem tracking and locating, data extraction, data processing, and the like. The more detailed the log record, the more information is contained, but the larger the corresponding log volume will be.
Currently, most distributed systems for transactions generate log files over time. For example, data in all transaction logs in one hour are put into one log file, so that thousands of log files are generated, no targeted processing is performed on the transaction data in the transaction log, and the processing precision of the transaction log is low.
Disclosure of Invention
The application provides a log processing method, device and equipment based on transaction data and a storage medium, which are used for improving the processing precision of transaction logs.
In a first aspect, the present application provides a log processing method based on transaction data, including:
acquiring a transaction log and determining target data in the transaction log; the transaction log comprises transaction data of a transaction, wherein the target data is used for representing transaction data of a preset first data type in the transaction log;
Generating a blank log file corresponding to the target data, and distributing a file identifier for the blank log file; the log file is used for storing transaction data in the transaction log;
storing the transaction data in the blank log file to obtain a log file containing the transaction data;
storing the target data and the file identification association corresponding to the log file containing the transaction data in a preset database, and storing the log file containing the transaction data and the corresponding file identification association in a preset backup log server; the backup log server is used for carrying out backup storage on the log file containing the transaction data.
In a second aspect, the present application provides a log processing device based on transaction data, comprising:
the data determining module is used for acquiring a transaction log and determining target data in the transaction log; the transaction log comprises transaction data of a transaction, wherein the target data is used for representing transaction data of a preset first data type in the transaction log;
the file generation module is used for generating a blank log file corresponding to the target data and distributing a file identifier for the blank log file; the log file is used for storing transaction data in the transaction log;
The data storage module is used for storing the transaction data in the blank log file to obtain a log file containing the transaction data;
the file backup module is used for storing the association of the target data and the file identifier corresponding to the log file containing the transaction data in a preset database, and storing the association of the log file containing the transaction data and the corresponding file identifier in a preset backup log server; the backup log server is used for carrying out backup storage on the log file containing the transaction data.
In a third aspect, the present application provides an electronic device comprising: a processor, and a memory communicatively coupled to the processor;
the memory stores computer-executable instructions;
the processor executes computer-executable instructions stored in the memory to implement the transaction data based log processing method according to the first aspect of the present application.
In a fourth aspect, the present application provides a computer-readable storage medium having stored therein computer-executable instructions which, when executed by a processor, are adapted to carry out a transaction data based log processing method according to the first aspect of the present application.
In a fifth aspect, the present application provides a computer program product comprising a computer program which, when executed by a processor, implements the transaction data based log processing method of the first aspect of the present application.
The application provides a log processing method, device, equipment and storage medium based on transaction data, which are used for acquiring transaction logs in a period of time, wherein each transaction log can comprise a plurality of transaction data. For each transaction log, target data is determined from a plurality of transaction data. The target data corresponds to a log file, and transaction data of a transaction log to which the target data belongs is stored in the corresponding log file to obtain an association relationship between the target data and the log file. Each log file is provided with a unique file identification, the target data and the file identifications of the corresponding log files are associated and stored in a preset database, and the log files and the corresponding file identifications are associated and stored in a preset backup log server, so that the log files can be conveniently searched from the backup log server according to the target data. The method avoids mixing a plurality of transaction logs in one file for storage, realizes targeted processing of the transaction logs according to different target data, and improves log processing precision.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the application and together with the description, serve to explain the principles of the application.
Fig. 1 is a flow chart of a log processing method based on transaction data according to an embodiment of the present application;
fig. 2 is a flow chart of a log processing method based on transaction data according to an embodiment of the present application;
fig. 3 is a block diagram of a log processing device based on transaction data according to an embodiment of the present application;
fig. 4 is a block diagram of a log processing device based on transaction data according to an embodiment of the present application;
fig. 5 is a block diagram of an electronic device according to an embodiment of the present application;
fig. 6 is a block diagram of an electronic device according to an embodiment of the present application.
Specific embodiments of the present application have been shown by way of the above drawings and will be described in more detail below. The drawings and the written description are not intended to limit the scope of the inventive concepts in any way, but rather to illustrate the inventive concepts to those skilled in the art by reference to the specific embodiments.
Detailed Description
Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numbers in different drawings refer to the same or similar elements, unless otherwise indicated. The implementations described in the following exemplary examples do not represent all implementations consistent with the application. Rather, they are merely examples of apparatus and methods consistent with aspects of the application as detailed in the accompanying claims.
It should be noted that, the user information (including but not limited to user equipment information, user personal information, etc.) and the data (including but not limited to data for analysis, stored data, presented data, etc.) related to the present application are information and data authorized by the user or fully authorized by each party, and the collection, use and processing of the related data need to comply with related laws and regulations and standards, and provide corresponding operation entries for the user to select authorization or rejection.
It should be noted that the log processing method, device, equipment and storage medium based on transaction data of the present application can be used in the big data field, and also can be used in any field other than the big data field.
The transaction log is used for recording all transaction data of a transaction from the beginning to the end, for example, the transaction log can record client operation information, background access records, application resource conditions and the like. Transaction logs are used for a variety of purposes including problem tracking and locating, data extraction, data processing, and the like. The more detailed the transaction log record, the more information is contained, but the greater the corresponding log file volume will be.
Current distributed systems most partition transaction logs by time, for example, placing a transaction log of all transactions that occur within an hour into a log file. On a daily basis, a distributed system may generate thousands of log files, and log backups are also simple and rough copies to backup servers. That is, the transaction logs are not subjected to targeted classification processing, the processing process of the transaction logs is only uniformly stored, and the targeting and the accuracy of the processing are low. If the transaction record of a certain customer on a certain day before is to be called, the log files need to be extracted one by one from thousands of log files on the day, and carpet search is performed, so that the search efficiency and the search precision are affected.
The application provides a log processing method, device and equipment based on transaction data and a storage medium, and aims to solve the technical problems in the prior art.
The following describes the technical scheme of the present application and how the technical scheme of the present application solves the above technical problems in detail with specific embodiments. The following embodiments may be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments. Embodiments of the present application will be described below with reference to the accompanying drawings.
Fig. 1 is a flow chart of a log processing method based on transaction data according to an embodiment of the present application, where the method is performed by a log processing device based on transaction data. As shown in fig. 1, the method comprises the steps of:
s101, acquiring a transaction log and determining target data in the transaction log; the transaction log comprises transaction data of a transaction, and the target data is used for representing the transaction data of a preset first data type in the transaction log.
By way of example, the user performs transactions of various service types through a preset transaction Application, which may be a bank APP (Application program) or the like. For example, the user may conduct a transaction such as a transfer. Each transaction corresponds to a transaction log, and the transaction log comprises various transaction data generated from the beginning to the end of the transaction. For example, the transaction data may include information such as user account number, transaction type, transaction time, and transaction amount.
For a large distributed system, multiple nodes of transaction applications may be included. Each transaction application may generate a transaction log in real-time based on the user's transactions. When processing the transaction log, a preset log processing system can be adopted, and the log processing system can be used for executing the log processing method based on the transaction data in the embodiment of the application. The log processing system and the distributed system where the transaction application is located are independent and asynchronous systems, so that the time consumption and the success rate of the transaction are not affected when the log processing is carried out.
The log processing system may obtain a transaction log of transactions, for example, may obtain a transaction log of one transaction, or may obtain a transaction log of a plurality of transactions. For each transaction log, target data may be determined from the transaction data of the transaction log. The data category of the target data may be preset as the first data category. And determining the transaction data in the first data category in the transaction log as target data. For example, if the first data class is the user account, the user account in the transaction log is determined to be the target data. The first data category may be determined in advance from different dimensions, e.g., may be set from a customer dimension, a transaction dimension, a resource dimension, and the like. The customer dimension may be a user account of the customer, the transaction dimension may be a transaction type of the transaction, and the resource dimension may be a CPU, a memory, a disk, a time consuming, and the like used by the transaction.
In this embodiment, obtaining the transaction log includes: and acquiring the transaction log generated in the data acquisition period from the preset transaction application according to the preset data acquisition period.
Specifically, a data acquisition period is preset, and every other data acquisition period, a transaction log generated in the data acquisition period can be acquired from a preset transaction application. For example, the data acquisition period may be one hour. The log processing system may obtain a transaction log generated by a transaction application every other hour. Each transaction log obtained may be at least one, and each transaction log corresponds to its own target data. The target data for each transaction log may be the same or different.
The log quantity of the distributed system is large, information is disordered, the efficiency and the accuracy of log acquisition can be improved by automatically acquiring transaction logs in batches, log processing is converted from online processing to asynchronous processing, and the influence on the application using process is reduced.
S102, generating a blank log file corresponding to target data, and distributing a file identifier for the blank log file; the log file is used for storing transaction data in the transaction log.
Illustratively, the transaction data in the transaction log is stored in a log file, and thus, a blank log file needs to be generated to store the transaction data. The format of the log file may be preset, and a blank log file corresponding to the target data is generated, for example, for each target data, a corresponding blank log file is generated; alternatively, for a plurality of target data, a corresponding blank log file is generated. Each blank log file is assigned a unique file identification, which can be used to represent the corresponding log file.
If each target data corresponds to a log file, each log file may store a plurality of transaction data in a transaction log corresponding to the target data. If the plurality of target data corresponds to one log file, transaction data in a transaction log corresponding to the plurality of target data can be stored in one log file. For example, transaction data in a plurality of transaction logs in which target data is the same is stored in one log file.
In this embodiment, the transaction logs are at least two; generating a blank log file corresponding to the target data, including: determining target data in the same first data category as the same category data; performing numerical comparison on the same-class data to determine the same-class data in the same-class data; and generating a blank log file corresponding to the same numerical value data.
Specifically, there may be a plurality of obtained transaction logs, that is, a plurality of target data. The first data category of the target data of each transaction log may be the same or different. For example, two first data categories may be preset for 10 transaction logs, one for the first five transaction logs and the other for the last five transaction logs. In this embodiment, the first data category of each target data may be set according to the actual requirement.
After determining the target data for each transaction log, it is determined whether the first data category for each target data is the same. And determining the target data under the same first data category as the same category data. For example, there are 10 pieces of target data, and if the first data type of the first five pieces of target data is the same, the first five pieces of target data are the same type data. And determining the specific numerical content of the same class data, for example, determining the represented user account when the same class data is the data representing the user account. And comparing the numerical values of the same class data, judging whether the same class data are the same, and if so, determining the same class data with the same numerical value as the same numerical value data. Generating a blank log file corresponding to the same numerical value data, for example, three target data are all user accounts, wherein two user accounts are 001, and the other user account is 002, and if the data of the two user accounts of 001 are the same numerical value data, a corresponding log file can be generated for the two target data. That is, one log file may be generated for a plurality of target data.
The method has the advantages that for the same target data, transaction data corresponding to the target data can be stored in one log file, so that all transaction data related to the target data can be found conveniently, each transaction log does not need to correspond to one log file, information storage is excessively scattered, pertinence of log processing is improved, and further efficiency of log query is improved.
In this embodiment, the method further includes: if the same type of data does not exist in the target data, generating a blank log file corresponding to each target data; and if the same numerical value data does not exist in the same type of data, generating a blank log file corresponding to each target data in the same type of data.
Specifically, if the same type of data does not exist in the target data, that is, the first data types of the target data are different, a corresponding blank log file is generated for each target data. If the same type of data exists in the target data, one or more target data also exist, and the first data type of the other target data is different from that of the other target data, a corresponding blank log file is generated for each target data which is not the same type of data. For example, there are ten target data, wherein the first data types of eight target data are the same, and the first data types of the other two target data are different, and a corresponding blank log file is generated for each of the two target data.
After determining that the target category data exist, determining whether the numerical values of all the target category data are the same, and if the numerical values of all the target category data are different, generating a corresponding blank log file for all the target category data; if the same numerical value data exists in the target category data and target data different from the numerical value exists, generating a corresponding blank log file for the target data which is not the same numerical value data. For example, there are four same category data, where there are two same numerical data, and for the other two same category data that are not the same numerical data, a corresponding blank log file is generated.
The method has the advantages that for target data with different categories or different values, the transaction data can be stored differently by generating the corresponding log files, so that data confusion is avoided, and the log processing precision is improved.
S103, storing the transaction data in the blank log file to obtain the log file containing the transaction data.
In an exemplary embodiment, transaction data in a transaction log to which target data belongs is determined, and the transaction data is stored in a blank log file corresponding to the target data, so as to obtain a log file containing the transaction data. All transaction data in the transaction log to which the target data belongs may be stored in the corresponding log file, or part of the transaction data may be stored in the corresponding log file. Transaction data for one transaction may be stored in one log file, or transaction data for a plurality of transactions may be stored in one log file. For example, if the target data is a user account and the target data of two transaction logs are the same user account, the transaction data of the two transactions may be stored in one log file, where the log file is a log file corresponding to the user account.
S104, storing the file identification association corresponding to the target data and the log file containing the transaction data in a preset database, and storing the log file containing the transaction data and the corresponding file identification association in a preset backup log server; the backup log server is used for carrying out backup storage on the log file containing the transaction data.
For example, after obtaining the log file containing the transaction data, the log file may be stored, for example, the log file may be stored in a preset database, or the log file may be backed up. When the log file is stored in the database, all data of the log file can be not stored, so that the occupation of storage space is reduced. Target data and file identifications corresponding to the log files can be determined, and the target data and the corresponding file identifications are stored in a preset database in an associated mode. In addition to storing the association relationship between the target data and the file identifier, part or all of the transaction data in the log file may be stored, the transaction data required to be stored in the database may be preset, and for example, the transaction code, the transaction time, and the like may be stored in the database.
And presetting a backup log server, and carrying out backup storage on the log file containing the transaction data. The backup log server can store complete log files and file identifications corresponding to the log files, so that the log files can be quickly positioned according to the file identifications, and data in the log files can be obtained.
In this embodiment, a time stamp is marked in the log file, and the time stamp is used to represent the generation time of the log file; after storing the log file containing the transaction data and the corresponding file identification in a preset backup log server in an associated manner, the method further comprises the following steps: acquiring current time, and determining the time difference between the current time and the generation time of the log file according to the time stamp of the log file; and if the time difference is equal to or greater than the preset time difference threshold, deleting the log file from the backup log server.
Specifically, each log file may be marked with a time stamp, and the time stamp may be used to represent the time of generation of the log file. After the log files are stored in the backup log server, the log files in the backup log server can be updated in real time or at regular time, so that the data redundancy in the backup log server is avoided.
The current time can be acquired in real time or at regular time, and the generation time of the log file is determined according to the time stamp of the log file. The time difference between the current time and the generation time is determined and can be used as the storage time when the backup log server has stored the log file. A time difference threshold is preset, and the calculated time difference is compared with the time difference threshold. If the time difference is smaller than the time difference threshold, the backup log server continues to store the log file; if the time difference is equal to or greater than the time difference threshold, the log file can be deleted from the backup log server, so that the backup log server can store other log files conveniently.
The backup log server has the beneficial effects that the backup log server can be updated in time, data confusion caused by excessive stored data is avoided, subsequent storage of new log files and quick searching of transaction data are facilitated, and the efficiency and the accuracy of data searching are improved.
In this embodiment, the method further includes: responding to the log file inquiring instruction, and determining data to be inquired from the log file inquiring instruction; the data to be queried is used for representing target data corresponding to the log file to be queried; searching a file identifier corresponding to the data to be queried from a database, wherein the file identifier is a target identifier; and searching the log file corresponding to the target identifier from the backup log server to obtain the log file to be queried.
Specifically, the user can query the stored log file at any time, and the user can send a log file query instruction, wherein the log file query instruction can comprise target data corresponding to the log file to be queried as the data to be queried. For example, if a user wants to query a transaction of a certain user, the data to be queried may be the account number of the user, i.e. query all transactions performed under the account number of the user. When a user sends out a log file inquiry command, the user can input the data to be inquired to search the log file corresponding to the data to be inquired.
Responding to the log file inquiring instruction, and determining the data to be inquired, namely the target data corresponding to the log file to be inquired, from the log file inquiring instruction. The database stores the association relation between the target data and the file identifications, and the file identifications corresponding to the data to be queried are searched from the database and used as the target identifications. The backup server stores the log files and the corresponding file identifications, namely the log files corresponding to the target identifications can be quickly positioned from the backup log server according to the target identifications and used as the log files to be queried, so that transaction data are searched from the log files to be queried.
The method has the advantages that the query request of the external business task is received, the position of the log file is rapidly positioned according to the information recorded by the database and the dimension to be queried, the information in the log file is extracted, and the query precision and efficiency of the data are improved.
The embodiment of the application provides a log processing method based on transaction data, which is used for acquiring transaction logs in a period of time, wherein each transaction log can comprise a plurality of transaction data. For each transaction log, target data is determined from a plurality of transaction data. The target data corresponds to a log file, and transaction data of a transaction log to which the target data belongs is stored in the corresponding log file to obtain an association relationship between the target data and the log file. Each log file is provided with a unique file identification, the target data and the file identifications of the corresponding log files are associated and stored in a preset database, and the log files and the corresponding file identifications are associated and stored in a preset backup log server, so that the log files can be conveniently searched from the backup log server according to the target data. The method avoids mixing a plurality of transaction logs in one file for storage, realizes targeted processing of the transaction logs according to different target data, and improves log processing precision.
Fig. 2 is a flow chart of a log processing method based on transaction data according to an embodiment of the present application, where the embodiment is an alternative embodiment based on the foregoing embodiment.
In this embodiment, determining the target data in the transaction log may be refined as: acquiring preset log processing configuration information; the log processing configuration information comprises data types of transaction data to be extracted, wherein the data types are first data types; and extracting the transaction data of the first data category from the transaction log according to the log processing configuration information, and taking the transaction data as target data.
As shown in fig. 2, the method comprises the steps of:
s201, acquiring a transaction log and acquiring preset log processing configuration information; the log processing configuration information comprises data types of transaction data to be extracted, and the data types are first data types.
Illustratively, transaction logs of a plurality of transactions are periodically obtained, whereby the transaction logs are data processed. When processing the transaction log, specific transaction data needs to be determined from the transaction log as target data. The method comprises the steps of presetting log processing configuration information, wherein the log processing configuration information is a rule configured for processing data in a transaction log. The log processing configuration information may include a data type of transaction data to be extracted, as a first data type, and the transaction data to be extracted is target data. That is, the log processing configuration information may be provided with a data category of the target data. For example, a worker may manually enter a data category and save log processing configuration information. The latest log processing configuration information stored in advance can be obtained each time the transaction log is processed.
S202, extracting transaction data of a first data category from the transaction log according to log processing configuration information, and taking the transaction data as target data.
Illustratively, after the log processing configuration information is obtained, a preset first data category is determined from the log processing configuration information. Transaction data of the first data category is extracted from the transaction log as target data. One or more first data categories may be set in the log processing configuration information, the target data of each transaction log may be one or more, and the target data of different transaction logs may be different. For example, for a transaction log, the target data may be a transaction type and a user account, and the transaction data in the transaction log may be stored in a log file corresponding to the transaction type or in a log file corresponding to the user account. When the data is queried, the corresponding transaction data under a certain transaction type can be searched, and the corresponding transaction data under a certain user account can be queried. That is, the transaction data in one transaction log may be stored in only one log file or may be stored in a plurality of log files.
The target data of different transaction logs may be different, and in this embodiment, log processing configuration information may be set according to actual requirements.
In this embodiment, according to log processing configuration information, extracting transaction data of a first data class from a transaction log includes: determining the corresponding transaction time of the transaction log; according to the association relation between the preset transaction time and the first data category in the log processing configuration information, determining the first data category associated with the transaction time corresponding to the transaction log as a target category; transaction data of the target category is extracted from the transaction log.
Specifically, the transaction data in the transaction log may include transaction time, and after obtaining the transaction log, determining the transaction time corresponding to the transaction log. The log processing configuration information may be preset with an association relationship between a transaction time and a first data category, for example, a transaction generated at a time point a, where the first data category is a user account; the transaction generated at time point B, the first data category is the transaction type. According to the association relation between the preset transaction time and the first data category in the log processing configuration information, the first data category associated with the transaction time corresponding to the transaction log is determined and used as the target category of the transaction log. And determining transaction data corresponding to the target category from the transaction log as target data of the transaction log.
The method has the advantages that for transactions generated at different moments, the first data types of the target data can be different, and the pertinence and the flexibility of log processing are improved.
S203, generating a blank log file corresponding to the target data, and distributing a file identifier for the blank log file; the log file is used for storing transaction data in the transaction log.
For example, this step may refer to step S102, and will not be described in detail.
S204, storing the transaction data in the blank log file to obtain the log file containing the transaction data.
Illustratively, a blank log file corresponding to each target data is determined, i.e., a blank log file corresponding to each transaction log is determined. And storing the transaction data in the transaction log in a corresponding blank log file to obtain a log file containing the transaction data.
In this embodiment, the log processing configuration information includes a data class of transaction data to be stored, which is a second data class; storing the transaction data in a blank log file to obtain a log file containing the transaction data, including: determining transaction data of a second data category in the transaction log according to preset log processing configuration information; and storing the transaction data of the second data category in the blank log file to obtain the log file containing the transaction data.
Specifically, all transaction data in the transaction log may be stored in a corresponding log file, or some transaction data in the transaction log may be stored in a corresponding log file. The data category of the transaction data to be stored in the log file may be set in advance in the log processing configuration information, that is, the data category of the transaction data to be stored may be included in the log processing configuration information as the second data category. For example, there is a large amount of transaction data in the transaction log, where only the user account number, the transaction type, the transaction amount, and the transaction time need to be stored in the log file, then the four categories of the user account number, the transaction type, the transaction amount, and the transaction time are the second data category.
And determining a second data category according to the preset log processing configuration information, and acquiring transaction data of the second data category from the transaction log. And storing the transaction data of the second data category in the corresponding blank log file to obtain the log file containing the transaction data.
The beneficial effects of setting up like this are that can select a part of data to store from the transaction log, but not store all transaction data, satisfy actual log processing's demand, practice thrift storage space, be convenient for follow-up quick the looking up.
In this embodiment, the method further includes: and responding to the information configuration instruction, and receiving the first data category and the second data category input by the user to obtain log processing configuration information.
Specifically, the staff can update the log processing configuration information at any time according to the actual requirement, and can send out an information configuration instruction when the log processing configuration information needs to be changed. The information configuration instruction may include a first data category and/or a second data category to be updated, e.g., the user may input the first data category and the second data category. The information configuration instruction may further include an association relationship between the transaction time and the first data category, and the like.
And responding to the information configuration instruction, and determining a first data category and/or a second data category input by a user according to the information configuration instruction. And updating the log processing configuration information according to the first data category and/or the second data category input by the user, obtaining the latest log processing configuration information and storing the latest log processing configuration information. For example, the user may add or subtract the first data category and/or the second data category, and may replace the first data category and/or the second data category.
The beneficial effects of the arrangement are that the log processing configuration information can be dynamically modified, the actual business requirements are met, and the flexibility and the accuracy of log processing are improved.
S205, storing the file identification association corresponding to the target data and the log file containing the transaction data in a preset database, and storing the log file containing the transaction data and the corresponding file identification association in a preset backup log server; the backup log server is used for carrying out backup storage on the log file containing the transaction data.
For example, this step may refer to step S104 described above, and will not be described in detail.
The embodiment of the application provides a log processing method based on transaction data, which is used for acquiring transaction logs in a period of time, wherein each transaction log can comprise a plurality of transaction data. For each transaction log, target data is determined from a plurality of transaction data. The target data corresponds to a log file, and transaction data of a transaction log to which the target data belongs is stored in the corresponding log file to obtain an association relationship between the target data and the log file. Each log file is provided with a unique file identification, the target data and the file identifications of the corresponding log files are associated and stored in a preset database, and the log files and the corresponding file identifications are associated and stored in a preset backup log server, so that the log files can be conveniently searched from the backup log server according to the target data. The method avoids mixing a plurality of transaction logs in one file for storage, realizes targeted processing of the transaction logs according to different target data, and improves log processing precision.
Fig. 3 is a block diagram of a log processing device based on transaction data according to an embodiment of the present application, and for convenience of explanation, only a portion related to an embodiment of the present disclosure is shown. Referring to fig. 3, the apparatus includes: a data determination module 301, a file generation module 302, a data storage module 303, and a file backup module 304.
A data determining module 301, configured to obtain a transaction log, and determine target data in the transaction log; the transaction log comprises transaction data of a transaction, wherein the target data is used for representing transaction data of a preset first data type in the transaction log;
the file generating module 302 is configured to generate a blank log file corresponding to the target data, and allocate a file identifier to the blank log file; the log file is used for storing transaction data in the transaction log;
a data storage module 303, configured to store the transaction data in the blank log file, to obtain a log file containing the transaction data;
the file backup module 304 is configured to store the target data and a file identifier corresponding to the log file containing the transaction data in a preset database in an associated manner, and store the log file containing the transaction data and the corresponding file identifier in a preset backup log server in an associated manner; the backup log server is used for carrying out backup storage on the log file containing the transaction data.
Fig. 4 is a block diagram of a log processing device based on transaction data according to an embodiment of the present application, and on the basis of the embodiment shown in fig. 3, as shown in fig. 4, a data determining module 301 includes an information obtaining unit 3011 and a data extracting unit 3012.
An information acquisition unit 3011 for acquiring log processing configuration information set in advance; the log processing configuration information comprises data types of transaction data to be extracted, wherein the data types are first data types;
and a data extraction unit 3012, configured to extract, from the transaction log, the transaction data of the first data class as the target data according to the log processing configuration information.
In one example, the data extraction unit 3012 is specifically configured to:
determining the transaction time corresponding to the transaction log;
determining a first data category associated with the transaction time corresponding to the transaction log as a target category according to the association relation between the transaction time preset in the log processing configuration information and the first data category;
and extracting the transaction data of the target category from the transaction log.
In one example, the log processing configuration information includes a data category of transaction data to be stored, which is a second data category;
The data storage module 303 is specifically configured to:
determining transaction data of a second data category in the transaction log according to preset log processing configuration information;
and storing the transaction data of the second data category in the blank log file to obtain a log file containing the transaction data.
In one example, the apparatus further comprises:
and the information configuration module is used for responding to the information configuration instruction, receiving the first data category and the second data category input by the user and obtaining the log processing configuration information.
In one example, the transaction log is at least two;
the file generating module 302 is specifically configured to:
generating a blank log file corresponding to the target data, including:
determining target data in the same first data category as the same category data;
performing numerical comparison on the same category data to determine the same numerical data in the same category data;
and generating a blank log file corresponding to the same numerical value data.
In one example, the apparatus further comprises:
the data judging module is used for generating a blank log file corresponding to each target data if the same category data does not exist in the target data;
And if the same numerical value data does not exist in the same type of data, generating a blank log file corresponding to each target data in the same type of data.
In one example, the data determination module 301 further includes:
the log acquisition unit is used for acquiring the transaction log generated in the data acquisition period from a preset transaction application according to the preset data acquisition period.
In one example, a time stamp is marked in the log file, and the time stamp is used for representing the generation time of the log file;
the apparatus further comprises:
the time acquisition module is used for acquiring the current time after the log file containing the transaction data and the corresponding file identifier are stored in a preset backup log server in a correlated manner, and determining the time difference between the current time and the generation time of the log file according to the time stamp of the log file;
and the backup updating module is used for deleting the log file from the backup log server if the time difference is equal to or larger than a preset time difference threshold value.
In one example, the apparatus further comprises:
the log inquiry module is used for responding to a log file inquiry instruction and determining data to be inquired from the log file inquiry instruction; the data to be queried is used for representing target data corresponding to the log file to be queried;
Searching a file identifier corresponding to the data to be queried from the database, wherein the file identifier is a target identifier;
and searching the log file corresponding to the target identifier from the backup log server, wherein the log file is the log file to be queried.
Fig. 5 is a block diagram of an electronic device according to an embodiment of the present application, where, as shown in fig. 5, the electronic device includes: a memory 51, a processor 52; memory 51, a memory for storing instructions executable by processor 52.
Wherein the processor 52 is configured to perform the method as provided by the above-described embodiments.
The electronic device further comprises a receiver 53 and a transmitter 54. The receiver 53 is configured to receive instructions and data transmitted from other devices, and the transmitter 54 is configured to transmit instructions and data to external devices.
Fig. 6 is a block diagram of an electronic device, which may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, an exercise device, a personal digital assistant, etc., in accordance with an exemplary embodiment.
The device 600 may include one or more of the following components: a processing component 602, a memory 604, a power component 606, a multimedia component 608, an audio component 610, an input/output (I/O) interface 612, a sensor component 614, and a communication component 616.
The processing component 602 generally controls overall operation of the device 600, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing component 602 may include one or more processors 620 to execute instructions to perform all or part of the steps of the methods described above. Further, the processing component 602 can include one or more modules that facilitate interaction between the processing component 602 and other components. For example, the processing component 602 may include a multimedia module to facilitate interaction between the multimedia component 608 and the processing component 602.
The memory 604 is configured to store various types of data to support operations at the device 600. Examples of such data include instructions for any application or method operating on device 600, contact data, phonebook data, messages, pictures, videos, and the like. The memory 604 may be implemented by any type or combination of volatile or nonvolatile memory devices such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disk.
The power supply component 606 provides power to the various components of the device 600. The power components 606 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 600.
The multimedia component 608 includes a screen between the device 600 and the user that provides an output interface. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensor may sense not only the boundary of a touch or slide action, but also the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 608 includes a front camera and/or a rear camera. The front-facing camera and/or the rear-facing camera may receive external multimedia data when the device 600 is in an operational mode, such as a shooting mode or a video mode. Each front camera and rear camera may be a fixed optical lens system or have focal length and optical zoom capabilities.
The audio component 610 is configured to output and/or input audio signals. For example, the audio component 610 includes a Microphone (MIC) configured to receive external audio signals when the device 600 is in an operational mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals may be further stored in the memory 604 or transmitted via the communication component 616. In some embodiments, audio component 610 further includes a speaker for outputting audio signals.
The I/O interface 612 provides an interface between the processing component 602 and peripheral interface modules, which may be a keyboard, click wheel, buttons, etc. These buttons may include, but are not limited to: homepage button, volume button, start button, and lock button.
The sensor assembly 614 includes one or more sensors for providing status assessment of various aspects of the device 600. For example, the sensor assembly 614 may detect the on/off state of the device 600, the relative positioning of the components, such as the display and keypad of the device 600, the sensor assembly 614 may also detect a change in position of the device 600 or a component of the device 600, the presence or absence of user contact with the device 600, the orientation or acceleration/deceleration of the device 600, and a change in temperature of the device 600. The sensor assembly 614 may include a proximity sensor configured to detect the presence of nearby objects in the absence of any physical contact. The sensor assembly 614 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 614 may also include an acceleration sensor, a gyroscopic sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.
The communication component 616 is configured to facilitate communication between the device 600 and other devices, either wired or wireless. The device 600 may access a wireless network based on a communication standard, such as WiFi,2G or 3G, or a combination thereof. In one exemplary embodiment, the communication component 616 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication component 616 further includes a Near Field Communication (NFC) module to facilitate short range communications. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, ultra Wideband (UWB) technology, bluetooth (BT) technology, and other technologies.
In an exemplary embodiment, the apparatus 600 may be implemented by one or more Application Specific Integrated Circuits (ASICs), digital Signal Processors (DSPs), digital Signal Processing Devices (DSPDs), programmable Logic Devices (PLDs), field Programmable Gate Arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic elements for executing the methods described above.
In an exemplary embodiment, a non-transitory computer readable storage medium is also provided, such as memory 604, including instructions executable by processor 620 of device 600 to perform the above-described method. For example, the non-transitory computer readable storage medium may be ROM, random Access Memory (RAM), CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.
A non-transitory computer readable storage medium, which when executed by a processor of a terminal device, causes the terminal device to perform the above-described log processing method of the terminal device based on transaction data.
Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. This application is intended to cover any variations, uses, or adaptations of the application following, in general, the principles of the application and including such departures from the present disclosure as come within known or customary practice within the art to which the application pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the application being indicated by the following claims.
It is to be understood that the application is not limited to the precise arrangements and instrumentalities shown in the drawings, which have been described above, and that various modifications and changes may be effected without departing from the scope thereof. The scope of the application is limited only by the appended claims.
Claims (13)
1. A log processing method based on transaction data, comprising:
acquiring a transaction log and determining target data in the transaction log; the transaction log comprises transaction data of a transaction, wherein the target data is used for representing transaction data of a preset first data type in the transaction log;
Generating a blank log file corresponding to the target data, and distributing a file identifier for the blank log file; the log file is used for storing transaction data in the transaction log;
storing the transaction data in the blank log file to obtain a log file containing the transaction data;
storing the target data and the file identification association corresponding to the log file containing the transaction data in a preset database, and storing the log file containing the transaction data and the corresponding file identification association in a preset backup log server; the backup log server is used for carrying out backup storage on the log file containing the transaction data.
2. The method of claim 1, wherein determining target data in the transaction log comprises:
acquiring preset log processing configuration information; the log processing configuration information comprises data types of transaction data to be extracted, wherein the data types are first data types;
and extracting the transaction data of the first data category from the transaction log according to the log processing configuration information, and taking the transaction data as the target data.
3. The method of claim 2, wherein extracting transaction data of the first data category from the transaction log based on the log processing configuration information comprises:
determining the transaction time corresponding to the transaction log;
determining a first data category associated with the transaction time corresponding to the transaction log as a target category according to the association relation between the transaction time preset in the log processing configuration information and the first data category;
and extracting the transaction data of the target category from the transaction log.
4. The method of claim 2, wherein the log processing configuration information includes a data category of transaction data to be stored as a second data category;
storing the transaction data in the blank log file to obtain a log file containing the transaction data, wherein the method comprises the following steps of:
determining transaction data of a second data category in the transaction log according to preset log processing configuration information;
and storing the transaction data of the second data category in the blank log file to obtain a log file containing the transaction data.
5. The method according to claim 4, wherein the method further comprises:
and responding to the information configuration instruction, and receiving a first data category and a second data category input by a user to obtain the log processing configuration information.
6. The method of claim 1, wherein the transaction log is at least two;
generating a blank log file corresponding to the target data, including:
determining target data in the same first data category as the same category data;
performing numerical comparison on the same category data to determine the same numerical data in the same category data;
and generating a blank log file corresponding to the same numerical value data.
7. The method of claim 6, wherein the method further comprises:
if the same type of data does not exist in the target data, generating a blank log file corresponding to each target data;
and if the same numerical value data does not exist in the same type of data, generating a blank log file corresponding to each target data in the same type of data.
8. The method of claim 1, wherein obtaining a transaction log comprises:
And acquiring the transaction log generated in the data acquisition period from a preset transaction application according to the preset data acquisition period.
9. The method according to claim 1, wherein the log file is marked with a time stamp, and the time stamp is used for representing the generation time of the log file;
after the log file containing the transaction data is associated with the corresponding file identifier and stored in a preset backup log server, the method further comprises the steps of:
acquiring current time, and determining a time difference between the current time and generation time of the log file according to the time stamp of the log file;
and if the time difference is equal to or larger than a preset time difference threshold, deleting the log file from the backup log server.
10. The method according to any one of claims 1-9, further comprising:
responding to a log file inquiring instruction, and determining data to be inquired from the log file inquiring instruction; the data to be queried is used for representing target data corresponding to the log file to be queried;
searching a file identifier corresponding to the data to be queried from the database, wherein the file identifier is a target identifier;
And searching the log file corresponding to the target identifier from the backup log server, wherein the log file is the log file to be queried.
11. A log processing device based on transaction data, comprising:
the data determining module is used for acquiring a transaction log and determining target data in the transaction log; the transaction log comprises transaction data of a transaction, wherein the target data is used for representing transaction data of a preset first data type in the transaction log;
the file generation module is used for generating a blank log file corresponding to the target data and distributing a file identifier for the blank log file; the log file is used for storing transaction data in the transaction log;
the data storage module is used for storing the transaction data in the blank log file to obtain a log file containing the transaction data;
the file backup module is used for storing the association of the target data and the file identifier corresponding to the log file containing the transaction data in a preset database, and storing the association of the log file containing the transaction data and the corresponding file identifier in a preset backup log server; the backup log server is used for carrying out backup storage on the log file containing the transaction data.
12. An electronic device, comprising: a processor, and a memory communicatively coupled to the processor;
the memory stores computer-executable instructions;
the processor executes computer-executable instructions stored in the memory to implement the transaction data based log processing method of any one of claims 1 to 10.
13. A computer readable storage medium having stored therein computer executable instructions which when executed by a processor are for implementing a transaction data based log processing method according to any of claims 1 to 10.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310702171.8A CN116737682A (en) | 2023-06-13 | 2023-06-13 | Log processing method, device, equipment and storage medium based on transaction data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310702171.8A CN116737682A (en) | 2023-06-13 | 2023-06-13 | Log processing method, device, equipment and storage medium based on transaction data |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116737682A true CN116737682A (en) | 2023-09-12 |
Family
ID=87909211
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310702171.8A Pending CN116737682A (en) | 2023-06-13 | 2023-06-13 | Log processing method, device, equipment and storage medium based on transaction data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116737682A (en) |
-
2023
- 2023-06-13 CN CN202310702171.8A patent/CN116737682A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106445189B (en) | Candidate word display method and device | |
| CN112131466A (en) | Group display method, device, system and storage medium | |
| CN112416219A (en) | Target object processing method, device and system, electronic equipment and storage medium | |
| CN110442844B (en) | Data processing method, device, electronic equipment and storage medium | |
| CN111046040A (en) | Method and device for determining index, electronic equipment and storage medium | |
| CN112134987B (en) | Information processing method and device and electronic equipment | |
| CN110908814A (en) | Message processing method and device, electronic equipment and storage medium | |
| CN106454781B (en) | Method and device for identifying source of communication message | |
| CN116737682A (en) | Log processing method, device, equipment and storage medium based on transaction data | |
| CN107515853B (en) | Cell word bank pushing method and device | |
| CN110489155B (en) | Data file management method, device, electronic equipment and medium | |
| CN114090738A (en) | Method, device and equipment for determining scene data information and storage medium | |
| CN114036917A (en) | Report generation method and device, computer equipment and storage medium | |
| CN112800084A (en) | Data processing method and device | |
| CN110209775B (en) | Text processing method and device | |
| CN115225716B (en) | Data processing method and device and electronic equipment | |
| CN114422811B (en) | Method, device, equipment and storage medium for task allocation and task template generation | |
| CN115237925A (en) | Data processing method, device, equipment, storage medium and product | |
| CN117149704A (en) | File management method, device, equipment and storage medium based on performance test | |
| CN117194738A (en) | Method, device and equipment for inquiring receipt of transaction | |
| CN107544965B (en) | Method for optimizing size of static lexicon and electronic equipment | |
| CN117112567A (en) | Data storage method and device and electronic equipment | |
| CN116910087A (en) | Method and device for determining service circulation information, electronic equipment and storage medium | |
| CN116643673A (en) | Information collection method, device and equipment | |
| CN115525635A (en) | Session data processing method, device, equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |