CN116483795A - Log data processing method and device, electronic equipment and storage medium - Google Patents
Log data processing method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN116483795A CN116483795A CN202310449714.XA CN202310449714A CN116483795A CN 116483795 A CN116483795 A CN 116483795A CN 202310449714 A CN202310449714 A CN 202310449714A CN 116483795 A CN116483795 A CN 116483795A
- Authority
- CN
- China
- Prior art keywords
- log
- task
- file
- exported
- information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title abstract description 17
- 238000000034 method Methods 0.000 claims abstract description 77
- 230000005540 biological transmission Effects 0.000 claims abstract description 33
- 238000012545 processing Methods 0.000 claims abstract description 31
- 238000004458 analytical method Methods 0.000 claims abstract description 16
- 238000012216 screening Methods 0.000 claims description 39
- 230000008569 process Effects 0.000 claims description 24
- 238000004590 computer program Methods 0.000 claims description 14
- 238000012546 transfer Methods 0.000 claims description 8
- 238000006243 chemical reaction Methods 0.000 claims description 7
- 238000013500 data storage Methods 0.000 abstract description 6
- 230000006870 function Effects 0.000 description 13
- 238000010586 diagram Methods 0.000 description 11
- 238000004891 communication Methods 0.000 description 6
- 238000007726 management method Methods 0.000 description 6
- 238000007405 data analysis Methods 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 3
- 238000013523 data management Methods 0.000 description 2
- 238000007418 data mining Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000012512 characterization method Methods 0.000 description 1
- 238000013144 data compression Methods 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000005055 memory storage Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/602—Providing cryptographic facilities or services
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Computational Linguistics (AREA)
- Health & Medical Sciences (AREA)
- Bioethics (AREA)
- General Health & Medical Sciences (AREA)
- Computer Hardware Design (AREA)
- Computer Security & Cryptography (AREA)
- Software Systems (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention provides a log data processing method, a device, electronic equipment and a storage medium, and relates to the technical field of big data processing, wherein the method comprises the following steps: receiving log export task information sent by a log storage system, and determining a corresponding original log file according to a task query language and a task identifier in the log export task information; and carrying out file analysis and encryption on the original log file, and exporting the encrypted log file to a corresponding file transmission platform. The invention improves the data storage capacity of the 5G message, optimizes the traditional database processing method, improves the database processing efficiency of enterprises, improves the accessibility and availability of data, facilitates the user to inquire and manage the data, and ensures that the enterprises can better analyze and mine the data. In addition, by encrypting the log data, the security of the log data can be improved, and sensitive information leakage can be prevented.
Description
Technical Field
The invention relates to the technical field of big data processing, and also can be used in the financial field, in particular to a log data processing method, a log data processing device, electronic equipment and a storage medium.
Background
The 5G message is a rich media message, is established based on GSMA international communication standard (RCS), can provide direct, convenient, high-capacity and content-rich video rich media service, and has wide application prospect.
However, with the development of 5G message services, a large amount of history log data will be gradually accumulated in the database, but due to the deficiency of the storage and retrieval mechanism of the current 5G message database, the data is difficult to be effectively managed and utilized at this time, and valuable reference and support cannot be provided for subsequent services. In addition, because the 5G message has the characteristics of large data volume, high real-time requirement and the like, the traditional data storage and processing mode cannot meet the requirement. Accordingly, new techniques and methods are needed to optimize the storage and management of 5G message databases.
Disclosure of Invention
In view of the above, the present invention provides a log data processing method, apparatus, electronic device and readable storage medium, so as to solve at least the problems in the related art.
According to a first aspect of an embodiment of the present invention, there is provided a log data processing method, the method including:
receiving log export task information sent by a log storage system, wherein the log export task information is generated by a log to be exported, which is selected by a user through the log storage system;
Determining a corresponding original log file according to a task query language and a task identifier in the log derived task information;
carrying out file analysis and encryption on the original log file to obtain an encrypted log file;
and exporting the encrypted log file to a corresponding file transmission platform.
In combination with any embodiment of the present invention, the determining the corresponding original log file according to the task query language and the task identifier in the log derived task information includes:
deriving a task query language and a task identifier in the task information according to the log, and generating a file name of the binary format log;
and downloading the corresponding binary format log from the log storage system according to the file name of the binary format log.
In combination with any one of the embodiments of the present invention, the performing file parsing and encrypting on the original log file to obtain an encrypted log file includes:
and carrying out file analysis and encryption on the binary format log to obtain an encrypted comma separated value format log.
In combination with any one of the embodiments of the present invention, the process of selecting the log to be exported includes:
Screening logs to be exported in the task list according to start-stop time input by a user;
and receiving a log to be exported, which is selected by a user based on the screening result.
In combination with any one of the embodiments of the present invention, the process of selecting the log to be exported includes:
screening logs to be exported in the task list according to the user permission of the user;
and receiving a log to be exported, which is selected by a user based on the screening result.
In combination with any embodiment of the present invention, after receiving the log to be exported selected by the user based on the screening result, the method further includes:
setting a distributed lock for the selected log to be exported;
in the process of screening the logs to be exported in the task table, the method further comprises:
and screening the task to be processed which is not provided with the distributed lock in the task table.
In combination with any one of the embodiments of the present invention, after exporting the encrypted log file to a corresponding file transfer platform, the method further includes:
responding to the success of the export to the corresponding file transmission platform, and updating task success information to a server;
responding to failure of exporting to a corresponding file transmission platform, and updating task failure information to a server;
The task success information comprises at least one of a task state, a current file name and a server returned file name;
the task failure information comprises at least one of task state, current file name and error information.
According to a second aspect of an embodiment of the present invention, there is provided a log data processing apparatus, the apparatus including:
an information receiving module for: receiving log export task information sent by a log storage system, wherein the log export task information is generated by a log to be exported, which is selected by a user through the log storage system;
a file determining module, configured to: determining a corresponding original log file according to a task query language and a task identifier in the log derived task information;
a file conversion module for: carrying out file analysis and encryption on the original log file to obtain an encrypted log file;
a file export module, configured to: and exporting the encrypted log file to a corresponding file transmission platform.
In combination with any one of the embodiments of the present invention, when determining a corresponding original log file according to a task query language and a task identifier in the log-derived task information, the file determining module is specifically configured to:
Deriving a task query language and a task identifier in the task information according to the log, and generating a file name of the binary format log;
and downloading the corresponding binary format log from the log storage system according to the file name of the binary format log.
In combination with any one of the embodiments of the present invention, when the file conversion module performs file analysis and encryption on the original log file to obtain an encrypted log file, the file conversion module is specifically configured to:
and carrying out file analysis and encryption on the binary format log to obtain an encrypted comma separated value format log.
In combination with any embodiment of the present invention, in the process of selecting the log to be exported, the method is specifically used for:
screening logs to be exported in the task list according to start-stop time input by a user;
and receiving a log to be exported, which is selected by a user based on the screening result.
In combination with any embodiment of the present invention, in the process of selecting the log to be exported, the method is specifically used for:
screening logs to be exported in the task list according to the user permission of the user;
and receiving a log to be exported, which is selected by a user based on the screening result.
In combination with any embodiment of the present invention, after receiving the log to be exported selected by the user based on the screening result, the apparatus further includes a distributed lock setting module, configured to:
Setting a distributed lock for the selected log to be exported;
in the process of screening the logs to be exported in the task table, the method further comprises:
and screening the task to be processed which is not provided with the distributed lock in the task table.
In combination with any one of the embodiments of the present invention, after exporting the encrypted log file to a corresponding file transmission platform, the apparatus further includes an information update module configured to:
responding to the success of the export to the corresponding file transmission platform, and updating task success information to a server;
responding to failure of exporting to a corresponding file transmission platform, and updating task failure information to a server;
the task success information comprises at least one of a task state, a current file name and a server returned file name;
the task failure information comprises at least one of task state, current file name and error information.
According to a third aspect of an embodiment of the present invention, there is provided an electronic apparatus including:
a memory for storing the processor-executable instructions;
a processor configured to execute executable instructions in the memory to implement the steps of the method of any of the embodiments of the first aspect described above.
According to a fourth aspect of embodiments of the present invention, there is provided a computer readable storage medium having stored thereon a computer program which when executed by a processor performs the steps of the method according to any of the embodiments of the first aspect described above.
According to the scheme, the big data lake server receives the log export task information sent by the log storage system, analyzes and encrypts the original log file obtained by determination, and then forwards the original log file to the corresponding file transmission platform. The method has the advantages of improving the data storage capacity of the 5G message, optimizing the traditional database processing method, improving the database processing efficiency of enterprises, improving the accessibility and availability of data, facilitating the user to query and manage the data, and enabling the enterprises to better analyze and mine the data. In addition, by encrypting the log data, the security of the log data can be improved, and sensitive information leakage can be prevented.
Drawings
In order to more clearly illustrate the embodiments of the invention or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of a log data processing method according to an embodiment of the present invention;
FIG. 2 is a flow chart of another log data processing method according to an embodiment of the present invention;
FIG. 3 illustrates a task table identification reference diagram of an embodiment of the present invention;
FIG. 4 is a schematic diagram of a log data processing apparatus according to an embodiment of the present invention;
fig. 5 shows a schematic diagram of a computer device suitable for use in implementing embodiments of the invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The terms "first," "second," … …, and the like, as used herein, do not denote a particular order or sequence, nor are they intended to be limiting of the invention, but rather are merely used to distinguish one element or operation from another in the same technical terms.
As used herein, the terms "comprising," "including," "having," "containing," and the like are intended to be inclusive and mean an inclusion, but not limited to.
As used herein, "and/or" includes any or all combinations of such things.
It should be noted that, in the technical scheme of the invention, the acquisition, storage, use, processing and the like of the data all conform to the relevant regulations of laws and regulations.
It should be noted that the log data processing method, apparatus, electronic device and storage medium disclosed in the present application may be used in a distributed field, and may also be used in any field other than the distributed field, and the application fields of the log data processing method, apparatus, electronic device and storage medium disclosed in the present application are not limited.
The scheme of the invention is applied to the big data lake server, and the big data lake server characterizes a digital platform capable of providing a series of big data services such as data management, data analysis, data mining and the like, and is used for helping enterprises or individual users manage and utilize mass data.
By way of example, the big data lake server may include a service platform widely used in the fields of data analysis and big data processing, such as BDSP (Big Data Service Platform, enterprise class big data service platform), ADLS (Microsoft Azure Data Lake Store, microsoft data lake storage), GCS (Google Cloud Storage, gu Geyun storage), COS (IBM Cloud Object Storage ), and the like.
FIG. 1 is a flowchart of a log data processing method according to an exemplary embodiment of the present invention, the method specifically includes the following steps:
s101, receiving log export task information sent by a log storage system, wherein the log export task information is generated by a log to be exported, which is selected by a user through the log storage system.
Wherein the log storage system characterizes a software system that can be used to collect, store, and manage log information, which can include system logs, application logs, web logs, and the like. By way of example, the log storage system may include a bank teller operating system for storing log information of teller's login, customer's account information, transaction amount, transaction status, etc., for subsequent querying and auditing.
The task information characterizes the relevant information of the logs needing to be exported, including information such as task query language and task identifier, wherein the task query language is used for describing the filtering condition of the logs needing to be screened by the user, such as designating the time range of the log data needing to be exported, specific service scenes and the like, and the task identifier characterizes the unique identification of the selected logs, thereby facilitating the subsequent processing and management.
In step S101, the big data lake server may receive log export task information sent from the log storage system, where the task information is generated based on a log to be exported selected by a user in the log storage system. Illustratively, the user may select the log to be exported by performing a corresponding operation in the log storage system. And then, the log storage system can generate a corresponding export task through the selected log information and receive the task information through the big data lake server.
S102, determining a corresponding original log file according to the task query language and the task identifier in the log export task information.
The original log file characterizes a file recorded with key information such as system running conditions, operation records, abnormal information and the like, and can be stored in various formats including a text format, a binary (bin) format and the like. The binary format has higher data compression rate and faster reading speed compared with the text format, and is suitable for scenes needing to process a large amount of data. Therefore, the log file corresponding to the binary format can be stored in the log storage system as the original log file and is acquired by the big data lake server.
In step S102, the big data lake server determines a corresponding original log file according to the task query language and the task identifier in the task information. Illustratively, the task query language may be parsed into SQL statements and queried through a database connection with the log storage system. The query results may contain an original log file that matches the task identifier and relevant information (e.g., filename, path, etc.) of the original log file is recorded in server memory for subsequent processing.
S103, carrying out file analysis and encryption on the original log file to obtain an encrypted log file.
The big data lake server can analyze the selected original log file and encrypt the log data, wherein the encryption process can adopt a symmetric encryption algorithm or an asymmetric encryption algorithm to ensure the security and the integrity of the data.
S104, exporting the encrypted log file to a corresponding file transmission platform.
The big data lake server can export the encrypted log file to a corresponding file transmission platform so as to complete the complete export process of the log data. The file transfer platform may be established based on FTP (file transfer protocol), SFTP (secure file transfer protocol), SCP (secure copy protocol) or other protocols supporting file transfer.
The GTP (Globally Unique Temporary Identifier) file server is used as a high-availability and high-performance file transmission server, is generally used for tasks such as internal file transmission, file storage and backup of enterprises, and can realize cross-platform and high-efficiency file transmission. In the method of the invention, the GTP file server can be used as the corresponding file transmission platform so as to realize reliable and safe file transmission.
According to the scheme, the big data lake server receives the log export task information sent by the log storage system, analyzes and encrypts the original log file obtained by determination, and then forwards the original log file to the corresponding file transmission platform. The data storage capacity of the 5G message is improved, the traditional database processing method is optimized, the database processing efficiency of enterprises is improved, the accessibility and the availability of data are improved, the user can conveniently inquire and manage the data, and the enterprises can better perform data analysis and mining. In addition, by encrypting the log data, the security of the log data can be improved, and sensitive information leakage can be prevented.
In an optional embodiment, the determining the corresponding original log file according to the task query language and the task identifier in the log derived task information includes:
Deriving a task query language and a task identifier in the task information according to the log, and generating a file name of the binary format log;
and downloading the corresponding binary format log from the log storage system according to the file name of the binary format log.
For example, first, the big data lake server may derive a task query language and a task identifier in task information from the log, and generate a file name of the corresponding binary format log. The file name is generated based on the query conditions and identifiers contained in the task information, and consistency of the downloaded log file and the user requirements is ensured.
Then, the big data lake server can use the file name to download the corresponding binary format log from the log storage system and store the binary format log on a local disk for subsequent processing.
According to the embodiment, the big data lake server can quickly acquire the needed log information through the query condition and the identifier in the log export task information, so that efficient query and positioning of massive log data are realized, and the data storage and management mode is optimized.
In an optional embodiment, the performing file parsing and encrypting on the original log file to obtain an encrypted log file includes:
and carrying out file analysis and encryption on the binary format log to obtain an encrypted comma separated value format log.
For example, first, the big data lake server may parse the binary log file according to a predefined log format specification, extract data information therein, and convert the data information according to a defined Comma Separated Value (CSV) format. Then, an encryption algorithm can be used for encrypting the log data so as to ensure the security of the data in the transmission and storage processes. And finally generating an encrypted comma separated value format log file, and outputting the encrypted comma separated value format log file as a final log file.
According to the embodiment, the original binary format log is converted into the comma separated value format log, and encryption processing is carried out, so that the safety of data is improved. Meanwhile, comma separation value format logs are more convenient for data analysis, searching and storage, and the readability and operability of data are improved.
In an alternative embodiment, in the process of selecting the log to be exported, the method includes:
Screening logs to be exported in the task list according to start-stop time input by a user;
and receiving a log to be exported, which is selected by a user based on the screening result.
For example, the log storage system may query a task table according to a time range input by a user and filter logs to be exported therein. And then, the screened result can be returned to the user, so that the user can select the log which the user needs to export. Therefore, the user can quickly screen and select the logs to be exported according to the own requirements, the data volume to be processed by the user is reduced, and the processing efficiency is improved. Meanwhile, the burden of the system is reduced, and a large amount of useless log data is prevented from being exported and processed.
In an alternative embodiment, in the process of selecting the log to be exported, the method includes:
screening logs to be exported in the task list according to the user permission of the user;
and receiving a log to be exported, which is selected by a user based on the screening result.
For example, the log storage system may filter logs to be exported from the log storage system according to the rights of the user. And then, the screened result can be returned to the user, so that the user can select the log which the user needs to export. Only users with corresponding rights can access and export the corresponding logs, so that the data security of the log storage system is improved.
Taking a log storage system of a bank teller as an example, when a common teller needs to check the files in the limited area, the system can screen the task list according to the authority of the common teller, and only the log conforming to the authority of the common teller is returned, so that the common teller is ensured to check the files conforming to the authority of the common teller, and the confidentiality of sensitive data is ensured. The super teller has the right to view all files, and can view any file in the system without any obstacle.
By applying the method to file access authority management of a bank teller, the access authority of the teller to the file can be controlled more finely, and therefore the security of sensitive data in the bank is guaranteed.
In an optional embodiment, after receiving the log to be exported selected by the user based on the screening result, the method further includes:
setting a distributed lock for the selected log to be exported;
in the process of screening the logs to be exported in the task table, the method further comprises:
and screening the task to be processed which is not provided with the distributed lock in the task table.
For example, after the log storage system receives the log to be exported selected by the user based on the screening result, a distributed lock may be set on the selected log to be exported, so as to ensure that only one user may conduct export operation to avoid conflicts and data errors caused by that a plurality of users operate the same log file at the same time.
In addition, in the process of screening the logs to be exported in the task table, the task to be processed, which is not provided with the distributed lock, can be screened, that is, the log storage system can firstly judge whether the task is locked by other users, and if the task is locked, the task cannot be selected for exporting. If the task is not locked, it may be selected for export operations. So as to effectively avoid conflict and data error caused by that a plurality of users conduct export operation on the same task at the same time.
In an alternative embodiment, after exporting the encrypted log file to the corresponding file transfer platform, the method further comprises:
responding to the success of the export to the corresponding file transmission platform, and updating task success information to a server;
responding to failure of exporting to a corresponding file transmission platform, and updating task failure information to a server;
the task success information comprises at least one of a task state, a current file name and a server returned file name;
the task failure information comprises at least one of task state, current file name and error information.
For example, when the encrypted log file is successfully exported to the corresponding file transfer platform, the big data lake server may update the task successfully. The task success information comprises at least one of a task state, a current file name and a server returned file name, and is used for identifying relevant information of the task, so that follow-up tracking and management are facilitated.
If the encrypted log file fails to be exported to the corresponding file transmission platform, the big data lake server can update task failure information. The task failure information includes at least one of a task status, a current file name, and error information for identifying a specific cause of the export failure.
In the above embodiment, after the task success or failure information is updated to the server, statistics and analysis may be performed based on the information, so as to better understand the processing state of the log data. For example, the number of task successes and failures may be counted and optimized based on the data, maintaining subsequent log data processing flows.
In the following, taking a log data processing scenario of a bank teller as an example, the invention provides a log data processing method, as shown in fig. 2, which specifically includes the following steps:
s201, inquiring whether a log to be exported exists in the task table.
Illustratively, first, the journal storage system may determine whether there are any pending TASKs by querying the TASK table eiop_omg_download_task.
FIG. 3 is a task table identification reference diagram illustrating one example embodiment of the present invention:
FuNCID is a function ID, indicating that the task belongs to a teller export or a function group export.
TASKTYPE is the export type, meaning that the task belongs to a top page list or log export.
TASKSTATUS is a task state and represents the state of the task at different stages. Wherein 1 indicates to be executed, 2 indicates that in execution, 3 indicates downloadable, 4 indicates that execution has failed, 5 indicates that execution has completed, 6 indicates that it is sent to the BDSP server but not returned, and 7 indicates that download has failed.
Wherein, the fields of the log may include: information such as teller number, name, teller type, location area, affiliated institution, personnel affiliated institution, first registration time, affiliated function group, log update time, personnel status, submitting teller, submitting time, auditing teller, auditing time, operation type and the like.
In step S201, the conditions for querying the task in the data lake are: task=1, function=0001, task type=2, where task=1 indicates that the task is to be processed, function=0001 indicates that the task belongs to the teller log export function, and task type=2 indicates that the task type is teller log export.
S202, receiving logs to be exported selected by a user based on screening results, and setting distributed locks on the selected logs to be exported.
For example, after receiving the log to be exported selected by the user based on the screening result, a distributed lock may be set on the selected log to be exported, so as to ensure that only one user may conduct export operation, so as to avoid conflicts and data errors caused by that a plurality of users operate the same log file at the same time.
S203, a task query language and a task identifier of the log to be exported are sent.
First, it may be determined whether the task state is "to be executed" (taskstatus=1). If yes, the task state is updated to be 'in execution' (task status=2), and SQL sentences are assembled according to the task parameters to inquire the teller log data meeting the conditions. And then splicing the event list number SERIALNO and the 'omg _TellerInfoManager' into a JOBID, and sending the JOBID to the BDSP server as a unique identification. After successful transmission, the update task status is "send BDSP, but not return" (taskstatus=6), and the JOBID is recorded. If the transmission fails, the task state is updated to "execution failure" (taskstatus=4), and the failure cause is recorded. If the task state is not "to be executed," no operation is performed.
S204, generating file names of binary format logs according to the task query language and the task identifier.
S205, downloading the corresponding binary format log according to the file name of the binary format log.
The purpose of the two steps is to obtain the file name of the generated binary format log based on the transmitted JOBID, and then use the name to download and parse log files containing relevant field data in the teller information derived log. Illustratively, the binary format log name generated by the job de-splice may be named "F-eiop#job#1. Bin". If the download and parsing of the binary format log is successful, then the next step may continue. If so, the task state is updated to "execution failure" (taskstatus=4), and the failure cause is recorded.
S206, carrying out file analysis and encryption on the binary format log to obtain an encrypted comma separated value format log.
The binary format log may be downloaded and parsed according to the file name and task state (taskstatus=6), and finally an encrypted comma separated value format log may be generated. Illustratively, the comma separated value format log has a file name format of String filename= "F-omg_event ticket number_timestamp_type (bdsp). Csv).
Wherein, "F-OMG_": representing file types, the F-OMG token is a teller operation management file, and different types of files have different prefixes.
"event sheet number": representing event list number corresponding to current operation
"timestamp": the time stamp of the current file when being generated is characterized, and the time stamp is accurate to millisecond, so that the uniqueness of the file name can be ensured.
"type (bdsp)": the file format and source are characterized, wherein the file format is CSV, and the source is BDSP.
And S207, deriving the encrypted comma separated value format log.
Illustratively, the generated comma separated value format log is uploaded to a GTP file server, and corresponding success/failure related fields of the taskStatus, fileName, error msg, podId are updated to the server for subsequent task tracking and management.
The task status is a task status, fileName is a file name, error msg is error information, and podId is a file ID returned after uploading to the GTP file server. The information characterization is an important parameter and state of the task, and other modules can judge the execution condition and the processing result of the task according to the parameters by updating the important parameter and state of the task to the server.
In the above embodiment, aiming at the problems that the storage amount of log data in a bank teller system is large and quick search is required, a log data processing method based on a BDSP server is provided, log export task information sent by a log storage system is received through the BDSP server, and the obtained original log file is analyzed, encrypted and forwarded to a corresponding GTP file server. The method has the advantages that the 5G message data storage capacity is improved, the traditional database processing method is optimized, the database processing efficiency of a bank is improved, the accessibility and the availability of data are improved, the teller can conveniently inquire and manage the data, and the bank can better analyze and mine the data. In addition, by encrypting the log data, the security of the log data can be improved, and sensitive information leakage can be prevented.
For the foregoing method embodiments, for simplicity of explanation, the methodologies are shown as a series of acts, but one of ordinary skill in the art will appreciate that the present invention is not limited by the order of acts, as some steps may, in accordance with the present invention, occur in other orders or concurrently.
Further, those skilled in the art will also appreciate that the embodiments described in the specification are alternative embodiments, and that the acts and modules referred to are not necessarily required for the present invention.
Corresponding to the embodiment of the application function implementation method, the invention also provides an embodiment of the application function implementation device and a corresponding terminal.
As shown in fig. 4, the present invention further provides a log data processing apparatus 400, which includes:
an information receiving module 401, configured to: receiving log export task information sent by a log storage system, wherein the log export task information is generated by a log to be exported, which is selected by a user through the log storage system;
a file determination module 402, configured to: determining a corresponding original log file according to a task query language and a task identifier in the log derived task information;
A file conversion module 403, configured to: carrying out file analysis and encryption on the original log file to obtain an encrypted log file;
a file export module 404, configured to: and exporting the encrypted log file to a corresponding file transmission platform.
In combination with any one of the embodiments of the present invention, when determining a corresponding original log file according to a task query language and a task identifier in the log-derived task information, the file determining module is specifically configured to:
deriving a task query language and a task identifier in the task information according to the log, and generating a file name of the binary format log;
and downloading the corresponding binary format log from the log storage system according to the file name of the binary format log.
In combination with any one of the embodiments of the present invention, when the file conversion module performs file analysis and encryption on the original log file to obtain an encrypted log file, the file conversion module is specifically configured to:
and carrying out file analysis and encryption on the binary format log to obtain an encrypted comma separated value format log.
In combination with any embodiment of the present invention, in the process of selecting the log to be exported, the method is specifically used for:
Screening logs to be exported in the task list according to start-stop time input by a user;
receiving logs to be exported selected by a user based on screening results
In combination with any embodiment of the present invention, in the process of selecting the log to be exported, the method is specifically used for:
screening logs to be exported in the task list according to the user permission of the user;
and receiving a log to be exported, which is selected by a user based on the screening result.
In combination with any embodiment of the present invention, after receiving the log to be exported selected by the user based on the screening result, the apparatus further includes a distributed lock setting module, configured to:
setting a distributed lock for the selected log to be exported;
in the process of screening the logs to be exported in the task table, the method further comprises:
and screening the task to be processed which is not provided with the distributed lock in the task table.
In combination with any one of the embodiments of the present invention, after exporting the encrypted log file to a corresponding file transmission platform, the apparatus further includes an information update module configured to:
responding to the success of the export to the corresponding file transmission platform, and updating task success information to a server;
Responding to failure of exporting to a corresponding file transmission platform, and updating task failure information to a server;
the task success information comprises at least one of a task state, a current file name and a server returned file name;
the task failure information comprises at least one of task state, current file name and error information.
For the device embodiments, reference is made to the description of the method embodiments for the relevant points, since they essentially correspond to the method embodiments. The apparatus embodiments described above are merely illustrative, wherein the elements described above as separate elements may or may not be physically separate, and the elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purposes of the present invention. Those of ordinary skill in the art will understand and implement the present invention without undue burden.
Referring now to FIG. 5, there is illustrated a schematic diagram of a computer device 500 suitable for use in implementing embodiments of the present application.
As shown in fig. 5, the computer device 500 includes a Central Processing Unit (CPU) 501, which can perform various appropriate works and processes according to a program stored in a Read Only Memory (ROM) 502 or a program loaded from a storage section 508 into a Random Access Memory (RAM) 503. In the RAM503, various programs and data required for the operation of the system 500 are also stored. The CPU501, ROM502, and RAM503 are connected to each other through a bus 504. An input/output (I/O) interface 505 is also connected to bus 504.
The following components are connected to the I/O interface 505: an input section 506 including a keyboard, a mouse, and the like; an output portion 507 including a Cathode Ray Tube (CRT), a liquid crystal feedback device (LCD), and the like, and a speaker, and the like; a storage portion 508 including a hard disk and the like; and a communication section 509 including a network interface card such as a LAN card, a modem, or the like. The communication section 509 performs communication processing via a network such as the internet. The drive 510 is also connected to the I/O interface 505 as needed. A removable medium 511 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 510 as needed, so that a computer program read therefrom is mounted as needed as the storage section 508.
In particular, according to embodiments of the present invention, the processes described above with reference to flowcharts may be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program tangibly embodied on a machine-readable medium, the computer program comprising program code for performing the method shown in the flowchart. In such an embodiment, the computer program may be downloaded and installed from a network via the communication portion 509, and/or installed from the removable media 511.
Computer readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. Computer-readable media, as defined herein, does not include transitory computer-readable media (transmission media), such as modulated data signals and carrier waves.
For convenience of description, the above devices are described as being functionally divided into various units, respectively. Of course, the functions of each element may be implemented in one or more software and/or hardware elements when implemented in the present application.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article or apparatus that comprises the element.
It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The application may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The application may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.
In this specification, each embodiment is described in a progressive manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for system embodiments, since they are substantially similar to method embodiments, the description is relatively simple, as relevant to see a section of the description of method embodiments.
The foregoing is merely exemplary of the present application and is not intended to limit the present application. Various modifications and changes may be made to the present application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. which are within the spirit and principles of the present application are intended to be included within the scope of the claims of the present application.
Claims (10)
1. A method of log data processing, the method comprising:
receiving log export task information sent by a log storage system, wherein the log export task information is generated by a log to be exported, which is selected by a user through the log storage system;
determining a corresponding original log file according to a task query language and a task identifier in the log derived task information;
carrying out file analysis and encryption on the original log file to obtain an encrypted log file;
and exporting the encrypted log file to a corresponding file transmission platform.
2. The method of claim 1, wherein said determining a corresponding original log file from a task query language and a task identifier in the log derived task information comprises:
deriving a task query language and a task identifier in the task information according to the log, and generating a file name of the binary format log;
and downloading the corresponding binary format log from the log storage system according to the file name of the binary format log.
3. The method according to claim 2, wherein the performing file parsing and encrypting on the original log file to obtain an encrypted log file includes:
And carrying out file analysis and encryption on the binary format log to obtain an encrypted comma separated value format log.
4. The method according to claim 1, wherein in selecting the log to be exported, the method comprises:
screening logs to be exported in the task list according to start-stop time input by a user;
and receiving a log to be exported, which is selected by a user based on the screening result.
5. The method according to claim 1, wherein in selecting the log to be exported, the method comprises:
screening logs to be exported in the task list according to the user permission of the user;
and receiving a log to be exported, which is selected by a user based on the screening result.
6. The method according to claim 4 or claim 5, further comprising, after receiving a log to be exported selected by a user based on the screening result:
setting a distributed lock for the selected log to be exported;
in the process of screening the logs to be exported in the task table, the method further comprises:
and screening the task to be processed which is not provided with the distributed lock in the task table.
7. The method of claim 1, wherein after exporting the encrypted log file to a corresponding file transfer platform, the method further comprises:
Responding to the success of the export to the corresponding file transmission platform, and updating task success information to a server;
responding to failure of exporting to a corresponding file transmission platform, and updating task failure information to a server;
the task success information comprises at least one of a task state, a current file name and a server returned file name;
the task failure information comprises at least one of task state, current file name and error information.
8. A log data processing apparatus, the apparatus comprising:
an information receiving module for: receiving log export task information sent by a log storage system, wherein the log export task information is generated by a log to be exported, which is selected by a user through the log storage system;
a file determining module, configured to: determining a corresponding original log file according to a task query language and a task identifier in the log derived task information;
a file conversion module for: carrying out file analysis and encryption on the original log file to obtain an encrypted log file;
a file export module, configured to: and exporting the encrypted log file to a corresponding file transmission platform.
9. An electronic device, the electronic device comprising:
a memory for storing processor-executable instructions;
a processor configured to execute executable instructions in the memory to implement the steps of the method of any one of claims 1 to 7.
10. A computer readable storage medium, on which a computer program is stored, characterized in that the program, when being executed by a processor, implements the steps of the method of any of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310449714.XA CN116483795A (en) | 2023-04-24 | 2023-04-24 | Log data processing method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310449714.XA CN116483795A (en) | 2023-04-24 | 2023-04-24 | Log data processing method and device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116483795A true CN116483795A (en) | 2023-07-25 |
Family
ID=87220905
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310449714.XA Pending CN116483795A (en) | 2023-04-24 | 2023-04-24 | Log data processing method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116483795A (en) |
-
2023
- 2023-04-24 CN CN202310449714.XA patent/CN116483795A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111258989B (en) | Database migration evaluation method and device, storage medium and computer equipment | |
| US9135322B2 (en) | Environment classification | |
| CN107220142B (en) | Method and device for executing data recovery operation | |
| US20110302277A1 (en) | Methods and apparatus for web-based migration of data in a multi-tenant database system | |
| US10521407B2 (en) | Grouping of database objects | |
| CN111460241B (en) | Data query method and device, electronic equipment and storage medium | |
| CN112988770B (en) | Method, device, electronic equipment and storage medium for updating serial number | |
| CN113420026B (en) | Database table structure changing method, device, equipment and storage medium | |
| CN110837535A (en) | Data synchronization method, device, equipment and medium | |
| US11748495B2 (en) | Systems and methods for data usage monitoring in multi-tenancy enabled HADOOP clusters | |
| CN111737227A (en) | Data modification method and system | |
| US11068496B2 (en) | System and method for data management | |
| CN116303628B (en) | Alarm data query method, system and equipment based on elastic search | |
| CN116483795A (en) | Log data processing method and device, electronic equipment and storage medium | |
| CN115757642A (en) | Data synchronization method and device based on filing log file | |
| CN112749197B (en) | Data fragment refreshing method, device, equipment and storage medium | |
| CN111563083B (en) | Report data query method, device and system | |
| CN114925044A (en) | Data synchronization method, device and equipment based on cloud storage and storage medium | |
| US20150195180A1 (en) | Workfile monitor | |
| CN115934670B (en) | Method and device for verifying copy placement strategy of HDFS (Hadoop distributed File System) multi-machine room | |
| US11514017B2 (en) | Systems and methods for provisioning a new secondary IdentityIQ instance to an existing IdentityIQ instance | |
| CN116455678B (en) | Network security log tandem method and system | |
| CN116150279A (en) | Database synchronization method and device, storage medium and electronic device | |
| CN116662373A (en) | Data access control method, device, equipment and medium | |
| CN116126819A (en) | System log processing method, device and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |