CN117708077A - Information processing method, information processing system, information processing device, storage medium and electronic equipment - Google Patents

Information processing method, information processing system, information processing device, storage medium and electronic equipment Download PDF

Info

Publication number
CN117708077A
CN117708077A CN202311707524.XA CN202311707524A CN117708077A CN 117708077 A CN117708077 A CN 117708077A CN 202311707524 A CN202311707524 A CN 202311707524A CN 117708077 A CN117708077 A CN 117708077A
Authority
CN
China
Prior art keywords
log files
information
log
processing
target
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202311707524.XA
Other languages
Chinese (zh)
Inventor
宇文晓硕
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Industrial and Commercial Bank of China Ltd ICBC
Original Assignee
Industrial and Commercial Bank of China Ltd ICBC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Industrial and Commercial Bank of China Ltd ICBC filed Critical Industrial and Commercial Bank of China Ltd ICBC
Priority to CN202311707524.XA priority Critical patent/CN117708077A/en
Publication of CN117708077A publication Critical patent/CN117708077A/en
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/18—File system types
    • G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
    • G06F16/1815—Journaling file systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
    • G06F16/33—Querying
    • G06F16/332—Query formulation
    • G06F16/3329—Natural language query formulation or dialogue systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
    • G06F16/35—Clustering; Classification
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/54—Interprogram communication
    • G06F9/546—Message passing systems or structures, e.g. queues
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2209/00—Indexing scheme relating to G06F9/00
    • G06F2209/54—Indexing scheme relating to G06F9/54
    • G06F2209/548—Queue

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Computational Linguistics (AREA)
  • Human Computer Interaction (AREA)
  • Artificial Intelligence (AREA)
  • Debugging And Monitoring (AREA)

Abstract

The application discloses an information processing method, an information processing system, an information processing device, a storage medium and electronic equipment, and relates to the field of financial science and technology or other related fields. The method comprises the following steps: receiving problem information sent by a target object; acquiring N log files based on the problem information, wherein the N log files at least comprise service data related to a target service; according to the target dimension, classifying N log files through at least one message queue to obtain N log files after classifying; according to the N log files after the classification processing, M log files for analyzing the problem information are determined, and the problem information is analyzed based on the M log files to obtain answer information corresponding to the problem information. By the method and the device, the problem that the efficiency of analyzing the business problems presented by the user is low due to the fact that the log files for analyzing the business problems presented by the user of the financial institution are difficult to accurately acquire in multiple dimensions in the related art is solved.

Description

Information processing method, information processing system, information processing device, storage medium and electronic equipment
Technical Field
The present invention relates to the field of financial science and technology or other related fields, and in particular, to an information processing method, an information processing system, an information processing device, a storage medium, and an electronic apparatus.
Background
In the related art, a log for analyzing service problems presented by users of a financial institution generally generates an hour-level text log according to a time of day, then obtains a required full log file through an FTP (File Transfer Protocol ) or a log platform, and determines the service problems presented by users of the financial institution by analyzing the obtained full log file. However, when analyzing the log, only the test data for analyzing the service problem that is currently used should be paid attention to, and redundant invalid data is not needed, but it is difficult to accurately obtain the log file for analyzing the service problem in a multi-dimension manner in the related art, only a full log file is obtained, and a large amount of redundant invalid data exists in the obtained full log file, so when determining the service problem presented by the user of the financial institution through analyzing the log, the use of the obtained full log file may interfere or block the progress of the analysis of the problem, and further may result in low efficiency of analyzing the service problem presented by the user.
Aiming at the problem that the log file for analyzing the business problems of the financial institution, which are presented by the user, is difficult to accurately acquire in a multi-dimension manner in the related art, so that the efficiency of analyzing the business problems of the user is low, no effective solution is presented at present.
Disclosure of Invention
The main object of the present invention is to provide an information processing method, system, device, storage medium and electronic apparatus, so as to solve the problem that in the related art, it is difficult to accurately obtain log files for analyzing service problems presented by users of financial institutions in multiple dimensions, resulting in lower efficiency in analyzing service problems presented by users.
In order to achieve the above object, according to one aspect of the present application, there is provided an information processing method. The method comprises the following steps: receiving problem information sent by a target object, wherein the problem information is information of a problem related to a target service in a financial institution; acquiring N log files based on the problem information, wherein the N log files at least comprise service data related to the target service, and N is a positive integer greater than 1; classifying the N log files through at least one message queue according to the target dimension to obtain N log files after classification; and determining M log files for analyzing the problem information according to the N log files after the classification processing, and analyzing the problem information based on the M log files to obtain answer information corresponding to the problem information, wherein M is a positive integer which is less than or equal to N and more than or equal to 1.
Further, based on the problem information, obtaining N log files includes: based on the problem information, acquiring T log files related to the target service, wherein T is a positive integer greater than N; s log files are determined from the T log files, wherein the S log files at least comprise target data, and the target data is at least one of the following: invalid data and data which do not meet the specification, wherein S is a positive integer smaller than T; and filtering the S log files from the T log files through an interceptor to obtain the N log files.
Further, according to the target dimension, classifying the N log files through at least one message queue, where obtaining N log files after the classifying includes: obtaining target information of each log file in the N log files, wherein the target information is at least one of the following: content information of each log file, generation time information of each log file, and name information of each log file; determining the target dimension according to the target information of each log file in the N log files; and classifying the N log files according to the target dimension through the at least one message queue to obtain N log files after classification.
Further, determining M log files for analyzing the problem information according to the N log files after the classification processing includes: determining a file storage system for storing the N log files after the classification processing; storing the N log files subjected to the classification processing into the file storage system; the M log files for analyzing the problem information are determined from the file storage system.
Further, storing the N log files after the sorting process in the file storage system includes: processing the N log files after the classification processing according to a preset processing mode to obtain N log files after the processing, wherein the preset processing mode is at least one of the following: a processing mode of partition processing and a processing mode of compression processing; and storing the processed N log files into the file storage system.
Further, if the preset processing mode is a partition processing mode and a compression processing mode, processing the N log files after the classification processing according to the preset processing mode, where the obtaining the N processed log files includes: acquiring generation time information of each log file in the N log files; partitioning the N log files subjected to the classification processing according to the generation time information of each log file in the N log files to obtain N log files subjected to the partition processing; determining log files of K areas based on the N log files processed by the partition, wherein K is a positive integer greater than 1; compressing the log files of each region to obtain N compressed log files; and taking the N log files after compression processing as the N log files after processing.
In order to achieve the above object, according to another aspect of the present application, there is provided an information processing system for executing the information processing method described in any one of the above, the information processing system comprising: the system comprises at least one production end, a database and a database, wherein the production end is used for acquiring N log files according to problem information sent by a target object, the problem information is information of problems related to target service in a financial institution, the N log files at least comprise service data related to the target service, and N is a positive integer greater than 1; the message queue is used for classifying the N log files according to the target dimension to obtain N classified log files; at least one consumer end, which is used for storing the N log files after the classification processing into a file storage system; and the application platform is used for downloading M log files for analyzing the problem information from the file storage system, analyzing the problem information based on the M log files and obtaining answer information corresponding to the problem information, wherein M is a positive integer which is less than or equal to N and more than or equal to 1.
In order to achieve the above object, according to another aspect of the present application, there is provided an information processing apparatus. The device comprises: a first receiving unit, configured to receive issue information sent by a target object, where the issue information is information of an issue related to a target service in a financial institution; a first obtaining unit, configured to obtain N log files based on the problem information, where the N log files at least include service data related to the target service, and N is a positive integer greater than 1; the first processing unit is used for classifying the N log files through at least one message queue according to the target dimension to obtain N log files after classification; and the first determining unit is used for determining M log files for analyzing the problem information according to the N log files after the classification processing, and analyzing the problem information based on the M log files to obtain answer information corresponding to the problem information, wherein M is a positive integer which is less than or equal to N and is more than or equal to 1.
Further, the first acquisition unit includes: a first obtaining subunit, configured to obtain T log files related to the target service based on the problem information, where T is a positive integer greater than N; the first determining subunit is configured to determine S log files from the T log files, where the S log files at least include target data, and the target data is at least one of the following: invalid data and data which do not meet the specification, wherein S is a positive integer smaller than T; and the first filtering subunit is used for filtering the S log files from the T log files through an interceptor to obtain the N log files.
Further, the first processing unit includes: the second obtaining subunit is configured to obtain target information of each log file in the N log files, where the target information is at least one of the following: content information of each log file, generation time information of each log file, and name information of each log file; the second determining subunit is used for determining the target dimension according to the target information of each log file in the N log files; and the first processing subunit is used for classifying the N log files through the at least one message queue according to the target dimension to obtain N log files after the classification processing.
Further, the first determination unit includes: a third determining subunit, configured to determine a file storage system for storing the N log files after the classification processing; the first storage subunit is used for storing the N log files subjected to the classification processing into the file storage system; and a fourth determining subunit configured to determine the M log files for analyzing the problem information from the file storage system.
Further, the first storage subunit includes: the first processing module is used for processing the N log files after the classification processing according to a preset processing mode to obtain N log files after the processing, wherein the preset processing mode is at least one of the following: a processing mode of partition processing and a processing mode of compression processing; and the first storage module is used for storing the processed N log files into the file storage system.
Further, if the preset processing mode is a processing mode of partition processing and a processing mode of compression processing, the first processing module includes: the first acquisition sub-module is used for acquiring the generation time information of each log file in the N log files; the first processing sub-module is used for carrying out partition processing on the N log files subjected to the classification processing according to the generation time information of each log file in the N log files to obtain N log files subjected to the partition processing; the first determining submodule is used for determining log files of K areas based on the N log files processed by the partition, wherein K is a positive integer greater than 1; the second processing submodule is used for compressing the log files of each region to obtain N compressed log files; and the second determining submodule is used for taking the N log files after the compression processing as the N log files after the processing.
In order to achieve the above object, according to another aspect of the present application, there is provided a computer-readable storage medium storing a program, wherein the program performs the information processing method of any one of the above.
In order to achieve the above object, according to another aspect of the present application, there is provided an electronic device including one or more processors and a memory for storing one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement any one of the information processing methods described above.
Through the application, the following steps are adopted: receiving problem information sent by a target object, wherein the problem information is information of problems related to target business in a financial institution; based on the problem information, N log files are obtained, wherein the N log files at least comprise service data related to a target service, and N is a positive integer greater than 1; according to the target dimension, classifying N log files through at least one message queue to obtain N log files after classifying; according to the N log files after the classification processing, M log files for analyzing the problem information are determined, and the problem information is analyzed based on the M log files to obtain answer information corresponding to the problem information, wherein M is a positive integer which is less than or equal to N and is greater than or equal to 1, and the problem that in the related art, the log files for analyzing the service problems proposed by users of financial institutions are difficult to accurately obtain in a multidimensional manner, so that the efficiency of analyzing the service problems proposed by the users is low is solved. The method comprises the steps of receiving problem information about a service, which is proposed by a customer of a financial institution, based on the problem information, obtaining a plurality of log files comprising service data, classifying the plurality of log files according to a target dimension through at least one message queue to obtain a plurality of classified log files, determining a plurality of log files for analyzing the problem information according to the plurality of classified log files, and analyzing the problem information based on the determined plurality of log files to obtain answer information corresponding to the problem information, so that the log files for analyzing the service problem which is proposed by the user of the financial institution can be accurately obtained in multiple dimensions, and further the effect of improving the efficiency of analyzing the service problem which is proposed by the user is achieved.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application, illustrate and explain the application and are not to be construed as limiting the application. In the drawings:
FIG. 1 is a flow chart of an information processing method provided according to an embodiment of the present application;
FIG. 2 is a schematic diagram of a log data collection module in an embodiment of the present application;
fig. 3 is a schematic diagram of a component configuration of a production-side jump in a log data acquisition module in an embodiment of the present application;
fig. 4 is a schematic diagram of a component configuration of a consumer-side jump in a log data acquisition module in an embodiment of the present application;
fig. 5 is a schematic diagram of an information processing apparatus provided according to an embodiment of the present application;
fig. 6 is a schematic diagram of an electronic device provided according to an embodiment of the present application.
Detailed Description
It should be noted that, in the case of no conflict, the embodiments and features in the embodiments may be combined with each other. The present application will be described in detail below with reference to the accompanying drawings in conjunction with embodiments.
In order to make the present application solution better understood by those skilled in the art, the following description will be made in detail and with reference to the accompanying drawings in the embodiments of the present application, it is apparent that the described embodiments are only some embodiments of the present application, not all embodiments. All other embodiments, which can be made by one of ordinary skill in the art based on the embodiments herein without making any inventive effort, shall fall within the scope of the present application.
It should be noted that the terms "first," "second," and the like in the description and claims of the present application and the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate in order to describe the embodiments of the present application described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
It should be noted that, related information (including, but not limited to, user equipment information, user personal information, etc.) and data (including, but not limited to, data for presentation, analyzed data, etc.) related to the present disclosure are information and data authorized by a user or sufficiently authorized by each party. For example, an interface is provided between the system and the relevant user or institution, before acquiring the relevant information, the system needs to send an acquisition request to the user or institution through the interface, and acquire the relevant information after receiving the consent information fed back by the user or institution.
For convenience of description, the following will describe some terms or terms related to the embodiments of the present application:
hadoop: is a tool for storing and processing large-scale data sets, and HDFS (Hadoop Distributed File System ) in a core component provides a distributed storage function of mass data, and MapReduce (a parallel programming model for processing large-scale data) provides a distributed computing function of mass data. Hadoop has the advantages of high reliability, high expansibility, high fault tolerance and the like.
Jume: the system is a framework based on a distributed architecture and can realize collection, aggregation and transmission of massive log data with high reliability.
Kafka: the distributed message queue based on the publish/subscribe mode has the advantages of high reliability, high throughput, strong expansibility, low fault tolerance and the like.
The present invention will be described with reference to preferred implementation steps, and fig. 1 is a flowchart of an information processing method according to an embodiment of the present application, as shown in fig. 1, and the method includes the following steps:
in step S101, question information sent by the target object is received, where the question information is information of a question related to a target service in the financial institution.
For example, the target object may be a customer of a financial institution, and the target service may be a transfer service handled by the customer in the financial institution, a balance inquiry service, or the like. The problem information may be data problems and environmental problems related to the business handled by the customer in the financial institution, etc., and for example, the problems may be problems related to transfer failure, inquiry failure, system prompt error, etc. presented by the customer. For example, when a customer of a financial institution handles a transfer service and the system prompts that the transfer has failed, a question (the question information described above) of the transfer failure fed back by the customer may be received.
Step S102, based on the problem information, N log files are obtained, wherein the N log files at least comprise service data related to a target service, and N is a positive integer greater than 1.
For example, the log file may be a log containing business data (transaction data) related to business handled by the customer in the financial institution. For example, after receiving a service problem (the above-described problem information) fed back by a customer, a plurality of log files for recording service data related to a service handled by the customer (the above-described target service) may be acquired.
Step S103, classifying the N log files according to the target dimension through at least one message queue to obtain N log files after classification.
For example, the message queue may be a Kafka message queue, and the target dimension may be a number of a client, a mobile phone number of the client, a date and time of generation of a log, a program file name of each business line, and the like. For example, after a plurality of log files for recording service data are acquired, the plurality of log files may be classified by one or more Kafka message queues according to different dimensions (number of clients, number of mobile phones of clients, date and time of generation of log, program file name of each service line, etc.), and log files of different dimensions may be obtained.
Step S104, determining M log files for analyzing the problem information according to the N log files after the classification processing, and analyzing the problem information based on the M log files to obtain answer information corresponding to the problem information, wherein M is a positive integer less than or equal to N and greater than or equal to 1.
For example, when the log generation date and time is used as a dimension, a plurality of log files are classified according to the log generation date and time, and then a log corresponding to 1 month generation date, a log corresponding to 2 months generation date, a log corresponding to 3 months generation date, and the like (N log files after the above classification processing) can be obtained. For example, when analyzing a business problem presented by a client, only a log file with a generation date of 2 months is needed, the log file with the generation date of 2 months can be screened out, the log files (the M log files) with the generation date of 2 months are analyzed to obtain an answer (the answer information) corresponding to the business problem presented by the client, and then the answer obtained by analysis is sent to the client.
It should be noted that the information processing method provided in the embodiment of the present application may be applied to a financial scenario.
Through the steps S101 to S104, the plurality of log files including the service data are obtained by receiving the problem information about the service provided by the customer of the financial institution and based on the problem information provided by the customer, and then the plurality of log files are classified according to the target dimension through at least one message queue, so as to obtain the plurality of classified log files, and then the plurality of log files for analyzing the problem information are determined according to the plurality of classified log files, and the problem information is analyzed based on the determined plurality of log files, so as to obtain the answer information corresponding to the problem information, thereby being capable of accurately obtaining the log files for analyzing the service problem provided by the user of the financial institution in a multidimensional manner, and further achieving the effect of improving the efficiency of analyzing the service problem provided by the user.
Optionally, in the information processing method provided in the embodiment of the present application, acquiring N log files based on the problem information includes: based on the problem information, acquiring T log files related to the target service, wherein T is a positive integer greater than N; s log files are determined from the T log files, wherein the S log files at least comprise target data, and the target data is at least one of the following: invalid data and data which do not meet the specification, wherein S is a positive integer smaller than T; and filtering the S log files from the T log files through an interceptor to obtain N log files.
For example, according to a business problem posed by a customer of a financial institution, a plurality of log files (the above-described T log files) related to the business problem may be acquired first, and then the log files (the above-described S log files) containing invalid dirty data (invalid data, data that does not meet specifications, etc.) may be determined among the log files. And these log files (the above-mentioned S log files) containing invalid dirty data (invalid data and data not conforming to specifications, etc.) can be filtered using an interceptor.
In summary, by using the interceptor to filter the invalid dirty data log file, the useful log file can be quickly and accurately obtained, and the influence of the invalid data in the log file on the subsequent analysis problem is avoided.
Optionally, in the information processing method provided in the embodiment of the present application, classifying, according to the target dimension, the N log files through at least one message queue, where obtaining the N log files after the classification includes: obtaining target information of each log file in the N log files, wherein the target information is at least one of the following: content information of each log file, generation time information of each log file, and name information of each log file; determining a target dimension according to the target information of each log file in the N log files; and classifying the N log files according to the target dimension through at least one message queue to obtain N log files after classification.
For example, the content information of each log file may include the number of the client and the mobile phone number of the client, so the target information may be the number of the client, the mobile phone number of the client, the date of generation of each log file, the name of each log file, and the like. Then, the number of the client in each log file, the mobile phone number of the client, the generation date of each log file and the name of each log file are respectively used as a dimension (the target dimension), and then according to the number of the client in each log file, the mobile phone number of the client, the generation date of each log file and the name of each log file, the plurality of log files are classified through one or more Kafka message queues, and log files with different dimensions (N log files after the classification processing) can be obtained.
Through the scheme, the obtained plurality of log files can be classified rapidly and accurately according to different dimensions.
Optionally, in the information processing method provided in the embodiment of the present application, determining, according to the N log files after the classification processing, M log files for analyzing problem information includes: determining a file storage system for storing N log files after the classification processing; storing the N log files subjected to the classification treatment into a file storage system; m log files for analyzing problem information are determined from the file storage system.
For example, the above-mentioned file storage system may be an HDFS system, and the logs classified according to different dimensions may be stored in the HDFS system, that is, log files of various dimensions (N log files after the above-mentioned classification process) are stored in the HDFS system, and then log files (M log files above) required for analyzing service problems raised by clients may be downloaded from the HDFS system through the log data acquisition module.
By the scheme, the log files required by analyzing the service problems can be quickly and accurately downloaded from the file storage system.
Optionally, in the information processing method provided in the embodiment of the present application, storing N log files after the classification processing in the file storage system includes: processing the N log files subjected to the classification processing according to a preset processing mode to obtain N log files subjected to the processing, wherein the preset processing mode is at least one of the following: a processing mode of partition processing and a processing mode of compression processing; and storing the processed N log files into a file storage system.
For example, after log files of various dimensions (N log files after the above classification processing) are written into the HDFS, the log files may be subjected to partition and compression processing, and when the log files are subjected to compression processing, LZO compression (a lossless compression algorithm) may be adopted for compression and then dropped into the HDFS.
By the scheme, the log files in the file storage system can be conveniently partitioned and compressed, so that the required log files can be conveniently and conveniently found in the follow-up process, and meanwhile, the storage space of the storage system occupied by the log files can be reduced by compression.
Optionally, in the information processing method provided in the embodiment of the present application, if the preset processing mode is a partition processing mode and a compression processing mode, processing the N log files after the classification processing according to the preset processing mode, where the obtaining the N log files after the processing includes: acquiring generation time information of each log file in N log files; carrying out partition processing on the N log files subjected to partition processing according to the generation time information of each log file in the N log files to obtain N log files subjected to partition processing; determining log files of K areas based on N log files after partition processing, wherein K is a positive integer greater than 1; compressing the log files of each region to obtain N compressed log files; and taking the N log files after compression processing as N log files after processing.
For example, when the log file is partitioned, the partition may be performed according to the generation time of the log. For example, the partition can be performed according to hours, the log file distribution generated from 11 hours to 12 hours can be stored in an area A of the HDFS, and the log file distribution generated from 12 hours to 13 hours can be stored in an area B of the HDFS; then, the log file in the area a can be compressed by adopting an LZO compression mode (a lossless compression algorithm), and then the log file in the area B can be compressed by adopting an LZO compression mode (a lossless compression algorithm).
By the scheme, the log file can be rapidly and accurately partitioned according to the generation time of the log.
In summary, in the information processing method provided by the embodiment of the present application, problem information sent by a target object is received, where the problem information is information of a problem related to a target service in a financial institution; based on the problem information, N log files are obtained, wherein the N log files at least comprise service data related to a target service, and N is a positive integer greater than 1; according to the target dimension, classifying N log files through at least one message queue to obtain N log files after classifying; according to the N log files after the classification processing, M log files for analyzing the problem information are determined, and the problem information is analyzed based on the M log files to obtain answer information corresponding to the problem information, wherein M is a positive integer which is less than or equal to N and is greater than or equal to 1, and the problem that in the related art, the log files for analyzing the service problems proposed by users of financial institutions are difficult to accurately obtain in a multidimensional manner, so that the efficiency of analyzing the service problems proposed by the users is low is solved. The method comprises the steps of receiving problem information about a service, which is proposed by a customer of a financial institution, based on the problem information, obtaining a plurality of log files comprising service data, classifying the plurality of log files according to a target dimension through at least one message queue to obtain a plurality of classified log files, determining a plurality of log files for analyzing the problem information according to the plurality of classified log files, and analyzing the problem information based on the determined plurality of log files to obtain answer information corresponding to the problem information, so that the log files for analyzing the service problem which is proposed by the user of the financial institution can be accurately obtained in multiple dimensions, and further the effect of improving the efficiency of analyzing the service problem which is proposed by the user is achieved.
The embodiment of the application also provides an information processing system, and it should be noted that the information processing system provided by the embodiment of the application can be used for executing the information processing method provided by the embodiment of the application. Also, the information processing system includes: the system comprises at least one production end, a database and a database, wherein the production end is used for acquiring N log files according to problem information sent by a target object, the problem information is information of problems related to target business in a financial institution, the N log files at least comprise business data related to the target business, and N is a positive integer greater than 1; the message queue is used for classifying the N log files according to the target dimension to obtain N log files after classification; at least one consumer end, which is used for storing the N log files after the classification processing into a file storage system; and the application platform is used for downloading M log files for analyzing the problem information from the file storage system, and analyzing the problem information based on the M log files to obtain answer information corresponding to the problem information, wherein M is a positive integer which is less than or equal to N and more than or equal to 1.
For example, the at least one production end may be one or more production end flues, the at least one message queue may be one or more Kafka message queues, the at least one consumption end may be one or more consumption end flues, and the application platform may be a platform for downloading log files. For example, one or more production side Flume can be used to collect a plurality of log files related to the business according to business problems set by customers of a financial institution, and then Kafka's topic (dimension) can be set according to different dimensions (such as custom, mobile phone number, date and time, program file name of each business line, etc.) through one or more Kafka message queues and pushed to the corresponding consumption side Flume; one or more consumer-side Flume may be reused to write data (log files) into HDFS (file storage system described above); then, log files (the above M log files) required for analyzing the service questions presented by the clients are downloaded from the HDFS through the application platform, and the downloaded log files are analyzed, and then answers (the above answer information) corresponding to the service questions presented by the clients are obtained.
In summary, through the constructed information processing system, the log files for analyzing the service questions can be accurately obtained according to different dimensions, and answer information corresponding to the service questions can be rapidly and accurately obtained through analysis of the log files.
In summary, in the information processing system provided in the embodiment of the present application, N log files are obtained according to problem information sent by a target object through at least one production end, where the problem information is information of a problem related to a target service in a financial institution, the N log files at least include service data related to the target service, and N is a positive integer greater than 1; at least one message queue classifies N log files according to the target dimension to obtain N log files after classification; at least one consumer stores the N log files subjected to the classification processing into a file storage system; the application platform downloads M log files for analyzing the problem information from the file storage system, analyzes the problem information based on the M log files to obtain answer information corresponding to the problem information, wherein M is a positive integer which is less than or equal to N and is greater than or equal to 1, and the problem that the efficiency of analyzing the service problem which is presented by a user is lower due to the fact that the log files for analyzing the service problem which is presented by the user of a financial institution are difficult to accurately acquire in a multidimensional manner in the related technology is solved. The method comprises the steps of receiving problem information about a service, which is proposed by a customer of a financial institution, based on the problem information, obtaining a plurality of log files comprising service data, classifying the plurality of log files according to a target dimension through at least one message queue to obtain a plurality of classified log files, determining a plurality of log files for analyzing the problem information according to the plurality of classified log files, and analyzing the problem information based on the determined plurality of log files to obtain answer information corresponding to the problem information, so that the log files for analyzing the service problem which is proposed by the user of the financial institution can be accurately obtained in multiple dimensions, and further the effect of improving the efficiency of analyzing the service problem which is proposed by the user is achieved.
For example, the method provided by the embodiment of the application can accurately acquire the log file required by the developer and the tester in a multi-dimensional manner.
In addition, the system for acquiring the multidimensional journal in real time provided by the embodiment can comprise three modules, namely a journal data system module, a journal data acquisition module and a journal data acquisition module. The log data system module is a log server system of the current stock; the log data acquisition module can be a module constructed by adopting a mode of a flame-Kafka-flame; the log data acquisition module may still employ an inventory of front-end platforms, such as conventional log downloads in an operation and maintenance self-service platform (a platform for downloading log files) or log downloads from application-side containers. The log data acquisition module is described below.
For example, the log data acquisition module may adopt a mode of a scale-Kafka-scale, designs a structure comprising a plurality of production-side scales and a plurality of consumption-side scales, and adds a Kafka cluster in the middle, wherein the scales of the production-side scales, the consumption-side scales and the Kafka cluster can be reasonably configured according to the log data volume. For example, fig. 2 is a schematic diagram of a log data collection module in the embodiment of the present application, as shown in fig. 2, where the log data collection module includes a plurality of production end Flume, kafka clusters, and a plurality of consumption end Flume, and log file in fig. 2 represents a log file. A plurality of log files can be acquired from a log server through a plurality of production end flums, and then the topic (dimension) of the Kafka is set according to different dimensions (such as a custom, a mobile phone number, a date and time, a program file name of each business strip line and the like) through a Kafka cluster and is pushed to a corresponding consumption end Flume; the data (log file) is then written into the HDFS using multiple consumer-side flues.
In addition, the reason why the log data collection module adopts the structure shown in fig. 2 is as follows:
1. the plurality of production ends Flame are used for collecting data simultaneously, a plurality of log files can be monitored, the collection efficiency of the data is improved, the breakpoint continuous transmission can be realized after faults occur, and the integrity of the data is guaranteed.
2. The use of Kafka clusters has the following effects:
(1) The topic (dimension) of Kafka can be set according to different dimensions (such as a custom, a mobile phone number, a date and time, a program file name of each business strip line and the like) and pushed to a corresponding consumption end Flume;
(2) When the data volume acquired by the production end Flume is too large, the consumption end Flume can not process log data in time, so that network congestion or system breakdown can be caused, and the Kafka cluster is used as a buffer zone to play a role in peak clipping.
3. The use of multiple consumption-side flues to write data into the HDFS can avoid generating a large number of small files, and at the same time, partition and compression processing can be performed on the data, so as to reduce the use rate of the disk and the consumption of network IO (Input/Output) streams.
4. The log files of the consumption end according to various dimensions are stored in the HDFS, and the required log files can be downloaded through a log data acquisition module.
Therefore, the adoption of the mode of the Flume-Kafka-Flume can not only increase the throughput of data acquisition, but also ensure the high reliability and high performance of the system.
For example, fig. 3 is a schematic diagram of a component configuration of a production side flime in the log data acquisition module in the embodiment of the present application, and the component configuration of the production side flime is shown in fig. 3. The Source (a mode for selecting and collecting) selects a TailDir Source (a plug-in for monitoring and collecting log files under a specified directory in real time and realizing breakpoint continuous transmission), and monitors log files of a log server in real time and can realize breakpoint continuous transmission; channel is selected from Kafka Channel (a component for transferring messages from one place to another), which directly transmits the collected data in different dimensions into the Topic named topic_log in the Kafka cluster; an interceptor named LogInterceptor (an interceptor used for intercepting log output in an application program) is customized between Source and Channel, invalid dirty data log files can be filtered, and the invalid dirty data can be non-specification data sent to a client for debugging in a development process.
For example, fig. 4 is a schematic diagram of a component configuration of the consumer-side flime in the log data acquisition module in the embodiment of the present application, and the component configuration of the consumer-side flime is shown in fig. 4. Source is selected from Kafka Source (a data Source used to extract data from a message queue), and log data is collected directly from Topic called topic_log in Kafka cluster; the Channel is selected as a File Channel (which represents a Channel connected to a File and does not need filtering), so that the data can be stored in an instantiated mode, and the integrity of the data is ensured; the Sink (output end) is selected to be an HDFS Sink (used for writing data into a Hadoop distributed file system), and collected data is compressed by adopting an LZO (lossless compression algorithm) compression mode and then is dropped into the HDFS. And the content corresponding to HDFS in fig. 4 represents a path of storing log files.
Moreover, by the method provided by the embodiment of the application, the log file can be flexibly and accurately acquired according to various dimensions; and the trouble of opening a log file on a white screen for acquiring a large file log and a local Ue (text editor) is avoided; and the efficiency of analyzing logs and locating problems can be effectively improved. In addition, the multi-dimensional log real-time acquisition system provided by the embodiment has good expandability, migration and reusability.
It should be noted that the steps illustrated in the flowcharts of the figures may be performed in a computer system such as a set of computer executable instructions, and that although a logical order is illustrated in the flowcharts, in some cases the steps illustrated or described may be performed in an order other than that illustrated herein.
The embodiment of the application also provides an information processing device, and it should be noted that the information processing device of the embodiment of the application can be used for executing the information processing method provided by the embodiment of the application. The information processing apparatus provided in the embodiment of the present application is described below.
Fig. 5 is a schematic diagram of an information processing apparatus provided according to an embodiment of the present application. As shown in fig. 5, the apparatus includes: a first receiving unit 501, a first acquiring unit 502, a first processing unit 503, and a first determining unit 504.
Specifically, the first receiving unit 501 is configured to receive issue information sent by a target object, where the issue information is information of an issue related to a target service in a financial institution;
a first obtaining unit 502, configured to obtain N log files based on the problem information, where N log files at least include service data related to a target service, and N is a positive integer greater than 1;
A first processing unit 503, configured to perform classification processing on the N log files through at least one message queue according to the target dimension, to obtain N log files after the classification processing;
the first determining unit 504 is configured to determine M log files for analyzing the problem information according to the N log files after the classification processing, and analyze the problem information based on the M log files to obtain answer information corresponding to the problem information, where M is a positive integer less than or equal to N and greater than or equal to 1.
In summary, the information processing apparatus provided in the embodiments of the present application receives, through the first receiving unit 501, issue information sent by a target object, where the issue information is information of an issue related to a target service in a financial institution; the first obtaining unit 502 obtains N log files based on the problem information, where N is a positive integer greater than 1, and the N log files at least include service data related to the target service; the first processing unit 503 classifies the N log files according to the target dimension through at least one message queue to obtain N log files after classification; the first determining unit 504 determines M log files for analyzing the problem information according to the N log files after the classification processing, and analyzes the problem information based on the M log files to obtain answer information corresponding to the problem information, where M is a positive integer less than or equal to N and greater than or equal to 1, so as to solve the problem that in the related art, it is difficult to accurately obtain the log files for analyzing the service problem presented by the user of the financial institution in a multidimensional manner, which results in lower efficiency of analyzing the service problem presented by the user. The method comprises the steps of receiving problem information about a service, which is proposed by a customer of a financial institution, based on the problem information, obtaining a plurality of log files comprising service data, classifying the plurality of log files according to a target dimension through at least one message queue to obtain a plurality of classified log files, determining a plurality of log files for analyzing the problem information according to the plurality of classified log files, and analyzing the problem information based on the determined plurality of log files to obtain answer information corresponding to the problem information, so that the log files for analyzing the service problem which is proposed by the user of the financial institution can be accurately obtained in multiple dimensions, and further the effect of improving the efficiency of analyzing the service problem which is proposed by the user is achieved.
Optionally, in the information processing apparatus provided in the embodiment of the present application, the first acquisition unit includes: the first acquisition subunit is used for acquiring T log files related to the target service based on the problem information, wherein T is a positive integer greater than N; the first determining subunit is configured to determine S log files from the T log files, where the S log files at least include target data, and the target data is at least one of the following: invalid data and data which do not meet the specification, wherein S is a positive integer smaller than T; and the first filtering subunit is used for filtering the S log files from the T log files through the interceptor to obtain N log files.
Optionally, in the information processing apparatus provided in the embodiment of the present application, the first processing unit includes: the second obtaining subunit is configured to obtain target information of each log file in the N log files, where the target information is at least one of the following: content information of each log file, generation time information of each log file, and name information of each log file; the second determining subunit is used for determining a target dimension according to the target information of each log file in the N log files; the first processing subunit is configured to perform classification processing on the N log files through at least one message queue according to the target dimension, so as to obtain N log files after the classification processing.
Optionally, in the information processing apparatus provided in the embodiment of the present application, the first determining unit includes: a third determining subunit, configured to determine a file storage system for storing the N log files after the classification processing; the first storage subunit is used for storing the N log files subjected to the classification processing into a file storage system; and a fourth determining subunit for determining M log files for analyzing the problem information from the file storage system.
Optionally, in the information processing apparatus provided in the embodiment of the present application, the first storage subunit includes: the first processing module is used for processing the N log files after the classification processing according to a preset processing mode to obtain the N log files after the processing, wherein the preset processing mode is at least one of the following: a processing mode of partition processing and a processing mode of compression processing; and the first storage module is used for storing the processed N log files into the file storage system.
Optionally, in the information processing apparatus provided in the embodiment of the present application, if the preset processing mode is a processing mode of partition processing and a processing mode of compression processing, the first processing module includes: the first acquisition sub-module is used for acquiring the generation time information of each log file in the N log files; the first processing submodule is used for carrying out partition processing on the N log files subjected to partition processing according to the generation time information of each log file in the N log files to obtain N log files subjected to partition processing; the first determining submodule is used for determining log files of K areas based on N log files after partition processing, wherein K is a positive integer greater than 1; the second processing submodule is used for compressing the log files of each region to obtain N compressed log files; and the second determining submodule is used for taking the N log files after compression processing as N log files after processing.
The information processing apparatus includes a processor and a memory, and the first receiving unit 501, the first acquiring unit 502, the first processing unit 503, the first determining unit 504, and the like described above are stored as program units in the memory, and the processor executes the program units stored in the memory to realize the corresponding functions.
The processor includes a kernel, and the kernel fetches the corresponding program unit from the memory. The kernel can be provided with one or more than one, and the efficiency of analyzing the service problems proposed by the user is improved by adjusting the kernel parameters.
The memory may include volatile memory, random Access Memory (RAM), and/or nonvolatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM), among other forms in computer readable media, the memory including at least one memory chip.
An embodiment of the present invention provides a computer-readable storage medium having stored thereon a program which, when executed by a processor, implements the information processing method.
The embodiment of the invention provides a processor for running a program, wherein the information processing method is executed when the program runs.
As shown in fig. 6, an embodiment of the present invention provides an electronic device, where the device includes a processor, a memory, and a program stored in the memory and executable on the processor, and when the processor executes the program, the following steps are implemented: receiving problem information sent by a target object, wherein the problem information is information of a problem related to a target service in a financial institution; acquiring N log files based on the problem information, wherein the N log files at least comprise service data related to the target service, and N is a positive integer greater than 1; classifying the N log files through at least one message queue according to the target dimension to obtain N log files after classification; and determining M log files for analyzing the problem information according to the N log files after the classification processing, and analyzing the problem information based on the M log files to obtain answer information corresponding to the problem information, wherein M is a positive integer which is less than or equal to N and more than or equal to 1.
The processor also realizes the following steps when executing the program: based on the problem information, obtaining N log files includes: based on the problem information, acquiring T log files related to the target service, wherein T is a positive integer greater than N; s log files are determined from the T log files, wherein the S log files at least comprise target data, and the target data is at least one of the following: invalid data and data which do not meet the specification, wherein S is a positive integer smaller than T; and filtering the S log files from the T log files through an interceptor to obtain the N log files.
The processor also realizes the following steps when executing the program: according to the target dimension, classifying the N log files through at least one message queue, wherein the obtaining N log files after classifying comprises the following steps: obtaining target information of each log file in the N log files, wherein the target information is at least one of the following: content information of each log file, generation time information of each log file, and name information of each log file; determining the target dimension according to the target information of each log file in the N log files; and classifying the N log files according to the target dimension through the at least one message queue to obtain N log files after classification.
The processor also realizes the following steps when executing the program: according to the N log files after the classification processing, determining M log files for analyzing the problem information includes: determining a file storage system for storing the N log files after the classification processing; storing the N log files subjected to the classification processing into the file storage system; the M log files for analyzing the problem information are determined from the file storage system.
The processor also realizes the following steps when executing the program: storing the N log files after the classification processing in the file storage system comprises the following steps: processing the N log files after the classification processing according to a preset processing mode to obtain N log files after the processing, wherein the preset processing mode is at least one of the following: a processing mode of partition processing and a processing mode of compression processing; and storing the processed N log files into the file storage system.
The processor also realizes the following steps when executing the program: if the preset processing mode is a partition processing mode and a compression processing mode, processing the N log files after the classification processing according to the preset processing mode, wherein the N log files after the processing comprise: acquiring generation time information of each log file in the N log files; partitioning the N log files subjected to the classification processing according to the generation time information of each log file in the N log files to obtain N log files subjected to the partition processing; determining log files of K areas based on the N log files processed by the partition, wherein K is a positive integer greater than 1; compressing the log files of each region to obtain N compressed log files; and taking the N log files after compression processing as the N log files after processing.
The device herein may be a server, PC, PAD, cell phone, etc.
The present application also provides a computer program product adapted to perform, when executed on a data processing device, a program initialized with the method steps of: receiving problem information sent by a target object, wherein the problem information is information of a problem related to a target service in a financial institution; acquiring N log files based on the problem information, wherein the N log files at least comprise service data related to the target service, and N is a positive integer greater than 1; classifying the N log files through at least one message queue according to the target dimension to obtain N log files after classification; and determining M log files for analyzing the problem information according to the N log files after the classification processing, and analyzing the problem information based on the M log files to obtain answer information corresponding to the problem information, wherein M is a positive integer which is less than or equal to N and more than or equal to 1.
When executed on a data processing device, is further adapted to carry out a program initialized with the method steps of: based on the problem information, obtaining N log files includes: based on the problem information, acquiring T log files related to the target service, wherein T is a positive integer greater than N; s log files are determined from the T log files, wherein the S log files at least comprise target data, and the target data is at least one of the following: invalid data and data which do not meet the specification, wherein S is a positive integer smaller than T; and filtering the S log files from the T log files through an interceptor to obtain the N log files.
When executed on a data processing device, is further adapted to carry out a program initialized with the method steps of: according to the target dimension, classifying the N log files through at least one message queue, wherein the obtaining N log files after classifying comprises the following steps: obtaining target information of each log file in the N log files, wherein the target information is at least one of the following: content information of each log file, generation time information of each log file, and name information of each log file; determining the target dimension according to the target information of each log file in the N log files; and classifying the N log files according to the target dimension through the at least one message queue to obtain N log files after classification.
When executed on a data processing device, is further adapted to carry out a program initialized with the method steps of: according to the N log files after the classification processing, determining M log files for analyzing the problem information includes: determining a file storage system for storing the N log files after the classification processing; storing the N log files subjected to the classification processing into the file storage system; the M log files for analyzing the problem information are determined from the file storage system.
When executed on a data processing device, is further adapted to carry out a program initialized with the method steps of: storing the N log files after the classification processing in the file storage system comprises the following steps: processing the N log files after the classification processing according to a preset processing mode to obtain N log files after the processing, wherein the preset processing mode is at least one of the following: a processing mode of partition processing and a processing mode of compression processing; and storing the processed N log files into the file storage system.
When executed on a data processing device, is further adapted to carry out a program initialized with the method steps of: if the preset processing mode is a partition processing mode and a compression processing mode, processing the N log files after the classification processing according to the preset processing mode, wherein the N log files after the processing comprise: acquiring generation time information of each log file in the N log files; partitioning the N log files subjected to the classification processing according to the generation time information of each log file in the N log files to obtain N log files subjected to the partition processing; determining log files of K areas based on the N log files processed by the partition, wherein K is a positive integer greater than 1; compressing the log files of each region to obtain N compressed log files; and taking the N log files after compression processing as the N log files after processing.
It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In one typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include volatile memory in a computer-readable medium, random Access Memory (RAM) and/or nonvolatile memory, etc., such as Read Only Memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.
Computer readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. Computer-readable media, as defined herein, does not include transitory computer-readable media (transmission media), such as modulated data signals and carrier waves.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article or apparatus that comprises an element.
It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The foregoing is merely exemplary of the present application and is not intended to limit the present application. Various modifications and changes may be made to the present application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. which are within the spirit and principles of the present application are intended to be included within the scope of the claims of the present application.

Claims (10)

1. An information processing method, characterized by comprising:
receiving problem information sent by a target object, wherein the problem information is information of a problem related to a target service in a financial institution;
acquiring N log files based on the problem information, wherein the N log files at least comprise service data related to the target service, and N is a positive integer greater than 1;
Classifying the N log files through at least one message queue according to the target dimension to obtain N log files after classification;
and determining M log files for analyzing the problem information according to the N log files after the classification processing, and analyzing the problem information based on the M log files to obtain answer information corresponding to the problem information, wherein M is a positive integer which is less than or equal to N and more than or equal to 1.
2. The method of claim 1, wherein obtaining N log files based on the problem information comprises:
based on the problem information, acquiring T log files related to the target service, wherein T is a positive integer greater than N;
s log files are determined from the T log files, wherein the S log files at least comprise target data, and the target data is at least one of the following: invalid data and data which do not meet the specification, wherein S is a positive integer smaller than T;
and filtering the S log files from the T log files through an interceptor to obtain the N log files.
3. The method of claim 1, wherein classifying the N log files by at least one message queue according to the target dimension, the obtaining N classified log files comprises:
Obtaining target information of each log file in the N log files, wherein the target information is at least one of the following: content information of each log file, generation time information of each log file, and name information of each log file;
determining the target dimension according to the target information of each log file in the N log files;
and classifying the N log files according to the target dimension through the at least one message queue to obtain N log files after classification.
4. The method of claim 1, wherein determining M log files for analyzing the problem information from the N log files after the classification process comprises:
determining a file storage system for storing the N log files after the classification processing;
storing the N log files subjected to the classification processing into the file storage system;
the M log files for analyzing the problem information are determined from the file storage system.
5. The method of claim 4, wherein storing the categorized N log files into the file storage system comprises:
Processing the N log files after the classification processing according to a preset processing mode to obtain N log files after the processing, wherein the preset processing mode is at least one of the following: a processing mode of partition processing and a processing mode of compression processing;
and storing the processed N log files into the file storage system.
6. The method according to claim 5, wherein if the preset processing mode is a partition processing mode and a compression processing mode, processing the N log files after the classification processing according to the preset processing mode, and obtaining the N processed log files includes:
acquiring generation time information of each log file in the N log files;
partitioning the N log files subjected to the classification processing according to the generation time information of each log file in the N log files to obtain N log files subjected to the partition processing;
determining log files of K areas based on the N log files processed by the partition, wherein K is a positive integer greater than 1;
compressing the log files of each region to obtain N compressed log files;
And taking the N log files after compression processing as the N log files after processing.
7. An information processing system for executing the information processing method according to any one of the preceding claims 1 to 6, comprising:
the system comprises at least one production end, a database and a database, wherein the production end is used for acquiring N log files according to problem information sent by a target object, the problem information is information of problems related to target service in a financial institution, the N log files at least comprise service data related to the target service, and N is a positive integer greater than 1;
the message queue is used for classifying the N log files according to the target dimension to obtain N classified log files;
at least one consumer end, which is used for storing the N log files after the classification processing into a file storage system;
and the application platform is used for downloading M log files for analyzing the problem information from the file storage system, analyzing the problem information based on the M log files and obtaining answer information corresponding to the problem information, wherein M is a positive integer which is less than or equal to N and more than or equal to 1.
8. An information processing apparatus, characterized by comprising:
a first receiving unit, configured to receive issue information sent by a target object, where the issue information is information of an issue related to a target service in a financial institution;
a first obtaining unit, configured to obtain N log files based on the problem information, where the N log files at least include service data related to the target service, and N is a positive integer greater than 1;
the first processing unit is used for classifying the N log files through at least one message queue according to the target dimension to obtain N log files after classification;
and the first determining unit is used for determining M log files for analyzing the problem information according to the N log files after the classification processing, and analyzing the problem information based on the M log files to obtain answer information corresponding to the problem information, wherein M is a positive integer which is less than or equal to N and is more than or equal to 1.
9. A computer-readable storage medium storing a program, wherein the program executes the information processing method according to any one of claims 1 to 6.
10. An electronic device comprising one or more processors and a memory for storing one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the information processing method of any of claims 1-6.
CN202311707524.XA 2023-12-12 2023-12-12 Information processing method, information processing system, information processing device, storage medium and electronic equipment Pending CN117708077A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202311707524.XA CN117708077A (en) 2023-12-12 2023-12-12 Information processing method, information processing system, information processing device, storage medium and electronic equipment

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202311707524.XA CN117708077A (en) 2023-12-12 2023-12-12 Information processing method, information processing system, information processing device, storage medium and electronic equipment

Publications (1)

Publication Number Publication Date
CN117708077A true CN117708077A (en) 2024-03-15

Family

ID=90161753

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202311707524.XA Pending CN117708077A (en) 2023-12-12 2023-12-12 Information processing method, information processing system, information processing device, storage medium and electronic equipment

Country Status (1)

Country Link
CN (1) CN117708077A (en)

Similar Documents

Publication Publication Date Title
CN106897206B (en) Service testing method and device
EP3545419B1 (en) Distributed code tracing system
US20220004480A1 (en) Log data collection method, log data collection device, storage medium, and log data collection system
CN113360554B (en) Method and equipment for extracting, converting and loading ETL (extract transform load) data
CN105224445A (en) Distributed tracking system
CN113656245B (en) Data inspection method and device, storage medium and processor
CN111352903A (en) Log management platform, log management method, medium, and electronic device
CN109951326B (en) Data processing method, device, storage medium and processor
CN110557291A (en) Network service monitoring system
CN109359109B (en) Data processing method and system based on distributed stream computing
CN111367803A (en) Method and system for improving testing efficiency of client software
CN110334011B (en) Method and device for executing test case
CN108228432A (en) A kind of distributed link tracking, analysis method and server, global scheduler
CN110968487A (en) Abnormal data analysis method and device
CN113342806A (en) Big data processing method and device, storage medium and processor
CN117539754A (en) Pressure testing method and device, storage medium and electronic equipment
CN117708077A (en) Information processing method, information processing system, information processing device, storage medium and electronic equipment
CN116566873A (en) ELK-based automatic log analysis method, system and storage medium
CN115664992A (en) Network operation data processing method and device, electronic equipment and medium
CN114895851A (en) Data processing method, data processing device, storage medium and electronic equipment
CN114895852A (en) Data processing method, system, device, storage medium and electronic equipment
CN110597603A (en) Scheduling method and system of distributed scheduling tasks
CN117853221A (en) Method, device and system for monitoring consumption credit model
US20180176291A1 (en) Automated server deployment platform
CN114448832B (en) Pressure testing system for trading platform on stock line

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination