CN111651481A - Log processing method, device, equipment and computer readable storage medium - Google Patents
Log processing method, device, equipment and computer readable storage medium Download PDFInfo
- Publication number
- CN111651481A CN111651481A CN202010475917.2A CN202010475917A CN111651481A CN 111651481 A CN111651481 A CN 111651481A CN 202010475917 A CN202010475917 A CN 202010475917A CN 111651481 A CN111651481 A CN 111651481A
- Authority
- CN
- China
- Prior art keywords
- service
- processed
- data
- line data
- type
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2453—Query optimisation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q40/00—Finance; Insurance; Tax strategies; Processing of corporate or income taxes
- G06Q40/08—Insurance
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The invention provides a log processing method, a log processing device, log processing equipment and a computer readable storage medium, wherein the method comprises the following steps: acquiring service information corresponding to at least one service to be processed, wherein the service information comprises all log information corresponding to the service to be processed and an identifier of the service to be processed; analyzing all the log information to obtain a message set corresponding to all the log information, wherein the message set comprises at least one row of data; traversing the message set, and determining the message type corresponding to each row of data; and determining all the line data corresponding to each service to be processed according to the message type and the identification of the service to be processed, and performing data processing operation on all the line data. Therefore, all the line data corresponding to the service to be processed can be determined according to the message type of the line data, whether any two lines of data belong to the same table or not is determined from the database according to the policy number, and the pressure of the database is reduced.
Description
Technical Field
The present invention relates to the field of data processing, and in particular, to a log processing method, apparatus, device, and computer-readable storage medium.
Background
In a binary log (binlog) synchronous application scenario, for example, in an underwriting service, data of an underwriting service includes policy master table data, applicant table data, insured table data, target table data and the like, and after the underwriting is successful, the data of all the tables are inserted into a mysql database in one transaction.
When receiving the binlog generated by the service insertion database, the existing log processing method needs to analyze the binlog. However, since the analyzed data of each table is atomized, it is impossible to determine whether the data of the two tables belong to the same service. For example, the row data of a policy master table and the row data of a policy insured person are received, but the two row data cannot be determined to be in the same service, if all data of a service are required, all data of the service are inquired in the database according to the policy number in the row data of the master table, and the pressure of the database is increased.
Disclosure of Invention
The invention provides a log processing method, a log processing device, log processing equipment and a computer readable storage medium, which are used for solving the technical problems that when the existing log processing method is used for determining whether two table-row data are the same service, all data of the service need to be inquired from a database through a single number of a table, the inquiry efficiency is low, and the database is stressed greatly.
The first aspect of the present invention provides a log processing method, including:
acquiring service information corresponding to at least one service to be processed, wherein the service information comprises all log information corresponding to the service to be processed and an identifier of the service to be processed;
analyzing all the log information to obtain a message set corresponding to all the log information, wherein the message set comprises at least one row of data;
traversing the message set, and determining the message type corresponding to each row of data;
and determining all the line data corresponding to each service to be processed according to the message type and the identification of the service to be processed, and performing data processing operation on all the line data.
In one possible design, the message types include a start type, an end type, and a line data type.
In a possible design, the obtaining service information corresponding to at least one service to be processed includes:
acquiring current service information corresponding to the current batch of to-be-processed services;
if the historical service information of the previous batch of services to be processed is determined to exist, acquiring the historical service information, wherein the current service information and the historical service information form the service information;
accordingly, the set of messages includes: and the data of the row corresponding to the current batch of the service to be processed and the data of the row corresponding to the last batch of the service to be processed.
In a possible design, the determining, according to the message type and the identifier of the service to be processed, all the lines of data corresponding to each service to be processed includes:
if the message type is a starting type and the identifier of the service to be processed is the identifier of the service to be processed in the current batch, creating a new row data set according to the identifier of the service to be processed;
storing the line data with the message type as the starting type into the new line data set;
and storing the line data type and the ending type of the service to be processed with the same identification into the new line data set.
In a possible design, the determining all the line data corresponding to each service to be processed according to the message type and the identifier of the service to be processed includes:
if the message type of the line data is the line data type and the identifier of the service to be processed is the identifier of the service to be processed in the previous batch, acquiring a line data set corresponding to the service to be processed in the previous batch according to the identifier of the service to be processed;
storing the line data of the line data type into a line data set corresponding to the service to be processed in the last batch;
and storing the data of the ending type corresponding to the service to be processed in the last batch into the data set corresponding to the service to be processed in the last batch.
In one possible design, the method further includes:
and if the fact that the row data set corresponding to the current batch of the to-be-processed services does not comprise the row data of the ending type is determined, the to-be-processed services are subjected to retention operation, and all the corresponding row data are determined together with the to-be-processed services with the same identification in the next batch.
In one possible design, the to-be-processed service is a to-be-processed underwriting service, and the performing data processing operation on all the line data includes:
and storing all the row data corresponding to each service to be processed into a database so that the back-end server acquires all the row data corresponding to the same service to be processed.
A second aspect of the present invention provides a log processing apparatus including:
the system comprises an acquisition module, a processing module and a processing module, wherein the acquisition module is used for acquiring service information corresponding to at least one service to be processed, and the service information comprises all log information corresponding to the service to be processed and an identifier of the service to be processed;
the analysis module is used for analyzing all the log information to obtain a message set corresponding to all the log information, wherein the message set comprises at least one row of data;
the determining module is used for traversing the message set and determining the message types corresponding to the data of each row;
and the processing module is used for determining all the line data corresponding to each service to be processed according to the message type and the identification of the service to be processed and carrying out data processing operation on all the line data.
In one possible design, the message types include a start type, an end type, and a line data type.
In one possible design, the obtaining module is specifically configured to:
acquiring current service information corresponding to the current batch of to-be-processed services;
if the historical service information of the previous batch of services to be processed is determined to exist, acquiring the historical service information, wherein the current service information and the historical service information form the service information;
accordingly, the set of messages includes: and the data of the row corresponding to the current batch of the service to be processed and the data of the row corresponding to the last batch of the service to be processed.
In one possible design, the processing module is specifically configured to:
if the message type is a starting type and the identifier of the service to be processed is the identifier of the service to be processed in the current batch, creating a new row data set according to the identifier of the service to be processed;
storing the line data with the message type as the starting type into the new line data set;
and storing the line data type and the ending type of the service to be processed with the same identification into the new line data set.
In one possible design, the processing module is specifically configured to:
if the message type of the line data is the line data type and the identifier of the service to be processed is the identifier of the service to be processed in the previous batch, acquiring a line data set corresponding to the service to be processed in the previous batch according to the identifier of the service to be processed;
storing the line data of the line data type into a line data set corresponding to the service to be processed in the last batch;
and storing the data of the ending type corresponding to the service to be processed in the last batch into the data set corresponding to the service to be processed in the last batch.
In one possible design, the apparatus further includes:
and the retention module is used for performing retention operation on the service to be processed if the data set corresponding to the service to be processed in the current batch does not comprise the data of the ending type, so as to determine all corresponding data of the data together with the service to be processed with the same identification in the next batch.
In one possible design, the pending service is a pending underwriting service, and the processing module is configured to:
and storing all the row data corresponding to each service to be processed into a database so that the back-end server acquires all the row data corresponding to the same service to be processed.
A third aspect of the present invention provides a log processing apparatus including: a memory, a processor;
a memory; a memory for storing the processor-executable instructions;
wherein the processor is configured to perform the log processing method of the first aspect by the processor.
A fourth aspect of the present invention provides a computer-readable storage medium having stored therein computer-executable instructions for implementing the log processing method according to the first aspect when the computer-executable instructions are executed by a processor.
According to the log processing method, the log processing device, the log processing equipment and the computer readable storage medium, the line data after the log information corresponding to the service to be processed is analyzed are classified according to the preset type, so that all the line data corresponding to the service to be processed can be determined according to the message type of the line data, whether any two lines of data belong to the same table or not is determined from the database according to the policy number, and the pressure of the database is reduced.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings can be obtained by those skilled in the art according to the drawings.
FIG. 1 is a schematic diagram of a network architecture on which the present invention is based;
fig. 2 is a schematic flowchart of a log processing method according to an embodiment of the present invention;
fig. 3 is a schematic flowchart of a log processing method according to a second embodiment of the present invention;
FIG. 4 is a schematic flow chart provided by an embodiment of the present invention;
fig. 5 is a schematic flowchart of a log processing method according to a third embodiment of the present invention;
fig. 6 is a schematic structural diagram of a log processing apparatus according to a fourth embodiment of the present invention;
fig. 7 is a schematic structural diagram of a log processing device according to a fifth embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other examples obtained based on the examples in the present invention are within the scope of the present invention.
The invention provides a log processing method, a device, equipment and a computer readable storage medium, aiming at the technical problems that when the existing log processing method is used for determining whether two table row data are the same service, all data of the service need to be inquired from a database through a single number of the table, the inquiry efficiency is low, and the database is stressed greatly.
It should be noted that the log processing method, device, apparatus, and computer-readable storage medium provided in the present application may be applied to a scenario of processing various logs.
Fig. 1 is a schematic diagram of a network architecture based on the present invention, and as shown in fig. 1, the network architecture based on the present invention at least includes: a data server 1 and a log processing device 2. The data server 1 can be a MySQL database in which a large amount of data is stored; the log processing device 2 is written by C/C + +, Java, Shell or Python.
Fig. 2 is a schematic flowchart of a log processing method according to an embodiment of the present invention, as shown in fig. 1, the method includes:
The execution subject of the present embodiment is a log processing apparatus. The log processing device can be in communication connection with the data server, so that information interaction can be carried out with the data server, service information can be acquired, and the like. In practical application, all log information corresponding to the underwriting service can be stored in a data server, such as policy master table data, applicant table data, insured table data, target table data, and the like. In order to clearly identify the corresponding relationship between the log information and the service to be processed, the stored service information includes all log information corresponding to the service to be processed and the identification of the service to be processed. Correspondingly, the log processing device can obtain the service information corresponding to at least one service to be processed from the data server.
And 102, analyzing all the log information to obtain a message set corresponding to all the log information, wherein the message set comprises at least one row of row data.
In this embodiment, after the service information corresponding to at least one service to be processed is obtained from the data server, the service information may be analyzed to obtain a message set corresponding to all log information. It should be noted that any analysis method may be adopted to implement the analysis operation on the log information, for example, the log information may be analyzed according to the MySQL protocol, which is not limited in the present invention. The message set comprises at least one row of data corresponding to at least the log information.
Different from the prior art, after the log information is analyzed, only a plurality of line data can be obtained, so that whether any two line data belong to the same table cannot be determined. In this embodiment, the message type of each line of data may be determined first, and the line of data may be classified according to the message type. Specifically, the message set may be traversed, and a message type corresponding to each line of data in the message combination is determined, where the message type may be specifically divided into a start type, an end type, and a line data type, where the start type indicates that the line of data is a line of data at the beginning of the current table, the end type indicates that the line of data is a line of data at the end of the current table, and correspondingly, the line data type indicates that the line of data is a line of data in the middle of the current table.
And step 104, determining all the line data corresponding to each service to be processed according to the message type and the identification of the service to be processed, and performing data processing operation on all the line data.
In this embodiment, all the row data corresponding to the service to be processed may be determined according to the message type and the identifier of the service to be processed, so that it is not necessary to determine whether any two rows of data belong to the same table from the database according to the policy number. And then corresponding data processing operation can be carried out on all the line data.
Further, on the basis of the first embodiment, the step 101 specifically includes:
acquiring current service information corresponding to the current batch of to-be-processed services;
if the historical service information of the previous batch of services to be processed is determined to exist, acquiring the historical service information, wherein the current service information and the historical service information form the service information;
accordingly, the set of messages includes: and the data of the row corresponding to the current batch of the service to be processed and the data of the row corresponding to the last batch of the service to be processed.
In this embodiment, if the service information corresponding to the pending service of the current batch is not processed completely, a leave operation may be performed on the service information to wait for the pending service of the next batch to be processed together. Therefore, after the current service information corresponding to the current batch of the to-be-processed service is acquired, whether historical service information that the previous batch of the to-be-processed service is not processed exists can be determined. If the historical service information exists, the historical service information can be obtained, and the current service information and the historical service information form the service information. Correspondingly, the message set generated after the service information is analyzed comprises the row data corresponding to the current batch of service to be processed and the row data corresponding to the last batch of service to be processed.
According to the log processing method provided by the embodiment, the line data after the log information corresponding to the service to be processed is analyzed is classified according to the preset type, so that all the line data corresponding to the service to be processed can be determined according to the message type of the line data, whether any two lines of data belong to the same table or not is determined from the database according to the policy number, and the pressure of the database is reduced.
Fig. 3 is a schematic flowchart of a log processing method according to a second embodiment of the present invention, where on the basis of any of the foregoing embodiments, step 104 specifically includes:
step 203, storing the line data of the same identified line data type and ending type of the service to be processed into the new line data set.
In this embodiment, the message type of the line data may specifically be a start type, and the start type indicates that the line data is the line data at the beginning of the current table. When the message type of the line data is determined to be the starting type, the last service is indicated to be ended, and the data of another service is newly started to be processed, so that a new line data set can be created according to the identification of the service to be processed. And storing the line data with the message type as the starting type into the new line data set. By the message type, the line data corresponding to the current service to be processed can be accurately distinguished from the line data corresponding to the previous service, and a basis is provided for acquiring all data corresponding to log information. Further, the row data of the same identified to-be-processed service and the row data of the ending type may both be stored in the row data set.
According to the log processing method provided by the embodiment, when the message type of the line data is determined to be the starting type, a new line data set is created according to the identifier of the service to be processed, so that the line data corresponding to the current service to be processed can be accurately distinguished from the line data corresponding to the previous service, the line data corresponding to two services in the same line data set is avoided, and the pressure of a database can be further reduced.
Further, on the basis of any of the above embodiments, the step 104 specifically includes:
if the message type of the line data is the line data type and the identifier of the service to be processed is the identifier of the service to be processed in the previous batch, acquiring a line data set corresponding to the service to be processed in the previous batch according to the identifier of the service to be processed;
storing the line data of the line data type into a line data set corresponding to the service to be processed in the last batch;
and storing the data of the ending type corresponding to the service to be processed in the last batch into the data set corresponding to the service to be processed in the last batch.
In this embodiment, the message type of the line data may specifically be a line data type, and the line data type indicates that the line data is a line data in the middle of the current table. At this time, the line of data may be stored in a line of data set corresponding to the identifier of the service to be processed. Specifically, if the message type of the row data is the row data type and the identifier of the service to be processed is the identifier of the service to be processed in the previous batch, the row data set corresponding to the service to be processed in the previous batch is obtained according to the identifier of the service to be processed. And storing the line data of the line data type into a line data set corresponding to the service to be processed in the previous batch. And storing the data of the ending type corresponding to the service to be processed in the last batch into the data set corresponding to the service to be processed in the last batch.
Optionally, on the basis of any of the above embodiments, the method further includes:
and if the fact that the row data set corresponding to the current batch of the to-be-processed services does not comprise the row data of the ending type is determined, the to-be-processed services are subjected to retention operation, and all the corresponding row data are determined together with the to-be-processed services with the same identification in the next batch.
In this embodiment, if it is determined that the line data set corresponding to the current batch of pending services does not include the line data of the ending type, the pending services may be left, and when the next batch of pending services is obtained, all the corresponding line data is determined together with the pending services with the same identifier in the next batch.
According to the log processing method provided by the embodiment, the row data of which the current message type is the row data type and all the row data corresponding to the previous service to be processed are stored in the row data set, so that the connection between the current service to be processed and the previous service to be processed can be ensured, and the database pressure can be further reduced.
Further, on the basis of any of the above embodiments, the step 104 specifically includes:
and if the message type of the line data is an ending type, storing the line data into a line data set corresponding to the identifier of the service to be processed, and acquiring all line data corresponding to the service to be processed.
In this embodiment, the message type of the line data may specifically be an end type, where the end type represents that the line data is the line data at the end of the current table. If the message type of the current line data is the ending type, the line data can be directly stored into the line data set corresponding to the identifier of the service to be processed, and all line data corresponding to the service to be processed are obtained.
Further, on the basis of any of the foregoing embodiments, the to-be-processed service is a to-be-processed underwriting service, and step 104 includes:
and storing all the row data corresponding to each service to be processed into a database so that the back-end server acquires all the row data corresponding to the same service to be processed.
In this embodiment, after all the row data corresponding to each service to be processed is obtained, all the row data corresponding to each service to be processed may be stored in the database, so that the subsequent back-end server may directly obtain all the row data corresponding to the required service to be processed from the database according to actual requirements.
Fig. 4 is a schematic flow chart provided in an embodiment of the present invention, and as shown in fig. 4, service information corresponding to at least one service to be processed may be obtained, and all the service information may be analyzed to obtain a message set. And circularly judging the type of each row of data in the message set, and if the type is a starting type, establishing a new row of data set according to the identification of the service to be processed. If the data type is the row data type, a row data set corresponding to the last batch of to-be-processed services can be obtained; and storing the line data of the line data type into a line data set corresponding to the service to be processed in the last batch. If the type of the line data is the ending type, the line data can be stored in a line data set corresponding to the identifier of the service to be processed. And detecting whether the current message set is traversed completely, if so, detecting whether the row data of which the transaction is not finished exist, and if not, finishing the processing. If present, it can be left to process with the next batch of pending traffic. Otherwise, if the traversal is not finished, the process of the circular traversal is repeatedly executed until the traversal of the message set is finished.
According to the log processing method provided by the embodiment, the line data with the current message type as the ending type is stored in the line data set corresponding to the identifier of the service to be processed, and all line data corresponding to the service to be processed is obtained, so that the line data corresponding to the current service to be processed can be accurately distinguished from the line data corresponding to the next service, the line data corresponding to two services in the same line data set is avoided, and the pressure of a database can be further reduced.
Fig. 5 is a schematic flowchart of a log processing method according to a third embodiment of the present invention, where on the basis of any of the foregoing embodiments, after step 104, as shown in fig. 5, the method further includes:
and 302, if yes, analyzing the new log information, and processing the analyzed row data as the row data of the next service to be processed.
In this embodiment, after the message set is circularly traversed, if it is detected that there is an incomplete transaction, that is, new log information, the new log information may be analyzed and kept, and when a next service to be processed is received, the new log information and a next batch of messages are processed together, so that a last line of data in the current service to be processed can be prevented from being lost.
In the log processing method provided by this embodiment, whether the to-be-processed service generates new log information is detected, and when new log information exists, the new log information is analyzed, and the analyzed row data is processed as row data of the next to-be-processed service, so that loss of the last row data in the current to-be-processed service can be avoided, and integrity of the row data corresponding to the to-be-processed service data is ensured.
Fig. 6 is a schematic structural diagram of a log processing apparatus according to a fourth embodiment of the present invention, and as shown in fig. 6, the apparatus includes: the system comprises an acquisition module 41, an analysis module 42, a determination module 43 and a processing module 44, wherein the acquisition module is configured to acquire service information corresponding to at least one service to be processed, and the service information includes all log information corresponding to the service to be processed and an identifier of the service to be processed; the analysis module is used for analyzing all the log information to obtain a message set corresponding to all the log information, wherein the message set comprises at least one row of data; the determining module is used for traversing the message set and determining the message types corresponding to the data of each row; and the processing module is used for determining all the line data corresponding to each service to be processed according to the message type and the identification of the service to be processed and carrying out data processing operation on all the line data.
Further, on the basis of any of the above embodiments, the message types include a start type, an end type, and a line data type.
Further, on the basis of any of the above embodiments, the obtaining module is specifically configured to:
acquiring current service information corresponding to the current batch of to-be-processed services;
if the historical service information of the previous batch of services to be processed is determined to exist, acquiring the historical service information, wherein the current service information and the historical service information form the service information;
accordingly, the set of messages includes: and the data of the row corresponding to the current batch of the service to be processed and the data of the row corresponding to the last batch of the service to be processed.
Further, on the basis of any of the above embodiments, the processing module is specifically configured to:
if the message type is a starting type and the identifier of the service to be processed is the identifier of the service to be processed in the current batch, creating a new row data set according to the identifier of the service to be processed;
storing the line data with the message type as the starting type into the new line data set;
and storing the line data type and the ending type of the service to be processed with the same identification into the new line data set.
Further, on the basis of any of the above embodiments, the processing module is specifically configured to:
if the message type of the line data is the line data type and the identifier of the service to be processed is the identifier of the service to be processed in the previous batch, acquiring a line data set corresponding to the service to be processed in the previous batch according to the identifier of the service to be processed;
storing the line data of the line data type into a line data set corresponding to the service to be processed in the last batch;
and storing the data of the ending type corresponding to the service to be processed in the last batch into the data set corresponding to the service to be processed in the last batch.
Further, on the basis of any one of the above embodiments, the apparatus further includes:
and the retention module is used for performing retention operation on the service to be processed if the data set corresponding to the service to be processed in the current batch does not comprise the data of the ending type, so as to determine all corresponding data of the data together with the service to be processed with the same identification in the next batch.
Further, on the basis of any of the above embodiments, the service to be processed is a service to be processed, and the processing module is configured to:
and storing all the row data corresponding to each service to be processed into a database so that the back-end server acquires all the row data corresponding to the same service to be processed.
Fig. 7 is a schematic structural diagram of a log processing device according to a fifth embodiment of the present invention, as shown in fig. 7, including: a memory 51, a processor 52;
a memory 51; a memory 51 for storing instructions executable by the processor 52;
wherein the processor 52 is configured to execute the log processing method according to any one of the above embodiments by the processor 52.
Still another aspect of the present invention provides a computer-readable storage medium, in which computer-executable instructions are stored, and the computer-executable instructions are executed by a processor to implement the log processing method according to any one of the above embodiments.
The memory 51 stores programs. In particular, the program may include program code comprising computer operating instructions. The memory 51 may comprise high-speed RAM memory, and may also include non-volatile memory (non-volatile memory), such as at least one disk memory.
The processor 52 may be a Central Processing Unit (CPU), an Application Specific Integrated Circuit (ASIC), or one or more Integrated circuits configured to implement embodiments of the present invention.
Alternatively, in a specific implementation, if the memory 51 and the processor 52 are implemented independently, the memory 51 and the processor 52 may be connected to each other through a bus and perform communication with each other. The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, an Extended ISA (enhanced Industry Standard Architecture) bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown in FIG. 6, but this is not intended to represent only one bus or type of bus.
Alternatively, in a specific implementation, if the memory 51 and the processor 52 are integrated on a chip, the memory 51 and the processor 52 may complete the same communication through an internal interface.
It is clear to those skilled in the art that, for convenience and brevity of description, the specific working process of the apparatus described above may refer to the corresponding process in the foregoing method embodiment, and is not described herein again.
Those of ordinary skill in the art will understand that: all or a portion of the steps of implementing the above-described method embodiments may be performed by hardware associated with program instructions. The program may be stored in a computer-readable storage medium. When executed, the program performs steps comprising the method embodiments described above; and the aforementioned storage medium includes: various media that can store program codes, such as ROM, RAM, magnetic or optical disks.
Finally, it should be noted that: the above embodiments are only used to illustrate the technical solution of the present invention, and not to limit the same; while the invention has been described in detail and with reference to the foregoing embodiments, it will be understood by those skilled in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present invention.
Claims (10)
1. A log processing method, comprising:
acquiring service information corresponding to at least one service to be processed, wherein the service information comprises all log information corresponding to the service to be processed and an identifier of the service to be processed;
analyzing all the log information to obtain a message set corresponding to all the log information, wherein the message set comprises at least one row of data;
traversing the message set, and determining the message type corresponding to each row of data;
and determining all the line data corresponding to each service to be processed according to the message type and the identification of the service to be processed, and performing data processing operation on all the line data.
2. The method of claim 1, wherein the message types include a start type, an end type, and a line data type.
3. The method according to claim 2, wherein the obtaining of the service information corresponding to the at least one service to be processed includes:
acquiring current service information corresponding to the current batch of to-be-processed services;
if the historical service information of the previous batch of services to be processed is determined to exist, acquiring the historical service information, wherein the current service information and the historical service information form the service information;
accordingly, the set of messages includes: and the data of the row corresponding to the current batch of the service to be processed and the data of the row corresponding to the last batch of the service to be processed.
4. The method according to claim 3, wherein the determining all the line data corresponding to each service to be processed according to the message type and the identifier of the service to be processed comprises:
if the message type is a starting type and the identifier of the service to be processed is the identifier of the service to be processed in the current batch, creating a new row data set according to the identifier of the service to be processed;
storing the line data with the message type as the starting type into the new line data set;
and storing the line data type and the ending type of the service to be processed with the same identification into the new line data set.
5. The method according to claim 3, wherein the determining all the line data corresponding to each service to be processed according to the message type and the identifier of the service to be processed comprises:
if the message type of the line data is the line data type and the identifier of the service to be processed is the identifier of the service to be processed in the previous batch, acquiring a line data set corresponding to the service to be processed in the previous batch according to the identifier of the service to be processed;
storing the line data of the line data type into a line data set corresponding to the service to be processed in the last batch;
and storing the data of the ending type corresponding to the service to be processed in the last batch into the data set corresponding to the service to be processed in the last batch.
6. The method according to any one of claims 3-5, further comprising:
and if the fact that the row data set corresponding to the current batch of the to-be-processed services does not comprise the row data of the ending type is determined, the to-be-processed services are subjected to retention operation, and all the corresponding row data are determined together with the to-be-processed services with the same identification in the next batch.
7. The method according to any one of claims 1 to 5, wherein the service to be processed is a service to be processed, and the performing data processing operation on all the line data comprises:
and storing all the row data corresponding to each service to be processed into a database so that the back-end server acquires all the row data corresponding to the same service to be processed.
8. A log processing apparatus, comprising:
the system comprises an acquisition module, a processing module and a processing module, wherein the acquisition module is used for acquiring service information corresponding to at least one service to be processed, and the service information comprises all log information corresponding to the service to be processed and an identifier of the service to be processed;
the analysis module is used for analyzing all the log information to obtain a message set corresponding to all the log information, wherein the message set comprises at least one row of data;
the determining module is used for traversing the message set and determining the message types corresponding to the data of each row;
and the processing module is used for determining all the line data corresponding to each service to be processed according to the message type and the identification of the service to be processed and carrying out data processing operation on all the line data.
9. A log processing apparatus characterized by comprising: a memory, a processor;
a memory; a memory for storing the processor-executable instructions;
wherein the processor is configured to perform the log processing method of any one of claims 1-7 by the processor.
10. A computer-readable storage medium having stored therein computer-executable instructions for implementing the log processing method of any one of claims 1 to 7 when executed by a processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010475917.2A CN111651481B (en) | 2020-05-29 | 2020-05-29 | Log processing method, device, equipment and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010475917.2A CN111651481B (en) | 2020-05-29 | 2020-05-29 | Log processing method, device, equipment and computer readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111651481A true CN111651481A (en) | 2020-09-11 |
| CN111651481B CN111651481B (en) | 2023-08-22 |
Family
ID=72344838
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010475917.2A Active CN111651481B (en) | 2020-05-29 | 2020-05-29 | Log processing method, device, equipment and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111651481B (en) |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104580310A (en) * | 2013-10-21 | 2015-04-29 | 腾讯科技（深圳）有限公司 | Log processing method and server |
| US20170185576A1 (en) * | 2015-12-28 | 2017-06-29 | International Business Machines Corporation | Categorizing Log Records at Run-Time |
| US20180089304A1 (en) * | 2016-09-29 | 2018-03-29 | Hewlett Packard Enterprise Development Lp | Generating parsing rules for log messages |
| US20190146754A1 (en) * | 2017-11-15 | 2019-05-16 | Accenture Global Solutions Limited | Optimized construction of a sample imprint for selecting a sample dataset for comparison testing |
| CN110134648A (en) * | 2019-05-22 | 2019-08-16 | 中国联合网络通信集团有限公司 | Log processing method, device, equipment, system and computer readable storage medium |
| CN110297846A (en) * | 2019-05-28 | 2019-10-01 | 北京奇艺世纪科技有限公司 | A kind of log feature processing system, method, electronic equipment and storage medium |
| CN110489433A (en) * | 2019-07-11 | 2019-11-22 | 中国平安人寿保险股份有限公司 | A kind of message treatment method and device, electronic equipment based on MongoDB |
| CN111010290A (en) * | 2019-11-22 | 2020-04-14 | 泰康保险集团股份有限公司 | Service log processing method and device, electronic equipment and storage medium |
| CN111144697A (en) * | 2019-11-29 | 2020-05-12 | 泰康保险集团股份有限公司 | Data processing method, data processing device, storage medium and electronic equipment |
-
2020
- 2020-05-29 CN CN202010475917.2A patent/CN111651481B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104580310A (en) * | 2013-10-21 | 2015-04-29 | 腾讯科技（深圳）有限公司 | Log processing method and server |
| US20170185576A1 (en) * | 2015-12-28 | 2017-06-29 | International Business Machines Corporation | Categorizing Log Records at Run-Time |
| US20180089304A1 (en) * | 2016-09-29 | 2018-03-29 | Hewlett Packard Enterprise Development Lp | Generating parsing rules for log messages |
| US20190146754A1 (en) * | 2017-11-15 | 2019-05-16 | Accenture Global Solutions Limited | Optimized construction of a sample imprint for selecting a sample dataset for comparison testing |
| CN110134648A (en) * | 2019-05-22 | 2019-08-16 | 中国联合网络通信集团有限公司 | Log processing method, device, equipment, system and computer readable storage medium |
| CN110297846A (en) * | 2019-05-28 | 2019-10-01 | 北京奇艺世纪科技有限公司 | A kind of log feature processing system, method, electronic equipment and storage medium |
| CN110489433A (en) * | 2019-07-11 | 2019-11-22 | 中国平安人寿保险股份有限公司 | A kind of message treatment method and device, electronic equipment based on MongoDB |
| CN111010290A (en) * | 2019-11-22 | 2020-04-14 | 泰康保险集团股份有限公司 | Service log processing method and device, electronic equipment and storage medium |
| CN111144697A (en) * | 2019-11-29 | 2020-05-12 | 泰康保险集团股份有限公司 | Data processing method, data processing device, storage medium and electronic equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111651481B (en) | 2023-08-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107122290B (en) | Method and device for recording log information | |
| CN109559226B (en) | Block chain transaction execution method and system, electronic device and storage medium | |
| CN109614433B (en) | Method, device, equipment and storage medium for identifying data blooding margin between business systems | |
| CN111583000B (en) | Method and device for identifying behavior of surrounding mark and string mark, computer equipment and storage medium | |
| CN112214366A (en) | Test method, device, system, equipment and medium | |
| CN111651481A (en) | Log processing method, device, equipment and computer readable storage medium | |
| CN107016613B (en) | Data modification method and device | |
| CN111625342B (en) | Data tracing method, device and server | |
| CN116645082A (en) | System inspection method, device, equipment and storage medium | |
| CN116431522A (en) | Automatic test method and system for low-code object storage gateway | |
| CN112131180B (en) | Data reporting method, device and storage medium | |
| CN112235363B (en) | Data processing method, device, electronic equipment, storage medium and system | |
| CN112035169B (en) | Jump processing method and device, computer equipment and computer readable storage medium | |
| CN115065510A (en) | Login method, device, system, electronic equipment and readable storage medium | |
| CN113052688A (en) | Credit card handling method and device based on block chain | |
| CN114071450B (en) | Recognition method and device for card changing behavior of machine changing | |
| CN111488236A (en) | Order abnormity processing method, server, storage medium and processing device | |
| CN113313594B (en) | Order processing method, order processing device, electronic equipment, storage medium and program product | |
| CN115643314B (en) | System alternating information confirmation method and system | |
| CN112116452B (en) | Transaction processing method and device | |
| CN114637657A (en) | Log processing method, device, equipment and computer readable storage medium | |
| CN115277880B (en) | Network message analysis method and device | |
| CN108984285B (en) | Data collision flow analysis method and device, storage medium and terminal | |
| CN115952172A (en) | Data matching method and device based on temporary table of database | |
| CN115712578A (en) | Interface automation test method, test device and test system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |