Disclosure of Invention
The embodiment of the invention provides a log data processing method and device. The technical problems of low efficiency and low accuracy of log data analysis in the prior art can be solved.
In a first aspect, the present application provides a log data processing method, including:
firstly, acquiring service log data of application operation from an application server, and acquiring interface sorting data from a data platform, wherein the interface sorting data comprises the calling times of each log processing mode in a plurality of log processing modes; then selecting a target log processing mode from the multiple log processing modes according to the calling times of each log processing mode; and finally, processing the service log data by using a target log processing mode, determining the log processing mode by acquiring analysis hotspot requirements counted by a data platform, and preprocessing the log data by adopting the determined log processing mode before analyzing the log data, so that the number of the service log data is reduced, and the log data analysis efficiency and the log analysis accuracy are improved.
In another possible design, the multiple log processing modes at least include retaining fields required in the log analysis stage in the service log data, counting contents matched with preset keywords in the service log data, counting and time-stamping the same contents in the service log data within a preset time range, and establishing indexes of designated fields in the service log data.
In another possible design, the priority of each log processing mode may be determined according to the number of calls of each log processing mode; and selecting the log processing mode with the highest priority from the plurality of log processing modes as the target log processing mode, so that the log processing mode is selected by sensing the analysis hotspot requirement of a data platform, and the real-time performance of log data analysis is met.
In another possible design, after business log data of application operation is acquired from an application server, a read-write operation mode of the business log data can be determined; when the read-write operation mode of the service log data is data write-in operation, executing the operation step of acquiring interface sequencing data from the data platform; and when the read-write operation mode of the service log data is data read operation, the following operation steps are not executed.
In another possible design, the processed service log data is stored in the data platform, so that the local log storage space of the application server is saved, and the problem that the storage and transmission of the local log consume a CPU is solved.
In a second aspect, the present application provides a log data processing apparatus configured to implement the method and the functions performed by the log data processing apparatus in the first aspect, and implemented by hardware/software, where the hardware/software includes units corresponding to the functions.
In a third aspect, the present application provides a log data processing apparatus, including: the processing method comprises a processor, a memory and a communication bus, wherein the communication bus is used for realizing connection communication between the processor and the memory, and the processor executes a program stored in the memory for realizing the steps in the processing method for processing log data provided by the first aspect.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Referring to fig. 1, fig. 1 is a schematic structural diagram of a log processing system according to an embodiment of the present invention. As shown in the figure, the system in the embodiment of the present invention includes an application server, a coprocessor and a data platform, where the application server may include a service module and a log module, the service module is used to execute various service logics, and the log module is used to print log data when the service module executes the service logics; the coprocessor, such as an ARM64(Advanced RISC processors) or an FPGA (Field Programmable Gate Array), may include a log pass-through module, a log preprocessing module, and an interface sorting module, where the interface sorting module is configured to obtain interface sorting data from a data platform and transmit the interface sorting data to the log preprocessing module, the log preprocessing module is configured to select a log processing mode according to the interface sorting data and preprocess the log data using the selected log processing mode, and the log pass-through module is configured to transmit the preprocessed log data to the data platform; the data platform can comprise a log storage module and a log analysis module, wherein the log storage module is used for storing the preprocessed service log data, and the log analysis module is used for counting the call data of the log processing mode used by the coprocessor.
Referring to fig. 2, fig. 2 is a flowchart illustrating a log processing method according to a first embodiment of the present invention. As shown in the figure, the method in the embodiment of the present invention includes:
s201, acquiring service log data of application operation from an application server, and acquiring interface sequencing data from a data platform, wherein the interface sequencing data comprises the calling times of each log processing mode in a plurality of log processing modes.
In the specific implementation, the interface sequencing data is the calling times of a log processing mode adopted by all processors associated with the data platform to process the service log data, after the application server receives a service request, the application server runs various applications and prints the service log data, and outputs the service log data to the coprocessor; the data platform stores a large number of times of calling of a log processing mode adopted by the processor for processing the service log data, and after the coprocessor acquires the service log data transmitted by the application server, the statistical data can be acquired from the data platform.
Optionally, after acquiring service log data of application operation from the application server, a read-write operation mode of the service log data may be determined; when the read-write operation mode of the service log data is data write-in operation, executing the operation step of acquiring interface sequencing data from the data platform; and when the read-write operation mode of the service log data is data read operation, the following operation steps are not executed.
S202, selecting a target log processing mode from the multiple log processing modes according to the calling times of each log processing mode.
In a specific implementation, the priority of each log processing mode can be determined according to the calling times of each log processing mode; and selecting the log processing mode with the highest priority from the plurality of log processing modes as the target log processing mode. Furthermore, if the number of times of calling a certain log processing mode is the largest, the priority of the log processing mode can be set to be the highest priority, so that the log processing mode is selected by sensing the analysis hotspot requirement of a data platform, and the real-time performance of log data analysis is met.
Optionally, the priorities of multiple log processing modes may be sorted, and a log processing mode with the top N bits of priorities is selected from the multiple log processing modes, where N is an integer greater than or equal to 1.
S203, the service log data is processed by using the target log processing mode.
In a specific implementation, the multiple log processing modes at least include retaining fields required in a log analysis stage in the service log data, counting contents matched with preset keywords in the service log data, counting and time-stamping the same contents in the service log data within a preset time range, and establishing an index of a designated field in the service log data. At least one log processing mode can be selected, and the selected log processing mode is adopted to preprocess the service log data, so that the output of the log data is reduced, and the indexing speed of the log data is accelerated.
For example: as shown in fig. 1, the log preprocessing module includes a plurality of log processing interfaces (Top1 interface, Top2 interface, Top3 interface, … …), each of which uses different log processing modes to process the service log data, and may select a Top1 interface, a Top2 interface, and a Top3 interface to process the service log data at the same time, and obtain a result after each of the log processing interfaces processes the log service data, and finally transmit the result to all data platforms.
The method can be applied to a user behavior analysis system, a security intrusion analysis system, a traffic flow prediction system and a shopping recommendation system, and the service log data output by the user behavior analysis system, the security intrusion analysis system, the traffic flow prediction system and the shopping recommendation system are preprocessed by the method, so that the log output quantity and the data quantity are reduced, the performance loss of log dotting of an application server is reduced, the log storage quantity is reduced, and the data analysis performance is accelerated.
Referring to fig. 3, fig. 3 is a flowchart illustrating a log processing method according to a second embodiment of the present invention. As shown in the figure, the method in the embodiment of the present invention includes:
s301, the application server sends the service log data of the application operation to the coprocessor. The same steps as those in the above embodiments are performed, and the description of this embodiment is omitted.
S302, the data platform sends interface sequencing data to the co-processing. The same steps as in the previous embodiment are performed and the present embodiment is no longer cumbersome.
And S303, the coprocessor selects a target log processing mode from the multiple log processing modes according to the calling times of each log processing mode. The same steps as those in the above embodiments are performed, and the description of this embodiment is omitted.
S304, the coprocessor uses the target log processing mode to process the service log data. The same steps as those in the above embodiments are performed, and the description of this embodiment is omitted.
S305, the coprocessor stores the processed service log data to the data platform, so that the local log storage space of the application server is saved, and the problem that the local log storage and transmission consume a CPU is solved.
In the embodiment of the invention, firstly, business log data of application operation is obtained from an application server, interface sequencing data is obtained from a data platform, and then a target log processing mode is selected from a plurality of log processing modes according to the calling times of each log processing mode; and finally, processing the service log data by using a target log processing mode, determining the log processing mode by acquiring analysis hotspot requirements counted by a data platform, and preprocessing the log data by adopting the determined log processing mode before analyzing the log data, so that the number of the service log data is reduced, and the log data analysis efficiency and the log analysis accuracy are improved.
As shown in fig. 4, fig. 4 is a schematic structural diagram of a log data processing apparatus according to an embodiment of the present invention. As shown in the figures, the apparatus in the embodiment of the present invention includes:
the data obtaining module 401 is configured to obtain service log data of application operation from an application server, and obtain interface sorting data from a data platform, where the interface sorting data includes the number of times of calling each log processing mode in multiple log processing modes.
In the specific implementation, the interface sequencing data is the calling times of a log processing mode adopted by all processors associated with the data platform to process the service log data, after the application server receives a service request, the application server runs various applications and prints the service log data, and outputs the service log data to the coprocessor; the data platform stores a large number of times of calling of a log processing mode adopted by the processor for processing the service log data, and after the coprocessor acquires the service log data transmitted by the application server, the statistical data can be acquired from the data platform.
Optionally, after acquiring service log data of application operation from the application server, a read-write operation mode of the service log data may be determined; and when the read-write operation mode of the service log data is data write-in operation, executing the operation step of acquiring interface sequencing data from the data platform.
A mode selecting module 402, configured to select a target log processing mode from the multiple log processing modes according to the number of calls of each log processing mode.
In a specific implementation, the priority of each log processing mode can be determined according to the calling times of each log processing mode; and selecting the log processing mode with the highest priority from the plurality of log processing modes as the target log processing mode. Furthermore, if the number of times of calling a certain log processing mode is the largest, the priority of the log processing mode can be set to be the highest priority, so that the log processing mode is selected by sensing the analysis hotspot requirement of a data platform, and the real-time performance of log data analysis is met.
Optionally, the priorities of multiple log processing modes may be sorted, and a log processing mode with the top N bits of priorities is selected from the multiple log processing modes, where N is an integer greater than or equal to 1.
A data processing module 403, configured to process the service log data by using the target log processing manner.
In a specific implementation, the multiple log processing modes at least include retaining fields required in a log analysis stage in the service log data, counting contents matched with preset keywords in the service log data, counting and time-stamping the same contents in the service log data within a preset time range, and establishing an index of a designated field in the service log data. At least one log processing mode can be selected, and the selected log processing mode is adopted to preprocess the service log data, so that the output of the log data is reduced, and the indexing speed of the log data is accelerated.
For example: as shown in fig. 1, the log preprocessing module includes a plurality of log processing interfaces (Top1 interface, Top2 interface, Top3 interface, … …), each of which uses different log processing modes to process the service log data, and may select a Top1 interface, a Top2 interface, and a Top3 interface to process the service log data at the same time, and obtain a result after each of the log processing interfaces processes the log service data, and finally transmit the result to all data platforms.
Optionally, as shown in fig. 4, the apparatus in the embodiment of the present invention may further include:
and a data sending module 404, configured to store the processed service log data in the data platform, so as to save a local log storage space of the application server, and solve a problem that the storage and transmission of the local log consume a CPU.
In the embodiment of the invention, firstly, business log data of application operation is obtained from an application server, interface sequencing data is obtained from a data platform, and then a target log processing mode is selected from a plurality of log processing modes according to the calling times of each log processing mode; and finally, processing the service log data by using a target log processing mode, determining the log processing mode by acquiring analysis hotspot requirements counted by a data platform, and preprocessing the log data by adopting the determined log processing mode before analyzing the log data, so that the number of the service log data is reduced, and the log data analysis efficiency and the log analysis accuracy are improved.
Referring to fig. 5, fig. 5 is a schematic structural diagram of a log data processing apparatus according to the present invention. As shown, the apparatus may include: at least one processor 501, such as a CPU, at least one communication interface 502, at least one memory 503, and at least one communication bus 504. Wherein a communication bus 504 is used to enable the connection communication between these components. The communication interface 502 of the device in the embodiment of the present invention is used for performing signaling or data communication with other node devices. The memory 503 may be a high-speed RAM memory or a non-volatile memory (non-volatile memory), such as at least one disk memory. The memory 503 may optionally be at least one storage device located remotely from the processor 501. A set of program codes is stored in the memory 503 and the processor 501 executes the program stored in the memory 503, performs the method performed by the log data device or implements the function implemented by the log data device.
It should be noted that, for simplicity of description, the above-mentioned embodiments of the method are described as a series of acts or combinations, but those skilled in the art will recognize that the present invention is not limited by the order of acts, as some steps may occur in other orders or concurrently in accordance with the invention. Further, those skilled in the art should also appreciate that the embodiments described in the specification are preferred embodiments and that the acts and modules referred to are not necessarily required by the invention.
In the above embodiments, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.
Those skilled in the art will appreciate that all or part of the steps in the methods of the above embodiments may be implemented by associated hardware instructed by a program, which may be stored in a computer-readable storage medium, and the storage medium may include: flash Memory disks, Read-Only memories (ROMs), Random Access Memories (RAMs), magnetic or optical disks, and the like.
The content downloading method, the related device and the system provided by the embodiment of the present invention are described in detail above, and a specific example is applied in the text to explain the principle and the embodiment of the present invention, and the description of the above embodiment is only used to help understanding the method and the core idea of the present invention; meanwhile, for a person skilled in the art, according to the idea of the present invention, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present invention.