CN114896222A - Log data processing method and device, computer equipment and system - Google Patents

Log data processing method and device, computer equipment and system Download PDF

Info

Publication number
CN114896222A
CN114896222A CN202210635332.1A CN202210635332A CN114896222A CN 114896222 A CN114896222 A CN 114896222A CN 202210635332 A CN202210635332 A CN 202210635332A CN 114896222 A CN114896222 A CN 114896222A
Authority
CN
China
Prior art keywords
target
log data
log
data processing
application service
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210635332.1A
Other languages
Chinese (zh)
Inventor
武文轩
王豪赞
周文泽
程鹏
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Industrial and Commercial Bank of China Ltd ICBC
Original Assignee
Industrial and Commercial Bank of China Ltd ICBC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Industrial and Commercial Bank of China Ltd ICBC filed Critical Industrial and Commercial Bank of China Ltd ICBC
Priority to CN202210635332.1A priority Critical patent/CN114896222A/en
Publication of CN114896222A publication Critical patent/CN114896222A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/18—File system types
    • G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
    • G06F16/1815—Journaling file systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/245—Query processing
    • G06F16/2455—Query execution
    • G06F16/24553—Query execution of query operations
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
    • G06F9/5011—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
    • G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
    • G06F9/505—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the load
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Computational Linguistics (AREA)
  • Debugging And Monitoring (AREA)

Abstract

The application provides a log data processing method, a log data processing device, computer equipment, a log data processing system, a computer readable storage medium and a computer program product, and relates to the technical field of cloud computing. The log data processing method is applied to a log processing cloud server in a log data processing system in the log data processing system, and the log data processing system further comprises an application server for running application services. After the application server runs the application service to generate original data, the application server sends the acquired original data to the log processing cloud server. After the log processing cloud server obtains original data sent by a target application server, a target log data processing strategy corresponding to a target application service is inquired according to a corresponding relation between a pre-stored and configured application service and the log data processing strategy, and the original data are converted into log data according to the target log data processing strategy.

Description

Log data processing method and device, computer equipment and system
Technical Field
The application relates to the technical field of cloud computing, in particular to a log data processing method, a log data processing device, computer equipment and a log data processing system.
Background
In the application of the Internet, log data occupies an important position, and troubleshooting, data analysis, intranet safety monitoring and the like all depend on the log data. In different application scenarios, requirements, formats, and contents of log data may be different, and generally, the acquired raw data needs to be processed as required to obtain corresponding log data.
In the related art, when an application service is deployed on a server, a log data processing service is deployed at the same time, and after the application server runs original data generated by the application service, the log data processing service on the application server converts the original data into corresponding log data according to configuration and then stores the corresponding log data into a corresponding database. The original data is unprocessed data generated when the application server runs the application service, and corresponds to log data in an expected form, and the original data is data in an unexpected form.
However, the resources of the application server are limited, and the log data processing service occupies part of the resources of the application service, so that the resources of the application service are limited and the service quality is reduced.
Disclosure of Invention
In view of the foregoing, it is desirable to provide a log data processing method, apparatus, computer device, computer readable storage medium, and computer program product capable of guaranteeing application service resources and service quality.
In a first aspect, the present application provides a log data processing method. The method is applied to a log processing cloud server in a log data processing system, and the log data processing system also comprises an application server for running application services; the method comprises the following steps:
acquiring original data generated by a target application server running a target application service;
inquiring a target log data processing strategy corresponding to the target application service according to a pre-stored corresponding relation between the application service and the log data processing strategy;
and converting the original data into log data according to the target log data processing strategy.
In one embodiment, the obtaining raw data generated by the target application server running the target application service includes:
receiving compressed data sent by a target application server; the compressed data is obtained by compressing original data generated by running a target application service by the target application server;
and decompressing the compressed data to obtain the original data generated by the target server running the target application service.
In one embodiment, the converting the raw data into log data according to the log data processing policy includes:
determining a target conversion type corresponding to the original data according to the log data processing strategy;
determining a plurality of process groups corresponding to the target conversion type in a pre-established process group, and selecting a target process group with a working state as a load unsaturated state from the plurality of process groups;
and calling the target process group to convert the original data into log data.
In one embodiment, the method further comprises:
inquiring the working states of a plurality of process groups corresponding to the target conversion type;
and under the condition that the working states of the process groups corresponding to the target conversion type are all in a load supersaturation state, creating the process group with the conversion type as the target conversion type.
In one embodiment, the method further comprises:
and under the condition that the working states of the process groups corresponding to the target conversion type are all in a load supersaturation state, determining the application service corresponding to the target conversion type, and sending a current limiting instruction to the server where the application service is located, wherein the current limiting instruction is used for indicating the server where the application service is located to reduce the rate of sending original data.
In one embodiment, the method further comprises:
and deleting at least one process group in the load unsaturated state in the plurality of process groups corresponding to the target conversion type under the condition that the number of the process groups in which the working state is the load unsaturated state reaches a preset threshold value.
In a second aspect, the application also provides a log data processing system. The log data processing system at least comprises an application server for running application service and a log processing cloud server; the method comprises the following steps:
the application server is used for acquiring original data generated by running an application service and sending the original data to the log processing cloud server;
the log processing cloud server is used for acquiring original data generated when the application server operates the application service; inquiring a target log data processing strategy corresponding to the target application service according to a corresponding relation between the pre-stored and configured application service and the log data processing strategy; and converting the original data into log data according to the target log data processing strategy.
In a third aspect, the present application further provides a log data processing apparatus. The device is applied to a log processing cloud server in a log data processing system, and the log data processing system also comprises an application server for running application services; the conversion device includes:
the acquisition module is used for acquiring original data generated when the target application server operates the target application service;
the query module is used for querying a target log data processing strategy corresponding to the target application service according to the corresponding relation between the pre-stored application service and the log data processing strategy;
and the conversion module is used for converting the original data into log data according to the target log data processing strategy.
In one embodiment, the obtaining module includes a receiving unit and a decompressing unit, wherein:
the receiving unit is used for receiving compressed data sent by a target application server; the compressed data is obtained by compressing original data generated by running a target application service by the target application server;
and the decompression unit is used for decompressing the compressed data to obtain the original data generated by the target server running the target application service.
In one embodiment, the conversion module includes a determination unit, a selection unit, and a call unit, wherein:
the determining unit is used for determining a target conversion type corresponding to the original data according to the log data processing strategy;
the selecting unit is configured to determine, in a process group created in advance, a plurality of process groups corresponding to the target transition type, and select, from the plurality of process groups, a target process group whose working state is a load unsaturation state;
and the calling unit is used for calling the target process group to convert the original data into log data.
In one embodiment, the log data processing apparatus further includes:
the state query module is used for querying the working states of a plurality of process groups corresponding to the target conversion type;
and the creating module is used for creating the process group with the conversion type as the target conversion type under the condition that the working states of the process groups corresponding to the target conversion type are all in the load supersaturation state.
In one embodiment, the log data processing apparatus further includes:
the instruction sending module is used for determining the application service corresponding to the target conversion type and sending a current limiting instruction to a server where the application service is located under the condition that the working states of the process groups corresponding to the target conversion type are all in a load oversaturation state; and the current limiting instruction is used for indicating the server where the application service is located to reduce the rate of sending the original data.
In one embodiment, the log data processing apparatus further includes:
and the deleting module is used for deleting at least one process group in the load unsaturated state in the plurality of process groups corresponding to the target conversion type under the condition that the number of the process groups in which the working state is the load unsaturated state reaches a preset threshold value.
In a fourth aspect, the present application further provides a computer device. The computer device comprises a memory storing a computer program and a processor implementing the following steps when executing the computer program:
acquiring original data generated by a target application server running a target application service;
inquiring a target log data processing strategy corresponding to the target application service according to a pre-stored corresponding relation between the application service and the log data processing strategy;
and converting the original data into log data according to the target log data processing strategy.
In a fifth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of:
acquiring original data generated by a target application server running a target application service;
inquiring a target log data processing strategy corresponding to the target application service according to a pre-stored corresponding relation between the application service and the log data processing strategy;
and converting the original data into log data according to the target log data processing strategy.
In a sixth aspect, the present application further provides a computer program product. The computer program product comprising a computer program which when executed by a processor performs the steps of:
acquiring original data generated by a target application server running a target application service;
inquiring a target log data processing strategy corresponding to the target application service according to a pre-stored corresponding relation between the application service and the log data processing strategy;
and converting the original data into log data according to the target log data processing strategy.
The log data processing method is applied to a log processing cloud server in a log data processing system in the log data processing system, and the log data processing system further comprises an application server for running application services. After the application server runs the application service to generate original data, the application server sends the acquired original data to the log processing cloud server. After the log processing cloud server obtains original data sent by a target application server, a target log data processing strategy corresponding to a target application service is inquired according to a corresponding relation between a pre-stored and configured application service and the log data processing strategy, and the original data are converted into log data according to the target log data processing strategy.
According to the log data processing method and device, the computer equipment, the storage medium and the computer program product, the log data processing service is processed in a separated mode, the independent log processing cloud server is set to convert original data into log data, on one hand, occupation of the log data service on application service resources is reduced, and on the other hand, the processing efficiency of the log data is guaranteed.
Drawings
FIG. 1 is a block diagram illustrating a log data processing system according to one embodiment of the present application;
FIG. 2 is a flowchart illustrating a log data processing method according to an embodiment of the present application;
fig. 3 is a schematic diagram illustrating a deployment structure of a log processing cloud server according to an embodiment of the present application;
FIG. 4 is a block diagram illustrating a log data processing apparatus according to an embodiment of the present application;
FIG. 5 is a diagram of an internal structure of a computer device according to one embodiment of the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.
In the related art, the log collection service is generally deployed as an independent service on the application server. The collection, processing, output and the like of the log data are completed on the application server. For example, there are many log collection tools such as fluent and logstack, and the collection processing flow mostly reads raw data and performs corresponding data processing through an analysis and filtering component, and finally sends the data to a designated terminal or a database according to the sending requirement.
Because the log data processing and the application service are synchronously deployed on the application server, the log data processing has a problem that resources on the application server are mutually crowded with the increase of the traffic of the application service. Therefore, in the related art, most schemes are performed through a delay processing mechanism, and when the resources of the application server are sufficient, the related processing of the log data is performed. However, this results in the problem of untimely log data processing, and for some presentations requiring timely log review or monitoring of data, the delay is unacceptably high.
Based on this, the application provides a log data processing method, which is applied to a log processing cloud server in a log data processing system, wherein the log data processing system further comprises an application server for running an application service. After the application server runs the application service to generate original data, the application server sends the acquired original data to the log processing cloud server. After the log processing cloud server obtains original data sent by a target application server, a target log data processing strategy corresponding to a target application service is inquired according to a corresponding relation between a pre-stored and configured application service and the log data processing strategy, and the original data are converted into log data according to the target log data processing strategy.
According to the log data processing method, the log data processing services are processed separately, and the independent log processing cloud server is arranged to convert the original data into the log data, so that on one hand, occupation of the log data services on application service resources is reduced, and on the other hand, the processing efficiency of the log data is guaranteed.
The application provides a log data processing system and also provides a log data processing method, a log processing cloud server applied to the log data processing system, and a device, a computer readable storage medium and a computer program product corresponding to the log processing cloud server.
First, the log data processing system provided in the present application will be described in detail.
Fig. 1 is a schematic structural diagram of a log data processing system shown in the present application. The log processing cloud server is used for running a plurality of application servers of different application services.
The log processing cloud server can be implemented by an independent server or a server cluster consisting of a plurality of servers. The target application server may be implemented by an independent server or a server cluster composed of a plurality of servers, and may be implemented by a cloud server, which is not limited in this application.
In one embodiment, the target application server is used for collecting original data generated by running the target application service and sending the original data to the log processing cloud server.
In one embodiment, the log processing cloud server is configured to acquire original data generated when the target application server operates the target application service, query a target log data processing policy corresponding to the target application service according to a correspondence between a pre-stored configured application service and the log data processing policy, and finally convert the acquired original data into log data according to the queried target log data processing policy.
It should be noted that the application environment of the log data processing method of the present application is generally a cloud service environment, and after a cloud provider deploys a log processing cloud server on the cloud, the cloud provider performs relevant configuration on the log processing cloud server on the cloud according to the needs of a user, that is, configures a corresponding log data processing policy for original data corresponding to different application services on the log processing cloud server and stores the log data processing policy in the log processing cloud server.
In one embodiment, as shown in fig. 2, there is provided a log data processing method, which is applied to a log processing cloud server in the log data processing system shown in fig. 1, and includes the following steps:
step 201, obtaining original data generated by the target application server running the target application service.
In different application scenarios, formats, contents and the like of required log data are different, original data are unprocessed data generated when a target application server operates a target application service and correspond to log data in an expected form, the original data are data in an unexpected form, for example, in an e-commerce application scenario, the original data are each transaction flow, and after the transaction flow needs to be analyzed, decoded and the like, corresponding transaction logs are obtained, and in a social application scenario, the original data are operation flows of different users, and the operation flows need to be analyzed, decoded and the like, so that operation logs of different users are obtained.
In one embodiment, a collector for the target application service may be deployed at the target application server, and is used for collecting raw data generated when the target application server runs the target application service, and then sending the raw data to the log processing cloud server.
In an embodiment, a collector for the target application service may be deployed at the target application server, and is used to collect original data generated when the target application server operates the target application service, and then store the original data to the local, the log processing cloud server periodically sends a request for obtaining the original data generated when the target application server operates the target application service to the target application server, and after receiving the request, the target application server sends the original data stored in the local and generated when the target application server operates the target application service to the log processing cloud server.
Step 203, inquiring a target log data processing strategy corresponding to the target application service according to the corresponding relationship between the pre-stored application service and the log data processing strategy.
In different application scenarios, the content and the form of the original data are different, and the format and the content of the log data obtained by processing are also different, so that the processing process of converting the original data into the log data is different.
In one embodiment, after the log processing cloud server configures corresponding log data processing strategies for original data corresponding to different application services and stores the log data processing strategies into the log processing cloud server, the log processing cloud server queries a target log data processing strategy corresponding to a target application service according to a corresponding relationship between the stored application service and the log data processing strategies after receiving the original data corresponding to the target application service. For example, in the log processing cloud server, the corresponding log data processing policy configured for the application Service a is Strategy a, the corresponding log data processing policy configured for the application Service B is Strategy B, the corresponding log data processing policy configured for the application Service C is Strategy C, and when the log processing cloud server receives the original data corresponding to the Service B, that is, the target application Service is Service B, the target log data processing policy is queried and obtained as Strategy B.
The log data processing strategy comprises an execution process of converting original data into log data, and the execution process comprises the processes of analysis, translation, transcoding, filtering, sending and the like, wherein the analysis is generally the analysis process of the original data, the translation and the transcoding are generally the conversion process of converting the analyzed data into fixed format data, the filtering is generally the filtering process of the processed log data, and the sending is generally the sending of the converted log data to a fixed terminal or a fixed database. In the related art, a detailed description of a log data processing strategy for obtaining corresponding log data by processing original data can be referred to the description of the log data processing strategy in the related art.
Step 205, converting the original data into log data according to the target log data processing policy.
In one embodiment, after the log processing cloud server determines a target log data processing strategy of the raw data, the raw data is converted into the log data according to the log data processing strategy. For example, the original data is a plurality of files, and corresponding content needs to be read from the files, analyzed, and converted into corresponding log data according to the format and content requirements of the log data. For example, each operation process of a user is stored in a file, the log processing cloud server needs to read the content in the file, determine the position, content and the like of each operation process in the file, and then form a corresponding operation log according to the format requirement of operation log data.
In one embodiment, the log processing cloud server may create a process group in advance, and when original data generated by running a target application service and sent by a target application server is received, the process group is called to process the received original data to obtain log data corresponding to the original data.
In one embodiment, after the log processing cloud server converts the original data into the log data, the original data can be stored in a database connected with the log processing cloud server for being queried and checked by relevant personnel, and the original data can be sent to a database connected with an application server, so that the relevant personnel can directly access the database connected with the application server to query the corresponding log data.
In the embodiment, all the original data generated by the target application service is sent to the log processing cloud server, the log processing cloud server completes conversion from the original data to the log data, occupation of the log data processing service on target application service resources is reduced, and conversion efficiency from the original data to the log data is ensured when the application service is busy, so that related personnel can acquire the log data obtained by conversion from the original data in time.
In one embodiment, step 201 specifically includes:
step 201A, receiving compressed data sent by a target application server.
And the compressed data is obtained by compressing original data generated by running the target application service by the target application server.
In one embodiment, a collector for collecting raw data may be deployed at a target application server. The target application server runs the target application service to generate original data, an acquisition unit deployed in the target application server acquires the original data, the acquired original data are compressed after a preset time length, compressed data are obtained, and the compressed data are sent to the log processing cloud server.
When compressing the original data, a compression algorithm in the related art may be used, such as a commonly used compression algorithm corresponding to zip, rar, and the like, which is not limited in this application.
Step 201B, decompressing the compressed data to obtain the original data generated by the target server running the target application service.
In one embodiment, after receiving compressed data sent by a target application server, a log processing cloud server determines a decompression algorithm corresponding to the compressed data, for example, reads a flag bit of the compression algorithm in the compressed data to determine a corresponding decompression algorithm, and then decompresses the compressed data by using the decompression algorithm to obtain original data generated when the target application server operates a target application service.
The decompression process corresponds to the compression process, that is, the decompression algorithm corresponds to the compression algorithm, and the decompression algorithm corresponding to the compression algorithm is required to decompress the compressed data.
In the embodiment, the occupation of the bandwidth resource of the application server is reduced as much as possible, and the influence of the log data processing service on the application service is reduced.
In an embodiment, the log processing cloud server may create process groups of different conversion types in advance, and at least one process group of each conversion type is created, and after receiving raw data generated by running a target application service and sent by a target application server, determine a conversion type of the raw data, and then call a process group of a corresponding conversion type to process the received raw data to obtain log data corresponding to the raw data, so step 205 may specifically include:
step 205A, determining a target conversion type corresponding to the original data according to the log data processing policy.
The conversion type is different from the conversion process of raw data to log data, for example, the process of converting transaction flow into transaction log is different from the process of converting operation flow into operation log, such as the parsing, translating, transcoding, and filtering processes described above, on one hand, converting raw data into log data does not necessarily require all processes to be performed, and according to the difference of raw data and the user requirement, a part of processes is selected to be performed, for example, the conversion process of converting raw data into log data may not include the filtering process, on the other hand, for raw data from different application servers, the parsing, translating, transcoding processes may also be different, for example, raw data sent by some application servers is in a file form, and raw data sent by some application servers is a data stream, not organized in file form.
In one embodiment, after the log processing cloud server determines the log data processing strategy of the original data, the target conversion type corresponding to the original data is determined according to processes which need to be executed in the conversion process from the original data to the log data in the strategy, such as whether analysis, translation, transcoding, filtering and the like are needed, and specific processes of analysis, translation, transcoding and filtering.
Step 205B, in the pre-created process groups, determining a plurality of process groups corresponding to the target conversion type, and selecting a target process group whose working state is a load unsaturation state from the plurality of process groups.
In one embodiment, after the log processing cloud server determines a target conversion type corresponding to original data, a process group with a conversion type of the target conversion type is determined, and if only one process group with the target conversion type exists, the process group is directly called to convert the original data into log data.
In one embodiment, after determining a target conversion type corresponding to original data, a log processing cloud server determines a process group of which the conversion type is the target conversion type, if multiple process groups of which the target conversion type exists, determines working states of the multiple process groups corresponding to the target conversion type, and selects a target process group of which the working state is a load unsaturation state as a process group for converting the original data into log data.
Step 205C, the target process group is called to convert the original data into log data.
In one embodiment, after the log processing cloud server selects a target process group with a working state being a load unsaturation state, the target process group is called to convert original data into log data.
In one embodiment, the log processing cloud server may create a process group for acquiring raw data in advance, the process group being used for receiving the raw data sent by the application server, and after the log processing cloud server determines a target process group for converting the raw data into log data, the process group for acquiring the raw data sends the raw data to the target process group.
In this embodiment, corresponding process groups are configured for different conversion strategies, and the same process group can be reused for the original data of the same log data processing strategy, thereby improving the resource utilization rate.
In one embodiment, a process group includes a plurality of processes, each process is used for executing one execution process of the conversion process of the original data to the target data, for example, a process group includes four processes, which are respectively used for executing the parsing process, the translation process, the transcoding process and the filtering process.
Each process in the process group has an execution sequence, for example, after the execution of the corresponding process is completed, the analyzed data is sent to the process corresponding to the translation, after the execution of the process corresponding to the translation is completed, the translated data is sent to the process corresponding to the transcoding, after the execution of the process corresponding to the transcoding is completed, the transcoded data is sent to the process corresponding to the filtering, after the execution of the process corresponding to the filtering is completed, the final log data is obtained, and the conversion from the original data to the log data is completed.
In one embodiment, the method further comprises:
and step 207, inquiring the working states of a plurality of process groups corresponding to the target conversion types.
In an embodiment, the log processing cloud server may determine, every fixed time interval, a process group of which the conversion type is the target conversion type, and query the working state of each process group corresponding to the target conversion type.
In one embodiment, the log processing cloud server can query the CPU occupancy rate, the memory occupancy rate, the disk read-write rate, and the like of the process group in the query, and determine the working state of the process group according to the CPU occupancy rate, the memory occupancy rate, the disk read-write rate, and the like of the process group.
And 209, under the condition that the working states of the process groups corresponding to the target conversion type are all in the load supersaturation state, creating the process group with the conversion type being the target conversion type.
In one embodiment, the log processing cloud server queries the CPU occupancy, the memory occupancy, the disk read-write rate, and the like of the process group, determines whether the CPU occupancy, the memory occupancy, the disk read-write rate, and the like of the process group reach corresponding thresholds, determines that the working state of the process group is in a load supersaturation state if the CPU occupancy, the memory occupancy, the disk read-write rate, and the like of the process group reach corresponding thresholds, creates a process group of which the conversion type is the target conversion type, and builds load pressure of the process group of other target conversion types.
In this embodiment, when the working states of the process groups all satisfy the overload condition, it indicates that the existing process groups at this time are not enough to process the original data received at this time, and a new process group is created.
In an embodiment, when the working states of the process groups corresponding to the target transition type are all in the load supersaturation state, on one hand, the number of the process groups of the target transition type is increased, and on the other hand, the number of the original data that needs to be processed by the process groups of the target transition type can be reduced, so the method further includes:
and step 211, determining the application service corresponding to the target conversion type under the condition that the working states of the plurality of process groups corresponding to the target conversion type are all in the load oversaturation state, and sending a current limiting instruction to a server where the application service is located.
And the current limiting instruction is used for indicating the server where the application service is located to reduce the rate of sending the original data. The conversion type corresponds to the log data processing strategy one by one, each application service is configured with the corresponding log data processing strategy, and the corresponding relation between the conversion type and the application service is determined according to the corresponding relation between the conversion type and the log data processing strategy and the corresponding relation between the log data processing strategy and the application service, namely the application service corresponding to the target conversion type is determined.
In one embodiment, the log processing cloud server determines the application service corresponding to the target conversion type according to the corresponding relationship between the conversion type and the log data processing policy and the corresponding relationship between the log data processing policy and the application service under the condition that the working state of each process group corresponding to the target conversion type is determined to meet the overload condition, and sends a current limiting instruction to the server where the application service is located. And under the condition that the log processing cloud server determines that the working state of each process group corresponding to the target conversion type is good, namely under the condition that the working state of each process group corresponding to the target conversion type is in a load unsaturated state, determining the application service corresponding to the target conversion type, and sending a current limitation relieving instruction to the server where the application service is located, wherein the current limitation relieving instruction is used for indicating the server where the application service is located to restore the rate of sending the original data.
In one embodiment, after the server where the application service is located receives the current limiting instruction, the collected original data can be stored locally, and the rate of sending the original data is reduced. And after a current limiting relieving instruction sent by the log service is received, the rate of sending the original data is recovered, the original data stored to the local is sent to the log processing cloud server, and the newly acquired original data is directly sent to the log processing cloud server without being stored to the local.
In this embodiment, when the working states of the process groups all satisfy the overload condition, it indicates that the existing process groups are not enough to process the original data received at this time, and the acquisition rate of the original data is reduced.
In one embodiment, the method further comprises:
and step 213, deleting at least one process group in the load unsaturated state in the plurality of process groups corresponding to the target conversion type when the number of the process groups in which the working state is the load unsaturated state reaches a preset threshold value.
For example, when the process group is in the load saturation state (full load state), the CPU occupancy, the memory occupancy, the disk read-write rate, and the like of the process group have corresponding thresholds, and when the process group exceeds the corresponding thresholds, the process group is in the overload state, and satisfies the overload condition, and when the process group is lower than the corresponding thresholds, the process group is in the load unsaturation state.
In one embodiment, the log processing cloud server determines whether the process group is in a load unsaturated state or not by querying the CPU occupancy rate, the memory occupancy rate, the disk read-write rate, and the like of the process group, and if the number of the process groups in which the working states of the process groups corresponding to the target conversion type are in the load unsaturated state reaches a preset threshold, it indicates that the number of the process groups in which the conversion type is the target conversion type is too many at this time, and deletes at least one process group in which the load unsaturated state is in the plurality of process groups corresponding to the target conversion type.
In this embodiment, when the number of the process groups in which the working state of the process group is the load unsaturation state reaches the preset threshold, it indicates that the number of the process groups having the transition type is too large, so that the number of the process groups having the transition type is reduced, and the idle rate of the process groups having the transition type is reduced.
Next, a detailed description will be given of a specific embodiment provided in the present application. Fig. 3 is a schematic diagram illustrating a deployment structure of a log data processing service at a log processing cloud server according to an embodiment of the present application. The system comprises a data acquisition layer, a data scheduling layer, an execution layer, a resource monitoring layer and a control layer.
The data acquisition layer comprises a plurality of collectors, the collectors are deployed in each application server, original data generated when the application servers run the application services are read and compressed, and the compressed original data are sent to the data scheduling layer after compression is completed.
And after receiving the compressed original data sent by the application server, the data scheduling layer decompresses the original data and sends the original data to a corresponding process group according to a log data processing strategy issued by the control layer and aiming at each application service.
The execution layer is composed of a plurality of process instances with different functions and types and used for responding to the request sent by the data scheduling layer and providing corresponding processes aiming at various processing requirements of the original data, such as analysis, translation, transcoding, filtering, sending and the like. The execution layer may also perform capacity expansion or capacity reduction on the process group according to a related instruction issued by the control layer, that is, increase or decrease the process group.
The resource monitoring layer is used for acquiring corresponding resource use condition data, such as CPU use rate, memory occupancy rate, disk read-write rate, network bandwidth, connection quantity and the like of the process group from the execution layer at regular time.
The controller layer is used for configuring the log data processing strategies of the original data corresponding to the application services according to the log data processing requirements of the different application services, and issuing the configuration to the data acquisition layer, so that the data acquisition layer sends the original data to the corresponding process group according to the log data processing strategies issued by the controller and aiming at the application services. According to the corresponding resource use condition data of the execution layer acquired by the resource monitoring layer, whether the capacity expansion or the capacity reduction is needed for each process group of the execution layer is determined, and corresponding instructions are issued to the execution layer when the capacity expansion or the capacity reduction is needed.
The log processing cloud server generally executes the following processes:
1. after the data scheduling layer receives the log data, determining a log data processing strategy corresponding to the received original data, judging whether a process group of a conversion type corresponding to the log data processing strategy exists, if not, feeding back a process group of a start-up corresponding conversion type to the controller, and meanwhile, the controller finds an instruction for suspending data acquisition to an acquisition unit corresponding to the original data, so as to avoid data blockage. And after the process group of the conversion type corresponding to the log data processing strategy is started, issuing an acquisition recovery instruction to the acquisition device.
2. And under the condition that the process group of the conversion type corresponding to the log data processing strategy exists, forwarding the original data to a load balancer corresponding to the process group of the conversion type, and performing further data processing by load balancing.
3. And the load balancer forwards the original data to the process group with the optimal load condition according to the working state of the process group of the conversion type, so that the corresponding process group converts the original data into log data.
4. The controller is communicated with each process group regularly, and when the process group does not respond, the process group is deleted, and a new process group is newly built to replace the deleted process group.
5. The controller determines the working state of each process group of the execution layer according to the corresponding resource use condition data of the execution layer acquired by the resource monitoring layer, determines whether each process group needs capacity expansion or capacity reduction, and issues a corresponding instruction to the execution layer when capacity expansion or capacity reduction is needed.
6. The controller determines the working state of each process group of the execution layer according to the corresponding resource use condition data of the execution layer acquired by the resource monitoring layer, and if the working state is not good, the controller issues an expansion instruction to the execution layer and issues a current limiting instruction to a collector corresponding to the collection layer, so that the original data collection rate is reduced. If the process group in the load unsaturated state is too many, the controller issues an instruction for deleting the process group, namely a capacity reduction instruction to the actuator layer.
It should be understood that, although the steps in the flowcharts related to the embodiments as described above are sequentially displayed as indicated by arrows, the steps are not necessarily performed sequentially as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least a part of the steps in the flowcharts related to the embodiments described above may include multiple steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, and the execution order of the steps or stages is not necessarily sequential, but may be rotated or alternated with other steps or at least a part of the steps or stages in other steps.
The application also provides a log data processing system, wherein the log data processing system at least comprises an application server for running the application service and a log processing cloud server. And the application server is used for acquiring the original data generated by running the application service and sending the acquired original data to the log processing cloud server. The log processing cloud server is used for acquiring original data generated by the application server running the application service, inquiring a target log data processing strategy corresponding to a target application service according to a corresponding relation between the application service and the log data processing strategy which is pre-stored and configured, and converting the original data into log data according to the target log data processing strategy.
The specific limitations and descriptions of the execution steps of the log processing cloud server in the log data processing system provided by the present application may refer to the limitations of the log data processing method executed by the log processing cloud server in the foregoing, and are not described herein again.
Based on the same inventive concept, the embodiment of the present application further provides a log data processing apparatus for implementing the log data processing method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the method, so specific limitations in one or more log data processing device embodiments provided below can refer to the limitations on the log data processing method in the foregoing, and details are not described herein again.
In one embodiment, as shown in fig. 4, there is provided a log data processing apparatus 400 applied to a log processing cloud server in a log data processing system further including an application server running an application service; the conversion device includes:
an obtaining module 401, configured to obtain original data generated when a target application server runs a target application service;
the query module 403 is configured to query a target log data processing policy corresponding to the target application service according to a pre-stored correspondence between the application service and the log data processing policy;
a conversion module 405, configured to convert the original data into log data according to the target log data processing policy.
In one embodiment, the obtaining module 401 includes a receiving unit 4011 (not shown in the figure), a decompressing unit 4013 (not shown in the figure), wherein:
the receiving unit 4011 is configured to receive compressed data sent by a target application server; the compressed data is obtained by compressing original data generated by running a target application service by the target application server;
the decompressing unit 4013 is configured to decompress the compressed data to obtain original data generated when the target server runs the target application service.
In one embodiment, the converting module 405 includes a determining unit 4051 (not shown), a selecting unit 4053 (not shown), and a calling unit 4055 (not shown), wherein:
the determining unit 4051 is configured to determine, according to the log data processing policy, a target conversion type corresponding to the original data;
the selecting unit 4053 is configured to determine, in a process group created in advance, a plurality of process groups corresponding to the target conversion type, and select, from the plurality of process groups, a target process group whose working state is an unsaturated load state;
the invoking unit 4055 is configured to invoke the target process group to convert the original data into log data.
In one embodiment, the log data processing apparatus further includes:
a state query module 407 (not shown in the figure), configured to query the working states of the process groups corresponding to the target conversion type;
a creating module 409 (not shown in the figure) configured to create a process group with a transition type as the target transition type when the working states of the process groups corresponding to the target transition type are all in a load over-saturation state.
In one embodiment, the log data processing apparatus further includes:
an instruction sending module 411 (not shown in the figure), configured to determine, when working states of multiple process groups corresponding to the target conversion type are all in a load supersaturation state, an application service corresponding to the target conversion type, and send a current limiting instruction to a server where the application service is located; and the current limiting instruction is used for indicating the server where the application service is located to reduce the rate of sending the original data.
In one embodiment, the log data processing apparatus further includes:
a deleting module 413 (not shown in the figure), configured to delete at least one process group in the load unsaturation state from the multiple process groups corresponding to the target transition type when the number of process groups in which the working state is the load unsaturation state reaches a preset threshold.
The respective modules in the log data processing apparatus described above may be implemented in whole or in part by software, hardware, and a combination thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.
In one embodiment, a computer device is provided, which may be a server, the internal structure of which may be as shown in fig. 5. The computer device includes a processor, a memory, and a network interface connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The database of the computer device is used for storing log data. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a log data processing method.
Those skilled in the art will appreciate that the architecture shown in fig. 5 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.
In one embodiment, a computer device is further provided, which includes a memory and a processor, the memory stores a computer program, and the processor implements the steps of the above method embodiments when executing the computer program.
In an embodiment, a computer-readable storage medium is provided, on which a computer program is stored which, when being executed by a processor, carries out the steps of the above-mentioned method embodiments.
In an embodiment, a computer program product is provided, comprising a computer program which, when being executed by a processor, carries out the steps of the above-mentioned method embodiments.
It should be noted that, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, presented data, etc.) referred to in the present application are information and data authorized by the user or sufficiently authorized by each party.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, database, or other medium used in the embodiments provided herein may include at least one of non-volatile and volatile memory. The nonvolatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical Memory, high-density embedded nonvolatile Memory, resistive Random Access Memory (ReRAM), Magnetic Random Access Memory (MRAM), Ferroelectric Random Access Memory (FRAM), Phase Change Memory (PCM), graphene Memory, and the like. Volatile Memory can include Random Access Memory (RAM), external cache Memory, and the like. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM), among others. The databases referred to in various embodiments provided herein may include at least one of relational and non-relational databases. The non-relational database may include, but is not limited to, a block chain based distributed database, and the like. The processors referred to in the embodiments provided herein may be general purpose processors, central processing units, graphics processors, digital signal processors, programmable logic devices, quantum computing based data processing logic devices, etc., without limitation.
The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the present application. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present application shall be subject to the appended claims.

Claims (11)

1. The log data processing method is applied to a log processing cloud server in a log data processing system, and the log data processing system further comprises an application server for running application services; the method comprises the following steps:
acquiring original data generated by a target application server running a target application service;
inquiring a target log data processing strategy corresponding to the target application service according to a pre-stored corresponding relation between the application service and the log data processing strategy;
and converting the original data into log data according to the target log data processing strategy.
2. The method of claim 1, wherein obtaining raw data generated by the target application server running the target application service comprises:
receiving compressed data sent by a target application server; the compressed data is obtained by compressing original data generated by running a target application service by the target application server;
and decompressing the compressed data to obtain the original data generated by the target server running the target application service.
3. The method of claim 1, wherein converting the raw data into log data according to the log data processing policy comprises:
determining a target conversion type corresponding to the original data according to the log data processing strategy;
determining a plurality of process groups corresponding to the target conversion type in a pre-established process group, and selecting a target process group with a working state as a load unsaturated state from the plurality of process groups;
and calling the target process group to convert the original data into log data.
4. The method of claim 3, further comprising:
inquiring the working states of a plurality of process groups corresponding to the target conversion type;
and under the condition that the working states of the process groups corresponding to the target conversion type are all in a load supersaturation state, creating the process group with the conversion type as the target conversion type.
5. The method according to any one of claims 3 or 4, further comprising:
under the condition that the working states of a plurality of process groups corresponding to the target conversion type are all in a load supersaturation state, determining an application service corresponding to the target conversion type, and sending a current limiting instruction to a server where the application service is located; and the current limiting instruction is used for indicating the server where the application service is located to reduce the rate of sending the original data.
6. The method of claim 4, further comprising:
and deleting at least one process group in the load unsaturated state in the plurality of process groups corresponding to the target conversion type under the condition that the number of the process groups in which the working state is the load unsaturated state reaches a preset threshold value.
7. The log data processing system is characterized by at least comprising an application server for running an application service and a log processing cloud server; the method comprises the following steps:
the application server is used for acquiring original data generated by running an application service and sending the original data to the log processing cloud server;
the log processing cloud server is used for acquiring original data generated when the application server operates the application service; inquiring a target log data processing strategy corresponding to the target application service according to a corresponding relation between the pre-stored and configured application service and the log data processing strategy; and converting the original data into log data according to the target log data processing strategy.
8. The log data processing device is applied to a log processing cloud server in a log data processing system, and the log data processing system further comprises an application server for running an application service; the conversion device includes:
the acquisition module is used for acquiring original data generated when the target application server operates the target application service;
the query module is used for querying a target log data processing strategy corresponding to the target application service according to the corresponding relation between the pre-stored application service and the log data processing strategy;
and the conversion module is used for converting the original data into log data according to the target log data processing strategy.
9. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor realizes the steps of the method of any one of claims 1 to 6 when executing the computer program.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 6.
11. A computer program product comprising a computer program, characterized in that the computer program realizes the steps of the method of any one of claims 1 to 6 when executed by a processor.
CN202210635332.1A 2022-06-07 2022-06-07 Log data processing method and device, computer equipment and system Pending CN114896222A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210635332.1A CN114896222A (en) 2022-06-07 2022-06-07 Log data processing method and device, computer equipment and system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210635332.1A CN114896222A (en) 2022-06-07 2022-06-07 Log data processing method and device, computer equipment and system

Publications (1)

Publication Number Publication Date
CN114896222A true CN114896222A (en) 2022-08-12

Family

ID=82727600

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210635332.1A Pending CN114896222A (en) 2022-06-07 2022-06-07 Log data processing method and device, computer equipment and system

Country Status (1)

Country Link
CN (1) CN114896222A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115757041A (en) * 2022-11-28 2023-03-07 安超云软件有限公司 Dynamically configurable multi-cluster log collection method and application

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115757041A (en) * 2022-11-28 2023-03-07 安超云软件有限公司 Dynamically configurable multi-cluster log collection method and application
CN115757041B (en) * 2022-11-28 2023-11-21 安超云软件有限公司 Method for collecting dynamically configurable multi-cluster logs and application

Similar Documents

Publication Publication Date Title
CN109032521B (en) Storage volume creation method, device, server and storage medium
CN101820426B (en) Data compression method in on-line backup service software
CN102859961B (en) There is the distributed video transcoding system of adaptive file process
US10055134B2 (en) Data compression method and storage system
CN114896222A (en) Log data processing method and device, computer equipment and system
CN113900810A (en) Distributed graph processing method, system and storage medium
WO2009097710A1 (en) Method for organizing and retrieving files, module and system for organizing files and storage media thereof
CN110321364B (en) Transaction data query method, device and terminal of credit card management system
CN115665284A (en) Message processing method and device based on distributed configuration center and computer equipment
CN113407340A (en) Service control system, gateway service method, service request forwarding method and device
CN115905168B (en) Self-adaptive compression method and device based on database, equipment and storage medium
CN112019604A (en) Edge data transmission method and system
CN116760698A (en) Service processing method, device, computer equipment and storage medium
CN114168672B (en) Log data processing method, device, system and medium
US20150269086A1 (en) Storage System and Storage Method
CN113934510A (en) Mirror image processing method and device, electronic equipment and computer readable storage medium
CN111125161B (en) Method, device, equipment and storage medium for processing data in real time
CN113872814A (en) Information processing method, device and system for content distribution network
CN114003602A (en) Power grid monitoring data processing system
CN114020218A (en) Mixed repeating data deleting and scheduling method and system
CN111694705A (en) Monitoring method, device, equipment and computer readable storage medium
CN111274203A (en) Call bill storage system and method
CN116384497B (en) Reading and writing system, related method, device and equipment for quantum computing experimental result
CN112231296B (en) Distributed log processing method, device, system, equipment and medium
CN111143280B (en) Data scheduling method, system, device and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination