CN114817192A - Log processing method, device, equipment and storage medium - Google Patents

Log processing method, device, equipment and storage medium Download PDF

Info

Publication number
CN114817192A
CN114817192A CN202210537517.9A CN202210537517A CN114817192A CN 114817192 A CN114817192 A CN 114817192A CN 202210537517 A CN202210537517 A CN 202210537517A CN 114817192 A CN114817192 A CN 114817192A
Authority
CN
China
Prior art keywords
log
sending
running
database cluster
address
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210537517.9A
Other languages
Chinese (zh)
Inventor
冯洋
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Ping An Technology Shenzhen Co Ltd
Original Assignee
Ping An Technology Shenzhen Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Ping An Technology Shenzhen Co Ltd filed Critical Ping An Technology Shenzhen Co Ltd
Priority to CN202210537517.9A priority Critical patent/CN114817192A/en
Publication of CN114817192A publication Critical patent/CN114817192A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/18—File system types
    • G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
    • G06F16/1815—Journaling file systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/14—Error detection or correction of the data by redundancy in operation
    • G06F11/1402—Saving, restoring, recovering or retrying
    • G06F11/1446—Point-in-time backing up or restoration of persistent data
    • G06F11/1448—Management of the data involved in backup or backup restore
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
    • G06F9/45533—Hypervisors; Virtual machine monitors
    • G06F9/45558—Hypervisor-specific management and integration aspects

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • Databases & Information Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Quality & Reliability (AREA)
  • Computing Systems (AREA)
  • Debugging And Monitoring (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The invention relates to the technical field of computers, and discloses a log processing method, a device, equipment and a storage medium, wherein the method comprises the following steps: collecting the running logs of the target container through a log collection component running in the target container; acquiring a sending mode and a sending address of the running log from a log acquisition component; the sending address is an address in a preset database cluster; the database cluster comprises a plurality of database servers; a database cluster is preset with a designated data persistence mode; and sending the running log to a sending address through a sending mode so as to store the running log through the database cluster and process the running log based on a data persistence mode. In the mode, a sufficient storage environment is provided for running the log, the log data are rapidly stored and updated, the resource consumption of the log acquisition component is reduced, and the acquisition efficiency of the log acquisition component is greatly improved.

Description

Log processing method, device, equipment and storage medium
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a log processing method, apparatus, device, and storage medium.
Background
In the K8S container cluster, a machine runs a large number of containers, each container corresponds to a registry file and is used for recording the processing progress of a container log by a log acquisition component; in the operation process of the container, the log collection component needs to frequently update a large number of registry files, and the efficiency of log collection processing is reduced.
Disclosure of Invention
The invention provides a log processing method, a log processing device, log processing equipment and a log processing storage medium, which are used for storing and updating information of log acquisition progress and improving acquisition processing efficiency of a log acquisition assembly.
To achieve the above object, a first aspect of the present invention provides a log processing method applied to a device running a plurality of containers, the method including: collecting the running logs of the target container through a log collection component running in the target container; acquiring a sending mode and a sending address of the running log from the log acquisition component; the sending address is an address in a preset database cluster; the database cluster comprises a plurality of database servers; a database cluster is preset with a designated data persistence mode; and sending the running log to a sending address through a sending mode so as to store the running log through the database cluster and process the running log based on a data persistence mode.
Optionally, in a first implementation manner of the first aspect of the present invention, before the step of acquiring the operation log of the target container by using the log acquisition component operating in the target container, the method further includes: acquiring a configuration file of a log acquisition component; deleting default configuration in the configuration file, and setting a sending mode and a sending address of the target log in the configuration file; wherein the default configuration comprises: recording the log processing progress of a target container in the equipment through a preset registry file; and setting a directory of a target file which needs to be subjected to log processing in the configuration file.
Optionally, in a second implementation manner of the first aspect of the present invention, the step of collecting the operation log of the target container by using the log collection component operating in the target container includes: starting a log acquisition process in the target container, and calling a log acquisition component through the log acquisition process; scanning a directory of a target file through a log acquisition component; if the target file is monitored to be updated, collecting log information of the target file; wherein the log information includes: node information, file directory, inode, and offset information of the target file where the update occurred.
Optionally, in a third implementation manner of the first aspect of the present invention, the step of obtaining a sending manner and a sending address of the operation log from the log collecting component includes: acquiring a sending port of the running log and an appointed IP address in the database cluster from a configuration file of the log acquisition component; wherein the transmit port comprises a 6379 port.
Optionally, in a fourth implementation manner of the first aspect of the present invention, after the step of sending the operation log to the sending address by the sending means, the method further includes: and if receiving the information that the running log returned by the database cluster is successfully sent, sending an updating record of the running log to the database cluster so as to save the updating record through the database cluster and record the log processing progress of the target container.
Optionally, in a fifth implementation manner of the first aspect of the present invention, the data persistence manner includes a snapshot manner; the step of sending the operation log to the sending address by the sending mode so as to save the operation log by the database cluster and process the operation log based on the data persistence mode includes: and sending the running log to a sending address by a sending mode so as to generate snapshot information of the running log through the database cluster and store the snapshot information into a preset backup file.
Optionally, in a sixth implementation manner of the first aspect of the present invention, after the step of sending the operation log to the sending address by the sending means, the method further includes: generating a log search request based on a preset log search condition; sending the log search request to a database cluster so as to return a target log meeting the log search condition through the database cluster; and receiving a target log returned by the database cluster.
The second aspect of the present invention provides a log processing apparatus, which is provided in a device running a plurality of containers; the log processing method comprises the following steps: the first acquisition module is used for acquiring the running logs of the target container through the log acquisition component running in the target container; the first acquisition module is used for acquiring the sending mode and the sending address of the running log from the log acquisition component; the sending address is an address in a preset database cluster; the database cluster comprises a plurality of database servers; a database cluster is preset with a designated data persistence mode; and the first processing module is used for sending the running log to the sending address in a sending mode so as to store the running log through the database cluster and process the running log based on a data persistence mode.
A third aspect of the present invention provides a log processing apparatus including: a memory and at least one processor, the memory having instructions stored therein; at least one processor invokes instructions in memory to cause the log processing device to perform the log processing device method as described above.
A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon instructions which, when executed by a processor, implement the log processing apparatus method as described above.
In the technical scheme provided by the invention, the method is applied to equipment operating with a plurality of containers and comprises the following steps: collecting the running logs of the target container through a log collection component running in the target container; acquiring a sending mode and a sending address of the running log from the log acquisition component; the sending address is an address in a preset database cluster; the database cluster comprises a plurality of database servers; a database cluster is preset with a designated data persistence mode; and sending the running log to a sending address through a sending mode so as to store the running log through the database cluster and process the running log based on a data persistence mode. In the method, the operation log containing the container operation progress state is sent to the address of a preset database cluster for storage in a designated sending mode, data persistence is achieved, and the log collection component does not need to perform frequent updating of log files for each container locally. In the mode, sufficient storage space is provided for running the log, the log information is rapidly stored and updated, the resource consumption of the log acquisition assembly is reduced, and the acquisition efficiency of the log acquisition assembly is greatly improved.
Drawings
FIG. 1 is a schematic diagram of an embodiment of a log processing method in an embodiment of the present invention;
FIG. 2 is a schematic diagram of another embodiment of a log processing method in the embodiment of the present invention;
FIG. 3 is a diagram of an embodiment of a log processing apparatus according to the embodiment of the present invention;
fig. 4 is a schematic diagram of an embodiment of a log processing device in the embodiment of the present invention.
Detailed Description
The terms "first," "second," "third," "fourth," and the like in the description and in the claims, as well as in the drawings, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It will be appreciated that the data so used may be interchanged under appropriate circumstances such that the embodiments described herein may be practiced otherwise than as specifically illustrated or described herein. Furthermore, the terms "comprises," "comprising," or "having," and any variations thereof, are intended to cover non-exclusive inclusions, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
Container (Container) is a lightweight, flexible way of virtualization that packages everything needed by an application, including all code, various environment dependent data, and even the operating system, so that the application can run almost anywhere. When the application program is large, a plurality of containers are needed to deploy the application, so that the aims of improving the concurrent access capability and avoiding single point of failure are fulfilled. When a plurality of containers with the same application function as a whole provide services to the outside, the containers are clusters, each container is a node in the cluster, for example, a kubernets (K8S for short) cluster, and K8S is an open source system for automatically deploying, scaling and containerizing management applications. When a program is deployed into a cluster, K8S may intelligently distribute program work to various nodes for unified management of processing containerized applications.
When some operations are performed on a device running a plurality of containers, for example, a device running a K8S cluster, the log file usually records the content of the operations of each container node in real time, and the log file data can provide great help in terms of fast positioning problems, tracking the program running process, and the like. Therefore, log collection management in clusters is becoming increasingly important. When a log collection component in a container is used for collecting logs, for example, open source components such as filebot or flash and the like, in order to ensure a sending state, a registry region is stored on a local disk, so that the component can write updated state information for processing log files into the registry file to ensure that the component can continuously process unprocessed data after restarting, in a k8s cluster container environment, dozens to hundreds of containers are operated on one machine, due to the limitation of disk space, each container can switch logs, a large number of log files are generated every hour, and therefore the log collection component needs to update a large number of registry files frequently, and accordingly, the work progress in the aspect of log collection is slowed down.
Based on the above, the invention provides a log processing method, a log processing device, log processing equipment and a log processing storage medium, which can avoid the situation that the log acquisition processing speed is slowed down due to a large number of frequent updates of the registry files, and improve the log processing efficiency of the log acquisition component.
For convenience of understanding, a detailed flow of an embodiment of the present invention is described below, and referring to fig. 1, a log processing method according to an embodiment of the present invention is applied to an apparatus running a plurality of containers, and an embodiment includes:
101. and collecting the running log of the target container through a log collection component running in the target container.
Specifically, the target container contains the language and framework of all applications when the application is fully run, except for the application itself, so that the application can be deployed in any environment. The target container may be a List container, a Set container, a Map container, etc. and implements the network service by means of port mapping, etc. When the container runs, various log files in the service running process need to be output externally, the log collection assembly serves as an open-source log data processing pipeline, the collection of the log files of the container can be achieved when the log collection assembly runs in the corresponding container, and the running logs of the log files can be obtained through content detection of directories of the log files. The operation log may be understood as the latest log data collected for the operation status of the container, and includes the progress status information of the operation of the container, for example: a file directory of a log file where an update occurs, container node information of a file where an update occurs, and the like.
In this step, the log collection component running in the target container collects the running log generated by the container to obtain the latest log data of the running state of the container, wherein the log collection component may be an open source component such as logstack, filebed, fluent, and the like.
102. Acquiring a sending mode and a sending address of the running log from the log acquisition component; the sending address is an address in a preset database cluster; the database cluster comprises a plurality of database servers; the database cluster is preset with a designated data persistence mode.
The database cluster uses two or more database servers to form a virtual single database logical image to provide transparent data service for the client. Data persistence refers to the persistence of data to storage on a storage medium that can be persisted. When a database cluster can provide certain data persistence functions, for example: a REmote DIctionary Server (REmote DIctionary Server) database cluster and the like, and data can be stored in a storage medium capable of being persistently stored by adopting a persistent mode such as snapshot (snapshot) and AOF (application Only File) File, so as to realize data persistence.
The log collection component comprises a plurality of configuration files, collects and reads log file information based on configuration information in the configuration files and sends the log file information to a designated address for storage. Therefore, the sending mode and the sending address of the running log can be obtained from the configuration file contained in the log collection component of the target container, the running log can be sent according to the specified port sending data, and the sending address is an address in a preset database cluster. The preset database cluster comprises a plurality of database servers, provides services such as loading, packaging and storing of running logs for the log collection assembly, and meanwhile, the database cluster needs to provide certain data persistence functions, such as: redis database cluster, and the like. The log information can be stored in a storage medium capable of being persistently stored based on a preset snapshot (snapshot) and a specified data persistence mode recorded to an AOF file, so that data persistence is realized.
In the step, a sending mode and a sending address of the running log are obtained from a configuration file of the log acquisition component, the sending address of the running log is set as a preset address in a database cluster, and the database cluster is preset with a specified data persistence mode.
103. And sending the running log to a sending address through a sending mode so as to store the running log through the database cluster and process the running log based on a data persistence mode.
That is, the operation log is sent to a sending address according to a sending mode preset by a configuration file in the log collection component, and after the operation log is sent successfully, the operation log can be saved through the database cluster and processed based on a data persistence mode.
In the step, the operation file is sent to a database cluster capable of providing data persistence, database services such as loading, packaging and storing of the operation log are provided for the log collection assembly based on the database cluster, log data are persisted, and information of the operation progress state of the container is rapidly stored and updated. Resource consumption for the log collection component is reduced. The collection efficiency of the log collection assembly is greatly improved.
The log processing method applied to the equipment running with the plurality of containers comprises the following steps: collecting the running logs of the target container through a log collection component running in the target container; acquiring a sending mode and a sending address of the running log from the log acquisition component; the sending address is an address in a preset database cluster; the database cluster comprises a plurality of database servers; a database cluster is preset with a designated data persistence mode; and sending the running log to a sending address through a sending mode so as to store the running log through the database cluster and process the running log based on a data persistence mode. In the method, the log collection component running in the target container is used for collecting the running log of the target container, the progress information of the updated log file is sent to the preset database cluster in a corresponding sending mode to be stored and processed based on the data persistence mode, the problem that the log collection processing speed is slowed down due to frequent updating of a large number of registry files by the log collection component is avoided, and the collection efficiency of the log collection component is greatly improved.
The present solution may also provide an exemplary embodiment, taking a Redis database cluster configured in a K8S cluster as an example, to describe the log processing method in detail. The Redis database cluster is a database cluster formed by Redis databases serving as nodes and comprises a plurality of Redis database servers, wherein the Redis database is the current fastest memory database supporting persistence. Data in the memory can be persisted to a hard disk, the read performance is expanded by copying, and the write performance is expanded by slicing. Referring to fig. 2, an embodiment of the log processing method will be described.
201. Acquiring a configuration file of a log acquisition component; deleting default configuration in the configuration file, and setting a sending mode and a sending address of the target log in the configuration file; wherein the default configuration comprises: recording the log processing progress of a target container in the equipment through a preset registry file; and setting a directory of a target file which needs to be subjected to log processing in the configuration file.
As mentioned above, the K8S cluster is composed of multiple containers, and the log collection component in a container includes multiple configuration files, and generally, the configuration files include default configurations, including: recording the log processing progress of the target container through a preset registry file (such as a registry) in the equipment and setting a directory of the target file needing log processing in a configuration file. Here, the default configuration in the configuration file is deleted, and the transmission manner and the transmission address of the target log are set in the configuration file. Taking a filebolt log collection component as an example, modifying the registry part in the filebolt.yaml of the configuration file of the filebolt, removing the original registry configuration, adding the registry configuration sending address host and the port, modifying the collected directory, and configuring the directory as the directory of the target file needing log processing.
202. Starting a log collection process in a target container, calling a log collection component through the log collection process, scanning a directory of a target file through the log collection component, and collecting log information of the target file if the target file is monitored to be updated; wherein the log information includes: node information, file directory, inode, and offset information of the target file where the update occurred.
The node information refers to container node information in a cluster corresponding to the target file, such as node identification and the like; the file directory refers to data of a corresponding relation between a file name and file attribute information; the index node is used for storing file attribute information, such as file size, user identification (user ID), read-write permission, file type, modification time and the like; the offset information indicates position information offset from the where position of the file, and includes: offset size, etc., where the where indicates the location of the file offset.
Starting a log collection process in a target container, calling a log collection component to scan a directory of a target file after the collection process is started, and starting to collect log information of the target file to acquire latest log data when the log file under the directory is monitored to be updated, wherein the log information comprises: node information, file directory, inode, and offset information of the target file where the update occurred.
203. Acquiring a sending port of the running log and an appointed IP address in the database cluster from a configuration file of the log acquisition component; wherein the transmission port comprises a 6379 port.
Redis is an open source, written in ANSI C Language (American National Standards Institute Combined Programming Language, National Standards Institute C Language), memory-based and persisted journaled databases, with the default port number for the Redis initiation service being 6379. In addition, regarding the IP address in the database cluster, it may be a default IP address in the redis configuration file, for example: bind 127.0.0.1, to receive all access requests; or, the IP can be appointed and set for each node of the redis database, so that the redis database cluster only receives the access request of the address.
Therefore, a sending port of the running log and a specified IP address in the database cluster are obtained from the configuration file of the log collection component, so that the running log can be sent to the Redis database cluster, in a specific manner, the sending address may be the specified IP address of the Redis database cluster, and the sending port is the default port 6379 of the Redis database.
204. And sending the running log to a sending address by a sending mode so as to generate snapshot information of the running log through the database cluster and store the snapshot information into a preset backup file.
And after the sending mode is determined, sending the operation log to a sending address based on the sending mode. In one mode, according to address information obtained from a configuration file of a log collection component, after sending an operation log to a specified IP address of a redis database cluster by using a specified port number, the operation log is saved to a preset backup file by using a snapshot mode, so as to implement persistent processing of operation log data, the so-called snapshot mode is that data in memory data is periodically synchronized to a disk. In particular, the persistence configuration of the redis cluster may be configured as RDB, periodically backing up snapshots of the data set into the dump.
In this step, the database records of Reids in the memory are backed up to the RDB on the disk at regular time, so as to implement data persistence.
205. And if receiving the information that the running log returned by the database cluster is successfully sent, sending an updating record of the running log to the database cluster so as to save the updating record through the database cluster and record the log processing progress of the target container.
And when the information that the running log sent back by the database cluster is successfully sent is received, the operation of saving the running log in the database cluster is successfully realized. And further, sending the update record of the running log to the database cluster so as to continuously save the update record through the database cluster and record the log processing progress of the target container.
In this step, when receiving information that the running log returned by the database cluster is successfully sent, the plurality of acquisition components can rapidly store and update the acquired information through the redis cluster, so that the situation that the registry file falls to the disk to become a bottleneck is avoided.
In addition, the database cluster also has a log retrieval function, and specifically generates a log search request based on a preset log search condition; and sending the log search request to a database cluster, returning a target log meeting the log search condition through the database cluster, and receiving the target log returned by the database cluster.
In one mode, the log search condition includes attribute information of a target log to be searched, such as a generation time of log data, an event type of the log data, or an application system generating the log data. The log search service unit receives log search conditions of K8S cluster users, generates a log search request, sends the log search request to the database cluster, searches in the database cluster according to the search conditions, returns target logs meeting the log search conditions through the database cluster, and determines target retrieval logs, so that the K8S cluster can quickly receive the target logs returned by the database cluster, and a log collection component can quickly respond to the target logs and continuously process unprocessed data.
In the method, by using the log retrieval function in the database cluster, the target retrieval log can be quickly determined by searching information through the log, so that the user experience is improved, and the value and the meaning of the data query result are improved.
According to the log processing method, the operation log containing the operation progress state of the container is sent to the address of the preset database cluster in a specified sending mode to be stored, and data persistence is achieved. In the mode, a sufficient storage environment is provided for running the log, the log data is rapidly stored and updated, and the resource consumption of the log acquisition component is reduced; meanwhile, the log retrieval function based on the database cluster can quickly determine the target retrieval log, and the acquisition efficiency of the log acquisition assembly is greatly improved.
Referring to fig. 3, a log processing apparatus according to an embodiment of the present invention is described below, where an embodiment of the log processing apparatus according to the present invention includes:
the first acquisition module 301 is configured to acquire an operation log of a target container through a log acquisition component operating in the target container;
a first obtaining module 302, configured to obtain a sending mode and a sending address of the running log from the log collection component; the sending address is an address in a preset database cluster; the database cluster comprises a plurality of database servers; a database cluster is preset with a designated data persistence mode;
the first processing module 303 is configured to send the operation log to a sending address in a sending manner, so as to store the operation log by the database cluster, and process the operation log based on a data persistence manner.
The log processing method, device, equipment and storage medium comprise: collecting the running logs of the target container through a log collection component running in the target container; acquiring a sending mode and a sending address of the running log from the log acquisition component; the sending address is an address in a preset database cluster; the database cluster comprises a plurality of database servers; a database cluster is preset with a designated data persistence mode; and sending the running log to a sending address through a sending mode so as to store the running log through the database cluster and process the running log based on a data persistence mode. In the method, the log collection component running in the target container is used for collecting the running log of the target container and acquiring the sending mode and the sending address of the running log, the progress information of the updated log file is sent to the preset database cluster through the corresponding sending mode to be stored and processed based on the data persistence mode, the problem that the log collection and processing speed is slowed down due to frequent updating of a large number of registry files by the log collection component is avoided, and the collection efficiency of the log collection component is greatly improved.
The above apparatus further comprises a first setting module configured to: acquiring a configuration file of a log acquisition component; deleting default configuration in the configuration file, and setting a sending mode and a sending address of the target log in the configuration file; wherein the default configuration comprises: recording the log processing progress of a target container in the equipment through a preset registry file; and setting a directory of a target file which needs to be subjected to log processing in the configuration file.
The first acquisition module is further configured to: starting a log acquisition process in the target container, and calling a log acquisition component through the log acquisition process; scanning a directory of a target file through a log acquisition component; if the target file is monitored to be updated, collecting log information of the target file; wherein the log information includes: node information, file directory, inode, and offset information of the target file where the update occurred.
The first obtaining module. And is also used for: acquiring a sending port of the running log and an appointed IP address in the database cluster from a configuration file of the log acquisition component; wherein the transmission port comprises a 6379 port.
The apparatus further comprises a first recording module configured to: and if receiving the information that the running log returned by the database cluster is successfully sent, sending an updating record of the running log to the database cluster so as to save the updating record through the database cluster and record the log processing progress of the target container.
The data persistence mode comprises a snapshot mode; the first processing module is further configured to send the operation log to a sending address in a sending manner, so as to generate snapshot information of the operation log through the database cluster, and store the snapshot information in a preset backup file.
The apparatus further comprises a first receiving module configured to: generating a log search request based on a preset log search condition; sending the log search request to a database cluster so as to return a target log meeting the log search condition through the database cluster; and receiving a target log returned by the database cluster.
Fig. 3 above describes the log processing apparatus in the embodiment of the present invention in detail from the perspective of modularization, and the log processing apparatus in the embodiment of the present invention is described in detail from the perspective of hardware processing.
Fig. 4 is a schematic structural diagram of a log processing apparatus 400 according to an embodiment of the present invention, where the log processing apparatus 400 may have a relatively large difference due to different configurations or performances, and may include one or more processors (CPUs) 410 (e.g., one or more processors) and a memory 420, and one or more storage media 430 (e.g., one or more mass storage devices) storing an application 433 or data 432. Memory 420 and storage medium 430 may be, among other things, transient or persistent storage. The program stored in the storage medium 430 may include one or more modules (not shown), each of which may include a series of instruction operations in the log processing apparatus 400. Further, the processor 410 may be configured to communicate with the storage medium 430, and execute a series of instruction operations in the storage medium 430 on the log processing device 400.
The log processing apparatus 400 may also include one or more power supplies 440, one or more wired or wireless network interfaces 450, one or more input-output interfaces 460, and/or one or more operating systems 431, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, and so forth. Those skilled in the art will appreciate that the log processing device configuration shown in FIG. 4 does not constitute a limitation of the log processing device, and may include more or fewer components than those shown, or some components in combination, or a different arrangement of components.
The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium, and which may also be a volatile computer-readable storage medium, having stored therein instructions, which, when executed on a computer, cause the computer to perform the steps of the log processing method.
The invention further provides a log processing device, which includes a memory and a processor, wherein the memory stores instructions, and the instructions, when executed by the processor, cause the processor to execute the steps of the log processing method in the above embodiments.
Further, the computer-readable storage medium may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function, and the like; the storage data area may store data created according to the use of the blockchain node, and the like.
The block chain is a novel application mode of computer technologies such as distributed data storage, point-to-point transmission, a consensus mechanism, an encryption algorithm and the like. A block chain (Blockchain), which is essentially a decentralized database, is a series of data blocks associated by using a cryptographic method, and each data block contains information of a batch of network transactions, so as to verify the validity (anti-counterfeiting) of the information and generate a next block. The blockchain may include a blockchain underlying platform, a platform product service layer, an application service layer, and the like.
It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a read-only memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
The above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.

Claims (10)

1. A log processing method is characterized in that the log processing method is applied to equipment which runs a plurality of containers; the log processing method comprises the following steps:
collecting a running log of a target container through a log collection component running in the target container;
acquiring a sending mode and a sending address of the running log from the log acquisition component; the sending address is an address in a preset database cluster; the database cluster comprises a plurality of database servers; the database cluster is preset with a designated data persistence mode;
and sending the running log to the sending address through the sending mode so as to save the running log through the database cluster, and processing the running log based on the data persistence mode.
2. The log processing method according to claim 1, wherein, before the step of collecting the operation log of the target container by the log collection component operating in the target container, the method further comprises:
acquiring a configuration file of the log acquisition component;
deleting default configuration in the configuration file, and setting a sending mode and a sending address of the target log in the configuration file; wherein the default configuration comprises: recording the log processing progress of the target container in the equipment through a preset registry file;
and setting a directory of a target file which needs to be subjected to log processing in the configuration file.
3. The log processing method according to claim 1, wherein the step of collecting the operation log of the target container by a log collection component operating in the target container comprises:
starting a log collection process in the target container, and calling the log collection component through the log collection process;
scanning a directory of a target file through the log collection component;
if the target file is monitored to be updated, collecting log information of the target file; wherein the log information includes: and updating node information, file directory, index node and offset information of the target file.
4. The log processing method according to claim 1, wherein the step of obtaining the sending mode and the sending address of the running log from the log collection component comprises:
acquiring a sending port of the running log and a specified IP address in the database cluster from a configuration file of the log acquisition component; wherein the transmission port comprises a 6379 port.
5. The log processing method according to claim 1, wherein after the step of sending the operation log to the sending address by the sending means, the method further comprises:
and if the information that the running log returned by the database cluster is successfully sent is received, sending an updating record of the running log to the database cluster, so that the updating record is saved by the database cluster, and the log processing progress of the target container is recorded.
6. A log processing method as defined in claim 1, wherein the data persistence mode comprises a snapshot mode; the step of sending the running log to the sending address through the sending mode so as to save the running log through the database cluster and process the running log based on the data persistence mode comprises the following steps:
and sending the running log to the sending address through the sending mode, so as to generate snapshot information of the running log through the database cluster, and storing the snapshot information into a preset backup file.
7. The log processing method according to claim 1, wherein after the step of sending the operation log to the sending address by the sending means, the method further comprises:
generating a log search request based on a preset log search condition;
sending the log search request to the database cluster so as to return a target log meeting the log search condition through the database cluster;
and receiving the target log returned by the database cluster.
8. A log processing apparatus, characterized in that the apparatus is provided to a device in which a plurality of containers are operated; the device includes:
the first acquisition module is used for acquiring the running logs of the target container through a log acquisition component running in the target container;
the first acquisition module is used for acquiring the sending mode and the sending address of the running log from the log acquisition component; the sending address is an address in a preset database cluster; the database cluster comprises a plurality of database servers; the database cluster is preset with a designated data persistence mode;
and the first processing module is used for sending the running log to the sending address through the sending mode, saving the running log through the database cluster, and processing the running log based on the data persistence mode.
9. A log processing apparatus characterized by comprising: a memory and at least one processor, the memory having instructions stored therein;
the at least one processor invokes the instructions in the memory to cause the cryptographic security detection device to perform the log processing method of any of claims 1-7.
10. A computer-readable storage medium having instructions stored thereon, wherein the instructions, when executed by a processor, implement a log processing method according to any one of claims 1 to 7.
CN202210537517.9A 2022-05-18 2022-05-18 Log processing method, device, equipment and storage medium Pending CN114817192A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210537517.9A CN114817192A (en) 2022-05-18 2022-05-18 Log processing method, device, equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210537517.9A CN114817192A (en) 2022-05-18 2022-05-18 Log processing method, device, equipment and storage medium

Publications (1)

Publication Number Publication Date
CN114817192A true CN114817192A (en) 2022-07-29

Family

ID=82515240

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210537517.9A Pending CN114817192A (en) 2022-05-18 2022-05-18 Log processing method, device, equipment and storage medium

Country Status (1)

Country Link
CN (1) CN114817192A (en)

Similar Documents

Publication Publication Date Title
EP2847678B1 (en) Decentralized distributed computing system
US9213706B2 (en) Live restore for a data intelligent storage system
US9384207B2 (en) System and method for creating deduplicated copies of data by tracking temporal relationships among copies using higher-level hash structures
EP2488949B1 (en) De-duplication storage system with multiple indices for efficient file storage
KR101544717B1 (en) Software-defined network attachable storage system and method
US8299944B2 (en) System and method for creating deduplicated copies of data storing non-lossy encodings of data directly in a content addressable store
US8788769B2 (en) System and method for performing backup or restore operations utilizing difference information and timeline state information
WO2017049828A1 (en) Method, device and system for data processing based on linux
US20150019556A1 (en) System and method for managing deduplicated copies of data using temporal relationships among copies
US20130036091A1 (en) Incremental copy performance between data stores
US11461140B2 (en) Systems and methods for controller-worker architecture for searching a storage system
WO2012067964A1 (en) Systems and methods for data management virtualization
US11782878B2 (en) Systems and methods for searching deduplicated data
CN113391952B (en) Automatic backup method based on LVM snapshot in cloud native environment
JP2022155462A (en) System and method for distributed-agent backup of virtual machines
EP3405871B1 (en) Versioned records management using restart era
CN114817192A (en) Log processing method, device, equipment and storage medium
CN116719604A (en) Container migration method and device, storage medium and electronic equipment

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination