CN118012732B

CN118012732B - Log management method and device and electronic equipment

Info

Publication number: CN118012732B
Application number: CN202410417629.XA
Authority: CN
Inventors: 请求不公布姓名
Original assignee: Hozon New Energy Automobile Co Ltd
Current assignee: Hozon New Energy Automobile Co Ltd
Priority date: 2024-04-08
Filing date: 2024-04-08
Publication date: 2024-06-28
Anticipated expiration: 2044-04-08
Also published as: CN118012732A

Abstract

The application relates to the technical field of computers, in particular to a log management method, a log management device and electronic equipment, which are used for realizing quick log landing. In the embodiment of the application, in response to the operation triggered by the first user on the target equipment, the obtained logs are respectively stored into the corresponding memory blocks through a plurality of business threads contained in at least one business process; collecting logs in memory blocks corresponding to at least one business process through at least one log collecting thread, and determining log landing addresses of the collected logs corresponding to each business process based on address continuous rules; based on the log landing addresses of the logs corresponding to the at least one business process, the corresponding logs are respectively and continuously landed in the log files associated with the at least one business process.

Description

Log management method and device and electronic equipment

Technical Field

The present application relates to the field of computer technologies, and in particular, to a log management method, a log management device, and an electronic device.

Background

A log is a record file generated by a computer system, application program, or network device that records various operations, events, and state information of the system. The log can be used for fault detection, safety monitoring, performance optimization, compliance, business decision and other aspects

However, the existing log management method has obvious defects in the log-landing process, for example, the defects of different log management methods in the log-landing process are as follows:

in the log management method, the process stores the log in the queue first, then stores the log in the queue to the hard disk, and finishes log landing. However, the log is not enough in the process of landing a disc: when a process crashes, the log stored in the queue may be lost.

In the log management method, the log is directly stored in the hard disk, so that log landing is completed. However, since the hard disk is composed of a magnetic head, a motor and a plurality of disks, when the log is required to be stored in a certain area of the hard disk, the hard disk is required to move the motor and the disks to corresponding positions for storage. That is, the log is not enough in the process of landing: when log landing is executed each time, the hard disk can store the log after the motor and the disk are moved to corresponding positions through seek, so that the log storage speed is low, the obtained log cannot be stored to the hard disk in time, the log processing efficiency is affected, and the service cannot normally run easily.

Thus, there are a number of disadvantages to existing log management methods.

Disclosure of Invention

The application aims to provide a log management method, a log management device and electronic equipment, which are used for realizing quick log landing.

In a first aspect, the present application provides a log management method, the method including:

Responding to the operation triggered by the first user to the target equipment, and respectively storing the obtained logs into corresponding memory blocks through a plurality of business threads contained in at least one business process; the memory block is obtained after the memory is divided based on a preset dividing rule; each memory block corresponds to a service thread; one of the business processes comprises at least one business thread;

Collecting logs in the memory blocks corresponding to the at least one business process through at least one log collecting thread, and determining log landing addresses of the collected logs corresponding to each business process based on address continuous rules; based on the log landing addresses of the logs corresponding to the at least one business process, the corresponding logs are respectively and continuously landed in the log files associated with the at least one business process.

In a possible embodiment, the determining, based on the address continuous rule, a log landing address of the collected log corresponding to each service process includes:

The following operations are respectively executed for the logs in the memory blocks corresponding to each business process:

determining the size of the logs respectively acquired by at least one target log acquisition thread in the at least one log acquisition thread respectively; the target log acquisition thread is used for acquiring logs in a memory block corresponding to a business process;

Determining, by the at least one target log acquisition thread, whether a log corresponding to the at least one target log acquisition thread meets a first log file writing condition based on a size of a log acquired by the corresponding target log acquisition thread and a log to-be-written address in a first log file associated with the one business process;

And determining a log landing address of the first log in the first log file by the corresponding target log acquisition thread according to the first log meeting the first log file writing condition in the log corresponding to the at least one target log acquisition thread and utilizing the address continuous rule.

In a possible embodiment, the determining, by the at least one target log collection thread, whether the log corresponding to the at least one target log collection thread meets the first log file writing condition based on the size of the log collected by the corresponding target log collection thread and the log to-be-written address in the first log file associated with the one business process, respectively includes:

the following steps are respectively executed for the logs collected by each target log collecting thread:

Determining, by a target log acquisition thread, a reference capacity corresponding to a first log file associated with the one business process based on a size of a log acquired by the target log acquisition thread and a log to-be-written address in the first log file; reference capacity characterization of the first log file: writing the log acquired by the target log acquisition thread into the first log file, wherein the current size of the first log file is the same as that of the first log file;

When the reference capacity corresponding to the first log file is determined to be in accordance with the capacity threshold range corresponding to the first log file through the target log acquisition thread, determining that the log corresponding to the target log acquisition thread is in accordance with the first log file writing condition;

Otherwise, determining that the log corresponding to the target log acquisition thread does not accord with the first log file writing condition.

In a possible embodiment, the determining, by the corresponding target log collection thread, the log landing address of the first log in the first log file according to the address continuation rule includes:

If one first log exists currently, using a current log to-be-written address in the first log file as a log landing address corresponding to the one first log through a corresponding target log acquisition thread;

If a plurality of first logs exist currently, using a corresponding target log acquisition thread to take the current log to-be-written address in the first log file as a log landing address corresponding to the first log in the landing sequence based on the landing sequence of the plurality of first logs; determining a log landing address of the next first log based on the end address corresponding to each first log according to the landing sequence; the end address corresponding to each first log is determined based on the log to-be-written address in the first log file, the size of the corresponding first log and the size of the first log before the disc-dropping sequence of the corresponding first log.

In one possible embodiment, based on the log landing addresses of the logs corresponding to the at least one service process, respectively, the continuous landing of the corresponding logs into the log files associated with the at least one service process includes:

When a plurality of first journals exist currently, respectively landing the first journals to corresponding journal landing addresses in the first journal file through corresponding target journal collection threads; the first log file supports a plurality of target log acquisition threads to make a first log drop in the first log file.

In one possible embodiment, after the respective logs are continuously dropped into the log file associated with the at least one business process, further comprising:

Responding to a log reading operation triggered by a second user, acquiring a target log from a log file associated with at least one business process through a log display thread, and displaying the target log to the second user; and the target log accords with the reading condition corresponding to the log reading operation.

In one possible embodiment, if multiple target logs are obtained by the log presentation thread; after the target log is obtained from the log included in the at least one file by the log showing thread, and before the target log is shown to the second user, the method further comprises:

And sequencing the plurality of target logs based on the corresponding time stamps of the plurality of target logs through the log presentation thread.

In a second aspect, the present application provides a log management apparatus comprising:

The storage module is used for responding to the operation triggered by the first user on the target equipment and respectively storing the obtained logs into the corresponding memory blocks through a plurality of business threads contained in at least one business process; the content block is obtained after the memory is divided based on a division rule; each memory block corresponds to a service thread; one of the business processes comprises at least one business thread;

The system comprises a log-drop module, a log-drop module and a log-drop module, wherein the log-drop module is used for acquiring logs in memory blocks corresponding to at least one business process through at least one log-acquisition thread, and determining the log-drop address of the acquired log corresponding to each business process based on an address continuous rule; based on the log landing addresses of the logs corresponding to the at least one business process, the corresponding logs are respectively and continuously landed in the log files associated with the at least one business process.

In one possible embodiment, the landing module includes:

In one possible embodiment, the landing module further comprises:

In one possible embodiment, if multiple target logs are obtained by the log presentation thread; the landing tray module further comprises:

In a third aspect, an embodiment of the present application provides an electronic device, including a processor and a memory, where the memory stores program code that, when executed by the processor, causes the processor to perform the steps of any of the methods of the first aspect.

In a fourth aspect, embodiments of the present application provide a computer storage medium storing computer instructions that, when run on a computer, cause the computer to perform the steps of any of the methods of the first aspect.

The embodiment of the application adopts the technical scheme and has at least the following technical effects:

In the embodiment of the application, the obtained logs are respectively stored into the corresponding memory blocks through the plurality of business threads contained in at least one business process, so that the logs can be quickly stored through the memory blocks, and the logs stored in the memory are not lost when the process crashes; according to the embodiment of the application, the logs in the memory blocks corresponding to at least one business process are collected through at least one log collection thread, and the log landing addresses of the collected logs corresponding to each business process are determined based on the address continuous rule, so that the log landing addresses of the logs corresponding to each business process are determined through the address continuous rule, the address continuity of the log landing addresses of the logs corresponding to each business process is ensured, the at least one log collection thread is enabled to store the corresponding logs to the continuous positions in the hard disk based on the log landing addresses of the logs corresponding to at least one business process, the seek time of the hard disk is reduced, and the efficiency of processing the logs is improved.

Drawings

The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain the application and do not constitute a limitation on the application. In the drawings:

Fig. 1 is a schematic diagram of an application scenario according to an embodiment of the present application;

FIG. 2 is a flowchart of a log management method according to an embodiment of the present application;

FIG. 3 is a schematic diagram of a memory block according to an embodiment of the present application;

FIG. 4 is a flow chart of memory block storage according to an embodiment of the present application;

FIG. 5 is a schematic diagram of memory block storage according to an embodiment of the present application;

FIG. 6 is a schematic diagram of a log storage mode according to an embodiment of the present application;

FIG. 7 is a schematic diagram of a log landing tray according to an embodiment of the present application;

FIG. 8 is a flowchart of determining a log-drop address according to an embodiment of the present application;

FIG. 9 is a flowchart of determining whether a first log file writing condition is met according to an embodiment of the present application;

FIG. 10 is a schematic diagram of determining a log-drop address according to an embodiment of the present application;

FIG. 11 is a schematic diagram of a log landing tray according to an embodiment of the present application;

FIG. 12 is a flow chart of a log-drop in accordance with an embodiment of the present application;

FIG. 13 is a schematic diagram of adding a target log to a heap root according to an embodiment of the present application;

FIG. 14 is a schematic diagram illustrating a log process according to an embodiment of the present application;

FIG. 15 is a schematic diagram of a log management device according to an embodiment of the present application;

Fig. 16 is a schematic structural diagram of an electronic device according to an embodiment of the present application;

Fig. 17 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed Description

For the purpose of making the objects, technical solutions and advantages of the embodiments of the present application more apparent, the technical solutions of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application, and it is apparent that the described embodiments are some embodiments of the technical solutions of the present application, but not all embodiments. All other embodiments, based on the embodiments described in the present document, which can be obtained by a person skilled in the art without any creative effort, are within the scope of protection of the technical solutions of the present application.

In order to facilitate a better understanding of the technical solutions of the present application, some terms related to the present application will be described below.

1. Ubuntu operating system (Ubuntu OS) represents the shift pattern operating system, which is one of Linux operating systems.

2. MMAP is used to map a file or other object into memory.

3. Atomic locks are a synchronization mechanism used to protect code blocks that access shared resources.

The word "exemplary" is used hereinafter to mean "serving as an example, embodiment, or illustration. Any embodiment described as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.

The terms "first," "second," and the like herein are used for descriptive purposes only and are not to be construed as either explicit or implicit relative importance or to indicate the number of technical features indicated. Thus, a feature defining "a first" or "a second" may explicitly or implicitly include one or more such feature, and in the description of embodiments of the application, unless otherwise indicated, the meaning of "a plurality" is two or more.

The following briefly describes the design concept of the embodiment of the present application:

Thus, there are a number of disadvantages to existing log management methods.

In view of the above, the embodiment of the application provides a log management method, a log management device and an electronic device. In the embodiment of the application, the obtained logs are respectively stored into the corresponding memory blocks through a plurality of business threads contained in at least one business process, so that the logs can be quickly stored through the memory blocks, and the logs stored in the memory are not lost when the process crashes; according to the embodiment of the application, the logs in the memory blocks corresponding to at least one business process are collected through at least one log collection thread, and the log landing addresses of the collected logs corresponding to each business process are determined based on the address continuous rule, so that the log landing addresses of the logs corresponding to each business process are determined through the address continuous rule, the address continuity of the log landing addresses of the logs corresponding to each business process is ensured, the at least one log collection thread is enabled to store the corresponding logs to the continuous positions in the hard disk based on the log landing addresses of the logs corresponding to at least one business process, the seek time of the hard disk is reduced, and the efficiency of processing the logs is improved.

The preferred embodiments of the present application will be described below with reference to the accompanying drawings of the specification, it being understood that the preferred embodiments described herein are for illustration and explanation only, and not for limitation of the present application, and that the embodiments of the present application and the features of the embodiments may be combined with each other without conflict.

In an alternative implementation manner, an embodiment of the present application provides an application scenario schematic diagram. As shown in fig. 1, the log management method in the embodiment of the present application may be applied to a vehicle. The log management method in the embodiment of the application is executed by an intelligent driving system in the vehicle.

Alternatively, the log management method in the embodiment of the present application may be applied to other electronic devices besides the vehicle shown in fig. 1. Exemplary such electronic devices may be personal computers, cell phones, tablet computers, notebooks, e-book reading, intelligent voice interaction devices, smart home, etc. with certain computing capabilities, as the application is not limited in this regard.

In another alternative implementation manner, the log management method in the embodiment of the present application may be performed by the electronic device and the server together, which is not limited by the present application.

The log management method provided by the exemplary embodiments of the present application will be described below with reference to the accompanying drawings in conjunction with the application scenarios described above, and it should be noted that the application scenarios described above are only shown for the convenience of understanding the spirit and principles of the present application, and embodiments of the present application are not limited in this respect.

Referring to fig. 2, a flowchart of a log management method according to an embodiment of the present application is shown, and specific steps of the log management method are as follows:

step 201, in response to an operation triggered by the first user to the target device, the obtained logs are respectively stored to the corresponding memory blocks through a plurality of service threads included in at least one service process.

The memory block is obtained after the memory is divided based on a preset dividing rule; each memory block corresponds to a service thread; one business process includes at least one business thread.

It should be noted that, when the log management method in the embodiment of the present application is applied to an electronic device, the target device in the embodiment of the present application is the electronic device.

When the log management method in the embodiment of the application is applied to a vehicle, the target device is the vehicle.

Optionally, in the embodiment of the present application, before the obtained log is stored in the corresponding memory block, memory division may be performed based on a preset division rule.

In an exemplary embodiment, in the process of performing memory division based on a preset division rule, a preset number of memory blocks may be first obtained by division, and after the preset number of memory blocks are allocated to corresponding service threads, new memory blocks are obtained by division.

The embodiment of the application can preset the size of each memory block, and divide the memory blocks with corresponding sizes based on the preset size of the memory blocks. For example, the memory block size may be set to 512kb.

As shown in fig. 3, a schematic diagram of a memory block according to an embodiment of the present application is shown. Wherein a number of memory blocks are illustrated in fig. 3.

It should be noted that, in the embodiment of the present application, the memory block may be expressed as: log Block.

Based on the memory blocks obtained by division shown in fig. 3, as shown in fig. 4, a flowchart of memory block storage according to an embodiment of the present application includes the following specific steps:

step S401, respectively receiving memory block addresses corresponding to the memory blocks through a plurality of service threads, and determining the memory blocks corresponding to the service threads.

In the embodiment of the application, each service thread receives a memory block address corresponding to a memory block, the memory block corresponding to the memory block address is used as the memory block corresponding to the service thread, and the memory block address is used as the address of the service thread storage log.

Step S402, storing the logs corresponding to the business threads into the corresponding memory blocks through the business threads.

In the embodiment of the application, each service thread stores the corresponding log into the memory block corresponding to the service thread.

For example, the embodiment of the application can store the log into the corresponding memory block based on the MMAP memory mapping mechanism.

It should be noted that, in the embodiment of the present application, the memory block is managed by the kernel log management module, which is implemented by relying on the Ubuntu OS underlying mechanism.

In the embodiment of the application, a memory management mechanism based on thread level is realized in a kernel layer, business threads in each application layer correspond to a Log Block mapped by the kernel layer when writing logs, read-write is performed by using a Ring Buffer mechanism, and Log data with longest writing time can be covered in the writing process.

It should be noted that, in the embodiment of the present application, the process of storing the log into the memory block by the service thread is equivalent to the above-mentioned log writing.

In the embodiment of the application, in order to prevent fluctuation of system performance caused by the fact that a large number of business threads cannot be started to acquire Log blocks in time when a large number of business processes are started, the embodiment of the application can pre-open up the preset number of Log blocks in advance based on an MMAP memory mapping mechanism, and dynamically open up memory blocks when the Log blocks used by subsequent business are insufficient, so that both the system stability and the business end performance are considered.

As shown in fig. 5, a schematic diagram of memory block storage according to an embodiment of the present application is shown. Taking 3 business processes as an example, log storage is performed, and the logs are respectively stored into corresponding memory blocks through business threads included in each business process. For example, if the service thread 11 included in the service process 1 corresponds to the memory block 1, the service thread 11 stores the log into the memory block 1.

After the first user triggers the operation on the target device, a corresponding log is generated by operating a corresponding component in the target device. The log producer in fig. 5 corresponds to the corresponding component in the target device that is operated in response to the operation after the operation in the embodiment of the present application. In the embodiment of the application, the generated logs are respectively obtained through a plurality of business threads contained in at least one business process, and the logs are stored in the corresponding memory blocks.

It should be noted that, at least one service process is included in each component of the target device, which is not limited by the present application, and the number of service threads included in each service process in fig. 5 is only an example, which is not limited by the present application.

In the process that the service thread stores the log into the corresponding memory block, the embodiment of the application can provide a unified log writing operation interface for a log producer through the log public component, and the service thread of the application layer stores the log into the memory block of the kernel layer through the log public component.

The embodiment of the application is based on a kernel log management mechanism, and can directly write the log into the mapped physical memory when the business thread writes the log, and when the process is suddenly hung up, the last log section can exist in the physical memory and cannot be lost due to the fact that the process is hung up. Therefore, the application can realize the effect that log data is less easy to lose.

In the embodiment of the application, the Log Block related interface externally provided by the kernel layer is safe, and the common component module needs to control each service thread to acquire the memory Block address respectively and then perform concurrent writing lock-free operation.

It should be noted that, in the embodiment of the present application, each service thread stores the log into the corresponding memory block, so that when each service thread executes the write operation, it is not required to receive the influence of other service threads, and multiple service threads can execute the write operation concurrently.

Alternatively, in an embodiment of the present application, the common component module may define a protocol for writing logs, for example: each log comprises a check mark, a log serial number, a log header length, a log length, a host name, a log time, a log type and the like, and the log can be correctly analyzed based on a defined protocol when the kernel log is consumed later.

Illustratively, the log may contain a check flag of: a Magic Number (Magic Number) is used to mark the format of a file or protocol, many of which have Magic marks to indicate the format of the file.

Optionally, in the embodiment of the present application, the logs acquired by each service thread correspond to respective levels, and in the embodiment of the present application, the log common component determines whether the level corresponding to each log meets the log level requirement, and stores the log meeting the log level requirement into the memory block.

Otherwise, the logs which do not meet the log level requirements are not stored, so that the data volume of the logs is reduced, and the burden of storage, transportation and maintenance is reduced.

The level corresponding to the log comprises: TRACE, DEBUG, INFO, WARN, ERROR, CRITICAL, OFF.

It should be noted that, the level corresponding to each log may be manually predetermined, which is not limited by the present application.

It should be noted that, taking a vehicle as an example, the log producer may be each application software in the intelligent driving system in the vehicle, for example, the application software such as path planning, fault diagnosis, etc. may be the log producer.

Optionally, the log level requirements are manually preset.

For example, the log level requirements set manually in advance may be: the level to which the log corresponds belongs to one of TRACE, DEBUG, INFO.

Optionally, in the embodiment of the present application, a storage mode of the log may be set through the log public component, and log output is performed through a write log interface of the log public component.

Exemplary, as shown in fig. 6, a log storage mode is shown in an embodiment of the present application. The log storage mode comprises a terminal mode, a file mode, a remote TCP service mode and a kernel mode.

Under different log storage modes, the log modes output through the log writing operation interface are different, namely the log storage mode influences the mode of the second user for checking the log.

In the embodiment of the application, the first user and the second user can be the same or different.

Optionally, in the embodiment of the present application, the log storage mode may be set by the first user or the second user.

In implementation, in the embodiment of the application, by setting the log storage mode to be the kernel mode, the log public component stores the log into the memory block of the kernel layer through the log write operation interface.

It should be noted that, in the embodiment of the present application, the kernel mode is used for: and setting a mapping relation between a physical memory address and a virtual memory address corresponding to the memory in the kernel component.

Step S202, collecting logs in memory blocks corresponding to at least one business process through at least one log collection thread, and determining log landing addresses of the collected logs corresponding to each business process based on address continuous rules; based on the log landing addresses of the logs corresponding to the at least one business process, the corresponding logs are respectively and continuously landed in the log files associated with the at least one business process.

It should be noted that, the log in the memory block corresponding to the business process is: and the logs in the memory blocks corresponding to each business thread included in the business process.

The number of log collection threads can be preset manually.

In the embodiment of the application, the log acquisition thread is contained in a log acquisition process, and the log acquisition thread is managed through the log acquisition process.

As shown in fig. 7, a log-drop schematic diagram according to an embodiment of the present application is shown. The method comprises the steps of collecting logs in a memory block through a log collecting thread in a log collecting process, and storing the logs into a log file associated with a business process through the log collecting thread.

For example, if the log file 1 is associated with the business process 1, the log in the memory block corresponding to the business process 1 is stored in the log file 1.

It should be noted that, when collecting the logs in the memory blocks, the embodiment of the application does not limit the number of log collection processes, the number of log collection threads included in one log collection process, and the number of log collection threads for collecting one memory block. The corresponding relationship between the memory block and the log acquisition process shown in fig. 7 is only an example of the present application, and is not limited to the present application, and the embodiment of the present application may set the corresponding relationship between the log acquisition process and the memory block, or may not set the corresponding relationship, and does not affect the implementation of the present application.

According to the embodiment of the application, different Log blocks are divided according to the application layer threads to perform Log writing operation, so that lock expenditure caused by writing the same file or memory by multiple threads is avoided, and meanwhile, due to the fact that the Log blocks adopt a Ring Buffer mechanism, the phenomenon that when full data are not consumed in time, the application threads are blocked, and the performance of an application layer process is influenced is avoided. Particularly, when the hard disk is damaged and cannot be written, the normal operation of the application process is not affected, and only log data is lost. Thus, the present application enables higher log throughput.

As shown in fig. 8, a flowchart for determining a log-drop address according to an embodiment of the present application includes the following specific steps:

step S801, determining the sizes of logs acquired by at least one target log acquisition thread in the at least one log acquisition thread respectively.

Taking one business process in each business process as an example, the target log acquisition thread is used for acquiring the log in the memory block corresponding to the business process.

For example, if there are 6 log collection threads, and after the logs in the memory block corresponding to one service process are collected by the thread 1 and the thread 2, the thread 1 and the thread 2 are target log collection threads for collecting the logs in the memory block corresponding to the service process in the 6 log collection threads.

Step S802, determining whether the log corresponding to at least one target log acquisition thread accords with a first log file writing condition or not according to the size of the log acquired by the corresponding target log acquisition thread and the log to-be-written address in the first log file associated with one business process through the at least one target log acquisition thread.

It should be noted that, each log file associated with the business process maintains a current log address to be written.

By way of example, in the embodiment of the present application, the target log collection thread manages the log address to be written in the log file by using the global file atom offset.

Taking the writing of the first log file by the target log acquisition thread 1 as an example, the embodiment of the application needs to carry out addition operation on the offset and the size of the log corresponding to the target log acquisition thread 1 through an atomic lock, and judges whether the first log file writing condition is met.

As shown in fig. 9, a flowchart for determining whether the first log file writing condition is met according to the embodiment of the present application includes the following specific steps:

Step S901, determining, by a target log collection thread, a reference capacity corresponding to a first log file based on a size of a log collected by the target log collection thread and a log address to be written in the first log file associated with a business process.

Wherein, the reference capacity of the first log file is characterized: and writing the log acquired by one target log acquisition thread into the first log file, wherein the current size of the first log file.

In the embodiment of the application, the current capacity of the first log file is determined based on the log to-be-written address in the first log file associated with the business process, and the reference capacity corresponding to the first log file is determined by adding the current capacity of the first log file and the size of the log acquired by the target log acquisition thread.

For example, if the size of the log collected by the target log collection thread is 4 bytes and the current size of the first log file is 10 bytes, the current size of the first log file and the size of the log collected by the target log collection thread are added, and then the reference size corresponding to the first log file is determined to be 14 bytes.

Step 902, through the target log collection thread, when it is determined that the reference capacity corresponding to the first log file meets the capacity threshold range corresponding to the first log file, it is determined that the log corresponding to the target log collection thread meets the first log file writing condition.

For example, taking the reference capacity corresponding to the first log file as 14 bytes and the capacity threshold range as not more than 50 bytes as an example, comparing the determined reference capacity corresponding to the first log file with the capacity threshold range corresponding to the first log file, and determining that the log corresponding to the target log acquisition thread meets the first log file writing condition.

Step 903, if not, determining that the log corresponding to the target log collection thread does not meet the first log file writing condition.

For example, taking the reference capacity corresponding to the first log file as 100 bytes and the capacity threshold range as not greater than 50 bytes as an example, it is determined that the log corresponding to the target log acquisition thread does not conform to the first log file writing condition.

Step 803, determining, by using an address continuous rule, a log landing address of the first log in the first log file according to the first log meeting the writing condition of the first log file in the logs corresponding to the at least one target log acquisition thread through the corresponding target log acquisition thread.

Optionally, the address continuation rule of the embodiment of the present application is:

If a first journal exists currently, using a current journal to-be-written address in the first journal file as a journal landing address corresponding to the first journal through a corresponding target journal acquisition thread.

If a plurality of first journals exist currently, using a corresponding target journal acquisition thread, based on the landing sequence of the plurality of first journals, taking the current journal address to be written in the first journal file as a journal landing address corresponding to the first journal in the landing sequence; and determining the log landing address of the next first log based on the end address corresponding to each first log according to the landing sequence.

The end address corresponding to each first log in the embodiment of the present application is determined based on the to-be-written address of the log in the first log file, the size of the corresponding first log, and the size of the first log before the disc-dropping sequence of the corresponding first log.

Optionally, in the embodiment of the present application, the order of dropping the plurality of first logs may be determined according to any one of the following manners: the method comprises the steps of determining according to the time stamp of each trigger drop operation of a plurality of first journals, determining by CPU scheduling and setting by people.

As shown in fig. 10, a schematic diagram of determining a log-drop address according to an embodiment of the present application is shown. Taking the current existence of a plurality of first journals as an example, if the current journal to be written address of a first journal file is the Head (Head) 1 in the track 3, the journal landing address corresponding to the first journal in the landing sequence is the Head 1 in the track 3, and if the size of the first journal is 3 bytes, the end address of the first journal is determined to be the Head 3 in the track 3; if the second first log is 4 bytes in size based on the end address and the sequence of the disk drops of the first log, it is determined that the head 4 in the track 3 is the log disk drop address corresponding to the second first log, and it is determined that the end address of the second first log is the head 7 in the track 3.

It should be noted that, in the embodiment of the present application, when there are a plurality of first journals currently, the plurality of first journals are respectively dropped to corresponding journal drop addresses in the first journal file through corresponding target journal collection threads; the first log file supports a plurality of target log acquisition threads to make a first log drop in the first log file.

For example, the method and the device can collect threads by the target log corresponding to each first log based on the determined log landing addresses corresponding to the first logs, and simultaneously land the first logs collected to the corresponding log landing addresses.

Optionally, after storing the plurality of first journals in the first journal file, the journal pending address of the first journal file may be updated based on the end address of the last first journal in the landing order.

Step S804, for the logs that do not meet the first log file writing condition in the logs corresponding to at least one target log collecting thread, determining, by the corresponding target log collecting thread, the log landing address of the second log meeting the second log file writing condition in the second log file based on the size of the log that does not meet the first log file writing condition and the log to be written address in the second log file associated with one business process by using the address continuous rule.

As shown in fig. 11, a log-drop schematic diagram is provided in an embodiment of the present application. Based on the process of storing the log into the log file by the log collection thread in fig. 7, taking the log file 1 associated with the business process 1 in fig. 7 as an example, the embodiment of the present application stores the first log meeting the first log file writing condition into the first log file, and stores the second log not meeting the first log file writing condition and meeting the second log file writing condition into the second log file.

Optionally, the embodiment of the application can support functions of volume splitting, compression, rollback and the like according to the size of the log file.

The principle of log landing for log files associated with other business processes according to the embodiment of the present application is similar to the principle described in fig. 11, and the present application is not repeated.

It should be noted that, the embodiment of the present application does not limit the number of log files, if there are logs that do not conform to the second log file, a third log file may be newly created, and whether the third log file writing condition is met is determined, and so on, the present application is not repeated here.

The log acquisition module in the embodiment of the application maintains a single thread, and is used for receiving relevant data such as a state mark of a log file written by each log acquisition thread, judging whether the log file is written or not based on the state mark, and judging whether operations such as log file compression can be performed or not.

In the embodiment of the application, when each log acquisition thread starts writing a log file and ends writing a log file, a state mark is sent to a thread which is maintained independently, and each log file carries out exclusive OR operation based on the state mark sent by the log acquisition thread to judge whether the log file is completely written.

And compressing the written log file according to the configured compression mode. And maintaining a queue for limiting the number of the files for the log files of the log acquisition process of the public components of different services, and judging whether to perform log deletion operation or not every time a newly written log file is added.

Optionally, in the embodiment of the present application, a log drop configuration file is set in a log collection process, and is used for managing configuration of all log collection processes, for example: the process application ID, log level, log storage path, log file size, log file number, etc., and the log collection process will manage the log files according to these configuration requirements.

For example, the process application ID may be the ID of the business process.

By way of example, the embodiment of the application can carry out log landing in a mode of directly writing files based on std:: fwrite, or log landing in a mode of mapping files based on MMAP.

It should be noted that the first method is mainly used for a scene where a single log file is relatively small, and the second method is mainly used for a scene where a single log file is relatively large.

As shown in fig. 12, a flowchart of a log-drop disc according to an embodiment of the present application includes the following specific steps.

Step S1201, for each memory block, collecting logs in the memory block through at least one target log collecting thread, and determining the sizes of the collected logs through the corresponding target log collecting threads;

step S1202, respectively determining whether the acquired logs conform to the first log file writing condition or not based on the log to-be-written address in the first log file associated with the business process and the size of the acquired logs through the corresponding target log acquisition thread; if yes, go to step S1203; if not, executing step S1204;

It should be noted that, in the embodiment of the present application, when the log acquisition process performs multi-line Cheng Lapan log, the location of the log to be written in the file is managed by using the global file atom offset.

Step S1203, storing a first log meeting the writing condition of the first log file into the first log file through a corresponding target log acquisition thread, and sending a status flag to a file flag queue after storing the first log into the first log file;

Step S1204, creating a second log file and sending a status flag to a file flag queue; storing a second log meeting the writing condition of a second log file into the second log file through a corresponding target log acquisition thread, and sending a status mark to a file mark queue after storing the second log into the second log file;

It should be noted that, in the embodiment of the present application, each log file to be written maintains an offset of a current position to be written, when a log acquisition thread needs to write the log file, an atomic lock performs an addition operation on the offset and the size of the log to be written, to determine whether the size exceeds a capacity threshold range of the log file to be written, if the size does not exceed the capacity threshold range of the log file, the size is written into the log file; if the capacity threshold range of the log file is exceeded, closing the log file, re-creating a new log file according to the file naming rule, and writing the log into the new log file, thereby realizing the multi-thread mutually exclusive write file.

According to the embodiment of the application, the lock-free operation of the write log is realized through the atomic lock control offset, the throughput of the write operation is improved, and the more efficient multithread lock-free drop log is realized.

Optionally, in the embodiment of the present application, each log collection thread sends a status flag to a thread separately maintained when writing a log file and writing a log file is completed, and each log file performs an exclusive-or operation through the status flag pushed by the log collection thread to determine whether the log is completely written.

Step S1205, determining whether the corresponding log file is in a completed state or not based on the received state mark through the file mark queue; if yes, go to step 1206; if not, executing step S1201;

Optionally, in the embodiment of the present application, the log collection process maintains a separate thread, which is mainly used for receiving relevant data such as a status flag of each log collection thread writing out a log file, and determining whether a log file is in a completed state, that is, whether operations such as log compression can be performed.

Step S1206, compressing the corresponding log file;

according to the embodiment of the application, the log files in the completed state are compressed according to the configured compression mode.

Step S1207, judging whether the number of the log files exceeds a threshold value; if yes, go to step S1208; if not, step S1205 is executed.

Step S1208, performing log file deletion operation.

Wherein, after step S1208 is performed, step S1205 is continued.

It should be noted that, in the embodiment of the present application, for the log files corresponding to different service processes, a queue is maintained to limit the number of log files, and each time a newly written log file is added, it is determined whether a log deletion operation is required.

In an optional implementation manner, after the log is dropped into the corresponding log file, the embodiment of the application may further obtain the target log from the log file in response to a log reading operation triggered by the second user, and display the target log to the second user.

In another optional implementation manner, after the log is stored in the corresponding memory block, the embodiment of the present application may further obtain the target log from the memory block in response to a log reading operation triggered by the second user, and display the target log to the second user.

The target log accords with the reading condition corresponding to the log reading operation.

Optionally, taking the example of obtaining the target log from the log file, in the embodiment of the present application, in response to a log reading operation triggered by the second user, the target log is obtained from the log file associated with at least one service process through the log display thread, and the target log is displayed to the second user.

Optionally, the log reading operation in the embodiment of the present application at least includes: screening out a log file or a memory block corresponding to at least one target business process; screening logs in a target time period from the log files or the memory blocks; and screening the logs of the target log level from the log files or the memory blocks.

In the embodiment of the application, one log display process comprises at least one log display thread.

It should be noted that, the target service process triggers the selected service process for the second user.

The target log is: and the log corresponding to the target business process selected by the second user.

For the inclusion in the log read operation: screening out the log file corresponding to at least one target business process, screening out the log in the target time period from the log file, and screening out the log of the target log level from the log file, all trigger the process of obtaining the target log from the log file in the first embodiment.

For the inclusion in the log read operation: screening out a memory block corresponding to at least one target service process, screening out a log in a target time period from the memory block, and screening out a log of a target log level from the memory block, all trigger a process of acquiring a target log from the memory block in the second embodiment.

Optionally, taking the log in the log file as an example, for the log reading operation of screening the log file corresponding to at least one target service process, in the embodiment of the present application, the log reading operation of the log file triggered by the second user for a certain target service process may be responded, or the log reading operation of the log file triggered by the second user for a plurality of target service processes may be responded.

Taking the example of obtaining the target log from the log file as an example, in the embodiment of the present application, if the second user selects the business process 1, the triggered log reading operation is: the log files corresponding to the business process 1 are screened out, and the target log is: a log in a log file corresponding to the business process 1; if the second user selects the business processes 1 and 2, the triggered log reading operation is as follows: and screening log files corresponding to the business processes 1 and 2, wherein the target logs are as follows: the business processes 1 and 2 correspond to the logs in the log files.

Optionally, for the log reading operation of screening the log in the target time period from the log files, the embodiment of the application can compare the reference timestamp corresponding to each log file with the target time period set by the second user, if the reference timestamp is in the target time period, the corresponding log file is the log file selected by the second user through the log reading operation, and the log in the corresponding log file is the target log.

It should be noted that, the reference time stamp corresponding to each log file may be the creation time of each log file, or may be the completion time of the log file.

Aiming at the embodiment of acquiring the target log from the memory block, the corresponding log reading operation of screening the log in the target time period from the log file has similar principle to that in the embodiment of acquiring the target log from the log file, and the application is not repeated here; aiming at the log reading operation of screening the logs in the target time period from the memory block, the method judges whether the corresponding logs accord with the reading conditions corresponding to the log reading operation by judging whether the reference time stamp corresponding to each log is positioned in the target time period or not based on the reference time stamp corresponding to each log in the memory block.

It should be noted that, the reference timestamp corresponding to each log in the memory block may be a writing time of each log in the memory block, or may be a time of completing writing of the log.

Optionally, if multiple target logs are obtained by the log exhibition thread; the embodiment of the application further comprises the following steps after the target log is obtained from the log included in at least one file through the log showing thread and before the target log is shown to the second user:

and ordering the plurality of target logs based on the corresponding time stamps of the plurality of target logs through the log presentation thread.

Taking a target log obtained from a log file as an example, firstly determining the log file to be checked according to a log reading operation triggered by a second user, grouping the log file based on a process application ID corresponding to a log public component module, sorting the files in each group based on time, sequentially opening the obtained content according to the files during reading, additionally creating a separate thread to read the file content according to the process application ID sequence, limiting the number of logs read in one cycle by each process application ID, creating a linked list based on the thread ID, forming N thread log linked lists, obtaining the head nodes of the N thread linked lists to create a root stack, and specially responsible for log sorting results. The output thread can acquire the data of the small root heap to output the logs, each acquired log can find the next log node through the linked list and re-add the small root heap, and the process is circulated until the content of the log file is read, and the small root heap is finished when the size of the small root heap is zero. Such as fig. 13.

In fig. 13, data represents each log, and "data1, data2, and data3" shown in fig. 13 represent the arrangement order of the logs, which is only one embodiment of the present application.

Based on the log-drop schematic diagrams shown in fig. 5 and fig. 7, an overall schematic diagram of log processing according to an embodiment of the present application is shown in fig. 14.

It should be noted that, because the log files of the log collection module landing tray are stored binary data and are out of order among the multiple threads, the log sorting tool can be used for sorting.

According to the embodiment of the application, after the log landing operation is unified and concentrated to the log acquisition module, the log landing operation can be carried out on batch data, so that the IO operation is reduced, and meanwhile, the number of data copying times is reduced in the whole process, so that the CPU occupation is reduced, and fewer IO and CPU resources can be occupied by the log landing operation system.

Optionally, the log sorting tool in the embodiment of the present application may also be used to control a log output mode when the second user views the log, for example, output to a terminal, a file, or various modes such as a file and a terminal.

It should be noted that different log output modes affect the display mode of the second user.

Aiming at the embodiment of the drop log file, the application needs to control the log output mode to be a file mode or a mode of a file and a terminal by using a log ordering tool.

According to the embodiment of the application, by defining the log protocol and utilizing the sequencing tool, the log can be flexibly customized according to the needs when the log is checked later, for example, the logs among multiple processes are combined and checked, the logs are combined according to time periods, and the like, so that a more flexible log checking mode is realized.

In the embodiment of the application, all links from the beginning of the log writing to the final viewing can be flexibly controlled, and flexible customization is performed according to the service scene, so that a more flexible log management system is realized.

For convenience of description, the above parts are respectively described as functionally divided into modules. Of course, the functions of each module may be implemented in the same piece or pieces of software or hardware when implementing the present application.

Those skilled in the art will appreciate that the various aspects of the application may be implemented as a system, method, or program product. Accordingly, aspects of the application may be embodied in the following forms, namely: an entirely hardware embodiment, an entirely software embodiment (including firmware, micro-code, etc.) or an embodiment combining hardware and software aspects may be referred to herein as a "circuit," module "or" system.

The specific implementation manner of each module in the apparatus in the above embodiment has been described in detail in the embodiment related to the method, and will not be described in detail herein.

Based on the same inventive concept, the embodiments of the present application provide a log management device, which solves the problem on the similar principle as the method of the above embodiments, so that the implementation of the device can refer to the implementation of the above method, and the repetition is omitted.

As shown in fig. 15, a log management device 1500 provided in an embodiment of the present application includes a storage module 1501 and a drop tray module 1502.

A storage module 1501, configured to respond to an operation triggered by the first user to the target device, and store the obtained logs to corresponding memory blocks through a plurality of service threads included in at least one service process respectively; the content block is obtained after the memory is divided based on a division rule; each memory block corresponds to a service thread; one of the business processes comprises at least one business thread;

A landing module 1502, configured to collect, by using at least one log collection thread, logs in memory blocks corresponding to each of the at least one service process, and determine, based on an address continuous rule, a log landing address of a log corresponding to each collected service process; based on the log landing addresses of the logs corresponding to the at least one business process, the corresponding logs are respectively and continuously landed in the log files associated with the at least one business process.

In one possible embodiment, the drop module 1502 includes:

In one possible embodiment, the drop module 1502 further includes:

And sequencing the log data in the target log based on the time stamp corresponding to each log data in the target log through the log presentation thread.

The principle of solving the problem of the electronic device is similar to that of the method of the above embodiment, so that the implementation of the electronic device can refer to the implementation of the method, and the repetition is omitted.

Referring to fig. 16, the electronic device 160 may include at least a processor 161 and a memory 162. Wherein the memory 162 stores program code that, when executed by the processor 161, causes the processor 161 to perform the log management method in the above-described embodiment of the present application.

An electronic device 170 according to this embodiment of the present application is described below with reference to fig. 17. The electronic device 170 of fig. 17 is merely an example and should not be construed as limiting the functionality and scope of use of embodiments of the present application.

As shown in fig. 17, the electronic device 170 is embodied in the form of a general-purpose electronic device. Components of the electronic device 170 may include, but are not limited to: at least one processing unit 171, the above-mentioned at least one memory unit 172, a bus 173 connecting the different system components, including the memory unit 172 and the processing unit 171.

Bus 173 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a processor, or a local bus using any of a variety of bus architectures.

The storage unit 172 may include readable media in the form of volatile memory, such as Random Access Memory (RAM) 1721 and/or cache memory 1722, and may further include Read Only Memory (ROM) 1723.

The storage unit 172 may also include a program/utility 1725 having a set (at least one) of program modules 1724, such program modules 1724 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of which may include an implementation of a network environment.

The electronic device 170 may also communicate with one or more external devices 174 (e.g., keyboard, pointing device, etc.), with one or more devices that enable objects to interact with the electronic device 170, and/or with any device (e.g., router, modem, etc.) that enables the electronic device 170 to communicate with one or more other electronic devices. Such communication may occur through an input/output (I/O) interface 175. Also, the electronic device 170 may communicate with one or more networks such as a Local Area Network (LAN), a Wide Area Network (WAN) and/or a public network, such as the Internet, through a network adapter 176. As shown, the network adapter 176 communicates with other modules for the electronic device 170 over the bus 173. It should be appreciated that although not shown, other hardware and/or software modules may be used in connection with electronic device 170, including, but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, data backup storage systems, and the like.

Based on the same inventive concept as the above-described method embodiments, various aspects of the advertisement presentation method provided by the present application may also be implemented in the form of a program product comprising program code for causing an electronic device to perform the advertisement presentation method according to various exemplary embodiments of the present application as described above in the present specification, when the program product is run on the electronic device.

The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium would include the following: an electrical connection having one or more wires, a portable disk, a hard disk, random Access Memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

While preferred embodiments of the present application have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. It is therefore intended that the following claims be interpreted as including the preferred embodiments and all such alterations and modifications as fall within the scope of the application.

It will be apparent to those skilled in the art that various modifications and variations can be made to the present application without departing from the spirit or scope of the application. Thus, it is intended that the present application also include such modifications and alterations insofar as they come within the scope of the appended claims or the equivalents thereof.

Claims

1. A method of log management, the method comprising:

Collecting the logs in the memory blocks corresponding to the at least one business process through at least one log collecting thread, and respectively executing the following operations for the logs in the memory blocks corresponding to each business process: determining the size of the logs respectively acquired by at least one target log acquisition thread in the at least one log acquisition thread respectively; the target log acquisition thread is used for acquiring logs in a memory block corresponding to a business process; determining, by the at least one target log acquisition thread, whether a log corresponding to the at least one target log acquisition thread meets a first log file writing condition based on a size of a log acquired by the corresponding target log acquisition thread and a log to-be-written address in a first log file associated with the one business process; determining a log landing address of a first log in a first log file by using the address continuous rule through a corresponding target log acquisition thread aiming at the first log meeting the first log file writing condition in the logs corresponding to the at least one target log acquisition thread;

Based on the log landing addresses of the logs corresponding to the at least one business process, respectively and continuously landing the corresponding logs into log files associated with the at least one business process;

the determining, by the corresponding target log collection thread and using the address continuous rule, a log landing address of the first log in the first log file includes:

if a first journal exists currently, using a current journal to-be-written address in the first journal file as a journal landing address corresponding to the first journal through a corresponding target journal acquisition thread;

If a plurality of first journals exist currently, using a corresponding target journal acquisition thread, based on the landing sequence of the plurality of first journals, taking the current journal to-be-written address in the first journal file as a journal landing address corresponding to the first journal in the landing sequence; determining a log landing address of the next first log based on the end address corresponding to each first log according to the landing sequence; the end address corresponding to each first log is determined based on the log to-be-written address in the first log file, the size of the corresponding first log and the size of the first log before the disc-dropping sequence of the corresponding first log.

2. The method of claim 1, wherein the determining, by the at least one target log collection thread, whether the log corresponding to the at least one target log collection thread meets the first log file writing condition based on the size of the log collected by the corresponding target log collection thread and the log to-be-written address in the first log file associated with the one business process, respectively, comprises:

3. The method of claim 1, wherein based on log landing addresses of the logs corresponding to the at least one business process, respectively, successively landing the corresponding logs into log files associated with the at least one business process, comprising:

4. The method of claim 1, further comprising, after the respective logs are continuously dropped into log files associated with the at least one business process:

5. The method of claim 4, wherein if multiple target logs are obtained by a log presentation thread; after the target log is obtained from the log file associated with the at least one business process by the log presentation thread, and before the target log is presented to the second user, the method further comprises:

6. A log management apparatus, comprising:

The storage module is used for responding to the operation triggered by the first user on the target equipment and respectively storing the obtained logs into the corresponding memory blocks through a plurality of business threads contained in at least one business process; the memory block is obtained after the memory is divided based on a division rule; each memory block corresponds to a service thread; one of the business processes comprises at least one business thread;

The system comprises a tray dropping module, a log collecting module and a log storing module, wherein the tray dropping module is used for collecting logs in memory blocks corresponding to at least one business process respectively through at least one log collecting thread, and the following operations are respectively executed for the logs in the memory blocks corresponding to each business process: determining the size of the logs respectively acquired by at least one target log acquisition thread in the at least one log acquisition thread respectively; the target log acquisition thread is used for acquiring logs in a memory block corresponding to a business process; determining, by the at least one target log acquisition thread, whether a log corresponding to the at least one target log acquisition thread meets a first log file writing condition based on a size of a log acquired by the corresponding target log acquisition thread and a log to-be-written address in a first log file associated with the one business process; determining a log landing address of a first log in a first log file by using the address continuous rule through a corresponding target log acquisition thread aiming at the first log meeting the first log file writing condition in the logs corresponding to the at least one target log acquisition thread; based on the log landing addresses of the logs corresponding to the at least one business process, respectively and continuously landing the corresponding logs into log files associated with the at least one business process;

wherein, the tray module is specifically used for:

7. An electronic device comprising at least one processor and at least one memory, wherein the memory stores a computer program that, when executed by the processor, causes the processor to perform the method of any of claims 1-5.

8. A computer readable storage medium storing computer instructions which, when run on a computer, cause the computer to perform the method of any one of claims 1 to 5.