CN112948136A - Method for implementing asynchronous log record of embedded operating system - Google Patents
Method for implementing asynchronous log record of embedded operating system Download PDFInfo
- Publication number
- CN112948136A CN112948136A CN202110142606.9A CN202110142606A CN112948136A CN 112948136 A CN112948136 A CN 112948136A CN 202110142606 A CN202110142606 A CN 202110142606A CN 112948136 A CN112948136 A CN 112948136A
- Authority
- CN
- China
- Prior art keywords
- log
- thread
- operating system
- embedded operating
- threads
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 42
- 238000004891 communication Methods 0.000 claims abstract description 10
- 238000012545 processing Methods 0.000 claims description 6
- 238000003491 array Methods 0.000 claims description 3
- 238000012360 testing method Methods 0.000 abstract description 4
- 230000008569 process Effects 0.000 description 6
- 230000004044 response Effects 0.000 description 5
- 230000006870 function Effects 0.000 description 4
- 238000010586 diagram Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000015572 biosynthetic process Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000004321 preservation Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/52—Program synchronisation; Mutual exclusion, e.g. by means of semaphores
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/50—Indexing scheme relating to G06F9/50
- G06F2209/5011—Pool
Abstract
The invention provides a method for realizing asynchronous log record of an embedded operating system, which comprises the following steps: step 1: creating a log record execution thread, and running a log output statement and a business logic statement in different threads; step 2: establishing a communication frame for data exchange among concurrent threads, wherein different execution threads can be established by different log outputs; and step 3: and creating a log output execution thread, wherein the log event executes log writing operation in a separate thread according to the configuration of the log. The method for realizing asynchronous log recording of the embedded operating system can greatly improve the log recording throughput, and tests the log throughputs of different thread quantities by taking the log4j and the logback frame as comparison objects.
Description
Technical Field
The invention relates to the technical field of communication, in particular to a method for realizing asynchronous log record of an embedded operating system.
Background
Each large application project comprises a logging system, wherein the logging system is an important part in the development process and provides accurate context information about the operation of the application program, and the logs can be stored in a permanent storage medium for later reference and research. However, many business systems, particularly real-time business systems, require printing detailed contexts and parameters to track problems; if the message is formatted before logging, the performance of the logging output is greatly affected, possibly slowing down the application and further burdening the system. Log4j is a java-based logging tool, and by defining the level of each piece of Log information, the Log generation process can be controlled more finely. But the log formation optimization aspect is general. The Logback rewrites part of kernel codes on the basis of Log4j, the performance is improved by more than 10 times on some key paths, and the time for initializing memory loading is shorter. A certain improvement in data throughput and delay is achieved. But still cannot meet the real-time requirements of real-time systems.
In view of the above-mentioned related art, the inventor considers that there is a problem of low throughput in a multi-thread environment, and therefore, a technical solution is needed to improve the above technical problem.
Disclosure of Invention
Aiming at the defects in the prior art, the invention aims to provide a method for realizing asynchronous log recording of an embedded operating system.
The method for realizing the asynchronous log record of the embedded operating system comprises the following steps:
step 1: creating a log record execution thread, and running a log output statement and a business logic statement in different threads;
step 2: establishing a communication frame for data exchange among concurrent threads, wherein different execution threads can be established by different log outputs;
and step 3: and creating a log output execution thread, wherein the log event executes log writing operation in a separate thread according to the configuration of the log.
Preferably, the log output statement and the service logic statement in step 1 run in different threads, a special thread is used for performing the log output operation, and a main thread processing the service logic executes the subsequent service logic.
Preferably, the step 2 comprises the steps of:
step 2.1: the core data structure queue in the framework is RingBuffer and is designed into a lock-free ring queue;
step 2.2: the concurrent thread accesses the ring queue by adopting a CAS lock-free strategy method;
step 2.3: the performance problem that a plurality of variables share the same cache line is solved by using a cache line filling method.
Preferably, the lock-free circular queues in step 2.1 are connected end to end, and are used cyclically for queue usage, and are saved by using arrays.
Preferably, the concurrent inter-thread communication in step 2.2 is considered a producer and consumer model.
Preferably, the user thread is a producer, the log output execution thread is a consumer, the producer and the consumer each maintain their own pointer, and locking is not required.
Preferably, the producer is one, the consumer is multiple, and the consumer controls own pointer to sequentially read the data corresponding to the respective serial numbers without locking.
Preferably, the producer is a plurality of threads sharing a write pointer, and the CAS method is adopted.
Preferably, the access speed of the memory in the step 2.3 is lower than the running speed of the CPU, a Cache is added between the memory and the CPU, the CPU accesses data in the Cache, and the Cache misses and accesses data in the memory.
Preferably, the log output in step 3 is a file or a network data packet; by adopting the method of the thread pool, the creation of the thread pool is completed before the log is output.
Compared with the prior art, the invention has the following beneficial effects:
1. the method can greatly improve the log record throughput, and takes log4j and a logback frame as comparison objects to test the log throughput of different thread numbers.
2. The response time of the invention is obviously improved, the response time refers to the time spent on recording the log, and the shorter the response time is, the smaller the log recording delay is. This time is equal to the sum of the service time and the wait time, which is the time that the logging event waits for the service to execute. This time increases significantly during peak service and should not be ignored. The response delay time under a certain load was tested using log4j and logback frames as comparison objects.
Drawings
Other features, objects and advantages of the invention will become more apparent upon reading of the detailed description of non-limiting embodiments with reference to the following drawings:
FIG. 1 is a block diagram of the asynchronous logging logic of the present invention;
FIG. 2 is a bar chart of log throughput test results of the present invention;
FIG. 3 is a data diagram of response time test results of the present invention;
Detailed Description
The present invention will be described in detail with reference to specific examples. The following examples will assist those skilled in the art in further understanding the invention, but are not intended to limit the invention in any way. It should be noted that it would be obvious to those skilled in the art that various changes and modifications can be made without departing from the spirit of the invention. All falling within the scope of the present invention.
The invention provides a method for realizing asynchronous log record of an embedded operating system, which is mainly implemented in an initialization process, wherein the initialization process comprises three parts: creating a ring queue for receiving log events sent by a user; creating a log event processing thread for acquiring and processing the log event from the annular queue; and creating a thread pool for processing the specific output process of the log according to the log configuration, wherein the specific output process comprises a network, a file and the like. After initialization is completed, a user firstly creates a log recording object, then configures the object, and finally completes the recording function of log information according to the requirement.
The invention comprises the following steps:
step 1: a logging execution thread is created. The log output statement and the service logic statement of the program are often run in the same thread, that is, when the log is output, the following service logic statement can be executed only after the log output statement is executed. In the method, the log output statement and the service logic statement do not run in the same thread, but a special thread is used for log output operation, and a main thread for processing the service logic can execute the subsequent service logic without waiting.
Step 2: and creating a communication framework for data exchange among concurrent threads. Each user thread calls a log API interface, different log outputs create different execution threads, and the communication framework can be abstracted into producer and consumer models. The user thread is a producer and the log output execution thread is a consumer.
Wherein, the step 2 comprises the following steps:
step 2.1: the internal core data structure queue of the framework is RingBuffer and is designed into a lock-free ring queue. Meaning end to end, the columns can be recycled and saved using arrays. The life cycle of the circular queue is always perpetual, and the pressure of dynamic memory application and release is reduced. The circular queue has no tail pointer and maintains only one sequence number that points to the next available element in the array. Because the circular queue does not control whether overlap is required, there is no need to delete data in the queue until there is a new data overlay. The preservation of the array structure is CPU cache friendly, i.e. at the hardware level, the elements in the array are preloaded, so that in accessing the elements in the array, the CPU does not need to load the next element in the array to memory each time.
Step 2.2: the access of the concurrent threads to the circular queue adopts a CAS (conditional access system) lock-free strategy method, and the method can effectively improve the data throughput aiming at the scene of large data exchange among the concurrent threads. Concurrent inter-thread communication can be viewed as a producer and consumer model.
1. When a single producer and a single consumer, no locking is required since the producer and the consumer each maintain their own pointers.
2. When there is a producer and multiple consumers, each consumer controls its own pointer, and reads the data corresponding to its own serial number in turn without locking.
3. When there are multiple producers, multiple threads share a write pointer, and for this case, we use the CAS method to ensure the write security of multiple threads. The CAS operation contains 3 operands: CAS, the function is described as: and comparing the value of the address A with the value of the address B, assigning C to the address A if the value of the address A is equal to the value of the address B, otherwise, updating the values of A and B, and re-executing the CAS operation until the assignment result is achieved. The CAS is a lightweight instruction implemented by hardware, while the CPU guarantees atomicity of secondary operations.
And the concurrent access of multiple producers and multiple consumers to the ring queue is realized through the CAS. The CAS behaves as an optimistic lock, which avoids performance penalties due to lock contention, thread switching, etc. The throughput of data communication between concurrent threads can be effectively improved.
Step 2.3: the performance problem that a plurality of variables share the same cache line is solved by using a cache line filling method. The access speed of the memory is far lower than the running speed of the CPU, the Cache is added between the memory and the CPU, the CPU firstly accesses the data in the Cache, and the Cache misses the data in the memory. Cache is stored in Cache line units, and when multiple threads modify variables that are independent of each other, if the variables share the same Cache line, the performance of each other is affected unintentionally.
The following method for filling code cache lines is to add 7 long variables before and after the instance variable respectively and to change the time by space.
And step 3: and creating a log output execution thread, wherein the log event can execute log writing operation in a separate thread according to the configuration of the log. The log output may be a file or a network packet. The step adopts a thread pool method, the thread pool is established before the log is output, and the influence of dynamically establishing threads on the performance of the log system during the log output is avoided. In addition, log output may encounter a bottleneck problem of hardware speed, and the thread pool can ensure that log output of different configurations does not influence each other.
Those skilled in the art will appreciate that, in addition to implementing the system and its various devices, modules, units provided by the present invention as pure computer readable program code, the system and its various devices, modules, units provided by the present invention can be fully implemented by logically programming method steps in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers and the like. Therefore, the system and various devices, modules and units thereof provided by the invention can be regarded as a hardware component, and the devices, modules and units included in the system for realizing various functions can also be regarded as structures in the hardware component; means, modules, units for performing the various functions may also be regarded as structures within both software modules and hardware components for performing the method.
The foregoing description of specific embodiments of the present invention has been presented. It is to be understood that the present invention is not limited to the specific embodiments described above, and that various changes or modifications may be made by one skilled in the art within the scope of the appended claims without departing from the spirit of the invention. The embodiments and features of the embodiments of the present application may be combined with each other arbitrarily without conflict.
Claims (10)
1. A method for realizing asynchronous log record of an embedded operating system is characterized by comprising the following steps:
step 1: creating a log record execution thread, and running a log output statement and a business logic statement in different threads;
step 2: establishing a communication frame for data exchange among concurrent threads, wherein different execution threads can be established by different log outputs;
and step 3: and creating a log output execution thread, wherein the log event executes log writing operation in a separate thread according to the configuration of the log.
2. The method for implementing asynchronous log record of embedded operating system according to claim 1, wherein the log output statement and the service logic statement in step 1 run in different threads, a special thread is used for performing log output operation, and a main thread processing the service logic executes subsequent service logic.
3. The method for implementing asynchronous logging of embedded operating system according to claim 1, wherein said step 2 comprises the steps of:
step 2.1: the core data structure queue in the framework is RingBuffer and is designed into a lock-free ring queue;
step 2.2: the concurrent thread accesses the ring queue by adopting a CAS lock-free strategy method;
step 2.3: the performance problem that a plurality of variables share the same cache line is solved by using a cache line filling method.
4. The method for implementing asynchronous logging of embedded operating system as recited in claim 3, wherein said lockless circular queues in step 2.1 are connected end to end, are used for queue recycle, and are saved by arrays.
5. The method of claim 3, wherein the concurrent inter-thread communication in step 2.2 is considered as a producer and consumer model.
6. The method of claim 5, wherein the user thread is a producer, the log output execution thread is a consumer, and the producer and the consumer each maintain their own pointers without locking.
7. The method for implementing asynchronous log record of embedded operating system according to claim 5, wherein the producer is one, the consumer is multiple, and the consumer controls its own pointer to read the data corresponding to their respective serial numbers in sequence without locking.
8. The method as claimed in claim 5, wherein the producer is a plurality of threads sharing a write pointer, and the CAS method is used.
9. The method for implementing asynchronous log record of the embedded operating system according to claim 3, wherein the access speed of the memory in the step 2.3 is lower than the running speed of the CPU, a Cache is added between the memory and the CPU, the CPU accesses data in the Cache, the Cache misses, and accesses data in the memory.
10. The method for implementing asynchronous logging of embedded operating system according to claim 1, wherein the log output in step 3 is a file or a network packet; by adopting the method of the thread pool, the creation of the thread pool is completed before the log is output.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110142606.9A CN112948136A (en) | 2021-02-02 | 2021-02-02 | Method for implementing asynchronous log record of embedded operating system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110142606.9A CN112948136A (en) | 2021-02-02 | 2021-02-02 | Method for implementing asynchronous log record of embedded operating system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112948136A true CN112948136A (en) | 2021-06-11 |
Family
ID=76241520
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110142606.9A Pending CN112948136A (en) | 2021-02-02 | 2021-02-02 | Method for implementing asynchronous log record of embedded operating system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112948136A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113434908A (en) * | 2021-08-30 | 2021-09-24 | 四川新网银行股份有限公司 | Data desensitization method based on Logback log component |
| CN115328755A (en) * | 2022-10-11 | 2022-11-11 | 中化现代农业有限公司 | Concurrent log recording method and system |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2005165932A (en) * | 2003-12-05 | 2005-06-23 | Matsushita Electric Ind Co Ltd | Log management device |
| CN104794040A (en) * | 2015-04-30 | 2015-07-22 | 百度在线网络技术（北京）有限公司 | Service monitoring method, device and system |
| CN105224444A (en) * | 2015-10-14 | 2016-01-06 | 深圳市金证科技股份有限公司 | Daily record generation method and device |
| CN106708578A (en) * | 2016-12-23 | 2017-05-24 | 北京五八信息技术有限公司 | Dual-thread-based journal output method and device |
| CN110377486A (en) * | 2019-07-09 | 2019-10-25 | 福建南威软件有限公司 | The asynchronous task processing method of stable high-throughput is realized based on kafka |
| CN110807062A (en) * | 2018-08-02 | 2020-02-18 | 中兴通讯股份有限公司 | Data synchronization method and device and database host |
| CN111045782A (en) * | 2019-11-20 | 2020-04-21 | 北京奇艺世纪科技有限公司 | Log processing method and device, electronic equipment and computer readable storage medium |
| CN111290919A (en) * | 2020-02-27 | 2020-06-16 | 平安国际智慧城市科技股份有限公司 | Log file generation method and device, computer equipment and storage medium |
| US20200250041A1 (en) * | 2019-01-31 | 2020-08-06 | EMC IP Holding Company, LLC | System and method for log metadata automatic recovery on dual controller storage system |
-
2021
- 2021-02-02 CN CN202110142606.9A patent/CN112948136A/en active Pending
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2005165932A (en) * | 2003-12-05 | 2005-06-23 | Matsushita Electric Ind Co Ltd | Log management device |
| CN104794040A (en) * | 2015-04-30 | 2015-07-22 | 百度在线网络技术（北京）有限公司 | Service monitoring method, device and system |
| CN105224444A (en) * | 2015-10-14 | 2016-01-06 | 深圳市金证科技股份有限公司 | Daily record generation method and device |
| CN106708578A (en) * | 2016-12-23 | 2017-05-24 | 北京五八信息技术有限公司 | Dual-thread-based journal output method and device |
| CN110807062A (en) * | 2018-08-02 | 2020-02-18 | 中兴通讯股份有限公司 | Data synchronization method and device and database host |
| US20200250041A1 (en) * | 2019-01-31 | 2020-08-06 | EMC IP Holding Company, LLC | System and method for log metadata automatic recovery on dual controller storage system |
| CN110377486A (en) * | 2019-07-09 | 2019-10-25 | 福建南威软件有限公司 | The asynchronous task processing method of stable high-throughput is realized based on kafka |
| CN111045782A (en) * | 2019-11-20 | 2020-04-21 | 北京奇艺世纪科技有限公司 | Log processing method and device, electronic equipment and computer readable storage medium |
| CN111290919A (en) * | 2020-02-27 | 2020-06-16 | 平安国际智慧城市科技股份有限公司 | Log file generation method and device, computer equipment and storage medium |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113434908A (en) * | 2021-08-30 | 2021-09-24 | 四川新网银行股份有限公司 | Data desensitization method based on Logback log component |
| CN115328755A (en) * | 2022-10-11 | 2022-11-11 | 中化现代农业有限公司 | Concurrent log recording method and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9619430B2 (en) | Active non-volatile memory post-processing | |
| US9195786B2 (en) | Hardware simulation controller, system and method for functional verification | |
| CN101421711B (en) | Virtual execution system for resource-constrained devices | |
| US10095556B2 (en) | Parallel priority queue utilizing parallel heap on many-core processors for accelerating priority-queue-based applications | |
| US20080271042A1 (en) | Testing multi-thread software using prioritized context switch limits | |
| US11900113B2 (en) | Data flow processing method and related device | |
| CN113918101B (en) | Method, system, equipment and storage medium for writing data cache | |
| AU2013361244A1 (en) | Paraller priority queue utilizing parallel heap on many-core processors for accelerating priority-queue-based applications | |
| CN110888727A (en) | Method, device and storage medium for realizing concurrent lock-free queue | |
| CN114168271B (en) | Task scheduling method, electronic device and storage medium | |
| CN112948136A (en) | Method for implementing asynchronous log record of embedded operating system | |
| CN111666210A (en) | Chip verification method and device | |
| CN109656868B (en) | Memory data transfer method between CPU and GPU | |
| EP1760580B1 (en) | Processing operation information transfer control system and method | |
| US20180046474A1 (en) | Method for executing child kernels invoked on device side utilizing dynamic kernel consolidation and related non-transitory computer readable medium | |
| KR102205899B1 (en) | Method and apparatus for avoiding bank conflict in memory | |
| CN112559403B (en) | Processor and interrupt controller therein | |
| Lázaro-Muñoz et al. | A tasks reordering model to reduce transfers overhead on GPUs | |
| US20230236878A1 (en) | Efficiently launching tasks on a processor | |
| CN105700913A (en) | Method for creation and parallel operation of lightweight bare chip codes | |
| US20220413849A1 (en) | Providing atomicity for complex operations using near-memory computing | |
| US9619277B2 (en) | Computer with plurality of processors sharing process queue, and process dispatch processing method | |
| Singh | Communication Coroutines For Parallel Program Using DW26010 Many Core Processor | |
| US20040034858A1 (en) | Programming a multi-threaded processor | |
| US20130166887A1 (en) | Data processing apparatus and data processing method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210611 |