CN110750372A - Log system based on shared memory and log management method - Google Patents
Log system based on shared memory and log management method Download PDFInfo
- Publication number
- CN110750372A CN110750372A CN201910990615.6A CN201910990615A CN110750372A CN 110750372 A CN110750372 A CN 110750372A CN 201910990615 A CN201910990615 A CN 201910990615A CN 110750372 A CN110750372 A CN 110750372A
- Authority
- CN
- China
- Prior art keywords
- log
- shared memory
- log information
- shared
- linked list
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000007726 management method Methods 0.000 title claims abstract description 12
- 238000000034 method Methods 0.000 claims abstract description 50
- 238000010586 diagram Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000013467 fragmentation Methods 0.000 description 1
- 238000006062 fragmentation reaction Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000035755 proliferation Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/544—Buffers; Shared memory; Pipes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
Abstract
The invention relates to a log system and a log management method based on a shared memory, wherein the log system comprises: two containers of a shared memory pool and a shared memory queue and a shared memory linked list; the shared memory pool is used for distributing log information to each business process; the shared memory queue is used for storing the log information distributed in the shared memory pool in a pointer mode; the shared memory linked list stores the control information of the log, and the output form of the log information can be accessed and controlled by using a tool or a webpage through the control information stored by the shared memory linked list. The invention not only outputs the log with high efficiency in the server, but also effectively solves the problem that the log is lost when the server releases the version and the service is abnormally crashed, thereby causing difficulty in accurately positioning the bug, and the problem that the unified interface outputs and pushes the log to the log center server, and simultaneously, the invention can flexibly control the output form of the log according to the service requirement.
Description
Technical Field
The invention relates to the technical field of computers, in particular to a log system and a log management method based on a shared memory.
Background
For a log recording system developed by a high-performance server, each service process generally manages and outputs own log, and along with the application of multiple threads, the server adds log information to a log queue container and starts threads to manage log output. But also brings the problem of log loss when the process crashes and the problem of log convergence of all business processes.
With the proliferation and complexity of internet and mobile internet services, when an online release version service crashes, the loss of key logs often makes it difficult to accurately and quickly locate problem points and context information, so that the service cannot be quickly recovered and the loss can be stopped; in addition, the log output of "war by war" inevitably also results in increasing the difficulty of pushing the log to the log center server.
Disclosure of Invention
The present invention provides a log system and a log management method based on a shared memory, which aims at the above problems. The invention stores all the service process log information into the shared memory queue, not only has high-performance log operation, but also solves the problem that a large amount of logs are lost when the process crashes, thereby being capable of quickly positioning the problem and quickly stopping loss, and simultaneously solving the problem that no uniform interface is used for pushing the logs to a log center.
In order to achieve the purpose, the specific technical scheme of the invention is as follows:
a shared memory based log system, the system comprising: two containers of a shared memory pool and a shared memory queue and a shared memory linked list;
the shared memory pool is used for distributing log information to each business process and marking the occupied space corresponding to each log information as a used state before reading;
the shared memory queue is used for storing the log information distributed in the shared memory pool in a pointer mode, and when a process in the log center reads the corresponding log information from the shared memory queue, the occupied space corresponding to the log information in the shared memory pool is marked as an unused state;
the shared memory linked list stores the control information of the log, and the output form of the log information can be accessed and controlled by using a tool or a webpage through the control information stored by the shared memory linked list.
Preferably, the shared memory linked list stores the log name of the control information of the log, the size of the current output document, the lowest level of output, the size and number of the compressed packets, and the tracking ID according to the service.
Preferably, the log information is extracted from the pop in the shared memory queue by the log center process, and after the log information is extracted, the process can judge whether to store the log information locally and in what manner, or judge whether to push the log information to the kafka cluster for collection by the log center server according to the log control information.
Preferably, in a plurality of service processes including the log center process, each process corresponds to an initialized shared memory log information, and accordingly, after the log information is written into the linked list, the shared memory linked list controls and updates the initialized shared memory log information.
The invention also provides a log management method based on the shared memory, which comprises the following steps:
s1, distributing log information to each business process, and placing the log information into a shared memory pool container;
s2, generating a pointer for the log information corresponding to each service process in the shared memory pool and storing the pointer in the shared memory queue;
and S3, writing the control information of the log information into the linked list to generate a shared memory linked list, and accessing the external tool or the webpage through the shared memory linked list to control the output form of the log information.
Preferably, in step S1, in the multiple service processes including the log-centric process, each process corresponds to an initialized shared memory log information, and accordingly, after the log information is written into the linked list, the shared memory linked list controls to update the initialized shared memory log information.
Preferably, in step S3, the shared memory linked list stores the log name of the control information, the size of the current output document, the lowest level of output, the size and number of the compressed packets, and the tracking ID according to the service.
Preferably, in step S3, the log information is extracted from the shared memory queue pop by the log center process, and after the log information is extracted, the process may determine whether to store the log information locally and in what manner, or whether to push the log information to the kafka cluster for collection by the log center server according to the log control information.
The invention has the beneficial effects that:
by using the log system based on the shared memory, the log is output in the server with high efficiency, the problem that the bug is difficult to accurately position due to the loss of the log when the service is abnormally crashed in the version released by the server is effectively solved, the problem that the log is output by a unified interface and pushed to a log center server is solved, and meanwhile, the output form of the log can be flexibly controlled according to the service requirement.
Drawings
FIG. 1 is a schematic diagram of the operation of the present invention;
FIG. 2 is a schematic diagram of the cooperation of a shared memory linked list and each service process in the present invention;
FIG. 3 is a flow chart of a log management method of the present invention.
Detailed Description
In order that those skilled in the art can understand and implement the present invention, the following embodiments of the present invention will be further described with reference to the accompanying drawings.
Referring to fig. 1 to fig. 2, a log system based on a shared memory provided by the present invention includes: the shared memory pool and the shared memory queue comprise two containers and a shared memory linked list.
The key technology of the shared memory based log system lies in the realization of a high-performance shared memory pool, a shared memory queue and a shared memory linked list, and the uniform output form of the log.
In order to achieve a high-performance log output system, the invention adopts two containers of a shared memory pool and a shared memory queue and a shared memory linked list, and the operations are based on the memory and are matched with a reasonable design, so the log output system has the characteristic of high performance.
The shared memory pool is mainly used for distributing log record items to all business processes, namely, elements in the shared memory pool are marked as used states so as to avoid the problems of memory fragmentation and time consumption caused by frequent dynamic application of memory space, when the elements are read from the shared memory queue by a log center process, the elements are released, and the elements in the shared memory pool are marked as unused states.
The shared memory queue is mainly used for storing log information of each business process, the log information is a pointer distributed from a shared memory pool, a large amount of direct memory copy operations of a structural body are avoided through pointer transmission, and in addition, a lock-free queue with multiple producers and multiple consumers is realized inside the shared memory queue, so that the access of a business layer is facilitated, and the performance is extremely high.
The shared memory linked list stores some log control information, such as log name, current output document size, output lowest level, compressed packet size and number, and tracking ID based on service, and the log output mode can be controlled by using tool or web page via the stored control information.
The extraction of the log information is carried out from the pop in the shared memory queue by the log center process, and after the log is extracted, the process can judge whether to store the log to the local and in what way to store the log to the local by the log control information, and can also judge whether to push the log to the kafka cluster so that the log center server can collect the log.
In the invention, in a plurality of service processes including a log center process, each process corresponds to an initialized shared memory log information, and the shared memory linked list controls and updates the initialized shared memory log information after writing in the linked list.
Referring to fig. 3, the present invention further provides a log management method based on a shared memory, including the following steps:
s1, distributing log information to each business process, and placing the log information into a shared memory pool container;
s2, generating a pointer for the log information corresponding to each service process in the shared memory pool and storing the pointer in the shared memory queue;
and S3, writing the control information of the log information into the linked list to generate a shared memory linked list, and accessing the external tool or the webpage through the shared memory linked list to control the output form of the log information.
In step S1, each process corresponds to an initialized shared memory log information in the multiple service processes including the log center process, and the shared memory linked list controls and updates the initialized shared memory log information after writing the initialized shared memory log information into the linked list.
In step S3, the shared memory linked list stores the log name of the control information of the log, the size of the currently output document, the lowest level of output, the size and number of the compressed packets, and the tracking ID according to the service.
In step S3, the log information is extracted from the pop in the shared memory queue by the log center process, and after the log information is extracted, the process may determine whether to store the log information locally or in what manner, or whether to push the log information to the kafka cluster for collection by the log center server according to the log control information.
By the log system and the log management method, the log is efficiently output in the server, the problem that the bug is difficult to accurately position due to log loss when the server is abnormally broken down in the version released by the server is effectively solved, the problem that the bug is difficult to accurately position due to log loss when the server is abnormally broken down is solved, the output of the interface is unified and the blog is pushed to the log center server, and meanwhile, the output form of the log can be flexibly controlled according to the service requirement.
The above-mentioned embodiments only express several embodiments of the present invention, and the description thereof is more specific and detailed, but not construed as limiting the scope of the present invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the inventive concept, which falls within the scope of the present invention. Therefore, the protection scope of the present patent shall be subject to the appended claims.
Claims (8)
1. A shared memory based logging system, the system comprising: two containers of a shared memory pool and a shared memory queue and a shared memory linked list;
the shared memory pool is used for distributing log information to each business process and marking the occupied space corresponding to each log information as a used state before reading;
the shared memory queue is used for storing the log information distributed in the shared memory pool in a pointer mode, and when a process in the log center reads the corresponding log information from the shared memory queue, the occupied space corresponding to the log information in the shared memory pool is marked as an unused state;
the shared memory linked list stores the control information of the log, and the output form of the log information can be accessed and controlled by using a tool or a webpage through the control information stored by the shared memory linked list.
2. The shared-memory based log system of claim 1, wherein the shared-memory linked list stores log names of control information of the log, a size of a currently output document, a lowest level of output, a size and number of compressed packets, and a tracking ID according to a service.
3. The shared-memory based log system as claimed in claim 1 or 2, wherein the log information is extracted from the shared-memory queue by the log-centric process, and when the log information is extracted, the process can determine whether to store the log information locally and in what manner according to the log control information, and also determine whether to push the log information to the kafka cluster for collection by the log-centric server.
4. The shared-memory-based log system according to claim 1 or 2, wherein each of the plurality of business processes including the log-centric process corresponds to an initialized shared-memory log information, and the shared-memory linked list controls updating of the initialized shared-memory log information after writing into the linked list.
5. A log management method based on a shared memory is characterized by comprising the following steps:
s1, distributing log information to each business process, and placing the log information into a shared memory pool container;
s2, generating a pointer for the log information corresponding to each service process in the shared memory pool and storing the pointer in the shared memory queue;
and S3, writing the control information of the log information into the linked list to generate a shared memory linked list, and accessing the external tool or the webpage through the shared memory linked list to control the output form of the log information.
6. The shared-memory-based log management method according to claim 5, wherein in step S1, in the plurality of service processes including the log-centric process, each process corresponds to an initialized shared memory log information, and the shared memory linked list controls updating of the initialized shared memory log information after writing into the linked list.
7. The log management method based on the shared memory as claimed in claim 5, wherein in step S3, the shared memory linked list stores the log name of the control information, the size of the current output document, the lowest level of output, the size and number of the compressed packets, and the tracking ID according to the service.
8. The method for managing logs based on shared memory according to claim 5, wherein in step S3, the log information is extracted from the shared memory queue pop by the log-centric process, and when the log information is extracted, the process can determine whether to store the log information locally and in what manner, and also determine whether to push the log information to the kafka cluster for collection by the log-centric server.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910990615.6A CN110750372A (en) | 2019-10-16 | 2019-10-16 | Log system based on shared memory and log management method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910990615.6A CN110750372A (en) | 2019-10-16 | 2019-10-16 | Log system based on shared memory and log management method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110750372A true CN110750372A (en) | 2020-02-04 |
Family
ID=69278823
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910990615.6A Pending CN110750372A (en) | 2019-10-16 | 2019-10-16 | Log system based on shared memory and log management method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110750372A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111614622A (en) * | 2020-04-20 | 2020-09-01 | 武汉思普崚技术有限公司 | Method and system for sending audit logs of internet behavior management product |
| CN111831515A (en) * | 2020-07-22 | 2020-10-27 | 江苏云柜网络技术有限公司 | Application system running state monitoring system and monitoring method thereof |
| CN112162879A (en) * | 2020-09-30 | 2021-01-01 | 中国科学院计算技术研究所 | Log system of real-time multi-core DSP software |
| CN115460054A (en) * | 2022-08-26 | 2022-12-09 | 深圳技威时代科技有限公司 | Cloud service management and release method and system based on shared memory |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1859325A (en) * | 2006-02-14 | 2006-11-08 | 华为技术有限公司 | News transfer method based on chained list process |
| CN102004675A (en) * | 2010-11-11 | 2011-04-06 | 福建星网锐捷网络有限公司 | Cross-process data transmission method, device and network equipment |
| CN102891873A (en) * | 2011-07-21 | 2013-01-23 | 腾讯科技（深圳）有限公司 | Method for storing log data and log data storage system |
| CN103514053A (en) * | 2013-09-22 | 2014-01-15 | 中国科学院信息工程研究所 | Shared-memory-based method for conducting communication among multiple processes |
| US20160094620A1 (en) * | 2014-09-26 | 2016-03-31 | Lenovo Enterprise Solutions (Singapore) Pte, Ltd. | Scalable logging control for distributed network devices |
| CN106708711A (en) * | 2015-11-17 | 2017-05-24 | 阿里巴巴集团控股有限公司 | Log information output control method and device |
| CN108509327A (en) * | 2018-04-20 | 2018-09-07 | 深圳市文鼎创数据科技有限公司 | A kind of log-output method, device, terminal device and storage medium |
| CN109471743A (en) * | 2018-11-12 | 2019-03-15 | 深圳前海微众银行股份有限公司 | Log collecting method, device and computer readable storage medium |
| CN109508246A (en) * | 2018-06-25 | 2019-03-22 | 广州多益网络股份有限公司 | Log recording method, system and computer readable storage medium |
-
2019
- 2019-10-16 CN CN201910990615.6A patent/CN110750372A/en active Pending
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1859325A (en) * | 2006-02-14 | 2006-11-08 | 华为技术有限公司 | News transfer method based on chained list process |
| CN102004675A (en) * | 2010-11-11 | 2011-04-06 | 福建星网锐捷网络有限公司 | Cross-process data transmission method, device and network equipment |
| CN102891873A (en) * | 2011-07-21 | 2013-01-23 | 腾讯科技（深圳）有限公司 | Method for storing log data and log data storage system |
| CN103514053A (en) * | 2013-09-22 | 2014-01-15 | 中国科学院信息工程研究所 | Shared-memory-based method for conducting communication among multiple processes |
| US20160094620A1 (en) * | 2014-09-26 | 2016-03-31 | Lenovo Enterprise Solutions (Singapore) Pte, Ltd. | Scalable logging control for distributed network devices |
| CN106708711A (en) * | 2015-11-17 | 2017-05-24 | 阿里巴巴集团控股有限公司 | Log information output control method and device |
| CN108509327A (en) * | 2018-04-20 | 2018-09-07 | 深圳市文鼎创数据科技有限公司 | A kind of log-output method, device, terminal device and storage medium |
| CN109508246A (en) * | 2018-06-25 | 2019-03-22 | 广州多益网络股份有限公司 | Log recording method, system and computer readable storage medium |
| CN109471743A (en) * | 2018-11-12 | 2019-03-15 | 深圳前海微众银行股份有限公司 | Log collecting method, device and computer readable storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 连仁包等: ""基于共享内存的松耦合日志系统研究和设计"", 《计算机应用与软件》, no. 06, 15 June 2013 (2013-06-15), pages 14 - 17 * |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111614622A (en) * | 2020-04-20 | 2020-09-01 | 武汉思普崚技术有限公司 | Method and system for sending audit logs of internet behavior management product |
| CN111831515A (en) * | 2020-07-22 | 2020-10-27 | 江苏云柜网络技术有限公司 | Application system running state monitoring system and monitoring method thereof |
| CN112162879A (en) * | 2020-09-30 | 2021-01-01 | 中国科学院计算技术研究所 | Log system of real-time multi-core DSP software |
| CN115460054A (en) * | 2022-08-26 | 2022-12-09 | 深圳技威时代科技有限公司 | Cloud service management and release method and system based on shared memory |
| CN115460054B (en) * | 2022-08-26 | 2024-04-19 | 深圳技威时代科技有限公司 | Cloud service management and release method and system based on shared memory |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110750372A (en) | Log system based on shared memory and log management method | |
| CN107169083B (en) | Mass vehicle data storage and retrieval method and device for public security card port and electronic equipment | |
| CN103379159B (en) | A kind of method that distributed Web station data synchronizes | |
| CN101686209A (en) | Method and device for storing message in message retransmission system | |
| CN102782683A (en) | Buffer pool extension for database server | |
| TW200725298A (en) | System and method for storage management | |
| US20140156603A1 (en) | Method and an apparatus for splitting and recovering data in a power system | |
| CN103051671A (en) | Repeating data deletion method for cluster file system | |
| CN102508913A (en) | Cloud computing system with data cube storage index structure | |
| CN108491332A (en) | A kind of real-time buffering updating method and system based on Redis | |
| CN103678603A (en) | Multi-source heterogeneous data efficient converging and storing frame system | |
| CN109710185A (en) | Data processing method and device | |
| CN103164525A (en) | Method and device for WEB application release | |
| CN102708158A (en) | PostgreSQL (postgres structured query language) cloud storage filing and scheduling system | |
| CN102521114A (en) | File system log storage system under virtualization environment | |
| CN103516549A (en) | File system metadata log mechanism based on shared object storage | |
| CN111177143A (en) | Key value data storage method and device, storage medium and electronic equipment | |
| CN110807038A (en) | CMDB information full-text retrieval method based on elastic search | |
| CN102073690B (en) | Method for constructing memory database supporting historical Key information | |
| CN108647278B (en) | File management method and system | |
| CN112464044B (en) | File data block change information monitoring and management system and method thereof | |
| CN107066281B (en) | Method for rapidly loading large data group based on DSS data system | |
| CN107329695B (en) | Distributed storage memory management method, system and computer storage medium | |
| CN111984198B (en) | Message queue implementation method and device and electronic equipment | |
| CN105095502A (en) | Log collection method of cluster storage system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |