CN110750372B - Log system and log management method based on shared memory - Google Patents
Log system and log management method based on shared memory Download PDFInfo
- Publication number
- CN110750372B CN110750372B CN201910990615.6A CN201910990615A CN110750372B CN 110750372 B CN110750372 B CN 110750372B CN 201910990615 A CN201910990615 A CN 201910990615A CN 110750372 B CN110750372 B CN 110750372B
- Authority
- CN
- China
- Prior art keywords
- shared memory
- log
- log information
- linked list
- information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000007726 management method Methods 0.000 title abstract description 9
- 238000000034 method Methods 0.000 claims abstract description 45
- 238000000605 extraction Methods 0.000 claims 1
- 238000010586 diagram Methods 0.000 description 2
- 230000002776 aggregation Effects 0.000 description 1
- 238000004220 aggregation Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 239000012634 fragment Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000035755 proliferation Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/544—Buffers; Shared memory; Pipes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention relates to a log system and a log management method based on a shared memory, wherein the log system comprises: the shared memory pool and the shared memory queue are two containers and one shared memory linked list; the shared memory pool is used for distributing log information to each business process; the shared memory queue is used for storing the log information distributed in the shared memory pool in the form of a pointer; the shared memory linked list stores the control information of the log, and the tool or the webpage can be used for accessing and controlling the output form of the log information through the control information stored in the shared memory linked list. The invention not only outputs the log in the server with high efficiency, but also effectively solves the problem that the log is lost when the server release version is abnormally crashed, so that the bug is difficult to be positioned accurately, and the problem that the log is output by a unified interface and pushed to the log center server, and meanwhile, the output form of the log can be flexibly controlled according to the service requirement.
Description
Technical Field
The invention relates to the technical field of computers, in particular to a log system based on a shared memory and a log management method.
Background
For a log record system developed by a high-performance server, it is common that each business process manages and outputs own log, and along with multi-thread application, the server adds log information to a log queue container and starts a thread to manage log output. But the problem of log loss during process crash and the problem of log aggregation of each business process are also brought.
With the proliferation and complexity of internet and mobile internet services, when online release version service collapses, it is often difficult to accurately and rapidly locate problem points and context information due to the loss of key logs, so that service and loss stop cannot be rapidly recovered; in addition, the difficulty of pushing the log to the log center server is necessarily increased due to the log output of the 'individual combat'.
Disclosure of Invention
The invention provides a log system and a log management method based on a shared memory aiming at the problems. By storing all the business process log information into the shared memory queue, the method not only has high-performance log operation, but also solves the problem that a large amount of logs are lost when the process crashes, thereby being capable of rapidly positioning the problem and rapidly stopping the loss, and simultaneously solving the problem that no unified interface pushes the logs to a log center.
In order to achieve the above purpose, the specific technical scheme of the invention is as follows:
A shared memory based logging system, the system comprising: the shared memory pool and the shared memory queue are two containers and one shared memory linked list;
The shared memory pool is used for distributing log information to each business process, and marking the occupied space corresponding to each log information as a used state before reading;
The shared memory queue is used for storing the log information distributed in the shared memory pool in the form of a pointer, and when a certain process in the log center reads the corresponding log information from the shared memory queue, the occupied space corresponding to the log information in the shared memory pool is marked as an unused state;
The shared memory linked list stores the control information of the log, and the tool or the webpage can be used for accessing and controlling the output form of the log information through the control information stored in the shared memory linked list.
Preferably, the shared memory linked list stores control information log names of logs, current output document size, output lowest level, compressed packet size and number, and tracking ID according to service.
Preferably, the log information is extracted from the shared memory queue by the log center process, and after the log information is extracted, the process can determine whether the log information is stored locally and in what manner, and can also determine whether the log information needs to be pushed to the kafka cluster for collection by the log center server.
Preferably, in a plurality of service processes including the log center process, each process corresponds to an initialized shared memory log information, and after writing the initialized shared memory log information into the linked list according to the initialized shared memory log information, the shared memory linked list is controlled to update the initialized shared memory log information.
The invention also provides a log management method based on the shared memory, which comprises the following steps:
S1, distributing log information to each business process, and placing the log information in a shared memory pool container;
s2, generating pointers for log information corresponding to each service process in the shared memory pool and storing the pointers in a shared memory queue;
s3, writing control information of the log information into the linked list to generate a shared memory linked list, and accessing and controlling an output form of the log information by an external tool or a webpage through the shared memory linked list.
Preferably, in step S1, in a plurality of service processes including the log center process, each process corresponds to an initialized shared memory log information, and after writing the log information into the linked list according to the initialized shared memory log information, the shared memory linked list is controlled to update the initialized shared memory log information.
Preferably, in step S3, the shared memory linked list stores the control information log name of the log, the current output document size, the output lowest level, the compressed packet size and number, and the tracking ID according to the service.
Preferably, in step S3, the log information is extracted from the shared memory queue by the log center process, and after the log information is extracted, the process can determine whether to store the log information locally and in what manner, and can also determine whether to push the log information to the kafka cluster for collection by the log center server.
The invention has the beneficial effects that:
Through the log system based on the shared memory, the log is output in the server with high efficiency, the problem that the log is lost when the server release version is abnormally crashed, so that the bug is difficult to accurately position is solved, the problem that the log is output through a unified interface and is pushed to the log center server is solved, and meanwhile, the output form of the log can be flexibly controlled according to the service requirement.
Drawings
FIG. 1 is a schematic diagram of the operation of the present invention;
FIG. 2 is a schematic diagram of the cooperation of a shared memory linked list and each business process in the present invention;
FIG. 3 is a flow chart of a log management method of the present invention.
Detailed Description
In order that those of ordinary skill in the art will readily understand and practice the invention, embodiments of the invention will be further described with reference to the drawings.
Referring to fig. 1 to 2, the present invention provides a log system based on a shared memory, the system includes: the shared memory pool and the shared memory queue are two containers and one shared memory linked list.
The log system based on the shared memory has the key technology of realizing a high-performance shared memory pool, a shared memory queue, a shared memory linked list and a unified output form of the log.
In order to achieve the high-performance log output system, the invention adopts two containers of the shared memory pool and the shared memory queue and a shared memory linked list, and has the characteristics of high performance because the operations are based on the memory and are matched with reasonable design.
The shared memory pool is mainly used for distributing log record items to each business process, namely, marking elements in the shared memory pool as used states, so as to avoid the problems of memory fragments and time consumption caused by frequent dynamic application of memory space, and when the log center process reads the elements from the shared memory queue, the elements are released, namely, marking the elements in the shared memory pool as unused states.
The shared memory queue is mainly used for storing log information of each business process, the log information is a pointer distributed from a shared memory pool, direct and large-scale memory copy operation of a structural body is avoided through pointer transmission, in addition, the lock-free queue of multiple producers and multiple consumers is realized in the shared memory queue, and the access of a business layer is convenient, and the performance is extremely high.
The shared memory linked list stores control information of some logs, such as log name, current output document size, output lowest level, compressed packet size and number, and tracking ID according to business, etc., and the tool or web page can be used to control the output form of the log through the control information stored in the linked list.
The log information is extracted from the shared memory queue pop by the log center process, and after the log is extracted, the process can judge whether the log is stored locally or not and in what way, and can also judge whether the log is pushed to the kafka cluster or not so as to be collected by the log center server.
In the invention, in a plurality of business processes including a log center process, each process corresponds to an initialized shared memory log information, and after the log information is written into a linked list according to the initialized shared memory log information, the shared memory linked list is controlled to update the initialized shared memory log information.
Referring to fig. 3, the present invention further provides a log management method based on a shared memory, which includes the following steps:
S1, distributing log information to each business process, and placing the log information in a shared memory pool container;
s2, generating pointers for log information corresponding to each service process in the shared memory pool and storing the pointers in a shared memory queue;
s3, writing control information of the log information into the linked list to generate a shared memory linked list, and accessing and controlling an output form of the log information by an external tool or a webpage through the shared memory linked list.
In step S1, each process corresponds to an initialized shared memory log information in a plurality of service processes including a log center process, and after writing the log information into a linked list, the shared memory linked list controls to update the initialized shared memory log information.
In step S3, the shared memory linked list stores the control information log name of the log, the current output document size, the output lowest level, the compressed packet size and number, and the tracking ID according to the service.
In step S3, the log information is extracted from the shared memory queue by the log center process, and after the log information is extracted, the process can determine whether to store the log information locally and in what manner, and can also determine whether to push the log information to the kafka cluster for the log center server to collect.
By the log system and the log management method, the log is output in the server with high efficiency, the problem that the log is lost when the server release version is abnormally crashed, so that the bug is difficult to accurately position is solved, the problem that the log is output through a unified interface and is pushed to the log center server is solved, and meanwhile, the output form of the log can be flexibly controlled according to the service requirement.
The foregoing examples illustrate only a few embodiments of the invention and are described in detail herein without thereby limiting the scope of the invention. It should be noted that it will be apparent to those skilled in the art that several variations and modifications can be made without departing from the spirit of the invention, which are all within the scope of the invention. Accordingly, the scope of protection of the present invention is to be determined by the appended claims.
Claims (1)
1. A shared memory-based log system, the system comprising: the shared memory pool and the shared memory queue are two containers and one shared memory linked list;
the shared memory pool is used for distributing log information to each business process, placing the log information into a shared memory pool container, and marking the occupied space corresponding to each log information as a used state before reading;
The shared memory queue is used for generating a pointer for the log information corresponding to each service process in the shared memory pool, storing the log information distributed in the shared memory pool in the form of the pointer, and marking the occupation space corresponding to the log information in the shared memory pool as an unused state when a certain process in the log center reads the corresponding log information from the shared memory queue;
Writing control information of the log information into a linked list to generate a shared memory linked list, accessing and controlling an output form of the log information by using a tool or a webpage through the control information stored in the shared memory linked list;
The extraction of the log information is that the log center process pops out of the shared memory queue, and after the log information is extracted, the process judges whether the log information is stored locally or not and in what mode, and judges whether the log information needs to be pushed to the kafka cluster so as to be collected by the log center server;
The shared memory linked list stores the control information log name of the log, the current output document size, the output lowest level, the compressed packet size and number and tracking ID according to the service;
In a plurality of business processes including a log center process, each process corresponds to an initialized shared memory log information, and after control information of the initialized shared memory log information is written into a linked list, the shared memory linked list controls and updates the initialized shared memory log information.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910990615.6A CN110750372B (en) | 2019-10-16 | 2019-10-16 | Log system and log management method based on shared memory |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910990615.6A CN110750372B (en) | 2019-10-16 | 2019-10-16 | Log system and log management method based on shared memory |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110750372A CN110750372A (en) | 2020-02-04 |
| CN110750372B true CN110750372B (en) | 2024-05-14 |
Family
ID=69278823
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910990615.6A Active CN110750372B (en) | 2019-10-16 | 2019-10-16 | Log system and log management method based on shared memory |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110750372B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111614622A (en) * | 2020-04-20 | 2020-09-01 | 武汉思普崚技术有限公司 | Method and system for sending audit logs of internet behavior management product |
| CN111831515A (en) * | 2020-07-22 | 2020-10-27 | 江苏云柜网络技术有限公司 | Application system running state monitoring system and monitoring method thereof |
| CN112162879A (en) * | 2020-09-30 | 2021-01-01 | 中国科学院计算技术研究所 | Log system of real-time multi-core DSP software |
| CN115460054B (en) * | 2022-08-26 | 2024-04-19 | 深圳技威时代科技有限公司 | Cloud service management and release method and system based on shared memory |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1859325A (en) * | 2006-02-14 | 2006-11-08 | 华为技术有限公司 | News transfer method based on chained list process |
| CN102004675A (en) * | 2010-11-11 | 2011-04-06 | 福建星网锐捷网络有限公司 | Cross-process data transmission method, device and network equipment |
| CN102891873A (en) * | 2011-07-21 | 2013-01-23 | 腾讯科技（深圳）有限公司 | Method for storing log data and log data storage system |
| CN103514053A (en) * | 2013-09-22 | 2014-01-15 | 中国科学院信息工程研究所 | Shared-memory-based method for conducting communication among multiple processes |
| CN106708711A (en) * | 2015-11-17 | 2017-05-24 | 阿里巴巴集团控股有限公司 | Log information output control method and device |
| CN108509327A (en) * | 2018-04-20 | 2018-09-07 | 深圳市文鼎创数据科技有限公司 | A kind of log-output method, device, terminal device and storage medium |
| CN109471743A (en) * | 2018-11-12 | 2019-03-15 | 深圳前海微众银行股份有限公司 | Log collecting method, device and computer readable storage medium |
| CN109508246A (en) * | 2018-06-25 | 2019-03-22 | 广州多益网络股份有限公司 | Log recording method, system and computer readable storage medium |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9807154B2 (en) * | 2014-09-26 | 2017-10-31 | Lenovo Enterprise Solutions (Singapore) Pte, Ltd. | Scalable logging control for distributed network devices |
-
2019
- 2019-10-16 CN CN201910990615.6A patent/CN110750372B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1859325A (en) * | 2006-02-14 | 2006-11-08 | 华为技术有限公司 | News transfer method based on chained list process |
| CN102004675A (en) * | 2010-11-11 | 2011-04-06 | 福建星网锐捷网络有限公司 | Cross-process data transmission method, device and network equipment |
| CN102891873A (en) * | 2011-07-21 | 2013-01-23 | 腾讯科技（深圳）有限公司 | Method for storing log data and log data storage system |
| CN103514053A (en) * | 2013-09-22 | 2014-01-15 | 中国科学院信息工程研究所 | Shared-memory-based method for conducting communication among multiple processes |
| CN106708711A (en) * | 2015-11-17 | 2017-05-24 | 阿里巴巴集团控股有限公司 | Log information output control method and device |
| CN108509327A (en) * | 2018-04-20 | 2018-09-07 | 深圳市文鼎创数据科技有限公司 | A kind of log-output method, device, terminal device and storage medium |
| CN109508246A (en) * | 2018-06-25 | 2019-03-22 | 广州多益网络股份有限公司 | Log recording method, system and computer readable storage medium |
| CN109471743A (en) * | 2018-11-12 | 2019-03-15 | 深圳前海微众银行股份有限公司 | Log collecting method, device and computer readable storage medium |
Non-Patent Citations (1)
| Title |
|---|
| "基于共享内存的松耦合日志系统研究和设计";连仁包等;《计算机应用与软件》;20130615(第06期);第14-17+21页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110750372A (en) | 2020-02-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110750372B (en) | Log system and log management method based on shared memory | |
| CN102521269B (en) | Index-based computer continuous data protection method | |
| CN104040481B (en) | Method and system for merging, storing and retrieving incremental backup data | |
| US8738673B2 (en) | Index partition maintenance over monotonically addressed document sequences | |
| US9069704B2 (en) | Database log replay parallelization | |
| CN101964820B (en) | Method and system for keeping data consistency | |
| US8356050B1 (en) | Method or system for spilling in query environments | |
| CN103049317B (en) | Based on the high concurrent data no write de-lay system and method for queue under cloud environment | |
| CN102782683A (en) | Buffer pool extension for database server | |
| CN102843396A (en) | Data writing and reading method and device in distributed caching system | |
| CN101944119B (en) | Real-time event management method for intelligent electronic equipment | |
| US20120290595A1 (en) | Super-records | |
| CN103370691A (en) | Managing buffer overflow conditions | |
| CN103516549B (en) | A kind of file system metadata log mechanism based on shared object storage | |
| CN108491332A (en) | A kind of real-time buffering updating method and system based on Redis | |
| JP6432805B2 (en) | REDO logging for partitioned in-memory data sets | |
| CN103051671A (en) | Repeating data deletion method for cluster file system | |
| CN102779138A (en) | Hard disk access method of real time data | |
| CN106681660B (en) | IO scheduling method and IO scheduling device | |
| CN110311817B (en) | Container log processing system for Kubernetes cluster | |
| CN114706836B (en) | Data life cycle management method based on airborne embedded database | |
| CN103778120A (en) | Global file identification generation method, generation device and corresponding distributed file system | |
| CN106874343B (en) | Data deletion method and system for time sequence database | |
| CN112464044B (en) | File data block change information monitoring and management system and method thereof | |
| CN107329695B (en) | Distributed storage memory management method, system and computer storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |