CN110750372B - Log system and log management method based on shared memory - Google Patents

Log system and log management method based on shared memory Download PDF

Info

Publication number
CN110750372B
CN110750372B CN201910990615.6A CN201910990615A CN110750372B CN 110750372 B CN110750372 B CN 110750372B CN 201910990615 A CN201910990615 A CN 201910990615A CN 110750372 B CN110750372 B CN 110750372B
Authority
CN
China
Prior art keywords
shared memory
log
log information
linked list
information
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201910990615.6A
Other languages
Chinese (zh)
Other versions
CN110750372A (en
Inventor
陈贤平
叶志斌
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shenzhen Gwelltimes Technology Co ltd
Original Assignee
Shenzhen Gwelltimes Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shenzhen Gwelltimes Technology Co ltd filed Critical Shenzhen Gwelltimes Technology Co ltd
Priority to CN201910990615.6A priority Critical patent/CN110750372B/en
Publication of CN110750372A publication Critical patent/CN110750372A/en
Application granted granted Critical
Publication of CN110750372B publication Critical patent/CN110750372B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00Arrangements for program control, e.g. control units
    • G06F9/06Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46Multiprogramming arrangements
    • G06F9/54Interprogram communication
    • G06F9/544Buffers; Shared memory; Pipes
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00Arrangements for program control, e.g. control units
    • G06F9/06Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46Multiprogramming arrangements
    • G06F9/54Interprogram communication
    • G06F9/546Message passing systems or structures, e.g. queues

Landscapes

  • Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Debugging And Monitoring (AREA)

Abstract

The invention relates to a log system and a log management method based on a shared memory, wherein the log system comprises: the shared memory pool and the shared memory queue are two containers and one shared memory linked list; the shared memory pool is used for distributing log information to each business process; the shared memory queue is used for storing the log information distributed in the shared memory pool in the form of a pointer; the shared memory linked list stores the control information of the log, and the tool or the webpage can be used for accessing and controlling the output form of the log information through the control information stored in the shared memory linked list. The invention not only outputs the log in the server with high efficiency, but also effectively solves the problem that the log is lost when the server release version is abnormally crashed, so that the bug is difficult to be positioned accurately, and the problem that the log is output by a unified interface and pushed to the log center server, and meanwhile, the output form of the log can be flexibly controlled according to the service requirement.

Description

Log system and log management method based on shared memory
Technical Field
The invention relates to the technical field of computers, in particular to a log system based on a shared memory and a log management method.
Background
For a log record system developed by a high-performance server, it is common that each business process manages and outputs own log, and along with multi-thread application, the server adds log information to a log queue container and starts a thread to manage log output. But the problem of log loss during process crash and the problem of log aggregation of each business process are also brought.
With the proliferation and complexity of internet and mobile internet services, when online release version service collapses, it is often difficult to accurately and rapidly locate problem points and context information due to the loss of key logs, so that service and loss stop cannot be rapidly recovered; in addition, the difficulty of pushing the log to the log center server is necessarily increased due to the log output of the 'individual combat'.
Disclosure of Invention
The invention provides a log system and a log management method based on a shared memory aiming at the problems. By storing all the business process log information into the shared memory queue, the method not only has high-performance log operation, but also solves the problem that a large amount of logs are lost when the process crashes, thereby being capable of rapidly positioning the problem and rapidly stopping the loss, and simultaneously solving the problem that no unified interface pushes the logs to a log center.
In order to achieve the above purpose, the specific technical scheme of the invention is as follows:
A shared memory based logging system, the system comprising: the shared memory pool and the shared memory queue are two containers and one shared memory linked list;
The shared memory pool is used for distributing log information to each business process, and marking the occupied space corresponding to each log information as a used state before reading;
The shared memory queue is used for storing the log information distributed in the shared memory pool in the form of a pointer, and when a certain process in the log center reads the corresponding log information from the shared memory queue, the occupied space corresponding to the log information in the shared memory pool is marked as an unused state;
The shared memory linked list stores the control information of the log, and the tool or the webpage can be used for accessing and controlling the output form of the log information through the control information stored in the shared memory linked list.
Preferably, the shared memory linked list stores control information log names of logs, current output document size, output lowest level, compressed packet size and number, and tracking ID according to service.
Preferably, the log information is extracted from the shared memory queue by the log center process, and after the log information is extracted, the process can determine whether the log information is stored locally and in what manner, and can also determine whether the log information needs to be pushed to the kafka cluster for collection by the log center server.
Preferably, in a plurality of service processes including the log center process, each process corresponds to an initialized shared memory log information, and after writing the initialized shared memory log information into the linked list according to the initialized shared memory log information, the shared memory linked list is controlled to update the initialized shared memory log information.
The invention also provides a log management method based on the shared memory, which comprises the following steps:
S1, distributing log information to each business process, and placing the log information in a shared memory pool container;
s2, generating pointers for log information corresponding to each service process in the shared memory pool and storing the pointers in a shared memory queue;
s3, writing control information of the log information into the linked list to generate a shared memory linked list, and accessing and controlling an output form of the log information by an external tool or a webpage through the shared memory linked list.
Preferably, in step S1, in a plurality of service processes including the log center process, each process corresponds to an initialized shared memory log information, and after writing the log information into the linked list according to the initialized shared memory log information, the shared memory linked list is controlled to update the initialized shared memory log information.
Preferably, in step S3, the shared memory linked list stores the control information log name of the log, the current output document size, the output lowest level, the compressed packet size and number, and the tracking ID according to the service.
Preferably, in step S3, the log information is extracted from the shared memory queue by the log center process, and after the log information is extracted, the process can determine whether to store the log information locally and in what manner, and can also determine whether to push the log information to the kafka cluster for collection by the log center server.
The invention has the beneficial effects that:
Through the log system based on the shared memory, the log is output in the server with high efficiency, the problem that the log is lost when the server release version is abnormally crashed, so that the bug is difficult to accurately position is solved, the problem that the log is output through a unified interface and is pushed to the log center server is solved, and meanwhile, the output form of the log can be flexibly controlled according to the service requirement.
Drawings
FIG. 1 is a schematic diagram of the operation of the present invention;
FIG. 2 is a schematic diagram of the cooperation of a shared memory linked list and each business process in the present invention;
FIG. 3 is a flow chart of a log management method of the present invention.
Detailed Description
In order that those of ordinary skill in the art will readily understand and practice the invention, embodiments of the invention will be further described with reference to the drawings.
Referring to fig. 1 to 2, the present invention provides a log system based on a shared memory, the system includes: the shared memory pool and the shared memory queue are two containers and one shared memory linked list.
The log system based on the shared memory has the key technology of realizing a high-performance shared memory pool, a shared memory queue, a shared memory linked list and a unified output form of the log.
In order to achieve the high-performance log output system, the invention adopts two containers of the shared memory pool and the shared memory queue and a shared memory linked list, and has the characteristics of high performance because the operations are based on the memory and are matched with reasonable design.
The shared memory pool is mainly used for distributing log record items to each business process, namely, marking elements in the shared memory pool as used states, so as to avoid the problems of memory fragments and time consumption caused by frequent dynamic application of memory space, and when the log center process reads the elements from the shared memory queue, the elements are released, namely, marking the elements in the shared memory pool as unused states.
The shared memory queue is mainly used for storing log information of each business process, the log information is a pointer distributed from a shared memory pool, direct and large-scale memory copy operation of a structural body is avoided through pointer transmission, in addition, the lock-free queue of multiple producers and multiple consumers is realized in the shared memory queue, and the access of a business layer is convenient, and the performance is extremely high.
The shared memory linked list stores control information of some logs, such as log name, current output document size, output lowest level, compressed packet size and number, and tracking ID according to business, etc., and the tool or web page can be used to control the output form of the log through the control information stored in the linked list.
The log information is extracted from the shared memory queue pop by the log center process, and after the log is extracted, the process can judge whether the log is stored locally or not and in what way, and can also judge whether the log is pushed to the kafka cluster or not so as to be collected by the log center server.
In the invention, in a plurality of business processes including a log center process, each process corresponds to an initialized shared memory log information, and after the log information is written into a linked list according to the initialized shared memory log information, the shared memory linked list is controlled to update the initialized shared memory log information.
Referring to fig. 3, the present invention further provides a log management method based on a shared memory, which includes the following steps:
S1, distributing log information to each business process, and placing the log information in a shared memory pool container;
s2, generating pointers for log information corresponding to each service process in the shared memory pool and storing the pointers in a shared memory queue;
s3, writing control information of the log information into the linked list to generate a shared memory linked list, and accessing and controlling an output form of the log information by an external tool or a webpage through the shared memory linked list.
In step S1, each process corresponds to an initialized shared memory log information in a plurality of service processes including a log center process, and after writing the log information into a linked list, the shared memory linked list controls to update the initialized shared memory log information.
In step S3, the shared memory linked list stores the control information log name of the log, the current output document size, the output lowest level, the compressed packet size and number, and the tracking ID according to the service.
In step S3, the log information is extracted from the shared memory queue by the log center process, and after the log information is extracted, the process can determine whether to store the log information locally and in what manner, and can also determine whether to push the log information to the kafka cluster for the log center server to collect.
By the log system and the log management method, the log is output in the server with high efficiency, the problem that the log is lost when the server release version is abnormally crashed, so that the bug is difficult to accurately position is solved, the problem that the log is output through a unified interface and is pushed to the log center server is solved, and meanwhile, the output form of the log can be flexibly controlled according to the service requirement.
The foregoing examples illustrate only a few embodiments of the invention and are described in detail herein without thereby limiting the scope of the invention. It should be noted that it will be apparent to those skilled in the art that several variations and modifications can be made without departing from the spirit of the invention, which are all within the scope of the invention. Accordingly, the scope of protection of the present invention is to be determined by the appended claims.

Claims (1)

1. A shared memory-based log system, the system comprising: the shared memory pool and the shared memory queue are two containers and one shared memory linked list;
the shared memory pool is used for distributing log information to each business process, placing the log information into a shared memory pool container, and marking the occupied space corresponding to each log information as a used state before reading;
The shared memory queue is used for generating a pointer for the log information corresponding to each service process in the shared memory pool, storing the log information distributed in the shared memory pool in the form of the pointer, and marking the occupation space corresponding to the log information in the shared memory pool as an unused state when a certain process in the log center reads the corresponding log information from the shared memory queue;
Writing control information of the log information into a linked list to generate a shared memory linked list, accessing and controlling an output form of the log information by using a tool or a webpage through the control information stored in the shared memory linked list;
The extraction of the log information is that the log center process pops out of the shared memory queue, and after the log information is extracted, the process judges whether the log information is stored locally or not and in what mode, and judges whether the log information needs to be pushed to the kafka cluster so as to be collected by the log center server;
The shared memory linked list stores the control information log name of the log, the current output document size, the output lowest level, the compressed packet size and number and tracking ID according to the service;
In a plurality of business processes including a log center process, each process corresponds to an initialized shared memory log information, and after control information of the initialized shared memory log information is written into a linked list, the shared memory linked list controls and updates the initialized shared memory log information.
CN201910990615.6A 2019-10-16 2019-10-16 Log system and log management method based on shared memory Active CN110750372B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201910990615.6A CN110750372B (en) 2019-10-16 2019-10-16 Log system and log management method based on shared memory

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201910990615.6A CN110750372B (en) 2019-10-16 2019-10-16 Log system and log management method based on shared memory

Publications (2)

Publication Number Publication Date
CN110750372A CN110750372A (en) 2020-02-04
CN110750372B true CN110750372B (en) 2024-05-14

Family

ID=69278823

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201910990615.6A Active CN110750372B (en) 2019-10-16 2019-10-16 Log system and log management method based on shared memory

Country Status (1)

Country Link
CN (1) CN110750372B (en)

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111614622A (en) * 2020-04-20 2020-09-01 武汉思普崚技术有限公司 Method and system for sending audit logs of internet behavior management product
CN111831515A (en) * 2020-07-22 2020-10-27 江苏云柜网络技术有限公司 Application system running state monitoring system and monitoring method thereof
CN112162879A (en) * 2020-09-30 2021-01-01 中国科学院计算技术研究所 Log system of real-time multi-core DSP software
CN115460054B (en) * 2022-08-26 2024-04-19 深圳技威时代科技有限公司 Cloud service management and release method and system based on shared memory

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN1859325A (en) * 2006-02-14 2006-11-08 华为技术有限公司 News transfer method based on chained list process
CN102004675A (en) * 2010-11-11 2011-04-06 福建星网锐捷网络有限公司 Cross-process data transmission method, device and network equipment
CN102891873A (en) * 2011-07-21 2013-01-23 腾讯科技(深圳)有限公司 Method for storing log data and log data storage system
CN103514053A (en) * 2013-09-22 2014-01-15 中国科学院信息工程研究所 Shared-memory-based method for conducting communication among multiple processes
CN106708711A (en) * 2015-11-17 2017-05-24 阿里巴巴集团控股有限公司 Log information output control method and device
CN108509327A (en) * 2018-04-20 2018-09-07 深圳市文鼎创数据科技有限公司 A kind of log-output method, device, terminal device and storage medium
CN109471743A (en) * 2018-11-12 2019-03-15 深圳前海微众银行股份有限公司 Log collecting method, device and computer readable storage medium
CN109508246A (en) * 2018-06-25 2019-03-22 广州多益网络股份有限公司 Log recording method, system and computer readable storage medium

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9807154B2 (en) * 2014-09-26 2017-10-31 Lenovo Enterprise Solutions (Singapore) Pte, Ltd. Scalable logging control for distributed network devices

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN1859325A (en) * 2006-02-14 2006-11-08 华为技术有限公司 News transfer method based on chained list process
CN102004675A (en) * 2010-11-11 2011-04-06 福建星网锐捷网络有限公司 Cross-process data transmission method, device and network equipment
CN102891873A (en) * 2011-07-21 2013-01-23 腾讯科技(深圳)有限公司 Method for storing log data and log data storage system
CN103514053A (en) * 2013-09-22 2014-01-15 中国科学院信息工程研究所 Shared-memory-based method for conducting communication among multiple processes
CN106708711A (en) * 2015-11-17 2017-05-24 阿里巴巴集团控股有限公司 Log information output control method and device
CN108509327A (en) * 2018-04-20 2018-09-07 深圳市文鼎创数据科技有限公司 A kind of log-output method, device, terminal device and storage medium
CN109508246A (en) * 2018-06-25 2019-03-22 广州多益网络股份有限公司 Log recording method, system and computer readable storage medium
CN109471743A (en) * 2018-11-12 2019-03-15 深圳前海微众银行股份有限公司 Log collecting method, device and computer readable storage medium

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
"基于共享内存的松耦合日志系统研究和设计";连仁包等;《计算机应用与软件》;20130615(第06期);第14-17+21页 *

Also Published As

Publication number Publication date
CN110750372A (en) 2020-02-04

Similar Documents

Publication Publication Date Title
CN110750372B (en) Log system and log management method based on shared memory
CN102521269B (en) Index-based computer continuous data protection method
CN104040481B (en) Method and system for merging, storing and retrieving incremental backup data
US8738673B2 (en) Index partition maintenance over monotonically addressed document sequences
US9069704B2 (en) Database log replay parallelization
CN101964820B (en) Method and system for keeping data consistency
US8356050B1 (en) Method or system for spilling in query environments
CN103049317B (en) Based on the high concurrent data no write de-lay system and method for queue under cloud environment
CN102782683A (en) Buffer pool extension for database server
CN102843396A (en) Data writing and reading method and device in distributed caching system
CN101944119B (en) Real-time event management method for intelligent electronic equipment
US20120290595A1 (en) Super-records
CN103370691A (en) Managing buffer overflow conditions
CN103516549B (en) A kind of file system metadata log mechanism based on shared object storage
CN108491332A (en) A kind of real-time buffering updating method and system based on Redis
JP6432805B2 (en) REDO logging for partitioned in-memory data sets
CN103051671A (en) Repeating data deletion method for cluster file system
CN102779138A (en) Hard disk access method of real time data
CN106681660B (en) IO scheduling method and IO scheduling device
CN110311817B (en) Container log processing system for Kubernetes cluster
CN114706836B (en) Data life cycle management method based on airborne embedded database
CN103778120A (en) Global file identification generation method, generation device and corresponding distributed file system
CN106874343B (en) Data deletion method and system for time sequence database
CN112464044B (en) File data block change information monitoring and management system and method thereof
CN107329695B (en) Distributed storage memory management method, system and computer storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant