CN107733667B - Log management method and system - Google Patents
Log management method and system Download PDFInfo
- Publication number
- CN107733667B CN107733667B CN201610649483.7A CN201610649483A CN107733667B CN 107733667 B CN107733667 B CN 107733667B CN 201610649483 A CN201610649483 A CN 201610649483A CN 107733667 B CN107733667 B CN 107733667B
- Authority
- CN
- China
- Prior art keywords
- log
- user
- file
- lustre
- file system
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/069—Management of faults, events, alarms or notifications using logs of notifications; Post-processing of notifications
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Debugging And Monitoring (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a log management method and a log management system, wherein the method comprises the following steps: configuring a mount point of a Lustre file system on equipment where a software system is located in advance; after the software system generates a log file, writing the log file into a storage node OST of the Lustre file system through the mounting point configured on the device where the software system is located; after a log server receives a log viewing request of a user, utilizing a device S where a software system to which a log to be queried indicated by the log viewing request belongsiIP address of, the user being at the device SiThe user name and the password enter the device SiBy said device SiAnd the mounted point configured on the system acquires the log file which needs to be checked by the user from the corresponding storage node of the Lustre file system and sends the log file to the user. By adopting the method and the device, the problems of complex deployment, long viewing time delay, inconvenience for a hadoop platform to access and the like in the traditional log management scheme can be solved.
Description
Technical Field
The invention relates to a computer application technology, in particular to a log management method and a log management system.
Background
At present, most of company IT systems are composed of a plurality of mutually independent software systems, wherein log management becomes the key work of software operation and maintenance management, and logs can objectively reflect the real operation condition of the software systems, thereby providing powerful support for software performance tuning and troubleshooting. As software systems have increased, logs have been stored in a distributed manner within the computer in which the software systems reside, and managing these separate logs has become complicated. Currently, a log centralized management method based on a distributed technology is mostly adopted to manage logs in a centralized manner, and it is seen from the whole that logs generated by each software system are located in a unified logical view and have a unified access entry, and actually, log data are distributed and stored on different storage nodes, and the specific implementation methods are various.
At present, there are a plurality of widely applied mature schemes for log centralized management, which, although implemented differently, are all composed of a local software system (generating logs), a local log collection component, a log centralized collection storage component, and a web viewing component as shown in fig. 1. The centralized collection and storage component of the log also applies a distributed storage technology to store the log on different storage nodes in a distributed manner, but the log is stored in a centralized manner when viewed from the whole log system.
In the above prior art, a distributed file system is not installed in a local operating system in which a software system is located, logs generated by the software system are stored in a local disk and need to be transmitted to a centralized storage system through a log collection component, which is completed by an application program on the operating system, the log collection component needs to configure from which source file the logs are read and then send to which log centralized collection storage component, where a large number of configurations are involved, which increases complexity, and in addition, the above process needs to be repeated for each software system generating the logs, which obviously is more complicated. And the centralized storage component adopts a distributed storage technology, and also needs a large amount of configuration. In order to reduce such complicated configuration, a configuration tool is often customized to complete the configuration, and developing such a configuration tool itself increases many development difficulties and time.
In addition, by adopting the log management scheme, a certain time is required for the log collection component to synchronize the logs in the local operating system environment to the log centralized storage component, so that for the web viewing component for viewing the log file, a certain delay time is required for viewing the log file. In a big data era, data analysis becomes very important, logs usually contain important information such as user access behaviors, log data stored in a centralized manner are usually processed and analyzed by using a hadoop platform, and based on the log management scheme, when the hadoop platform processes the log data, the data in a centralized storage component needs to be copied into hdfs first, and then the log data copied into hdfs is processed, so that the storage space is consumed, and the complexity of a system is increased.
Therefore, the problems that the conventional log management scheme is complex in deployment and configuration, long in viewing time delay, inconvenient for the hadoop platform to process data and the like exist.
Disclosure of Invention
In view of the above, the main purpose of the present invention is to provide a log management method and system, which can solve the problems of complex deployment, long viewing time delay, and inconvenience for a hadoop platform to access in the conventional log management scheme.
In order to achieve the purpose, the technical scheme provided by the invention is as follows:
a log management method, comprising:
configuring a mount point of a Lustre file system on equipment where a software system is located in advance;
after the software system generates a log file, writing the log file into a storage node OST of the Lustre file system through the mounting point configured on the device where the software system is located;
after a log server receives a log viewing request of a user, utilizing a device S where a software system to which a log to be queried indicated by the log viewing request belongsiIP address of, the user being at the device SiThe user name and the password enter the device SiBy said device SiAnd the mounted point configured on the system acquires the log file which needs to be checked by the user from the corresponding storage node of the Lustre file system and sends the log file to the user.
A log management system, comprising: software system equipment, a log server and a Lustre file system;
the software system equipment is used for writing the log file into a storage node OST of the Lustre file system through a mounting point of the Lustre file system configured on the equipment after the log file is generated by the software system;
a log server for receiving the user's dayAfter the log viewing request, utilizing the device S of the software system to which the log to be queried indicated by the log viewing request belongsiIP address of, the user being at the device SiThe user name and the password enter the device SiBy said device SiAnd the mounted point configured on the system acquires the log file which needs to be checked by the user from the corresponding storage node of the Lustre file system and sends the log file to the user.
An apparatus for log management, comprising:
a memory; and
a processor coupled to the memory, the processor configured to perform any of the methods described above based on instructions stored in the memory.
A computer-readable storage medium, on which a computer program is stored, which program, when being executed by a processor, is adapted to carry out the method of any of the preceding claims.
In summary, the journal management method and system provided by the invention introduce the Lustre file system to perform centralized management on the journals on the software system devices, and the journals of the software systems are not required to be stored on the devices, but are directly stored on the servers of the Lustre file system, so that the problems of complex deployment, long viewing time delay, inconvenience for the convenient access of the hadoop platform and the like in the traditional journal management scheme can be solved.
Drawings
FIG. 1 is a schematic diagram of a prior art log management scheme;
FIG. 2 is a schematic flow chart of a method according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of a system configuration according to an embodiment of the present invention; .
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention will be described in further detail with reference to the accompanying drawings and specific embodiments.
The core idea of the invention is as follows: the method has the advantages that the lustre file system is introduced, log files of the software system are directly written into a server of the lustre file system, and the lustre file system is used for centrally managing the log files, so that deployment complexity caused by the need of configuring source files and collecting and storing components in a log centralized mode in the existing scheme can be avoided, log viewing time delay can be reduced, in addition, a hadoop platform can directly access data on the lustre file system, log data do not need to be copied into the hdfs system, and therefore occupation of storage space of the hdfs system can be avoided.
Fig. 2 is a schematic flowchart of a first embodiment of the present invention, and as shown in fig. 1, a log management method implemented in this embodiment mainly includes:
It should be noted here that the Lustre file system is a storage architecture facing a large-scale cluster, conforms to the GNU GPL v2 protocol based on an open source cluster parallel file system of a Linux platform, has the greatest characteristics of high extensibility and high performance, and can support tens of thousands of client systems, PB-level storage capacity, and GB/s-level I/O throughput. The Lustre file system is used in a large number of top-level supercomputer systems around the world.
The Lustre file system is a distributed file system, and can effectively solve the storage and management problems of data: a certain file system fixed at a certain place is expanded to any multiple places/multiple file systems, and a plurality of nodes form a file system network. Each node may be distributed at different locations, with communication and data transfer between nodes over the network. When the distributed file system is used, a user does not need to care which node the data is stored on or obtained from, and only needs to configure the mount point of the Lustre file system on the local device (the Lustre client needs to be installed when the mount point of the Lustre file system is configured), so that the data in the Lustre file system can be managed and stored just like using the local file system by using the mount point of the Lustre file system installed locally.
In the step, the mounting point of the Lustre file system is configured on the device where the software system is located in advance, and then the log file of the software system can be stored into the storage node of the Lustre file system by using the mounting point, so that, the journal files of all software systems can be uniformly stored and managed by the Lustre file system, and therefore, the complexity of the prior art solutions that require configuration source files and centralized collection of log storage components can be avoided, and, when the user views the log, the user can directly inquire and read the log stored in the Lustre file system through the mounting point of the equipment where each software system is positioned, thus, for the hadoop platform, the log data on the lustre file system can be directly accessed without copying the log data into the hdfs system, therefore, the occupation of the storage space of the hdfs system can be effectively avoided, and the log viewing time delay is reduced.
Specifically, the Lustre file system includes a management node MDS and a storage node OST, and the construction of the Lustre file system and the configuration of mount points of the Lustre file system are all grasped by those skilled in the art. For example, the following method can be specifically adopted to implement the configuration of the management node, the storage node and the mount point of the Lustre file system.
1. Installation management node MDS example:
firstly, a lustre kernel module is installed on a linux system by adopting the following instructions:
modprobe lustre
modprobe ldiskfs
modprobe lnet'network＝"tcp0(eth0)"'
then, a free disk partition is formatted as a lustre partition, such as select/dev/sdb 1 as lustre partition, and the command statements are as follows:
mkfs.lustre--fsname＝log--mdt--mgs/dev/sdb1
finally, mount the partition into a directory, such as/mnt/lustre, with the command statements as follows:
mount.lustre/dev/sdb1/mnt/lustre
the installation of the MDS nodes can be completed through the processes.
2. Install storage node OST example:
the storage node OST is used for storing actual data in the lustre and can be mounted in a lustre file system through an mkfs lustre command, and the command statements are as follows:
mkfs.lustre--fsname＝log--ost--mgsnode＝192.168.1.247@tcp0/dev/sdb1
wherein 192.168.1.247 is the IP address of the MDS, and/dev/sdb 1 is the partition directory of the MDS.
3. Configuration of mount point example of Lustre file system:
the lustre file system is mounted in the local operating system of the application program through the following command, so that the application program can directly write log information into the/mnt directory.
mount.lustre 192.168.1.247@tcp0:/log/mnt
The above is merely an example of the construction of the Lustre file system and the mount point configuration of the Lustre file system, and is provided for reference only.
In this step, after the software system generates the log file, the file access API provided by the operating system may be directly or indirectly called to write the file, and since the Lustre file system is locally installed, the operating system writes the log data written by the software system into the Lustre OST node according to the Lustre file system access protocol, thereby implementing the storage and management of the log file by the Lustre file system.
In this step, when the user needs to check the software systemWhen the log file is unified, the log server accesses the device S where the log to be checked is locatediTriggering its remote access to the device SiIn order to use the apparatus SiAnd uploading the configured mounting points to obtain the log to be checked. It should be noted here that the equipment S is a deviceiThe mount point of the Lustre file system is configured, so that although the log file of the software system installed thereon is actually saved in the storage node of the Lustre file system, the log file saved in the storage node of the Lustre file system can be viewed as if the local file is accessed by using the mount point.
Specifically, the user can access the log server through a web browser to realize interaction with the log server.
Preferably, the following method can be adopted, by means of said device SiThe configured mount point acquires the log which needs to be queried by the user from the corresponding storage node of the Lustre file system and sends the log to the user:
the log server connects the device SiThe log catalog under the mounting point is notified to the user, and the user is triggered to select the log to be checked;
the log server triggers the device S according to the selection result of the useriAnd acquiring the log file which needs to be checked by the user from the corresponding storage node of the Lustre file system, and sending the acquired log file to the user.
Fig. 3 shows a log management system corresponding to the above method, and as shown in fig. 3, the system mainly includes: software system equipment, a log server and a Lustre file system.
The software system equipment is used for writing the log file into a storage node OST of the Lustre file system through a mount point of the Lustre file system configured on the equipment after the log file is generated by the software system.
A log server for receiving a log viewing request from a user and using the log viewing requestDevice S for displaying the location of the software system to which the log to be queried belongsiIP address of, the user being at the device SiThe user name and the password enter the device SiBy said device SiAnd the mounted point configured on the system acquires the log file which needs to be checked by the user from the corresponding storage node of the Lustre file system and sends the log file to the user.
Preferably, the log server is configured to interact with the user through a web browser of the user.
Preferably, the log server is configured to store the device SiThe log catalog under the mounting point is notified to the user, and the user is triggered to select the log to be checked; triggering the device S according to the selection result of the useriAnd acquiring the log file which needs to be checked by the user from the corresponding storage node of the Lustre file system, and sending the acquired log file to the user.
According to the scheme, the invention can obtain the following technical effects:
a. saving the storage space:
for the whole system, log data is stored only by lustre, and the storage space is saved.
b. And the complexity of installation and deployment is reduced:
the lustre file system is directly installed on the operating system, the deployment complexity caused by the traditional method is omitted, and the performance and the system stability are favorably improved due to the fact that few interactive nodes are provided.
c. Being convenient for being integrated with Hadoop:
the Hadoop can directly access the lustre file system without copying the log file to hdfs, so that the storage space is saved.
d. And (3) fully utilizing a linux tool to process the log:
the log is written into the equipment for mounting the lustre file system, and after logging in the equipment, the log processing can be carried out by using a tool carried by linux, files managed on the lustre file system are not different from files managed by a local disk from the view point of an operating system, and the operating system provides a uniform access interface, so that various tools with rich functions under linux can be normally used.
In summary, the above description is only a preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (6)
1. A log management method, comprising:
configuring a mount point of a Lustre file system on equipment where a software system is located in advance;
after the software system generates a log file, writing the log file into a storage node OST of the Lustre file system through the mounting point configured on the device where the software system is located;
after a log server receives a log viewing request of a user, utilizing a device S where a software system to which a log to be queried indicated by the log viewing request belongsiIP address of, the user being at the device SiThe user name and the password enter the device SiThe device SiThe log catalog under the mounting point is notified to the user, and the user is triggered to select the log to be checked; triggering the device S according to the selection result of the useriAnd acquiring the log file which needs to be checked by the user from the corresponding storage node of the Lustre file system, and sending the acquired log file to the user.
2. The method of claim 1, further comprising:
the user interacts with the log server through a web browser.
3. A log management system, comprising: software system equipment, a log server and a Lustre file system; wherein the content of the first and second substances,
the software system equipment is used for writing the log file into a storage node OST of the Lustre file system through a mounting point of the Lustre file system configured on the equipment after the log file is generated by the software system;
the log server is used for receiving a log viewing request of a user, and then utilizing the equipment S of the software system where the log to be inquired indicated by the log viewing request belongsiIP address of, the user being at the device SiThe user name and the password enter the device SiThe device SiThe log catalog under the mounting point is notified to the user, and the user is triggered to select the log to be checked; triggering the device S according to the selection result of the useriAnd acquiring the log file which needs to be checked by the user from the corresponding storage node of the Lustre file system, and sending the acquired log file to the user.
4. The system of claim 3, wherein the log server is configured to interact with the user via a web browser of the user.
5. An apparatus for log management, comprising:
a memory; and
a processor coupled to the memory, the processor configured to perform the method of any of claims 1-2 based on instructions stored in the memory.
6. A computer-readable storage medium, on which a computer program is stored, which program, when being executed by a processor, is adapted to carry out the method of any one of claims 1-2.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610649483.7A CN107733667B (en) | 2016-08-10 | 2016-08-10 | Log management method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610649483.7A CN107733667B (en) | 2016-08-10 | 2016-08-10 | Log management method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107733667A CN107733667A (en) | 2018-02-23 |
| CN107733667B true CN107733667B (en) | 2021-05-25 |
Family
ID=61199990
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610649483.7A Active CN107733667B (en) | 2016-08-10 | 2016-08-10 | Log management method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107733667B (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102546780A (en) * | 2011-12-28 | 2012-07-04 | 山东大学 | Operation method for file distributed storage based on thin client |
| CN103747064A (en) * | 2013-12-26 | 2014-04-23 | 广东中科遥感技术有限公司 | Mounting method, client and system based on MooseFS distributed file system |
| CN105138615A (en) * | 2015-08-10 | 2015-12-09 | 北京思特奇信息技术股份有限公司 | Method and system for building big data distributed log |
| CN105183868A (en) * | 2015-09-15 | 2015-12-23 | 浪潮(北京)电子信息产业有限公司 | Storage method and device for logs of file system |
| CN105224256A (en) * | 2015-10-13 | 2016-01-06 | 浪潮(北京)电子信息产业有限公司 | A kind of storage system |
| CN105260377A (en) * | 2015-09-01 | 2016-01-20 | 浪潮(北京)电子信息产业有限公司 | Updating method and system based on hierarchical storage |
| CN105430096A (en) * | 2015-12-22 | 2016-03-23 | 曙光信息产业(北京)有限公司 | Automatic installation method and device for parallel file system |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120310892A1 (en) * | 2004-12-21 | 2012-12-06 | Dam Tru Q | System and method for virtual cluster file server |
-
2016
- 2016-08-10 CN CN201610649483.7A patent/CN107733667B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102546780A (en) * | 2011-12-28 | 2012-07-04 | 山东大学 | Operation method for file distributed storage based on thin client |
| CN103747064A (en) * | 2013-12-26 | 2014-04-23 | 广东中科遥感技术有限公司 | Mounting method, client and system based on MooseFS distributed file system |
| CN105138615A (en) * | 2015-08-10 | 2015-12-09 | 北京思特奇信息技术股份有限公司 | Method and system for building big data distributed log |
| CN105260377A (en) * | 2015-09-01 | 2016-01-20 | 浪潮(北京)电子信息产业有限公司 | Updating method and system based on hierarchical storage |
| CN105183868A (en) * | 2015-09-15 | 2015-12-23 | 浪潮(北京)电子信息产业有限公司 | Storage method and device for logs of file system |
| CN105224256A (en) * | 2015-10-13 | 2016-01-06 | 浪潮(北京)电子信息产业有限公司 | A kind of storage system |
| CN105430096A (en) * | 2015-12-22 | 2016-03-23 | 曙光信息产业(北京)有限公司 | Automatic installation method and device for parallel file system |
Non-Patent Citations (1)
| Title |
|---|
| lustre体系结构;海岛;《检索自互联网：<url:http://blog.sina.com.cn/s/blog_6cd476240100tu39.html>》;20110719;全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107733667A (en) | 2018-02-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11178234B1 (en) | Method and apparatus for web based storage on-demand distribution | |
| CN107818112B (en) | Big data analysis operating system and task submitting method | |
| US9288266B1 (en) | Method and apparatus for web based storage on-demand | |
| CN104714946A (en) | Large-scale Web log analysis system based on NoSQL | |
| US9106677B2 (en) | Method and apparatus for web based storage on-demand | |
| CN111538590A (en) | Distributed data acquisition method and system based on CS framework | |
| KR102092921B1 (en) | Simulation apparatus and Distributed simulation system | |
| CN111274223A (en) | One-key deployment big data and deep learning container cloud platform and construction method thereof | |
| US11500755B1 (en) | Database performance degradation detection and prevention | |
| CN116263694A (en) | Warehouse cluster deployment method and device and computing equipment | |
| CN114356521A (en) | Task scheduling method and device, electronic equipment and storage medium | |
| CN115357198B (en) | Mounting method and device of storage volume, storage medium and electronic equipment | |
| CN111683091A (en) | Method, device, equipment and storage medium for accessing cloud host console | |
| CN107733667B (en) | Log management method and system | |
| Khalel et al. | Enhanced load balancing in kubernetes cluster by minikube | |
| US20150172131A1 (en) | Method and system for verifying quality of server | |
| AU2021268828B2 (en) | Secure data replication in distributed data storage environments | |
| CN111385334B (en) | Data distribution method, device, equipment and storage medium | |
| US20140122676A1 (en) | Method and Apparatus For Web Based Storage On Demand | |
| CN113407504B (en) | Data processing method, user space file system and storage medium | |
| US11924031B2 (en) | Highly scalable container network interface operation to reduce startup overhead of functions | |
| US20210383006A1 (en) | Distribution of user specific data elements in a replication environment | |
| US10318155B1 (en) | Mechanism for non-disruptive virtual tape drives removal and reconfiguration | |
| CN117459376A (en) | Log collection method, device, equipment and readable storage medium | |
| Raghav | Improving response time for Geographically distributed caches using query result caching |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |