CN112306744A - Log storage backup method, device, server and medium - Google Patents

Log storage backup method, device, server and medium Download PDF

Info

Publication number
CN112306744A
CN112306744A CN201910683252.1A CN201910683252A CN112306744A CN 112306744 A CN112306744 A CN 112306744A CN 201910683252 A CN201910683252 A CN 201910683252A CN 112306744 A CN112306744 A CN 112306744A
Authority
CN
China
Prior art keywords
log
nodes
level
node
target
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201910683252.1A
Other languages
Chinese (zh)
Other versions
CN112306744B (en
Inventor
吴会堂
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Zhejiang Uniview Technologies Co Ltd
Original Assignee
Zhejiang Uniview Technologies Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Zhejiang Uniview Technologies Co Ltd filed Critical Zhejiang Uniview Technologies Co Ltd
Priority to CN201910683252.1A priority Critical patent/CN112306744B/en
Publication of CN112306744A publication Critical patent/CN112306744A/en
Application granted granted Critical
Publication of CN112306744B publication Critical patent/CN112306744B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/14—Error detection or correction of the data by redundancy in operation
    • G06F11/1402—Saving, restoring, recovering or retrying
    • G06F11/1446—Point-in-time backing up or restoration of persistent data
    • G06F11/1448—Management of the data involved in backup or backup restore
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/0608—Saving storage space on storage systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0638—Organizing or formatting or addressing of data
    • G06F3/0643—Management of files
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/0671—In-line storage system
    • G06F3/0683—Plurality of storage devices
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Quality & Reliability (AREA)
  • Debugging And Monitoring (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The embodiment of the invention discloses a log storage backup method, a device, a server and a medium. Wherein, the method comprises the following steps: if the log generation event is detected, determining the target log grade and the target function type of the new log; determining a target node associated with the new log according to the target log grade and the target function type of the new log and the log grade and the function type associated with the nodes in the cluster; the log grades and the function types associated with the nodes are determined according to the number of the cluster nodes, the number of the grade grades of the log files and the number of the function types of the nodes; the new log is stored in the target node associated with the new log. According to the technical scheme, the newly generated log files are stored in each cluster node in a scattered mode according to the log grades and the function types, and therefore the space of a system disk occupied by the logs is saved.

Description

Log storage backup method, device, server and medium
Technical Field
The embodiment of the invention relates to the technical field of log processing of cloud storage systems, in particular to a log storage backup method, a log storage backup device, a server and a medium.
Background
At present, a general cloud storage system is composed of several or even dozens of hundreds of node devices, when nodes of the cloud storage system are more and more, logs can be increased by times, and currently, the general realization is that each node is in a mode of being respectively administrative, each node only manages and records the logs of the node, repeated printing and repeated coverage can exist, when the nodes of the system are more and more, the logs are more and more, the workload of log analysis can be increased by times, and the system is disastrous to system maintenance personnel for log analysis.
In order to facilitate searching, the prior art stores logs with different characteristics or logs of different objects in different files respectively. However, since each storage node in the prior art stores logs of all properties or objects, a large number of duplicate logs are generated, a large amount of system space is occupied, and a phenomenon that useful logs are flushed by being covered by other logs also occurs.
Disclosure of Invention
The embodiment of the invention provides a log storage backup method, a log storage backup device, a server and a medium, which are used for saving the storage space of a system and storing more useful log information.
In a first aspect, an embodiment of the present invention provides a log storage backup method, where the method includes:
if the log generation event is detected, determining the target log grade and the target function type of the new log;
determining a target node associated with the new log according to the target log grade and the target function type of the new log and the log grade and the function type associated with the nodes in the cluster; the log grades and the function types associated with the nodes are determined according to the number of the cluster nodes, the number of the grade grades of the log files and the number of the function types of the nodes;
the new log is stored in the target node associated with the new log.
In a second aspect, an embodiment of the present invention provides a log storage backup apparatus, where the apparatus includes:
the attribute determining module is used for determining the target log grade and the target function type of the new log if the log generating event is detected;
the target node determining module is used for determining a target node associated with the new log according to the target log grade and the target function type of the new log and the log grade and the function type associated with the nodes in the cluster; the log grades and the function types associated with the nodes are determined according to the number of the cluster nodes, the number of the grade grades of the log files and the number of the function types of the nodes;
and the storage module is used for storing the new log in the target node associated with the new log.
In a third aspect, an embodiment of the present invention further provides a server, where the server includes:
one or more processors;
storage means for storing one or more programs;
when executed by the one or more processors, cause the one or more processors to implement the log storage backup method of any of the first aspects.
In a fourth aspect, an embodiment of the present invention further provides a medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the log storage backup method described in any of the first aspects.
According to the log storage backup method, device, server and medium provided by the embodiment of the invention, the target log grade and the target function type of the new log are determined by detecting the log generation time, the target node associated with the new log is determined by the log grade and the function type associated with the cluster nodes, and the new log is stored in the target node associated with the new log. According to the technical scheme of the embodiment, the newly generated log files are stored in the cluster nodes in a scattered manner according to the log grades and the function types, so that the situation that all the nodes repeatedly store the same log is avoided, the space of a system disk occupied by the log is saved, and more useful log information can be stored.
Drawings
Fig. 1 is a flowchart of a log storage backup method according to an embodiment of the present invention;
fig. 2 is a schematic diagram illustrating a node log level initialization according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of a target log generation process according to an embodiment of the present invention;
fig. 4 is a flowchart of another log storage backup method according to an embodiment of the present invention;
FIG. 5 is a diagram illustrating deletion of a history log according to an embodiment of the present invention;
fig. 6 is a flowchart of another log storage backup method according to an embodiment of the present invention;
fig. 7 is a schematic diagram of log backup provided in the embodiment of the present invention;
fig. 8 is a schematic diagram of backup node search according to an embodiment of the present invention;
fig. 9 is a block diagram of a log storage backup apparatus according to an embodiment of the present invention;
fig. 10 is a schematic structural diagram of a server according to an embodiment of the present invention.
Detailed Description
The embodiments of the present invention will be described in further detail with reference to the drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the embodiments of the invention and that no limitation of the invention is intended. It should be further noted that, for convenience of description, only some structures, not all structures, relating to the embodiments of the present invention are shown in the drawings.
Fig. 1 is a flowchart of a log storage backup method according to an embodiment of the present invention, which is applicable to a situation of performing storage backup on a log. Typically, the method can be applied to log storage backup in a cloud storage system and is executed by a management node in a cluster. The method can be executed by the log storage backup device provided by the embodiment of the invention, and the device can be realized in a hardware and/or software mode. Referring to fig. 1, the method may specifically include:
and step 110, if the log generation event is detected, determining the target log level and the target function type of the new log.
The log is key information for recording hardware, software and system problems in the system, and can monitor events occurring in the system. The user and the maintenance personnel can check the reason of the system error through the log or look for traces left by an attacker when the system is attacked.
The log level typically includes hierarchical methods such as info, warning, error, critical, etc. Regardless of Linux, windows or other systems, system logs are classified and recorded in a grading mode, different grades represent different problems, and a system administrator confirms the operation condition and the problem reason of the system by checking and analyzing the logs of the grades.
Specifically, each node in the cluster includes multiple functional categories including, but not limited to, disk, array, logic resource, OSD, MON, alarm, etc. Each generated log corresponds to a respective log level and function type, and when a log file generation event is detected, a target level and a target function type of a new log are determined.
And step 120, determining a target node associated with the new log according to the target log level and the target function type of the new log and the log level and the function type associated with the node in the cluster.
And determining the log grade and the function type associated with the node according to the number of the cluster nodes, the number of the log file grades and the number of the function types of the node.
Specifically, determining the log level and the function type associated with the node according to the number of the cluster nodes, the number of the log file levels and the number of the function types of the node includes: determining the log grade associated with the node according to the number of the cluster nodes and the number of the grade of the log file; and determining the function type associated with the node according to the function type number of the node and the log level associated with the node.
Determining the log level associated with the nodes according to the number of the cluster nodes and the number of the log file levels comprises the following steps: the cluster node sets with the number being the maximum integral multiple of the number of the log file levels are distributed to each log file level to obtain the log levels associated with the nodes; and distributing other nodes except the cluster node set to the log file level according to the importance degree of the log file level so as to obtain the log levels associated with other nodes.
Specifically, the log levels may include a first level, a second level, a third level, and a fourth level from low to high in importance.
In this embodiment, all nodes have at least one role in four levels, and if the number of nodes is N, if N is an integral multiple of the number of log levels, the four log levels are respectively and equally distributed to each node, that is, each node is associated with one level role.
Illustratively, if the number of the nodes is 8, and the number of the nodes is 2 times of the number of the log levels at this time, the four log levels, i.e., the first level, the second level, the third level, and the fourth level, are respectively and equally distributed to each node, and at this time, each node is correspondingly associated with one log level, and each log level is respectively corresponding to two log nodes.
And if the N is not the maximum integral multiple of the number of the log levels, distributing the cluster nodes of which the number is the maximum integral multiple of the number of the log file levels to each log file level to obtain the log level associated with the node. And distributing other nodes except the cluster node set to the log file level according to the importance degree of the log file level so as to obtain the log levels associated with other nodes.
For example, if the number of nodes is 9, the log levels are allocated according to the above-mentioned average division principle for 8 cluster nodes whose number is the maximum integral multiple of the number of the log file levels, and all the log levels are allocated to the remaining 1 node.
For example, when the number of log levels is four, if the number N of nodes is not an integer multiple of 4, the number of other nodes may be one, two, or three. If the number of the other nodes is one, all log grades are allocated to the other nodes; if the number of the other nodes is two, assigning a first level and a second level to one node, and assigning a third level and a fourth level to the other node; if the number of the other nodes is three, the first grade and the second grade are assigned to one node, the third grade is assigned to one node, and the fourth grade is assigned to another node.
Initializing cluster nodes by using the log grades according to the obtained log grades associated with the nodes, specifically referring to fig. 2, judging whether the current node is endowed with the grades for the first time, and if not, endowing the current node with the log grades according to the log grades associated with the current node; if the log level of the current node is distributed, judging whether all the log levels have nodes, if so, finishing the level initialization of the nodes, if not, further checking which level has no nodes at present, and endowing the corresponding nodes with the log levels according to the log file levels associated with the nodes.
After determining the log file level associated with the node, determining the function type associated with the node according to the function type number of the node and the log level associated with the node, so as to obtain a storage node of a new log.
Specifically, determining the function type associated with the node according to the function type number of the node and the log level associated with the node includes: generating log files of different functional types of each log file level according to the functional type number and the log file level number of the nodes; and distributing the log files of different functional types of the log file level to the nodes associated with the log level to obtain the functional types associated with the nodes.
Distributing the log files of different functional types of the log file level to the nodes associated with the log level, including: and distributing the log files of different functional types of the log file level to the nodes associated with the log file level.
For example, it is assumed that the number of nodes of the log is 4, the number of function types of each node is 8, and the number of log level is 4, where each node corresponds to a different log level, and each log file level generates 8 log files of different function types. When the log files are distributed, the log files of different functional types at the same level are distributed to the nodes related to the log level by adopting a level priority principle, namely the log files of 8 different functional types generated by the first level are distributed to the nodes with the first level of level roles.
If the number of the nodes associated with the same log level is multiple, the log files of different functional types generated at the same level are equally distributed to all the associated nodes, so that the distributed storage of the log files is realized, the log files of the same level are ensured to be stored on the nodes associated with the log file level, and the searching by operation and maintenance personnel is facilitated.
Referring to fig. 3, when detecting that a log file of a certain function type is generated, first, it needs to be determined whether all log files of each level of the function are generated, and if all log files are generated, the log file of the function type is not generated, and a log backup process is performed; if not, continuously checking which level of the log of the function type is not generated, searching the node of the corresponding level, further judging whether the node of the corresponding level generates the log of the same level of the function, if not, generating a log file of the function, and if so, entering a log backup process.
Step 130, storing the new log in the target node associated with the new log.
Correspondingly, the generated new log is stored in the target node associated with the new log.
According to the technical scheme provided by the embodiment of the invention, the target log grade and the target function type of the new log are determined by detecting the log generation event, the target node associated with the new log is determined by the log grade and the function type associated with the nodes in the cluster, and the new log is stored in the target node associated with the new log. According to the technical scheme of the embodiment, the newly generated log files are stored in the cluster nodes in a scattered manner according to the log grades and the function types, so that the situation that all the nodes repeatedly store the same log is avoided, the space of a system disk occupied by the log is saved, and more useful log information can be stored. In addition, by means of the distributed storage of the log files, log write-in flow impact caused by long-term high-pressure log write-in of the system disks of all the nodes is avoided, the service life of the system disks is prolonged, all logs of all the nodes are stored in a classified mode, operation and maintenance personnel can find effective information timely, and the efficiency of positioning problems is improved.
Fig. 4 is a flowchart of another log storage backup method according to an embodiment of the present invention, where on the basis of the foregoing embodiment, the present embodiment further includes, after step 120: determining whether to delete the log file stored on the target node or not according to the residual storage space of the target node associated with the new log; and if the residual storage space on the target node is smaller than the preset threshold, determining the log file to be deleted in the target node and deleting the log file according to the grade of the log file stored on the target node and the storage duration of the log file. Referring to fig. 4, the method may specifically include:
step 210, if a log generation event is detected, determining a target log level and a target function type of the new log.
Step 220, determining a target node associated with the new log according to the target log grade and the target function type of the new log and the log grade and the function type associated with the node in the cluster;
and determining the log grade and the function type associated with the node according to the number of the cluster nodes, the number of the log file grades and the number of the function types of the node.
And step 230, determining whether to delete the log file stored on the target node according to the remaining storage space of the target node associated with the new log.
In this embodiment, the storage space of the cluster node is limited, and when the remaining storage space of the node is insufficient, the log file already stored on the node needs to be deleted, so as to provide a storage space for the newly generated log file.
And step 250, if the residual storage space on the target node is smaller than a preset threshold, determining the log file to be deleted in the target node according to the grade of the log file stored on the target node and the storage duration of the log file, and deleting the log file.
Specifically, when the number of nodes is small, some nodes may store logs of multiple functional types or multiple levels, and when the usage rate of the log space reaches or exceeds a certain proportion, the historical logs of a lower level are deleted in sequence from old to new so as to make room for the logs of a higher level.
Referring to fig. 5, firstly, whether to delete the history log is judged according to the remaining storage space of the target node, if the storage space of the target node does not exceed the storage threshold, a new log is generated, a log backup process is started, and whether to generate a backup log is checked. If the storage space of the target node exceeds the storage threshold, checking whether other grades of logs exist on the target node, and if not, deleting the same grade of log with the longest storage time; if yes, judging whether the logs of other levels stored on the target node are lower than the current log level, if yes, deleting a low-level log file compression packet with the longest storage time; if not, continuously checking whether the same-level backup log exists on the current node, and if so, deleting the same-level backup log with the longest storage duration; if not, deleting the log file compression packet or the high-level backup log with the longest storage time at the same level.
According to the grade and the storage time length of the log files stored on the target node, the log files with low grade and long storage time length are selected for deletion, and the storage period of the important log files with high grade can be ensured.
Step 250, storing the new log in the target node associated with the new log.
According to the technical scheme of the embodiment of the invention, the log file to be deleted in the target node is determined according to the residual space of the target plus point associated with the new log, the stored log file grade of the target node and the storage time of the log file, and the low-grade historical log with long storage time is deleted, so that the storage period of the important log is ensured, the risk that the important log is covered by the non-important log is avoided, and the maintainer can acquire effective information more quickly.
Fig. 6 is a flowchart of another log storage backup provided by an embodiment of the present invention. On the basis of the above embodiment, the present embodiment further includes, after step 250: and if the number of the backup logs of the target log file level is less than the backup log threshold value, generating a backup log of a new log. Referring to fig. 6, the method may specifically include the following steps:
and step 310, if the log generation event is detected, determining the target log level and the target function type of the new log.
And step 320, determining a target node associated with the new log according to the target log level and the target function type of the new log and the log level and the function type associated with the node in the cluster.
And step 330, determining whether to delete the log file stored on the target node according to the remaining storage space of the target node associated with the new log.
And 340, if the residual storage space on the target node is smaller than a preset threshold, determining a log file to be deleted in the target node according to the grade of the log file stored on the target node and the storage duration of the log file, and deleting the log file.
Step 350, storing the new log in the target node associated with the new log.
And step 360, if the number of the backup logs of the target log file level is less than the backup log threshold value, generating a backup log of a new log.
The backup log threshold value is determined according to the number of cluster nodes, the number of log file grades and the number of functional types of the nodes.
The backup log threshold of the logs of different levels is different according to the number of cluster nodes, the number of log files and the number of functional types of the nodes.
Specifically, determining the backup log threshold according to the number of cluster nodes, the number of log file levels, and the number of functional types of the nodes includes:
if the number of the cluster nodes is less than the number of the log file grades, the log files of the first grade are not backed up, and the log files of the other grades are backed up by one copy;
if the number of the cluster nodes is more than or equal to the number of the log file grades and less than or equal to the number of the functional types of the nodes, backing up two log files of a fourth grade, and backing up one log file of each of the other grades;
if the number of the cluster nodes is larger than or equal to the number of the functional types of the nodes, one part of the log file of the first level and one part of the log file of the second level are backed up, and two parts of the log file of the third level and one part of the log file of the fourth level are backed up.
And starting a log backup process according to the backup number of the log files at each level, wherein a specific process refers to fig. 7, and after a new log is generated, whether a backup file is generated is judged according to the level and the node number of the new log. If the number of the cluster nodes is less than the number of the grades of the log files, and the grade of the newly generated log is the first grade, no backup file is generated; if not, further judging whether a backup file needs to be generated according to the number of backup logs needing to be generated by the corresponding level logs, entering a backup node searching process after the backup file is generated, and storing the generated backup file by using the searched backup node.
Specifically, the backup log node of the new log is selected from nodes associated with other log levels except the target log level.
With reference to fig. 8, fig. 8 is a flowchart for searching for a backup node according to an embodiment of the present invention. As shown in fig. 8, the backup node is preferentially stored in the node whose log space is empty, and if there is no node whose log space is empty, the backup file is stored in the node associated with the other log level different from the target log level. More specifically, the node associated with the other log level with the largest remaining storage space may be selected to store the backup log, and if the storage spaces of the other log levels are the same, the backup node is continuously selected according to the storage conditions of the backup logs at the other levels. By dispersedly storing the backup logs on other logs with different grades from the target log, when some nodes in the cluster nodes have problems, the log backup files on the deleted nodes can be found on other nodes, so that the safety of the log files is guaranteed.
According to the technical scheme provided by the embodiment of the invention, the backup log of the new log is generated by judging the number of the backup logs of the target log file level and the size of the backup log threshold value. And determining a backup log threshold value according to the number of cluster nodes, the number of log file levels and the number of functional types of the nodes, and performing backup on a plurality of nodes on the important log to ensure the safety of important log information and avoid the loss of the important information caused by the downtime of a certain node.
Fig. 9 is a block diagram of a log storage backup apparatus according to an embodiment of the present invention. As shown in fig. 9, the apparatus may include:
an attribute determining module 410, configured to determine, if a log generation event is detected, a target log level and a target function type of the new log;
a target node determining module 420, configured to determine a target node associated with the new log according to the target log level and the target function type of the new log, and the log level and the function type associated with the node in the cluster; the log grades and the function types associated with the nodes are determined according to the number of the cluster nodes, the number of the grade grades of the log files and the number of the function types of the nodes;
and a storage module 430, configured to store the new log in a target node associated with the new log.
According to the technical scheme provided by the embodiment of the invention, the target log grade and the target function type of the new log are determined by detecting the log generation time, the target node associated with the new log is determined by the log grade and the function type associated with the nodes of the cluster, and the new log is stored in the target node associated with the new log. According to the technical scheme of the embodiment, the newly generated log files are stored in the cluster nodes in a scattered manner according to the log grades and the function types, so that the situation that all the nodes repeatedly store the same log is avoided, the space of a system disk occupied by the log is saved, and more useful log information can be stored.
On the basis of the above-described embodiment, the target node determining module 420 includes a log level determining unit and a function category determining unit.
Wherein the log level determination unit is configured to: and determining the log level associated with the nodes according to the number of the cluster nodes and the number of the log file levels.
The function type determination unit is configured to: and determining the function type associated with the node according to the function type number of the node and the log level associated with the node.
On the basis of the foregoing embodiment, the log level determination unit is specifically configured to: the cluster node sets with the number being the maximum integral multiple of the number of the log file levels are distributed to each log file level to obtain the log levels associated with the nodes;
and distributing other nodes except the cluster node set to the log file level according to the importance degree of the log file level so as to obtain the log levels associated with other nodes.
The log grades comprise a first grade, a second grade, a third grade and a fourth grade from low to high according to the importance degree;
correspondingly, allocating other nodes except the cluster node set to the log file level according to the importance degree of the log file level, including:
if the number of the other nodes is one, all log grades are allocated to the other nodes;
if the number of the other nodes is two, assigning a first level and a second level to one node, and assigning a third level and a fourth level to the other node;
if the number of the other nodes is three, the first grade and the second grade are assigned to one node, the third grade is assigned to one node, and the fourth grade is assigned to another node.
The function type determination unit is specifically configured to: generating log files of different functional types of each log file level according to the functional type number and the log file level number of the nodes;
and distributing the log files of different functional types of the log file level to the nodes associated with the log level to obtain the functional types associated with the nodes.
On the basis of the above embodiment, the log storage backup apparatus further includes: the log deleting device comprises a log deleting judging module and a log deleting module.
And the log deletion judging module is used for determining whether to delete the log file stored on the target node according to the residual storage space of the target node associated with the new log.
The log deleting module is used for: and if the residual storage space on the target node is smaller than the preset threshold, determining the log file to be deleted in the target node and deleting the log file according to the grade of the log file stored on the target node and the storage duration of the log file.
On the basis of the above embodiment, the log storage backup device further includes a backup log generation module, configured to generate a backup log of a new log if the number of backup logs of the target log file level is smaller than a backup log threshold; the backup log threshold value is determined according to the number of cluster nodes, the number of log file grades and the number of functional types of the nodes.
The backup log generation module is further specifically configured to: if the number of the cluster nodes is less than the number of the log file grades, the log files of the first grade are not backed up, and the log files of the other grades are backed up by one copy;
if the number of the cluster nodes is more than or equal to the number of the log file grades and less than or equal to the number of the functional types of the nodes, backing up two log files of a fourth grade, and backing up one log file of each of the other grades;
if the number of the cluster nodes is larger than or equal to the number of the functional types of the nodes, one part of the log file of the first level and one part of the log file of the second level are backed up, and two parts of the log file of the third level and one part of the log file of the fourth level are backed up.
On the basis of the above embodiment, the log storage backup apparatus further includes a backup node selection module, configured to select a backup log node of a new log from nodes associated with other log levels except the target log level.
The log storage backup device provided by the embodiment of the invention can execute the log storage backup method provided by any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.
Fig. 10 is a schematic structural diagram of a server according to an embodiment of the present invention, and fig. 10 shows a block diagram of an exemplary server suitable for implementing an embodiment of the present invention. The server shown in fig. 10 is only an example, and should not bring any limitation to the function and the scope of use of the embodiments of the present invention.
As shown in FIG. 10, the server 12 is in the form of a general purpose computing device. The components of the server 12 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 that couples various system components including the system memory 28 and the processing unit 16.
Bus 18 represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example, such architectures include, but are not limited to, Industry Standard Architecture (ISA) bus, micro-channel architecture (MAC) bus, enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.
The server 12 typically includes a variety of computer system readable media. Such media may be any available media that is accessible by server 12 and includes both volatile and nonvolatile media, removable and non-removable media.
The system memory 28 may include computer system readable media in the form of volatile memory, such as Random Access Memory (RAM)30 and/or cache memory 32. The server 12 may further include other removable/non-removable, volatile/nonvolatile computer system storage media. By way of example only, storage system 34 may be used to read from and write to non-removable, nonvolatile magnetic media (not shown in FIG. 10, and commonly referred to as a "hard drive"). Although not shown in FIG. 10, a magnetic disk drive for reading from and writing to a removable, nonvolatile magnetic disk (e.g., a "floppy disk") and an optical disk drive for reading from or writing to a removable, nonvolatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 by one or more data media interfaces. System memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the invention.
A program/utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which examples or some combination thereof may comprise an implementation of a network environment. Program modules 42 generally carry out the functions and/or methodologies of embodiments described herein.
The server 12 may also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), with one or more devices that enable a user to interact with the server 12, and/or with any devices (e.g., network card, modem, etc.) that enable the server 12 to communicate with one or more other computing devices. Such communication may be through an input/output (I/O) interface 22. Also, the server 12 may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network, such as the Internet) via the network adapter 20. As shown in FIG. 10, the network adapter 20 communicates with the other modules of the server 12 via the bus 18. It should be understood that although not shown in the figures, other hardware and/or software modules may be used in conjunction with the server 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, among others.
The processing unit 16 executes various functional applications and data processing by executing programs stored in the system memory 28, for example, implementing a log storage backup method provided by an embodiment of the present invention.
An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program (or referred to as computer-executable instructions) is stored, where the computer program is used for executing a log storage backup method when executed by a processor, and the method includes:
if the log generation event is detected, determining the target log grade and the target function type of the new log;
determining a target node associated with the new log according to the target log grade and the target function type of the new log and the log grade and the function type associated with the nodes in the cluster; the log grades and the function types associated with the nodes are determined according to the number of the cluster nodes, the number of the grade grades of the log files and the number of the function types of the nodes;
the new log is stored in the target node associated with the new log.
Computer storage media for embodiments of the invention may employ any combination of one or more computer-readable media. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Computer program code for carrying out operations for embodiments of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the embodiments of the present invention have been described in more detail through the above embodiments, the embodiments of the present invention are not limited to the above embodiments, and many other equivalent embodiments may be included without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims (12)

1. A method of log storage backup performed by a management node in a cluster, the method comprising:
if the log generation event is detected, determining the target log grade and the target function type of the new log;
determining a target node associated with the new log according to the target log grade and the target function type of the new log and the log grade and the function type associated with the nodes in the cluster; the log grades and the function types associated with the nodes are determined according to the number of the cluster nodes, the number of the grade grades of the log files and the number of the function types of the nodes;
the new log is stored in the target node associated with the new log.
2. The method of claim 1, wherein determining the log level and the function type associated with the node according to the number of cluster nodes, the number of log file levels, and the number of function types of the node comprises:
determining the log grade associated with the node according to the number of the cluster nodes and the number of the grade of the log file;
and determining the function type associated with the node according to the function type number of the node and the log level associated with the node.
3. The method of claim 2, wherein determining the log level associated with the node based on the number of cluster nodes and the number of log file levels comprises:
the cluster node sets with the number being the maximum integral multiple of the number of the log file levels are distributed to each log file level to obtain the log levels associated with the nodes;
and distributing other nodes except the cluster node set to the log file level according to the importance degree of the log file level so as to obtain the log level associated with the other nodes.
4. The method of claim 3, wherein the log levels comprise a first level, a second level, a third level, and a fourth level from low to high in importance;
correspondingly, allocating other nodes except the cluster node set to the log file level according to the importance degree of the log file level, including:
if the number of the other nodes is one, all log grades are allocated to the other nodes;
if the number of the other nodes is two, assigning a first level and a second level to one node, and assigning a third level and a fourth level to the other node;
if the number of the other nodes is three, the first grade and the second grade are assigned to one node, the third grade is assigned to one node, and the fourth grade is assigned to another node.
5. The method of claim 2, wherein determining the class of function associated with the node based on the number of classes of function of the node and the log level associated with the node comprises:
generating log files of different functional types of each log file level according to the functional type number and the log file level number of the nodes;
and distributing the log files of different functional types of the log file level to the nodes associated with the log level to obtain the functional types associated with the nodes.
6. The method according to any one of claims 1 to 5, wherein after determining the target node associated with the new log according to the target log level and the target function type of the new log and the log level and the function type associated with the node in the cluster, the method further comprises:
determining whether to delete the log file stored on the target node or not according to the residual storage space of the target node associated with the new log;
and if the residual storage space on the target node is smaller than the preset threshold, determining the log file to be deleted in the target node and deleting the log file according to the grade of the log file stored on the target node and the storage duration of the log file.
7. The method of claim 1, further comprising, after storing the new log in the target node associated with the new log:
if the number of the backup logs of the target log file level is smaller than the backup log threshold value, generating a backup log of a new log; the backup log threshold value is determined according to the number of cluster nodes, the number of log file grades and the number of functional types of the nodes.
8. The method of claim 7, wherein the log levels comprise a first level, a second level, a third level, and a fourth level from low to high in importance;
correspondingly, determining a backup log threshold according to the number of cluster nodes, the number of log file levels and the number of functional types of the nodes comprises the following steps:
if the number of the cluster nodes is less than the number of the log file grades, the log files of the first grade are not backed up, and the log files of the other grades are backed up by one copy; if the number of the cluster nodes is more than or equal to the number of the log file grades and less than or equal to the number of the functional types of the nodes, backing up two log files of a fourth grade, and backing up one log file of each of the other grades;
if the number of the cluster nodes is larger than or equal to the number of the functional types of the nodes, one part of the log file of the first level and one part of the log file of the second level are backed up, and two parts of the log file of the third level and one part of the log file of the fourth level are backed up.
9. The method of claim 1, wherein after storing the new log in the target node associated with the new log, further comprising:
and selecting the backup log node of the new log from the nodes associated with the other log levels except the target log level.
10. An apparatus for log storage backup, the apparatus comprising:
the attribute determining module is used for determining the target log grade and the target function type of the new log if the log generating event is detected;
the target node determining module is used for determining a target node associated with the new log according to the target log grade and the target function type of the new log and the log grade and the function type associated with the nodes in the cluster; the log grades and the function types associated with the nodes are determined according to the number of the cluster nodes, the number of the grade grades of the log files and the number of the function types of the nodes;
and the storage module is used for storing the new log in the target node associated with the new log.
11. A server, comprising:
one or more processors;
a memory for storing one or more programs;
when executed by the one or more processors, cause the one or more processors to implement the log storage backup method of any of claims 1-9.
12. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out a log storage backup method according to any one of claims 1 to 9.
CN201910683252.1A 2019-07-26 2019-07-26 Log storage backup method, device, server and medium Active CN112306744B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201910683252.1A CN112306744B (en) 2019-07-26 2019-07-26 Log storage backup method, device, server and medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201910683252.1A CN112306744B (en) 2019-07-26 2019-07-26 Log storage backup method, device, server and medium

Publications (2)

Publication Number Publication Date
CN112306744A true CN112306744A (en) 2021-02-02
CN112306744B CN112306744B (en) 2024-02-06

Family

ID=74329857

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201910683252.1A Active CN112306744B (en) 2019-07-26 2019-07-26 Log storage backup method, device, server and medium

Country Status (1)

Country Link
CN (1) CN112306744B (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114579368A (en) * 2022-05-07 2022-06-03 武汉四通信息服务有限公司 Backup management method for continuous data protection, computer equipment and storage medium

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7743030B1 (en) * 2006-09-29 2010-06-22 Emc Corporation Methods and apparatus for declarative log collection
CN103645984A (en) * 2013-12-24 2014-03-19 山东中创软件工程股份有限公司 Method and device for optimizing logs
US20160162366A1 (en) * 2013-08-02 2016-06-09 International Business Machines Corporation A method for a logging process in a data storage system
CN108846070A (en) * 2018-06-07 2018-11-20 北京京东尚科信息技术有限公司 Log processing method, system, electronic equipment and the storage medium of distributed system

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7743030B1 (en) * 2006-09-29 2010-06-22 Emc Corporation Methods and apparatus for declarative log collection
US20160162366A1 (en) * 2013-08-02 2016-06-09 International Business Machines Corporation A method for a logging process in a data storage system
CN103645984A (en) * 2013-12-24 2014-03-19 山东中创软件工程股份有限公司 Method and device for optimizing logs
CN108846070A (en) * 2018-06-07 2018-11-20 北京京东尚科信息技术有限公司 Log processing method, system, electronic equipment and the storage medium of distributed system

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114579368A (en) * 2022-05-07 2022-06-03 武汉四通信息服务有限公司 Backup management method for continuous data protection, computer equipment and storage medium
CN114579368B (en) * 2022-05-07 2022-08-02 武汉四通信息服务有限公司 Backup management method for continuous data protection, computer equipment and storage medium

Also Published As

Publication number Publication date
CN112306744B (en) 2024-02-06

Similar Documents

Publication Publication Date Title
US8676568B2 (en) Information processing apparatus and message extraction method
US8595564B2 (en) Artifact-based software failure detection
US9274714B2 (en) Method and system for managing storage capacity in a storage network
US11520649B2 (en) Storage mounting event failure prediction
US9563360B2 (en) Management of extent migration on tiered storage
JP2017091512A (en) Method, system, and program for reducing reactivation time of services
US10209905B2 (en) Reusing storage blocks of a file system
US10581668B2 (en) Identifying performance-degrading hardware components in computer storage systems
US20230123303A1 (en) Adjusting resources within a hyperconverged infrastructure system based on environmental information
US9213618B2 (en) Storage management systems and methods in hierarchical storage systems
CN112306744B (en) Log storage backup method, device, server and medium
US10365998B2 (en) Modifying monitoring configurations that support analytics programs
JP2017102922A (en) Method, program and processing system for selective retention of data
US9952773B2 (en) Determining a cause for low disk space with respect to a logical disk
US20230236922A1 (en) Failure Prediction Using Informational Logs and Golden Signals
CN110837428A (en) Storage device management method and device
US20220188424A1 (en) Anomaly detection with impact assessment
US9678854B2 (en) Application-centric analysis of leak suspect operations
CN114579357A (en) Servicing data storage devices in a data storage array
CN115237334A (en) Hard disk management method and device and electronic equipment
US20210157507A1 (en) Storage alteration monitoring
US11556446B2 (en) Programmatic performance anomaly detection
US11630754B1 (en) Identification and remediation of memory leaks using rule-based detection of anomalous memory usage patterns
CN113535412B (en) Method, apparatus and computer program product for tracking locks
CN117235043A (en) Database migration method, device, electronic equipment and medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant