CN114036121A - Log file processing method, device, system, equipment and storage medium - Google Patents

Log file processing method, device, system, equipment and storage medium Download PDF

Info

Publication number
CN114036121A
CN114036121A CN202111318978.9A CN202111318978A CN114036121A CN 114036121 A CN114036121 A CN 114036121A CN 202111318978 A CN202111318978 A CN 202111318978A CN 114036121 A CN114036121 A CN 114036121A
Authority
CN
China
Prior art keywords
log file
log
file group
preset
files
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202111318978.9A
Other languages
Chinese (zh)
Inventor
赵宇
刘彬
马欢
侯雪峰
王东
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Kingsoft Cloud Network Technology Co Ltd
Original Assignee
Beijing Kingsoft Cloud Network Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Kingsoft Cloud Network Technology Co Ltd filed Critical Beijing Kingsoft Cloud Network Technology Co Ltd
Priority to CN202111318978.9A priority Critical patent/CN114036121A/en
Publication of CN114036121A publication Critical patent/CN114036121A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/18—File system types
    • G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
    • G06F16/1815—Journaling file systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/30—Monitoring
    • G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
    • G06F11/3466—Performance evaluation by tracing or monitoring
    • G06F11/3476—Data logging
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/17—Details of further file system functions
    • G06F16/172—Caching, prefetching or hoarding of files
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/18—File system types
    • G06F16/182—Distributed file systems

Abstract

The embodiment of the invention provides a log file processing method, a device, a system, equipment and a storage medium, wherein a host in a distributed cluster acquires log files generated by each deployed component; determining a log file group to which each acquired log file belongs; one log file group comprises log files of the same type generated by the same component in the host; and aiming at each log file group, when a preset uploading condition is met, sending the log files in the log file group to a data storage server. The data storage server stores the received log files in the same log file group into the same local storage area in an object storage mode; the preset uploading condition is based on: and presetting the uploading time, and/or determining the size of the log file to be uploaded currently in the log file group. Based on the processing, the log files of the components deployed in the hosts in the distributed cluster can be uniformly managed.

Description

Log file processing method, device, system, equipment and storage medium
Technical Field
The present invention relates to the field of internet technologies, and in particular, to a method, an apparatus, a system, a device, and a storage medium for processing a log file.
Background
With the rapid development of big data technology, distributed clusters based on big data technology are widely applied in various fields. A distributed cluster includes multiple hosts (e.g., servers) in each of which multiple big data technology-based components can be deployed to implement corresponding functionality. For example, an Hbase (distributed database) component is deployed on each host in a distributed cluster to implement the functionality of distributed storage of data.
During the operation of each component deployed in the host, a log file for recording the operation condition of the component is generated. Log files generated by the components may be collected for monitoring the operational status and performance of the components. In the related art, a log file of each component is stored to a host where the component is deployed.
As can be seen, in the related art, the log files of the components deployed in the hosts in the distributed cluster cannot be managed uniformly.
Disclosure of Invention
Embodiments of the present invention provide a method, an apparatus, a system, a device, and a storage medium for processing a log file, so as to implement unified management on log files of components deployed in hosts in a distributed cluster. The specific technical scheme is as follows:
in a first aspect, to achieve the above object, an embodiment of the present invention provides a log file processing system, where the system includes a distributed cluster and a data storage server, where:
the host in the distributed cluster is used for acquiring log files generated by each component deployed in the host; determining a log file group to which each acquired log file belongs; for each log file group, when a preset uploading condition is met, sending the log files in the log file group to a data storage server; one log file group comprises log files of the same type generated by the same component in the host; the preset uploading condition is based on: presetting uploading time and/or determining the size of the current log file to be uploaded in the log file group;
and the data storage server is used for storing the received log files in the same log file group into the same local storage area according to an object storage mode.
In a second aspect, to achieve the above object, an embodiment of the present invention provides a log file processing method, where the method is applied to a host in a distributed cluster, and the method includes:
acquiring log files generated by each component deployed in the host;
determining a log file group to which each acquired log file belongs; one log file group comprises log files of the same type generated by the same component in the host;
for each log file group, when a preset uploading condition is met, sending the log files in the log file group to a data storage server, so that the data storage server stores the received log files in the same log file group to the same local storage area in an object storage mode; wherein the preset uploading condition is based on: and presetting the uploading time, and/or determining the size of the log file to be uploaded currently in the log file group.
Optionally, the obtaining log files generated by each component deployed in the host includes:
monitoring metadata of a log file generated by each component through a log acquisition process aiming at each component deployed in the host;
and acquiring the log file generated by the component from the storage position recorded in the monitored metadata.
Optionally, before determining the log file group to which each acquired log file belongs, the method further includes:
adding each acquired log file and attribute information of the log file to a preset queue correspondingly through the log acquisition thread; wherein the attribute information of one log file includes: generating an identifier of the component of the log file, an identifier of a host to which the component of the log file belongs, and a type identifier of the log file;
sequentially acquiring each log file and attribute information of the log file from the preset queue through a log processing process;
the determining of the log file group to which each acquired log file belongs includes:
and determining a log file group to which the log file belongs according to the attribute information of each log file acquired from the preset queue through the log processing process.
Optionally, the method further includes:
and for each log file in the preset queue, when the time length for adding the log file and the attribute information of the log file to the preset queue from the log acquisition process reaches a preset time length and the log processing process does not acquire the log file and the attribute information of the log file from the preset queue, deleting the log file and the attribute information of the log file from the preset queue.
Optionally, for each log file group, when a preset upload condition is met, sending the log file in the log file group to the data storage server includes:
for each log file group, when each log file belonging to the log file group is obtained, judging whether the log file to be uploaded in the current log file group reaches a preset size or not;
if the current log file to be uploaded in the log file group reaches a preset size, compressing the log file in the log file group according to a compression format corresponding to the log file group, and sending the compressed log file to a data storage server;
if the log file to be uploaded in the current log file group does not reach the preset size, judging whether the current log file group reaches the preset uploading moment or not;
and if the current uploading moment is reached, compressing the log files in the log file group according to the compression format corresponding to the log file group, and transmitting the compressed log files in the log file group to the data storage server.
In a third aspect, to achieve the above object, an embodiment of the present invention provides a log file processing method, where the method is applied to a data storage server, and the method includes:
receiving a log file sent by a host in a distributed cluster; wherein the log file is: when the host meets a preset uploading condition, sending the log files in each log file group to the data storage server; one log file group comprises log files of the same type generated by the same component in the host; the preset uploading condition is based on: presetting uploading time and/or determining the size of the current log file to be uploaded in the log file group;
and storing the received log files in the same log file group into the same local storage area according to an object storage mode.
Optionally, the method further includes:
and aiming at each locally stored log file, if the time length from the last time of accessing the log file reaches a first preset time length, storing the log file to a data archiving server.
Optionally, the method further includes:
receiving an access request of a client to a log file stored in the data archiving server;
and acquiring a target log file indicated by the access request from the data archiving server, and sending the target log file to the client.
Optionally, the method further includes:
and deleting the log file if the storage time of the log file reaches a second preset time aiming at each currently stored log file.
In a fourth aspect, to achieve the above object, an embodiment of the present invention provides a log file processing apparatus, where the apparatus is applied to a host in a distributed cluster, and the apparatus includes:
the first acquisition module is used for acquiring the log files generated by the components deployed in the host;
the determining module is used for determining the log file group to which each acquired log file belongs; one log file group comprises log files of the same type generated by the same component in the host;
the sending module is used for sending the log files in the log file group to the data storage server when the preset uploading condition is met aiming at each log file group, so that the data storage server stores the received log files in the same log file group to the same local storage area in an object storage mode; wherein the preset uploading condition is based on: and presetting the uploading time, and/or determining the size of the log file to be uploaded currently in the log file group.
Optionally, the first obtaining module is specifically configured to, for each component deployed in the host, monitor, through a log collection process, metadata of a log file generated by the component;
and acquiring the log file generated by the component from the storage position recorded in the monitored metadata.
Optionally, the apparatus further comprises:
the adding module is used for executing the log collection thread to correspondingly add each acquired log file and the attribute information of the log file to a preset queue before the determining module determines the log file group to which each acquired log file belongs; wherein the attribute information of one log file includes: generating an identifier of the component of the log file, an identifier of a host to which the component of the log file belongs, and a type identifier of the log file;
the second acquisition module is used for sequentially acquiring each log file and the attribute information of the log file from the preset queue through a log processing process;
the determining module is specifically configured to determine, through the log processing process, a log file group to which the log file belongs according to the attribute information of each log file acquired from the preset queue.
Optionally, the apparatus further comprises:
and the deleting module is used for deleting the log file and the attribute information of the log file from the preset queue when the time for adding the log file and the attribute information of the log file to the preset queue from the log collecting process reaches the preset time and the log processing process does not acquire the log file and the attribute information of the log file from the preset queue.
Optionally, the sending module is specifically configured to, for each log file group, determine whether a log file to be uploaded in the current log file group reaches a preset size when each log file belonging to the log file group is acquired;
if the current log file to be uploaded in the log file group reaches a preset size, compressing the log file in the log file group according to a compression format corresponding to the log file group, and sending the compressed log file to a data storage server;
if the log file to be uploaded in the current log file group does not reach the preset size, judging whether the current log file group reaches the preset uploading moment or not;
and if the current uploading moment is reached, compressing the log files in the log file group according to the compression format corresponding to the log file group, and transmitting the compressed log files in the log file group to the data storage server.
In a fifth aspect, in order to achieve the above object, an embodiment of the present invention provides a log file processing apparatus, where the apparatus is applied to a data storage server, and the apparatus includes:
the first receiving module is used for receiving the log file sent by the host in the distributed cluster; wherein the log file is: when the host meets a preset uploading condition, sending the log files in each log file group to the data storage server; one log file group comprises log files of the same type generated by the same component in the host; the preset uploading condition is based on: presetting uploading time and/or determining the size of the current log file to be uploaded in the log file group;
and the storage module is used for storing the received log files in the same log file group into the same local storage area according to an object storage mode.
Optionally, the apparatus further comprises:
and the archiving module is used for storing each locally stored log file to the data archiving server if the time length from the last time of accessing the log file reaches a first preset time length.
Optionally, the apparatus further comprises:
the second receiving module is used for receiving an access request of a client to the log file stored in the data archiving server;
and acquiring a target log file indicated by the access request from the data archiving server, and sending the target log file to the client.
Optionally, the apparatus further comprises:
and the deleting module is used for deleting the log file according to each currently stored log file if the storage time of the log file reaches a second preset time.
The embodiment of the invention also provides electronic equipment which comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete mutual communication through the communication bus;
a memory for storing a computer program;
a processor, configured to implement the log file processing method steps of the second aspect or the log file processing method steps of the third aspect when executing a program stored in a memory.
An embodiment of the present invention further provides a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the method for processing a log file according to the second aspect is implemented, or the method for processing a log file according to the third aspect is implemented.
An embodiment of the present invention further provides a computer program product including instructions, which, when run on a computer, causes the computer to execute the log file processing method according to the second aspect, or the log file processing method according to the third aspect.
According to the technical scheme provided by the embodiment of the invention, a host in a distributed cluster acquires log files generated by each deployed component; determining a log file group to which each acquired log file belongs; one log file group comprises log files of the same type generated by the same component in the host; and aiming at each log file group, when a preset uploading condition is met, sending the log files in the log file group to a data storage server. The data storage server stores the received log files in the same log file group into the same local storage area in an object storage mode; the preset uploading condition is based on: and presetting the uploading time, and/or determining the size of the log file to be uploaded currently in the log file group.
Based on the processing, the log files of the components deployed in the hosts in the distributed cluster can be uploaded to the data storage server, and the received log files are managed through the data storage server, that is, the log files of the components deployed in the hosts in the distributed cluster can be uniformly managed, and the received log files are stored according to an object storage mode, so that the storage cost can be saved, the stability of the data storage server is improved, the log files of the same log file group are stored in the same area, and the log files can be conveniently managed subsequently.
Of course, not all of the advantages described above need to be achieved at the same time in the practice of any one product or method of the invention.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other embodiments can be obtained by referring to these drawings.
Fig. 1 is a flowchart of a log file processing method according to an embodiment of the present invention;
fig. 2 is a flowchart of another log file processing method according to an embodiment of the present invention;
fig. 3 is a flowchart of another log file processing method according to an embodiment of the present invention;
fig. 4 is a flowchart of another log file processing method according to an embodiment of the present invention;
fig. 5 is a flowchart of another log file processing method according to an embodiment of the present invention;
fig. 6 is a flowchart of another log file processing method according to an embodiment of the present invention;
fig. 7 is a flowchart of another log file processing method according to an embodiment of the present invention;
fig. 8 is a flowchart of another log file processing method according to an embodiment of the present invention;
fig. 9 is a flowchart of another log file processing method according to an embodiment of the present invention;
fig. 10 is a structural diagram of a log file processing apparatus according to an embodiment of the present invention;
fig. 11 is a block diagram of another log file processing apparatus according to an embodiment of the present invention;
fig. 12 is a block diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived from the embodiments given herein by one of ordinary skill in the art, are within the scope of the invention.
In the related art, in order to monitor the operating state and performance of components deployed in hosts in a distributed cluster, a log file of each component is stored to the host where the component is deployed. As can be seen, in the related art, the log files of the components deployed in the hosts in the distributed cluster cannot be managed uniformly.
In order to solve the above technical problem, an embodiment of the present invention provides a log file processing system, where the system includes a distributed cluster and a data storage server, where:
the system comprises a host computer in the distributed cluster, a server and a server, wherein the host computer is used for acquiring log files generated by each component deployed in the host computer; determining a log file group to which each acquired log file belongs; for each log file group, when a preset uploading condition is met, sending the log files in the log file group to a data storage server; one log file group comprises log files of the same type generated by the same component; the preset uploading condition is based on: and presetting the uploading time, and/or determining the size of the log file to be uploaded currently in the log file group.
And the data storage server is used for storing the received log files in the same log file group into the same local storage area according to an object storage mode.
Based on the log file processing system provided by the embodiment of the invention, the log files of the components deployed in each host in the distributed cluster can be uploaded to the data storage server, and the received log files are managed through the data storage server, namely, the log files of the components deployed in each host in the distributed cluster can be uniformly managed, and the received log files are stored in an object storage mode, so that the storage cost can be saved, the stability of the data storage server is improved, the log files of the same log file group are stored in the same area, and the log files can be conveniently managed subsequently.
For other embodiments of the log file processing system, reference may be made to the following description of related methods for distributed clusters and data storage servers.
Referring to fig. 1, fig. 1 is a flowchart of a log file processing method according to an embodiment of the present invention, where the method may be applied to a host in a distributed cluster in the log file processing system, and the method may include the following steps:
s101: and acquiring log files generated by each component deployed in the host.
S102: and determining the log file group to which each acquired log file belongs.
One log file group comprises log files of the same type generated by the same component in the host.
S103: and aiming at each log file group, when a preset uploading condition is met, sending the log files in the log file group to a data storage server, so that the data storage server stores the received log files in the same log file group to the same local storage area in an object storage mode.
Wherein the preset uploading condition is based on: and presetting the uploading time, and/or determining the size of the log file to be uploaded currently in the log file group.
Based on the log file processing method provided by the embodiment of the invention, the log files of the components deployed in each host in the distributed cluster can be uploaded to the data storage server, and the received log files are managed through the data storage server, namely, the log files of the components deployed in each host in the distributed cluster can be uniformly managed, and the received log files are stored in an object storage mode, so that the storage cost can be saved, the stability of the data storage server is improved, the log files of the same log file group are stored in the same area, and the log files can be conveniently managed subsequently.
For step S101, a plurality of components, for example, an Hbase (an open source non-relational database) component, a Hadoop (an infrastructure of a distributed system) component, an Elastic Search (a distributed database) component, and the like, may be deployed in each host in the distributed cluster.
For each component deployed in the host, the component generates a corresponding log file in the running process, and the log file generated by the component can be acquired in order to monitor the running state and performance of the component.
In an embodiment of the present invention, on the basis of fig. 1, referring to fig. 2, step S101 may include the following steps:
s1011: and monitoring the metadata of the log file generated by each component through a log collection process aiming at each component deployed in the host.
S1012: and acquiring the log file generated by the component from the storage position recorded in the monitored metadata.
In one implementation, for each component, the component generates metadata for the log file as it generates the log file, i.e., the component generates an inode (inode) record for the log file. The log file has recorded in its metadata: the storage location of the log file, the size of the log file, the generation time of the log file, and the like.
Therefore, for each component, the host can listen to the metadata of the log file generated by the component through the log collection process. Further, the host may retrieve the log file generated by the component from the storage location recorded in the metadata that was snooped.
With respect to step S102, since a plurality of components are deployed in the host, and each component may generate a plurality of types of log files, for example, a log file recording errors occurring during the operation of the component, a log file recording user information related to the component, and the like.
In order to manage the log files conveniently, the host may divide the log files of the same type generated by the same component into a log file group. Therefore, after each log file is acquired, the host can determine the log file group to which the log file belongs according to the component generating the log file and the type of the log file.
In an embodiment of the present invention, on the basis of fig. 1, referring to fig. 3, before step S102, the method may further include the steps of:
s104: and adding each acquired log file and attribute information of the log file to a preset queue correspondingly through a log acquisition thread.
Wherein the attribute information of one log file includes: the log file generation method comprises the steps of generating an identifier of a component of the log file, generating an identifier of a host to which the component of the log file belongs, and identifying the type of the log file.
S105: and sequentially acquiring each log file and the attribute information of the log file from a preset queue through a log processing process.
Step S102 may include the steps of:
s1021: and determining a log file group to which the log file belongs through a log processing process according to the attribute information of each log file acquired from the preset queue.
The identification of the component that generated the log file may be the name of the component. The identity of the host to which the component that generated the log file belongs may be the name of the component. The type identification of the log file may be the name of the log file.
The log files with the same attribute information are the log files of the same type generated by the same component.
The log files and corresponding attribute information stored in the preset queue are as follows: the host is added when each log file is acquired through the log acquisition thread.
The host can sequentially acquire each log file and the attribute information of the log file from a preset queue through a log processing process. For each acquired log file, the host may determine a log file group to which the log file having the same attribute information as the log file belongs, and the determined log file group is the log file group to which the log file belongs.
Based on the above processing, when the number of the obtained log files is large, the host may not be able to process all the log files at the same time, and after storing each log file in the preset queue, the host may sequentially obtain each log file from the preset queue, and sequentially determine the log file group to which each log file belongs, so that the processing pressure of the host may be reduced.
In one embodiment of the invention, the method may further comprise the steps of: and for each log file in a preset queue, when the time length for adding the log file and the attribute information of the log file to the preset queue from the log acquisition process reaches the preset time length and the log processing process does not acquire the log file and the attribute information of the log file from the preset queue, deleting the log file and the attribute information of the log file from the preset queue.
In one implementation, if the log processing process is abnormal, the log file and the attribute information of the log file cannot be acquired from the preset queue, which may result in a long time of the added log file in the preset queue, and if the added log file is more, the log acquisition process may not add the newly acquired log file to the preset queue.
Therefore, the host can delete the log files which have the duration reaching the preset duration and are not acquired in the log processing process from the preset queue according to the duration of each log file in the preset queue.
For step S103, the preset upload condition may include: and when the preset uploading time is reached, and/or the log file to be uploaded in one log file group reaches the preset size.
The preset upload time and the preset size may be set by a technician based on experience. For example, the host may periodically send the log file to the data storage server, the preset upload time may be an upload time of each period, the period duration may be 10 minutes, and the preset size may be 1GB, but is not limited thereto.
If the preset uploading condition includes that the preset uploading time is reached, the host can judge whether the preset uploading time is reached currently or not for each log file group, and if the preset uploading time is reached currently, the host can send the log files in the log file group to the data storage server.
If the preset uploading condition includes that the log file to be uploaded in one log file group reaches a preset size, the host computer can judge whether the log file to be uploaded in the log file group reaches the preset size or not for each log file group, and if the log file to be uploaded in the log file group reaches the preset size, the host computer can send the log file in the log file group to the data storage server.
When the preset uploading only includes that the preset uploading time is reached, if more log files are generated by the component, more log files are stored locally by the host, and more storage space is occupied. When the preset uploading condition only includes that the current log file to be uploaded in the log file group reaches the preset size, if the log files generated by the assembly are less, the log files are uploaded only by waiting for less time, and the log files are not uploaded timely.
If the preset uploading condition comprises the following steps: when the preset uploading time is reached and the log file to be uploaded in one log file group reaches the preset size, the host can judge whether any one of the preset conditions is met, and corresponding processing is carried out according to the judgment result.
In an embodiment of the present invention, on the basis of fig. 1, referring to fig. 4, step S103 may include the following steps:
s1031: and for each log file group, when each log file belonging to the log file group is acquired, judging whether the current log file to be uploaded in the log file group reaches a preset size, if so, executing step S1032, and if not, executing step S1033.
S1032: and compressing the log files in the log file group according to the compression format corresponding to the log file group, and sending the compressed log files in the log file group to a data storage server.
S1033: and judging whether the current uploading time reaches a preset uploading moment, and if so, executing the step S1034.
S1034: and compressing the log files in the log file group according to the compression format corresponding to the log file group, and sending the compressed log files in the log file group to a data storage server.
In one implementation, after determining a log file group to which the log file belongs for each obtained log file, the host may determine whether the log file to be uploaded in the log file group reaches a preset size. If the log file to be uploaded in the log file group reaches a preset size, the host can compress the log file in the log file group according to the compression format corresponding to the log file group and send the compressed log file in the log file group to the data storage server.
If the log file to be uploaded in the log file group does not reach the preset size, the host can judge whether the current uploading time is reached, and if the current uploading time is reached, the host can compress the log file in the log file group according to the compression format corresponding to the log file group and send the compressed log file in the log file group to the data storage server. And if the preset uploading time is not reached currently, the host computer does not process the data.
The compression format corresponding to a group of log files may be set empirically by a skilled person. For example, the compressed format corresponding to a log file group may be Gzip or may also be queue, but is not limited thereto.
Correspondingly, when the data storage server receives the log file sent by the host in the distributed cluster, the data storage server stores the received log file in the same log file group to the same local storage area according to the object storage mode.
The specific processing manner of the data storage server can be referred to in the related description of the following embodiments.
Based on the processing, the problems that the log files are not uploaded timely and the host locally stores more log files and occupies more storage space can be avoided to a certain extent.
Referring to fig. 5, fig. 5 is a flowchart of a log file processing method, where the method is applied to a data storage server in the log file processing system, and the method may include the following steps:
s501: and receiving the log file sent by the host in the distributed cluster.
Wherein, the received log file is: when the host meets the preset uploading condition, sending the log files in each log file group to the data storage server; one log file group comprises log files of the same type generated by the same component in the host; the preset uploading condition is based on: and presetting the uploading time, and/or determining the size of the log file to be uploaded currently in the log file group.
S502: and storing the received log files in the same log file group into the same local storage area according to an object storage mode.
Based on the log file processing method provided by the embodiment of the invention, the log files of the components deployed in each host in the distributed cluster can be uploaded to the data storage server, and the received log files are managed through the data storage server, namely, the log files of the components deployed in each host in the distributed cluster can be uniformly managed, and the received log files are stored in an object storage mode, so that the storage cost can be saved, the stability of the data storage server is improved, the log files of the same log file group are stored in the same area, and the log files can be conveniently managed subsequently.
For step S501 and step S502, a plurality of components may be deployed in each host in the distributed cluster, each component may generate a corresponding log file in the running process, and in order to monitor the running state and performance of the component, the host may obtain the log file generated by the component and send the obtained log file to the data storage server.
For a specific processing manner of the hosts in the distributed cluster, reference may be made to the related description of the subsequent embodiments.
Accordingly, the data storage server may receive log files sent by hosts in the distributed cluster. Then, the data storage server may store the received log files in the same log file group into the same local storage area in an object storage manner, that is, store the log files in the same log file group into the same storage directory.
The object store may be: KS3(Kingsoft Standard Storage Service, kunskin cloud Object store), or BOS (Baidu Object Storage), but is not limited thereto.
Based on the processing, the log file is stored locally in an object storage mode, so that the cost of data storage can be saved, and the stability of the data storage server is improved.
In an embodiment of the present invention, on the basis of fig. 5, referring to fig. 6, the method may further include the steps of:
s503: and aiming at each locally stored log file, if the time length from the last time of accessing the log file reaches a first preset time length, storing the log file to a data archiving server.
In one implementation, after storing the received log file locally, for each log file, the data storage server may further determine whether the log file meets a preset archive condition. The preset filing conditions are as follows: and the time length from the last time of accessing the log file reaches a first preset time length.
The first preset duration may be set by a technician based on experience. For example, the first preset time period may be 1 month, or may also be 2 months, but is not limited thereto.
For each log file, the data storage server may obtain metadata of the log file, where the metadata of the log file records: the time when the log file is received, the time when the log file was last accessed, etc. Further, the data storage server may determine a time period if the log file was last accessed based on a time of last access to the log file.
And if the time length from the last time of accessing the log file reaches a first preset time length, indicating that the frequency of reading the log file is lower. Therefore, in order to save the storage space of the data storage server, the data storage server may dump the log file from a normal object storage to an archive object storage, where the normal object storage refers to the log file being stored in the data storage server, and the archive object storage refers to the log file being stored in the data archive server.
That is, for each log file, if the time length from the last access to the log file reaches a first preset time length, the data storage server may store the log file to the data archiving server.
In an embodiment of the present invention, on the basis of fig. 6, referring to fig. 7, the method may further include the steps of:
s504: an access request of a client to a log file stored in a data archiving server is received.
S505: and acquiring the target log file indicated by the access request from the data archiving server, and sending the target log file to the client.
In one implementation, a data storage server may receive an access request sent by a client for a target log file. If the target log file is stored locally, the data storage server may send the target log file to the client.
If the target log file is not stored locally, the data storage server can acquire the target log file from the data archiving server and send the target log file to the client.
In one embodiment of the invention, the method may further comprise the steps of: and deleting the log file if the storage time of the log file reaches a second preset time aiming at each currently stored log file.
The stored log file includes: a log file stored locally by the data storage server and a log file stored in the data archiving server.
In one implementation, for each stored log file, if the storage duration of the log file reaches a second preset duration, the log file is an expired log file, and in order to save the storage space, the log file may be deleted.
The second preset duration is greater than the first preset duration. For example, the first preset time period is 1 month, and the second preset time period may be 1 year, but is not limited thereto.
Referring to fig. 8, fig. 8 is a flowchart of another log file processing method according to an embodiment of the present invention.
Multiple components, e.g., component 1, component 2, component 3, and component 4, are deployed in each host in the distributed cluster. The host may monitor metadata of a log file generated by each component through a log collection process, and obtain the log file generated by the component from a storage location recorded in the monitored metadata, where "log stream" in fig. 8 represents a plurality of obtained log files.
After each log file is acquired, the host can correspondingly add the log file and the attribute information of the log file to a preset queue through a log acquisition process. Then, the host can sequentially acquire each log file and the attribute information of the log file from the preset queue through a log processing process. The host can determine the log file group to which the log file belongs according to the attribute information of each log file acquired from the preset queue through the log processing process, and store the log file to the local.
Furthermore, the host can judge whether the preset uploading condition is met currently or not for each log file group, compress the log files in the log file group when the preset uploading condition is met currently, and send the compressed log files in the log file group to the data storage server.
Correspondingly, when the data storage server receives the log file sent by the host in the distributed cluster, the log file in the same received log file group can be stored in the same local storage area through the log management process according to the object storage mode.
Based on the log file processing method provided by the embodiment of the invention, the log files of the components deployed in each host in the distributed cluster can be uploaded to the data storage server, and the received log files are managed through the data storage server, namely, the log files of the components deployed in each host in the distributed cluster can be uniformly managed, and the received log files are stored in an object storage mode, so that the storage cost can be saved, the stability of the data storage server is improved, the log files of the same log file group are stored in the same area, and the log files can be conveniently managed subsequently.
Referring to fig. 9, fig. 9 is a flowchart of another log file processing method according to an embodiment of the present invention.
S901: the monitoring metadata reads the log file.
The distributed cluster system comprises a plurality of components, wherein each host in the distributed cluster system is deployed with a plurality of components, and for each component, the host can monitor metadata of a log file generated by each component through a log collection process and acquire the log file generated by the component from a storage position recorded in the monitored metadata.
S902: and uploading to a preset queue.
After each log file is acquired, the host can correspondingly add the log file and the attribute information of the log file to a preset queue through a log acquisition process.
S903: and acquiring the log file from a preset queue.
The host can sequentially acquire each log file and the attribute information of the log file from the preset queue through a log processing process.
S904: the log file is cached locally.
The host can determine the log file group to which the log file belongs according to the attribute information of each log file acquired from the preset queue through the log processing process, and store the log file to the local.
S905: and judging whether the log file reaches a preset size.
For each log file group, the host can judge whether the log files to be uploaded in the log file group reach a preset size or not, and perform corresponding processing according to a judgment result.
S906: and judging whether the preset uploading moment is reached.
And aiming at each log file group, when the log file to be uploaded in the log file group does not reach the preset size, the host can judge whether the current time reaches the preset uploading moment or not, and performs corresponding processing according to the judgment result.
S907: the log file is compressed.
For each log file group, when the log file to be uploaded in the log file group reaches a preset size, the host can compress the log file to be uploaded in the log file group to obtain the compressed log file in the log file group. Or, when the current preset uploading time is reached, the host may compress the log file to be uploaded in the log file group to obtain the compressed log file in the log file group.
S908: and uploading to a data storage server.
For each log file group, the host may send the compressed log files in the log file group to the data storage server.
S909: the log management process starts monitoring.
The data storage server can receive log files sent by the host computers in the distributed cluster, store the received log files in the same log file group into the same local storage area, and start a log management process to monitor the locally stored log files.
S9010: and judging whether a preset filing condition is reached.
For each locally stored log file, the data storage server may determine whether the log file meets a preset archiving condition, where the preset archiving condition is: and the time length from the last time of accessing the log file reaches a first preset time length.
S9011: waiting for a period of retry.
For each locally stored log file, if the log file does not reach the preset archiving condition, the data storage server may wait for a period of time, and then judge again whether the log file reaches the preset archiving condition, so as to perform corresponding processing according to the judgment result.
S9012: and storing the log file to a data archiving server.
For each locally stored log file, if the log file meets a preset archiving condition, the data storage server may store the log file to the data archiving server.
S9013: and judging whether the expiration is reached.
For each log file that is stored, it can be determined whether the log file is an expired log file. For example, whether the time length for storing the log file reaches a second preset time length is judged, so that corresponding processing is performed according to the judgment result.
S9014: waiting for a period of retry.
For each stored log file, if the log file is not an expired log file, the data storage server may wait for a period of time, and then determine again whether the log file is an expired log file, so as to perform corresponding processing according to the determination result.
S9015: and clearing and deleting.
For each stored log file, if the log file is an expired log file, the log file may be cleared, that is, the log file may be deleted.
Based on the log file processing method provided by the embodiment of the invention, the log files of the components deployed in each host in the distributed cluster can be uploaded to the data storage server, and the received log files are managed through the data storage server, namely, the log files of the components deployed in each host in the distributed cluster can be uniformly managed, and the received log files are stored in an object storage mode, so that the storage cost can be saved, the stability of the data storage server is improved, the log files of the same log file group are stored in the same area, and the log files can be conveniently managed subsequently.
Corresponding to the embodiment of the method in fig. 1, referring to fig. 10, fig. 10 is a structural diagram of a log file processing apparatus, which is applied to a host in a distributed cluster, according to an embodiment of the present invention, and the apparatus includes:
a first obtaining module 1001, configured to obtain log files generated by respective components deployed in the host;
a determining module 1002, configured to determine a log file group to which each obtained log file belongs; one log file group comprises log files of the same type generated by the same component in the host;
a sending module 1003, configured to send, for each log file group, a log file in the log file group to a data storage server when a preset upload condition is met, so that the data storage server stores the received log files in the same log file group in the same local storage area in an object storage manner; wherein the preset uploading condition is based on: and presetting the uploading time, and/or determining the size of the log file to be uploaded currently in the log file group.
Optionally, the first obtaining module 1001 is specifically configured to, for each component deployed in the host, monitor metadata of a log file generated by the component through a log collection process;
and acquiring the log file generated by the component from the storage position recorded in the monitored metadata.
Optionally, the apparatus further comprises:
an adding module, configured to execute the log collection thread to correspondingly add each acquired log file and attribute information of the log file to a preset queue before the determining module 1002 determines the log file group to which each acquired log file belongs; wherein the attribute information of one log file includes: generating an identifier of the component of the log file, an identifier of a host to which the component of the log file belongs, and a type identifier of the log file;
the second acquisition module is used for sequentially acquiring each log file and the attribute information of the log file from the preset queue through a log processing process;
the determining module 1002 is specifically configured to determine, through the log processing process, a log file group to which the log file belongs according to the attribute information of each log file acquired from the preset queue.
Optionally, the apparatus further comprises:
and the deleting module is used for deleting the log file and the attribute information of the log file from the preset queue when the time for adding the log file and the attribute information of the log file to the preset queue from the log collecting process reaches the preset time and the log processing process does not acquire the log file and the attribute information of the log file from the preset queue.
Optionally, the sending module 1003 is specifically configured to, for each log file group, determine whether a log file to be uploaded in the current log file group reaches a preset size when each log file belonging to the log file group is acquired;
if the current log file to be uploaded in the log file group reaches a preset size, compressing the log file in the log file group according to a compression format corresponding to the log file group, and sending the compressed log file to a data storage server;
if the log file to be uploaded in the current log file group does not reach the preset size, judging whether the current log file group reaches the preset uploading moment or not;
and if the current uploading moment is reached, compressing the log files in the log file group according to the compression format corresponding to the log file group, and transmitting the compressed log files in the log file group to the data storage server.
Based on the log file processing device provided by the embodiment of the invention, the log files of the components deployed in each host in the distributed cluster can be uploaded to the data storage server, and the received log files are managed through the data storage server, namely, the log files of the components deployed in each host in the distributed cluster can be uniformly managed, and the received log files are stored in an object storage mode, so that the storage cost can be saved, the stability of the data storage server is improved, the log files of the same log file group are stored in the same area, and the log files can be conveniently managed subsequently.
Corresponding to the embodiment of the method in fig. 5, referring to fig. 11, fig. 11 is a structural diagram of a log file processing apparatus according to an embodiment of the present invention, where the apparatus is applied to a data storage server, and the apparatus includes:
a first receiving module 1101, configured to receive a log file sent by a host in a distributed cluster; wherein the log file is: when the host meets a preset uploading condition, sending the log files in each log file group to the data storage server; one log file group comprises log files of the same type generated by the same component in the host; the preset uploading condition is based on: presetting uploading time and/or determining the size of the current log file to be uploaded in the log file group;
the storage module 1102 is configured to store the received log files in the same log file group in the same local storage area according to an object storage manner.
Optionally, the apparatus further comprises:
and the archiving module is used for storing each locally stored log file to the data archiving server if the time length from the last time of accessing the log file reaches a first preset time length.
Optionally, the apparatus further comprises:
the second receiving module is used for receiving an access request of a client to the log file stored in the data archiving server;
and acquiring a target log file indicated by the access request from the data archiving server, and sending the target log file to the client.
Optionally, the apparatus further comprises:
and the deleting module is used for deleting the log file according to each currently stored log file if the storage time of the log file reaches a second preset time.
Based on the log file processing device provided by the embodiment of the invention, the log files of the components deployed in each host in the distributed cluster can be uploaded to the data storage server, and the received log files are managed through the data storage server, namely, the log files of the components deployed in each host in the distributed cluster can be uniformly managed, and the received log files are stored in an object storage mode, so that the storage cost can be saved, the stability of the data storage server is improved, the log files of the same log file group are stored in the same area, and the log files can be conveniently managed subsequently.
An embodiment of the present invention further provides an electronic device, as shown in fig. 12, including a processor 1201, a communication interface 1202, a memory 1203, and a communication bus 1204, where the processor 1201, the communication interface 1202, and the memory 1203 complete mutual communication through the communication bus 1204,
a memory 1203 for storing a computer program;
the processor 1201 is configured to, when executing the program stored in the memory 1203, implement the log file processing method steps applied to the hosts in the distributed cluster in the foregoing embodiment, or the log file processing method steps applied to the data storage server in the foregoing embodiment.
The communication bus mentioned in the electronic device may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown, but this does not mean that there is only one bus or one type of bus.
The communication interface is used for communication between the electronic equipment and other equipment.
The Memory may include a Random Access Memory (RAM) or a Non-Volatile Memory (NVM), such as at least one disk Memory. Optionally, the memory may also be at least one memory device located remotely from the processor.
The Processor may be a general-purpose Processor, including a Central Processing Unit (CPU), a Network Processor (NP), and the like; but also Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components.
In yet another embodiment provided by the present invention, a computer-readable storage medium is further provided, in which a computer program is stored, and the computer program, when executed by a processor, implements the log file processing method steps applied to the hosts in the distributed cluster in the above embodiments, or the log file processing method steps applied to the data storage server in the above embodiments.
In a further embodiment provided by the present invention, there is also provided a computer program product containing instructions which, when run on a computer, cause the computer to perform the log file processing method steps applied to the hosts in the distributed cluster in the above-described embodiment, or the log file processing method steps applied to the data storage server in the above-described embodiment.
In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, cause the processes or functions described in accordance with the embodiments of the invention to occur, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, from one website site, computer, server, or data center to another website site, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that incorporates one or more of the available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
All the embodiments in the present specification are described in a related manner, and the same and similar parts among the embodiments may be referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the apparatus, system, electronic device, computer-readable storage medium, and computer program product embodiments, the description is relatively simple as it is substantially similar to the method embodiments, and reference may be made to some descriptions of the method embodiments for related points.
The above description is only for the preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention shall fall within the protection scope of the present invention.

Claims (14)

1. A log file processing system, the system comprising distributed clusters and data storage servers, wherein:
the host in the distributed cluster is used for acquiring log files generated by each component deployed in the host; determining a log file group to which each acquired log file belongs; for each log file group, when a preset uploading condition is met, sending the log files in the log file group to a data storage server; one log file group comprises log files of the same type generated by the same component in the host; the preset uploading condition is based on: presetting uploading time and/or determining the size of the current log file to be uploaded in the log file group;
and the data storage server is used for storing the received log files in the same log file group into the same local storage area according to an object storage mode.
2. A method for processing log files, the method being applied to hosts in a distributed cluster, the method comprising:
acquiring log files generated by each component deployed in the host;
determining a log file group to which each acquired log file belongs; one log file group comprises log files of the same type generated by the same component in the host;
for each log file group, when a preset uploading condition is met, sending the log files in the log file group to a data storage server, so that the data storage server stores the received log files in the same log file group to the same local storage area in an object storage mode; wherein the preset uploading condition is based on: and presetting the uploading time, and/or determining the size of the log file to be uploaded currently in the log file group.
3. The method of claim 2, wherein the obtaining log files generated by respective components deployed in the host comprises:
monitoring metadata of a log file generated by each component through a log acquisition process aiming at each component deployed in the host;
and acquiring the log file generated by the component from the storage position recorded in the monitored metadata.
4. The method according to claim 2, wherein before the determining the log file group to which each obtained log file belongs, the method further comprises:
adding each acquired log file and attribute information of the log file to a preset queue correspondingly through the log acquisition thread; wherein the attribute information of one log file includes: generating an identifier of the component of the log file, an identifier of a host to which the component of the log file belongs, and a type identifier of the log file;
sequentially acquiring each log file and attribute information of the log file from the preset queue through a log processing process;
the determining of the log file group to which each acquired log file belongs includes:
and determining a log file group to which the log file belongs according to the attribute information of each log file acquired from the preset queue through the log processing process.
5. The method of claim 4, further comprising:
and for each log file in the preset queue, when the time length for adding the log file and the attribute information of the log file to the preset queue from the log acquisition process reaches a preset time length and the log processing process does not acquire the log file and the attribute information of the log file from the preset queue, deleting the log file and the attribute information of the log file from the preset queue.
6. The method of claim 2, wherein for each log file group, when a preset uploading condition is met, sending the log files in the log file group to the data storage server comprises:
for each log file group, when each log file belonging to the log file group is obtained, judging whether the log file to be uploaded in the current log file group reaches a preset size or not;
if the current log file to be uploaded in the log file group reaches a preset size, compressing the log file in the log file group according to a compression format corresponding to the log file group, and sending the compressed log file to a data storage server;
if the log file to be uploaded in the current log file group does not reach the preset size, judging whether the current log file group reaches the preset uploading moment or not;
and if the current uploading moment is reached, compressing the log files in the log file group according to the compression format corresponding to the log file group, and transmitting the compressed log files in the log file group to the data storage server.
7. A log file processing method is applied to a data storage server and comprises the following steps:
receiving a log file sent by a host in a distributed cluster; wherein the log file is: when the host meets a preset uploading condition, sending the log files in each log file group to the data storage server; one log file group comprises log files of the same type generated by the same component in the host; the preset uploading condition is based on: presetting uploading time and/or determining the size of the current log file to be uploaded in the log file group;
and storing the received log files in the same log file group into the same local storage area according to an object storage mode.
8. The method of claim 7, further comprising:
and aiming at each locally stored log file, if the time length from the last time of accessing the log file reaches a first preset time length, storing the log file to a data archiving server.
9. The method of claim 8, further comprising:
receiving an access request of a client to a log file stored in the data archiving server;
and acquiring a target log file indicated by the access request from the data archiving server, and sending the target log file to the client.
10. The method of claim 7, further comprising:
and deleting the log file if the storage time of the log file reaches a second preset time aiming at each currently stored log file.
11. An apparatus for processing log files, the apparatus being applied to hosts in a distributed cluster, the apparatus comprising:
the first acquisition module is used for acquiring the log files generated by the components deployed in the host;
the determining module is used for determining the log file group to which each acquired log file belongs; one log file group comprises log files of the same type generated by the same component in the host;
the sending module is used for sending the log files in the log file group to the data storage server when the preset uploading condition is met aiming at each log file group, so that the data storage server stores the received log files in the same log file group to the same local storage area in an object storage mode; wherein the preset uploading condition is based on: and presetting the uploading time, and/or determining the size of the log file to be uploaded currently in the log file group.
12. A log file processing apparatus, wherein the apparatus is applied to a data storage server, the apparatus comprising:
the first receiving module is used for receiving the log file sent by the host in the distributed cluster; wherein the log file is: when the host meets a preset uploading condition, sending the log files in each log file group to the data storage server; one log file group comprises log files of the same type generated by the same component in the host; the preset uploading condition is based on: presetting uploading time and/or determining the size of the current log file to be uploaded in the log file group;
and the storage module is used for storing the received log files in the same log file group into the same local storage area according to an object storage mode.
13. An electronic device is characterized by comprising a processor, a communication interface, a memory and a communication bus, wherein the processor and the communication interface are used for realizing mutual communication by the memory through the communication bus;
a memory for storing a computer program;
a processor for implementing the method steps of claims 2-6, or any of claims 7-10, when executing a program stored in a memory.
14. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the method steps of claims 2 to 6 or of any of claims 7 to 10.
CN202111318978.9A 2021-11-09 2021-11-09 Log file processing method, device, system, equipment and storage medium Pending CN114036121A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202111318978.9A CN114036121A (en) 2021-11-09 2021-11-09 Log file processing method, device, system, equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202111318978.9A CN114036121A (en) 2021-11-09 2021-11-09 Log file processing method, device, system, equipment and storage medium

Publications (1)

Publication Number Publication Date
CN114036121A true CN114036121A (en) 2022-02-11

Family

ID=80143664

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202111318978.9A Pending CN114036121A (en) 2021-11-09 2021-11-09 Log file processing method, device, system, equipment and storage medium

Country Status (1)

Country Link
CN (1) CN114036121A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116701336A (en) * 2023-05-19 2023-09-05 国网物资有限公司 Power data log processing method, electronic device and computer readable medium

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116701336A (en) * 2023-05-19 2023-09-05 国网物资有限公司 Power data log processing method, electronic device and computer readable medium

Similar Documents

Publication Publication Date Title
US10803016B2 (en) Predictive models of file access patterns by application and file type
US9454321B1 (en) Workload-driven storage configuration management
CN109947668B (en) Method and device for storing data
US11422990B2 (en) Content item purging
US11232070B2 (en) Metadata compaction in a distributed storage system
CN110046133B (en) Metadata management method, device and system for storage file system
US20190213171A1 (en) Directory Deletion Method and Apparatus, and Storage Server
CN110958300B (en) Data uploading method, system, device, electronic equipment and computer readable medium
EP3584716B1 (en) Storage constrained synchronization of shared content items
US8943364B2 (en) Appliance for storing, managing and analyzing problem determination artifacts
CN112650728A (en) Interface information storage method and device, server and distributed service system
CN110502510B (en) Real-time analysis and duplicate removal method and system for WIFI terminal equipment trajectory data
CN114036121A (en) Log file processing method, device, system, equipment and storage medium
WO2017088701A1 (en) Mass picture management method and apparatus
CN112256649A (en) Medical file storage method and device
US10719532B2 (en) Storage constrained synchronization engine
JP5956064B2 (en) Computer system, data management method, and computer
CN114238018B (en) Method, system and device for detecting integrity of log collection file and storage medium
CN115348160A (en) Backtracking data storage method and device and computer equipment
CN106921536B (en) Data processing method and device based on client version information
CN111858498A (en) Storage type conversion method, system, device and equipment
CN111291127A (en) Data synchronization method, device, server and storage medium
CN114238008A (en) Data acquisition method, device and system, electronic equipment and storage medium
US9251153B1 (en) Systems and methods for populating and maintaining a local cache with archived data items
JP6437691B2 (en) File management system, file management method, collection program, and non-transitory computer-readable information recording medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination