CN110309109B - Data monitoring method, device, computer equipment and storage medium - Google Patents
Data monitoring method, device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN110309109B CN110309109B CN201910432722.7A CN201910432722A CN110309109B CN 110309109 B CN110309109 B CN 110309109B CN 201910432722 A CN201910432722 A CN 201910432722A CN 110309109 B CN110309109 B CN 110309109B
- Authority
- CN
- China
- Prior art keywords
- data file
- data
- node
- directory
- file
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000012544 monitoring process Methods 0.000 title claims abstract description 78
- 238000000034 method Methods 0.000 title claims abstract description 31
- 239000003607 modifier Substances 0.000 claims abstract description 56
- 238000012986 modification Methods 0.000 claims abstract description 51
- 230000004048 modification Effects 0.000 claims abstract description 51
- 238000004590 computer program Methods 0.000 claims description 14
- 230000008859 change Effects 0.000 claims description 13
- 238000012806 monitoring device Methods 0.000 claims description 12
- 238000009825 accumulation Methods 0.000 claims description 3
- 238000012545 processing Methods 0.000 abstract description 6
- 238000005516 engineering process Methods 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 238000011161 development Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/1734—Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data monitoring method, a device, computer equipment and a storage medium, which relate to the technical field of data processing and have the main technical scheme that: selecting a target directory to be monitored according to brief information corresponding to the data file directory under each node in the distributed file system stored on the monitoring node; acquiring storage information, modification time and/or modifier of a data file under a corresponding node directory in the distributed file system through the target directory; storing the obtained storage information, modification time and/or modifier of the data file under each node directory in a local file; and analyzing the storage information, the modification time and/or the modifier of the data file under the directory of each node in the local file according to the preset monitoring requirement to determine whether the data file reaches an alarm line. The method is mainly used for summarizing, counting and monitoring the whole storage condition of the distributed file system in real time.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a data monitoring method, a data monitoring device, a computer device, and a storage medium.
Background
Along with the continuous improvement of social informatization technology and the rapid popularization of internet technology, data to be processed are also larger and larger, and the requirements of various fields on mass data processing are also increased. In the background that the single-machine storage space and the computing capability cannot meet the requirement of people on mass data processing, distributed computing and parallel computing begin to develop and apply rapidly. The monitoring information of the distributed system in a large scale is massive, the monitoring resources are multi-level and multi-source, and the dynamic property and complexity of the big data bring a plurality of difficulties to the big data monitoring system. How to effectively monitor the data in the big data and predict the data in time, and taking corresponding measures before failure is a key for improving the service quality of the big data, and is also the key point of current research.
The main characteristics of big data are distributed file system and distributed computation, so that when big data are to be used for storage, storage space is required, the stored data are increased along with the development of enterprises and the change of business, and therefore, the stored change is monitored in real time so as to expand capacity and clear junk files when required.
The existing monitoring devices are all arranged on nodes in the distributed system, so that the real-time operation condition of each node can be monitored, the whole operation condition of the distributed system is not summarized and monitored, and the summarizing statistics and the real-time monitoring are not achieved aiming at the whole storage condition.
Disclosure of Invention
The invention provides a data monitoring method, a data monitoring device, computer equipment and a storage medium, which realize summarizing statistics and real-time monitoring aiming at the integral storage condition of a distributed file system.
The embodiment of the invention provides a data monitoring method which is applied to any monitoring node in a distributed file system, and comprises the following steps:
selecting a target directory to be monitored according to brief information corresponding to data file directories under each node in the distributed file system stored on the monitoring node, wherein the data file directories are directories corresponding to data files under different nodes in the distributed file system;
acquiring storage information, modification time and/or modifier of a data file under a corresponding node directory in the distributed file system through the target directory;
storing the obtained storage information, modification time and/or modifier of the data file under each node directory in a local file;
and analyzing the storage information, the modification time and/or the modifier of the data file under the directory of each node in the local file according to the preset monitoring requirement to determine whether the data file reaches an alarm line.
The embodiment of the invention provides a data monitoring device, which comprises:
the acquisition module is used for selecting a target directory to be monitored according to brief information corresponding to data file directories under each node in the distributed file system stored on the monitoring node, wherein the data file directories are directories corresponding to data files under different nodes in the distributed file system;
the acquisition module is further used for acquiring storage information, modification time and/or modifier of the data file under the corresponding node directory in the distributed file system through the target directory;
the storage module is used for storing the storage information, the modification time and/or the modifier of the data file under the obtained node list in the local file;
and the determining module is used for analyzing the storage information, the modification time and/or the modifier of the data file under the directory of each node in the local file according to the preset monitoring requirement so as to determine whether the data file reaches an alarm line or not.
A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, the processor implementing the data monitoring method described above when executing the computer program.
A computer readable storage medium storing a computer program which when executed by a processor implements the data monitoring method described above.
Compared with the data monitoring method, the device, the computer equipment and the storage medium which are used for performing data monitoring by arranging the monitoring devices on all nodes on the distributed system, the method and the device for monitoring the data selects the target directory which needs to be monitored through the brief information corresponding to the data file directory under all the nodes in the distributed file system stored on the monitoring nodes in the distributed file system, and then obtains the storage information, the modification time and/or the modifier of the data file under the corresponding node directory in the distributed file system through the target directory; and storing the obtained storage information, modification time and/or modifier of the data file under each node directory in the local file, and finally analyzing the storage information, modification time and/or modifier of the data file under each node directory in the local file according to preset monitoring requirements to determine whether the data file reaches an alarm line or not. The invention can acquire the running state and the storage space condition of the distributed file system through the scheduling task initiated by the monitoring node, and the monitoring can timely inform operators to maintain when the dangerous storage space of the distributed file system is insufficient or the running is wrong, thereby ensuring the stable running of the system and improving the data monitoring efficiency of the distributed file system.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings that are needed in the description of the embodiments of the present invention will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of a data monitoring method according to an embodiment of the invention;
FIG. 2 is a flow chart of determining whether a data file reaches an alarm line in an embodiment of the invention;
FIG. 3 is a flowchart of another method for determining whether a data file reaches an alarm line according to an embodiment of the present invention;
FIG. 4 is a flow chart of determining whether a data file reaches an alarm line in accordance with one embodiment of the present invention;
FIG. 5 is a schematic diagram of a data monitoring device according to an embodiment of the present invention;
FIG. 6 is a schematic diagram of a computer device in accordance with an embodiment of the invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are some, but not all embodiments of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
In one embodiment, as shown in fig. 1, a data monitoring method is provided, and the method is applied to any monitoring node in a distributed file system, and includes the following steps:
s10, selecting a target directory to be monitored through brief information corresponding to the data file directories under each node in the distributed file system stored on the monitoring node.
The data file catalogue is a catalogue corresponding to data files under different nodes in the distributed file system, and the brief information is used for briefly recording data contents stored under the corresponding data file catalogue. The target directory can be selected according to the actual requirement of the user, for example, a certain file needs to be stored by means of a file system with big data, but the file is important, backup, change and other operations are needed, so that only the size and the data change condition of the file can be monitored independently, and the data file directory needing to be monitored is the directory where the file is located. It should be noted that, the embodiment of the present invention is applied to a distributed file system, that is, the data monitoring may be performed by the embodiment of the present invention in a scenario where a large data cluster is needed, for example, when the large data cluster is used to store data, the large data cluster is used to perform a computing task, etc., where the embodiment of the present invention is not limited specifically. Wherein the distributed file system is an integral unit of cluster nodes, and a complete data file is stored separately on different node machines.
In the embodiment of the invention, the data file directory to be monitored can be obtained through the scheduling task arranged on any node of the distributed file system. It should be noted that, in the embodiment of the present invention, a data file directory to be monitored may be selected according to an actual requirement, and if a task to be monitored for the data file a exists, then the data file directory where the data file a is located under each node may be selected in the distributed file system according to the requirement.
The scheduling task arranged on the node is a lightweight task, the scheduling task is a task for acquiring the storage condition of the distributed system, and the program in the node is an executed shell script used for acquiring the storage condition of the distributed system. It should be noted that, the scheduling task set on the node stores the data file directory under each node in the distributed file system and the brief information corresponding to each data file directory, where the brief information is used to briefly record the data content stored under the corresponding data file directory, when the data file in the distributed file system needs to be monitored, the data file directory needing to be monitored is selected through the brief information corresponding to the data file directory under each node in the stored distributed file system, and then the storage information, the modification time and/or the modifier of each sub-data file under the corresponding node are called through the selected data file directory.
For the embodiment of the invention, the service program started by the lightweight scheduling task does not need to be too complex, and the pressure of the service program is smaller than that of a server of a traditional monitoring device, so that the used resources, occupied memory and the like are very small. For the embodiment of the invention, only a single small service is required to be started to receive the monitoring information sent by the scheduling task and display the information after the information is made into a visual chart interface, the needed program of the server is few, the server has little pressure, and the small server is enough to meet the requirement.
S20, obtaining storage information, modification time and/or modifier of the data files under the corresponding node directory in the distributed file system through the target directory.
The target directory is selected from the data file directories according to actual demands, and the node directory is a node group corresponding to the target directory and the node directory, namely, the data files under the corresponding nodes can be acquired from the distributed file system through the node directory. In the embodiment of the invention, after the target directory to be performed is acquired, the storage information, the modification time and/or the modifier of each sub-data file under the corresponding node directory are acquired through the acquired target directory. Specifically, a scheduling task is executed on any node in the distributed file system to monitor the storage condition of the distributed file system, the shell script executed by the scheduling task can acquire all directories of the distributed file system, and then the acquired directories are used for acquiring the information such as the size of a storage file under the corresponding directory, the newly built/modified time of the file under the directory, the modifier and the like.
And S30, storing the obtained storage information, modification time and/or modifier of the data file under each node directory in the local file.
It should be noted that, in the embodiment of the present invention, since a data monitoring task initiated by any node in a distributed file system is provided, after the node obtains storage information, modification time, and/or modifier of a data file under each node that needs to be monitored, the node needs to store the obtained storage information, modification time, and/or modifier of the data file under each node in a local file, and store the obtained storage information, modification time, and/or modifier in the local file, and the obtained information is stored in the local file of the node, so that the monitoring node monitors the data file that needs to be monitored.
S40, analyzing the storage information, the modification time and/or the modifier of the data file under the directory of each node in the local file according to the preset monitoring requirement to determine whether the data file reaches an alarm line.
For the embodiment of the invention, after the storage information, the modification time and/or the modification person of the data file under each node directory are stored in the local file, the storage information, the modification time and/or the modification person of the data file under each node directory in the local file are required to be analyzed according to the preset monitoring requirement so as to determine whether the data file reaches the alarm line. The preset monitoring requirement is set according to actual requirements, and if the storage capacity of the monitoring data file exceeds a certain numerical range, the preset monitoring requirement can be set so that the storage capacity of the data file does not exceed the preset numerical range; if the last modification time of the data file is monitored, the preset monitoring requirement can be set to be whether the last modification time of the data file exceeds the preset time, etc., and the embodiment of the invention is not particularly limited.
Compared with the existing data monitoring by arranging monitoring devices on all nodes of a distributed file system, the method provided by the embodiment of the invention selects a target directory to be monitored through brief information corresponding to data file directories under all nodes in the distributed file system stored in the monitoring nodes of the distributed file system, and then acquires storage information, modification time and/or modifier of the data files under the corresponding node directories in the distributed file system through the target directory; and storing the obtained storage information, modification time and/or modifier of the data file under each node directory in the local file, and finally analyzing the storage information, modification time and/or modifier of the data file under each node directory in the local file according to preset monitoring requirements to determine whether the data file reaches an alarm line or not. The invention can acquire the storage information, the modification time and/or the modification person condition of the distributed file system through the scheduling task initiated by the monitoring node, and the monitoring can timely inform operators to maintain when the dangerous storage space of the distributed file system is insufficient or the operation is wrong, thereby ensuring the stable operation of the system and improving the data monitoring efficiency of the distributed file system.
As shown in fig. 2, in one embodiment provided by the present invention, step S40: the step of determining whether the data file reaches an alarm line according to the stored information of the data file under the directory of each node in the local file according to the preset monitoring requirement comprises the following steps:
and S40A1, performing accumulation calculation according to the storage information under the directory of each node to obtain the total storage information.
In the embodiment of the invention, after the data file directory acquires the storage information, the modification time and/or the modifier of the data file under the corresponding node directory and stores the storage information and the modification time of the data file in the local file, the storage information of each sub data file is required to be accumulated and calculated according to the storage dimension information to obtain the total storage information of the data file.
S40A2, judging whether the total storage amount information exceeds a preset range.
And S40A3, if the data file exceeds the alarm line, determining that the data file reaches the alarm line.
The preset range is set according to actual requirements, for example, the preset range may be set such that the total storage amount information exceeds eighty percent, or eighty-five percent of the load of the total storage amount information, and the embodiment of the invention is not particularly limited.
For example, according to the target directory, the storage information of the data file under the node a directory is 11G, the storage information of the data file under the node b directory is 15G, the storage information of the data file under the node c directory is 15G, the storage information of each node directory is accumulated to obtain the total storage information of the data file as 41G, that is, the data file has been used with the storage capacity of 41G, if the available storage capacity of the data file is 50G and the preset range is eighty percent of the available storage capacity, the total storage information exceeds the available storage capacity by judging that the storage capacity of the data file is 40G, and at the moment, the data file can be determined to reach the alarm line, the system can automatically send the alarm information to the operator, or send a short message to inform the operator, or set to automatically dial a call to the operator.
As shown in fig. 3, in one embodiment provided by the present invention, step S40: the step of determining whether the data file reaches an alarm line by modifying the data file under the directory of each node in the local file according to the preset monitoring requirement comprises the following steps:
S40B1, acquiring the starting time and the ending time of writing the data file in the local file.
The modification time of the data file comprises a start time and an end time of writing under the data file, wherein the start time and the end time are respectively the time of writing the data file and the time of last writing the data file.
S40B2, calculating the writing time of the data file according to the ending time and the starting time.
For example, when the start time of writing the data file is 13:00 in 2018 12 month 5 days and the end time of writing the data file is 15:00 in 2018 12 month 5 days, the writing duration of the data file is calculated to be 2 hours according to the end time and the start time of the data file.
S40B3, judging whether the writing time length is smaller than the time length required for writing the data file in the distributed file system.
The required time length for writing the data file in the distributed file system is the actual time length required for writing the data file in the distributed file system, and the required time length can be determined according to the size of the data file and the speed for writing the data file in the distributed file system.
And S40B4, if the data file is smaller than the alarm line, determining that the data file reaches the alarm line.
In the embodiment of the invention, the starting time and the ending time of writing the data file under the data file in the local file are obtained, the writing time of the data file is calculated according to the ending time and the starting time, then whether the writing time is smaller than the time required for writing the data in the distributed file system is judged, if the writing time is smaller than the time required for writing the data in the distributed file system, the incomplete writing of the data file is indicated, at the moment, the data file is required to be determined to reach an alarm line, the written data file is required to be alarmed, and a background manager is timely informed of corresponding processing.
As shown in fig. 4, in one embodiment provided by the present invention, step S40: the step of analyzing the modifier of the data file under the directory of each node in the local file according to the preset monitoring requirement to determine whether the data file reaches an alarm line comprises the following steps:
S40C1, acquiring the user authority of the modifier and the operation authority for writing the data content.
It should be noted that different modifiers have different user rights, where the user rights are preset according to actual requirements, and the user rights may specifically be rights of writing data content, rights of reading data content, rights of modifying data content, and the like, and embodiments of the present invention are not limited specifically.
S40C2, judging whether the user permission of the modifier contains the operation permission.
And S40C3, if the data file does not contain the data file, determining that the data file reaches an alarm line.
For the embodiment of the invention, after the user permission of the modifier and the operation permission of writing the data content are obtained, judging whether the user permission of the modifier contains the operation permission of writing the data content, if so, indicating that the modifier has the permission of writing the data in the distributed system; if the data file does not contain the data file, the data file is determined to reach the alarm line, namely the person writing the data in the distributed system does not meet the requirement of the person which is not preset, and alarm processing is needed at the moment.
In the embodiment of the invention, the data which is determined to reach the alarm line is sent to a server, so that the server displays and alarms the data file which reaches the alarm line. The alarm line is mainly set based on the performance of the distributed system, the number of nodes of the distributed system and the storage size of the distributed system, and can trigger an alarm when a program to be processed exceeds 90% of the performance load capacity of the distributed system and data to be stored exceeds 80% of the storage space of the distributed system; when the warning line is reached to trigger the warning device, the system automatically sends warning information to an operator, and can also send a short message to inform the operator; automatic telephone dialing can also be set to operators.
In one embodiment provided by the present invention, the method further comprises: when a data file directory under a node and/or brief information corresponding to the data file directory are changed in the distributed file system, obtaining change information of the brief information change corresponding to the data file directory and/or the data file directory; and updating the data file catalogue of the corresponding node and/or the brief information corresponding to the data file catalogue stored on the monitoring node according to the change information.
It should be understood that the sequence number of each step in the foregoing embodiment does not mean that the execution sequence of each process should be determined by the function and the internal logic, and should not limit the implementation process of the embodiment of the present invention.
In an embodiment, a data monitoring device is provided, where the data monitoring device corresponds to the data monitoring method in the foregoing embodiment one by one. As shown in fig. 5, the data monitoring apparatus includes an acquisition module 10, a storage module 20, and a determination module 30. The functional modules are described in detail as follows:
the acquiring module 10 is configured to select a target directory to be monitored according to brief information corresponding to data file directories under each node in the distributed file system stored on a monitoring node, where the data file directories are directories corresponding to data files under different nodes in the distributed file system;
the obtaining module 10 is further configured to obtain, through the target directory, storage information, modification time, and/or a modifier of a data file under a corresponding node directory in the distributed file system;
the storage module 20 is configured to store the obtained storage information, modification time, and/or modifier of the data file under each node directory in a local file;
the determining module 30 is configured to analyze, according to a preset monitoring requirement, storage information, modification time, and/or a modifier of the data file under the directory of each node in the local file, so as to determine whether the data file reaches an alarm line.
Specifically, the determining module 30 includes:
a calculating unit 31, configured to perform accumulation calculation according to the storage information of the data files under the node directories, so as to obtain total storage information;
a judging unit 32 for judging whether the total storage amount information exceeds a preset range;
and a determining unit 33, configured to determine that the data file reaches an alarm line if the total storage amount information exceeds a preset range.
Specifically, the determining module 30 includes:
an obtaining unit 34 for obtaining a start time and an end time of writing the data file in the local file; the start time and the end time are respectively the time of writing the data file and the time of last writing the data file;
the calculating unit 31 is further configured to calculate a writing duration of the data file according to the ending time and the starting time;
the judging unit 32 is further configured to judge whether the writing time length is less than a time length required for writing the data file in the distributed file system;
the determining unit 33 is further configured to determine that the data file reaches an alarm line if the writing time length is determined to be less than a time length required for writing the data in the distributed file system.
Specifically, the determining unit 30 includes:
the acquiring unit 34 is configured to acquire a user right of the modifier and an operation right of writing the data content;
the judging unit 33 is configured to judge whether the user right of the modifier includes the operation right;
the determining unit 33 is configured to determine that the data file reaches an alarm line if the user right of the modifier does not include the operation right.
Further, the device further comprises:
and the sending module 40 is used for sending the information of the data file reaching the alarm line to a server, so that the server displays and alarms the data file reaching the alarm line.
Further, the device further comprises:
the obtaining module 10 is further configured to obtain, when there is a data file directory under a node in the distributed file system and/or a brief information corresponding to the data file directory changes, change information of the brief information change corresponding to the data file directory and/or the data file directory;
and the updating module is used for updating the data file catalogue of the corresponding node and/or the brief information corresponding to the data file catalogue stored on the monitoring node according to the change information.
For specific limitations of the data monitoring device, reference may be made to the above limitation of the data monitoring method, and no further description is given here. The respective modules in the data monitoring apparatus described above may be implemented in whole or in part by software, hardware, and combinations thereof. The above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.
In one embodiment, a computer device is provided, which may be a terminal, and the internal structure of which may be as shown in fig. 6. The computer device includes a processor, a memory, a network interface, a display screen, and an input device connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface of the computer device is for communicating with an external server via a network connection. The computer program, when executed by a processor, implements any of the data monitoring methods described above.
In one embodiment, a computer device is provided comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing the steps of when executing the computer program:
selecting a target directory to be monitored according to brief information corresponding to data file directories under each node in the distributed file system stored on the monitoring node, wherein the data file directories are directories corresponding to data files under different nodes in the distributed file system;
acquiring storage information, modification time and/or modifier of a data file under a corresponding node directory in the distributed file system through the target directory;
storing the obtained storage information, modification time and/or modifier of the data file under each node directory in a local file;
and analyzing the storage information, the modification time and/or the modifier of the data file under the directory of each node in the local file according to the preset monitoring requirement to determine whether the data file reaches an alarm line.
In one embodiment, a computer readable storage medium is provided having a computer program stored thereon, which when executed by a processor, performs the steps of:
selecting a target directory to be monitored according to brief information corresponding to data file directories under each node in the distributed file system stored on the monitoring node, wherein the data file directories are directories corresponding to data files under different nodes in the distributed file system;
acquiring storage information, modification time and/or modifier of a data file under a corresponding node directory in the distributed file system through the target directory;
storing the obtained storage information, modification time and/or modifier of the data file under each node directory in a local file;
and analyzing the storage information, the modification time and/or the modifier of the data file under the directory of each node in the local file according to the preset monitoring requirement to determine whether the data file reaches an alarm line.
Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in the various embodiments provided herein may include non-volatile and/or volatile memory. The nonvolatile memory can include Read Only Memory (ROM), programmable ROM (PROM), electrically Programmable ROM (EPROM), electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double Data Rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous Link DRAM (SLDRAM), memory bus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), among others.
It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-described division of the functional units and modules is illustrated, and in practical application, the above-described functional distribution may be performed by different functional units and modules according to needs, i.e. the internal structure of the apparatus is divided into different functional units or modules to perform all or part of the above-described functions.
The above embodiments are only for illustrating the technical solution of the present invention, and not for limiting the same; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention, and are intended to be included in the scope of the present invention.
Claims (5)
1. A data monitoring method, applied to any monitoring node in a distributed file system, the method comprising:
selecting a target directory to be monitored according to brief information corresponding to data file directories under each node in the distributed file system stored on the monitoring node, wherein the data file directories are directories corresponding to data files under different nodes in the distributed file system;
acquiring storage information, modification time and/or modifier of a data file under a corresponding node directory in the distributed file system through the target directory;
storing the obtained storage information, modification time and/or modifier of the data file under each node directory in a local file;
analyzing the storage information, the modification time and/or the modifier of the data file under the directory of each node in the local file according to the preset monitoring requirement to determine whether the data file reaches an alarm line or not;
the analyzing the storage information of the data file under the directory of each node in the local file according to the preset monitoring requirement to determine whether the data file reaches an alarm line, including:
accumulating according to the storage information of the data files under the node directory to obtain total storage information;
judging whether the total storage amount information exceeds a preset range or not;
if yes, determining that the data file reaches an alarm line;
the analyzing the modification time of the data file under the directory of each node in the local file according to the preset monitoring requirement to determine whether the data file reaches an alarm line, including:
acquiring the starting time and the ending time of writing the data file in the local file; the start time and the end time are respectively the time of writing the data file and the time of last writing the data file;
calculating the writing time length of the data file according to the ending time and the starting time;
judging whether the writing time length is smaller than the time length required by writing the data file in the distributed file system;
if the data file is smaller than the alarm line, determining that the data file reaches the alarm line;
the step of analyzing the modifier of the data file under the directory of each node in the local file according to the preset monitoring requirement to determine whether the data file reaches an alarm line comprises the following steps:
acquiring the user authority of the modifier and the operation authority for writing the data content;
judging whether the user authority of the modifier contains the operation authority;
if the data file does not contain the data file, determining that the data file reaches an alarm line;
after the determining whether the data file reaches the alarm line, the method further comprises:
and sending the information of the data file reaching the alarm line to a server, so that the server displays and alarms the data file reaching the alarm line.
2. The data monitoring method of claim 1, wherein the method further comprises:
when a data file directory under a node and/or brief information corresponding to the data file directory are changed in the distributed file system, obtaining change information of the brief information change corresponding to the data file directory and/or the data file directory;
and updating the data file catalogue of the corresponding node and/or the brief information corresponding to the data file catalogue stored on the monitoring node according to the change information.
3. A data monitoring device, the device comprising:
the acquisition module is used for selecting a target directory to be monitored according to brief information corresponding to data file directories under each node in the distributed file system stored on the monitoring node, wherein the data file directories are directories corresponding to data files under different nodes in the distributed file system;
the acquisition module is further used for acquiring storage information, modification time and/or modifier of the data file under the corresponding node directory in the distributed file system through the target directory;
the storage module is used for storing the storage information, the modification time and/or the modifier of the data file under the obtained node list in the local file;
the determining module is used for analyzing the storage information, the modification time and/or the modifier of the data file under the directory of each node in the local file according to the preset monitoring requirement so as to determine whether the data file reaches an alarm line or not;
the determining module includes:
the computing unit is used for carrying out accumulation computation according to the storage information of the data files under the mesh of each node to obtain total storage information;
the judging unit is used for judging whether the total storage amount information exceeds a preset range;
the determining unit is used for determining that the data file reaches an alarm line if the total storage amount information exceeds a preset range;
the acquisition unit is used for acquiring the starting time and the ending time of writing the data file in the local file; the start time and the end time are respectively the time of writing the data file and the time of last writing the data file;
the calculating unit is further used for calculating the writing time of the data file according to the ending time and the starting time;
the judging unit is further configured to judge whether the writing time length is less than a time length required for writing the data file in the distributed file system;
the determining unit is further configured to determine that the data file reaches an alarm line if the writing time length is determined to be less than a time length required for writing the data in the distributed file system;
the acquisition unit is used for acquiring the user authority of the modifier and the operation authority for writing the data content;
the judging unit is used for judging whether the user permission of the modifier contains the operation permission or not;
the determining unit is used for determining that the data file reaches an alarm line if the user permission of the modifier does not contain the operation permission;
and the sending module is used for sending the information of the data file reaching the alarm line to a server, so that the server displays and alarms the data file reaching the alarm line.
4. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the data monitoring method according to any of claims 1 to 2 when executing the computer program.
5. A computer readable storage medium storing a computer program, characterized in that the computer program, when executed by a processor, implements the data monitoring method according to any one of claims 1 to 2.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910432722.7A CN110309109B (en) | 2019-05-23 | 2019-05-23 | Data monitoring method, device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910432722.7A CN110309109B (en) | 2019-05-23 | 2019-05-23 | Data monitoring method, device, computer equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110309109A CN110309109A (en) | 2019-10-08 |
| CN110309109B true CN110309109B (en) | 2024-02-02 |
Family
ID=68075481
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910432722.7A Active CN110309109B (en) | 2019-05-23 | 2019-05-23 | Data monitoring method, device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110309109B (en) |
Families Citing this family (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111241122B (en) * | 2020-01-07 | 2024-01-19 | 广州虎牙科技有限公司 | Task monitoring method, device, electronic equipment and readable storage medium |
| CN111404735A (en) * | 2020-03-09 | 2020-07-10 | 北京思特奇信息技术股份有限公司 | Distributed application monitoring method and monitoring system |
| CN111935215A (en) * | 2020-06-29 | 2020-11-13 | 广东科徕尼智能科技有限公司 | Internet of things data management method, terminal, system and storage device |
| CN112068901A (en) * | 2020-09-09 | 2020-12-11 | 阳光保险集团股份有限公司 | Configuration information determining method and device |
| CN112799912A (en) * | 2021-01-27 | 2021-05-14 | 苏州浪潮智能科技有限公司 | Data monitoring method, device and system of AMS (automatic monitoring system) |
| CN113836503A (en) * | 2021-08-10 | 2021-12-24 | 中标慧安信息技术股份有限公司 | Internet of things data modification state recording method and system based on block chain |
| CN116561076A (en) * | 2023-05-10 | 2023-08-08 | 合芯科技(苏州)有限公司 | Monitoring method and device for distributed file system, computer equipment and medium |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2011003296A1 (en) * | 2009-07-07 | 2011-01-13 | 中兴通讯股份有限公司 | Distributed management monitoring system, monitoring method and creating method thereof |
| CN103152352A (en) * | 2013-03-15 | 2013-06-12 | 北京邮电大学 | Perfect information security and forensics monitoring method and system based on cloud computing environment |
| CN105610878A (en) * | 2014-10-30 | 2016-05-25 | 腾讯科技（深圳）有限公司 | Directory uploading method, data releasing method, terminal and server |
| WO2017000592A1 (en) * | 2015-06-30 | 2017-01-05 | 中兴通讯股份有限公司 | Data processing method, apparatus and system |
| WO2018103315A1 (en) * | 2016-12-09 | 2018-06-14 | 上海壹账通金融科技有限公司 | Monitoring data processing method, apparatus, server and storage equipment |
| CN108563550A (en) * | 2018-04-23 | 2018-09-21 | 上海达梦数据库有限公司 | A kind of monitoring method of distributed system, device, server and storage medium |
| CN108829560A (en) * | 2018-06-01 | 2018-11-16 | 平安科技（深圳）有限公司 | Data monitoring method, device, computer equipment and storage medium |
-
2019
- 2019-05-23 CN CN201910432722.7A patent/CN110309109B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2011003296A1 (en) * | 2009-07-07 | 2011-01-13 | 中兴通讯股份有限公司 | Distributed management monitoring system, monitoring method and creating method thereof |
| CN103152352A (en) * | 2013-03-15 | 2013-06-12 | 北京邮电大学 | Perfect information security and forensics monitoring method and system based on cloud computing environment |
| CN105610878A (en) * | 2014-10-30 | 2016-05-25 | 腾讯科技（深圳）有限公司 | Directory uploading method, data releasing method, terminal and server |
| WO2017000592A1 (en) * | 2015-06-30 | 2017-01-05 | 中兴通讯股份有限公司 | Data processing method, apparatus and system |
| WO2018103315A1 (en) * | 2016-12-09 | 2018-06-14 | 上海壹账通金融科技有限公司 | Monitoring data processing method, apparatus, server and storage equipment |
| CN108563550A (en) * | 2018-04-23 | 2018-09-21 | 上海达梦数据库有限公司 | A kind of monitoring method of distributed system, device, server and storage medium |
| CN108829560A (en) * | 2018-06-01 | 2018-11-16 | 平安科技（深圳）有限公司 | Data monitoring method, device, computer equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110309109A (en) | 2019-10-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110309109B (en) | Data monitoring method, device, computer equipment and storage medium | |
| CN110086666B (en) | Alarm method, device and system | |
| US20230385170A1 (en) | Systems and methods for collecting, tracking, and storing system performance and event data for computing devices | |
| CN109032824B (en) | Database verification method, database verification device, computer equipment and storage medium | |
| CN112910945B (en) | Request link tracking method and service request processing method | |
| CN109460343A (en) | System exception monitoring method, device, equipment and storage medium based on log | |
| CN110851159B (en) | Business rule updating method and device, computer equipment and storage medium | |
| CN109885624A (en) | Data processing method, device, computer equipment and storage medium | |
| CN110750443A (en) | Webpage testing method and device, computer equipment and storage medium | |
| CN112130996A (en) | Data monitoring control system, method and device, electronic equipment and storage medium | |
| CN108763046B (en) | Thread running monitoring method, device, computer equipment and storage medium | |
| CN112866319B (en) | Log data processing method, system and storage medium | |
| CN112685270A (en) | System monitoring log acquisition method and device, electronic equipment and medium | |
| CN113835921A (en) | Method, device, equipment and storage medium for processing interface service exception | |
| CN112311902A (en) | File sending method and device based on micro-service | |
| CN111949389A (en) | Slurm-based information acquisition method and device, server and computer-readable storage medium | |
| CN111538754A (en) | Data collection management system, method, device, equipment and storage medium | |
| CN113342284B (en) | Time sequence data storage method and device, computer equipment and storage medium | |
| CN106815223B (en) | Mass picture management method and device | |
| CN113419950A (en) | Method and device for generating UI automation script, computer equipment and storage medium | |
| CN113439424B (en) | Establishing a blockchain having blocks including an adjustable number of transaction blocks and a plurality of intermediate blocks | |
| CN116126859A (en) | Data management method and device, electronic equipment and storage medium | |
| CN114996057A (en) | Data backup method and device, electronic equipment and computer readable storage medium | |
| CN113824590A (en) | Method for predicting problem of micro service network, computer device and storage medium | |
| CN113672449A (en) | Intelligent operation and maintenance abnormity monitoring method and device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |