CN110309109A - Data monitoring method, device, computer equipment and storage medium - Google Patents
Data monitoring method, device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN110309109A CN110309109A CN201910432722.7A CN201910432722A CN110309109A CN 110309109 A CN110309109 A CN 110309109A CN 201910432722 A CN201910432722 A CN 201910432722A CN 110309109 A CN110309109 A CN 110309109A
- Authority
- CN
- China
- Prior art keywords
- data file
- data
- file
- modification
- information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000003860 storage Methods 0.000 title claims abstract description 99
- 238000012544 monitoring process Methods 0.000 title claims abstract description 76
- 238000000034 method Methods 0.000 title claims abstract description 38
- 230000004048 modification Effects 0.000 claims abstract description 104
- 238000012986 modification Methods 0.000 claims abstract description 104
- 238000004590 computer program Methods 0.000 claims description 15
- 238000012806 monitoring device Methods 0.000 claims description 13
- 230000008859 change Effects 0.000 claims description 6
- 238000009825 accumulation Methods 0.000 claims description 5
- 238000012545 processing Methods 0.000 abstract description 6
- 230000008569 process Effects 0.000 description 5
- 230000006870 function Effects 0.000 description 4
- 238000000151 deposition Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000009826 distribution Methods 0.000 description 2
- 239000004744 fabric Substances 0.000 description 2
- 230000008901 benefit Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000010485 coping Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000002360 preparation method Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/1734—Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a kind of data monitoring method, device, computer equipment and storage mediums, it is related to technical field of data processing, the main technical solution of the present invention are as follows: the target directory for needing to be monitored by monitoring the corresponding brief information selection of data file content in the distributed file system stored on node under each node；Storage information, modification time, and/or the modification people of data file under corresponding node catalogue in the distributed file system are obtained by the target directory；Storage information, modification time, and/or the modification people of data file under each nodal directory that will acquire are stored in local file；The storage information of the data file under each nodal directory in the local file, modification time, and/or modification people are analyzed according to preset monitoring demand, whether alarming line is reached with the determination data file.Accomplish collect statistics present invention is mainly used for the storage condition to distributed file system entirety and monitors in real time.
Description
Technical field
The present invention relates to technical field of data processing more particularly to a kind of data monitoring method, device, computer equipment and
Storage medium.
Background technique
As the continuous improvement of social informatization technology and Internet technology are quickly popularized, data to be treated are also got over
Come bigger, demand of the every field to mass data processing is also more and more.Single machine memory space and operational capability not
People are able to satisfy under the background of mass data processing demand, distributed computing and parallel computation start fast-developing and application.
The monitoring information of extensive lower distributed system is magnanimity, and monitoring resource is multi-level multi-source, the dynamic of big data,
Complexity brings numerous difficulties to the monitoring system of big data.How data in big data are effectively monitored, in time to data
It is predicted, corresponding measure is taken before failure, be the key that improve big data service quality, and the weight studied at present
Point.
The main feature of big data is exactly distributed file system and distributed computing, therefore to be stored using big data
When centainly need memory space, the data of storage can be increasing with development, the variation of business of enterprise, therefore wants
The variation of storage is monitored in real time to carry out dilatation and the file that removes rubbish when needed.
Current monitoring device is all on the node of device in a distributed system, such benefit be can monitor it is each
The real time execution situation of a node, but do not summarized and the operating condition of monitoring distributed system entirety, for entirety
Storage condition do not accomplish collect statistics even more and monitor in real time.
Summary of the invention
The present invention provides a kind of data monitoring method, device, computer equipment and storage medium, realizes for distribution
The storage condition of file system entirety is accomplished collect statistics and is monitored in real time.
The embodiment of the invention provides a kind of data monitoring methods, applied to any monitoring section in distributed file system
Point, this method comprises:
It is corresponding by the data file content monitored in the distributed file system stored on node under each node
Brief information selection need the target directory that is monitored, the data file content be in the distributed file system not
With catalogue corresponding to data file under node；
The storage of data file under corresponding node catalogue in the distributed file system is obtained by the target directory
Information, modification time, and/or modification people；
Storage information, modification time, and/or the modification people of data file under each nodal directory that will acquire are stored in this
In ground file；
Storage information, modification according to preset monitoring demand to the data file under each nodal directory in the local file
Time, and/or modification people analyze, and whether reach alarming line with the determination data file.
The embodiment of the invention provides a kind of data monitoring device, which includes:
Module is obtained, for the data file under each node in the distributed file system by storing on monitoring node
The target directory that the corresponding brief information selection of catalogue needs to be monitored, the data file content is the distributed document
Catalogue corresponding to data file under different nodes in system；
The acquisition module is also used to obtain corresponding node mesh in the distributed file system by the target directory
Record storage information, modification time, and/or the modification people of lower data file；
Memory module, the storage information of the data file under each nodal directory for will acquire, modification time, and/or
Modification people is stored in local file；
Determining module, for according to preset monitoring demand to the data file under each nodal directory in the local file
Storage information, modification time, and/or modification people analyze, and whether reach alarming line with the determination data file.
A kind of computer equipment, including memory, processor and storage are in the memory and can be in the processing
The computer program run on device, the processor realize above-mentioned data monitoring method when executing the computer program.
A kind of computer readable storage medium, the computer-readable recording medium storage have computer program, the meter
Calculation machine program realizes above-mentioned data monitoring method when being executed by processor.
Above-mentioned data monitoring method, device, computer equipment and storage medium, and at present by distributed system
Each node is respectively provided with monitoring device progress data monitoring and compares, and the present embodiment in distributed file system by monitoring on node
The corresponding brief information selection of data file content in the distributed file system of storage under each node needs to be monitored
Target directory, depositing for data file under corresponding node catalogue is then obtained in the distributed file system by target directory
Store up information, modification time, and/or modification people；And when storage information, the modification of the data file under each nodal directory that will acquire
Between, and/or modification people be stored in local file, finally according to preset monitoring demand to each nodal directory in the local file
Under the storage information of data file, modification time, and/or modification people analyze, whether reached with the determination data file
To alarming line.I.e. the present invention by monitoring node initiate scheduler task can obtain distributed file system operating status,
Memory space situation, when distributed file system be in danger memory space inadequate or operation report an error when monitoring can timely inform
Operation personnel safeguards, so that the stable operation of safeguards system, improves the data monitoring efficiency of distributed file system.
Detailed description of the invention
In order to illustrate the technical solution of the embodiments of the present invention more clearly, below by institute in the description to the embodiment of the present invention
Attached drawing to be used is needed to be briefly described, it should be apparent that, the accompanying drawings in the following description is only some implementations of the invention
Example, for those of ordinary skill in the art, without any creative labor, can also be according to these attached drawings
Obtain other attached drawings.
Fig. 1 is a flow chart of data monitoring method in one embodiment of the invention；
Fig. 2 is to determine whether data file reaches a flow chart of alarming line in one embodiment of the invention；
Fig. 3 is to determine whether data file reaches the another flow chart of alarming line in one embodiment of the invention；
Fig. 4 is to determine whether data file reaches the flow chart of alarming line again in one embodiment of the invention；
Fig. 5 is a schematic diagram of data monitoring device in one embodiment of the invention；
Fig. 6 is a schematic diagram of computer equipment in one embodiment of the invention.
Specific embodiment
Following will be combined with the drawings in the embodiments of the present invention, and technical solution in the embodiment of the present invention carries out clear, complete
Site preparation description, it is clear that described embodiments are some of the embodiments of the present invention, instead of all the embodiments.Based on this hair
Embodiment in bright, every other implementation obtained by those of ordinary skill in the art without making creative efforts
Example, shall fall within the protection scope of the present invention.
In one embodiment, as shown in Figure 1, providing a kind of data monitoring method, this method is applied to distributed field system
Any monitoring node in system, this method comprises the following steps:
S10 is corresponded to by monitoring the data file content in the distributed file system stored on node under each node
Brief information selection need the target directory that is monitored.
Wherein, the data file content is corresponding to data file under nodes different in the distributed file system
Catalogue, brief information are stored in the data content under corresponding data file content for summary record.Target directory can root
It is selected according to the actual demand of user, such as needs to store some file by the file system of big data, but this
File is again important, is backed up, and there are also the operations such as change, then one can be done individually for this file
Monitoring only monitors size, the data situation of this file, while the data file content for needing to be monitored is i.e.
For the catalogue where this document.It should be noted that the embodiment of the present invention is applied in distributed file system, that is, needing to make
It is carried out with can all carry out data monitoring in the scene of big data cluster through the embodiment of the present invention, such as using big data cluster
Calculating task etc. is carried out when storing data, using big data cluster, the embodiment of the present invention is not specifically limited.Wherein, distributed
File system is an entirety of clustered node composition, and a complete data file is stored separately to different nodes
On machine.
It in embodiments of the present invention, specifically can be by the scheduler task that is arranged in any node of distributed file system
Obtain the data file content for needing to monitor.It should be noted that the embodiment of the present invention can select to need according to actual needs
The data file content being monitored, such as the needing to be monitored data file A there are one of the task, then can be according to the need
It asks and is selected in distributed file system under each node about the data file content where data file A.
Wherein, the scheduler task being arranged on node is the task of a lightweight, and scheduler task is to obtain distributed system
The task of storage condition, the program of the inside is the shell script executed, for obtaining the storage condition of distributed system.It needs
Illustrate, the data file mesh in distributed file system under each node is stored in the scheduler task being arranged on node
Record, and brief information corresponding with each data file content, the brief information are stored in corresponding for summary record
Data content under data file content, when needing to be monitored the data file in distributed file system, by depositing
What the corresponding brief information selection of data file content in the distributed file system of storage under each node needed to be monitored
Then data file content transfers the storage letter of each subdata file under respective nodes by the data file content of selection
Breath, modification time, and/or modification people.
For the embodiment of the present invention, do not needed by the service routine that the scheduler task of the lightweight starts too multiple
Miscellaneous, service routine is smaller relative to the pressure of the server of traditional monitoring device, used resource, occupancy memory etc. all
Very little.For the embodiment of the present invention, it is only necessary to start an individual small service and be used to receive what scheduler task sended over
Monitoring information and information is made into behind visual chart interface show can, the needs of server here come out
Program it is seldom, server does not have pressure yet, and small-sized server is sufficient for demand.
S20 obtains in the distributed file system data file under corresponding node catalogue by the target directory
Store information, modification time, and/or modification people.
Wherein, target directory selects to obtain according to actual needs from data file content, and nodal directory be by
Node group corresponding to target directory and the catalogue, i.e., correspondence can be got from distributed file system by nodal directory
Data file under node.In embodiments of the present invention, after getting and needing to carry out target directory, pass through the target of acquisition
Catalogue obtains the storage information of each subdata file under its corresponding node catalogue, modification time and/or modification people.Specifically, dividing
Scheduler task is executed on any one node in cloth file system to realize the storage condition of monitoring distributed system, scheduling
The shell script of task execution can get all catalogues of distributed file system, then the catalogue acquisition pair by obtaining
Newly-built/modification time of file under the size of storage file, catalogue under catalogue is answered, and modifies the information such as people, the embodiment of the present invention
It is not specifically limited, by these information realizations to the data monitoring of distributed file system.
S30, storage information, modification time, and/or the modification people's storage of the data file under each nodal directory that will acquire
In local file.
It should be noted that since the embodiment of the present invention is the number that any one node being distributed in formula file system is initiated
Storage information, the modification of the data file under each node for needing to be monitored are got according to monitor task, therefore in the node
After time, and/or modification people are stored in local file, the storage of the data file under each node for needing will acquire is believed
Breath, modification time, and/or modification people are stored in local file and are stored in local file, that is, the information storage that will acquire is arrived
In the local file of the node, so that the data file for realizing that the monitoring node is monitored needs is monitored.
S40, according to preset monitoring demand to the storage information of the data file under each nodal directory in the local file,
Modification time, and/or modification people analyze, and whether reach alarming line with the determination data file.
For the embodiment of the present invention, by the storage information of the data file under each nodal directory, modification time, and/or
After modification people is stored in local file, need according to preset monitoring demand to the number under each nodal directory in local file
It is analyzed according to the storage information of file, modification time, and/or modification people, whether alarm is reached with the determination data file
Line.Wherein, preset monitoring demand is set according to actual needs, as monitoring data file memory capacity whether be more than
Certain numberical range, the then memory capacity that preset monitoring demand can be arranged to data file are no more than default value range；
Such as the last modification time of monitoring data file, can be by the last modification time that preset monitoring demand is arranged to data file then
No is more than preset time etc., and the embodiment of the present invention is not specifically limited.
The embodiment of the invention provides a kind of data monitoring method, and pass through each node in distributed system at present
It is respectively provided with monitoring device progress data monitoring to compare, the present embodiment is by monitoring point stored on node in distributed file system
The target mesh that the corresponding brief information selection of data file content in cloth file system under each node needs to be monitored
Record, then by target directory obtain data file under corresponding node catalogue in the distributed file system storage information,
Modification time, and/or modification people；And the storage information of the data file under each nodal directory that will acquire, modification time and/
Or modification people is stored in local file, finally according to preset monitoring demand to the number under each nodal directory in the local file
It is analyzed according to the storage information of file, modification time, and/or modification people, whether alarm is reached with the determination data file
Line.When i.e. the present invention can obtain storage information, the modification of distributed file system by the scheduler task that monitoring node is initiated
Between, and/or modification human feelings condition, when distributed file system be in danger memory space inadequate or operation report an error when monitoring can be timely
Informing operation personnel safeguard, thus the stable operation of safeguards system, improve the data monitoring effect of distributed file system
Rate.
As shown in Fig. 2, in one embodiment provided by the invention, step S40: it is described according to preset monitoring demand to institute
The storage information of the data file in local file under each nodal directory is stated, whether alarm is reached with the determination data file
Line, comprising:
S40A1 carries out accumulation calculating according to the storage information under each nodal directory, obtains total storage capacity information.
In embodiments of the present invention, in the case where data file content to be obtained to corresponding node catalogue, the storage of data file is believed
After breath, modification time, and/or modification people's storage are into local file, need according to storage dimensional information to each subdata text
The storage information of part carries out accumulation calculating and obtains the total storage capacity information of data file.
S40A2 judges whether the total storage capacity information is more than presetting range.
S40A3, if being more than, it is determined that the data file reaches alarming line.
Wherein, presetting range is set according to actual needs, for example, presetting range can be set to total storage capacity letter
Breath is more than 80 percent or 85 percent etc. of its load, and the embodiment of the present invention is not specifically limited.
For example, obtaining the storage information of data file under A nodal directory for 11G, under B node catalogue according to target directory
The storage information of data file is 15G, and the storage information of the data file under C nodal directory is 15G, is deposited to each nodal directory
The total storage capacity information that storage information add up to obtain data file is 41G, i.e. the data file storage that is used for 41G
Capacity, if the available memory capacity of the data file is 50G, presetting range is 80 the percent of available storage, then leads to
It crosses and judges that the memory capacity 41G of data file is more than available storage 40G, illustrate that always storing information is more than presetting range, this
When can determine that data file reaches alarming line, warning information can be sent mail to operation personnel automatically by system, can also be sent
Short message notifies operation personnel, automatic poking also can be set phones operation personnel.
As shown in figure 3, in one embodiment provided by the invention, step S40: it is described according to preset monitoring demand to institute
The modification time of the data file in local file under each nodal directory is stated, whether alarm is reached with the determination data file
Line, comprising:
S40B1, obtains at the beginning of the data file is written in the local file and the end time.
Wherein, the modification time of data file include under data file be written at the beginning of and the end time, it is described to open
Time beginning and the end time are respectively written the time of the data file and are ultimately written the time of the data file.
S40B2 calculates the write-in duration of the data file according to the end time and the time started.
For example, being 13:00 on December 5th, 2018 at the beginning of write-in data file, at the end of data file is written
Between be 15:00 on December 5th, 2018, then data file is calculated according to the end time of the data file and time started
A length of 2 hours when write-in.
S40B3, judges whether said write duration is less than needs and the data are written in the distributed file system
The required duration of file.
Wherein, the required duration that the data file is written in the distributed file system, is in distributed document
The practically necessary duration of data file is written in system, required duration can be distributed according to the size of data file and write-in
The speed of file system is determined.
S40B4, if being less than, it is determined that the data file reaches alarming line.
In embodiments of the present invention, it obtains at the beginning of data file is written in local file under data file and ties
The beam time, and the write-in duration of the data file is calculated according to the end time and the time started, then judge institute
It states write-in duration and whether is less than to need that the required durations of the data is written in the distributed file system and said if being less than
Bright data file write-in is imperfect, at this time it needs to be determined that data file reaches alarming line, needs to carry out the data file of write-in
Alarm, to notify backstage manager to make corresponding processing in time.
As shown in figure 4, in one embodiment provided by the invention, step S40: it is described according to preset monitoring demand to institute
Whether the modification people for stating the data file in local file under each nodal directory analyzes, reached with the determination data file
Alarming line, comprising:
S40C1 obtains the user right of the modification people and the operating right of the write-in data content.
It should be noted that different modification people has different user rights, which is previously according to reality
What demand was set, user right is specifically as follows the permission, the permission for reading data content, modification number of write-in data content
According to the permission etc. of content, the embodiment of the present invention is not specifically limited.
Whether S40C2 judges in the user right of the modification people comprising the operating right.
S40C3, if not including, it is determined that the data file reaches alarming line.
For the embodiment of the present invention, the user right for getting modification people and the operating right that data content is written it
Afterwards, judge to modify whether people's user right includes the operating right that data content is written, if comprising illustrating that modifying people has
The permission of data is written in distributed system；If not including, illustrates that modification people does not have and data are written in a distributed system
Permission, can determine that data file reaches alarming line at this time, i.e., the people that data are written in a distributed system is unsatisfactory for not being pre-
The people being first arranged needs to carry out alert process at this time.
In embodiments of the present invention, the data that will determine as reaching the alarming line send a server, so that the clothes
Business device is shown and the data file for reaching the alarming line of alarming.Alarming line be mainly based upon distributed system performance,
The number of nodes of distributed system, distributed system storage size be arranged, when program to be processed is beyond distributed system
The performance load ability of system 90% when, the memory space of the data to be stored beyond distributed system 80% when can all trigger
Alarm；After reaching warning line triggering alarm device, warning information can be sent mail to operation personnel automatically by system, can also be with
Short message is sent to notify operation personnel；Also automatic poking can be set and phone operation personnel.
In one embodiment provided by the invention, the method also includes: exist when in the distributed file system
When the corresponding brief information of data file content and/or the data file content under node changes, data file mesh is obtained
The modification information of record and/or the corresponding brief information change of the data file content；According to modification information update
The data file content and/or the corresponding brief information of the data file content of the corresponding node stored on monitoring node.
It should be understood that the size of the serial number of each step is not meant that the order of the execution order in above-described embodiment, each process
Execution sequence should be determined by its function and internal logic, the implementation process without coping with the embodiment of the present invention constitutes any limit
It is fixed.
In one embodiment, a kind of data monitoring device is provided, data are supervised in the data monitoring device and above-described embodiment
Prosecutor method corresponds.As shown in figure 5, the data monitoring device includes obtaining module 10, memory module 20 and determining module 30.
Detailed description are as follows for each functional module:
Module 10 is obtained, for the data text under each node in the distributed file system by storing on monitoring node
The target directory that the corresponding brief information selection of part catalogue needs to be monitored, the data file content are the distributed text
Catalogue corresponding to data file under different nodes in part system；
The acquisition module 10 is also used to obtain corresponding node in the distributed file system by the target directory
The storage information of data file, modification time, and/or modification people under catalogue；
Memory module 20, the storage information of the data file under each nodal directory for will acquire, modification time and/
Or modification people is stored in local file；
Determining module 30, for according to preset monitoring demand to the data file under each nodal directory in the local file
Storage information, modification time, and/or modification people analyze, whether alarming line is reached with the determination data file.
Specifically, the determining module 30, comprising:
Computing unit 31, for carrying out accumulation calculating according to the storage information of the data file under each nodal directory,
Obtain total storage capacity information；
Judging unit 32, for judging whether the total storage capacity information is more than presetting range；
Determination unit 33, if being more than presetting range for the total storage capacity information, it is determined that the data file reaches
Alarming line.
Specifically, the determining module 30, comprising:
Acquiring unit 34, takes at the beginning of the data file is written in the local file and the end time；It is described
Time started and the end time be respectively be written the time of the data file and be ultimately written the data file when
Between；
The computing unit 31 is also used to calculate the data file according to the end time and the time started
Duration is written；
The judging unit 32, is also used to judge whether said write duration is less than needs in the distributed file system
The required duration of the middle write-in data file；
The determination unit 33 needs in the distributed file system if being also used to judge that said write duration is less than
The required duration of the data is written, it is determined that the data file reaches alarming line.
Specifically, the determination unit 30, comprising:
The acquiring unit 34, for obtaining the user right of the modification people and the operating rights of the write-in data content
Limit；
The judging unit 33, for whether judging in the user right for modifying people comprising the operating right；
The determination unit 33, if for not including the operating right in the user right for modifying people, it is determined that
The data file reaches alarming line.
Further, described device further include:
Sending module 40 is sent to server for being up to the information of data file of the alarming line, so that described
Server is shown and the data file for reaching the alarming line of alarming.
Further, described device further include:
The acquisition module 10 is also used to when there are the data file contents under node in the distributed file system
And/or when the corresponding brief information change of the data file content, obtain data file content and/or the data file mesh
Record the modification information of corresponding brief information change；
Update module, for updating the data of the corresponding node stored on the monitoring node according to the modification information
File directory and/or the corresponding brief information of the data file content.
Specific about data monitoring device limits the restriction that may refer to above for data monitoring method, herein not
It repeats again.Modules in above-mentioned data monitoring device can be realized fully or partially through software, hardware and combinations thereof.On
Stating each module can be embedded in the form of hardware or independently of in the processor in computer equipment, can also store in a software form
In memory in computer equipment, the corresponding operation of the above modules is executed in order to which processor calls.
In one embodiment, a kind of computer equipment is provided, which can be terminal, internal structure
Figure can be as shown in Figure 6.The computer equipment includes processor, the memory, network interface, display connected by system bus
Screen and input unit.Wherein, the processor of the computer equipment is for providing calculating and control ability.The computer equipment is deposited
Reservoir includes non-volatile memory medium, built-in storage.The non-volatile memory medium is stored with operating system and computer journey
Sequence.The built-in storage provides environment for the operation of operating system and computer program in non-volatile memory medium.The calculating
The network interface of machine equipment is used to communicate with external server by network connection.When the computer program is executed by processor with
Realize any one aforementioned data monitoring method.
In one embodiment, a kind of computer equipment is provided, including memory, processor and storage are on a memory
And the computer program that can be run on a processor, processor perform the steps of when executing computer program
It is corresponding by the data file content monitored in the distributed file system stored on node under each node
Brief information selection need the target directory that is monitored, the data file content be in the distributed file system not
With catalogue corresponding to data file under node；
The storage of data file under corresponding node catalogue in the distributed file system is obtained by the target directory
Information, modification time, and/or modification people；
Storage information, modification time, and/or the modification people of data file under each nodal directory that will acquire are stored in this
In ground file；
Storage information, modification according to preset monitoring demand to the data file under each nodal directory in the local file
Time, and/or modification people analyze, and whether reach alarming line with the determination data file.
In one embodiment, a kind of computer readable storage medium is provided, computer program is stored thereon with, is calculated
Machine program performs the steps of when being executed by processor
It is corresponding by the data file content monitored in the distributed file system stored on node under each node
Brief information selection need the target directory that is monitored, the data file content be in the distributed file system not
With catalogue corresponding to data file under node；
The storage of data file under corresponding node catalogue in the distributed file system is obtained by the target directory
Information, modification time, and/or modification people；
Storage information, modification time, and/or the modification people of data file under each nodal directory that will acquire are stored in this
In ground file；
Storage information, modification according to preset monitoring demand to the data file under each nodal directory in the local file
Time, and/or modification people analyze, and whether reach alarming line with the determination data file.
Those of ordinary skill in the art will appreciate that realizing all or part of the process in above-described embodiment method, being can be with
Relevant hardware is instructed to complete by computer program, the computer program can be stored in a non-volatile computer
In read/write memory medium, the computer program is when being executed, it may include such as the process of the embodiment of above-mentioned each method.Wherein,
To any reference of memory, storage, database or other media used in each embodiment provided herein,
Including non-volatile and/or volatile memory.Nonvolatile memory may include read-only memory (ROM), programming ROM
(PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM) or flash memory.Volatile memory may include
Random access memory (RAM) or external cache.By way of illustration and not limitation, RAM is available in many forms,
Such as static state RAM (SRAM), dynamic ram (DRAM), synchronous dram (SDRAM), double data rate sdram (DDRSDRAM), enhancing
Type SDRAM (ESDRAM), synchronization link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM
(RDRAM), direct memory bus dynamic ram (DRDRAM) and memory bus dynamic ram (RDRAM) etc..
It is apparent to those skilled in the art that for convenience of description and succinctly, only with above-mentioned each function
Can unit, module division progress for example, in practical application, can according to need and by above-mentioned function distribution by different
Functional unit, module are completed, i.e., the internal structure of described device is divided into different functional unit or module, more than completing
The all or part of function of description.
Embodiment described above is merely illustrative of the technical solution of the present invention, rather than its limitations；Although referring to aforementioned reality
Applying example, invention is explained in detail, those skilled in the art should understand that: it still can be to aforementioned each
Technical solution documented by embodiment is modified or equivalent replacement of some of the technical features；And these are modified
Or replacement, the spirit and scope for technical solution of various embodiments of the present invention that it does not separate the essence of the corresponding technical solution should all
It is included within protection scope of the present invention.
Claims (10)
1. a kind of data monitoring method, which is characterized in that applied to any monitoring node in distributed file system, the side
Method includes:
Pass through the corresponding letter of data file content monitored in the distributed file system stored on node under each node
The target directory for wanting information selection to need to be monitored, the data file content are different sections in the distributed file system
Catalogue corresponding to the lower data file of point；
By the target directory obtain data file under corresponding node catalogue in the distributed file system storage information,
Modification time, and/or modification people；
Storage information, modification time, and/or the modification people of data file under each nodal directory that will acquire are stored in local text
In part；
When according to preset monitoring demand to the storage information of the data file under each nodal directory in the local file, modification
Between, and/or modification people analyze, whether alarming line is reached with the determination data file.
2. data monitoring method according to claim 1, which is characterized in that it is described according to preset monitoring demand to described
The storage information of data file in ground file under each nodal directory is analyzed, and whether reaches report with the determination data file
Alert line, comprising:
Accumulation calculating is carried out according to the storage information of the data file under each nodal directory, obtains total storage capacity information；
Judge whether the total storage capacity information is more than presetting range；
If being more than, it is determined that the data file reaches alarming line.
3. data monitoring method according to claim 1, which is characterized in that it is described according to preset monitoring demand to described
The modification time of data file in ground file under each nodal directory is analyzed, and whether reaches report with the determination data file
Alert line, comprising:
It obtains at the beginning of the data file is written in the local file and the end time；Time started and described
End time is respectively written the time of the data file and is ultimately written the time of the data file；
The write-in duration of the data file is calculated according to the end time and the time started；
Judge whether said write duration is less than needs and is written needed for the data file in the distributed file system
Duration；
If being less than, it is determined that the data file reaches alarming line.
4. data monitoring method according to claim 1, which is characterized in that it is described according to preset monitoring demand to described
The modification people of data file in ground file under each nodal directory analyzes, and whether reaches alarm with the determination data file
Line, comprising:
Obtain the user right of the modification people and the operating right of the write-in data content；
Whether judge in the user right of the modification people comprising the operating right；
If not including, it is determined that the data file reaches alarming line.
5. any one of -4 data monitoring method according to claim 1, which is characterized in that the determination data file is
It is no reach alarming line after, the method also includes:
The information for being up to the data file of the alarming line is sent to server, so that the server shows and alarms described
Reach the data file of the alarming line.
6. data monitoring method according to claim 1, which is characterized in that the method also includes:
When in the distributed file system there are under node data file content and/or the data file content it is corresponding
When brief information changes, the change of data file content and/or the corresponding brief information change of the data file content is obtained
Information；
The data file content of corresponding node stored on the monitoring node and/or described is updated according to the modification information
The corresponding brief information of data file content.
7. a kind of data monitoring device, which is characterized in that described device includes:
Module is obtained, for the data file content under each node in the distributed file system by storing on monitoring node
The target directory that corresponding brief information selection needs to be monitored, the data file content are the distributed file system
Catalogue corresponding to data file under middle difference node；
The acquisition module is also used to obtain in the distributed file system under corresponding node catalogue by the target directory
Storage information, modification time, and/or the modification people of data file；
Memory module, storage information, modification time, and/or the modification of the data file under each nodal directory for will acquire
People is stored in local file；
Determining module, for the storage according to preset monitoring demand to the data file under each nodal directory in the local file
Information, modification time, and/or modification people analyze, and whether reach alarming line with the determination data file.
8. data monitoring device according to claim 7, which is characterized in that the determining module, comprising:
Computing unit obtains total for carrying out accumulation calculating according to the storage information of the data file under each nodal directory
Amount of storage information；
Judging unit, for judging whether the total storage capacity information is more than presetting range；
Determination unit, if being more than presetting range for the total storage capacity information, it is determined that the data file reaches alarming line.
9. a kind of computer equipment, including memory, processor and storage are in the memory and can be in the processor
The computer program of upper operation, which is characterized in that the processor realized when executing the computer program as claim 1 to
Any one of 6 data monitoring methods.
10. a kind of computer readable storage medium, the computer-readable recording medium storage has computer program, and feature exists
In realization data monitoring method as described in any one of claim 1 to 6 when the computer program is executed by processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910432722.7A CN110309109B (en) | 2019-05-23 | 2019-05-23 | Data monitoring method, device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910432722.7A CN110309109B (en) | 2019-05-23 | 2019-05-23 | Data monitoring method, device, computer equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110309109A true CN110309109A (en) | 2019-10-08 |
| CN110309109B CN110309109B (en) | 2024-02-02 |
Family
ID=68075481
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910432722.7A Active CN110309109B (en) | 2019-05-23 | 2019-05-23 | Data monitoring method, device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110309109B (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111241122A (en) * | 2020-01-07 | 2020-06-05 | 广州虎牙科技有限公司 | Task monitoring method and device, electronic equipment and readable storage medium |
| CN111404735A (en) * | 2020-03-09 | 2020-07-10 | 北京思特奇信息技术股份有限公司 | Distributed application monitoring method and monitoring system |
| CN111935215A (en) * | 2020-06-29 | 2020-11-13 | 广东科徕尼智能科技有限公司 | Internet of things data management method, terminal, system and storage device |
| CN112068901A (en) * | 2020-09-09 | 2020-12-11 | 阳光保险集团股份有限公司 | Configuration information determining method and device |
| CN112799912A (en) * | 2021-01-27 | 2021-05-14 | 苏州浪潮智能科技有限公司 | Data monitoring method, device and system of AMS (automatic monitoring system) |
| CN113836503A (en) * | 2021-08-10 | 2021-12-24 | 中标慧安信息技术股份有限公司 | Internet of things data modification state recording method and system based on block chain |
| CN116561076A (en) * | 2023-05-10 | 2023-08-08 | 合芯科技(苏州)有限公司 | Monitoring method and device for distributed file system, computer equipment and medium |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2011003296A1 (en) * | 2009-07-07 | 2011-01-13 | 中兴通讯股份有限公司 | Distributed management monitoring system, monitoring method and creating method thereof |
| CN103152352A (en) * | 2013-03-15 | 2013-06-12 | 北京邮电大学 | Perfect information security and forensics monitoring method and system based on cloud computing environment |
| CN105610878A (en) * | 2014-10-30 | 2016-05-25 | 腾讯科技（深圳）有限公司 | Directory uploading method, data releasing method, terminal and server |
| WO2017000592A1 (en) * | 2015-06-30 | 2017-01-05 | 中兴通讯股份有限公司 | Data processing method, apparatus and system |
| WO2018103315A1 (en) * | 2016-12-09 | 2018-06-14 | 上海壹账通金融科技有限公司 | Monitoring data processing method, apparatus, server and storage equipment |
| CN108563550A (en) * | 2018-04-23 | 2018-09-21 | 上海达梦数据库有限公司 | A kind of monitoring method of distributed system, device, server and storage medium |
| CN108829560A (en) * | 2018-06-01 | 2018-11-16 | 平安科技（深圳）有限公司 | Data monitoring method, device, computer equipment and storage medium |
-
2019
- 2019-05-23 CN CN201910432722.7A patent/CN110309109B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2011003296A1 (en) * | 2009-07-07 | 2011-01-13 | 中兴通讯股份有限公司 | Distributed management monitoring system, monitoring method and creating method thereof |
| CN103152352A (en) * | 2013-03-15 | 2013-06-12 | 北京邮电大学 | Perfect information security and forensics monitoring method and system based on cloud computing environment |
| CN105610878A (en) * | 2014-10-30 | 2016-05-25 | 腾讯科技（深圳）有限公司 | Directory uploading method, data releasing method, terminal and server |
| WO2017000592A1 (en) * | 2015-06-30 | 2017-01-05 | 中兴通讯股份有限公司 | Data processing method, apparatus and system |
| WO2018103315A1 (en) * | 2016-12-09 | 2018-06-14 | 上海壹账通金融科技有限公司 | Monitoring data processing method, apparatus, server and storage equipment |
| CN108563550A (en) * | 2018-04-23 | 2018-09-21 | 上海达梦数据库有限公司 | A kind of monitoring method of distributed system, device, server and storage medium |
| CN108829560A (en) * | 2018-06-01 | 2018-11-16 | 平安科技（深圳）有限公司 | Data monitoring method, device, computer equipment and storage medium |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111241122A (en) * | 2020-01-07 | 2020-06-05 | 广州虎牙科技有限公司 | Task monitoring method and device, electronic equipment and readable storage medium |
| CN111241122B (en) * | 2020-01-07 | 2024-01-19 | 广州虎牙科技有限公司 | Task monitoring method, device, electronic equipment and readable storage medium |
| CN111404735A (en) * | 2020-03-09 | 2020-07-10 | 北京思特奇信息技术股份有限公司 | Distributed application monitoring method and monitoring system |
| CN111935215A (en) * | 2020-06-29 | 2020-11-13 | 广东科徕尼智能科技有限公司 | Internet of things data management method, terminal, system and storage device |
| CN112068901A (en) * | 2020-09-09 | 2020-12-11 | 阳光保险集团股份有限公司 | Configuration information determining method and device |
| CN112799912A (en) * | 2021-01-27 | 2021-05-14 | 苏州浪潮智能科技有限公司 | Data monitoring method, device and system of AMS (automatic monitoring system) |
| CN113836503A (en) * | 2021-08-10 | 2021-12-24 | 中标慧安信息技术股份有限公司 | Internet of things data modification state recording method and system based on block chain |
| CN116561076A (en) * | 2023-05-10 | 2023-08-08 | 合芯科技(苏州)有限公司 | Monitoring method and device for distributed file system, computer equipment and medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110309109B (en) | 2024-02-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110309109A (en) | Data monitoring method, device, computer equipment and storage medium | |
| CN106878064B (en) | Data monitoring method and device | |
| CN105005532B (en) | A kind of system and method for automatic test application programming interfaces stability | |
| US20070288275A1 (en) | It services architecture planning and management | |
| US20080155386A1 (en) | Network discovery system | |
| CN107329741A (en) | A kind of software distribution upgrade method and device based on fingerprint recognition | |
| WO2015090379A1 (en) | Resource prediction for cloud computing | |
| CN109412852A (en) | Alarm method, device, computer equipment and storage medium | |
| CN109639751A (en) | Block chain node monitoring method, device, system and computer storage medium | |
| CN109885624A (en) | Data processing method, device, computer equipment and storage medium | |
| CN108769153B (en) | Data processing method and system for network application | |
| CN109543891A (en) | Method for building up, equipment and the computer readable storage medium of capacity prediction model | |
| CN109710440A (en) | Abnormality eliminating method, device, storage medium and the terminal device of webpage front-end | |
| CN108052385A (en) | A kind of Container Management method, system, equipment and computer storage media | |
| CN113391901A (en) | RPA robot management method, device, equipment and storage medium | |
| CN114528085A (en) | Resource scheduling method, device, computer equipment, storage medium and program product | |
| CN113220540A (en) | Service management method, device, computer equipment and storage medium | |
| CN113382071B (en) | Link creation method and device based on hybrid cloud architecture | |
| CN112699136B (en) | Cross-link certificate storage method and related device | |
| CN106161122A (en) | A kind of automatization's Centralized Monitoring method for early warning and system | |
| CN110784347A (en) | Node management method, system, equipment and storage medium for container cluster | |
| CN108696519A (en) | A kind of webpage tamper resistant systems and method based on shared storage | |
| CN112561506A (en) | Live broadcast data processing method, system, device and medium based on virtual currency | |
| CN116126859A (en) | Data management method and device, electronic equipment and storage medium | |
| CN116012017A (en) | Resource information processing method, device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |