CN114238220A - File monitoring system - Google Patents
File monitoring system Download PDFInfo
- Publication number
- CN114238220A CN114238220A CN202111522798.2A CN202111522798A CN114238220A CN 114238220 A CN114238220 A CN 114238220A CN 202111522798 A CN202111522798 A CN 202111522798A CN 114238220 A CN114238220 A CN 114238220A
- Authority
- CN
- China
- Prior art keywords
- monitoring
- file
- event
- document
- scheduler
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/1734—Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The embodiment of the application provides a file monitoring system. The document monitoring system includes: a command line parser: the method comprises the steps that a configuration file of a monitoring job corresponding to a user command is read in a configuration file table in response to the user command, wherein parameters of the configuration file of the monitoring job comprise a file name rule, a monitoring condition, a monitoring event, a monitoring mode and file metadata; a scheduler: the configuration file is used for scheduling the execution sequence of the corresponding monitoring operation according to the configuration file; a file monitor: the monitoring system is used for monitoring the monitoring operation which is called to be executed, and triggering the corresponding monitoring event when the monitoring operation meets the corresponding monitoring condition; an event handler: and the monitoring device is used for executing a corresponding monitoring instruction when the monitoring event is received. The embodiment of the invention provides file integrity guarantee for the upstream and downstream components of the software system.
Description
Technical Field
The application relates to the technical field of computers, in particular to a file monitoring system.
Background
Document monitoring is one of the important technologies for communicating components upstream and downstream of a software system. When the upstream component creates, deletes, moves and modifies the file, the downstream component needs to sense the specified change of the file and execute corresponding processing. For example, a data collection component in a software system collects data from a specific data source and transmits the data to a designated directory, a file monitor notifies a data processing component, and the data processing component reads a corresponding file and executes certain processing. With the development of computer technology, file monitoring has been widely applied in the fields of operating systems, job scheduling systems, code auditing, hot-loading configuration files, automated deployment, and the like.
A common inotify or inotify packaging tool for file monitoring under the existing Linux operating system, such as fsnotify; the file monitoring under the Windows operating system is commonly used by two interfaces of FindFirstChangeNotification and ReadDirectoryChangesW; the file monitoring under the cross-Linux and Windows operating systems is the WatchService of common jnotify and Java NIO. They can monitor events under the directory, such as file creation, file deletion, file modification and file metadata modification, and can customize the processor of the event according to the events.
The existing file monitoring mode has the following disadvantages: 1) existing and completely transmitted files cannot be monitored; 2) the event that the file monitoring process is lost from accidental closing to process recovery cannot be processed; 3) and the integrity of the file is guaranteed without native support.
Disclosure of Invention
The embodiment of the application aims to provide a file monitoring method, a file monitoring device and a file monitoring system, so as to solve the problems existing in the existing file monitoring.
In order to achieve the above object, an embodiment of the present invention provides a file monitoring system, including: a command line parser: the method comprises the steps that a configuration file of a monitoring job corresponding to a user command is read in a configuration file table in response to the user command, wherein parameters of the configuration file of the monitoring job comprise a file name rule, a monitoring condition, a monitoring event, a monitoring mode and file metadata; a scheduler: the configuration file is used for scheduling the execution sequence of the corresponding monitoring operation according to the configuration file; a file monitor: the monitoring system is used for monitoring the monitoring operation which is called to be executed, and triggering the corresponding monitoring event when the monitoring operation meets the corresponding monitoring condition; an event handler: and the monitoring device is used for executing a corresponding monitoring instruction when the monitoring event is received.
Optionally, the file name rule includes a regular expression of the indication directory and the corresponding file name of the monitoring job.
Optionally, the monitoring event includes: file creation events, file modification events, file deletion events, and file arrival events.
Optionally, the monitoring mode includes a standard mode, a cyclic mode, and a template mode.
Optionally, the scheduler is further configured to: instantiating a configuration file for the monitoring job; and registering the event processor according to the monitoring mode.
Optionally, the command line parser is further configured to: detecting whether the format of the user command is legal or not; analyzing whether the format of the configuration file corresponding to the user command is legal or not; and if one of the format of the user command and the format of the corresponding configuration file is illegal, not responding to the user command.
Optionally, the scheduler is further configured to: adding the corresponding monitoring events into a monitoring queue according to the execution sequence of the monitoring operation to prepare for being called and executed; removing the monitoring event from the monitoring queue after the monitoring event in the monitoring operation is completed; when the monitoring events which are not completed exist in the monitoring operation, scheduling the execution sequence of the monitoring events; and when the monitoring operation is finished monitoring, removing the monitoring operation from the monitoring queue.
Optionally, the monitoring job that is invoked for execution, and when the monitoring job meets the corresponding monitoring condition, triggering the corresponding monitoring event includes: when the monitoring operation is called up and executed, scanning all files under the corresponding file name rule indication directory according to a monitoring strategy; filtering all files under the directory according to the file name rule and the monitoring condition; and triggering the monitoring event when the monitoring file exists after filtering.
Optionally, the file monitor is further configured to: recording the metadata of the metadata file, wherein the metadata of the file is the operating system data of the monitoring file, and the method comprises the following steps: the file name, the path, the owner, the authority information, the file creating time, the last modifying time and the file size of the monitoring file.
Optionally, the file monitor is further configured to: when the monitoring file exists after filtering, reporting the state of the monitoring operation to the scheduler, and sending a monitoring event to an event processor; reporting to the scheduler that there is no monitoring event when the monitoring file does not exist after filtering.
Through the technical scheme, the embodiment of the invention provides file integrity guarantee for the upstream component and the downstream component of the software system, only after the upstream component completely transmits the file, the file arrival event is triggered, and the monitoring event processor is executed according to the service requirement to inform the downstream component of processing the event, so that the data loss caused by the early processing of the incomplete file by the downstream component is avoided. The embodiment of the invention can be used by only declaring in the configuration file of the monitoring operation when the monitoring process is executed by continuously scanning the monitoring operation in the configuration file table, is convenient for development or service personnel to use, and is fast adapted to the requirement of high-speed change.
Additional features and advantages of embodiments of the present application will be described in detail in the detailed description which follows.
Drawings
The accompanying drawings, which are included to provide a further understanding of the embodiments of the disclosure and are incorporated in and constitute a part of this specification, illustrate embodiments of the disclosure and together with the description serve to explain the embodiments of the disclosure, but are not intended to limit the embodiments of the disclosure. In the drawings:
FIG. 1 is a schematic structural diagram of a document monitoring system according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of an example flow performed by the file monitor;
FIG. 3 is a schematic flow chart illustrating operation of an example file monitoring system.
Description of the reference numerals
10 Command line parser 20 scheduler
30 file monitor 40 event handler
Detailed Description
To make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions of the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it should be understood that the specific embodiments described herein are only used for illustrating and explaining the embodiments of the present application and are not used for limiting the embodiments of the present application. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
It should be noted that if directional indications (such as up, down, left, right, front, and back … …) are referred to in the embodiments of the present application, the directional indications are only used to explain the relative positional relationship between the components, the movement situation, and the like in a specific posture (as shown in the drawings), and if the specific posture is changed, the directional indications are changed accordingly.
In addition, if there is a description of "first", "second", etc. in the embodiments of the present application, the description of "first", "second", etc. is for descriptive purposes only and is not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In addition, technical solutions between various embodiments may be combined with each other, but must be realized by a person skilled in the art, and when the technical solutions are contradictory or cannot be realized, such a combination should not be considered to exist, and is not within the protection scope of the present application.
Before describing the embodiments of the present invention in detail, the drawbacks of the prior art and the design idea of the embodiments of the present invention are introduced.
A common inotify or inotify packaging tool for file monitoring under the existing Linux operating system, such as fsnotify; the file monitoring under the Windows operating system is commonly used by two interfaces of FindFirstChangeNotification and ReadDirectoryChangesW; the file monitoring under the cross-Linux and Windows operating systems is the WatchService of common jnotify and Java NIO. They can monitor events under the directory, such as file creation, file deletion, file modification and file metadata modification, and can customize the processor of the event according to the events.
In view of implementation effect, the above file monitoring depends on a specific operating system, and by registering the event type to be monitored with the operating system, the operating system writes the event into the cache after triggering the event of the specified type, and the monitoring program needs to poll and read the cache to obtain the event, and then process the event. Or the monitoring framework realizes the process, the monitoring program only needs to register the monitoring file and the corresponding event processor, and the monitoring framework can automatically acquire the event and execute the corresponding event processor.
Therefore, the existing file monitoring has the following problems:
1) the existing file monitoring is to monitor the state of the file in an increment mode, and for the existing and completely transmitted file, a corresponding event processor cannot be monitored and triggered according to the existing file monitoring program. Therefore, file monitoring needs to be started and continuously monitored before the upstream component starts to run.
2) When the file monitoring program stops the process due to software design defects, human accidents, system failures or nonreactive factors, and the upstream components work normally, events during the process recovery cannot be processed, and manual intervention is needed.
3) The existing file monitoring has no native support to judge whether the file is transmitted to the end. When an upstream component in a software system continues to transmit a particular file, if the file monitor receives a file creation event and feeds back to a downstream processing component, the downstream component may not completely process the data and the file is considered processed.
4) The existing file monitoring supports developers to customize event processors, but the originally configurable event processors are not provided, so that after business personnel need to clarify requirements with the developers, the developers can develop the file, and the cooperation efficiency is reduced. The event processor is generally hard-coded in the monitoring program, and with the change of service requirements in a software system, a new event processor needs to be re-developed, compiled and packaged, so that the service requirements cannot be quickly adapted in a mode of not modifying the monitoring program according to the addition, deletion and configuration information.
Fig. 1 is a schematic structural diagram of a document monitoring system according to an embodiment of the present invention, and it will be understood by those skilled in the art that the structure shown in fig. 1 is only a block diagram of a part of the structure related to the present application, and does not constitute a limitation to a computer device to which the present application is applied, and a specific computer device may include more or less components than those shown in the figure, or combine some components, or have a different arrangement of components.
Referring to fig. 1, the document monitoring system may include: the command line parser 10: the configuration file is used for responding to a user command and reading a configuration file of a monitoring job corresponding to the user command in a configuration file table; the scheduler 20: the configuration file is used for scheduling the execution sequence of the corresponding monitoring operation according to the configuration file; the file monitor 30: the monitoring system is used for monitoring the monitoring operation which is called to be executed, and triggering the corresponding monitoring event when the monitoring operation meets the corresponding monitoring condition; the event handler 40: and the monitoring device is used for executing a corresponding monitoring instruction when the monitoring event is received.
The embodiment of the invention carries out monitoring process control by taking monitoring operation as a unit, wherein the parameters of the configuration file of the monitoring operation can comprise a file name rule, a monitoring condition, a monitoring event, a monitoring mode and file metadata.
By way of example, the profile table ruleFile of the monitoring job may include profiles of a plurality of monitoring jobs, and may also be configured with some global parameters, and may be set as global parameters by default if the parameters of the profiles of the monitoring jobs do not override this parameter.
For the parameters of the configuration file, a brief explanation is here made, the details and the use of which are further explained at the reference:
the file name rule may include a regular expression indicating a directory and a corresponding file name of the monitoring job.
For example, filename/"/data/input/datatime/$", may instruct the file monitor to scan files under "/data/input/datatime/" directory that conform to such regular expression ". Lambda _ mydata $".
The file name rule may also indicate what monitoring policy (monitoring mode) the file monitor employs, specify a ruleFile path, and service system parameters, etc.
The business system parameters may be used as macros in the ruleFile. After configuration of the configuration file of the general monitoring operation is finished, the configuration file is not changed, and a user can monitor different file name rule scanning and output some specific values and the like only by modifying a user command.
The monitoring condition can comprise a statistical index and a non-statistical index. The statistical indexes comprise the number of continuous file monitoring times after the file size of the monitoring file is unchanged, and the non-statistical indexes comprise the file minimum size of the monitoring file, the time when the current time is the least time from the last modification time, and the like.
By way of example, minDetect:3, indicates that a file size is monitored for a minimum of 3 consecutive times without change, and a specific event can be issued for the file;
minSize is 10M, the minimum file reaches 10MB, and a specific event can be sent out aiming at the file;
the minor is 3min, which indicates that the current system time is at least 3 minutes away from the last modification time of the file, and a specific event can be sent out aiming at the file;
maxAge: "10 min," meaning that the current system time is no more than 10 minutes from the last time the file was modified, before a particular event can be issued for that file.
The monitoring events may include: file creation event, file modification event, file deletion event, and file arrival event:
1) file creation event, corresponding event triggered by the file monitor when the file is created.
2) And a file modification event, namely a corresponding event triggered by the file monitor when the file is modified.
3) And a file deleting event, wherein when the file is deleted, the file monitor triggers a corresponding event.
4) File arrival events, where the definition does not exist in other homogeneous file monitoring programs. In a narrow sense, when a file has no content, a file arrival event is a file creation event. Based on the semantics of file integrity, the file arrival time may refer to a corresponding event triggered by the file monitor after the file has been completely transmitted to the specified directory, which includes a file creation event and several file modification events. The semantics of file integrity may be defined by monitoring conditions. It can be understood that: a file creation event is a set of sub-events of a file arrival event.
The monitoring mode may be set according to a monitoring policy, which may decide how the scheduler schedules execution of the monitoring job. Preferably, the monitoring mode may include a standard mode, a cyclic mode, and a template mode, and the corresponding monitoring policy may include: a standard pattern strategy, a cyclic pattern strategy, and a template pattern strategy.
Wherein, 1) standard mode: after the monitoring operation is successful/failed (end state), the monitoring operation is not monitored, and the monitoring process is quitted after all the monitoring operation is finished.
2) Circulation mode: after the monitoring operation succeeds/fails, as long as the end time point expected by the user is not reached, the monitoring operation state and the index are reset, and the scheduling execution is continued.
3) Template mode: the template mode needs to be used in addition to any of the above two modes. A template of an event handler is defined in ruleFile, and when a monitoring job is instantiated, the corresponding event handler 40 is registered according to the specified template.
By way of example, the filename rule filename: "/data/input/^ data $",
mode: "CREATE" # expects to monitor file arrivals
thenAction:
- { doCmd: "monitoring successfully executed operating system terminal command" }
- { doCond: "monitor for successful execution and issue specific instruction to external system" }
elseAction:
- { doCmd: "monitoring failed operating system termination command executed" }
Among other things, the action and the elseAction serve as functions that the event handler 40 can perform. For example, the event handler 40 executes the then action upon successfully receiving a file arrival event; the execution of an elseAction is monitored for failure. The doCmd and doCond can be used as a processing sequence to be executed, a processing method predefined for a system program, and a user registration self-defined function can be supported, like defining a processing template corresponding to the event handler, and the corresponding template can be used through instantiation.
The file metadata may be operating system data of the monitoring file, which may include: the file name, the path, the owner, the authority information, the file creating time, the last modification time, the file size and the like of the monitoring file.
Preferably, the command line parser 10 is further operable to: detecting whether the format of the user command is legal or not; analyzing whether the format of the configuration file corresponding to the user command is legal or not; and if one of the format of the user command and the format of the corresponding configuration file is illegal, not responding to the user command.
By way of example, the command line parser 10 may read and parse a user command from the terminal, check whether the user command and the configuration file of the corresponding monitoring job are legal, and exit the program if not, so as to avoid passing an error of the user command to other modules.
Preferably, the scheduler 20 is further configured to: instantiating a configuration file for the monitoring job; and registering the event processor according to the monitoring mode.
By way of example, after the scheduler obtains the configuration file corresponding to the monitoring job, the monitoring job may be scheduled according to the monitoring mode indicated by the file name rule, that is, the execution sequence of the monitoring job is determined, and may also be scheduled according to a scheduling algorithm, such as first-in first-out FIFO, and the like; instantiating the monitoring operation and registering the corresponding event handler. If the monitoring mode indicates that the monitoring operation is in the template mode, an event handler can be generated according to the event handler template; if not, an event handler is generated from the description of each monitoring job.
Preferably, the scheduler 20 is further configured to: adding the corresponding monitoring events into a monitoring queue according to the execution sequence of the monitoring operation to prepare for being called and executed; removing the monitoring event from the monitoring queue after the monitoring event in the monitoring operation is completed; when the monitoring events which are not completed exist in the monitoring operation, scheduling the execution sequence of the monitoring events; and when the monitoring operation is finished monitoring, removing the monitoring operation from the monitoring queue.
By way of example, the scheduler 20 reads the configuration description of the monitoring job instantiated in ruleFile in real time, and after the monitoring job completes monitoring, the scheduler 20 may remove the monitoring job from the monitoring queue; when there is still a monitor job that is not completed, the scheduler 20 continues to schedule its execution order, and starts execution of the uncompleted tasks in order. When all monitoring jobs have completed monitoring, the scheduler 20 is responsible for cleaning jobs, exiting the file monitoring program, and the like.
Preferably, the monitor 30 may be configured to monitor the monitoring job that is invoked for execution, and when the monitoring job satisfies the corresponding monitoring condition, trigger the corresponding monitoring event, which may include: when the monitoring operation is called up and executed, scanning all files under the corresponding file name rule indication directory according to a monitoring strategy; filtering all files under the directory according to the file name rule and the monitoring condition; and triggering the monitoring event when the monitoring file exists after filtering.
Preferably, the file monitor 30 is further configured to: and recording the metafile data.
Further preferably, the file monitor 30 is further configured to: reporting a status of the monitoring job to the scheduler, the event handler; reporting to the scheduler that there is no monitoring event when the monitoring file does not exist after filtering.
By way of example, the file monitor may scan all files in the directory indicated by the file name rule according to the above monitoring policy (for example, scan files in "/data/input/datatime/" directory, which conform to "^ mydata $" like regular expression), find a file to be monitored when there is a monitored file after filtering, trigger a monitoring event (for example, modify the file), and record file metadata of the file.
If the file metadata is recorded, it indicates that the file to be monitored may have been previously monitored by the executed file, but due to some reason (e.g., program necrosis or device power failure), the file monitoring is interrupted, the file monitoring process may be restarted, the file metadata may be automatically reconstructed, and the statistical indicator may be recorded again.
After filtering the files to be monitored, the file monitor 30 reports to the scheduler 20 to inform the scheduler to schedule the monitoring event; at the same time, the event handler 40 is notified that the event handler 40 will perform the corresponding processing. When the file to be monitored is not filtered, the scheduler 20 is also notified to wait for the next scheduling by the scheduler.
The scheduler 20 will collect the status of the monitoring job, remove the successful or failed monitoring event or monitoring job from the scheduling queue, or reset the status of the job and all monitoring indexes under the condition that the user designates continuous monitoring, and reschedule; the file monitor 30 collects and counts the monitoring index of each job, and if the monitoring condition is satisfied, a corresponding event is triggered.
FIG. 2 is a schematic diagram of an example flow performed by the file monitor. It should be understood that although the steps in the flow chart of fig. 2 are not strictly limited to the order in which they are performed unless explicitly stated otherwise herein, the steps may be performed in other orders. Moreover, at least a portion of the steps in fig. 2 may include multiple sub-steps or multiple stages that are not necessarily performed at the same time, but may be performed at different times, and the order of performance of the sub-steps or stages is not necessarily sequential, but may be performed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.
Referring to fig. 2, the file monitoring method may include the following steps:
step S11: scanning the directory where the monitoring operation is located according to a file name rule indicated by a configuration file of the monitoring operation;
step S12: filtering all files in the directory according to a file name rule and a monitoring condition;
step S13: recording file metadata of the filtered file, and updating the file metadata when the file metadata already has recorded data;
step S14: whether the record information contains a monitoring event concerned by the monitoring operation or not is judged;
step S15: if the monitoring event exists, reporting the file which needs to be monitored to a scheduler, sending the monitoring event and simultaneously informing a monitoring event processor;
step S16: if the monitoring event does not exist, the file needing to be monitored is not found in the scheduling report, and the next scheduling of the scheduler is waited.
FIG. 3 is a schematic flow chart illustrating an exemplary operation of a document monitoring system, and it should be understood that although the steps are not strictly limited in order to be performed in the flow chart of FIG. 3 unless explicitly stated otherwise herein, the steps may be performed in other orders. Moreover, at least a portion of the steps in fig. 3 may include multiple sub-steps or multiple stages that are not necessarily performed at the same time, but may be performed at different times, and the order of performance of the sub-steps or stages is not necessarily sequential, but may be performed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.
Referring to fig. 3, a file monitoring process of the file monitoring system may include:
step S21: the command line parser receives and parses a user command from the terminal device, reads a specified ruleFile and checks its format. And when the ruleFile configuration format has no problem and the input parameter values of the user command are legal, transmitting the read ruleFile data to the scheduler.
Step S22: the scheduler starts instantiating the monitoring job in ruleFile, which can be scheduled according to a scheduling algorithm.
For example, a ruleFile configures two monitoring jobs, and the scheduler may instantiate two monitoring jobs from the ruleFile, named as job0 and job1, respectively, after initialization, and register event handlers for the two jobs. The scheduler may decide to schedule job0 according to a scheduling algorithm.
Step S23: when the job0 is invoked to execute, that is, the file monitor scans whether the directory of the specified file in the monitoring operation contains the monitoring event meeting the attention of the file, if yes, the monitoring event is sent out, and meanwhile, the state of the monitoring operation is reported to the scheduler; if not, reporting that the corresponding event is not monitored by the monitoring operation of the scheduler.
Taking the above example as a support, the file monitor scans the corresponding directory according to the file name rule of the monitoring job joba 0, and when a file meets its monitoring condition after scanning, a monitoring event is triggered; if not, directly reporting the corresponding event which is not monitored by the monitoring operation of the scheduler to the scheduler.
Step S24: and when the event processor receives the monitoring event, executing the event processor specified by the monitoring operation.
Taking the above example as a support, the event handler of job0 receives the corresponding monitoring event and executes the corresponding processing.
Step S25: when all monitoring jobs job0 have completed monitoring, scheduler 20 is responsible for cleaning jobs, exiting the file monitor, and so on.
In connection with the above example, if the monitoring method is in the standard mode, and if the monitoring job is not completed, the process returns to step S22, and the scheduler re-determines the scheduling; if the monitoring operation (success/failure) is completed, the monitoring operation is removed from the scheduling queue, and when the scheduling queue is empty, the cleaning operation is executed to finish the monitoring.
In the loop mode, if the monitoring job is completed, the monitoring job status and the monitoring index are reset, and the process returns to step S22, and the scheduler re-determines the scheduling until the end runtime configured in ruleFile ends the monitoring.
Therefore, the file monitoring method provided by the embodiment of the invention can be used only by declaring in the configuration file of the monitoring operation when the monitoring process is executed, and developers can also register the self-defined monitoring event processing method by writing the configuration file, so that the file monitoring method is convenient for development or business personnel to use, and can be quickly adapted to the requirement of high-speed change.
From the aspect of functions, compared with the situation that the monitoring of other files cannot be monitored, the file monitoring process can monitor the file which exists and is completely transmitted before the file monitoring process is started; the embodiment of the invention provides file integrity guarantee for the upstream and downstream components of the software system, only after the upstream component completely transmits the file, the file arrival event is triggered, and the monitoring event processor is executed according to the service requirement to inform the downstream component of processing the event, so that the data loss caused by the early processing of the incomplete file by the downstream component is avoided; the embodiment of the invention also provides an event recovery scheme after the monitoring process is closed accidentally, so that the robustness in the software system is ensured. From the aspect of development efficiency, the embodiment of the invention provides self-defined monitoring event processing, wherein the self-defined monitoring event processing is decoupled from a file monitoring program, and how to process the specific events monitored to specific files is defined in a configuration file mode.
Specifically, the embodiments of the present invention can effectively solve the aforementioned problems:
aiming at the problem 1), the embodiment of the invention continuously scans the directory where the monitored file in the ruleFile is located, is different from other files which are registered with the operating system and need to be monitored, and can not monitor the file because the file is not provided with the file creation event and the file modification event after the file is completely transmitted, so that the file can be monitored to exist and be completely transmitted.
Aiming at the problem 2), the file creation time, the file arrival event and the file deletion event under the scene that the file monitoring process is closed accidentally can also be identified and processed after the file monitoring process is recovered in the mode.
Aiming at the problem 3), a directory where the file is located is monitored by continuous scanning, and file metadata at the moment is recorded, wherein the file metadata comprises the size of the file and the last modification time of the file, when the file size is continuously monitored for multiple times and is not changed and the last modification time of the file is not changed, and the file size meets the minimum file size defined in ruleFile, the file is regarded as completely arrived, and a file arrival event is triggered.
For problem 4), the business logic is decoupled from the file monitor by defining event handlers in ruleFile. Meanwhile, a general event handler template can be defined through a configuration file of specific monitoring operation, and a file monitoring program can automatically read the configuration file when being started and enable the event handler templates to take effect, so that the general event handler templates can be used in a ruleFile, development of business requirements is simplified, business personnel can automatically realize the business requirements after simple training, and the cooperation efficiency is improved. For highly customized business requirements, furthermore, all parameters that may be used are defined as macros, and the invention native support invokes custom programs in event processing and passes these macros in, replacing the macros with specific parameter values before execution.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). The memory is an example of a computer-readable medium.
Computer-readable media, which include both non-transitory and non-transitory, removable and non-removable media, may implement the information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in the process, method, article, or apparatus that comprises the element.
The above are merely examples of the present application and are not intended to limit the present application. Various modifications and changes may occur to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.
Claims (10)
1. A document monitoring system, comprising:
a command line parser: the configuration file table is used for reading the configuration file of the monitoring job corresponding to the user command in response to the user command,
the parameters of the configuration file of the monitoring operation comprise a file name rule, a monitoring condition, a monitoring event, a monitoring mode and file metadata;
a scheduler: the configuration file is used for scheduling the execution sequence of the corresponding monitoring operation according to the configuration file;
a file monitor: the monitoring system is used for monitoring the monitoring operation which is called to be executed, and triggering the corresponding monitoring event when the monitoring operation meets the corresponding monitoring condition;
an event handler: and the monitoring device is used for executing a corresponding monitoring instruction when the monitoring event is received.
2. The file monitoring system of claim 1 wherein the filename rule comprises a regular expression of the monitoring job indicating a directory and corresponding filenames.
3. The document monitoring system of claim 1, wherein the monitoring event comprises: file creation events, file modification events, file deletion events, and file arrival events.
4. The document monitoring system according to claim 1, wherein the monitoring modes include a standard mode, a cyclic mode, and a template mode.
5. The document monitoring system of claim 4, wherein the scheduler is further configured to:
instantiating a configuration file for the monitoring job;
and registering the event processor according to the monitoring mode.
6. The document monitoring system of claim 1, wherein the command line parser is further configured to:
detecting whether the format of the user command is legal or not;
analyzing whether the format of the configuration file corresponding to the user command is legal or not;
and if one of the format of the user command and the format of the corresponding configuration file is illegal, not responding to the user command.
7. The document monitoring system of claim 1, wherein the scheduler is further configured to:
adding the corresponding monitoring events into a monitoring queue according to the execution sequence of the monitoring operation to prepare for being called and executed;
removing the monitoring event from the monitoring queue after the monitoring event in the monitoring operation is completed;
when the monitoring events which are not completed exist in the monitoring operation, scheduling the execution sequence of the monitoring events;
and when the monitoring operation is finished monitoring, removing the monitoring operation from the monitoring queue.
8. The document monitoring system according to claim 1, wherein the monitoring job that is invoked for execution, and when the monitoring job satisfies the corresponding monitoring condition, triggering the corresponding monitoring event includes:
when the monitoring operation is called up and executed, scanning all files under the corresponding file name rule indication directory according to a monitoring strategy;
filtering all files under the directory according to the file name rule and the monitoring condition;
and triggering the monitoring event when the monitoring file exists after filtering.
9. The document monitoring system of claim 8, wherein the document monitor is further configured to:
recording the metadata of the metadata file, wherein the metadata of the file is the operating system data of the monitoring file, and the method comprises the following steps: the file name, the path, the owner, the authority information, the file creating time, the last modifying time and the file size of the monitoring file.
10. The document monitoring system of claim 9, wherein the document monitor is further configured to:
when the monitoring file exists after filtering, reporting the state of the monitoring operation to the scheduler, and sending a monitoring event to an event processor;
reporting to the scheduler that there is no monitoring event when the monitoring file does not exist after filtering.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111522798.2A CN114238220A (en) | 2021-12-13 | 2021-12-13 | File monitoring system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111522798.2A CN114238220A (en) | 2021-12-13 | 2021-12-13 | File monitoring system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114238220A true CN114238220A (en) | 2022-03-25 |
Family
ID=80755492
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111522798.2A Pending CN114238220A (en) | 2021-12-13 | 2021-12-13 | File monitoring system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114238220A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115034770A (en) * | 2022-08-12 | 2022-09-09 | 平安银行股份有限公司 | Dispute clearing file scheduling method and device |
-
2021
- 2021-12-13 CN CN202111522798.2A patent/CN114238220A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115034770A (en) * | 2022-08-12 | 2022-09-09 | 平安银行股份有限公司 | Dispute clearing file scheduling method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112346829B (en) | Method and equipment for task scheduling | |
| CN104166590A (en) | Task scheduling method and system | |
| CN106201672B (en) | Timed task setting system and timed task running method thereof | |
| CN110427258B (en) | Resource scheduling control method and device based on cloud platform | |
| US20070124547A1 (en) | Method, apparatus and computer program product for managing access to storage | |
| EP2113859A1 (en) | Computer, operation rule application method, and operating system | |
| CN113326107A (en) | Kubernetes cluster-based periodic task scheduling method and electronic equipment | |
| CN114238220A (en) | File monitoring system | |
| CN116089040A (en) | Service flow scheduling method and device, electronic equipment and storage medium | |
| CN110764962B (en) | Log processing method and device | |
| CN113342554B (en) | IO multiplexing method, medium, device and operating system | |
| CN116302457A (en) | Cloud primary workflow engine implementation method, system, medium and electronic equipment | |
| CN110968406B (en) | Method, device, storage medium and processor for processing task | |
| US7127446B1 (en) | File system based task queue management | |
| CN115499493A (en) | Asynchronous transaction processing method and device, storage medium and computer equipment | |
| CN113010278B (en) | Batch processing method and system for financial insurance core system | |
| CN115617459A (en) | Method, device and equipment for resource scheduling | |
| CN112540886B (en) | CPU load value detection method and device | |
| EP4191444A1 (en) | Image configuration method, device and system, and storage medium | |
| CN114816662A (en) | Container arrangement method and system applied to Kubernetes | |
| CN113886342A (en) | File format conversion method and device, storage medium and processor | |
| JP2007058265A (en) | Log output controller and log output control program | |
| CN112286876A (en) | Log file capturing method and device and computer readable storage medium | |
| JP5465401B2 (en) | File management method, apparatus and program | |
| CN116302361A (en) | Mirror image construction method and device, processor and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |