CN107943942B

CN107943942B - Event-driven multi-turn file acquisition system

Info

Publication number: CN107943942B
Application number: CN201711185042.7A
Authority: CN
Inventors: 耐尔; 屈朝晖
Original assignee: Suzhou Yosemei Intelligent System Co ltd
Current assignee: Beijing Advanced Digital Technology Co ltd
Priority date: 2017-11-23
Filing date: 2017-11-23
Publication date: 2020-08-28
Anticipated expiration: 2037-11-23
Also published as: CN107943942A

Abstract

The invention relates to a multi-turn file acquisition system based on event driving, which comprises a detector, a monitor, a scheduler and a thread pool which are connected in sequence; the detector is used for detecting the newly generated rotation rule file, identifying the file and then sending a registration path to the listener; the monitor is used for monitoring the registration path of the rotation rule file in real time and reacting to the scheduler after receiving the registration path; the scheduler matches the registration path sent by the listener with the rotation rule file identified by the detector, and after the matching is successful, the rotation rule file identified by the registration path is called and distributed to the thread pool; the thread pool reads the rotation rule file distributed by the scheduler, the event-driven multi-rotation file acquisition system can realize real-time acquisition, and the acquisition task is initiated based on the event, so that the consumed memory is low, and the efficiency is high.

Description

Event-driven multi-turn file acquisition system

Technical Field

The invention relates to a multi-turn file acquisition system based on event driving, and belongs to the technical field of data acquisition.

Background

The method comprises the steps that log files are generated by the middleware and the service system at every moment, new files are automatically generated when the files reach a certain size or reach a set time, the log file with the earliest time is deleted when the number of the files reaches a set value, and the process is called log rotation. First, these log files usually have their own log rotation rules, and different systems have different log rotation rules. Secondly, the logs can grow in real time along with the operation of the system, and a mode for reading the growing logs in real time is needed. These complex situations and requirements all add difficulty to the collection of text data.

At present, operation and maintenance data of many systems adopt a mode of backing up generated logs to a server at a fixed time, and then starting a collector to read the fixed texts and insert the fixed texts into a distributed database (such as an HDFS). Meanwhile, the real-time flowing data volume is huge, and it is time-consuming to find out whether a new file is generated in a certain rotation file by checking each rotation file, so that the workload is huge.

Disclosure of Invention

The technical problem to be solved by the invention is as follows: in order to overcome the problems, the event-driven multi-turn file acquisition system is provided.

The technical scheme adopted by the invention for solving the technical problems is as follows:

a multi-turn file acquisition system based on event driving comprises a detector, a monitor, a scheduler and a thread pool which are sequentially connected;

the detector is used for detecting the newly generated rotation rule file, identifying the file and then sending a registration path to the listener;

the monitor is used for monitoring the registration path of the rotation rule file in real time and reacting to the scheduler after receiving the registration path;

the scheduler matches the registration path sent by the listener with the rotation rule file identified by the detector, and after the matching is successful, the rotation rule file identified by the registration path is called and distributed to the thread pool;

and the thread pool reads the rotation rule file distributed by the scheduler.

Preferably, the listener also generates an event containing the registered path of the rotation rule file and sends the event to the scheduler.

Preferably, after traversing the rotation rule file, the detector identifies a new rotation rule file, where the new rotation rule file includes at least two of the following:

the method comprises the steps that a rotation rule file is newly generated on the basis of an existing rotation rule file, a detector firstly identifies the rotation rule file and marks the rotation rule file, a scheduler matches the rotation rule file, and the rotation rule file is distributed to a thread pool to be read after matching is successful;

the newly generated rotation rule file is an independent rotation rule file, the detector identifies and marks the rotation rule file, the scheduler matches the rotation rule file, and the rotation rule file is classified into the whole rotation rule file after matching is completed and is distributed to a corresponding thread pool for reading.

The invention has the beneficial effects that: the event-driven multi-turn file acquisition system can realize real-time acquisition, the acquisition task is initiated based on the event, the consumed memory is low, the efficiency is high, and the system can simultaneously acquire a large number of turn rule files without overhigh load.

Drawings

The invention is further illustrated with reference to the following figures and examples.

FIG. 1 is a block diagram of the architecture of one embodiment of the present invention.

Detailed Description

The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic views illustrating only the basic structure of the present invention in a schematic manner, and thus show only the constitution related to the present invention.

Example 1

The event-driven multi-turn file acquisition system comprises a detector, a monitor, a scheduler and a thread pool which are sequentially connected, and the event-driven multi-turn file acquisition system not only comprises the modules, but also can be configured with other modules if needed;

the detector is used for detecting a newly generated rotation rule file, identifying the rotation rule file and sending out a registration path of the rotation rule file;

the monitor is used for monitoring the registration path of the rotating file in real time and reacting the registration path to the scheduler;

the scheduler matches the file path detected by the listener with the file identified by the detector, and after the matching is successful, the round robin rule file is called and distributed to the thread pool;

and the thread pool reads the rotation rule file distributed by the scheduler.

The event-driven multi-turn file acquisition system can realize real-time acquisition by adopting the detector, the acquisition task is initiated in the monitor based on the event, the consumed memory is low, the efficiency is high, and the task is more efficient by being distributed to a specific thread pool through the scheduler.

In a preferred embodiment, the listener generates an event containing a round robin file path and sends the event to the scheduler.

In a preferred embodiment, after the detector traverses the rotation rule file, that is, the detector accesses each rotation rule file once, the structure is not limited to the tree structure, and a new file is identified. The identified files are two in number;

the first is a file newly generated on the basis of the existing rotating file, firstly, a detector identifies the file and identifies the file, a scheduler matches the file, and the file is distributed to a thread pool to be read after the matching is successful;

and the second method is that the newly generated file is an independent rotating file, the detector identifies and marks the file, the scheduler matches the file, and the file is classified into the whole rotating file after matching is finished and is distributed to a corresponding thread pool for reading.

In a preferred embodiment, the event type is a preferred event type in the art, but is not limited to the two types, and other types of events can be selected according to needs.

Example 2

The embodiment provides a specific operation implementation mode on the basis of the embodiment 1, for example, a page configuration reads a round robin rule file/var/log/maillog, the scheduler only needs to encapsulate the path change into a read unit and register a directory part (/ var/log) of the path change to the listener, the listener informs the scheduler of the path (/ var/log/maillog) of the file and the event type after detecting the file change, the scheduler matches the path to find the read unit and encapsulates the read unit into a Task to be sent to a thread pool, and the thread pool can use one thread to perform final file reading.

As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

In light of the foregoing description of the preferred embodiment of the present invention, many modifications and variations will be apparent to those skilled in the art without departing from the spirit and scope of the invention. The technical scope of the present invention is not limited to the content of the specification, and must be determined according to the scope of the claims.

Claims

1. A multi-turn file acquisition system based on event driving is characterized by comprising a detector, a monitor, a scheduler and a thread pool which are sequentially connected;

the thread pool reads a rotation rule file distributed by the scheduler;

after traversing the rotation rule file, the detector identifies a new rotation rule file, wherein the new rotation rule file at least comprises the following two types:

2. The event-driven-based multi-turn file collection system of claim 1, wherein the listener further generates an event containing a registered path of the turn rule file and sends the event to the scheduler.