KR101736382B1 - Ems server and log data management method thereof - Google Patents

Abstract

The present invention relates to an MS server and a log data management method for enabling to store, monitor, and inquire log data in hundreds of thousands or more per second in real time.
An EM server according to the present invention manages receiving and storing packet data from external equipment and inquiring stored information according to a request of a client, the EM server comprising: a report database in which bulk insertion of the packet data is performed; And a real-time memory table in which data extracted by parsing the packet data is inserted and updated in a table structure and storage of indexing files corresponding to the top N items arranged in order of usage amount is performed in real time .

Description

[0001] DESCRIPTION [0002] EMS SERVER AND LOG DATA MANAGEMENT METHOD THEREOF [

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an MMS, and more particularly, to an MMS server and its log data management method capable of storing, monitoring, and inquiring log data in hundreds of thousands or more per second in real time.

In the public communication network operating a large scale integrated information communication network, convergence and automation have emerged as essential constituent elements of the communication network. Accordingly, it has become necessary to perform management and control functions for the elements constituting each communication network. ITU-T proposed an element management system (EMS) according to the necessity. That is, the element management system (EMS) performs control and network monitoring functions for each configuration of the communication network using the SNMP protocol while the EMS server is connected to the communication network by the TCP protocol.

1 is a system configuration diagram showing a conventional MS configuration. 1, the conventional MS comprises an external device 5, an MS server 1 and clients 6a and 6b. Particularly, the MS server 1 includes a data collector 2, a central database (3), and a Web server (4).

The conventional log data management method of the present invention is as follows. The data collector 2 collects packet data transmitted from the external equipment 5 and stores it in the central database 3, and the clients 6a and 6b When requesting the log information through the Web server 4, the MS server 1 inquires the central database 3 and outputs the result of the processing. The client may access the web server 4 through the browser 6a to inquire data or access the central database 3 through the C / S app 6b to request data.

The conventional MS server 1 collects the log data received from the external equipment 5 for a certain period of time, collectively stores the log data in the central database 3, and real-time inquiry is performed while storing the mass data It depends entirely on the performance of the server.

That is, since a single centralized database engine is used, there is a limit in that when a large amount of log data of hundreds of thousands or more is received, it can not be stored in seconds or can not be monitored or inquired in real time.

Accordingly, the conventional ES server only provides monitoring functions aggregated in minutes and analysis-oriented functions of aggregated static data.

SUMMARY OF THE INVENTION The present invention has been made to solve the above-mentioned problems, and it is an object of the present invention to provide an MS server capable of real- And a method for managing log data thereof.

According to an aspect of the present invention, there is provided an MS server for receiving and storing packet data from external equipment and managing the stored packet data according to a request from a client, the MS server comprising: ; And a real-time memory table in which data extracted by parsing the packet data is inserted and updated in a table structure and storage of indexing files corresponding to the top N items arranged in order of usage amount is performed in real time .

A method for managing log data of an MS server according to the present invention includes: receiving packet data from an external device; Dividing the received packet data by a plurality of applications that are distinguished from each other and storing the packet data; And receiving a request from the client.

The step of dividing the received packet data into a plurality of applications to be distinguished and stored may include a step of storing the packet data as a first use, and the step of storing the packet data includes parsing the received packet data A first step; A second step of inserting data extracted by the parsing into a table structure in a real-time memory table; A third step of selecting the top N items by sorting the items in order of usage; And a fourth step of generating an indexing file corresponding to the N items and storing the indexing file in the real-time memory table.

According to the present invention, it is possible to store and monitor log data of hundreds of thousands or more per local IP in real time, to retrieve historical data of a few months ago in seconds, And stability can be secured.

1 is a system configuration diagram showing a conventional MS configuration;
2 is a block diagram showing an MS structure including an MS server according to the present invention;
3 is an example showing data update of a real-time memory table according to the present invention.
4 is an example of data expressed in the json format of the indexing file generation process of the present invention.
5 is an example of an indexing file generated in the json format of the present invention.
6 is a block flow diagram of a method for managing log data of an MS server according to the present invention.
7 is a block flow diagram of a method of managing log data for a real-time memory table of the present invention.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, the terms "comprises" or "having" and the like refer to the presence of stated features, integers, steps, operations, elements, components, or combinations thereof, But do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or combinations thereof.

Also, in the present specification, the term " above or above "means to be located above or below the object portion, and does not necessarily mean that the object is located on the upper side with respect to the gravitational direction. It will also be understood that when a section of an area, plate, or the like is referred to as being "above or above another section ", this applies not only to the case where the other section is " And the like.

Also, in this specification, when an element is referred to as being "connected" or "connected" with another element, the element may be directly connected or directly connected to the other element, It should be understood that, unless an opposite description is present, it may be connected or connected via another element in the middle.

Also, in this specification, the terms first, second, etc. may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another.

In the following, preferred embodiments, advantages and features of the present invention will be described in detail with reference to the accompanying drawings.

2 is a block diagram illustrating an MS structure including an MS server according to the present invention. 2, the MMS server 100 according to the present invention receives and stores packet data from the external device 200 and manages the received packet data in order to inquire stored information of the client 300, A collector 10, an emulation control unit (not shown), data storage units 20, 30, 40, and 50, and a web server 60.

In particular, the MS server 100 of the present invention is characterized in that received packet data is divided into various forms or applications, and the processed data is stored in a data storage unit.

Accordingly, the data storage unit of the present invention can be divided into a plurality of data storage units. According to a preferred embodiment of the present invention, a real-time memory table 20 for storing data processed for real-time monitoring, The analysis database 40 for storing analysis data, and the NoSQL database 50 for storing various log data.

Here, the real-time memory table 20, the report database 30, the analysis database 40, and the NoSQL database 50 of the present invention are functionally separated and separated, Of course.

The MS server 100 is connected to each network element constituting the network through a communication network and receives and manages log information according to a QoS protocol, a DPI protocol, a TCP / IP protocol, or an SNMP protocol, And analysis functions.

Hereinafter, each configuration of the MMS server 100 will be described in detail.

The data collector 10 of the present invention collects packet data, that is, log information, transmitted from the external equipment 200, and packet data collected by the data collector 10 is processed And stored in any one of the configurations selected from the report database 30, the real-time memory table 20, the analysis database 40, and the NoSQL database 50 according to the processing form.

Here, the data packet structure transmitted to the MS server 100 includes a message header information, a monitoring header information, a link information, a local IP address, Information per application (APP), monitoring footer information, message (APP), information per application group (APPGRP), information per ether type (ETHE) Message footer information, and the like.

The data collector 10 may be configured to notify the client 300 in real time when an event occurs.

Here, an event may be a general event related to a network, a VOIP event, an event depending on a setting change, and the like.

In this case, if the event data is included in the packet data collected by the data collector 10, the collector collector 10 is configured to push an event occurrence notification or message to the client 300 through the messaging server in real time .

The real-time memory table 20 of the present invention stores data processed for real-time monitoring in a data table structure. The real-time memory table 20 stores the data of the indexed file generated by the parsing and sorting of the packet data received by the data collector 10 And the storage is performed in real time.

Specifically, the MS server 100 sequentially performs decompression, packet parsing, data update, sorting, selection of an upper item, and data extraction on the packet data received from the external device 200, And the indexing file thus generated is configured to be stored in the real-time memory table 20.

Here, decompression means decompressing the received binary data.

Packet parsing means parsing the original binary data to extract the data.

Updating the data means updating the data parsed in the information already stored in the real-time memory table 20. [ Therefore, the data extracted by the parsing is inserted and updated in the data table structure.

3 is an example showing data update of the real-time memory table according to the present invention. Referring to FIG. 3, the data updating process is configured to add, update, and delete new data while comparing the data with stored data in the real-time memory table 20. According to one embodiment, the MS server 100 may be configured to manage only 120 recently received data through the data updating process as described above.

Sorting means sorting in order of usage when data update is completed.

Selecting an upper item means selecting the top N items among the sorted items. According to a preferred embodiment, the top N items are made up of the top 100 items with high usage.

The data extraction means extracting detailed data of the top N items determined through the sorting and the upper item selection process.

The indexing file creation process is a process of creating an indexing file having a format suitable for real-time monitoring of data extracted through the data extraction process.

According to a preferred embodiment, the indexing file generated through the above-described indexing file generation process can be formed as a file of the 'json' format. 4 is an example of data expressed in the json format of the indexing file generation process of the present invention, and FIG. 5 is an example of the indexing file generated in the json format of the present invention.

Such indexing files in the 'json' format can be used as data by deserializing them from JavaScript to objects.

For example, when data is requested to the ES server 100 from AJAX in a web page, the ES server 100 reads the generated indexing file and transmits it to the corresponding web page, To a chart or a grid.

On the other hand, the decompression, packet parsing, data update, sorting, upper item selection, and data extraction processes described above can be performed by the MS control unit.

The report database 30 of the present invention is a configuration for storing processed data for reporting purposes and is characterized in that bulk insertion of packet data received by the data collector 10 is performed.

Specifically, the MS server 100 stores the packet data received from the external device 200 as a file, and converts the packet data into a file having a structure capable of bulk insertion in the report database 30.

A backup file for bulk insertion is created in advance, and the scheduler is configured to perform bulk insertion into the report database 30 in Sec (Sec) units. If there is no corresponding data in the report database 30 during the bulk insertion, the bulk file is restored through the backup path.

The analysis database 40 of the present invention is a configuration for storing analysis data. After the log data stored in the report database 30 is aggregated by the scheduler for a certain period of time, the analysis data calculated accordingly is periodically .

Therefore, when the clinician requests for the analysis of the log data, the web server 60 can retrieve and output the analysis date in units of week / week / month / year through the above-described analysis database 40 .

The NoSQL database 50 of the present invention stores and manages data by the NoSQL method. The NoSQL database 50 stores events of the packet data received from the external device 200 or system log data.

The web server 60 of the present invention inquires the indexing file of the real-time memory table 20 at the request of the client 300 and responds the processing result to the client 300, And is configured to perform monitoring on the data.

Particularly, since the data extracted by parsing and the indexing file generated based on the extracted data are inserted and updated in the real-time memory table 20, it is possible to implement log information monitoring in units of seconds.

Specifically, more than 100,000 log data per second can be monitored in real time.

The monitoring information that is inquired through the real-time memory table 20 includes the link, the local host, the application group, the local IP, the application, and the application type information.

When the client 300 requests the web server 60, the web server 60 inquires the reporting data through the report database 30 and responds the processing result to the client 300.

In particular, the report database 30 is configured to preliminarily create a backup file for bulk insertion, and then perform a bulk insertion into the report database 30 in units of Sec (sec) by a scheduler. In this case, You will be able to implement a reporting query.

The reporting information, which is inquired through the report database 30, includes classified information such as remote host, protocol, service port, Ether type, port, node group, VOIP, URL filtering,

When the client 300 requests the web server 60, the web server 60 inquires the analysis data through the analysis database 40 and responds the processing result to the client 300.

The data of the analysis database 40 is stored by accumulating the log data stored in the report database 30 by the scheduler for a certain period of time and the web server 60 stores the daily / It is possible to retrieve the analysis data in units of years and output it to the client 300.

As described above, the analysis information to be searched through the analysis database 40 includes a local host, an application, an application group, a protocol, a service, a URL filtering, a policy, a VOIP, a remote host, Can be set and inquired.

The client 300 that requests the web server 60 to inquire log information and output the processing result includes an electronic device that can be connected to the web server 60 through a wired / wireless communication network such as a computer, a smart phone, a PDA And these electric / electronic devices are equipped with a program such as a browser.

The program installed in the client 300 attempts to connect to the MS server 100 running on the network and the MS server 100 performs a task corresponding to the request of the client 300 through the predetermined interface And provides the processing result to the client 300.

Meanwhile, the client 300 may attempt to request log information to the web server 60 using any one of various known data transmission methods such as a RESTful method, an XML method, and a SOAP method.

The emulation control unit of the present invention processes the collected packet data in various forms according to the purpose and outputs the resultant data to the report database 30, the real-time memory table 20, the analysis database 40, And the NoSQL database 50 as shown in FIG.

For example, the MMS controller can perform processing for decompression, packet parsing, data update, sorting, parent selection, data extraction, and indexing file generation for storing information in the real-time memory table 20.

In order to store information in the report database 30, the MMS controller can also perform processing for storing the received packet data as a file as it is, and converting the packet data into a file having a structure capable of bulk insertion in the report database 30 .

Furthermore, the multimedia controller may include a scheduler for performing the above-described second-order scheduling, periodic aggregate scheduling.

Hereinafter, the log data management method of the MS server 100 described and illustrated above will be described.

6 is a flowchart of a log data management method of an MS server according to the present invention. Referring to FIG. 6, the method for managing log data of an MS server according to the present invention includes a step of receiving a packet data (S10), a storing step for each use (S20), a client request receiving step (S30) (S40), and a processing result output step (S50).

The packet data receiving step S10 is a step in which the data collector 10 of the MS server 100 collects packet data transmitted from the external equipment 200, that is, log information.

The usage classification storage step (S20) is a step of storing received packet data by dividing the packet data into a plurality of applications that are distinguished from each other.

According to a preferred embodiment, the plurality of uses include a first use for real-time monitoring in seconds, a second use for reporting inquiry in seconds, a third use for analysis inquiry, and a second use for event / 4 applications.

The client request receiving step S30 is a step of receiving a request for retrieving or searching log information from a client program connected to the web server 60 through a RESTful method, an XML method, a SOAP method, or the like.

The optional inquiry step S40 is a step of inquiring at least one data storage unit selected from the real-time memory table 20, the report database 30, the analysis database 40 and the NoSQL database 50 at the request of the client 300 to be.

The processing result output step S50 is a step of providing the client 300 with a result processed through the selective inquiry step.

Hereinafter, the storing step as the first use of the present invention will be described in detail.

7 is a block flow chart illustrating a method of managing log data for a real-time memory table according to the present invention. Referring to FIG. 7, the step of storing as the first use of the present invention includes a decompressing step S22 of receiving the packet S21, a packet parsing step S23, a data updating step S24, (S25), an upper item selection step (S26), a data extraction step (S27), and an indexing file generation step (S28).

The decompressing step S22 is a step of decompressing the received binary data, and the packet parsing step S23 is a step of extracting data by parsing the original binary data.

The data update (S24) is a step of updating the parsed data in the information stored in the real-time memory table 20. [ The data extracted by the parsing is inserted and updated in the data table structure.

The sorting step S25 is a step of sorting the data items in order of usage when the data updating is completed, and the upper item selecting step S26 is a step of selecting the upper N items among the sorted items. According to a preferred embodiment, the top N items are made up of the top 100 items with high usage.

The data extracting step S27 is a step of extracting detailed data of the upper N items determined through the sorting and the upper item selecting process.

The indexing file creation step S28 is a step of creating an indexing file having a format suitable for real-time monitoring of the extracted data through the data extraction process. According to a preferred embodiment, the indexing file generated through the indexing file creation step S28 may be formed of a file of the 'json' format.

In this case, upon receipt of the client 300 request for real-time monitoring, the selective inquiry step S40 is configured to inquire the indexing file stored in the real-time memory table 20 and respond to the client 300. [

Storing the received packet data as a file and converting the received packet data into a file having a structure capable of bulk insertion into the report database 30, And inserting and storing in the report database 30.

Here, the backup file for bulk insertion is prepared in advance, and the scheduler is configured to perform bulk insertion into the report database 30 in Sec (Sec) units. If the corresponding data does not exist in the report database 30 at the time of bulk insertion, the bulk file is restored from the backup path.

In this case, upon receipt of a client request for reporting inquiry in seconds, a selective inquiry step (S40) is configured to search the report database (30) and provide the processing result to the client (300).

The step of storing as the third use of the present invention includes a step of periodically accumulating the analytical data calculated after the log data stored in the report database 30 is collected by the scheduler for a certain period of time .

In this case, when receiving a client request for an analysis inquiry, the selective inquiry step (S40) searches the analytic database (40) for analytic data in units of day / week / month / year and outputs the processing result.

Finally, storing as a fourth use of the present invention includes storing event or system log data of the packet data in the NoSQL database 50.

As described above, the MMS server of the present invention is configured to process, store, and manage received packet data by dividing the received packet data into a plurality of applications that are separated from each other, thereby storing and monitoring log data of hundreds of thousands or more per second in real time , It is possible to retrieve historical data of a few months ago in seconds and to guarantee the performance and stability of the server.

While the preferred embodiments of the present invention have been described and illustrated above using specific terms, such terms are used only for the purpose of clarifying the invention, and it is to be understood that the embodiment It will be obvious that various changes and modifications can be made without departing from the spirit and scope of the invention.

Such modified embodiments should not be understood individually from the spirit and scope of the present invention, but should be regarded as being within the scope of the claims of the present invention.

10: Data Collector
20: Real-time memory table
30: Report database
40: Analysis database
50: NoSQL database
60: Web server
100: Ems server
200: External equipment
300: Client

Claims (11)

A method for managing log data of an MS server,
Receiving packet data from external equipment; Dividing the received packet data by a plurality of applications that are distinguished from each other and storing the packet data; And receiving a request from a client,
The step of dividing the received packet data into a plurality of applications to be distinguished from each other and storing the packet data includes storing the packet data as a first use and storing the packet data as a second usage,
The method of claim 1,
A first step of parsing the received packet data;
A second step of inserting the data extracted by the parsing into a table structure in a real-time memory table;
A third step of selecting the top N items by sorting the items in order of usage; And
And a fourth step of generating an index file corresponding to the upper N items and storing the generated index file in the real-time memory table,
The method of claim 1,
Converting the received packet data into a file of a structure capable of bulk insertion (hereinafter referred to as a " bulk insertion file "); And
And inserting the bulk insert file into the report database.
The method according to claim 1,
Further comprising the step of inquiring and responding to the indexing file stored in the real-time memory table upon receiving a request from the client.
delete The method according to claim 1,
Storing the received packet data as a third use in a step of dividing the received packet data according to a plurality of uses,
Wherein the step of storing as the third purpose comprises the step of storing analysis data aggregating data stored in the report database.
An MS server for receiving and storing packet data from external equipment and managing the storage information according to a request from a client,
A report database in which bulk insertion of the packet data is performed; And
And a real-time memory table in which data extracted by parsing the packet data is inserted and updated in a table structure in a real-time memory table, and storage of indexing files corresponding to the top N items sorted in order of usage amount is performed in real time Wherein the server comprises:
6. The method of claim 5,
Further comprising a web server receiving a request from the client and inquiring an indexing file corresponding to the top N items of the real-time memory table upon request of the client.
6. The method of claim 5,
Wherein the indexing file is generated in a json format.
6. The method of claim 5,
Wherein the report database is configured to perform bulk insertion of the packet data in sec units by a scheduler.
6. The method of claim 5,
Further comprising an analysis database for storing analysis data obtained by aggregating data stored in the report database for a predetermined period of time.
6. The method of claim 5,
Further comprising a NoSQL database for storing events or system log data of the packet data.
9. The method of claim 8,
Further comprising a web server for receiving a request from the client and for inquiring data of the second unit of the report database upon request of the client.

Images