JP6777582B2 - Management device, management method and management program - Google Patents

Description

The present invention relates to a management device, a management method and a management program.

Conventionally, there has been known a technique for improving processing efficiency by performing distributed processing on a huge data set (see Non-Patent Document 1). Generally, in distributed processing, a plurality of different processes are executed in parallel. In addition, the content of each process is divided into a plurality of processes, and the result of each process is used to execute the subsequent process.

Jeffrey Dean, Sanjay Ghemawat, “MapReduce: Simplified Data Processing on Large Clusters”, OSDI 2004, October 2004

However, in the conventional technique, the processing efficiency may decrease when the processing partially overlaps between a plurality of different types of processing performed on the same data set.

The present invention has been made in view of the above, and an object of the present invention is to improve the efficiency of a plurality of types of processing for the same data set.

In order to solve the above-mentioned problems and achieve the object, the management device according to the present invention acquires data and delivers the acquired data to a preprocessing unit that calculates a predetermined feature amount of the data, and the above-mentioned The pretreatment management unit that receives the feature amount calculated from the preprocessing unit and stores the received feature amount in the storage unit, and the main processing unit that performs a predetermined process using the feature amount are subjected to the processing. It is characterized by including a main processing management unit that extracts and delivers the corresponding feature amount from the storage unit.

According to the present invention, it is possible to improve the efficiency of a plurality of types of processing for the same data set.

FIG. 1 is an explanatory diagram for explaining an information processing system elephant to be processed by the management device according to the embodiment of the present invention. FIG. 2 is a schematic diagram illustrating a schematic configuration of the management device of the present embodiment. FIG. 3 is an explanatory diagram for explaining the processing of the processing management unit. FIG. 4 is a flowchart showing the management processing procedure of the present embodiment. FIG. 5 is a diagram illustrating a computer that executes a management program.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. The present invention is not limited to this embodiment. Further, in the description of the drawings, the same parts are indicated by the same reference numerals.

[Information processing system]
First, the information processing system to be processed by the management device according to the present embodiment will be described with reference to FIG. The information processing system 1 performs machine learning and artificial intelligence processing using data that changes in time series such as network information, updates statistics / learning models, and detects security threats / abnormalities.

In the present embodiment, the management device 10 receives data 20 that changes in time series such as network information and delivers it to a predetermined preprocessing unit 30. The preprocessing unit 30 is realized by a general-purpose computer or the like, and executes preprocessing for calculating a predetermined feature amount of the received data 20 for a predetermined time. For example, each of the plurality of preprocessing units 30 has data 20 such as the average number of clients per 5 minutes, the average number of flows per 5 minutes for each client, the average number of packets per flow, or the average number of bytes per packet. Calculate the feature amount.

The management device 10 receives the calculated preprocessing result and stores it in the storage unit. Further, the management device 10 extracts a predetermined preprocessing result from the storage unit and delivers it to the main processing unit 40. The plurality of main processing units 40 are realized by a general-purpose computer or the like, and each of them uses the preprocessing results to perform main processing such as updating a learning model and applying a detection algorithm, for example, batch processing such as malware infection detection and scan detection. To execute.

[Management device configuration]
Next, a schematic configuration of the management device according to the present embodiment will be described with reference to FIG. As shown in FIG. 2, the management device 10 according to the present embodiment is realized by a general-purpose computer such as a workstation or a personal computer, and has an input unit 11, an output unit 12, a communication control unit 13, a storage unit 14, and a control unit. It is provided with 15.

The input unit 11 is realized by using an input device such as a keyboard or a mouse, and inputs various instruction information to the control unit 15 in response to an input operation by the operator. The output unit 12 is realized by a display device such as a liquid crystal display, a printing device such as a printer, an information communication device, and the like, and outputs the result of management processing described later to the operator.

The communication control unit 13 is realized by a NIC (Network Interface Card) or the like, and is an external device and a control unit such as a preprocessing unit 30 or a main processing unit 40 via a telecommunication line such as a LAN (Local Area Network) or the Internet. Controls communication with 15.

The storage unit 14 is realized by a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory (Flash Memory), or a storage device such as a hard disk or an optical disk. The storage unit 14 stores in advance a processing program for operating the management device 10, data used during execution of the processing program, and the like, or temporarily stores each time the processing is performed. The storage unit 14 may be configured to communicate with the control unit 15 via the communication control unit 13.

In the present embodiment, the storage unit 14 functions as a database having so-called CRUD functions of Create (generate), Read (read), Update (update), and Delete (delete), and the preprocessing result 14a is performed by the management process described later. Is saved.

The preprocessing result 14a includes identification information, a time stamp, and a preprocessing result. The identification information means information such as a preprocessing name that identifies the content of the preprocessing performed by the preprocessing unit 30 or the preprocessing unit 30. The time stamp represents the date and time when the preprocessing was executed. The preprocessing result means a feature amount calculated by the preprocessing unit 30 using the input data 20.

In this way, in the pretreatment result 14a, the pretreatment results of the plurality of pretreatment units 30 are identified and stored. In addition to adding, updating, and deleting the preprocessing result 14a, it is possible to add a new preprocessing result of the preprocessing unit 30.

The control unit 15 serves as the pre-processing management unit 15a and the main processing management unit 15b as illustrated in FIG. 2 by executing a processing program stored in the memory by an arithmetic processing unit such as a CPU (Central Processing Unit). Function. Note that these functional units may be implemented in different hardware. Further, these functional units may be mounted on the same hardware as a part or all of the preprocessing unit 30 and the main processing unit 40.

The pre-processing management unit 15a acquires the data 20, delivers the acquired data 20 to the pre-processing unit 30 for calculating a predetermined feature amount of the data, and receives the feature amount calculated from the pre-processing unit 30. Then, the received feature amount is stored in the storage unit 14.

Specifically, the pre-processing management unit 15a is a time-series data 20 that changes in time series at predetermined time intervals such as every 5 minutes from a server or the like that manages network information via an input unit 11 or a communication control unit 13. Is acquired and delivered to a predetermined preprocessing unit 30. The preprocessing unit 30 receives the time series data 20 for a predetermined time, and receives, for example, the average number of clients per 5 minutes, the average number of flows per 5 minutes for each client, the average number of packets per flow, or the average number of packets per packet. The feature amount of the data 20 such as the average number of bytes is calculated.

Further, the pre-processing management unit 15a receives the calculated feature amount from the pre-processing unit 30, associates the identification information with the time stamp, and stores the calculated feature amount in the pre-processing result 14a of the storage unit 14. The preprocessing management unit 15a can generate the preprocessing result 14a, and can update or delete the preprocessing result to add a new preprocessing result. Further, the pre-processing management unit 15a adds identification information for identifying the pre-processing unit 30 to the feature amount of the pre-processing result in response to the addition, update or deletion of the pre-processing unit 30, and the pre-processing result of the storage unit 14. It is possible to save or delete in 14a.

The processing management unit 15b extracts the feature amount corresponding to the processing from the storage unit 14 and delivers it to the processing unit 40, which performs a predetermined processing using the feature amount. Specifically, the processing of the processing management unit 15b will be described with reference to FIG. The processing management unit 15b extracts a predetermined feature amount from the preprocessing result 14a of the storage unit 14 based on a predetermined condition at a predetermined time interval or in response to a request from the processing unit 40. And hand it over to the main processing unit 40.

For example, in FIG. 3, it is set as a condition that the preprocessing result 1 and the preprocessing result 2 are delivered to the main processing unit 40 that performs scan detection, for example, at predetermined time intervals such as every 3 hours. The case is illustrated. In this case, the processing management unit 15b determines the preprocessing result 1, that is, the average number of clients per 5 minutes, and the preprocessing result 2, that is, the average number of flows per 5 minutes for each client, from the preprocessing result 14a every 3 hours. And hand it over to the main processing unit 40 that performs scan detection.

The processing management unit 15b extracts the feature amount required for the processing from the preprocessing result 14a and delivers it to the processing unit 40 according to a predetermined condition predetermined internally to request the processing.

Alternatively, the processing management unit 15b may extract the requested feature amount from the preprocessing result 14a and deliver it to the processing unit 40 in response to the request from the processing unit 40. In this case, even if the delivery conditions are not set in advance, it is possible to deliver an appropriate preprocessing result to the main processing unit 40.

In addition, the processing management unit 15b accepts the designation of all the preprocessing results stored in the preprocessing result 14a or any of the preprocessing results within the specified range of the time stamp such as up to 3 hours before. It can be the preprocessing result of the delivery target.

Further, the processing management unit 15b may delete the preprocessing result that does not need to be delivered to the processing unit 40 from the preprocessing result 14a. This makes it possible to improve the usage efficiency of the storage unit 14.

The processing unit 40 uses the delivered preprocessing result, that is, the feature amount of the data 20, to perform the main processing such as updating the learning model and applying detection algorithms such as scan detection and malware infection detection.

[Management process]
Next, the management process by the management device 10 will be described with reference to FIG. FIG. 4 is a flowchart illustrating a management processing procedure. The flowchart of FIG. 4 is started, for example, at the timing when there is an operation input instructing the start of processing.

First, the preprocessing management unit 15a delivers the data 20 for a predetermined time to the predetermined preprocessing unit 30 at a predetermined time interval (step S1). Next, the preprocessing management unit 15a receives the feature amount which is the preprocessing result from the preprocessing unit 30 and stores it in the preprocessing result 14a of the storage unit 14 (step S2).

Further, the processing management unit 15b stores a predetermined feature amount in the processing unit 40 based on predetermined conditions at predetermined time intervals or in response to a request from the processing unit 40. It is extracted from the preprocessing result 14a and delivered (step S3). The processing unit 40 uses the delivered features to execute the main processing corresponding to batch processing such as scan detection and malware infection detection. As a result, a series of management processes is completed.

As described above, in the management device 10 of the present embodiment, the preprocessing management unit 15a acquires the data 20 and delivers it to the preprocessing unit 30 for calculating a predetermined feature amount of the data, and the calculation is performed. The feature amount is received and stored in the storage unit 14. In addition, the processing management unit 15b extracts the feature amount from the storage unit 14 and delivers it to the processing unit 40, which performs a predetermined process using the feature amount.

As a result, for example, preprocessing performed in real time or at short time intervals speeds up the digestion of network information, reduces the amount of information required for this processing, and shortens the time required for processing. Further, since the pretreatment result obtained in the pretreatment can be diverted to a plurality of main treatments, the processing efficiency is improved as a whole and the time required for the processing is shortened. As described above, according to the management process of the management device 10 of the present embodiment, it is possible to improve the efficiency of a plurality of types of the main process for the same data set.

Further, the processing management unit 15b may search the requested feature amount from the preprocessing result 14a and deliver it to the processing unit 40 in response to the request from the processing unit 40. As a result, it is possible to deliver an appropriate preprocessing result to the main processing unit 40 even if the delivery conditions are not set in advance.

Further, the processing management unit 15b may delete the feature amount that does not need to be delivered to the processing unit 40 from the preprocessing result 14a. This makes it possible to improve the usage efficiency of the storage unit 14.

The data 20 is not limited to data that changes in time series. For example, the time series order of the time stamps may be reversed. It can also be applied to processing a large amount of past data. It can also be applied to data that does not have a time stamp.

[program]
It is also possible to create a program in which the processing executed by the management device 10 according to the above embodiment is described in a language that can be executed by a computer. In one embodiment, the management device 10 can be implemented by installing a management program that executes the above management process as package software or online software on a desired computer. For example, by causing the information processing device to execute the above management program, the information processing device can function as the management device 10. The information processing device referred to here includes a desktop type or notebook type personal computer. In addition, the information processing device includes smartphones, mobile communication terminals such as mobile phones and PHS (Personal Handyphone System), and slate terminals such as PDAs (Personal Digital Assistants). Further, the terminal device used by the user may be used as a client, and the terminal device may be implemented as a server device that provides the service related to the above management process to the client. For example, the management device 10 is implemented as a server device that provides a management processing service that inputs data and outputs a preprocessing result used in this processing. In this case, the management device 10 may be implemented as a Web server, or may be implemented as a cloud that provides services related to the above management processing by outsourcing. An example of a computer that executes a management program that realizes the same functions as the management device 10 will be described below.

As shown in FIG. 5, the computer 1000 that executes the management program includes, for example, a memory 1010, a CPU 1020, a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface. It has 1070 and. Each of these parts is connected by a bus 1080.

The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM 1012. The ROM 1011 stores, for example, a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to the hard disk drive 1031. The disk drive interface 1040 is connected to the disk drive 1041. A removable storage medium such as a magnetic disk or an optical disk is inserted into the disk drive 1041. For example, a mouse 1051 and a keyboard 1052 are connected to the serial port interface 1050. For example, a display 1061 is connected to the video adapter 1060.

Here, as shown in FIG. 5, the hard disk drive 1031 stores, for example, the OS 1091, the application program 1092, the program module 1093, and the program data 1094. Each table described in the above embodiment is stored in, for example, the hard disk drive 1031 or the memory 1010.

Further, the management program is stored in the hard disk drive 1031 as, for example, a program module 1093 in which a command executed by the computer 1000 is described. Specifically, the program module 1093 in which each process executed by the management device 10 described in the above embodiment is described is stored in the hard disk drive 1031.

Further, the data used for information processing by the management program is stored as program data 1094 in, for example, the hard disk drive 1031. Then, the CPU 1020 reads the program module 1093 and the program data 1094 stored in the hard disk drive 1031 into the RAM 1012 as needed, and executes each of the above-described procedures.

The program module 1093 and program data 1094 related to the management program are not limited to the case where they are stored in the hard disk drive 1031. For example, they are stored in a removable storage medium and read by the CPU 1020 via the disk drive 1041 or the like. May be done. Alternatively, the program module 1093 and the program data 1094 related to the management program are stored in another computer connected via a network such as a LAN or WAN (Wide Area Network), and read by the CPU 1020 via the network interface 1070. You may.

Although the embodiment to which the invention made by the present inventor is applied has been described above, the present invention is not limited by the description and the drawings which form a part of the disclosure of the present invention according to the present embodiment. That is, all other embodiments, examples, operational techniques, and the like made by those skilled in the art based on the present embodiment are included in the scope of the present invention.

10 Management device 11 Input unit 12 Output unit 13 Communication control unit 14 Storage unit 14a Preprocessing result 15 Control unit 15a Preprocessing management unit 15b This processing management unit

Claims (5)

The data is acquired, the acquired data is delivered to a plurality of preprocessing units that calculate a predetermined feature amount of the data, the feature amount calculated from each of the preprocessing units is received, and the received feature is received. Pre-processing management unit that saves the amount in the storage unit,
The main processing management unit that extracts the feature amount corresponding to the processing from the storage unit and delivers it to the main processing unit that performs a predetermined process using the feature amount.
Equipped with a,
The processing management unit searches in response to a request from the processing unit, extracts the feature amount from the storage unit, and delivers it, or the processing management unit delivers the feature amount to the processing unit according to internally determined conditions. processing management apparatus characterized that you request the treatment delivery by extracting the feature amount required from the storage unit to. The pre-processing management unit, in response to adding or updating of the pre-processing unit, according to claim, characterized in that stored in the storage unit identification information identifying the pretreatment unit by applying to the feature quantity 1 management apparatus according to. Wherein the feature amount stored in the storage unit, the additional by prior processing manager, update, or delete, or claim 1 or 2, characterized in that it is possible to extract or delete according to the process controller The management device described in. It is a management method executed by the management device.
The data is acquired, the acquired data is delivered to a plurality of preprocessing units that calculate a predetermined feature amount of the data, the feature amount calculated from each of the preprocessing units is received, and the received feature is received. Pretreatment management process to save the amount in the storage unit,
The main processing management step of extracting the feature amount corresponding to the processing from the storage unit and handing it over to the main processing unit that performs a predetermined process using the feature amount.
Only including,
In the present processing management step, the feature amount is extracted from the storage unit and delivered by searching in response to a request from the main processing unit, or the said to the main processing unit according to internally determined conditions. A management method characterized in that the feature amount required for processing is extracted from the storage unit, delivered, and the processing is requested . The data is acquired, the acquired data is delivered to a plurality of preprocessing units that calculate a predetermined feature amount of the data, the feature amount calculated from each of the preprocessing units is received, and the received feature is received. Pre-processing management step to save the amount in the storage,
A main processing management step of extracting a feature amount corresponding to the processing from the storage unit and handing it over to the main processing unit that performs a predetermined process using the feature amount.
The processing management step searches in response to a request from the processing unit, extracts the feature amount from the storage unit, and delivers the feature amount to the processing unit, or according to an internally determined condition, the processing management step. A process of extracting the feature amount required for the process from the storage unit, delivering the feature amount, and requesting the process.
A management program characterized by having a computer execute.

Images