JP6695490B2 - Learning data management device and learning data management method - Google Patents

Description

  The present invention relates to a learning data management device and a learning data management method, and particularly to a service developed in “DevOps”, which is a development method in which a developer (Operation) and an operation manager (Operations) cooperate and cooperate with each other. And has a function suitable for management of learning data when machine learning is used.

  In recent years, the method of developing services and applications is changing. Conventionally, when providing services and applications, they have been provided to users after a long process of requirement definition, design and development. However, if such a long process is required, it is not possible to quickly respond to changes in user needs, so there is a conventional need to shorten the development cycle. In recent years, development methods such as agile development and DevOps have become popular. As a result, the frequency of updates can be increased and services and applications can be updated multiple times a day.

  When these services and applications are used as monitored systems, conventionally, baseline information indicating the normal range is generated from the metric values other than the failure occurrence period of the monitored systems among the metric values that represent the operating status of the monitored system. There is known a technique of doing (see Patent Document 1).

JP, 2013-066113, A

  In the above-mentioned conventional technique, baseline information is generated by distinguishing the failure occurrence period of the monitored system from the period other than that, but the behavior of the monitored system, that is, the distribution of the metric values frequently changes. It is not supposed to be done. On the other hand, in recent services and applications, it is conceivable that the behavior of the monitored system may change due to frequent updates by a development method such as DevOps described above.

  When the baseline information and the like are generated by machine learning, it is possible that the internal logic is changed by updating the monitored system. If the behavior of the target system changes due to the update, a difference may occur between the learning result created using the metric value before the update and the actual operation, and the prediction accuracy of the baseline information etc. may decrease. It was

  The present invention has been made in consideration of the above points, and has a function of proposing a learning data management device and a learning data management method capable of generating a prediction model with high prediction accuracy for future behavior of a monitored system. ..

  In order to solve such a problem, in the present invention, a monitoring data acquisition unit that acquires monitoring data from a monitoring target system as a monitoring target, and monitoring data that divides the acquired monitoring data according to the behavior of the monitoring target system. A division unit, a feature extraction unit that extracts features from the divided monitoring data, the extracted features are compared with the features of the monitoring data of the monitoring target system that is in operation at the time of processing, and the features that are close to each other are learned. And a prediction model generation unit that generates a prediction model using the selected learning data.

  Further, in the present invention, there is provided a learning data management method in a learning data management device for generating a prediction model using learning data, wherein the learning data management device acquires monitoring data from a monitoring target system as a monitoring target. A monitoring data acquisition step, a monitoring data dividing step in which the learning data management device divides the acquired monitoring data according to the behavior of the monitoring target system, and the learning data management device uses the divided monitoring data And a feature extraction step of extracting the learning data, the learning data management device compares the extracted features with the features of the monitoring data of the monitoring target system that is in operation at the time of executing the process, and learning data having similar features is used for learning. And a learning data selecting step for selecting the learning data selected by the learning data management device. And having a prediction model generating step of generating a prediction model using.

  According to the present invention, it is possible to generate a prediction model with high prediction accuracy for future behavior of the monitored system.

It is a system configuration diagram showing a configuration example of a learning data management device according to the first embodiment. It is a block diagram which shows the structural example of the deployment server shown in FIG. FIG. 2 is a block diagram showing in more detail a configuration example of the virtual machine shown in FIG. 1. FIG. 2 is a block diagram showing in more detail a configuration example of the service monitoring server shown in FIG. 1. FIG. 2 is a block diagram showing in more detail a configuration example of the management server shown in FIG. 1. It is a table block diagram which shows the structural example of the monitoring metric value table shown in FIG. It is a table block diagram which shows the structural example of the learning data table classified by version shown in FIG. It is a table block diagram which shows the structural example of the cluster gravity center position table shown in FIG. It is a table block diagram which shows the structural example of the prediction model table shown in FIG. It is a figure which shows an example of general prediction model information. FIG. 6 is a table calibration diagram showing a configuration example of a program setting table shown in FIG. 5. It is a flow chart which shows an example of learning processing by a 1st embodiment. 13 is a flowchart showing the learning data storage process shown in FIG. 12 in more detail. 13 is a flowchart showing the learning data selection processing shown in FIG. 12 in more detail. It is a figure which shows a calculation of a cluster centroid position and a comparison of cluster centroid positions. 13 is a flowchart showing an example of the prediction model generation process shown in FIG. 12. It is a figure which shows an example of the effect by 1st Embodiment. It is a block diagram which shows the structural example of the management server by 2nd Embodiment. It is a table block diagram which shows the manuscript rearrangement of the learning data table classified by date according to the second embodiment. 11 is a flowchart showing in more detail the learning data storage process by date according to the second embodiment.

  Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

(1) First Embodiment (1-1) Schematic Configuration FIG. 1 shows a configuration example of a learning data management device according to the first embodiment. In the present embodiment, the EC service, which is one type of Web application, is taken as an example of the system to be monitored, but the system is not limited to this. Further, the system to be monitored is not limited to the Web application, and can be used for the behavior of the server, for example, storage response performance prediction.

  Further, in the present embodiment, for example, the version of the monitored system is used as an element for dividing the behavior of the monitored system.

  FIG. 1 shows a configuration example of a computer system as a learning data management device according to the first embodiment. The computer system according to this embodiment includes a monitored system 100, a deployment server 101, a service monitoring server 102, a management server 103, a management terminal 105, and a development terminal 104. These are connected to the network 106 by their own communication interfaces, and are connected to each other via the network 106.

  In the present embodiment, as the monitored system 100, for example, a Web application, specifically, an EC (Electronic Commerce) service is exemplified.

  The management terminal 105 includes a communication interface 134, a processor 133, a storage device 135, and a memory 136, which are connected by an internal bus 145. An input device 137 and an output device 138 are connected to the internal bus 145. The operations manager 139 performs operations on the EC service 100, the deployment server 101, the service monitoring server 102, and the management server 103 via the input device 137 and the output device 138.

  The development terminal 104 includes a communication interface 129, a processor 130, a storage device 131, and a memory 132, which are connected by an internal bus 144. An input device 147 and an output device 148 are connected to the internal bus 144. The developer 140 develops an application using the development terminal 104. The source code of the developed application is stored in the storage device 119 of the deployment server 101 via the network 106.

  The EC service 100 includes a virtual machine 112 provided by virtualization software 111 operating on the physical server 110. The physical server 110 includes a communication interface 113, a processor 114, a storage device 115, and a memory 116. A part of the processor 114, the storage device 115, and the memory 116 of the physical machine 110 is assigned to the virtual machine 112. Operations on the virtual machine 112 are performed via the communication interface 113 of the physical server 110. The communication interface 113, the processor 114, the storage device 115, and the memory 116 are connected by an internal bus 146.

  The deployment server 101 includes a communication interface 117, a processor 118, a storage device 119, and a memory 120. The communication interface 117, the processor 118, the storage device 119, and the memory 120 are connected by the internal bus 141.

  The service monitoring server 102 includes a communication interface 121, a processor 122, a storage device 123, and a memory 124. The communication interface 121, the processor 122, the storage device 123, and the memory 124 are connected by an internal bus 142.

  The management server 103 includes a communication interface 125, a processor 126, a storage device 127, and a memory 128. The communication interface 125, the processor 126, the storage device 127, and the memory 128 are connected by the internal bus 143.

  FIG. 2 is a block diagram showing a configuration example of the deploy server 101. The deployment server 101 has a function of building the source code stored in the source code repository 201 and updating the application program 300 running on the virtual machine 112.

  A deploy program 200 is stored in the memory 120. A source code repository 201 is stored in the storage device 119. The source code repository 201 stores the source code of the application program 300 provided on the virtual machine 112. The source code is developed by the developer 140 using the development terminal 104 and is stored in the source code repository 201 via the network 106.

  When the deploy program 200 receives a deploy instruction via the input device 137 of the management terminal 105, the deploy program 200 builds the source code stored in the source code repository 201, and the execution file of the application program generated by such build. Is replaced with the application program 300 running on the virtual machine 112 to update the application.

  FIG. 3 shows a configuration diagram of the virtual machine 112 operating in the EC service 100. An application program 300 that provides the EC service 100 runs on the memory 116 assigned to the virtual machine 112.

  An application program 300 and a monitoring agent program 301 are stored in the memory 116. The storage device 115 stores a product data DB 303. The product data DB (database) 303 stores product information including product names, product prices, and product inventory numbers. The application program 300 acquires product information stored in the product data DB 303 and provides a service based on this product information.

  The application program 300 is open to the network 106. The monitoring agent program 301 acquires the operation information of the application program 300, and transmits the monitoring metric value obtained by the monitoring to the service management manager program 400 (FIG. 4) of the service monitoring server 102 via the network 106. ..

  FIG. 4 shows a configuration diagram of the service monitoring server 102. The service monitoring server 102 receives and saves the monitoring result from the monitoring manager program 301 running on the virtual machine 112.

  A service management manager program 400 is stored in the memory 124. The storage device 123 stores a monitoring metric value table 401. The service monitoring manager program 400 receives the monitoring metric value acquired by the monitoring agent program 300 running on the virtual machine 112 and stores it in the monitoring metric value table 401 in the storage device 123. Details of the monitoring metric value table 401 will be described later.

  FIG. 5 shows a configuration diagram of the management server 103. The management server 103 has a function of learning the monitoring metric value acquired by the service monitoring server 102 and creating a sales volume prediction model in the application program 300. The processor 126 predicts the number of sales in the EC service 100 using this sales number prediction model.

  The memory 128 stores a learning data storage program 500, a learning data selection program 501, a prediction model generation program 502, and an inventory management program 503. The storage device 127 stores a version-specific learning data table 504, a cluster centroid position table 505, a prediction model table 506, and a program setting table 507.

  The learning data storage program 500 reads the values from the monitoring metric value table 401 of the service monitoring server 102, executes the processing, and then stores the values in the version-based learning data table 504. The learning data selection program 501 calculates the cluster centroid position based on the learning data in the version-specific learning data table 504, and stores it in the cluster centroid position table 505. The learning data selection program 501 selects a table used for learning based on the cluster centroid position stored in this way. Note that although a record is stored in the normal table, in the present embodiment, for convenience of explanation, it is assumed that the version-specific learning data table 504 has a table corresponding to the record.

  The prediction model generation program 502 generates a prediction model based on the learning data of the table selected by the learning data selection program, and stores the prediction model in the prediction model table 506. Details of these processes will be described later.

  The inventory management program 503 acquires the latest prediction model from the prediction model table 506, and holds this acquired prediction model as the sales quantity prediction model 508. The inventory management program 503 predicts the sales quantity in the EC service 100 based on the sales quantity prediction model 508. The operation manager 139 adjusts the order quantity based on this prediction information. The setting file is used by the learning data storage program 500 and the learning data selection program 501.

  FIG. 6 is a diagram showing an example of the monitoring metric value table 401 stored in the storage device 123 of the service monitoring server 102. The management metric value table 401 manages a version 601, a date / time 602, an access count 603, a user count 604, a transition rate 605, and a purchase rate 606. Among these, the metric value in the access count 603 indicates a numerical value such as 5000 times, and the metric indicates the access count item itself.

  The monitoring data is a collection of metric values of each metric at a certain date and time. The monitoring agent program 301 running on the virtual machine 112 sends the metrics value of the application program 300 to the service monitoring server 102, and the monitoring manager program 400 stores the metrics value. The monitoring metric value table 401 stores a version 601, date and time 602, access count 603, user count 604, transition rate 605, and purchase rate 606.

  The version 601 indicates version information of the application program 300 running on the virtual machine 112. The date and time 602 indicates the date and time when the monitoring metric value was acquired, and the access count 603 indicates the number of times the introduction page of the product sold by the EC service 100 was accessed within a unit time. The number of users 604 indicates the number of users registered in the application program 300 when the metrics value is acquired. The transition rate 605 indicates the rate of transition from the product introduction page to the purchase page in the access count 603. The purchase rate 606 indicates the rate of purchasing the product among the access numbers. The number of sales indicates the number of products purchased.

  FIG. 7 is a diagram showing an example of the version-specific learning data table 504 stored in the storage device 127 of the management server 103. Here, the learning data of version 2.03, the learning data of version 2.04, and the learning data of version 2.05 are shown stored in different learning data tables 701, 702, and 703, respectively. ..

  The learning data refers to data stored in the version-based learning data table 504 by the learning data storage program 500. The contents of the version-specific learning data table 504 are normalized by the learning data storage program 500. The version-specific learning data table 504 stores normalized values by selecting only the metrics to be used for learning from the metrics in the monitoring metric value table 401. Items 705, 706, and 707 store normalized values of the number of accesses, the transition rate, and the purchase rate. The item 704 has the role of the ID of the learning data, and is therefore stored as it is without being normalized. In the present embodiment, the number of users 604 is not stored in the version-specific learning data table 504 because it is not used for learning.

  FIG. 8 is a diagram showing an example of the cluster centroid position table 505 stored in the storage device 127 of the management server 103. The cluster center-of-gravity position table 505 stores the calculation result 802 of the cluster center-of-gravity position in the table related to the version 801. The cluster center-of-gravity position refers to the average of coordinates obtained by mapping the learning data of the table of each version of the learning data table 504 in the coordinate space. The cluster centroid position table 505 is used and updated when the learning data selection program 501 is executed.

  FIG. 9 is a diagram showing an example of the prediction model table 506 stored in the storage device 127 of the management server 103. This prediction model table 506 stores the date and time 900 when the prediction model was generated, the version 901 used when generating the prediction model, and the prediction model information 902 generated thereby.

  When a plurality of versions are used for learning, each version is listed like version 901 [2.01] and [2.02]. Information on the prediction model itself is stored in the prediction model information. For example, when the prediction model is created using the neural network as shown in FIG. 10, the weight of each node is stored in the prediction model information. The prediction model table is updated when the prediction model generation program 502 is executed.

  FIG. 10 shows a configuration example of a neural network of the sales volume prediction model 508 created by the prediction model generation program, which is divided into, for example, an input layer, a hidden layer, and an output layer. In the present embodiment, as an example, the input is the number of accesses, the transition rate, and the purchase rate, and the output is the number of sales. The node 1 has one input, which is weighted as “w1_0”.

  On the other hand, with respect to the node N, there are five inputs, and weights “wN_0”, “wN_1”, “wN_2”, “wN_3” and “wN_4” are applied to the respective inputs. The value of this weight is stored in the prediction model information 903 and 904 shown in FIG.

  FIG. 11 is a diagram showing an example of the program setting table 1000 stored in the storage device 127 of the management server 103. In this program setting table 1000, settings used in the learning data storage program 500 and the learning data selection program 501 are stored.

  The processing time execution interval setting 1001 stores the execution interval of the learning process S1100 (FIG. 12) executed by the management server 103. The learning metric selection setting 1002 is used when the learning data selection program 501 selects a metric to be used for learning. The data number threshold setting 1003 is used when the learning data selection program 501 determines whether to execute a process. The cluster centroid position threshold setting 1004 is used when the learning data selection program 501 selects a learning data table to be used for learning.

  The setting of the program setting table 1000 is saved via the network 106 by the operation manager 139 using the input device 137 of the management terminal 105 before the learning process S1100 is executed.

  In the monitoring metric value table 401 described above, since the monitoring target is the EC service 100, the stored metrics are the number of accesses 603, the number of users 604, the transition rate 605, and the purchase rate 606.

  Although the present embodiment exemplifies the EC service as the monitored system 100, the present invention is not limited to this, and can also be applied to, for example, prediction of the response performance of the storage. When the storage response performance is predicted in this way, the monitoring metric value table 401 stores the processor usage rate, cache usage rate, cache size, and the like.

(1-2) Learning Process for Generating Prediction Model FIG. 12 is a flowchart showing an example of a learning process S1100 for generating a prediction model. This flowchart is executed by the management server 103.

  The learning data storage processing S1101 corresponds to the learning data storage program 500, the learning data selection processing S1102 corresponds to the learning data selection program 501, and the prediction model generation processing S1103 performs prediction model generation. It corresponds to the program 502. In the management server 103, these programs 500, 501, 502 are loaded in the memory 128 of the management server 103, and the processing included in each program 500, 501, 502 is executed by the processor 126.

  The learning process S1100 is executed at regular time intervals based on the process execution time interval setting 509 preset by the operations manager 139 (step S1105). In the process execution time interval setting 1001 of the program setting table 1000 shown in FIG. 11, only the process execution interval is described. If the processing execution time interval setting 1001 describes 1 hour, it means that the processing is executed every hour. In the process S1105, the date and time when the previous process was executed is output, and the date and time when the previous process was executed is used in the next learning data storage process S1101.

  In the management server 103, the processor 126 executes the learning data storage process (step S1101), and based on the date and time of the previous execution, reads the data increased from the execution of the previous process S1100 from the monitoring metrics table 401, The learning data table for each version is stored in the learning data table 504.

  Next, the processor 126 executes the learning data selection process (step S1102) and selects the learning data table used for generating the prediction model from the version-specific learning data table 504.

  In the learning data selection process, the processor 126 outputs the learning data table of the version-specific learning data table 504 used for learning (step S1102), then the processor 126 executes the prediction model generation process (step S1103), A new prediction model is generated using the learning data table of the version-specific learning data table 504 passed by the learning data selection process S1102, and is stored in the prediction model table 506.

  Further, the processor 126 acquires from the prediction model table 506 the prediction model generated in the prediction model generation process (step S1103) from the prediction model table 506, and the sales prediction model 507 of the inventory management program 503 is a new prediction model generated. (Step S1104).

  FIG. 13 is a flowchart showing details of the learning data storage process shown in FIG. In the management server 103, the processor 126 acquires, from the monitoring metric value table 401 of the service monitoring server 102 via the network 106, the amount of monitoring data increased from the previous execution (step S1201). At this time, the processor 126 uses the date and time when the previous process is executed and uses the date and time after the previous process is executed to determine whether or not the increased amount of the monitor data is used. To get.

  Next, the processor 126 refers to the learning metric selection setting 1002 set in advance by the operations manager 139 and selects a metric to be used for learning from the monitoring data of the monitoring metric value table 401 (step S1202). In the learning metric selection setting 1002, the metrics used for learning are listed, and for example, the three metrics of the number of accesses, the transition rate, and the purchase rate are listed.

  Next, the processor 126 normalizes the metric value of the metric selected as described above (step S1203). The normalization referred to here is to convert where the metric value is located between the maximum value and the minimum value that can be taken by the metric value of each metric, as indicated by a numerical value between 0 and 1. Finally, in step S1204, the normalized metrics value is stored in the version-specific learning data table 504 by version.

  For example, when the monitoring data of the date and time 2016/10/10 13:00 is acquired in the above step S1201 as the monitoring data of the increased amount from the monitoring metric value table 504, the processor 126 causes the processor 126 to access the number of metrics, transition rate and purchase. A rate is selected (step S1202).

  The processor 126 normalizes the metric value of the metric selected as described above (step S1203), and stores the normalized metric value in the table 703 of the version-specific learning data table 504 (step S1204).

  FIG. 14 is a flowchart showing details of the learning data selection process shown in FIG. First, in the management server 103, the processor 126 confirms whether or not there is a sufficient number of data of the version of the application program 300 running on the virtual machine 112 by referring to the version-specific learning data table 504 (step S1301). ). The determination as to whether or not the number of data is sufficient is based on the data number threshold setting 1003 determined by the operation manager 139 in advance.

  The data number threshold setting 1003 stores only a value indicating how many data items should be determined to be sufficient. For example, when the operation manager 139 has set “300” in advance, if the table 703 of the version-specific learning data table 504 has 300 or more learning data, it is determined that the number of data is sufficient. When the number of data is sufficient, the processor 126 calculates the cluster centroid position using the version-specific learning data table 504 of each version (step S1303).

  At this time, in step S1302 described above, the processor 126 compares the cluster centroid position calculation last time and the number of data used in the cluster centroid position calculation, and calculates the cluster centroid position only when the number of data increases. And the result is stored in the cluster centroid position table 505. The learning data selection program 501 holds the number of pieces of data used for the cluster centroid position calculation.

  After completing the calculation of the cluster centroid positions for all the versions, the processor 126 selects a learning data table to be used for learning (step S1304). In step S1304, the processor 126 follows the cluster centroid position threshold setting 1004 preset by the operations manager 139, and the threshold value is the “distance from the cluster centroid position” of the version of the application program 300 running on the virtual machine 112. The learning data table of the version-specific learning data table 504 that fits in the selected version is selected.

  The distance from the position of the center of gravity of the cluster indicates the difference between the values of the positions of the center of gravity of the clusters. Only the threshold value is stored in the cluster centroid position threshold setting 1004. The cluster centroid position of each version is acquired from the cluster centroid position table 505. When the number of learning data is not sufficient in step S1301, the processor 126 does not execute the cluster centroid position calculation process.

  In this case, the processor 126 selects the version selected at the time of executing the previous learning data selection process, and also selects the learning data table of the version running on the virtual machine 112 (step S1305). The selected learning data table is held by the learning data selection program 501. As described above, in step S1304 or step S1305, the processor 126 selects the learning data table and executes the following prediction model generation process (S1103).

  15A to 15C are conceptual diagrams showing an example of calculating the cluster centroid position in step S1303, and FIG. 15D shows an example of selecting learning data in step S1304. It is a conceptual diagram.

  In the management server 103, the processor 126 maps the learning data in the learning data table for each version into the coordinate space as shown in FIGS. 15 (A) to 15 (C), and determines the center of gravity of the mapped learning data. Calculate (step S1400).

  In FIGS. 15 (A) to 15 (C), the learning data of versions [2.03], [2.04], and [2.05] are mapped, and the position of the center of gravity is calculated. In FIG. 15D, only the cluster centroid position of each version is mapped, and the distance from version 2.05 is compared (step S1401). The threshold value in the figure is a threshold value of the cluster centroid position set by the operation manager 139.

  In the present embodiment, if “0.10” is set as the cluster centroid position threshold value, the cluster centroid position of version [2.05] shown in FIG. 15C is, for example, “0.61”. Therefore, while the cluster centroid position “0.56” of the version [2.03] shown in FIG. 15 (A) is within the threshold value, the version of the version [2.04] shown in FIG. 15 (B) is The cluster centroid position 0.72 is not within the threshold. Based on the above, the version [2.03] and the currently operating version [2.05] are selected as the learning data table.

  FIG. 16 is a flowchart showing details of the prediction model generation process (step S1103). First, in step S1501, the processor 126 refers to the prediction model table 506 and selects a past prediction model corresponding to the selected learning data table. An example of selecting a prediction model is shown below.

  In the learning data selection process (step S1102) described above, since the learning data tables of version [2.03] and version [2.05] are selected as described above, the processor 126 selects from the prediction model table 506. A prediction model generated from the learning data tables of versions [2.03] and [2.05] is searched. If this prediction model does not exist, the processor 126 searches for a prediction model generated in only one of the selected learning data tables.

  If such a prediction model does not exist, and if versions [2.03] and [2.05] are selected as described above, the prediction model generated in version [2.03], and , One of the prediction models generated in version [2.05] is applicable.

  On the other hand, when such a prediction model does not exist, if versions [2.01], [2.03] and [2.05] are selected, versions [2.01], [2. 03], [2.05], [2.01, 2.03], [2.01, 2.05], and [2.03, 2.05] correspond to the prediction models. If there are a plurality of applicable prediction models, the prediction model with the largest total number of data is selected.

  In the first embodiment, in the learning data selection process (step S1102), the processor 126 selects the version [2.03] learning data table 701 from the version-specific learning data table 504, and the version [2 .05] learning data table 703 is selected. In step S1501, the processor 126 selects the prediction model created by version 2.03 from the prediction model table 506.

  Next, in step S1502, the processor 126 determines whether or not the corresponding prediction model is selected in step S1501. When the corresponding prediction model is found and the past prediction model is available, the processor 126 trains difference learning data with respect to the past prediction model selected in step S1501 described above, and generates a new prediction model. (Step S1503), which is registered in the prediction model table 506 (see FIG. 9). The difference here indicates learning data after this date and time based on the item 900 (date and time when the prediction model was created) of the prediction model table 506.

  In the first embodiment, since the past prediction model created in version [2.03] is selected, a new prediction is obtained by additionally learning difference learning data with respect to this past prediction model. A model is generated, and the new prediction model is added to the prediction model table 506.

  On the other hand, when the corresponding prediction model does not exist and the past prediction model cannot be used, the processor 126 generates a prediction model using all the learning data included in the table selected by the learning data selection program 1102 ( In step S1504), this prediction model is added to the prediction model table 506.

  FIG. 17 is a diagram showing how the sales quantity indicated by the inventory management program 503 is predicted. The vertical axis represents the number of sales and the horizontal axis represents time. Below the horizontal axis is shown which version of the application program 300 was running. The sales volume change shown by the solid line is the actual measurement value, the sales volume change shown by the dotted line is the prediction value by the forecast model, and the sales volume change shown by the dashed line is the forecast value by the old prediction model. Is.

  At the current time, the prediction value is updated to the prediction model newly generated in step S1104 described above (corresponding to the dotted line in the figure). In the illustrated example, the version of the application program 300 has been updated from [2.04] to [2.05], and the predicted value by the old model has a large deviation from the actual measured value indicated by the solid line. By updating this with a new model, it becomes possible to make a prediction closer to the actual measurement value.

(1-3) Effects and the like of the first embodiment As described above, in the management server 103 according to the present embodiment, the processor 126 acquires the monitoring data from the monitoring target system 100, and the acquired monitoring data. Are divided according to the behavior of the monitored system 100. The processor 126 compares the characteristics extracted from the divided monitoring data with the characteristics of the monitoring data of the monitoring target system that is operating at the time of executing the process, and selects those having similar characteristics as learning data to be used for learning. A prediction model is generated using the selected learning data.

  According to the present embodiment, such learning data is managed for each behavior of the monitoring target system, and only learning data close to the behavior of the monitoring target system 100 that is currently moving is selected and learned, thereby generating by learning. The prediction accuracy of the prediction model used can be improved. This makes it possible to generate a prediction model with high prediction accuracy for future behavior of the monitored system, and improve prediction accuracy for future behavior.

(2) Second Embodiment Since the second embodiment is almost the same as the first embodiment, description of similar configurations and operations will be omitted, and different points will be mainly described.

(2-1) Features of the Second Embodiment In the second embodiment, the processor 126 uses the learning data saving program 501 to perform the learning data saving process S1101 in the processing S1204 (see FIG. 13). Instead of storing the learning data for each version in the learning data table 504 for each version as in the embodiment of the present invention, the learning data is learned by date and time for each date and time by dividing the case where the behavior is different depending on the time zone such as weekdays and holidays. It is stored in the data table 1700.

  The second embodiment is different from the first embodiment in that the processor 126 uses the learning data of the date / time learning data table 1700 for generating the prediction model, as described later. Hereinafter, it will be described more specifically.

  FIG. 18 is a block diagram showing a configuration example of the management server 103A according to the second embodiment. The management server 103A has substantially the same configuration as the management server 103 according to the first embodiment, but instead of the version-specific learning data table 504, the next date-specific learning data table 1700 capable of storing learning data by date and time. Is different.

  19 (A) to 19 (C) each show a table configuration example of the learning data table by date and time 1700 shown in FIG. The date-by-date learning data table 1700 is stored in the memory 128 of the management server 103. Each learning data table is given a table name including date and time, and indicates at which date and time the learning data was saved.

  Specifically, for example, FIG. 19A illustrates the learning data table 1701 at 9:00 on October 8, 2016, and in FIG. 19B, at 9:00 on October 9, 2016. 19C exemplifies the learning data table 1702 of FIG. 19C, and the learning data table 1703 at 9:00 on October 10, 2016 is illustrated. Each learning data table manages, for example, date and time 704, access count 705, transition rate 706, and purchase rate 707.

  FIG. 20 is a flowchart of the learning data storage process S1101A according to the second embodiment. Note that FIG. 20 in the second embodiment corresponds to FIG. 13 in the first embodiment, and steps S1201, S1202, and S1203 in the second embodiment correspond to those in the first embodiment. This corresponds to steps S1201, S1202 and S1203.

  This learning data storage processing S1101A is executed by the learning data storage program 500 instead of the learning data storage processing S1100 shown in FIGS. 12 and 13 according to the first embodiment. Similar to the first embodiment, the learning data storage program 500 is loaded in the memory 128 and executed by the processor 126. Note that steps S1201 to 1203 according to the second embodiment are the same as those in the first embodiment, so description thereof will be omitted.

  Unlike step S1204 in the first embodiment, the processor 126 divides the learning data by date and time instead of version. At this time, the processor 126 reads the table division setting 1805 preset by the operations manager 139, and divides the learning data based on the table division setting 1805.

  The table division setting 1805 is stored in the program setting table 507 of the management server 103, and the operation manager 139 makes settings via the network 106 using the input device 137 of the management terminal 105.

  The table division setting 1805 describes at which date and time the learning table is divided. Therefore, the learning table may be divided in the same time zone or may be divided in different time zones.

  Note that, for example, in the learning data table 1700 by date and time, the tables 1701, 1702, and 1703 are all divided at 9:00, but only the table 1702 is divided at 12:00 on October 9, 2016. May be. The data normalized by step S1203 is stored in the learning data table 1800 by date and time (step S1204A).

  When the learning data storage processing (step S1101A) is completed, the processor 126 of the second embodiment also executes the learning data selection processing shown in FIG. 12 as in the first embodiment. (Step S1102).

  In this learning data selection process, the processor 126 executes almost the same operation as in the first embodiment, but the learning data table to be processed is not the version-specific learning data table 504 but the date-specific learning data. The table 1700 is different from the first embodiment.

  As a result, in the second embodiment, the processor 126 performs substantially the same processing by using the date-specific learning data table 1700 instead of the version-specific learning data table 504 in step S1306 described above (see FIG. 14). Execution is performed and a table is selected almost in the same manner as in the first embodiment (steps S1304 and S1305).

  By executing the learning data selection process (step S1102) described above, the processor 126 selects a learning data table from the date / time-specific learning data table 1700 and generates a prediction model by using this learning data table as an input (step S1103). ).

(2-2) Effects of the Second Embodiment According to the above configuration, by managing the learning data by date and time, for example, weekdays and holidays, daytime and nighttime, and sale period Even when the behavior of the monitored system 100 is different as in other periods, the data can be cut and learned as described above, and thus a prediction model with high prediction accuracy can be generated in each situation.

(3) Other Embodiments The above-described embodiments are examples for explaining the present invention, and the present invention is not intended to be limited to these embodiments. The present invention can be implemented in various forms without departing from the spirit thereof. For example, in the above embodiment, the EC service is illustrated as an example of the monitored system 100, but the present invention is not limited to this, and various Web applications can be illustrated.

  The present invention is a method for managing learning data when machine learning is used for a service developed by "DevOps", which is a development method in which a developer and an operations manager cooperate and cooperate. It can be widely applied to a learning data management device using.

  103 ... Management server, 500 ... Data storage program, 501 ... Learning data selection program, 502 ... Prediction model generation program, 504 ... Version learning data table, 103 ... Cluster centroid position table, 506 ... Prediction model table, 1100 ... Learning processing, S1101 ... Learning data storage processing, S1102 ... Learning data selection processing, S1103 ... Prediction model generation processing, S1201 ... Monitoring metric value acquisition processing, S1202 ... Learning metrics Selection processing, S1203 ... Metric value normalization processing, S1204 ... Version learning data storage processing, S1303 ... Cluster centroid position calculation processing, S1305 ... Learning data table selection processing, S1503 ... Prediction model generation processing.

Claims (7)

A monitoring data acquisition unit that acquires monitoring data from a monitored system as a monitoring target,
A monitoring data dividing unit that divides the acquired monitoring data according to the behavior of the monitored system,
A feature extraction unit for extracting features from the divided monitoring data,
The extracted characteristics are compared with the characteristics of the monitoring data of the monitoring target system that is operating during processing execution, and the characteristics of the divided monitoring data are similar to the monitoring data of the monitoring target system that is operating during processing execution. A learning data selection unit that selects the monitoring data as learning data to be used for learning,
A prediction model generation unit that generates a prediction model using the selected learning data,
Equipped with
The monitoring data division unit,
A learning data management device, characterized in that the monitoring data is divided according to the version of an application program running in the monitoring target system . The monitored system is a Web application,
The prediction model generation unit,
The learning data management device according to claim 1, wherein the behavior of the Web application is predicted. The prediction model generation unit,
The learning data management device according to claim 1 , wherein an access count, a transition rate, and a purchase rate are used as the learning data. The prediction model generation unit,
The learning data management device according to claim 2 , wherein the behavior is predicted for a server on which the Web application operates as the monitored system. The prediction model generation unit,
The learning data management device according to claim 1 , wherein a processor usage rate, a cache usage rate, and a cache size are used as the learning data.   The learning data management device according to claim 1, further comprising a behavior prediction unit that predicts a behavior of the monitoring target system using the prediction model generated by the prediction model generation unit. A learning data management method in a learning data management device for generating a prediction model using learning data,
A monitoring data acquisition step in which the learning data management device acquires monitoring data from a monitoring target system as a monitoring target,
The learning data management device, a monitoring data dividing step of dividing the acquired monitoring data according to the behavior of the monitored system,
A feature extraction step in which the learning data management device extracts features from the divided monitoring data;
The learning data management device compares the extracted characteristics with the characteristics of the monitoring data of the monitoring target system that is operating during the processing execution, and of the divided monitoring data, the monitoring target system that is operating during the processing execution A learning data selection step of selecting monitoring data whose characteristics are close to those of the monitoring data as learning data to be used for learning,
The learning data management device, a prediction model generation step of generating a prediction model using the selected learning data,
Have a,
In the monitoring data dividing step,
The learning data management method, wherein the learning data management device divides the monitoring data according to the version of an application program running in the monitoring target system .

Images