JP2019168820A - Data analysis support system and data analysis support method - Google Patents

Abstract

To provide a data analysis support system revealing a data relation among tables.SOLUTION: In a data analysis support system comprising a computer, the computer comprises a memory in which a program is stored and a CPU executing the program stored in the memory, the CPU inputs multiple tables, extracts a keyword from each of the inputted multiple tables, conducts machine learning of the extracted keyword, extracts a characteristic component including a pair of the keyword and an appearance probability from each of the multiple tables as a result of the machine learning, identifies a combination of inter-table characteristic components based on inter-table matching of a keyword included in the extracted characteristic component, synthesizes an appearance probability of a characteristic component included in the identified combination to integrate analytical results, and outputs the integrated analytical results.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a data analysis support system and a data analysis support method.

  The demand to analyze data has been persistent for a long time. It is difficult to analyze a huge amount of data manually, and various data analysis techniques such as data mining have been developed. As one of them, machine learning is a technique for formulating some model based on statistics, probabilities, etc., and estimating the detailed parameters of the model from the actually observed data. Computer judgment based on a machine-learned model is regarded as intelligent judgment and can be useful for classification, prediction, recommendation, and the like.

  However, even if we directly analyze big data of boulders mixed without thinking, we can only obtain superficial estimation results without investing a huge amount of computer resources, and it is difficult to obtain potential results. . Because of this tendency, specific topics are often analyzed for specific data in order to obtain useful results.

  For example, Patent Literature 1 estimates a topic in which each account is interested with reference to a purchase information table, estimates a probability that a product is purchased for each topic, and purchases a product if the product is a new product. A technique for recommending new products with a high probability is described.

  Patent Document 2 discloses a technique for performing similarity learning between job information and resume information by using an analysis method based on a topic model.

JP 2012-242940 A JP 2017-134732 A

  If the technique disclosed in Patent Document 1 or Patent Document 2 is used, it is possible to analyze a specific topic for specific data. However, even when machine learning big data that consists of many tables and cannot define topics, the relationships between the data in the same table are strong and the relationships across the tables tend to be buried. is there.

  When the relationship in the table is noise (N) and the relationship across the table is a signal (S), simply increasing the number of records in the table to be learned does not improve the S / N ratio. In Patent Document 1 and Patent Document 2, no disclosure of the technology relating to the improvement of the SN ratio in such learning is found.

  In addition, it is difficult to take a bird's-eye view of data relationships in a data lake that stores big data that contains a large amount of various data. In particular, it is important for the data analysis worker to support the discovery of unknown patterns in the relationships of raw data, in other words, to extract data relationship information that is noticed.

  An object of the present invention is to provide a data analysis support system that makes the data relationship between tables obvious.

  A typical data analysis support system according to the present invention is a data analysis support system configured by a computer, the computer including a memory in which a program is stored, and a CPU that executes the program stored in the memory. The CPU inputs a plurality of tables, extracts a keyword from each of the input plurality of tables, performs machine learning on the extracted keyword, and uses the keyword and the appearance probability as a result of machine learning analysis. A feature component including a pair is extracted for each of a plurality of tables, a combination between feature component tables is identified based on a match between keyword tables included in the extracted feature component, and included in the identified combination This is characterized by combining the appearance probabilities of the feature components to be collected, aggregating the analysis results, and outputting the aggregated analysis results. .

  ADVANTAGE OF THE INVENTION According to this invention, it becomes possible to provide the data analysis support system which actualizes the data relationship between tables.

It is a figure which shows the example of a data analysis support system. It is a figure which shows the example of a computer. It is a figure which shows the example of the data classification using machine learning. It is a figure which shows the example of a conversion relationship definition table. It is a figure which shows the example of a quantity quality conversion table. It is a figure which shows the example of the relationship between a data value and a process. It is a figure which shows the example of combination optimization. It is a figure which shows the example of the processing flow of a data analysis support system. It is a figure which shows the example of a user interaction screen. It is a figure which shows the example of the processing flow of machine learning.

  Among the techniques disclosed in this specification, the configuration and operation according to one aspect are as follows, and an embodiment of the present invention will be described as an example with reference to the drawings.

  FIG. 1 is a diagram illustrating an example of a data analysis support system. The data analysis support system 101 includes a computer and is connected to a user terminal 102, a data lake 103, a business operation system 104, and a data warehouse 105 through a network 120.

  The user terminal 102, the data lake 103, the business operation system 104, and the data warehouse 105 may also be computers. The user accesses these computers including the data analysis support system 101 via the user terminal 102, and inputs data or displays data.

  In FIG. 1, a solid line network 120 represents a physical connection between computers, and a broken line represents a data flow (or control flow). However, the broken line indicates a typical data flow, and there may be a data flow other than the broken line shown in FIG.

  The data lake 103 is a facility for storing raw data before analysis. Operation management logs, maintenance records, and the like are stored from the business operation system 104 via the network 120 into the transaction DB 106 that is a database in the data lake 103. As a result, a large amount of records corresponding to the schema structure are generated in the transaction DB 106 in time series.

  The transaction DB 106 may include different types of tables that do not depend on time series. As a database other than the transaction DB 106, the data lake 103 has a master DB 107 in which a data encoding method and a schema structure are predetermined. Information of the master DB 107 may be stored in advance by the user.

  The memory of each computer including the business operation system 104 and the data lake 103 stores a communication program module in which a data transmission / reception procedure is determined in advance, and the communication program is executed by a CPU (Central Processing Unit) of each computer. Communicate by.

  The user selects data in the data lake 103, converts the selected data as necessary, and transfers the data to the data warehouse 105 for storage. Alternatively, a selective conversion analysis program module in which a selection procedure, a conversion procedure, and an analysis procedure are determined in advance may be stored in the memory of the data warehouse 105 in advance.

  The user interactively analyzes data using the data warehouse 105. For this analysis, the user selects data from the data in the data lake 103. The data analysis support system 101 provides the user with information on which data to select.

  In other words, the user receives a recommendation of data to be selected using the data analysis support system 101. In order to explain the role of the data analysis support system 101, the data warehouse 105 has been described as described above. However, the data warehouse 105 is not limited to the one described above.

  The data analysis support system 101 stores a program module in which an analysis support procedure is described in advance, and parameters such as conditions specified interactively by the user and predetermined initial values. An analysis support process is realized by executing a program describing the analysis support procedure stored in the memory by the CPU.

  The program module in which the analysis support procedure is described includes a plurality of submodules, and includes the overall control module 108, the learning control module 109, the aggregation control module 110, and the dialog control module 111. Here, each submodule may be a CPU that executes each program instead of the program itself.

  A database is constructed on a disk (storage device) or memory of the data analysis support system 101, and this database includes a management DB 112, a quality conversion DB 113, a learning DB 114, and an aggregation DB 115. In the management DB 112, information on the conversion relationship definition table (FIG. 4) and the access method to various data in the data lake 103 is created and stored in advance.

  In the quality conversion DB 113, a quality conversion table (FIG. 5) such as time and place is created and stored in advance. The learning DB 114 stores intermediate processing results as a learning table (FIG. 6). The aggregation DB 115 stores final processing results as an aggregation table (FIG. 6).

  The data analysis support system 101 starts processing in response to a request from the user, outputs a result to the user, and ends the processing. First, the overall control module 108 puts the dialog control module 111 into a state waiting for a request from the user in advance. It should be noted that the delivery of the following series of processes may be performed by the overall control of the overall control module 108.

  If it is determined that data analysis support is requested from the user terminal 102 by a user operation, the dialogue control module 111 passes the processing to the overall control module 108. The overall control module 108 reads an access method to various data based on the management information defined in the management DB 112 and hands over the processing to the learning control module 109.

  The learning control module 109 reads the data of the transaction DB 106 (FIG. 6) using the quality conversion DB 113 and stores a plurality of learning results in the learning DB 114. The aggregation control module 110 reads a plurality of learning results from the learning DB 114 and stores aggregated data obtained by aggregating the plurality of learning results in an aggregation table in the aggregation DB 115. The dialogue control module 111 reads the aggregation result from the aggregation DB 115 and outputs it to the user terminal 102.

  FIG. 2 is a diagram illustrating an example of a computer. The computer 201 is the data analysis support system 101, for example, and may be the other equipment shown in FIG. The computer 201 includes a CPU 202 having a calculation register, a memory 203, a disk 205, an input / output unit 206, a timer 204, and a sensor 207.

  The disk 205 may be a magnetic storage device or a non-volatile semiconductor storage device. The input / output unit 206 includes a display, a keyboard, a mouse, and a communication circuit with the network 120. Instead of the user terminal 102, the display, keyboard, and mouse of the input / output unit 206 may be used.

  FIG. 3 is a diagram illustrating an example of data sorting using machine learning. The data lake 103 includes a plurality of databases and a plurality of tables, and the tables include different types. Here, two tables having a common field (time etc.) are of the same type. Data of two tables of the same kind can be converted into data of a single table by table combination processing.

  On the other hand, two tables that do not have a common field are different from each other. Two different tables cannot be joined. For example, if there are four fields ABCD and three tables PQR, and they have data structures of P (A, B), Q (C, D), and R (A, C), the table P and the table Q are different types. P and table R are the same type, and table Q and table R are the same type. For this reason, the three tables PQR include heterogeneous tables.

  When only the table PQ is input, since the heterogeneous tables cannot be joined, the relationship between the data values included in the field B and the field D cannot be analyzed. On the other hand, when the table PQR is input, three tables can be combined into one table by first combining PR and then combining Q. Furthermore, the relationship between the data values in the fields B and D can be analyzed using the combined table.

  However, since there are a wide variety of tables in the data lake, the number of combinations that can be combined is enormous, and in the conventional work in the data lake, finding the combination route itself is a very heavy workload. In addition, since the analysis results may differ depending on the combination, finding an appropriate table joining method has depended on the analyst's trial and error and intuition.

  On the other hand, in the case of analysis processing using machine learning, even different types of tables can be analyzed together, and even if data of a plurality of different types of tables are input, the data is sorted based on some model. The relationship information between data can be output. A wide variety of methods have been developed and disclosed for such models of machine learning.

  However, no matter what model is used, the result of direct machine learning of data in different tables is strongly evaluated as having a relationship with the fact that the data in the same table is in the same table. The relationship with data in different tables tends to be weakly evaluated.

  FIG. 3 exemplifies a case where four tables of data are learned using a machine learning method in which data is analyzed and three classification is performed on three analysis axes of XYZ. If the data is directly machine-learned, the data in the first table tends to be biased on the X axis, and the data in the second table is biased on the Y axis.

  In this way, if the learned sorting results are biased for each table, the meaning of what to learn will be lessened even if many (multiple) (different) tables are analyzed. I can't.

  In the data analysis of this embodiment, as a first step, machine learning for performing three classifications is applied to each table, and each of the four tables is machine-learned independently. As a result, four bases or analysis axes (X1, Y1, Z1). . . (X4, Y4, Z4) is obtained.

  As a second step, the relationships between the analysis axes are optimally aggregated to obtain three new aggregation axes (Group1, Group2, Group3). Since the aggregation axis focuses on the relationship across tables including different tables, the data relationship between the tables becomes obvious and can be extracted.

  The data relationship between the heterogeneous tables is information obtained by using the data analysis support system 101 of the present embodiment, which is difficult for a data analysis operator who has previously recognized the enormous amount of data included in the heterogeneous table. Obtaining such information is a useful effect of this embodiment.

  In this embodiment, a topic model is used as an example of machine learning. Below, the analysis axis is called a topic. A topic is also a feature component. The topic model is a model mainly used for sorting documents, and one topic is formulated as a list of pairs of words and their appearance probabilities. The results of machine learning of a large number of documents are output as a plurality of topics. In this respect, it may be the same as general machine learning.

  The topic model is simply machine learning that inputs a document group and outputs the topic group. When inputting a document group into a topic model, morphological analysis is performed on each document to extract keywords in the document, and a list of the keywords is used as an input vector. The topic model formulates an appropriate topic (keyword and its appearance probability pair) to sort the input vector group by reading a plurality of input vectors.

  Individual topics show potential meanings that represent some documents, but their meanings are not expressed in a single word. For example, when a document group is classified into two topics, the first topic appears with a probability of 60%, the word B has 0%, and the word C has a probability of 40%, and the second topic has the word A. 0%, Word B 70%, and Word C appear with a probability of 30%.

  The topic to which each document belongs can be determined by using the inner product value or distance as an index for the word vector of the input document and the word vectors constituting the topic. That is, if a topic obtained by machine learning is used as an analysis axis, a topic model having the analysis axis can be used for classification and sorting of unknown document groups.

  On the other hand, when a database record group is input to the topic model, the morphological analysis described above is not necessary, and each record corresponds to one document, and each field value corresponds to a word. The machine learning process will be described later in detail with reference to FIG. However, the record group has a problem specific to transaction data.

  First, information in a database record is normally encoded to save disk, etc. If the encoded field value is used as an input vector as it is, the code values “0” and “1” are extremely small. The input vector appears frequently, and the learning machine cannot perform proper interpretation.

  In addition, machine learning becomes inefficient and requires a large amount of sample data. Therefore, when machine learning is performed after decoding a codeword (code value), learning efficiency is improved, and appropriate learning is possible even with a small number of samples.

  Secondly, quantitative variables such as temperature and time have a wide variation as word character strings, and the learning machine cannot perform proper interpretation. Even if machine learning is performed directly on quantitative variables, the machine learning requires an extremely large number of sample data and processing time, which is not realistic.

  In addition, it is difficult for the data analysis operator to interpret the generated topic. For example, when the word “3” becomes a keyword constituting a topic, it is ambiguous what the word means in which table. Therefore, after the quantitative variable is converted into the qualitative variable, the converted qualitative variable is machine-learned.

  For example, the weather code “3” is converted to “clear”. This conversion enables efficient machine learning with a relatively small number of samples, and such machine learning saves computational resources. Furthermore, reasonable learning is possible in a short period of time. And it becomes easy for a data analysis worker to understand a calculation result.

  FIG. 4 is a diagram illustrating an example of the conversion relationship definition table. The conversion relation definition table 401 includes fields of a transaction table name 402, a transaction field name 403, a master table name 404, a master field name 405, a quality conversion table name 406, and a quality conversion field name 407.

  For example, in the first record of the conversion relation definition table 401, the transaction table name 402 is “Table-B”, the transaction field name 403 is “date time”, and the quality conversion table name 406 is “date, time”.

  With this record, the quantitative value of the date and time in the “date and time” field of the transaction table “Table-B” is converted into the information of the quantitative conversion table “date” and the quantitative conversion table of “time”. Based on the information, a correspondence relationship of converting to a qualitative value is defined.

  The third record of the conversion relationship definition table 401 has a transaction table name 402 of “Table-B”, a transaction field name 403 of “alarm”, a master table name 404 of “Table-M1”, and a master field name 405 of “F”. In the “F” field of the master table “Table-M1”, code words such as “1” and “9” are associated with meaning (quality) as alarm codes.

  With this record and the master table, the alarm code in the “alarm” field of the transaction table “Table-B” can be changed from “1” to “1” based on the “F” field of the master table “Table-M1”. Correspondence that is converted into meanings such as “Caution” is defined.

  Using such a conversion relationship definition table 401, the learning control module 109 can decode the codeword and perform quality conversion before machine learning. In the example of FIG. 4, an example in which the master table and the quantity conversion table are rejected is shown, but both the master table and the quantity conversion table may be defined.

  When both the master table and the quality conversion table are defined in the conversion relationship definition table 401, either table may be selected according to preset information, or defined by the master table and the quality conversion table. The code word or the quantitative variable is contradictory, and either table may be selected according to the code word or the quantitative variable to be converted.

  FIG. 5 is a diagram illustrating an example of the quality conversion table. The qualitative conversion table is a table that defines a correspondence relationship for converting a quantitative variable into a qualitative variable. Quantitative variables include date, time, position, distance, price, temperature, age, etc., expressed numerically and implicitly continuous numerical values.

  For example, November 30, 2017 is Thursday. November 30, 2017 is quantitative, and the expressions fall and Thursday are considered qualitative. Converting a qualitative variable to a quantitative variable requires special constraints, but the reverse is possible.

  Among such qualitative conversion tables, the qualitative conversion table 501 is a table relating to time, and is a table that enables conversion from one quantitative variable to a plurality of qualitative variables. The quantitative conversion table 501 includes fields of a lower limit value 502, an upper limit value 503, a qualitative expression (1) 504, a qualitative expression (2) 505, a reverse lookup table name 506, and a record number 507.

  For example, in the first record of the quality conversion table 501, the lower limit 502 is “14:00:00”, the upper limit 503 is “14:59:59”, the qualitative expression (1) 504 is “daytime”, The qualitative expression (2) 505 is “2 o'clock”.

  With this record, if the time as a quantitative expression falls between “14:00:00” and “14:59:59”, the qualitative expression “daytime” and “2 o'clock” will be used. Correspondences to convert are defined. Further, fields for reverse lookup table name 506 and record number 507 are provided so that the conversion relationship definition table 401 shown in FIG.

  In the example shown in FIG. 5, the specific time of “14:06” has three qualitatives, “daytime”, “2pm”, and “afternoon” of the first record and the second record. Expressions are associated.

  The conversion process using the mass-quality conversion table 501 makes it easy for a machine to learn rough characteristics of data, makes machine learning with a small number of samples more efficient, and at the same time, reduces the consumption of computing resources, thereby realizing high-speed processing.

  Unlike the word sorting in the document, the data value of the transaction record includes various quantitative variables, and the quantitative variable data is the essential information of the record. In machine learning using a topic model, when a topic included in a record is estimated, a broad interpretation such as “afternoon” or “2 o'clock” is included rather than learning with the appearance frequency of the word “14:06”. Learning with the frequency of occurrence of the word is more likely to contribute to data analysis support.

  FIG. 6 is a diagram illustrating an example of the relationship between data and processing. Data in the transaction table of the transaction DB 106 in the data lake 103 is input to the data analysis support system 101. In the example illustrated in FIG. 6, the transaction DB 106 includes two transaction tables, Table-A 601 and Table-B 602.

  The Table-A 601 includes fields of a date time 603, a person in charge 604, a place 605, and a work 606, and stores a work record of the worker. Table-B 602 includes fields of date / time 607, alarm 608, and position 609, and stores a sensor output record of a certain facility.

  As described above, the example of FIG. 6 is an example in which Table-A 601 and Table-B 602, which are heterogeneous tables based on these information sources, provide analysis support through the relationship between date and time.

  The learning table 610 and the learning table 611 are tables of the learning DB 114 in which learning results based on the topic model are stored. Data stored in Table-A 601 is decoded, converted from a quantitative variable to a qualitative variable, and a learning result by machine learning of the converted qualitative variable is stored in a learning table 610.

  In addition, the data stored in Table-B 602 is decoded, converted from a quantitative variable to a qualitative variable, and a learning result by machine learning of the converted qualitative variable is stored in the learning table 611.

  The learning table 610 includes fields of topic number 612, keyword 613, and appearance probability 614, and is shown in an object-oriented format. The learning table 610 includes the same fields.

  In the numbering of the topic number 612, for example, the first topic for the first table (Table-A601) is numbered “# 1-1”. The topic is divided into 10 topics. Here, 10 may be a preset number or a calculated number.

  In this way, topic groups when Table-A 601 and Table-B 602 are sorted into 10 topics are shown in the learning table 610 and the learning table 611, respectively.

  The number of topics obtained by analyzing Table-A601 is ten. For example, a topic whose topic number 612 is “# 1-1” has a potential meaning based on keywords such as “Tuesday”, “2 o'clock”, “Hanka”, and “Osaka” as indicated by the keyword 613. It is a topic to show.

  Further, the appearance probability of each keyword of the keyword 613 is “0.31”, “0.2”, “0.04”, “0.03”, etc., as indicated by the appearance probability 614, and the topic number. The sum of the appearance probabilities in the topic 612 of “# 1-1” is 1.0.

  That is, when a certain record exists, “Tuesday” is included with an appearance probability of “0.31”, and “2 o'clock” is included with an appearance probability of “0.2”, the existing record is In principle, it is possible to determine that the topic number 612 is included in the topic “# 1-1”.

  However, in an actual record, data on “Tuesday” is included or not, and it is binary, and does not exist with a halfway appearance probability. Thus, the appearance probability only indicates the potential meaning of the topic.

  Similarly, there are 10 topics obtained by analyzing Table-B602. For example, the topic having the topic number “# 2-2” is a topic that indicates a potential meaning by keywords such as “Tuesday”, “Alarm (3)”, and “Osaka”. Further, the appearance probability of each keyword is “0.51”, “0.07”, “0.01”, and the like.

  The aggregation table 615 is a table of the aggregation DB 115 that stores the result of aggregating the optimal combination of topics through combination optimization and base aggregation processing described later. The aggregation table 615 includes a group number 616, a topic number 617, a keyword 618, and an appearance frequency 619 in each field, and these are object-oriented variable lengths.

  A topic group is information indicating the relationship of data across learning tables. In the topic group, one topic is selected from a plurality of topics constituting a certain learning table, and one topic is selected from a plurality of topics constituting the next learning table, and the learning table in the learning DB 114 is selected. Each topic is selected and combined sequentially.

  A topic group is identified by a group number 616. The topic group whose group number 616 is “# 1” includes “Topic # 1-1” and “Topic2-2”. This is a combination in which keywords such as “Tuesday” are matched and determined to be the optimum combination.

  Among the topic groups with the group number 616 of “# 1”, “Hanka” and “Alarm (3)” are information originally included in the Table-A601 and Table-B602, that is, the heterogeneous tables, and the contents of the table are the existing ones. It is difficult for data analysis workers to notice the relationship between the two simply by using a table browser.

  In addition to these, when there is a heterogeneous table storing the relationship between the location and the weather, the relationship between the plurality of tables is more difficult to notice. For example, the “alarm (3)” notifying the failure of the equipment may have occurred intermittently due to inappropriate parts replacement work of “Hanka” in rainy weather. Such a causal hypothesis can be the starting point for data discovery that data analysis workers can come up with.

  FIG. 7 is a diagram illustrating an example of combination optimization. The aggregation control module 110 inputs the learning table 610 and the learning table 611 illustrated in FIG. 6 and outputs the aggregation table 615. For this purpose, the aggregation control module 110 executes processing described below.

  For the sake of explanation, let P be the number of learning tables to be analyzed. When machine learning based on a topic model is performed, one learning table is sorted by a plurality of topics (analysis axes). Let Q be the number of topics that make up one learning table.

  One topic includes a plurality of keywords and corresponding appearance probability data. The number of keywords constituting the topic here is R. Here, for simplification of explanation, it is assumed that all learning tables include topics having the same number of topics and all topics have keywords having the same number of keywords, but they need not be the same.

  Assume that the r-th keyword constituting the q-th topic of the p-th learning table is w (p, q, r), and the appearance probability of the corresponding keyword is φ (p, q, r). The keywords w are sorted in descending order of appearance probability. w (p, q, 1) is the most frequent keyword in the qth topic of the pth table.

  The sum of the appearance probabilities of keywords constituting a certain topic is 1. In the following, the term “probability of appearance” will also indicate the probability of appearance of a keyword. The appearance probability is a real number format having a maximum value 1 and a minimum value 0. Keywords are in variable-length character string format.

  For example, in the example illustrated in FIG. 6, when the learning table 610 with p = 1 is analyzed and the topic “# 1-1” with the topic number 612 is obtained as the topic with q = 1, the appearance probability 614 is the maximum. The keyword 613 that becomes “0.31” is “Tuesday”. Therefore, w (p, q, 1) = “Tuesday” and φ (p, q, 1) = “0.31”.

  A topic group is obtained by grouping topics of a plurality of learning tables. Here, the number of topic groups is Q, which is the same as the number of topics in the learning table.

  An adjacency matrix x indicating whether or not the jth topic of the kth learning table belongs to the ith topic group can be defined. If the element value x (i, j, k) of the adjacency matrix x is “1”, the corresponding element belongs to the topic group, and if the value is “0”, the corresponding element is the topic group. Does not belong.

  The objective function is defined so that topics containing many keywords with high appearance probabilities belong to the same topic group. That is, as shown in FIG. 7, in a plurality of topics constituting a topic group, the object of optimization is to calculate the sum of the products of appearance probabilities in a range where the keywords of the topics match and to maximize the sum. .

  The objective function may be formulated as a linear expression of the adjacency matrix x by defining another function for character string matching in order to determine a range where the keywords match. Instead of matching keywords, it may be keyword approximation based on some preset criteria. Using the fact that the keywords w are sorted in descending order of appearance probability, the calculation related to the keyword w having the low appearance probability may be aborted.

  In an adjacency matrix in which topics are appropriately assigned to topic groups, only the corresponding element value is “1”, the other element values in the same row as the element value “1”, and “1” The other element value in the same column as the element value becomes “0”. That is, as shown in FIG. 7, as a constraint condition, the sum of the values in the row direction of the adjacency matrix is “1”, the sum of the values in the column direction is also “1”, and the element value x is binary.

  The optimization problem obtained by the above formulation is a 01 integer programming problem. By this formulation, the adjacency matrix x of the optimal solution can be obtained at high speed using a general optimization solver. For example, another solution procedure used for classifying students may be applied.

  FIG. 8 is a diagram illustrating an example of a processing flow of the data analysis support system. The overall processing flow shown in FIG. 8 is controlled by the overall control module 108. First, the dialog control module 111 waits for an input (request) from the user under the control of the overall control module 108.

  In step 801, the dialog control module 111 receives an input from the user, and passes the received input to the overall control module 108. The overall control module 108 reads the management DB 112 based on the input, initializes it, and opens the transaction DB 106.

  In step 802, the overall control module 108 refers to the conversion relationship definition table 401 of the management DB 112 and sets a table relationship for decoding codewords. The table relationship for decoding codewords may be set by the learning control module 109.

  In step 803, the learning control module 109 refers to the quantitative conversion table 501 of the quantitative conversion DB 113 and sets a table relationship for converting quantitative variables into qualitative variables. By setting the table relationship for decoding these codewords and setting the table relationship for converting quantitative variables to qualitative variables, decoding and conversion can be performed by the following processing.

  In step 804, the learning control module 109 repeats the processes in steps 805, 806, and 809 for each transaction table to be processed in the opened transaction DB 106.

  In step 805, the learning control module 109 calculates the topic number Q based on the number of qualitative variables. The number of topics Q may be set in advance by the user or may be determined using data samples. Data samples may be obtained from the i-th transaction table in the iteration of step 804, for which codewords may be decoded or quantitative variables may be converted to qualitative variables.

  Here, the configuration unique to the present embodiment is to determine the number Q of topics by referring to the quality conversion table 501. For example, when focusing only on the day of the week as a qualitative variable, the tendency for each day of the week can be easily revealed by setting Q ≧ 7.

  Quantitative variables representing locations can also be qualitatively converted. Regardless of the contents of the quantitative variable, if ((number of qualitative variables) +1) is used, the analyzed feature components will include one qualitative variable at a time, and noise components will be collected in the remaining feature components Therefore, the effect of classifying by the qualitative variable that has been noticed increases.

  In the quality conversion table, prepare data to sort time in x ways and place in y places, and define function f (): Q = f (x, y) = max (x, y) +1 To set the number of topics Q. x is a value obtained by creating a quality conversion table (date) in advance and counting the number of rows in the table. y is a value obtained by creating a quality conversion table (location) in advance and counting the number of rows in the table.

  Further, the function g (): Q = g (x, y, z) = max (x, y, z) +1 is defined where z is the number of rows of the qualitative conversion table (time), and the topic number Q is It may be set. Alternatively, when the number of topics Q is a value of the number of combinations such as Q = x * y * z, the user can comprehensively see the analysis results based on the combination of time and space.

  In step 806, the learning control module 109 repeats the processing in steps 807 and 808 for each of the records in the i-th transaction table in the repetition of step 804.

  In step 807, the learning control module 109 creates an input vector corresponding to the jth record in the repetition of step 806. In creating the input vector, a set of keywords is created while decoding the code word and converting the quantitative variable into the qualitative variable. Note that when the j-th record includes a quantitative variable, decoding may be skipped.

  In step 808, the learning control module 109 performs machine learning using the topic number Q and the created input vector. Note that the machine learning process itself may be executed by the learning control module 109, or may be executed outside the data analysis support system 101. Information on the topic number Q and the input vector may be sent to obtain a learning result. The machine learning process will be described later in detail with reference to FIG.

  In step 809, when the repetition of steps 807 and 808 is completed, the learning control module 109 stores the learning result of the i-th transaction table obtained by the repetition.

  In step 810, the aggregation control module 110 optimizes the combination of topic groups. As described with reference to FIG. 7, the optimum solution is in the form of the adjacency matrix x shown in FIG. With reference to the element value of the adjacency matrix x, if x (i, j, k) = 1, it is determined that the jth topic of the kth learning table belongs to the ith topic group.

  In step 811, the aggregation control module 110 aggregates various data (topic, keyword, and appearance probability) constituting the topic group. For this aggregation, the aggregation control module 110 first secures a topic list in the memory area for the index i (0 ≦ i ≦ Q) indicating each topic group, and the following second and third Repeat the procedure.

  Secondly, the aggregation control module 110 includes the r-th component constituting the j-th topic of the k-th learning table included in the i-th topic group, that is, the keyword w (k, j, r) and the appearance probability. Add φ (k, j, r) to the topic list and save it in memory.

  Third, the aggregation control module 110 integrates the keywords while avoiding duplication. For example, if the same keyword as the keyword in the topic list is a component in another learning table, the average value of the two occurrence probabilities is calculated and stored as the occurrence probability of the keyword, and the Are unique within the topic list.

  As a result, the analysis results for each learning table are aggregated, and the data relationship across the learning tables becomes apparent. The aggregation control module 110 stores the information stored in the memory, that is, the aggregated various data in the aggregation DB 115 as an aggregation table.

  In step 812, the dialogue control module 111 outputs the analysis result and the dialogue input area on the display of the user terminal 102. The analysis result is information obtained from the aggregation DB 115, and includes the keyword of the topic list and the appearance probability. Furthermore, the dialog input area includes a check box for accepting user selection in parallel with the keyword, and accepts input from the user.

  When the user selects a plurality of check boxes and inputs an instruction to output related information using the user terminal 102, the dialog control module 111 accepts the information of the selected check boxes, the reverse lookup information is searched, It outputs to the user terminal 102 as related information.

  The output of related information when the checked keyword is “2 o'clock” will be described with reference to FIG. The overall control module 108 receives the keyword “2 o'clock” and the related information search request from the dialogue control module 111, refers to the quality conversion table 501 based on the received keyword, and has the same qualitative characteristics as the received keyword. Search for records that contain expressions.

  By this search, the overall control module 108 finds “2 o'clock” in the qualitative expression (2) 505 of the first record, and displays “Table” of the reverse table name 506 registered in the found first record. -B "and record number 507" 1 "are obtained. Then, the overall control module 108 searches the transaction DB 106 for the first record in the transaction table “Table-B”.

  The overall control module 108 obtains the first record found by the search and sends it to the dialog control module 111. The dialogue control module outputs the received first record to the user terminal 102 as related information before the quality conversion. Note that the information of the reverse lookup table name 506 and the record number 507 may be output to the user terminal 102 as related information.

  FIG. 9 is a diagram illustrating an example of a user interaction screen. A screen 901 shown in FIG. 9 includes an analysis result output area 902 and a dialog input area 903, and is created and output in step 812 described with reference to FIG. In the example shown in FIG. 9, the analysis result output area 902 lists pairs of keywords and appearance probabilities such as “Tuesday 0.31” as analysis results of “Group 1”, that is, topic group 1.

  In the example illustrated in FIG. 9, in short, it is indicated that a record including “Tuesday” is highly likely to belong to the topic group 1. In addition, another keyword and appearance probability are listed in the topic group 2 of “Group 2”. The same keyword may appear in a plurality of topic groups.

  The dialog input area 903 is a check box, and is an input area where the user can select a keyword by checking the check box. In the example shown in FIG. 9, the check boxes of “Tuesday”, “Osaka”, and “Alarm (3)” are checked, and the keywords are selected by the user.

  The reverse lookup information 904 indicates information that is the basis of the analysis, and is created and output in step 812 as related information. The record including the numerical value interpreted as “Tuesday” by the quality conversion is a record having a record number of “transaction table Table-B” 1, 4, 5, etc., and a record of “maintenance table Table-H0”. It is a record whose number is 2, 3, 4, etc.

  From the above information shown in FIG. 9, the data analysis worker can find a tendency that “alarm (3)” appears simultaneously with “Tuesday”, “Osaka”, and “Sunny”. Even if only the “transaction table Table-B” containing the original alarm information is analyzed, “alarm (3)” occurs frequently in various places, and it is difficult to pay attention to a specific day or place. There is also a possibility.

  In such a case, when investigating the cause of an alarm due to a complex factor that the maintenance work method of a specific worker is inappropriate and an abnormality occurs only in a specific weather, conventionally such a hypothesis is used. A data analysis worker thought about it, and had to investigate related heterogeneous data by himself.

  In addition, there are many combinations of such hypotheses, and the working time of data analysis workers tends to be extremely long. In this embodiment, the possibility of such a hypothesis can be recommended to a data analysis worker by presenting a topic group. The time required for data analysis workers can be greatly reduced.

  That is, by collecting and analyzing heterogeneous tables such as “transaction table Table-B”, “maintenance table Table-H0”, and weather table, data analysis work Makes it easier for people to find.

The processing flow of machine learning in step 808 will be described in detail with reference to FIG.
Input information to the machine learning includes a parameter α indicating a bias in topic appearance frequency, a parameter β indicating a bias in word appearance frequency, a parameter S for the upper limit of sampling repetitions, a total number K of topics, and a record data set r.

  For the parameters α and β, a fixed distribution with a fixed value may be used as an initial value. The parameter S for the upper limit of the number of repetitions may be a fixed value. In addition, the following four assumptions are made for the variables: (1) The topic occurrence probability in the record follows the Dirichlet distribution of α, (2) The appearance probability of the data value in the k-th topic follows the β Dirichlet distribution, ( 3) The latent variable in which the i-th data value in the d-th record appears follows a multinomial distribution, and (4) the i-th data value in the d-th record follows a multinomial distribution.

  Also, two statistical variables: (1) the number of times topic k appears in the dth record, and (2) the number of times the kth topic is estimated for the data value index v for all records. Define.

  In step 1001, the subsequent processing is repeatedly controlled for all tables. In step 1002, the quantitative data values of all records are classified and converted into qualitative data values. All tables may handle the same number of records. Further, an index v uniquely corresponding to the data value w is created.

  In step 1003, the subsequent processing is repeatedly controlled. In step 1004, the subsequent processing is repeatedly controlled for all records. In step 1005, the subsequent processing is repeatedly controlled for all data value records. In step 1006, a latent variable is sampled based on the above assumption. In step 1007, two statistical variables are updated.

  In step 1008, the parameters α and β are updated. In step 1009, the appearance probability φ of the data value in the kth topic is output, and the generated data value w is also output. By repeating the above processing, it is possible to update and converge the accuracy of the initially input parameters α and β based on the statistical properties of actual data. In step 1010, combination optimization is performed for a plurality of topics (w and φ).

  Each step from step 1003 to step 1009 is equivalent to the processing of step 808 and is based on the existing technique of marginal Gibbs sampling.

101 Data Analysis Support System 109 Learning Control Module 110 Aggregation Control Module 113 Quantity Conversion DB
114 Learning DB
115 Aggregation DB

Claims (10)

A data analysis support system configured by a computer,
The calculator is
Memory where the program is stored;
A CPU for executing a program stored in the memory;
With
The CPU
Enter multiple tables
Extract keywords from each of multiple input tables,
Machine learning is performed on the extracted keywords, and as a result of machine learning analysis, feature components including pairs of keywords and appearance probabilities are extracted for each of a plurality of tables.
Based on the match between the keyword tables contained in the extracted feature components, identify the combinations between the feature component tables,
By combining the appearance probabilities of feature components included in the specified combination,
A data analysis support system characterized by outputting aggregated analysis results. The data analysis support system according to claim 1,
The calculator is
A disk storing a quantitative conversion table that associates a range of values of quantitative variables with values of one or more qualitative variables;
The CPU
Quantitative variable values included in each of the plurality of input tables are converted into one or more qualitative variable values using the quantitative conversion table, and the converted qualitative variable values are converted into values. Data analysis support system characterized by extracting as keywords. The data analysis support system according to claim 2,
The disc is
A conversion relationship definition table that associates the names of a plurality of input tables, the names of fields included in the plurality of input tables, and the quantitative conversion table;
The CPU
Identify the names of multiple input tables and field names, identify the quality conversion table to be used using the conversion relationship definition table, and input using the specified quality conversion table Data of converting a quantitative variable value included in each of the plurality of converted tables into one or more qualitative variable values, and extracting the converted qualitative variable values as keywords Analysis support system. The data analysis support system according to claim 3,
The CPU
The code word included in each of the plurality of input tables is converted into a value of a quantitative variable, and the converted quantitative variable value is converted into one or more qualitative variables using the quantitative conversion table. The data analysis support system is characterized in that it converts the value of the qualitative variable as a keyword and converts the value of the converted qualitative variable as a keyword. The data analysis support system according to claim 2,
The CPU
In each table among a plurality of input tables, the number of feature components is calculated using the number of types of values of each of a plurality of qualitative variables obtained from each table,
A data analysis support system characterized in that, in machine learning of extracted keywords, the calculated number of feature components is applied to machine learning, and the feature components of the calculated number of feature components are used as machine learning analysis results. . The data analysis support system according to claim 5,
The CPU
In each table among a plurality of input tables, the number of types of values of each of the plurality of qualitative variables obtained from each table is specified and specified to be the maximum among the plurality of qualitative variables. A data analysis support system for calculating the number of feature components by adding 1 to the maximum number. The data analysis support system according to claim 5,
The CPU
Data analysis support characterized in that the number of feature components is calculated by multiplying the number of value types of each of a plurality of qualitative variables obtained from each table in each of the plurality of input tables system. The data analysis support system according to claim 2,
The CPU
The value of the quantitative variable included in each of the plurality of input tables is converted into the value of one or more qualitative variables using the quantitative conversion table, and the value of the quantitative variable before the conversion is converted. Information that associates the name of the table to be included with the value of the converted qualitative variable is stored in the quantitative conversion table, and the value of the converted qualitative variable is extracted as a keyword,
Outputs the aggregated analysis results of feature components including keywords, accepts selection input for the output keywords, identifies the keywords selected by the accepted input, and extracts qualitative variables extracted as identified keywords A data analysis support system, wherein a name of a table associated with a value is retrieved from the quantitative conversion table and output. A data analysis support method by a computer,
The calculator is
Memory where the program is stored;
A CPU for executing a program stored in the memory;
With
The CPU
Enter multiple tables
Extract keywords from each of multiple input tables,
Machine learning is performed on the extracted keywords, and as a result of machine learning analysis, feature components including pairs of keywords and appearance probabilities are extracted for each of a plurality of tables.
Based on the match between the keyword tables contained in the extracted feature components, identify the combinations between the feature component tables,
By combining the appearance probabilities of feature components included in the specified combination,
A data analysis support method characterized by outputting an aggregated analysis result. The data analysis support method according to claim 9, comprising:
The calculator is
A disk storing a quantitative conversion table that associates a range of values of quantitative variables with values of one or more qualitative variables;
The CPU
Quantitative variable values included in each of the plurality of input tables are converted into one or more qualitative variable values using the quantitative conversion table, and the converted qualitative variable values are converted into values. A data analysis support method characterized by extracting as a keyword.

Images