JP6886935B2 - Data analysis support system and data analysis support method - Google Patents

Description

The present invention relates to a data analysis support system and a data analysis support method.

The demand for analyzing data has been strong for a long time. It is difficult to analyze a huge amount of data manually, and various data analysis techniques such as data mining have been developed. As one of them, machine learning is a technology that formulates some model based on statistics and probabilities, and conversely estimates the detailed parameters of the model from the actually observed data. Computer judgments based on machine-learned models are considered intellectual judgments and can be useful for classification, prediction, recommendation, etc.

However, even if the big data of cobblestone mixing is directly analyzed without thinking, it tends to be difficult to obtain potential results because only superficially estimable results can be obtained without investing a huge amount of computer resources. .. Because of this tendency, we often analyze specific topics for specific data in order to obtain useful results.

For example, in Patent Document 1, the topic of interest of each account is estimated by referring to the purchase information table, the probability that a product is purchased for each topic is estimated, and if the product is a new product, it is purchased. A technique for recommending a new product with a high probability of being patented is disclosed.

Further, Patent Document 2 discloses a technique for learning the degree of similarity between job information and resume information by using an analysis method based on a topic model.

Japanese Unexamined Patent Publication No. 2012-242940 JP-A-2017-134732

By using the technique disclosed in Patent Document 1 or Patent Document 2, it is possible to analyze a specific topic for a specific data. However, even if machine learning of big data, which consists of many tables and whose topic cannot be determined, the relationships between the data in the same table are strong, and the relationships across tables tend to be buried. is there.

When the relationship in the table is noise (N) and the relationship across tables is signal (S), simply increasing the number of records in the table to be learned does not improve this SN ratio. And, in Patent Document 1 and Patent Document 2, there is no disclosure of the technique relating to the improvement of the SN ratio in such learning.

In addition, it is difficult to get a bird's-eye view of the data relationship in a data lake that stores big data in which a large amount of diverse data is mixed. In particular, it is important to help data analysts discover unknown patterns in the relationships of raw data, in other words, to extract the data-related information that they are aware of.

An object of the present invention is to provide a data analysis support system that manifests data relationships between tables.

A typical data analysis support system according to the present invention is a data analysis support system configured by a computer, in which the computer includes a memory in which a program is stored and a CPU that executes a program stored in the memory. , The CPU inputs a plurality of tables, extracts a keyword from each of the input plurality of tables, machine-learns the extracted keyword, and as an analysis result of machine learning, the keyword and the appearance probability. The feature components including the pair are extracted for each of the plurality of tables, the combination between the table of the feature components is specified based on the match between the table of the keywords included in the extracted feature component, and the combination is included in the specified combination. It is characterized by synthesizing the appearance probabilities of the characteristic components, aggregating the analysis results, and outputting the aggregated analysis results.

According to the present invention, it is possible to provide a data analysis support system that manifests the data relationship between tables.

It is a figure which shows the example of the data analysis support system. It is a figure which shows the example of a computer. It is a figure which shows the example of data sorting using machine learning. It is a figure which shows the example of the transformation relation definition table. It is a figure which shows the example of the quantity quality conversion table. It is a figure which shows the example of the relationship between a data value and processing. It is a figure which shows the example of combinatorial optimization. It is a figure which shows the example of the processing flow of the data analysis support system. It is a figure which shows the example of the user dialogue screen. It is a figure which shows the example of the processing flow of machine learning.

Among the techniques disclosed in the present specification, configurations and operations according to one viewpoint are as follows, and embodiments of the present invention will be described as examples with reference to the drawings.

FIG. 1 is a diagram showing an example of a data analysis support system. The data analysis support system 101 is composed of a computer, and is connected to the user terminal 102, the data lake 103, the business operation system 104, and the data warehouse 105 by a network 120.

The user terminal 102, the data lake 103, the business operation system 104, and the data warehouse 105 may also be computers. The user accesses these computers including the data analysis support system 101 via the user terminal 102 to input data and display the data.

In FIG. 1, the solid network 120 represents the physical connection between the computers, and the dashed line represents the data flow (or control flow). However, the broken line indicates a typical data flow, and there may be a data flow other than the broken line shown in FIG.

The data lake 103 is a facility for storing raw data before analysis. The operation management log, maintenance record, and the like are stored in the transaction DB 106, which is a database in the data lake 103, from the business operation system 104 via the network 120. As a result, a large amount of records that occur in time series and correspond to the schema structure are accumulated in the transaction DB 106.

Note that the transaction DB 106 may include different types of tables that do not depend on the time series. Further, as a database other than the transaction DB 106, the data lake 103 has a master DB 107 in which a data encoding method and a schema structure are predetermined. The information of the master DB 107 may be stored in advance by the user.

A communication program module in which data transmission / reception procedures are predetermined is stored in the memory of each computer including the business operation system 104 and the data lake 103, and the communication program is executed by the CPU (Central Processing Unit) of each computer. Communicate by.

The user selects the data in the data lake 103, converts the selected data as necessary, and then transfers the selected data to the data warehouse 105 for storage. Alternatively, a selection conversion analysis program module in which selection procedures, conversion procedures, and analysis procedures are predetermined may be stored in the memory of the data warehouse 105 in advance.

The user interactively analyzes the data using the data warehouse 105. For this analysis, the user selects data from the data in the data lake 103, and the data analysis support system 101 provides the user with information on which data should be selected.

In other words, the user receives a recommendation of data to be selected using the data analysis support system 101. Although the data warehouse 105 has been described above in order to explain the role of the data analysis support system 101, the data warehouse 105 is not limited to the one described above.

In the data analysis support system 101, a program module in which an analysis support procedure is described in advance, and parameters such as conditions interactively specified by the user and predetermined initial values are stored in the memory. The analysis support process is realized by executing the program in which the analysis support procedure described in the memory is executed by the CPU.

The program module in which the analysis support procedure is described includes a plurality of submodules, and includes each submodule of the integrated control module 108, the learning control module 109, the centralized control module 110, and the interactive control module 111. Here, each submodule may be a CPU that executes each program, not the program itself.

A database is constructed on the disk (storage device) or memory of the data analysis support system 101, and this database includes a management DB 112, a quality conversion DB 113, a learning DB 114, and an aggregation DB 115. In the management DB 112, information about the conversion relationship definition table (FIG. 4) and the access method to various data in the data lake 103 is created and stored in advance.

In the quantity conversion DB 113, a quantity conversion table (FIG. 5) such as time and place is created and stored in advance. The learning DB 114 stores the intermediate processing result as a learning table (FIG. 6). The aggregation DB 115 stores the final processing result as an aggregation table (FIG. 6).

The data analysis support system 101 starts processing in response to a request from the user, outputs a result to the user, and ends the processing. First, the overall control module 108 puts the dialogue control module 111 into a state of waiting for a request from the user in advance. The delivery of the following series of processes may be performed by the integrated control of the integrated control module 108.

When it is determined that the data analysis support is requested from the user terminal 102 by the user operation, the dialogue control module 111 hands over the process to the integrated control module 108. The integrated control module 108 reads the access method to various data based on the management information defined in the management DB 112, and delivers the process to the learning control module 109.

The learning control module 109 reads the data of the transaction DB 106 (FIG. 6) using the quantity conversion DB 113, and stores a plurality of learning results in the learning DB 114. The aggregation control module 110 reads a plurality of learning results from the learning DB 114, and stores the aggregated data in which the plurality of learning results are aggregated in the aggregation table in the aggregation DB 115. The dialogue control module 111 reads the aggregation result from the aggregation DB 115 and outputs it to the user terminal 102.

FIG. 2 is a diagram showing an example of a computer. The computer 201 is, for example, a data analysis support system 101, and may be other equipment shown in FIG. The computer 201 includes a CPU 202 having a calculation register, a memory 203, a disk 205, an input / output unit 206, a timer 204, and a sensor 207.

The disk 205 may be a magnetic storage device or a non-volatile semiconductor storage device. The input / output unit 206 includes a display, a keyboard, a mouse, and a communication circuit with the network 120. Instead of the user terminal 102, the display, keyboard, and mouse of the input / output unit 206 may be used.

FIG. 3 is a diagram showing an example of data sorting using machine learning. There are a plurality of databases and a plurality of tables in the data lake 103, and the tables include different ones. Here, two tables having a common field (time, etc.) are of the same kind. The data of two tables of the same type can be converted into the data of a single table by the table join process.

In contrast, two tables that do not have a common field are different from each other. Two different types of tables cannot be joined. For example, if there are 4 fields ABCD and 3 tables PQR and they have P (A, B), Q (C, D), R (A, C) data structures, then table P and table Q are different, tables. P and table R are of the same type, and table Q and table R are of the same type. Therefore, the three tables PQR include heterogeneous tables.

When only the table PQ is input, the relationship between the data values included in the field B and the field D cannot be analyzed because the heterogeneous tables cannot be joined. On the other hand, when the table PQR is input, three tables can be joined to one table by first joining PR and then joining Q. Furthermore, it becomes possible to analyze the relationship between the data values in the fields B and D using the joined table.

However, since the tables in the data lake are diverse, the number of combinations that can be joined is enormous, and in the conventional work in the data lake, finding the combination route itself is a great amount of work. In addition, since the analysis results can differ depending on the combination, finding an appropriate table join method has been a matter of trial and error and intuition of the analyst.

On the other hand, in the case of analysis processing using machine learning, it is possible to analyze even heterogeneous tables collectively, and even if data from multiple heterogeneous tables is input, the data is sorted based on some model. It is possible to output relationship information between data. A wide variety of methods have been developed and disclosed for such models of machine learning.

However, no matter what model is used, the result of direct machine learning of the data of different types of tables is strongly evaluated for the relationship that the data in the same table is in the same table, and other Relationships with data in heterogeneous tables tend to be weakly evaluated.

FIG. 3 illustrates a case where four tables of data are learned by using a machine learning method that analyzes the data and performs three classifications on the three analysis axes of XYZ. When the data is directly machine-learned, the data in the first table is biased on the X-axis, the data in the second table is biased on the Y-axis, and so on.

In this way, if the learned sorting results are biased for each table, even if you bother to analyze many (plural) (heterogeneous) tables, the meaning of what to learn becomes less meaningful, and the effect of machine learning can be obtained. I can't.

In the data analysis of this embodiment, as the first step, machine learning that performs three classifications is applied to each table, and each of the four tables is machine-learned independently. As a result, four basis or analysis axes (X1, Y1, Z1). .. .. (X4, Y4, Z4) is obtained.

As a second step, the relationships between the analysis axes are optimally aggregated to obtain three new aggregation axes (Group1, Group2, Group3). Since the aggregation axis focuses on the relationship across tables including between different types of tables, the data relationship between tables becomes apparent and can be extracted.

The data relationship between different types of tables is difficult for a data analysis worker to notice in front of the huge amount of data contained in the different types of tables, and is information obtained by using the data analysis support system 101 of this embodiment. Obtaining such information is a useful effect of this embodiment.

In this embodiment, a topic model is used as an example of machine learning. In the following, the analysis axis will be referred to as a topic. The topic is also a feature component. The topic model is a model mainly used for sorting documents, and one topic is formulated as a list of words and their occurrence probability pairs. The result of machine learning a large number of documents is output as multiple topics. In this respect, it may be the same as general machine learning.

The topic model is simply machine learning that inputs a group of documents and outputs a group of topics. When inputting a group of documents into the topic model, each document is morphologically analyzed to extract keywords in the document, and the list of the keywords is used as an input vector. The topic model formulates an appropriate topic (a pair of a keyword and its appearance probability) that sorts the input vector group by reading a plurality of input vectors.

Individual topics show potential meanings that represent some set of documents, but their meanings are not shown in a single word. For example, when the document group is divided into two topics, word A appears with a probability of 60%, word B has a probability of 0%, and word C has a probability of 40% in the first topic, and word A appears in the second topic. The topic group is output in the form of 0%, word B 70%, word C appearing with a probability of 30%, and so on.

Which topic each document belongs to can be determined by using the internal product value or the distance as an index for the word vector of the input document and the word vector constituting the topic. That is, if the topic obtained by machine learning is used as the analysis axis, the topic model having the analysis axis can be used for classification and sorting of unknown document groups.

On the other hand, when inputting a database record group into the topic model, the above-mentioned morphological analysis is not necessary, each record corresponds to one document, and each field value corresponds to a word. The machine learning process will be described in detail later with reference to FIG. However, there are problems specific to transaction data in the record group.

First, the information in the database record is usually encoded to save disk space, and if the encoded field value is used as it is as an input vector, the code values "0" and "1" become extremely high. It becomes a frequent input vector, and the learning machine cannot interpret it properly.

In addition, machine learning becomes inefficient and requires a huge amount of sample data. Therefore, if machine learning is performed after decoding the code word (code value), the learning efficiency is improved and appropriate learning is possible even with a small number of samples.

Secondly, quantitative variables such as temperature and time have a wide variety as word strings, and learning machines cannot interpret them properly. Even if quantitative variables are directly machine-learned, the machine learning requires an extremely large amount of sample data and processing time, which is not realistic.

In addition, it makes it difficult for data analysts to interpret the generated topics. For example, when the word "3" becomes a keyword that constitutes a topic, it is ambiguous what the word means in which table. Therefore, after converting the quantitative variable into the qualitative variable, the converted qualitative variable is machine-learned.

For example, the weather code "3" is converted to "sunny". This conversion enables efficient machine learning with a relatively small number of samples, and such machine learning saves computational resources. Furthermore, reasonable learning becomes possible in a short period of time. Then, the calculation result becomes easy to understand even for the data analysis worker.

FIG. 4 is a diagram showing an example of a conversion relationship definition table. The conversion relationship definition table 401 includes the fields of transaction table name 402, transaction field name 403, master table name 404, master field name 405, quantity conversion table name 406, and quantity conversion field name 407.

For example, in the first record of the conversion relationship definition table 401, the transaction table name 402 is "Table-B", the transaction field name 403 is "date and time", and the quantity conversion table name 406 is "date and time".

By this record, the quantitative value of date and time in the field of "date and time" of the transaction table called "Table-B" is changed to the information of the quantity conversion table called "date" and the quantity conversion table called "time". A correspondence is defined that converts to a qualitative value based on the information.

Further, in the third record of the conversion relationship definition table 401, the transaction table name 402 is "Table-B", the transaction field name 403 is "alarm", the master table name 404 is "Table-M1", and the master field name 405 is. It is "F". Then, in the field of "F" of the master table called "Table-M1", code words such as "1" and "9" and meanings (quality) are associated with each other as alarm codes.

With this record and the master table, the alarm code in the "alarm" field of the transaction table "Table-B" can be changed from "1" to "1" based on the "F" field of the master table "Table-M1". Correspondence relation to convert to meaning such as "caution" is defined.

Using such a conversion relationship definition table 401, the learning control module 109 can decode codewords and perform quantitative conversion before machine learning. In the example of FIG. 4, the master table and the quantity conversion table are excluded, but both the master table and the quantity conversion table may be defined.

When both the master table and the quantity conversion table are defined in the conversion relationship definition table 401, either table may be selected according to the preset information, or the master table and the quantity conversion table are defined. The codeword or quantitative variable is an exclusion, and either table may be selected depending on the codeword or quantitative variable to be converted.

FIG. 5 is a diagram showing an example of a quantity conversion table. The quantitative conversion table is a table that defines the correspondence that converts a quantitative variable into a qualitative variable. Quantitative variables include date, time, position, distance, price, temperature, age, etc., which are expressed numerically and are implicitly continuous numerical values.

For example, November 30, 2017 is Thursday. November 30, 2017 is quantitative, and the expressions autumn and Thursday are considered qualitative. Converting a qualitative variable to a quantitative variable requires special constraints, but the reverse is possible.

Among such quantity conversion tables, the quantity conversion table 501 is a table related to time, and is a table that enables conversion from one quantitative variable to a plurality of qualitative variables. The quantity conversion table 501 includes fields of a lower limit value 502, an upper limit value 503, a qualitative expression (1) 504, a qualitative expression (2) 505, a reverse lookup table name 506, and a record number 507.

For example, in the first record of the quantity conversion table 501, the lower limit value 502 is "14:00:00", the upper limit value 503 is "14:59:59", and the qualitative expression (1) 504 is "daytime". Qualitative expression (2) 505 is "2 o'clock".

According to this record, if the time as a quantitative expression falls between "14:00:00" and "14:59:59", it becomes a qualitative expression of "daytime" and "2 o'clock". The correspondence of conversion is defined. Further, each field of the reverse lookup table name 506 and the record number 507 is provided so that the conversion relationship definition table 401 shown in FIG. 4 can be reverse-looked up.

Then, in the example shown in FIG. 5, the specific time of "14:06" has three qualitative values: "daytime", "2 o'clock", and "afternoon" of the first record and the second record. Representations are associated.

The conversion process using the quantity conversion table 501 makes it easier for the machine to learn the rough features of the data, streamlines machine learning with a small number of samples, and at the same time suppresses the consumption of computational resources and realizes high-speed processing.

Unlike the sorting of words in a document, the data value of a transaction record contains various quantitative variables, and the data of the quantitative variables is the essential information of the record. In machine learning using a topic model, when estimating a topic contained in a record, it includes a broader interpretation such as "afternoon" or "2 o'clock" rather than learning with the frequency of occurrence of the word "14:06". It is easier to contribute to data analysis support by learning by the frequency of occurrence of new words.

FIG. 6 is a diagram showing an example of the relationship between data and processing. The data of the transaction table of the transaction DB 106 in the data lake 103 is input to the data analysis support system 101. In the example shown in FIG. 6, the transaction DB 106 has two transaction tables, Table-A601 and Table-B602.

The Table-A601 includes fields of date and time 603, person in charge 604, place 605, and work 606, and stores the work record of the worker. The Table-B602 includes fields of date and time 607, alarm 608, and position 609, and stores a sensor output record of a facility.

As described above, the example of FIG. 6 is an example in which Table-A601 and Table-B602, which are heterogeneous tables based on these information sources, support analysis through the relationship between the date and time and the place.

The learning table 610 and the learning table 611 are tables of the learning DB 114 in which the learning results based on the topic model are stored. The data stored in the Table-A601 is decoded, converted from a quantitative variable to a qualitative variable, and the learning result of the converted qualitative variable by machine learning is stored in the learning table 610.

Further, the data stored in the Table-B602 is decoded, converted from a quantitative variable to a qualitative variable, and the learning result of the converted qualitative variable by machine learning is stored in the learning table 611.

The learning table 610 includes fields of topic number 612, keyword 613, and probability of occurrence 614 and is presented in an object-oriented format. The learning table 610 also includes the same fields.

In the numbering of topic number 612, for example, the first topic for the first table (Table-A601) is numbered as "# 1-1". The topic shall be divided into 10. Here, 10 may be a preset number or a calculated number.

In this way, the topic groups when the Table-A601 and the Table-B602 are divided into 10 topics are shown in the learning table 610 and the learning table 611, respectively.

The number of topics obtained by analyzing the Table-A601 is 10. For example, a topic whose topic number 612 is "# 1-1" has a potential meaning by keywords such as "Tuesday", "2 o'clock", "Hanka", and "Osaka" as indicated by keyword 613. This is the topic to show.

Further, the appearance probability of each keyword of the keyword 613 is "0.31", "0.2", "0.04", "0.03", etc. as shown by the appearance probability 614, and is a topic number. The total probability of appearance in the topic of "# 1-1" for 612 is 1.0.

That is, when a certain record exists, when "Tuesday" is included with the appearance probability of "0.31" and "2 o'clock" is included with the appearance probability of "0.2", the existing record is included. , It is possible in principle that the topic number 612 is determined to be included in the topic of "# 1-1".

However, in the actual record, it is binary whether the data of "Tuesday" is included or not, and it does not exist with a halfway appearance probability. In this way, the probability of appearance only indicates the potential meaning of the topic.

Similarly, there are 10 topics obtained by analyzing Table-B602. For example, a topic with a topic number of "# 2-2" is a topic whose potential meaning is indicated by keywords such as "Tuesday", "alarm (3)", and "Osaka". The appearance probability of each keyword is "0.51", "0.07", "0.01", and the like.

The aggregation table 615 is a table of the aggregation DB 115 in which the result of aggregating the optimum combinations of topics through the combinatorial optimization and basal aggregation processes described later is stored. The aggregate table 615 includes a group number 616, a topic number 617, a keyword 618, and a frequency of occurrence 619 for each field, which are object-oriented variable lengths.

A topic group is information that shows the relationship of data across learning tables. In the topic group, one topic is selected from a plurality of topics constituting a certain learning table, one topic is selected from a plurality of topics constituting the next learning table, and the learning table in the learning DB 114 is selected. One topic is selected and combined in order from.

Topic groups are identified by group number 616. The topic group whose group number 616 is "# 1" includes "Topic # 1-1" and "Topic2-2". This is a combination that is determined to be the optimum combination because the keywords such as "Tuesday" match.

In the topic group in which the group number 616 is "# 1", "Hanka" and "alarm (3)" are originally the information contained in the Table-A601 and the Table-B602, that is, the heterogeneous table, and the contents of the table are existing. It is difficult for data analysis workers to notice the relationship between the two just by looking at them using a table browser or the like.

In addition to these, when there are heterogeneous tables that store location-weather relationships, the relationships between multiple tables are even less noticeable. For example, the "alarm (3)" notifying the failure of this equipment may have occurred intermittently due to improper parts replacement work of "Hanka" in rainy weather. Such a causal hypothesis can be the starting point for data discovery that data analysts can come up with.

FIG. 7 is a diagram showing an example of combinatorial optimization. The aggregation control module 110 inputs the learning table 610 and the learning table 611 shown in FIG. 6, and outputs the aggregation table 615. For this purpose, the aggregation control module 110 executes the process described below.

For the sake of explanation, let P be the number of learning tables to be analyzed. When machine learning is performed based on a topic model, one learning table is sorted by a plurality of topics (analysis axes). Let Q be the number of topics that make up one learning table here.

One topic contains multiple keywords and corresponding probability of occurrence data. Let R be the number of keywords that make up the topic here. Here, for the sake of simplicity, it is assumed that all learning tables contain topics with the same number of topics, and all topics have keywords with the same number of keywords, but they do not have to be the same.

Let w (p, q, r) be the r-th keyword constituting the q-th topic of the p-th learning table, and let φ (p, q, r) be the appearance probability of the corresponding keyword. The keyword w is sorted in descending order of appearance probability. w (p, q, 1) is the most frequent keyword in the qth topic of the pth table.

The sum of the appearance probabilities of the keywords that make up a topic is 1. The probability of appearance of a keyword is also shown below when it is simply called the probability of appearance. The appearance probability is a real number format with a maximum value of 1 and a minimum value of 0. Keywords are in variable length string format.

For example, in the example shown in FIG. 6, when the learning table 610 with p = 1 is analyzed and the topic “# 1-1” with topic number 612 is obtained as the topic with q = 1, the appearance probability 614 is the maximum. The keyword 613 that becomes "0.31" is "Tuesday". Therefore, w (p, q, 1) = "Tuesday" and φ (p, q, 1) = "0.31".

A topic group is a grouping of topics in multiple learning tables. Here, the number of topic groups is Q, which is the same as the number of topics in the learning table.

An adjacency matrix x can be defined that indicates whether the j-th topic of the k-th learning table belongs to the i-th topic group. For the element value x (i, j, k) of the adjacency matrix x, if the value is "1", the corresponding element belongs to the topic group, and if the value is "0", the corresponding element belongs to the topic group. Does not belong.

The objective function is defined so that topics containing many keywords with high probability of occurrence belong to the same topic group. That is, as shown in FIG. 7, in a plurality of topics constituting the topic group, the purpose of optimization is to calculate the sum of the products of the appearance probabilities within the range where the keywords of the topics match, and to maximize the sum. ..

The objective function may be formulated as a linear expression of the adjacency matrix x by defining another function for string matching to determine the range in which the keywords match. Instead of matching keywords, it may be an approximation of keywords based on some preset criteria. By utilizing the fact that the keyword w is sorted in descending order of appearance probability, the calculation regarding the keyword w having a low appearance probability may be terminated.

In an adjacency matrix where topics are properly assigned to topic groups, only the associated element values are "1", the other element values in the same row as the element value that is "1", and "1". The other element value in the same column as the element value is "0". That is, as shown in FIG. 7, as a constraint condition, the sum of the values in the row direction of the adjacency matrix is "1", the sum of the values in the column direction is also "1", and the element value x is binary.

The optimization problem obtained by the above formulation is the 01 integer programming problem. By this formulation, the adjacency matrix x of the optimum solution can be obtained at high speed by using a general optimization solver. Alternative procedures, such as those used for student classification, may be applied.

FIG. 8 is a diagram showing an example of a processing flow of the data analysis support system. The entire processing flow shown in FIG. 8 is controlled by the integrated control module 108. First, under the control of the integrated control module 108, the dialogue control module 111 is put into an input (request) waiting state from the user.

In step 801 the dialogue control module 111 receives an input from the user and passes the received input to the overall control module 108. The integrated control module 108 reads the management DB 112 based on the input, makes initial settings, and opens the transaction DB 106.

In step 802, the overall control module 108 sets a table relationship for decoding the codeword with reference to the conversion relationship definition table 401 of the management DB 112. The table relation for decoding the codeword may be set by the learning control module 109.

In step 803, the learning control module 109 refers to the quantity conversion table 501 of the quantity conversion DB 113 and sets a table relationship for converting the quantitative variable into the qualitative variable. By setting the table relation for decoding these codewords and the table relation for converting quantitative variables to qualitative variables, decoding and conversion can be performed by the following processing.

In step 804, the learning control module 109 repeats the processing of steps 805, 806, and 809 for each transaction table to be processed in the opened transaction DB 106.

In step 805, the learning control module 109 calculates the number of topics Q based on the number of qualitative variables. The number of topics Q may be set in advance by the user or may be determined using a data sample. The data sample may be obtained from the i-th transaction table in the iteration of step 804, for which the codeword may be decoded or the quantitative variable may be converted to a qualitative variable.

Here, the configuration peculiar to this embodiment is to determine the number of topics Q with reference to the quantity conversion table 501. For example, when focusing only on the day of the week as a qualitative variable, setting Q ≧ 7 makes it easier for the tendency for each day of the week to emerge.

Quantitative variables representing places can also be quantitatively transformed. Regardless of the content of the quantitative variables, if ((number of qualitative variables) + 1) is used, the analyzed feature components include one qualitative variable, and the noisy components gather in the remaining feature components. Therefore, the effect of classifying by the qualitative variables of interest is enhanced.

In the quantity conversion table, prepare data for sorting time in x ways and places in y ways, and define a function f (): Q = f (x, y) = max (x, y) + 1. And set the number of topics Q. x is a value in which a quantity conversion table (date) is created in advance and the number of rows in the table is counted. y is a value in which a quantity conversion table (location) is created in advance and the number of rows in the table is counted.

Furthermore, the function g (): Q = g (x, y, z) = max (x, y, z) + 1 is defined with z as the number of rows in the quantity conversion table (time), and the number of topics Q is It may be set. Alternatively, when the number of topics Q is a value of the number of combinations such as Q = x * y * z, the user can comprehensively see the analysis result based on the combination of space-time.

In step 806, the learning control module 109 repeats the process of steps 807 and 808 for each of the records in the i-th transaction table in the repetition of step 804.

In step 807, the learning control module 109 creates an input vector corresponding to the jth record of the repetition of step 806. In the creation of this input vector, a set of keywords is created while decoding codewords and converting quantitative variables into qualitative variables. If the j-th record contains a quantitative variable, decoding may be skipped.

In step 808, the learning control module 109 performs machine learning using the number of topics Q and the created input vector. The machine learning process itself may be executed by the learning control module 109 or may be executed outside the data analysis support system 101, and the learning control module 109 when it is executed externally may be executed outside the data analysis support system 101. Information on the number of topics Q and the input vector may be sent to obtain a learning result. The machine learning process will be described in detail later with reference to FIG.

In step 809, when the repetition of steps 807 and 808 is completed, the learning control module 109 stores the learning result of the i-th transaction table obtained by the repetition.

In step 810, the aggregate control module 110 optimizes the combination of topic groups. As described with reference to FIG. 7, the optimal solution is in the form of the adjacency matrix x shown in FIG. With reference to the element value of the adjacency matrix x, if x (i, j, k) = 1, it is determined that the j-th topic of the k-th learning table belongs to the i-th topic group.

In step 811 the aggregation control module 110 aggregates various data (topics, keywords, and appearance probabilities) that make up the topic group. For this aggregation, the aggregation control module 110 first allocates a topic list in the memory area for the index i (0≤i≤Q) indicating each topic group, and the following second and third Repeat the processing procedure.

Second, the aggregate control module 110 includes the r-th component that constitutes the j-th topic of the k-th learning table included in the i-th topic group, that is, the keyword w (k, j, r) and the appearance probability. Add φ (k, j, r) to the topic list and save it in memory.

Thirdly, the aggregation control module 110 integrates by avoiding duplication of keywords at the time of storage. For example, if the same keyword as the keyword in the topic list is a component in another learning table, the average value of the two occurrence probabilities is calculated and saved as the appearance probability of the keyword in the topic list. Keywords are unique within the topic list.

As a result, the analysis results for each learning table are aggregated, and the data relationship across the learning tables becomes apparent. Then, the aggregation control module 110 stores the information stored in the memory, that is, various aggregated data, in the aggregation DB 115 as an aggregation table.

In step 812, the dialogue control module 111 outputs the analysis result and the dialogue input area to the display of the user terminal 102. The analysis result is the information obtained from the aggregated DB 115, and includes the keyword of the topic list and the appearance probability. Further, the interactive input area includes a check box for accepting user selection in parallel with the keyword, and accepts input from the user.

When the user selects a plurality of check boxes and inputs an instruction to output related information using the user terminal 102, the dialogue control module 111 accepts the information of the selected check boxes and searches for the reverse information. It is output to the user terminal 102 as related information.

The output of related information when the checked keyword is "2 o'clock" will be described with reference to FIG. The integrated control module 108 receives a request for searching the keyword "2 o'clock" and related information from the dialogue control module 111, refers to the quantity conversion table 501 based on the received keyword, and has the same qualitative as the received keyword. Search for records that contain expressions.

By this search, the integrated control module 108 finds "2 o'clock" in the qualitative expression (2) 505 of the first record, and "Table" of the reverse lookup table name 506 registered in the first record found. -B "and" 1 "of record number 507 are obtained. Then, the integrated control module 108 searches the transaction DB 106 for the first record of the transaction table called "Table-B".

The integrated control module 108 obtains the first record found by the search and sends it to the interactive control module 111. The dialogue control module outputs the first received record to the user terminal 102 as related information before the quantity conversion. The information of the reverse lookup table name 506 and the record number 507 may be output to the user terminal 102 as related information.

FIG. 9 is a diagram showing an example of a user dialogue screen. The screen 901 shown in FIG. 9 includes an analysis result output area 902 and an interactive input area 903, and is created and output in step 812 described with reference to FIG. In the example shown in FIG. 9, in the analysis result output area 902, as the analysis result of "Group 1", that is, the topic group 1, a pair of a keyword and an appearance probability is listed as "Tuesday 0.31".

In the example shown in FIG. 9, it is simply shown that the record including "Tuesday" is likely to belong to the topic group 1. In addition, another keyword and the appearance probability are listed in the topic group 2 of "Group2". The same keyword may appear in multiple topic groups.

The interactive input area 903 is a check box, which is an input area in which the user can check the check box and select a keyword. In the example shown in FIG. 9, the check boxes of "Tuesday", "Osaka", and "Alarm (3)" are checked, indicating that these keywords are selected by the user.

The reverse lookup information 904 indicates the information on which the analysis is based, and is created and output as related information in step 812. The record containing the numerical value interpreted as "Tuesday" by the quantity conversion is the record whose record number of "transaction table Table-B" is 1, 4, 5, etc., and further, the record of "maintenance table Table-H0". The numbers are records such as 2, 3, and 4.

From the above information shown in FIG. 9, the data analysis worker can find a tendency that "alarm (3)" appears at the same time as "Tuesday", "Osaka", and "fine". Even if only the "transaction table Table-B" containing the original alarm information is analyzed, "alarm (3)" occurs irregularly in various places and it is difficult to pay attention to a specific day or place. Is also possible.

In such a case, when investigating the cause of the alarm due to the compound factor that the maintenance work method of a specific worker is inappropriate and the abnormality occurs only in a specific weather, the conventional hypothesis is such. The data analysis worker had to think about this and investigate the related heterogeneous data on his own.

Moreover, there are numerous combinations of such hypotheses, and the working hours of data analysts tend to be extremely long. In this example, the possibility of such a hypothesis can be recommended to data analysts by presenting a topic group. It is possible to significantly reduce the work time of data analysis workers.

That is, by aggregating and analyzing different types of tables such as "transaction table Table-B", "maintenance table Table-H0", and weather table, useful patterns distributed among different types of tables can be analyzed by data analysis work. It makes it easier for people to find it.

The machine learning processing flow of step 808 will be described in detail with reference to FIG.
The input information for machine learning is a parameter α indicating the bias of the topic appearance frequency, a parameter β indicating the bias of the word appearance frequency, a parameter S of the upper limit of the number of sampling repetitions, a total number of topics K, and a set r of record data.

A stationary distribution of fixed values may be used as the initial value for the parameters α and β. The parameter S of the upper limit of the number of repetitions may be a fixed value. In addition, the following four assumptions are made for variables: (1) the appearance probability of a topic in a record follows the Dirichlet distribution of α, (2) the appearance probability of a data value in the kth topic follows the Dirichlet distribution of β, 3) The latent variable that caused the i-th data value in the d-th record to appear follows a multinomial distribution, and (4) the i-th data value in the d-th record follows a multinomial distribution.

Also, two statistical variables: (1) the number of times the topic k appears in the dth record, and (2) the number of times the kth topic is estimated for the data value index v for all records. Define.

In step 1001, the subsequent processing is repeatedly controlled for all the tables. In step 1002, the quantitative data values of all the records are classified and converted into qualitative data values. All tables may handle the same number of records. Further, an index v uniquely corresponding to the data value w is created.

In step 1003, the subsequent processing is repeatedly controlled. In step 1004, the subsequent processing is repeatedly controlled for all the records. In step 1005, the subsequent processing is repeatedly controlled for all the data value records. In step 1006, latent variables are sampled based on the above assumptions. In step 1007, two statistical variables are updated.

In step 1008, the parameters α and β are updated. In step 1009, the appearance probability φ of the data value in the kth topic is output, and the created data value w is also output. By repeating the above processing, the accuracy of the initially input parameters α and β can be updated and converged based on the statistical properties of the actual data. In step 1010, combinatorial optimization is performed for a plurality of topics (w and φ).

Each step from step 1003 to step 1009 is equivalent to the process of step 808 and is based on the existing technique of peripheral Gibbs sampling.

101 Data analysis support system 109 Learning control module 110 Aggregate control module 113 Quantitative conversion DB
114 Learning DB
115 Aggregate DB

Claims (10)

It is a data analysis support system composed of computers.
The calculator
The memory where the program is stored and
A CPU that executes a program stored in the memory,
With
The CPU
Enter multiple tables and
Extract keywords from each of the multiple entered tables
Machine learning is performed on the extracted keywords, and as a result of machine learning analysis, feature components including a pair of keywords and appearance probabilities are extracted for each of a plurality of tables.
Based on the matching between the table of keywords included in the extracted feature component, the combination between the table of feature components is identified, and the combination is identified.
Synthesize the appearance probabilities of the characteristic components contained in the specified combination, aggregate the analysis results, and
A data analysis support system characterized by outputting aggregated analysis results. The data analysis support system according to claim 1.
The calculator
An additional disk containing a qualitative conversion table that associates a range of quantitative variable values with the values of one or more qualitative variables.
The CPU
The value of the quantitative variable contained in each of the input multiple tables is converted into the value of one or more qualitative variables using the quantitative conversion table, and the value of the converted qualitative variable is converted. A data analysis support system characterized by extracting as keywords. The data analysis support system according to claim 2.
The disc is
A conversion relationship definition table that associates the names of the plurality of input tables, the names of the fields included in the plurality of input tables, and the quality conversion table is further stored.
The CPU
Specify the names of the plurality of input tables and the names of the fields, specify the quantity conversion table to be used using the conversion relationship definition table, and input using the specified quantity conversion table. Data characterized in that the values of quantitative variables contained in each of a plurality of tables are converted into the values of one or more qualitative variables, and the converted qualitative variable values are extracted as keywords. Analysis support system. The data analysis support system according to claim 3.
The CPU
The code word contained in each of the input plurality of tables is converted into the value of the quantitative variable, and the value of the converted quantitative variable is converted into one or more qualitative variables using the quality conversion table. A data analysis support system characterized by converting to the value of and extracting the value of the converted qualitative variable as a keyword. The data analysis support system according to claim 2.
The CPU
In each table among the multiple input tables, the number of feature components is calculated using the number of value types of each of the multiple qualitative variables obtained from each table.
In machine learning of extracted keywords, a data analysis support system characterized by applying the calculated number of feature components to machine learning and using the calculated number of feature components as the analysis result of machine learning. .. The data analysis support system according to claim 5.
The CPU
In each table in the input multiple tables, the number of value types of each of the multiple qualitative variables obtained from each table is specified by identifying the maximum number among the multiple qualitative variables. A data analysis support system characterized in that the number of characteristic components is calculated by adding 1 to the maximum number. The data analysis support system according to claim 5.
The CPU
Data analysis support characterized by calculating the number of feature components by multiplying the number of value types of each of the plurality of qualitative variables obtained from each table in each table among the plurality of input tables. system. The data analysis support system according to claim 2.
The CPU
The value of the quantitative variable contained in each of the input multiple tables is converted into the value of one or more qualitative variables using the quantitative conversion table, and the value of the quantitative variable before conversion is converted. Information that associates the name of the included table with the value of the qualitative variable after conversion is stored in the quantitative conversion table, and the value of the qualitative variable after conversion is extracted as a keyword.
Outputs the aggregated analysis results of feature components including keywords, accepts selection input for the output keyword, identifies the selected keyword by the accepted input, and of the qualitative variables extracted as the specified keyword. A data analysis support system characterized in that the name of a table associated with a value is searched for from the qualitative conversion table and output. It is a data analysis support method using a computer.
The calculator
The memory where the program is stored and
A CPU that executes a program stored in the memory,
With
The CPU
Enter multiple tables and
Extract keywords from each of the multiple entered tables
Machine learning is performed on the extracted keywords, and as a result of machine learning analysis, feature components including a pair of keywords and appearance probabilities are extracted for each of a plurality of tables.
Based on the matching between the table of keywords included in the extracted feature component, the combination between the table of feature components is identified, and the combination is identified.
Synthesize the appearance probabilities of the characteristic components contained in the specified combination, aggregate the analysis results, and
A data analysis support method characterized by outputting aggregated analysis results. The data analysis support method according to claim 9.
The calculator
An additional disk containing a qualitative conversion table that associates a range of quantitative variable values with the values of one or more qualitative variables.
The CPU
The value of the quantitative variable contained in each of the input multiple tables is converted into the value of one or more qualitative variables using the qualitative conversion table, and the value of the converted qualitative variable is converted. A data analysis support method characterized by extracting as a keyword.

Images