JP6563549B1 - Data trend analysis method, data trend analysis system, and narrowing and restoring device - Google Patents

Abstract

An object of the present invention is to obtain the same result as that analyzed using all data items with high accuracy while reducing the calculation load by reducing the data items to be analyzed. It also aims to make it easier to interpret the analysis results even if the number of items is large by collecting similar attributes. A computer extracts relationships between data items for a plurality of the data tables, generates related information of the data items, generates restoration information of the data items based on the related information, and the related information Generating a group of the data items based on the data, selecting a representative data item from the data items in the group based on the related information and the restoration information, and performing a trend analysis of the data corresponding to the representative data item Then, the data item of the group to which the representative data item belongs is restored based on the restoration information, and the result of the trend analysis of the data corresponding to the representative data item and the restoration result of the data item are output. [Selection] Figure 1

Description

  The present invention relates to data trend analysis.

  2. Description of the Related Art Conventionally, a technique for deriving data characteristics by performing data trend analysis on a large amount of data using correlation analysis or machine learning is known. As a trend analysis, a cluster method is known in which similar data is collected from given data and classified into several clusters or the degree of similarity is calculated.

  Patent Document 1 is known as a technique for extracting data characteristics and relationships between data items. Patent Document 1 discloses a technique for recommending a column (data item) serving as a primary key suitable for data shaping even if an analyst (or user) does not know the details of data in detail.

JP 2012-238153 A

  In big data analysis, etc., since the amount of data handled is large, there are problems such as insufficient resource performance during analysis and a huge amount of analysis processing time. Conventionally, input data is manually selected and analyzed. The target data was narrowed down.

  However, with the conventional technique of selecting input data at the discretion of a person (analyst), there is a possibility that important data items may be overlooked. There was a problem that the amount of work increased because it was necessary to manually select input data again.

  On the other hand, when data items of analysis target data are mechanically narrowed down and trend analysis is performed based on the narrowed down data, data items that are important to the analyst may not be obtained as output results.

  Therefore, the present invention has been made in view of the above problems, and it is possible to accurately obtain the same result as that analyzed using all data items while reducing the calculation load by narrowing down the data items to be analyzed. Objective. It also aims to make it easier to interpret the analysis results even if the number of items is large by collecting similar attributes.

  The present invention relates to a data trend analysis method for performing data trend analysis on a data table having data corresponding to a data item in a computer including a processor and a memory. A first step of generating association information of the data item, and a second step of the computer generating restoration information of the data item based on the association information, and the computer A third step of generating a group of the data items based on the related information; and a fourth step in which the computer selects a representative data item from the data items in the group based on the related information and the restoration information. A fifth step in which the computer performs a trend analysis of data corresponding to the representative data item; A sixth step of restoring the data item of the group to which the representative data item belongs based on the restoration information; and a result of trend analysis of data corresponding to the representative data item; And a seventh step of outputting a restoration result of the data item.

  Therefore, since the analysis result of the present invention presents similar items together, it is easy for a person to see the analysis result. Results close to the analysis of all items can be obtained (the analysis is not an analysis in which features are lost by selecting an item based on the subjective judgment of the analyst).

It is a block diagram which shows the Example of this invention and shows an example of a data trend analysis system. It is a block diagram which shows the Example of this invention and shows an example of a narrowing down and restoration | restoration server. It is a flowchart which shows the Example of this invention and shows an example of the process performed with a data trend analysis system. It is a figure which shows the Example of this invention and shows an example of a schema information table. It is a figure which shows the Example of this invention and shows an example of an analysis object data table. It is a figure which shows the Example of this invention and shows an example of the relationship result between data. It is a figure which shows the Example of this invention and shows an example of an independent data item table. It is a flowchart which shows the Example of this invention and shows an example of the grouping process of the input data item performed with a data trend analysis system. It is a figure which shows the Example of this invention and shows an example of a summary table. It is the first half of the flowchart which shows the Example of this invention and shows an example of the storing process to a summary table. It is a latter half part of the flowchart which shows the Example of this invention and shows an example of the storage process to a summary table. It is a graph of the previous period total assets total and total assets total which shows the Example of this invention and shows an example of a regression equation. It is a figure which shows the Example of this invention and shows an example of the summary table which allocated the group. It is a figure which shows the Example of this invention and shows an example of the relationship between a data item and a group. It is a flowchart which shows the Example of this invention and shows an example of the selection process of representative data. It is a flowchart which shows the Example of this invention and shows the other example of the selection process of representative data. It is a figure which shows the Example of this invention and shows an example of the division | segmentation of the node in the selection process of representative data. It is a figure which shows the Example of this invention and shows the other example of the division | segmentation of the node in the selection process of representative data. It is a figure which shows the Example of this invention and shows an example of a representative data table. It is a figure which shows the Example of this invention and shows an example of a restoration table. It is a flowchart which shows the Example of this invention and shows an example of a data trend analysis process. It is a figure which shows the Example of this invention and shows an example of the result table of the tendency analysis of data. It is a flowchart which shows the Example of this invention and shows an example of a data restoration process. It is a flowchart which shows the Example of this invention and shows an example of a data restoration calculation process. It is a figure which shows the Example of this invention and shows an example of the display screen of a final result table. It is a figure which shows the Example of this invention and shows an example of a candidate table.

  Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

<Configuration of data trend analysis system>
FIG. 1 is a block diagram illustrating an example of a data trend analysis system according to an embodiment of the present invention. The data trend analysis system 1 includes a database server 2 that stores information such as analysis target data and final results, a narrowing and restoration server 3 that narrows down data items of the analysis target data, and restores the data items, and an analysis target Data trend analysis server 4 that executes trend analysis processing on data, user terminals 6-1 to 6-3 that use the data trend analysis system 1, and a network that connects each server and user terminals 6-1 to 6-3 5 is included.

  In the following description, the symbol “6” in which “−” and the subsequent symbols are omitted is used when each user terminal is not specified. The same applies to the reference numerals of other components.

  The database server 2 is a computer that provides a database 20 and a synonym dictionary 22 and is connected to the network 5 via a network interface (I / F in the figure) 21. The database server 2 operates a database management system (DBMS) (not shown).

  The database 20 stores an analysis target data table 220 that stores a table of analysis target data, a schema information table 210 that stores schema information of the analysis target data, and a similarity calculation result between data items of the analysis target data. The inter-data relation calculation result 230, the independent data item table 240 for storing the data item name of the data table, the similarity table between the data items, the summary table 250 for storing the regression equation and the group, and the result of the data trend analysis. A result table 260 to be stored, a representative data table 270 that stores selected representative data, a restoration table 280 that stores a minimum path (to be described later) for selecting a representative data item, and a final calculation result are stored. A final result table 290 is included.

  The analysis target data table 220 can be composed of a plurality of data tables. The synonym dictionary 22 is a dictionary set in advance for name identification of data item names in the data table.

  The narrowing-down and restoration server 3 is based on the inter-data relationship analysis unit 310 that calculates the similarity between the data items and the similarity between the data corresponding to the data items, and the similarity between the data item names and the data similarity A group generation unit 320 that generates a group of data items, an input representative data selection unit 330 that selects a representative data item (representative data item) in each group, and a data item name included in the group from the representative data item A restoration formula generation unit 340 for generating a restoration formula (regression formula) to be performed, a data restoration unit 350 for restoring a related data item (group) based on the restoration formula, and a candidate table 360 for selecting representative data items Have.

  The data trend analysis server 4 has a data trend analysis unit 41. The data trend analysis unit 41 is the same as the conventional example, and performs a data trend analysis by applying a known or well-known technique such as correlation analysis or machine learning. In this embodiment, the trend analysis is executed using the representative data corresponding to the representative data item selected by the narrowing-down and restoration server 3.

  The user terminal 6 is a computer connected to the network 5 via a network interface (I / F in the figure) 61 and includes an input / output unit 62 that gives instructions and responses to the narrowing and restoration server 3. The input / output unit 62 includes an input device such as a keyboard, a mouse, and a touch panel, and an output device such as a display.

  The user terminal 6 performs data trend analysis using the data trend analysis system 1 including the database server 2, the narrowing and restoring server 3, and the data trend analysis server 4.

  In the present embodiment, an example is shown in which the database server 2, the narrowing / restoring server 3, and the data trend analysis server 4 are realized by different computers, but the present invention is not limited to this. For example, the functions of the database server 2, the narrowing and restoring server 3, and the data trend analysis server 4 may be provided by a single computer, or each server may be operated by a virtual computer. In the present embodiment, an example in which the data table to be analyzed is held in the data trend analysis system 1 is shown, but the present invention is not limited to this, and it may be read from an external device.

  FIG. 2 is a block diagram illustrating an example of the narrowing and restoration server 3. The narrowing and restoring server 3 is a computer including a processor 31, a memory 32, a storage 33, and a network interface 34.

  The memory 32 is loaded with an inter-data relation analysis unit 310, a group generation unit 320, an input representative data selection unit 330, a restoration formula generation unit 340, and a data restoration unit 350, and is executed by the processor 31.

  The functional units of the inter-data relation analysis unit 310, the group generation unit 320, the input representative data selection unit 330, the restoration formula generation unit 340, and the data restoration unit 350 are loaded into the memory 202 as programs.

  The processor 31 operates as a functional unit that provides a predetermined function by processing according to a program of each functional unit. For example, the processor 31 functions as the data relation analysis unit 310 by performing processing according to the data relation analysis program. The same applies to other programs. Furthermore, the processor 31 also operates as a function unit that provides each function of a plurality of processes executed by each program. A computer and a computer system are an apparatus and a system including these functional units.

  Information such as programs and tables for realizing the functions of the narrowing and restoration server 3 includes storage 33, nonvolatile semiconductor memory, hard disk drive, storage device such as SSD (Solid State Drive), IC card, SD card, DVD Etc., and can be stored in a computer readable non-transitory data storage medium.

<Outline of processing>
FIG. 3 is a flowchart illustrating an example of processing performed in the data trend analysis system 1. This processing is started when the narrowing-down and restoration server 3 receives an instruction for data trend analysis from the user terminal 6. The data analysis instruction can include schema information of the data table included in the analysis target data table 220 and designation of the analysis target data table and analysis target data.

  In the present embodiment, an example of analyzing a tendency of a company having a history of over 100 years as analysis target data will be described, and a case where financial statements, management information, etc. are designated from the schema information table 210 and the table name 212 will be described. The following description shows an outline of the processing, and details of the processing will be described later.

  In step S <b> 1, the narrowing and restoration server 3 groups data items (input data items) for the data table specified by the user terminal 6. Grouping is performed in the following procedure.

  First, the data relation analysis unit 310 reads the data items of the data table designated as the analysis target from the schema information table 210 or the analysis target data table 220, and generates the independent data item table 240. In the schema information table 210, schema information of tables stored in the analysis target data table 220 of the database server 2 is registered in advance. The inter-data relationship analysis unit 310 may acquire data items from the analysis target data table 220 of the designated data table.

  Next, the inter-data relationship analysis unit 310 reads the independent data item table 240 and the analysis target data table 220, and compares the item name (natural language) similarity between the data items and the data (numerical value) corresponding to the data item. ) And a variance value are calculated, and an inter-data relation calculation result 230 is generated.

  Then, the data relation analysis unit 310 extracts a pair of data items based on the similarity between the item names and the similarity between the data, and generates a summary table 250. Next, the restoration formula generation unit 340 generates a bidirectional restoration formula (regression formula) for the data item pair in the summary table 250 and stores it in the summary table 250. Further, the group generation unit 320 updates the summary table 250 by assigning a group to the data item pair.

  Next, in step S <b> 2, the input representative data selection unit 330 selects a data item representing the group from each group of the summary table 250 as a representative data item.

  Next, in step S3, the narrowing down and restoration server 3 notifies the data trend analysis server 4 of the selected representative data item and the data of the representative data item, and causes the data trend analysis to be executed. In the present embodiment, an example is shown in which the input representative data selection unit 330 notifies the data trend analysis server 4 of the data (data table, data item, etc.) of the representative data item. Also good.

  The data trend analysis server 4 reads the data of the representative data item notified from the narrowing and restoration server 3 from the database server 2 and executes a predetermined trend analysis (trend estimation). When the trend analysis is completed, the data trend analysis server 4 narrows down the result of the trend analysis and responds to the restoration server 3.

  In step S4, in the refinement and restoration server 3 that has received the result of the trend analysis, the data restoration unit 350 restores data items other than the representative data item based on the regression equation. Then, the data restoration unit 350 stores the restored data item related to the representative data item and the result of the trend analysis in the final result table 290. The data restoration unit 350 outputs the contents of the final result table 290 to the user terminal 6.

  Through the processing as described above, data items that are important for the analyst using the user terminal 6 can be accurately restored without depending on the subjective interpretation of the analyst, and appropriate analysis results can be obtained. Is possible.

  That is, in the data trend analysis system 1 of the present embodiment, for the related data items, the names of the data items are compared with both the natural language similarity and the numerical similarity of the data corresponding to the data item. Group data items to be narrowed down.

  Next, the data trend analysis system 1 selects a representative data item whose restoration accuracy by the restoration formula is high from the grouped data items. Then, the data trend analysis system 1 performs data trend analysis on the data of the selected data item. Thereby, the result of the trend analysis equivalent to the case where the trend analysis is performed on all the analysis target data can be obtained with a small amount of data corresponding to the narrowed data items.

  Finally, the data trend analysis system 1 restores the data items in the group of representative data items, and outputs them to the user terminal 6 together with the data trend analysis results, thereby allowing the analyst to narrow down the grounds and overall Can be presented.

  Note that the trend analysis (trend estimation) executed by the data trend analysis server 4 is not described in detail in this embodiment because a known or publicly known technique may be applied.

<Table configuration>
FIG. 4 is a diagram illustrating an example of the schema information table 210. In the schema information table 210, schema information of the data table stored in the analysis target data table 220 is set in advance.

  The schema information table 210 includes an identification number (# in the figure) 211, a table name 212, and an item name 213 in one entry. In the illustrated example, the example of two items is shown, but other items may be included.

  In the item name 213, a data item (field name) of an analysis target data table 220 described later is stored in a natural language.

  FIG. 5 is a diagram illustrating an example of the analysis target data table 220. The analysis target data table 220 is a table of analysis target data stored in advance in the database server 2. The analysis target data table 220 includes a plurality of data tables, and the illustrated example shows an example of a financial statement data table.

  The analysis target data table 220 includes a company code 221, sales 222, cost of sales 223, operating profit 224, current assets 225, fixed assets 226, current liabilities 227, fixed liabilities 228, and capital 229. And in one entry.

  The analysis target data table 220 has different data items for each data table, but numerical values are stored in the data of each field.

  FIG. 6 is a diagram illustrating an example of the inter-data relation calculation result 230. The inter-data relation calculation result 230 is a table generated by the inter-data relation analysis unit 310.

  The inter-data relation calculation result 230 includes an identification number (# in the figure) 231, a data item (x) 232, a data item (y) 233, an item name similarity 234, a variance difference 235, and an inter-data The similarity 236 is included in one entry.

  In the data item (x) 232 and the data item (y) 233, the value of the item name 242 of the independent data item table 240 (described later) as a pair is stored. The item name similarity 234 stores the item name similarity of the data item (x) 232 and the data item (y) 233 in the natural language.

  The variance difference 235 stores the difference between the data (numerical value) of the data item (x) 232 and the variance value of the data (numeric value) of the data item (y) 233. In the inter-data similarity 236, the similarity between the data (numerical value) of the data item (x) 232 and the data (numerical value) of the data item (y) 233 is stored.

  FIG. 7 is a diagram illustrating an example of the independent data item table 240. The independent data item table 240 is generated by the data relation analysis unit 310. The independent data item table 240 includes an identification number (# in the figure) 241 and an item name 242 in one entry.

  In the item name 242, the data items of the analysis target data table 220 are stored in the natural language as described above.

  FIG. 9 is a diagram illustrating an example of the summary table 250. The summary table 250 is updated by the group generation unit 320.

  The summary table 250 includes an identification number (# in the figure) 251, a data item (x) 252, a data item (y) 253, an inter-data similarity 254, a regression equation (reconstruction of x → y) 255, The regression equation (reconstruction of y → x) 256 and group # 257 are included in one entry.

  The data item (x) 252 and the data item (y) 253 store the contents of the data item (x) 232 and the data item (y) 233 of the inter-data relation calculation result 230. The inter-data similarity 254 stores the value of the inter-data similarity 236 of the inter-data relation calculation result 230.

  The regression equation (reconstruction of x → y) 255 stores a regression equation for restoring the data item (y) 253 from the data item (x) 252 data. The regression equation (reconstruction of y → x) 256 stores a regression equation for restoring the data item (x) 252 from the data item (y) 253 data.

  Group # 257 stores an identifier (group number) of a group to which a pair of data item (x) 252 and data item (y) 253 belongs.

  FIG. 13 is a diagram showing a state in which a group number is assigned to group # 257 of summary table 250 in FIG.

  FIG. 19 is a diagram illustrating an example of the representative data table 270. The representative data table 270 is generated by the input representative data selection unit 330. The representative data table 270 includes an identification number (# in the figure) 271 and an item name 272 in one entry.

  FIG. 20 is a diagram illustrating an example of the restoration table 280. The restoration table 280 is generated by the input representative data selection unit 330.

  The restoration table 280 includes an identification number (# in the figure) 281, a data item (x) 282, a data item (y) 283, and a regression equation (x → y restoration) 284 in one entry.

  The data item (x) 282 and the data item (y) 283 store the contents of the data item (x) 252 and the data item (y) 253 of the summary table 250. The regression equation (reconstruction of x → y) 284 stores the value of the regression equation (reconstruction of x → y) 255 of the summary table 250.

  FIG. 22 is a diagram illustrating an example of a result table 260 of data trend analysis. The data trend analysis result table 260 is generated by the data restoration unit 350.

  The data trend analysis result table 260 includes an identification number (# in the figure) 261, condition 1 (262), and condition 2 (263) in one entry. In this example, as a result of trend analysis of companies over 100 years old, it is highly likely that companies that satisfy both Condition 1 (262) and Condition 2 (263) can continue for more than 100 years since their founding. Indicates.

  FIG. 26 is a diagram illustrating an example of the candidate table 360. The candidate table 360 is a table managed by the input representative data selection unit 330.

  The candidate table 360 includes an identification number 361, an item name 362, and a variance difference 363 in one entry. The identification number 361 stores a value assigned by the input representative data selection unit 330. In the item name 362 and the variance difference 363, the representative data item candidate value selected by the input representative data selection unit 330 is stored.

<Details of processing>
FIG. 8 is a flowchart illustrating an example of data item grouping processing performed in the data trend analysis system 1. This process is performed in step S1 of FIG.

  The narrowing-down and restoration server 3 of the data trend analysis system 1 receives information on the data tables (the schema information table 210 and the analysis target data table 220) for performing the trend analysis from the user terminal 6. In this embodiment, as shown above, an example is shown in which a data table storing only the data of a company that has been continued for more than 100 years since its establishment is used.

  First, in step S11, the data relation analysis unit 310 reads the data items of the data table designated as the analysis target from the schema information table 210 or the analysis target data table 220, and generates the independent data item table 240.

  In the independent data item table 240, all data items of one or more data tables designated as analysis targets are stored in a natural language. Note that in the analysis target data table 220 of this embodiment, the names of the data items are described in natural language.

  In step S12, the data relation analysis unit 310 repeatedly executes the processing up to step S21 for all combinations of the item names 242 in the independent data item table 240. The data relation analysis unit 310 selects the two item names 242 from the independent data item table 240 and then executes the following processing.

  In step S <b> 13, the data relation analysis unit 310 calculates the similarity in natural language for the combination of the two selected item names 242. It should be noted that a known or publicly known technique can be applied to calculate the similarity between the two item names 242. For example, the similarity can be calculated using an algorithm called “Gestalt pattern matching” (https://docs.python.jp/3/library/difflib.html) by Ratcliff and Obershelp.

  In step S14, the data relation analysis unit 310 compares the calculated similarity with a preset threshold Th1, and determines whether or not the similarity is equal to or greater than the threshold Th1. If the similarity is greater than or equal to the threshold Th1, the process proceeds to step S15. If the similarity is less than the threshold Th1, the process proceeds to step S21, and the next combination of item names 242 is selected and the above process is repeated.

  In step S15, the data relation analysis unit 310 acquires data (numerical value) of the data item corresponding to the two selected item names 242 from the analysis target data table 220, and calculates the difference between the variance values of the data combination. calculate. The following equation (1) can be used to calculate the difference between the variance values.

  However, “A” indicates each data value of the data item (x), “B” indicates each data value of the data item (y), and “Aavg” indicates the addition average value of the data item (x). , “Bavg” indicates the addition average value of the data item (y), and “n” indicates the number of data. The data item (x) and the data item (y) indicate a combined data item.

  In step S16, the inter-data relationship analysis unit 310 compares the calculated variance value difference with a preset threshold value Ths, and determines whether or not the variance value difference is equal to or less than the threshold value Ths. If the variance value difference is equal to or smaller than the threshold value Ths, the process proceeds to step S17. If the variance value difference is higher than the threshold value Ths, the process proceeds to step S21, and the combination of the next item name 242 is selected and the above process is repeated.

  In step S <b> 17, the data relation analysis unit 310 acquires data items (numerical values) corresponding to the two selected item names 242 from the analysis target data table 220, and calculates the similarity between the data. As for the degree of similarity between data, “Gestalt pattern matching” can be used as in step S13.

  In step S18, the data relationship analysis unit 310 compares the calculated similarity with a preset threshold Th2, and determines whether the similarity is equal to or greater than the threshold Th2. If the similarity is greater than or equal to the threshold Th2, the process proceeds to step S19. If the similarity is less than the threshold Th1, the process proceeds to step S21, and the next combination of item names 242 is selected and the above process is repeated.

  In step S <b> 19, the data relation analysis unit 310 deletes two records including the currently selected item name 242 from the independent data item table 240.

  In step S20, the data relation analysis unit 310 stores the two currently selected item names 242 as a pair in the summary table 250 as described later. Further, the inter-data relation analysis unit 310 stores the two currently selected item names 242 in the data item (x) 232 and the data item (y) 233 of the inter-data relation calculation result 230, and stores the item name 242. The similarity is stored in the item name similarity 234, the variance difference is stored in the variance difference 235, the numerical similarity between the data is stored in the inter-data similarity 236, and an identification number 231 is assigned.

  In step S21, the inter-data relationship analysis unit 310 selects the next item name 242 pair and repeats the above processing. When all the combinations of the item names 242 in the independent data item table 240 are completed, the processing ends. .

  As a result of the above processing, a pair of item names 242 in which the similarity of the item names of the data items is greater than or equal to the threshold Th1, the difference in the variance between the data is less than or equal to the threshold Ths, and the similarity of the numerical values of the data is greater than or equal to the threshold Th2 Stored. Then, the item name 242 having no similar pair remains in the independent data item table 240.

  In the present embodiment, when the inter-data relation analysis unit 310 selects related data items, the names of the natural language data items and the data (numerical data) corresponding to the data items are compared and similar. Narrow down the data items to be processed. As a result, data items having similar names and similar values can be grouped as described later.

  In the above description, an example is shown in which the presence / absence of a data item name is determined based on the similarity in natural language. However, the present invention is not limited to this, and the synonym dictionary 22 is used to determine similarity in natural language. You may do it.

  Further, in this embodiment, only for data items having a similar natural language, it is possible to exclude data items having a similar natural language from related items by performing a similar determination on data (numerical values) corresponding to the data items. it can.

  10 and 11 are flowcharts showing an example of storage processing in the summary table 250. FIG. This process is executed in step S20 of FIG.

  In step S31 of FIG. 10, the restoration formula generation unit 340 stores the two item names 242 currently selected in the processing of FIG. 8 in the data item (x) 252 and the data item (y) 253 of the summary table 250. Then, the similarity between the data calculated in step S17 is stored in the data similarity 254, and an identification number 251 is assigned.

  The restoration formula generation unit 340 may acquire data corresponding to the two currently selected item names from the inter-data relation calculation result 230 and store the data in the summary table 250.

  In step S <b> 32, the restoration formula generation unit 340 calculates a regression formula (x → y) for restoring the data item (y) 253 from the data item (x) 252. Note that a known or publicly known technique may be applied for calculating the regression equation.

  For example, as shown in FIG. 12, when the data item (x) is the total asset for the previous period and the inter-data relation calculation result 230 (y) is the total asset, two amounts (total asset for the previous period, total asset) ) On the coordinate plane, a straight line approximating the distribution of data points can be represented by a regression equation.

  In step S <b> 33, the restoration formula generation unit 340 calculates a regression formula (y → x) for restoring the data item (x) 252 from the data of the data item (y) 253.

  In step S34, the restoration formula generation unit 340 uses the regression formula (x → y) for restoring the data item (y) 253 from the data of the data item (x) 252 as the regression formula of the summary table 250 (x → y restoration). The regression formula (y → x) for restoring the data item (x) 252 from the data of the data item (y) 253 is stored in the regression formula (reconstruction of y → x) 256 of the summary table 250.

  Next, in step S35 of FIG. 11, the group generation unit 320 repeatedly executes the processing up to step S44 for all of the data items (x) 252 and the data items (y) 253 of the summary table 250.

  In step S <b> 36, the group generation unit 320 selects the data item (x) 252 and the data item (y) 253 from the top entry of the summary table 250. Then, the group generation unit 320 compares the data item (x) 252 and the data item (y) 253 in the column direction with respect to the assignment status of the group # 257 in order from the top entry of the summary table 250.

  In step S37, the group generation unit 320 matches the group # 257 of another entry that matches the data item (x) 252 of the currently selected entry and the data item (y) 253 of the currently selected entry. It is determined whether the group # 257 of the entry is a different group.

  When the group # 257 of the data item (x) 252 and the group # 257 of the data item (y) 253 are different, the group generation unit 320 proceeds to step S38, and otherwise proceeds to step S39.

  In other words, the group generation unit 320 searches for an entry whose data item matches the data item (x) 252 of the currently selected entry in the column direction of the summary table 250 in FIG. If 257 is given, the group number is set in the variable Nx.

  Similarly, the group generation unit 320 searches in the column direction of the summary table 250 for an entry whose data item matches the data item (y) 253 of the currently selected entry, and the group # 257 is assigned to the matching entry. If so, the group number is set in the variable Ny.

  Then, the group generation unit 320 determines whether or not the variable Nx and the variable Ny match. If they do not match, the group generation unit 320 proceeds to step S38, otherwise proceeds to step S39.

  In step S38, the group generation unit 320 determines that the data item (x) 252 that is currently selected matches the group # 257 of the data item that matches the data item, and the data item (y) 253 that is currently selected matches the data item. The smaller one of the group numbers of the group # 257 is obtained, and the group # 257 of the entry whose data item matches the currently selected entry is aligned. Thereafter, the process proceeds to step S44.

  In step S39, the group generation unit 320 selects either the entry whose data item matches the currently selected data item (x) 252 or the entry whose data item matches the currently selected data item (y) 253. It is determined whether or not group # 257 is set.

  The group generation unit 320 proceeds to step S40 if the group # 257 is set in either one, and proceeds to step S41 otherwise.

  In step S40, the group generation unit 320 selects either the entry whose data item matches the currently selected data item (x) 252 or the entry whose data item matches the currently selected data item (y) 253. The value in which group # 257 is set is set in group # 257 of the current entry. Thereafter, the process proceeds to step S44.

  In step S41, the group generation unit 320 includes the entry ## 257 of the entry whose data item matches the currently selected data item (x) 252 and the data item (y) 253 whose data item matches. It is determined whether or not the group # 257 matches. The group generation unit 320 proceeds to step S42 if both groups # 257 match, and proceeds to step S43 if both groups # 257 are not set.

  In step S42, the group generation unit 320 sets the value of the group # 257 of the entry whose data item matches the currently selected data item (x) 252 to the group # 257 of the current entry. Thereafter, the process proceeds to step S44.

  In step S44, the data item (x) 252 and the data item (y) 253 of the next entry in the summary table 250 are selected, and then the process returns to step S36 to repeat the above processing.

  Through the above processing, the summary table 250 shown in FIG. 9 is given the group # 257 as shown in FIG.

  This will be described using a specific example. When the group generation unit 320 selects the first entry in the summary table 250 of FIG. 9, the data item (x) 252 = the group # 257 of the entry of the data item that matches “total assets in the previous period”, and the data item (y) 253 = No value is set in the group # 257 of the entry of the data item that matches “total liabilities / net assets”. Therefore, the group generation unit 320 proceeds from step S41 to step S43, and sets a new group number = 1 to the group # 257.

  The same applies to the second entry of the summary table 250. A new group number = 2 is set in the group # 257, and is updated as in the summary table 250 of FIG.

  In the third entry of the summary table 250, the entry whose data item matches the data item (x) 252 = “total assets in the previous period” is the identification number 251 = 4 and the group # 257 = 1. Further, in the third entry of the summary table 250, the data item (y) 253 = “total debt / net assets” and the data item that matches the data item have the identification number 251 = 1 and the group # 257 = 1.

  Therefore, in the third entry of the summary table 250, group # 257 = 1 is set by steps S41 and S42.

  With the above processing, as shown in FIG. 14, the group # 257 is set from the chain of the item names of the data item (x) 252 and the data item (y) 253 of the summary table 250. Note that the group # 257 of the entry having no data item that matches the data item (x) 252 or the data item (y) 253 is blank.

  As described above, the relation information calculated by the inter-data relation analysis unit 310 (difference between the pair of data relation calculation results 230 (data items (x), (y)) and the similarity and the variance value) and the analysis target data table From the data 220, the restoration formula generation unit 340 generates restoration information (restoration formula). Then, the group generation unit 320 groups the item names of the data items based on the related information.

  FIG. 15 is a flowchart illustrating an example of representative data selection processing. This process is performed in step S2 of FIG.

  In step S51, the input representative data selection unit 330 repeats the processing up to step S63 for each group # 257 in the summary table 250, and selects a data item representing each group # 257 as a representative data item from the candidate table 360. To do. First, the input representative data selection unit 330 acquires the group number of the group # 257.

  In step S52, the process up to step S60 is repeated for all entries of the summary table 250 belonging to the acquired group number, and the one with the smaller difference between the input data and the restored data is stored in the candidate table 360.

  In step S53, the input representative data selection unit 330 selects the first entry from the summary table 250 belonging to the acquired group number, and all the data corresponding to the data item (x) 252 are regressed (x → y (Restoration) 255 is applied, and the value of restored data = y is calculated.

  In step S54, the input representative data selection unit 330 calculates the difference diff by the following equation (2) using all the data corresponding to the data item (y) 253 as the input data for the entry selected in step S53. (2).

  However, “A” indicates each data value of the input data, “B” indicates each data value of the restored data, and “n” indicates the number of data.

  In step S55, the input representative data selection unit 330 applies the regression equation (restoration of y → x) 256 to all the data corresponding to the data item (y) 253 for the entry selected in step S53. Value = x is calculated.

  In step S56, the input representative data selection unit 330 calculates the difference diff by the above equation (2) using all data corresponding to the data item (x) 252 as input data for the entry selected in step S53. (4).

  In step S57, the input representative data selection unit 330 determines whether or not the calculated difference (2) exceeds the difference (4). The input representative data selection unit 330 proceeds to step S58 if the difference (2) exceeds the difference (4), and proceeds to step S59 if the difference (2) is equal to or less than the difference (4).

  In step S <b> 58, the input representative data selection unit 330 adds the item name of the data item (y) 253 and the difference (2) to the candidate table 360. Further, the input representative data selection unit 330 adds entries of the data item (x) 252, the data item (y) 253, and the regression equation (reconstruction of x → y) 255 to the restoration table 280.

  On the other hand, in step S <b> 59, the input representative data selection unit 330 adds the item name of the data item (x) 252 and the difference (4) to the candidate table 360.

  Thereafter, the process proceeds to step S60, and the input representative data selection unit 330 selects the next entry in the group number and repeats the above process. The input representative data selection unit 330 performs the above processing for all entries belonging to the group number, and then proceeds to step S61.

  In step S <b> 61, the input representative data selection unit 330 selects an entry having the largest difference 363 from the candidate table 360 and adds the item name 362 of the entry to the representative data table 270.

  In step S62, the input representative data selection unit 330 clears (initializes) the candidate table 360. Thereafter, the process proceeds to step S63, where the input representative data selection unit 330 selects the next group number in the summary table 250 and repeats the above processing. The input representative data selection unit 330 ends the above process for all group numbers in the summary table 250.

  By the above process, the original data having the smaller difference diff between the input data and the restored data can be set as the representative data item. As a result, among the data items collected in one group, representative data that can be restored as accurately as possible when restoring data items other than the representative data item within the same group from the output results of data trend analysis. The item is selected. The representative data item is used as a main data item to be input to the data trend analysis server 4.

  As described above, the input representative data selection unit 330 selects representative data items that can ensure the restoration accuracy based on the related information generated by the inter-data relationship analysis unit 310 and the restoration information generated by the restoration formula generation unit 340. Select for each group. Note that the input representative data selection unit 330 applies the restoration information to the data of the data item, compares the difference of the variance values of the restoration results with the pair of data items, and selects the one with the smaller variance value as the representative data item. It is determined that the restoration accuracy can be secured.

  FIG. 16 is a diagram illustrating an example of node division in the representative data selection process. The selection of the representative data item is not limited to the difference shown in FIG. 16, but can be performed by dividing the node. An example of node division as the second selection process of representative data will be described.

  In the following description, nodes, edges, and topologies are defined as follows. The node indicates the data item (x) 252 and the data item (y) 253 of the summary table 250. The edge indicates the relationship between the data item (x) 252 and the data item (y) 253 and is represented by an inter-data similarity 254. The topology is represented by a group number of group # 257.

  In step S <b> 71, the input representative data selection unit 330 repeats the processing up to step S <b> 79 for each group # 257 of the summary table 250, and stores data items representing each group # 257 in the representative data table 270. The input representative data selection unit 330 acquires the group number of group # 257.

  In step S72, the input representative data selection unit 330 calculates the minimum distance between the nodes (entries) belonging to the group number acquired in step S71. In this embodiment, the distance between nodes is represented by the inter-data similarity 254.

  In step S73, the input representative data selection unit 330 sets the path (data item (x) 252 and data item (y) 253) having the smallest inter-node distance calculated in step S72 as the minimum path, and sets the minimum path and regression equation. (Restore x → y) 255 is stored in the restore table 280.

  In step S74, it is determined whether or not the distance from the target node is equal to or greater than a predetermined threshold Th3 for the minimum path. The input representative data selection unit 330. If the distance from the target node is equal to or greater than the predetermined threshold Th3, the process proceeds to step S75. If the distance from the target node is less than the predetermined threshold Th3, the process proceeds to step S76.

  In step S75, the input representative data selection unit 330 divides the node into two at the edge where the distance from the target node is equal to or greater than the predetermined threshold Th3. And it returns to step S72 and repeats the said process.

  In step S76, the input representative data selection unit 330 calculates the sum (multiplication) of the distances between the nodes, and calculates the node having the minimum distance (multiplication) as the main node (primary node). Note that the minimum value of the sum (multiplication) of distances is calculated by the following equation (3).

  However, S shows a group and x shows a node.

  Next, in step S77, the input representative data selection unit 330 selects a node having the smallest total distance (multiplication) as the main node, and proceeds to step S78 to display the main node data item as the representative data item. Add to table 270.

  In step S79, the input representative data selection unit 330 selects the next group number and repeats the above process. When the selection of representative data items for all group numbers is completed, the process ends.

  By the above processing, the distance between all other nodes (similarity between data) is calculated from the target nodes in the group, and one of the nodes having the smallest total (multiplication) distance between nodes is set as the main node. . Then, the input representative data selection unit 330 determines that a range in which the multiplication value of the distance between the main node and other nodes (similarity between data) is greater than or equal to the threshold Th3 is one group. On the other hand, when the input representative data selection unit 330 is less than the threshold Th3, the input representative data selection unit 330 cuts with an edge (branch) that is lower than the threshold. If there are two nodes with the smallest total distance, they may be selected randomly.

  As described above, when the distance between the nodes is equal to or greater than the threshold Th3, the restoration accuracy decreases due to the restoration chain. Therefore, by selecting a representative data item that can ensure restoration accuracy by dividing the node (topology) can do.

  FIG. 17 is a diagram illustrating an example of node division in the representative data selection process. 17 shows an example of calculating representative data items from the similarity of data between nodes shown in FIG.

  The input representative data selection unit 330 calculates a multiplication value of the similarity (distance) between the main node and the target node (Step 1). Then, the input representative data selection unit 330 compares the distance from the main node (similarity multiplication value) and the threshold (0.7), cuts at an edge less than the threshold, and divides the topology into two (Step 2). .

  By repeating the above processing, a representative data item (main node) can be selected from the entries in the group number.

  FIG. 18 is a diagram illustrating another example of node division in the representative data selection process. An example in which a representative data item is calculated from the similarity between nodes illustrated in FIG. In this example, the node arrangement is different from the linear arrangement shown in FIG.

  In this case as well, the similarity between data with adjacent nodes is calculated in the same manner as in FIG. 17, and the similarity between data is also calculated between the diagonal nodes by multiplication (Step 1). Then, the input representative data selection unit 330 compares the distance from the main node (similarity multiplication value) and the threshold (0.7), cuts at an edge less than the threshold, and divides the topology into two (Step 2). .

  FIG. 21 is a flowchart illustrating an example of the data trend analysis process. This process is a process performed in step S3 of FIG.

  The input representative data selection unit 330 inputs the items of the independent data item table 240 and the representative data table 270 and the data (value) corresponding to the items to the data trend analysis server 4 and instructs the trend analysis (S81).

  In the data trend analysis server 4, the data trend analysis unit 41 performs trend analysis on the items and values received from the narrowing and restoration server 3, and responds to the narrowing and restoration server 3 with the result of the trend analysis.

  Through the above processing, the trend analysis is performed on the representative data items of group # 257 and the data of the independent data item table 240 that cannot be grouped.

  23 and 24 are flowcharts illustrating an example of the data restoration process. This process is a process performed in step S4 of FIG. 3, and restores data other than the representative data using the restoration table 280.

  The narrowing-down and restoration server 3 starts the flowchart of FIG. 23 after receiving the result of the trend analysis from the data trend analysis server 4. When the narrowing-down and restoration server 3 receives the result of the trend analysis from the data trend analysis server 4, the representative data selection unit 330 stores the result in the data trend analysis result table 260.

  In step S86 of FIG. 23, the data restoration unit 350 repeats the process up to step S88 for all the condition items in the data trend analysis result table 260. In step S87, the data restoration unit 350 acquires the data items of the condition items (262, 263) of the data trend analysis result table 260 as input items.

  In step S87, the data restoration unit 350 executes the restoration calculation process shown in FIG. 24 with reference to the restoration table 280 for the acquired input items.

  In step S88, the data restoration unit 350 selects the next condition from the result table 260 and repeats the above process. When the restoration calculation process is completed for all conditions, the process ends.

  In step S91 of FIG. 24, the data restoration unit 350 compares the input item acquired in step S86 with the data item (x) 282 of the restoration table 280, and searches for a matching item name.

  In step S92, the data restoration unit 350 determines whether there is a data item (x) 282 that matches the input item. If there is a data item (x) 282 that matches the input item, the process proceeds to step S93. If not, the process ends and the process proceeds to the next conditional expression.

  In step S93, the data restoration unit 350 calculates the data item (y) and the value = y from the data corresponding to the data item (x) 282 by using the regression equation (x → y restoration) 284 of the restoration table 280. To do. This process is executed for all the data items (x) 282 that match the input items.

  In step S94, the process up to step S97 is repeatedly executed for the data item (y) restored in step S93. The data restoration unit 350 selects one restored data item (y).

  In step S95, the data restoring unit 350 adds the selected data item (y) to the final result table 290. The data restoration unit 350 adds the data item (y) to the condition (292, 293) and the identification number 291 of the final result table 290 corresponding to the condition of the currently selected result table 260 and the identification number 261.

  In step S96, the data restoration unit 350 obtains the data corresponding to the restored data item (y) from the analysis target data table 220 and performs a predetermined calculation, and the result of the condition (in the final result table 290 in step S95) ( 292, 293) and the identification number 291. In the example of the final result table 290 in FIG. 25, an example in which the range of values corresponding to the data item (y) is calculated is shown.

  In step S97, the data restoration unit 350 selects the next data item (y) and repeats the above process. When the restoration calculation process is completed for all data items (y), the process ends, and the process of FIG. Return to.

  By the above processing, the data item (y) belonging to the same group and the value range are restored from the representative data item that matches the data item of the condition (262, 263) in the result table 260 of the data trend analysis.

  When the above processing ends, the data restoration unit 350 outputs the display screen 620 of the final result table 290 to the user terminal 6. The user (analyzer) of the user terminal 6 includes the data items important to the analyst by restoring the input data items and values collected in the representative data items after the result of the data trend analysis. It can be easily determined whether or not.

  In the display screen 620 of the final result table 290 in FIG. 25, the restoration result of the data item related to the representative data item (item name and data (range) in relation to the result of the data trend analysis by the representative data item shown in FIG. ) Is added to condition 1 (292)).

  In the illustrated example, the data items of “total assets” and “total liabilities / net assets” belonging to the same group # 257 are restored for the representative data item = “total assets in the previous period”, and the restored data items Data (numerical value) is also restored.

  The analyst using the user terminal 6 refers to the display screen 620 of the final result table 290 to determine unnecessary data items from the grouping restoration result performed by the narrowing and restoration server 3 and the data trend analysis result. Can be selected.

<Summary>
In this embodiment, when selecting related data items, similar data items are narrowed down by comparing both the name of the data item in the natural language and the data (numerical data) corresponding to the data item. As a result, data items having similar names and similar values can be grouped. In addition, the data item excluded from the representative data item data can be restored, and the restoration result can be presented to the analyst.

  The narrowing-down and restoration server 3 can secure the accuracy of trend analysis while reducing the processing load of data trend analysis by narrowing down the data items to be input to the data trend analysis server 4 to the representative data items of each group. it can. In other words, in this embodiment, it is possible to obtain the same result as inputting all data items while reducing the time required for the data trend analysis process by narrowing down to representative data items.

  Then, when selecting the representative data item, the narrowing and restoring server 3 selects a data item that can ensure the accuracy of restoring the data item excluded from the data of the representative data item as the representative data item. Thereby, after obtaining the trend analysis result of the data, the relationship between the representative data item and the excluded data item can be provided. An analyst using the user terminal 6 can know what data items are collected, and can confirm whether data items important to the analyst are obtained as a result of trend analysis. . In this embodiment, an analyst using the user terminal 6 can select unnecessary data items in the post-processing of trend analysis.

  In addition, this invention is not limited to an above-described Example, Various modifications are included. For example, the above-described embodiments are described in detail for easy understanding of the present invention, and are not necessarily limited to those having all the configurations described. Further, a part of the configuration of one embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of one embodiment. In addition, any of the additions, deletions, or substitutions of other configurations can be applied to a part of the configuration of each embodiment, either alone or in combination.

  Each of the above-described configurations, functions, processing units, processing means, and the like may be realized by hardware by designing a part or all of them with, for example, an integrated circuit. In addition, each of the above-described configurations, functions, and the like may be realized by software by the processor interpreting and executing a program that realizes each function. Information such as programs, tables, and files for realizing each function can be stored in a memory, a hard disk, a recording device such as an SSD (Solid State Drive), or a recording medium such as an IC card, an SD card, or a DVD.

  Further, the control lines and information lines indicate what is considered necessary for the explanation, and not all the control lines and information lines on the product are necessarily shown. Actually, it may be considered that almost all the components are connected to each other.

1 Data trend analysis system 2 Database server 3 Refinement and restoration server 4 Data trend analysis server 4
6 User terminals 6-1 to 6-3
20 Database 220 Analysis target data table 240 Independent data item table 260 Data trend analysis result table 250 Summary table 270 Representative data table 280 Restoration table 290 Final result table 310 Inter-data relation analysis unit 320 Group generation unit 330 Input representative data selection unit 340 Restoration Expression Generation Unit 350 Data Restoration Unit

Claims (15)

A data trend analysis method for performing a data trend analysis on a data table having data corresponding to a data item in a computer including a processor and a memory,
A first step in which the computer extracts associations between data items for a plurality of the data tables to generate association information of the data items;
A second step in which the computer generates restoration information of the data item based on the related information;
A third step in which the calculator generates the group of data items based on the related information;
A fourth step in which the computer selects a representative data item from the data items in the group based on the related information and the restoration information;
A fifth step in which the computer performs a trend analysis of data corresponding to the representative data item;
A sixth step in which the computer restores the data item of the group to which the representative data item belongs, based on the restoration information;
A seventh step in which the computer outputs a result of trend analysis of data corresponding to the representative data item, and a restoration result of the data item;
A data trend analysis method characterized by comprising: The data trend analysis method according to claim 1,
The first step includes
Extracting a pair of related data items from the similarity of the item name of the data item in natural language and the similarity of the value of the data corresponding to the data item, and generating the related information for the pair of the data item A data trend analysis method characterized by The data trend analysis method according to claim 2,
The second step includes
A first restoration formula for calculating the second data item from the data of the first data item for each of the first data item and the second data item constituting the data item pair; A data trend analysis method comprising: generating the restoration information including a second restoration formula for calculating the first data item from data of two data items. A data trend analysis method according to claim 3, wherein
The fourth step includes
A data trend analysis method, wherein a data item capable of ensuring restoration accuracy when a data item is restored from the restoration information among the pair of data items is selected for each group as a representative data item. A data trend analysis method according to claim 3, wherein
The fourth step includes
A data trend analysis method, comprising: dividing a group when a restoration accuracy falls below a predetermined threshold when restoring a data item from the restoration information in the pair of data items. The data trend analysis method according to claim 1,
The seventh step includes
A data trend analysis method comprising: outputting a restoration result of the data item in association with the representative data item included in the trend analysis result. A refinement and restoration server including a processor and memory;
A data trend analysis system having a data trend analysis server including a processor and a memory, and performing data trend analysis on a data table having data corresponding to a data item,
The narrowing and restoration server is:
An inter-data relation analysis unit that extracts relations between data items for a plurality of data tables and generates relation information of the data items;
A restoration formula generator for generating restoration information of the data item based on the related information;
A group generation unit for generating a group of the data items based on the related information;
A representative data item selection unit that selects a representative data item from the data items in the group based on the related information and the restoration information, and causes the data trend analysis server to perform a trend analysis of data corresponding to the representative data item. When,
Receiving the result of the trend analysis of the data corresponding to the representative data item from the data trend analysis server, restoring the data item of the group to which the representative data item belongs based on the restoration information, and restoring the data item A data restoration unit for outputting the result and the result of the trend analysis;
A data trend analysis system characterized by comprising: The data trend analysis system according to claim 7,
The inter-data relation analysis unit
Extracting a pair of related data items from the similarity of the item name of the data item in natural language and the similarity of the value of the data corresponding to the data item, and generating the related information for the pair of the data item A data trend analysis system characterized by The data trend analysis system according to claim 8,
The restoration formula generator is
A first restoration formula for calculating the second data item from the data of the first data item for each of the first data item and the second data item constituting the data item pair; 2. A data trend analysis system, comprising: generating restoration information including a second restoration formula for calculating the first data item from data of two data items. The data trend analysis system according to claim 9,
The representative data item selection unit is:
A data trend analysis system, wherein a data item that can ensure restoration accuracy is selected for each group as a representative data item when the data item is restored from the restoration information among the pair of data items. The data trend analysis system according to claim 9,
The representative data item selection unit is:
The data trend analysis system according to claim 1, wherein when a data item is restored from the restoration information among the pair of data items, the restoration of the restoration accuracy falls below a predetermined threshold. The data trend analysis system according to claim 7,
The data restoration unit
A data trend analysis system characterized by outputting the restoration result of the data item in association with the representative data item included in the trend analysis result. A refinement and restoration device including a processor and a memory,
A plurality of data tables having data corresponding to the data items, extracting relationships between the data items for the plurality of data tables, and generating an inter-data relationship analysis unit;
A restoration formula generator for generating restoration information of the data item based on the related information;
A group generation unit for generating a group of the data items based on the related information;
A representative data item selection unit that selects a representative data item from the data items in the group based on the related information and the restoration information, and requests an external trend analysis of data corresponding to the representative data item;
Upon receiving the result of the trend analysis of the data corresponding to the representative data item, the data item of the group to which the representative data item belongs is restored based on the restoration information, and the restoration result of the data item and the trend analysis A data restoration unit for outputting the results;
And a refining and restoring device. The narrowing and restoring device according to claim 13,
The inter-data relation analysis unit
Extracting a pair of related data items from the similarity of the item name of the data item in natural language and the similarity of the value of the data corresponding to the data item, and generating the related information for the pair of the data item A narrowing and restoring device characterized by the above. The narrowing and restoring device according to claim 14,
The restoration formula generator is
A first restoration formula for calculating the second data item from the data of the first data item for each of the first data item and the second data item constituting the data item pair; 2. A narrowing-down / restoring apparatus, comprising: generating restoration information including a second restoration formula for calculating the first data item from data of two data items.