JP6862969B2 - Information processing method, information processing device and information processing program for estimating data type - Google Patents

Description

The present invention relates to an information processing method, an information processing apparatus, and an information processing program for estimating a data type of the set from a set of numerical data.

With the development of data collection technology, data processing technology, and data storage technology, there are increasing opportunities to handle a wide variety of large amounts of information. Most of the data we handle is data collected by others for other purposes or data collected in the past. When collecting data, the meaning and specifications of the data are often not described in the table in which the data is collected or in the data itself. The meaning and specifications of the data are described in the legend of the material that summarizes the table and the specifications of the system. In general, such specification information may not be available, may not be updated to the latest state, or may be lost in some cases.

Generally, when dealing with numerical set data, the type of the data, the source of the data, the unit of the data, and the like are known. Therefore, it is easy to determine how to process the data.

However, in the process from when the data is collected until the person who processes the data obtains the data, some of the information necessary to process the data correctly is missing, and at a glance what the data represents. It may happen that it cannot be determined. For example, if there is no data on the table structure of the database and the column names are abbreviated, it is not possible to determine at a glance what the data represents. In such cases, measures such as analyzing the acquisition route and processing method are taken. However, it may take a long time to clarify the type of data because the contact with the person who collected the data is restricted or specialized knowledge is required.

For example, spreadsheet software table data often contains terms that only the creator and the user can understand. If the people involved at the time of creation are absent over time, there may be no clue as to what the data represents.

In the case of character data, it is possible to infer a certain meaning from the meaning of words and sentences, for example, an address or a name, but in the case of numerical data, it is difficult to make an inference even by looking at each piece of data. Further, as a method of determining the degree of similarity with respect to text data, a method of generating a multidimensional vector for each word and determining the similarity between the data is known. However, for numerical data, it is difficult to apply such a method because the numerical value itself cannot be used as a dimension.

Patent Document 1 describes a method of estimating which type of data is based on statistical values such as mean and variance.

Japanese Unexamined Patent Publication No. 2006-99236

However, in the invention described in Patent Document 1, since only statistical values (mean, variance, etc.) are used, the amount of calculation for obtaining a type that seems to be correct to some extent is large. Further, for example, although it is possible to obtain a type called "length", it is not possible to obtain a type lower than that (detailed type).

An object of the present invention is to make it possible to estimate a more detailed type of a set of numerical data.

In the information processing method according to the present invention, statistical values are calculated from numerical set data for learning to which known data types are associated with each other, additional data added to the numerical set data is extracted, and the statistical values and extraction are performed. A feature quantity vector with the added data as an element is generated, and the feature quantity vector is stored in the storage unit in a state of being layered according to the type of the numerical set data, and the numerical set data for the target to be estimated by the type. The statistical value is calculated from, the additional data added to the numerical set data is extracted, a feature quantity vector having the statistical value and the extracted additional data as elements is generated, and the generated feature quantity vector is generated. The feature is that the distance between the data and the feature amount vector stored in the storage unit is calculated, and the type of numerical set data for the target is estimated based on the calculated distance .

The information processing apparatus according to the present invention is an extraction means for calculating statistical values from learning numerical set data associated with known data types and extracting additional data added to the numerical set data, and statistics. A feature quantity vector generation means that generates a feature quantity vector having a value and extracted additional data as elements, and a feature quantity vector hierarchy that stores the feature quantity vector in a storage unit in a layered state according to the type of numerical set data. of storage means and Bei example and extraction means calculates a statistical value from value set data for target is a type of the estimation target, and extracts the additional data added to said numerical value set data, the feature vector generating The means generates a feature quantity vector having the statistical value and the extracted additional data as elements, and calculates and calculates the distance between the generated feature quantity vector and the feature quantity vector stored in the storage unit. It is further provided with an estimation means for estimating the type of numerical set data for an object based on the distance obtained.

The information processing program according to the present invention is a process of calculating statistical values from learning numerical set data associated with known data types and extracting additional data added to the numerical set data. , The process of generating a feature amount vector with statistical values and extracted additional data as elements, the process of saving the feature amount vector in the storage unit in a layered state according to the type of numerical set data, and the type estimation. A process of calculating a statistical value from the numerical value set data for the target object and extracting the additional data added to the numerical value set data, and a feature amount having the statistical value and the extracted additional data as elements. The process of generating a vector, the process of calculating the distance between the generated feature amount vector and the feature amount vector stored in the storage unit, and the process of estimating the type of numerical set data for the target based on the calculated distance. characterized in that to execute and.

According to the present invention, it is possible to estimate a more detailed type of a set of numerical data.

It is a block diagram which shows the data type estimation apparatus which is an example of the information processing apparatus for estimating a data type. It is explanatory drawing which shows an example of the learning data. It is explanatory drawing which shows an example of the estimation target data. It is a flowchart which shows the processing of a learning phase. It is explanatory drawing for demonstrating the feature quantity vector generated by the feature quantity vector generation part. It is explanatory drawing for demonstrating the hierarchical structure stored in the feature quantity vector layering preservation part. It is a flowchart which shows the processing of the estimation phase. It is explanatory drawing which shows an example of the extracted feature amount vector and the like. It is explanatory drawing which shows the extraction result including a parent type. It is a block diagram which shows the main part of the information processing apparatus by this invention. It is a block diagram which shows the main part of the information processing apparatus of another aspect by this invention.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

FIG. 1 is a block diagram showing a data type estimation device which is an example of an information processing device for estimating a data type (hereinafter, also simply referred to as “type”). FIG. 1 shows a data type estimation device 10, a learning numerical string data storage device 500, and an estimation target numerical string data storage device 600.

The data type estimation device 10 includes a feature amount vector extraction unit 100, a learning unit 200, an estimation unit 300, and a feature amount vector management unit 400.

The feature amount vector extraction unit 100 inputs numerical value set data including a type (specifically, data indicating the type) and additional information, and outputs a feature amount vector. The feature amount vector extraction unit 100 includes a statistical value calculation unit 101, an additional data extraction unit 102, and a feature amount vector generation unit 103.

The statistical value calculation unit 101 calculates statistical values such as the average and variance of the input numerical set data. The additional data extraction unit 102 extracts the unit of the feature amount vector and the characteristics of the information described together. The feature amount vector generation unit 103 combines the calculation result of the statistical value calculation unit 101 and the extraction result of the additional data extraction unit 102 to generate a feature amount vector for the input data.

The learning unit 200 inputs numerical value set data for learning (a set of numerical data), that is, learning data, and generates a feature amount vector. The learning unit 200 includes a learning data input unit 201, a data analysis unit 202, and a feature quantity vector output unit 203.

The learning data input unit 201 inputs learning data from the learning numerical string data storage device 500. The data analysis unit 202 uses the feature amount vector extraction unit 100 to acquire the feature amount vector for the learning data. The feature amount vector output unit 203 outputs the acquired feature amount vector and the type to the feature amount vector management unit 400.

The estimation unit 300 estimates the type of numerical set data. The estimation unit 300 includes an estimation target data input unit 301, a data analysis unit 302, a similar feature amount vector search unit 303, and a result display unit 304.

The estimation target data input unit 301 inputs the numerical set data of the type estimation target from the estimation target numerical string data storage device 600. The data analysis unit 302 uses the feature vector extraction unit 100 to generate a feature vector for the numerical set data of the type estimation target. The similar feature vector search unit 303 uses the feature vector management unit 400 to extract a feature vector having a distance close to the generated feature vector from the feature vector storage unit 403. The result display unit 304 displays the estimation result of the similar feature amount vector search unit 303 on the display unit (not shown). The feature amount vector stored in the feature amount vector storage unit 403 via the learning unit 200 is used as the learned feature amount vector.

The feature vector management unit 400 manages the feature vector. The feature amount vector management unit 400 includes a feature amount vector layered storage unit 401, a feature amount vector comparison unit 402, and a feature amount vector storage unit 403.

The feature vector layered storage unit 401 identifies the layer in which the feature vector should be stored while collating the type input from the feature vector output unit 203 with the stored contents of the feature vector storage unit 403. The feature amount vector is stored in the feature amount vector storage unit 403. The feature vector comparison unit 402 calculates the distance between the feature vector input from the similar feature vector search unit 303 and the learned feature vector stored in the feature vector storage unit 403, and features that are close to each other. The type of the quantity vector is extracted by giving a probability according to the distance. If there is a common type (parent type) in the extracted types, the parent type is also extracted.

At least statistical value calculation unit 101, additional data extraction unit 102, feature amount vector generation unit 103, data analysis unit 202, feature amount vector output unit 203, data analysis unit 302, similar feature amount vector search unit 303, feature amount vector layering. The storage unit 401 and the feature amount vector comparison unit 402 can be realized by one or a plurality of CPUs (Central Processing Units) executing processing based on the program stored in the program storage unit. However, they and other blocks (excluding storage) may be implemented in hardware (individual circuits or LSI (Large Scale Integration)).

Next, the operation of the data type estimation device 10 will be described.

Here, a case where the learning data shown in FIG. 2 is used for learning for estimation of the numerical set data and a case where the estimation target data shown in FIG. 3 is used to estimate the numerical set data are taken as examples.

The learning data illustrated in FIG. 2 is numerical set data regarding systolic blood pressure of an adult male in his twenties. The estimation target data shown in FIG. 3 is numerical set data related to blood pressure, and the data type is unknown.

In addition, the case of using tabular data will be taken as an example. Hereinafter, the numerical set data may be referred to as numerical string data. Input data may be spreadsheet software files, database tables, XML (eXtensible Markup Language) documents, CSV (Common Separated Value) format documents, HTML (Hypertext Markup Language) documents, and so on. However, the format of the input data does not matter as long as it can be decomposed into a set of numerical values and additional data.

In the present embodiment, the data type estimation device 10 executes the processing of the learning phase and the processing of the estimation phase in which the estimation is actually performed. The data type estimation device 10 hierarchically creates a feature vector created by calculating statistical values and extracting additional data for numerical set data whose data type is known in the learning phase, based on the given data type. Convert and save.

FIG. 4 is a flowchart showing the processing of the learning phase. FIG. 4A shows the processing of the learning unit 200. FIG. 4B shows the processing of the feature amount vector extraction unit 100.

In the learning phase, the learning data input unit 201 of the learning unit 200 inputs the learning numerical string data from the learning numerical string data storage device 500 as input data (step S11). The data analysis unit 202 uses the feature amount vector extraction unit 100 to acquire the feature amount vector for the learning data (step S12). The feature amount vector extraction unit 100 executes the process shown in FIG. 4 (B).

The statistical value calculation unit 101 calculates the statistical value of the input numerical string data (step B11). The statistics are, for example, mean, variance, kurtosis, skewness, and distribution type (for example, normal distribution, Poisson distribution, long tail distribution).

The additional data extraction unit 102 extracts the unit of the numerical value (for example, m, g, circle, ° C.) in the numerical string data and the characteristic (additional data) of the information written together in the numerical string data (step B12). .. Further, the additional data extraction unit 102 sets the individual identification information to "Yes" if there is data that can identify an individual such as a name and an ID in the numerical string data. In addition, as additional data, for example, latitude / longitude and time information can be considered. In the example shown in FIG. 2, "mmHg" is extracted as a unit. In addition, the individual identification information is "Yes".

The feature amount vector generation unit 103 combines the calculation result of the statistical value calculation unit 101 and the extraction result of the additional data extraction unit 102 to generate a feature amount vector for the input data, and the caller (in this case, the data). It is supplied to the analysis unit 202) (step B13).

FIG. 5 is an explanatory diagram for explaining the feature quantity vector generated by the feature quantity vector generation unit 103. On the right side of the equation shown in FIG. 5, the values obtained by the statistical value calculation unit 101 and the additional data extraction unit 102 with the numerical string data illustrated in FIG. 3 as input values are input as multidimensional quantities ( The state summarized as a vector) is shown.

The feature amount vector output unit 203 outputs the acquired feature amount vector and the type to the feature amount vector management unit 400.

In the feature amount vector storage unit 403, the feature amount vector is stored in a layered manner (clustered) with respect to the data type. The layering is executed by the feature vector layered storage unit 401, and the feature vector layered storage unit 401 saves the given data type in a layered state when the feature vector is stored. .. At that time, for example, "blood pressure of 20's man", if there is a two input data of "blood pressure of women in their 20s", may be stored in a type of "blood pressure in their 20s adult" that is common as a child type .. As a method of clustering, there are a method of collating with a dictionary of a specific field, a method of performing cluster analysis with a feature vector, and the like. The feature vector layered storage unit 401 may use any known layering method, but it is preferable to select a layering method that can obtain a desired estimation result.

The feature vector layered storage unit 401 in the feature vector management unit 400 reads out the hierarchical information indicating the hierarchical structure of the data type from the feature vector storage unit 403 (step S13).

The feature vector layered storage unit 401 identifies the layer in which the feature vector should be stored from the layer of the type of numerical string data as training data and the hierarchical structure read from the feature vector layered storage unit 401. (Step S14). The feature vector layered storage unit 401 stores the feature vector in the specified layer in the feature vector storage unit 403 (step S15).

On the left side of the equation shown in FIG. 5, a state in which the types assigned to the numerical string data illustrated in FIG. 3 are hierarchized is shown. As shown in FIG. 5, the types are layered in order from a broader concept (“pressure” in this example) to a narrower concept (“20s” in this example).

FIG. 6 is an explanatory diagram for explaining the hierarchical structure stored in the feature quantity vector layered storage unit 401. In the example shown in FIG. 6, the concept is represented in a tree shape from the widest concept to the narrowest concept in the hierarchical structure. In the examples shown in FIGS. 5 and 6, the process under step S14 identifies the hierarchy below "male".

FIG. 7 is a flowchart showing the processing of the estimation phase. FIG. 7A shows the processing of the estimation unit 300. FIG. 7B shows the processing of the feature amount vector extraction unit 100.

In the estimation phase, the estimation target data input unit 301 of the estimation unit 300 inputs the estimation target numerical string data from the estimation target numerical string data storage device 600 as input data (step S21). The data analysis unit 302 uses the feature vector extraction unit 100 to acquire a feature vector for the numerical string data to be estimated (step S22). The feature amount vector extraction unit 100 executes the process shown in FIG. 7B. The process shown in FIG. 7 (B) is the same as the process shown in FIG. 4 (B).

Next, the similar feature amount vector search unit 303 uses the feature amount vector management unit 400 to extract a learned feature amount vector having a distance close to the generated feature amount vector from the feature amount vector storage unit 403.

Specifically, the feature amount vector comparison unit 402 reads out a list (list) of feature amount vectors from the feature amount vector storage unit 403 (step S23). The feature amount vector comparison unit 402 inputs the feature amount vector acquired in the process of step S23 via the similar feature amount vector search unit 303, and the feature amount vector and the trained individual feature amount vectors (in the list). The distance from the existing feature vector) is calculated (step S24). At this time, it is preferable that the feature amount vector comparison unit 402 does not treat each element included in the feature amount vector equally, but treats the unit, the presence / absence of individual identification information, etc. with more weight than the statistical value. .. For example, if the type is blood pressure, the unit is not length (m) or weight (kg), so it is preferable to increase the influence of the difference in units (units other than mmHg) on the distance.

When dealing with target data related to foreign exchange (¥ ⇔ $), electric power (W⇔VA), etc., it is not always possible to aggregate one type and one unit. In such a case, the unit may be excluded, but each may be weighted to deal with it. Similarly, the presence or absence of individual identification information, position information, and time information is also adjusted so that a favorable estimation result can be obtained, such as by weighting the calculated statistical value.

The feature vector comparison unit 402 extracts n feature vectors having a small distance (step S25). Note that "n" is a predetermined positive integer. Then, the feature amount vector comparison unit 402 outputs n feature amount vectors to the similar feature amount vector search unit 303. At that time, the feature amount vector comparison unit 402 also outputs the probability according to the distance to the similar feature amount vector search unit 303. If the feature quantity vector comparison unit 402 has a common type (parent type) in the n feature quantity vectors, the feature quantity vector comparison unit 402 also extracts the parent type.

Various methods are known as methods for measuring the distance between vectors. For example, a method that treats the angle between vectors (cosine similarity) as a distance, as represented by the MinHash method, can be processed at high speed. , Suitable for handling large amounts of numerical set data.

In the present embodiment, the feature vector comparison unit 402 calculates the distance and the probability, but these functions may be included in the similar feature vector search unit 303.

The result display unit 304 presents the extracted n feature quantity vectors to the user by displaying them on the display unit (step S26).

FIG. 8 is an explanatory diagram showing an example (display example) of the extracted feature amount vector and the like. As shown in FIG. 8, the types are layered. Since the probabilities are calculated according to the individual distances, the total probability exceeds 1.

In the example shown in FIG. 8, "pressure / blood pressure / maximum / male" is a common type (parent type). In such a case, the feature vector comparison unit 402 includes "pressure / blood pressure / maximum / male" as the parent type in the extraction result in the process of step S25. Further, when the extraction result is displayed, it is preferable that the parent type is displayed at the highest level (see FIG. 9).

FIG. 9 is an explanatory diagram showing the extraction result including the parent type. In the example shown in FIG. 9, the difference between the distance of "20's" and the distance of "30's" is small. However, since it can be estimated that the blood pressure of a man is almost certain, the feature vector comparison unit 402 aggregates the results using the type hierarchy in order to improve the estimation accuracy. If the types are not layered, the relationship between the feature vectors with short distances cannot be specified, and the comprehensive types cannot be presented.

As described above, in the present embodiment, even if the user does not have knowledge about the target numerical set data, the data type estimation device 10 is characterized by additional information such as statistical values and units of the numerical set data. Since it is extracted as a quantity vector and the comparison result between the feature quantity vector and the feature quantity vector of the trained data is presented, the user can easily estimate the data type.

Further, since the numerical set data is numerical data, it does not completely match the training data, but the data type estimation device 10 calculates the distance between the feature quantity vectors and collates it with the hierarchical structure of the trained feature quantity data. By doing so, it is possible to extract the types corresponding to the learned feature vectors having a short distance and to estimate the common type (parent type) of the extracted types with high accuracy.

In the present embodiment, the data related to blood pressure is set as the numerical set data, but the present invention can also target other types of numerical set data. For example, when referring to past sales data, even if it is difficult to obtain specifications for the table structure and type at that time, by learning in advance the numerical data included in the current sales data, It is possible to automatically estimate which column in the numerical set data is the sales, which column is the discount amount, and so on.

FIG. 10 is a block diagram showing a main part of the information processing apparatus according to the present invention. The information processing device 1A shown in FIG. 10 calculates a statistical value from the numerical set data for learning, and extracts the additional data added to the numerical set data. Extraction means 2 (in the embodiment, the statistical value calculation unit 101). And the feature quantity vector generation means 3 (in the embodiment, realized by the feature quantity vector generation unit) that generates the feature quantity vector from the statistical value and the extracted additional data. ) And the feature quantity vector layered storage means for storing the feature quantity vector in the storage unit 5 (in the embodiment, the feature quantity vector storage unit 403) in a state of being layered according to the type of the numerical set data. 4 (in the embodiment, it is realized by the feature quantity vector layered storage unit 401).

FIG. 11 is a block diagram showing a main part of the information processing apparatus of another aspect according to the present invention. In the information processing apparatus 1B shown in FIG. 11, the extraction means calculates a statistical value from the numerical set data for the target which is the estimation target of the type, extracts the additional data added to the numerical set data, and features the feature amount. The vector generation means generates a feature amount vector from the statistical value and the extracted additional data, and calculates and calculates the distance between the generated feature amount vector and the feature amount vector stored in the storage unit 5. The estimation means 6 (in the embodiment, realized by the feature amount vector comparison unit 402) for estimating the type of the numerical set data for the target based on the set distance is provided.

1A, 1B Information processing device 2 Extraction means 3 Feature vector generation means 4 Feature vector layered storage means 5 Storage unit 6 Estimating means 10 Data type estimation device 100 Feature vector extraction unit 101 Statistical value calculation unit 102 Additional data extraction unit 103 Feature vector generation unit 200 Learning unit 201 Learning data input unit 202 Data analysis unit 203 Feature quantity vector output unit 300 Estimating unit 301 Estimating target data input unit 302 Data analysis unit 303 Similar feature quantity vector search unit 304 Result display unit 400 Feature vector management unit 401 Feature vector layered storage unit 402 Feature vector comparison unit 403 Feature vector storage unit 500 Learning numerical string data storage device 600 Estimated numerical string data storage device

Claims (7)

Statistical values are calculated from the numerical set data for learning to which known data types are associated, and the additional data added to the numerical set data is extracted.
A feature vector having the statistical value and the extracted additional data as elements is generated.
The feature quantity vector is stored in the storage unit in a state of being layered according to the type of numerical set data .
Statistical values are calculated from the numerical set data for the target that is the estimation target of the type, and the additional data added to the numerical set data is extracted.
Generate a feature vector with the statistical value and the extracted additional data as elements,
An information processing method that calculates the distance between the generated feature vector and the feature vector stored in the storage unit, and estimates the type of numerical set data for the target based on the calculated distance. Calculate the distance between the generated feature vector and each of the multiple feature vectors stored in the storage unit.
Distance information processing method according to claim 1, wherein outputting the feature vector is stored within the storage portion of the small predetermined matter. The information processing method according to claim 1 or 2 , wherein the numerical set data is tabular data. An extraction means that calculates statistical values from learning numerical set data associated with known data types and extracts additional data added to the numerical set data.
A feature vector generating means for generating a feature vector having the statistical value and the extracted additional data as elements, and a feature vector generating means.
E Bei a feature vector layered storage means for storing said feature vector in a storage unit in a state of being layered with the type of value set data,
The extraction means calculates a statistical value from the numerical set data for the target which is the estimation target of the type, extracts the additional data added to the numerical set data, and extracts the additional data.
The feature amount vector generation means generates a feature amount vector having the statistical value and the extracted additional data as elements.
Further provided is an estimation means for calculating the distance between the generated feature vector and the feature vector stored in the storage unit and estimating the type of numerical set data for the target based on the calculated distance. Information processing device. The estimation means calculates the distance between the generated feature vector and each of the plurality of feature vectors stored in the storage unit, and outputs the feature vector stored in a predetermined storage unit having a small distance. The information processing apparatus according to claim 4. On the computer
A process of calculating statistical values from numerical set data for learning in which known data types are associated with each other and extracting additional data added to the numerical set data.
A process of generating a feature vector having the statistical value and the extracted additional data as elements, and
A process of storing the feature vector in a storage unit in hierarchical state type of value set data,
A process of calculating statistical values from the numerical set data for the target that is the estimation target of the type and extracting the additional data added to the numerical set data.
Processing to generate a feature vector with the statistical value and the extracted additional data as elements,
To calculate the distance between the generated feature vector and the feature vector stored in the storage unit, and to execute the process of estimating the type of numerical set data for the target based on the calculated distance. Information processing program. On the computer
The process of calculating the distance between the generated feature vector and each of the plurality of feature vectors stored in the storage unit, and
The information processing program according to claim 6, wherein a process of outputting a feature amount vector stored in a predetermined storage unit having a small distance is executed.

Images