JP2019101902A - Data processing apparatus, data processing method, and data processing program - Google Patents

Abstract

To provide a data processing apparatus, a data processing method, and a data processing program which can divide data in a short period of time such that unintentional deviation of data contents is prevented from occurring between learning data and verification data.SOLUTION: The present invention is directed to a data processing apparatus 10 having an acquiring uni 12 for acquiring data having a plurality of records each including a plurality of fields corresponding to a plurality of items, an input unit 13 for receiving an input representing selection of an item among the plurality of items and division ratio of data, a dividing unit 15 for extracting a field corresponding to the selected item from each of the plurality of records and for dividing data into a plurality of data including learning data for machine learning and verification data in accordance with a division ratio on the basis of the plurality of extracted fields, and a learning processing unit for carrying out machine learning processing based on the learning data and for verifying a leaning model generated by the machine learning processing, on the basis of the verification data.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a data processing device, a data processing method, and a data processing program.

  In recent years, research has been advanced in which a learning model such as a neural network analyzes data and executes a predetermined task (for example, Patent Document 1 etc.). The learning model is subjected to learning processing so as to improve the task execution accuracy by causing the learning data to learn the relationship between the input and the output. The learned model generated by performing the learning process can analyze new data and perform various tasks. For example, a learned model is used to classify content by attribute and predict sales trends of products.

JP, 2017-168057, A

  The generation of a learned model uses learning data to make the learning model learn the relationship between inputs and outputs using learning data, and verifies whether the learned model can properly execute a task using verification data And the verification process. Therefore, data prepared for generating a learned model may be divided into learning data and verification data before performing learning processing.

  In the past, data division was sometimes performed while manually checking the data content so that unintended deviation of the data content did not occur in each of the learning data and the verification data. However, the data for generating a learned model is often enormous, and the division process may require a long working time. On the other hand, if division processing is performed without checking the data content, data can be divided in a short working time, but in that case, unintended deviation of data content occurs between learning data and verification data, In some cases, the learning process of the learning model can not be performed efficiently, or improper verification results can be obtained in the verification process.

  Therefore, according to the present invention, there is provided a data processing apparatus, data processing method, and data processing capable of performing data division in a short time so as to prevent unintended deviation of data contents between learning data and verification data. The purpose is to provide a program.

  A data processing apparatus according to an aspect of the present invention includes an acquisition unit for acquiring data having a plurality of records, each including a plurality of fields corresponding to a plurality of items, and selection and data of one of the plurality of items. And an input unit for receiving an input of the division ratio, and a field corresponding to the selected item are respectively extracted from the plurality of records, and based on the extracted plurality of fields, the data is A learning process that performs machine learning processing on the basis of learning data and a dividing unit that divides the data into a plurality of data including verification data, and verifies a learning model generated by the machine learning processing on the basis of the verification data Equipped with

  According to this aspect, between the learning data and the verification data, the data used for machine learning is divided into a plurality of data including the learning data and the verification data based on the value of the specific field. The data can be divided in a short time so that unintended deviation of the data content does not occur. Therefore, without preparing data prepared for learning, learning processing of a learning model can be efficiently performed using so-called big data as it is, and properly verifying the performance of the learned model in verification processing. Can.

  In the above aspect, the input unit further receives an input of an addition rule of the field, and the data processing device further comprises an addition unit that adds one non-empty field to each of the plurality of records by the addition rule, Based on the data, the division ratio may divide the data into a plurality of data including machine learning data and verification data.

  According to this aspect, even if the data is not included in the data suitable as a reference for data division, a field suitable for the reference for data division is newly added. Data division can be performed so as to prevent unintended deviation of data contents between learning data and verification data.

  In the above aspect, the acquisition unit acquires a plurality of data, and the input unit receives the selection of the item to which the combining condition is given among the plurality of data and the selection of the item to be combined, and the data processing device And a combining unit configured to combine the plurality of data into one data including fields corresponding to the items to be combined, based on the fields corresponding to the items giving.

  According to this aspect, it is possible to combine a plurality of separately prepared data into one data without including items that do not need to be combined in the combined data, and it is possible to perform learning processing and verification processing of a learning model. It is possible to easily generate the required set of data.

  In the above aspect, when the data includes an empty field, the input unit receives a selection of an empty field completion method, and based on the statistics according to the selected completion method for a field in which the empty field and the item are common. It may further include an editing unit that complements the empty field.

  According to this aspect, when the data is used as learning data or verification data because the record includes an empty field, the loss may occur even if appropriate learning processing or verification processing can not be performed. By complementing the values, learning processing and verification processing can be appropriately performed without deleting records including missing values.

  In the above aspect, the statistic is to find the mode, median, average, maximum value or minimum value of common fields when the data type of the empty fields is numeric, and the empty fields If the data type of is a character type, it may be to find the most frequent character of the common field.

  According to this aspect, an empty field can be complemented by an appropriate statistical value according to the data type, and data is not deviated from unintended data content between the learning data and the verification data. Can be divided.

  In the above aspect, the input unit may receive item selection and data type selection, and the editing unit may change the data type of the field corresponding to the selected item to the selected data type.

  According to this aspect, in the case where an error occurs due to an inappropriate data type, the error can be avoided by changing to an appropriate data type. In addition, by selecting a data type from among changeable data types, it is possible to prevent an erroneous change to the data type.

  In the above aspect, the input unit may receive an item selection and an input of a replacement rule, and the editing unit may replace the field corresponding to the selected item according to the replacement rule.

  According to this aspect, unit conversion and writing fluctuation can be reduced by replacing numerical values and characters of the field according to a predetermined substitution rule, and learning processing and verification processing of a learning model are performed more efficiently. be able to.

  In the above aspect, the input unit may receive selection of an item and input of an item name, and the editing unit may change the name of the selected item to the input item name.

  According to this aspect, the item name can be changed to any item name, and when an error occurs due to the character code of the character used for the item name, the character of the appropriate character code You can avoid the error by changing to. Therefore, the learning process of the learning model can be efficiently performed using the prepared data, and the performance of the learned model can be properly verified in the verification process.

  In the above aspect, the input unit may receive the selection of the target item and the horizontal axis item, and the editing unit may set the target item on the vertical axis and create a graph in which the horizontal axis item is set on the horizontal axis.

  According to this aspect, the relationship between the selected target item and the horizontal axis item can be visually displayed using a graph. Therefore, it is possible to easily identify the item that affects the target item, and it is possible to appropriately select the item to be included in the learning data.

  A data processing method according to another aspect of the present invention includes acquiring data having a plurality of records, each including a plurality of fields corresponding to a plurality of items, selecting one of a plurality of items, and selecting data Receiving the input of the division ratio of the data, and extracting fields corresponding to the selected items from the plurality of records, and based on the extracted plurality of fields, data for learning of machine learning by the division ratio and verification And dividing the data into a plurality of data including the data for learning, and performing machine learning processing based on the data for learning, and verifying the learning model generated by the machine learning processing based on the data for verification. .

  According to this aspect, between the learning data and the verification data, the data used for machine learning is divided into a plurality of data including the learning data and the verification data based on the value of the specific field. The data can be divided in a short time so that unintended deviation of the data content does not occur. Therefore, without preparing data prepared for learning, learning processing of a learning model can be efficiently performed using so-called big data as it is, and properly verifying the performance of the learned model in verification processing. Can.

  According to another aspect of the present invention, there is provided a data processing program including: an acquisition unit for acquiring data having a plurality of records including a computer and a plurality of fields respectively corresponding to a plurality of items; An input unit for receiving selection and input of a data division ratio, extracting fields corresponding to selected items from a plurality of records, and based on the extracted plurality of fields, data for learning of machine learning by division ratio A division unit that divides data and data for verification into multiple data, machine learning processing based on learning data, and verification of a learning model generated by machine learning processing based on verification data, learning Function as a processing unit.

  According to this aspect, between the learning data and the verification data, the data used for machine learning is divided into a plurality of data including the learning data and the verification data based on the value of the specific field. The data can be divided in a short time so that unintended deviation of the data content does not occur. Therefore, without preparing data prepared for learning, learning processing of a learning model can be efficiently performed using so-called big data as it is, and properly verifying the performance of the learned model in verification processing. Can.

  According to the present invention, it is possible to divide data in a short time so as not to cause unintended deviation of data contents between learning data and verification data.

It is a functional block diagram of a data processor concerning an embodiment of the present invention. It is a figure which shows the example of the data display by the data processor which concerns on embodiment of this invention. It is a figure which shows the example of the addition reception of the row | line by the data processor which concerns on embodiment of this invention. It is a figure which shows the example of the division reception of data by the data processor which concerns on embodiment of this invention. It is a flowchart which shows the flow of the machine learning process performed by the data processor which concerns on embodiment of this invention. It is a figure which shows the example of the coupling reception of the data by the data processor which concerns on embodiment of this invention. It is a flowchart which shows the flow of the machine learning process performed by the data processor which concerns on embodiment of this invention. It is a figure which shows the example of reception of the defect value complementation by the data processor which concerns on embodiment of this invention. It is a figure which shows the example of the reception of the data type change by the data processing apparatus which concerns on embodiment of this invention. It is a flowchart which shows the flow of the machine learning process performed by the data processor which concerns on embodiment of this invention. It is a figure which shows the example of the statistical information column produced by the data processor which concerns on embodiment of this invention, and a data graph.

  Embodiments of the present invention will be described with reference to the accompanying drawings. In addition, what attached the same code | symbol in each figure has the same or same structure.

  FIG. 1 is a functional block diagram of a data processing apparatus 10 according to an embodiment of the present invention. The data processing apparatus 10 can communicate with the user terminal 20 and the database 30 via the communication network N, and performs machine learning processing based on the data acquired from the database 30. In the present embodiment, the data processing apparatus 10 processes data acquired from the database 30 based on the information acquired from the user terminal 20. For example, the data processing apparatus 10 can divide the data acquired from the database 30 into machine learning data and verification data, and transmit the processed data to the database 30 again. Among the divided data, learning data is used for machine learning processing performed by the data processing device 10, and verification data is used for verification processing for verifying the performance of a learned model generated by the data processing device 10. . The communication unit 11 included in the data processing apparatus 10 is connected to the communication network N, and connected to the user terminal 20 or the database 30 via the communication network N.

  The learning data is data to be given to the learning model when performing the learning process on the learning model. Further, the verification data is data used to verify whether the output when the learned model executes the task achieves the desired accuracy. For example, a part of sales trend data of a product is used as learning data to create a learned model, and the remaining data is used as verification data to verify sales trends output by the learned model. It is possible to confirm whether the learned model gives appropriate output.

  The communication network N is a wired or wireless telecommunication line, such as the Internet. The communication network N may be a LAN (Local Area Network) or the like.

  The user terminal 20 includes at least an arithmetic circuit, a communication interface, an input unit, a display unit, and a storage unit, and is a terminal operated by a user. The user terminal 20 may be configured by, for example, a personal computer, a tablet, or a smartphone. The user inputs an instruction regarding data editing to the input unit while confirming the contents of the data displayed on the display unit of the user terminal 20. The input instruction on data editing is sent to the communication unit 11 of the data processing apparatus 10 via the communication network N.

  The database 30 stores, manages, etc. data used for machine learning. The database 30 may be provided by, for example, a cloud service. The database 30 transmits data to the data processing apparatus 10 based on an instruction from the user terminal 20.

  The data processing apparatus 10 is an apparatus that executes machine learning processing, and includes a communication unit 11, an acquisition unit 12, an input unit 13, an addition unit 14, a division unit 15, a combination unit 16, and an editing unit 17. The data processing apparatus 10 acquires data stored in the database 30, and performs data processing based on the instruction received by the input unit 13. After processing the data, the data processing device 10 performs machine learning processing using the data.

  Here, the data to be processed by the data processing apparatus 10 may be CSV data composed of a plurality of records including a plurality of fields, or tabular data in which cells are arranged in rows and columns. In the present embodiment, the case of processing CSV data will be described as an example. The CSV data has a second number of records, each including a first number of fields. Also, the first several fields included in each record correspond to the first several items. Here, the first number and the second number are arbitrary natural numbers, and the data has at least one or more fields and a record. In each record, a set of fields stored at common positions is called a column. For example, when a record contains three fields, the first field of each record forms one row. The middle and end fields of each record form a column as well. Although the data may include a first number of columns and a second number of rows, each row corresponds to a record, and an element of each column corresponds to a field. Also, the fields making up the column have common items. For example, when one field has an item "Year", fields in the same column as the field each have an item "Year". In the present embodiment, data editing processing will be described by using data including a plurality of records including three fields as an example unless otherwise noted. That is, in the present embodiment, unless otherwise stated, an example in which the first number before adding a field is 3 and the second number is an arbitrary natural number will be described. The field located at the beginning of the record has the item "Year," the field located at the middle has the item "Sales," and the field located at the end is the item "Genre." ". Note that the order of the fields in the record is arbitrary. For example, even if the field located in the middle has the item "Genre (genre)" and the field located at the end has the item "Sales (sales)" Good. Further, the field located at the end may have an item to be predicted. For example, when learning a learning model in which the fields of items "Year" and "Genre" are input and the fields of item "Sales" are output, the field located at the end is the item " The order of the fields may be changed so as to be “sales”.

  The data may include item names. The item name is the name given to the item, and the selection of the item name is the same as the selection of the item. The item name may directly represent the content of the item or may be different from the content of the item. Therefore, any name can be added to the item. In the present embodiment, “Year” is added as the item name corresponding to the item “Year”, “Genre” is added as the item name corresponding to the “Sales” item and the item “Genre” as the item name corresponding to the item “Sales”. ing. In addition, when the item name is not included in the data, the item name can be added. Further, as shown in the item name column 40 of FIG. 2, the item names are displayed on the display unit of the user terminal 20, and the user can select a column while checking the item names when performing data processing. .

  The communication unit 11 communicates with the user terminal 20 and the database 30 via the communication network N. The communication unit 11 receives an instruction of data processing from the user terminal 20 and transmits the instruction to the input unit 13. The communication unit 11 also receives data to be used for machine learning from the database 30 and transmits the data to the acquisition unit 12. Furthermore, the data processed by the dividing unit 15, the combining unit 16 or the editing unit 17 of the data processing apparatus 10 is acquired and transmitted to the database 30.

  The acquisition unit 12 acquires data from the communication unit 11 and transmits the data to the adding unit 14, the combining unit 16, or the editing unit 17. The communication unit 11 determines the data transmission destination based on the content of the data processing instruction received by the input unit 13. For example, when the input unit 13 receives an instruction to combine data, the data is transmitted to the combining unit 16.

  The input unit 13 receives an instruction of data processing from the communication unit 11. For example, the data processing instruction includes item selection, data division ratio, field addition rule, and the like. The data processing instruction received by the input unit 13 is transmitted to the adding unit 14, the dividing unit 15, the combining unit 16, or the editing unit 17 according to the content of the instruction. For example, when an instruction to add a field is received, the instruction is transmitted to the adding unit 14.

  The adding unit 14 adds one non-empty field to each of the second few records based on the adding rule of the field received from the input unit 13. The fields added by the adding unit 14 form a row. Here, the field addition rule refers to the nature of the field to be added, and there are, for example, additional rules such as sequential or random, but it is not limited thereto. If the add rule is sequential, successive numeric fields are added to each record, and if the add rule is random, a random or pseudorandom field is added to each record. More specifically, if the addition rule is sequential, an integer field incremented or decremented by 1 is added to each record, and if the addition rule is random, a random number in the range of 0 to 1 Alternatively, a pseudo random number field may be added to each record. That is, when the addition rule is sequential, adjacent records have fields of continuous numerical values, but when the addition rule is random, they are not necessarily continuous.

  The dividing unit 15 divides the data based on the instruction received from the input unit 13. The division unit 15 acquires an item serving as a division reference and a division ratio from the input unit 13 as a data processing instruction, and divides data at a division ratio based on fields corresponding to the items serving as a division reference. Data division will be described later with reference to FIG.

  The combining unit 16 combines a plurality of data based on the instruction received from the input unit 13. The combining unit 16 obtains an item giving the combining condition and an item to be combined from the input unit 13, acquires a plurality of data from the obtaining unit 12, and then combines the fields based on the fields corresponding to the items giving the combining condition. A plurality of data are combined so as to be one data including a field corresponding to the target item. The combination of data will be described later with reference to FIG.

  The editing unit 17 acquires the selected item to be edited and the editing content from the input unit 13, and edits the field corresponding to the selected item based on the editing content. For example, the contents of editing include field completion, data type change, field replacement, or item name change. Data editing will be described later with reference to FIGS.

  FIG. 2 is a view showing an example of data display by the data processing apparatus 10 according to the embodiment of the present invention. The data processing apparatus 10 displays the content of the data on the display unit of the user terminal 20 for each column. The figure shows a data table 100 displayed on the display unit of the user terminal 20. The data table 100 shows the contents of data for each column, and the data table 100 includes an item name column 40, a data type column 41, a unique value column 42, a missing value column 43, a column addition button 44, and a column An edit button 45 is included. The data contents displayed in the data table 100 are not limited to the contents shown in FIG.

  The item name column 40 is a column in which an item name is displayed. In the present embodiment, the item name column 40 includes four columns. In the item name column 40, the characters "Item Name", "Year", "Sales", and "Genre" are displayed in order from the upper column, and the user can distinguish the columns by using the item name as a mark. Can. If the data does not include an item name, a temporary item name such as "columnA" or "columnB" may be displayed in the item name column 40. The item name can be arbitrarily changed by the user.

  The data type column 41 is a column in which the data types of the fields constituting the column are displayed. In the data type column 41 of the present embodiment, “data type”, “INTEGER”, and “FLOAT” are sequentially arranged from the above column. , The letters "STRING" are displayed. Here, "INTEGER" means integer type, "FLOAT" means floating point type, and "STRING" means character data type. That is, the data type of the field included in the column of the item name "Year" is an integer type, the data type of the field included in the column of the item name "Sales" is a floating point type, and the column of the item name "genre" It can be seen that the data types of the fields included in are character types. The data type of the field is not limited to the integer type, the floating point type, and the character type, and may be any data type such as a boolean type, a time stamp type, and the like.

  Here, the integer type is a kind of numeric type, and is a data type that can store integers. The floating point type is a type of numeric type and is a data type that can store real numbers. The character type is a data type capable of storing a character string. The true / false type is a data type that can store a true / false value (such as “true” or “false”), and is used in condition determination or the like. The timestamp type is a data type that stores a time value composed of year, month, day, hour, minute, and second.

  The unique value column 42 is a column in which the value of the unique value is displayed. The unique value is the number of fields when counting overlapping fields (fields consisting of the same numerical value and the same string) as one field in each field constituting the column. For example, in a 5-field column, if two fields have the same numerical value, the unique value is 4. In the unique value column 42 shown in FIG. 2, the characters “unique value”, “50”, “46”, “2” are displayed in order from the upper column, and the unique value of the column of the item name “Year” Is 50, the unique value of the item name "Sales" is 46, and the unique value of the item name "Genre" is 2.

  The missing value field 43 is a field in which information on the missing value is displayed. Missing values are empty fields for which no data exists. In the missing value column 43, the number of empty fields included in the column and the ratio of the number of empty fields to the number of all fields included in the column are displayed. In the missing value column 43 shown in FIG. 2, numerical values of “1 (0.01%)”, “1 (0.01%)”, and “0 (0%)” are displayed in order from the upper column. . That is, one empty field exists in the column of the item name "Year", the ratio of the number of empty fields occupying in the whole column is 0.01%, and the column of the item name "Sales" is empty. There is one field, and the proportion of the number of empty fields in the whole column is 0.01%, and there is no empty field in the column of the item name "Genre", the empty fields in the whole column It can be seen that the proportion of the number of is 0%.

  The column addition button 44 is a button for adding a new column to the data. When the column addition button 44 is selected, a column addition menu 50 shown in FIG. 3 is displayed on the display unit of the user terminal 20. The addition of columns will be described later with reference to FIG. The column editing button 45 is a button for editing the fields included in the column. When the column editing button 45 is selected, a column editing menu 80 as shown in FIG. 8 is displayed. The column editing will be described later with reference to FIG.

  FIG. 3 is a diagram showing an example of the column addition acceptance by the data processing apparatus according to the embodiment of the present invention. The figure shows a column addition screen 200 displayed on the display unit of the user terminal 20. When the column addition button 44 is selected, a column addition menu 50 is displayed, and the contents of the column to be added by the addition unit 14 can be determined by the column addition menu 50. The column addition menu 50 includes an addition rule selection unit 51, an item name input unit 52, a cancel button 53, and an execution button 54.

  In the additional rule selection unit 51, characters of "sequential value" and "random value" are displayed. When the user selects "sequential value", a column of continuous values is added, and when "random value" is selected, a column of irregular values is added. The "value" mentioned here includes numerical values, character strings, symbols, and other values. For example, the irregular value may be a real number randomly selected from a predetermined numerical range. It may be an integer randomly selected from a predetermined numerical range or a numerical value randomly selected from several candidate values. In addition, as the irregular value, for example, a column of irregular date or time may be added. In the additional rule selection unit 51 shown in FIG. 3, since a random value is selected, for example, a sequence of irregular numerical values (including pseudo random numbers) is added. The item name of the column to be added can be input to the item name input unit 52. From the figure, it can be seen that the item name of the column to be added is "Add". Note that the option shown in FIG. 3 is an example, and the columns to be added are not limited to columns consisting of numerical values. For example, the data processing apparatus 10 may display an option such as an option “random character” on the additional rule selection unit 51, and may add a string of irregular character strings when the option is selected by the user. Also, for example, columns may be added by a column of alphabets or colors selected regularly or irregularly from a plurality of colors. In addition, when selecting a sequential value or a random value, the additional rule selection unit 51 of FIG. 3 may select or designate the type (numerical value, character string, symbol, etc.) of the value to be added.

  The cancel button 53 is a button for canceling the column addition operation, and when selected, the column addition menu 50 is closed. The execution button 54 is a button for executing a column adding operation, and when selected, the adding unit 14 of the data processing apparatus 10 adds a column based on the content input in the column adding menu 50.

  FIG. 4 is a diagram showing an example of data division acceptance by the data processing apparatus according to the embodiment of the present invention. The figure shows a data division screen 300 displayed on the display unit of the user terminal 20. The data division screen 300 includes a reference item selection unit 60, a sort condition selection unit 61, a division number input unit 62, a division data name input unit 63, and a division ratio input unit 64.

  The reference item selection unit 60 receives the selection of the item to be the reference of data division. The dividing unit 15 divides data based on the field of the item serving as the reference. For example, when dividing data including a column consisting of fields of integer values from 1 to 100 at a ratio of 8 to 2, by using the items of the column as the reference items, the alignment from 1 to 80 Data can be divided at a ratio of 8 to 2 by dividing into a record including a numeric field and a record including an integer value field of 81 to 100. Also, for example, when dividing data including a row consisting of random number fields from 0 to 1 at a ratio of 8 to 2, it is possible to use an item in the row as a reference item, thereby dividing the data from 0 to 0.8. Data can be divided at a ratio of 8 to 2 by dividing into records including less than fields and records including 0.8 or more and 1 or less fields. However, columns suitable for such division criteria are not necessarily included in the data. Therefore, in the data processing apparatus 10 according to the present embodiment, the adding unit 14 is configured to be able to add a row suitable for the division criterion. In the present embodiment, since the character “Add” is displayed in the reference item selection unit 60 of FIG. 4, data division is performed based on the column of the item name “Add” added by the addition unit 14.

  The sort condition selection unit 61 receives the sort condition of the field. For example, when the sort condition selection unit 61 is selected, the characters "ascending order" and "descending order" are displayed, and the user can select either. Since "ascending order" is selected in the sorting condition selecting unit 61 shown in FIG. 4, the dividing unit 15 rearranges the records so that the fields corresponding to the column of the item name "Add" are arranged in ascending order. In this embodiment, after a column (item name “Add”) consisting of random numerical values is added, sorting is performed based on the added column, so records included in the original data are mixed. It will be in the state.

  Moreover, the column used as the reference | standard of a sort is not restricted to the added column, It can sort by selecting the column of the item name "Year" and "Sales" contained in original data. For example, after sorting based on the column of the item “Year”, data division can be performed to divide data into old data and new data of the year. Also, by sorting the items included in the data, it becomes easy to confirm the data contents when the CSV data is output to a display or a paper medium. It is not necessary to perform sorting at the time of division, and whether or not to execute sorting is optional.

  The division number input unit 62 receives an input of the number for dividing data. Since “2” is input to the division number input unit 62 shown in FIG. 4, the data is divided into two by the division unit 15. The numerical value input to the division number input unit 62 is not limited to “2”, and may be an arbitrary value. For example, the data is divided into three, and two data of them are used as learning data and verification data when creating a learned model, and the result of the learned model outputting the remaining data is It may be used as comparison data to show how it conforms to reality. Also, for example, the remaining two data are used as learning data when creating a learned model, as the validation data used to divide the data into three and adjust one to adjust the hyper-parameters of the learning model. It may be used for verification data.

  The divided data name input unit 63 receives an input of the name of data after division. The division ratio input unit 64 also receives an input of data division ratio. The divided data name input unit 63 and the division ratio input unit 64 are displayed by the number input by the division number input unit 62. As shown in the figure, the original data is divided at a ratio of 8 to 2, and the names of the divided data are "learning data" and "verification data", respectively.

  FIG. 5 is a flowchart showing the flow of machine learning processing executed by the data processing apparatus according to the embodiment of the present invention. First, based on an instruction from the user terminal 20, the acquisition unit 12 acquires data for performing machine learning from the database 30 via the communication unit 11 (S10). Upon acquiring the data, the data processing apparatus 10 causes the display unit of the user terminal 20 to display the data division screen 300 as shown in FIG. 4 and the user knows how to divide and learn the acquired data. Accept instructions from At this time, the user may divide data using an existing field included in the acquired data, such as, for example, Year or Sales, but adds a new field. The data may be divided based on the field. When data division is performed without using an existing field, the data processing apparatus 10 causes the display unit of the user terminal 20 to display a column addition screen 200 as shown in FIG. 3 based on an instruction from the user terminal 20. Then, the input unit 13 receives the input of the addition rule and the item name of the column to be added (S11). The adding unit 14 adds a new field to the data acquired from the database 30 based on the received input content (S12). For example, a column of item name "Add" consisting of random numerical values is added.

  After adding a new field, the data processing apparatus 10 causes the display unit of the user terminal 20 to display the data division screen 300 as shown in FIG. 4 again, and divides the acquired data to perform learning. Accept instructions from the user. The input unit 13 selects an item name of a field serving as a reference when dividing data, selects a sorting condition of a field corresponding to the selected item name, inputs a division number, inputs a division data name, and divides a division ratio. The input is accepted (S13). The dividing unit 15 rearranges the data (S14) so that the fields corresponding to the selected item names are arranged according to the selected sort condition, and the number of divisions input based on the column of the selected item names The data is divided so as to be the division ratio and the divided data are named (S15). In this example, the dividing unit 15 rearranges the positions of the records so that the columns of the added item name “Add” are arranged in ascending order, and divides the data at a ratio of 8 to 2 based on the column of the item name “Add” Do. After dividing the data, give "Data for learning" to data that occupies 80% of the original data, and give the data name "Data for verification" to the data that occupies 20%. Thus, the processing by the data processing device 10 is completed. Thereafter, the data processing apparatus 10 performs machine learning processing based on the divided data (S16). For example, after a learning model is generated using data divided as learning data, the learning model is verified using data divided as verification data, and a learning model that can be verified to have a predetermined accuracy is used. It can be adopted. In addition, data processing device 10 itself may perform machine learning processing to perform machine learning processing, data processing device 10 shall support machine learning processing, and an external machine learning server (not shown) May perform machine learning processing. In the former case, the data processing device 10 may include a learning processing unit that performs machine learning processing and verification processing, and the learning processing unit may execute the machine learning processing. In the latter case, for example, data for learning and data for verification are passed from the data processing apparatus 10 to an external machine learning server via the network, and generated after the machine learning server is executed by the machine learning server. The data processing apparatus 10 may receive the learning model.

  According to the present embodiment, data used for machine learning is divided into a plurality of data including learning data and verification data based on the value of a specific field, whereby the learning data and the verification data are It is possible to divide data in a short time so as not to cause deviation of data contents among the data.

  Further, according to the present embodiment, even if a field suitable for use as a reference for data division is not included in the data in advance, as described above, a field consisting of a sequential value or a random numerical value, etc. By adding to each record, it is possible to divide data so as not to cause unintended deviation of data contents between learning data and verification data. Therefore, without preparing data prepared for learning, learning processing of a learning model can be efficiently performed using so-called big data as it is, and properly verifying the performance of the learned model in verification processing. Can.

  FIG. 6 is a diagram showing an example of acceptance of data combination by the data processing apparatus 10 according to the embodiment of the present invention. When data desired to be machine-learned in a learning model is divided and present as a plurality of data, it is inefficient to execute machine learning processing many times using each data. Therefore, in order to perform machine learning processing efficiently using one set of data, a plurality of data may be combined.

  The figure shows a data combination screen 400 displayed on the display unit of the user terminal 20. The data combining screen 400 includes a first combining condition selecting unit 70a, a second combining condition selecting unit 70b, a first combining target selecting unit 71a, a second combining target selecting unit 71b, and a combining result display unit 72. Data combining will be described by way of example of combining two pieces of data (hereinafter referred to as Data 1 and Data 2 respectively).

  The item names of the items included in Data 1 are displayed in the first combination condition selection unit 70 a, and the user can select an item to which a combination condition is to be provided. The item which gives the join condition is also selected from Data2. Here, the item which gives the combining condition is an item which becomes a reference of data combining, and usually, the item which a plurality of data to combine have in common is selected. The fields corresponding to the items for which the combining condition is given are integrated in the combined data to form one row. As shown in FIG. 6, in the first combining condition selection unit 70a, the item "Year" is selected as an item to which a combining condition is given.

  The second combining condition selecting unit 70b has the same configuration as the first combining condition selecting unit 70a, and the user can select an item to which the combining condition is to be provided. As shown in FIG. 6, in the second coupling condition selection unit 70b, the item "Year" is selected as the item to which the coupling condition is given.

  When combining data, the field of the item giving the combining condition selected from a plurality of data is checked to see if there is a field having a common content. When there is a field having common contents, a plurality of records having the common field are integrated into one record. Here, “content in common” includes cases in which the contents of data are common as well as cases in which the numerical values and characters of the fields completely match. For example, for a field representing a year, if one data field represents a year in year notation, and another data field represents the same year in Christian notation, it is assumed that the contents of both fields are common. It may be judged. Note that a record having no field whose content is common may be included in the combined data without being integrated with other records. In the present embodiment, the item “Year” is selected from Data 1 and Data 2 as the item for providing the combining condition. Therefore, when Data1 and Data2 are combined, the fields of the item "Year" included in each data are compared with each other, and a plurality of records having fields having common contents are integrated into one record. The integration of records may be performed, for example, by adding selected fields other than the item “Year” of Data 2 to the record of Data 1.

  In the first combination target selection unit 71a, check boxes are displayed for each of the name of the data to be combined, the item name, and the item name. In the first combination target selection unit 71a shown in FIG. 6, "Data 1" as the data name, "Year", "Sales" and "genre" as the item names are displayed, and a check box is provided for each item name. . The user can select an item included in the combined data by checking a check box provided for each item name. Items included in the combined data are referred to as items to be combined. From the figure, it can be seen that the data after combination includes the items of the item names "Year", "Sales" and "genre".

  The second combination target selection unit 71b has the same configuration as the first combination target selection unit 71a, and the second combination target selection unit 71b illustrated in FIG. 6 has “Data 2” as a data name and “Year” as an item name. And “Name” is displayed, and all check boxes provided for each item name are checked. From the figure, it can be seen that the combined data includes the items of the item names "Year" and "Name".

  The combined result display section 72 shows the name and item name of data after combined. From the combining result display unit 72 shown in FIG. 6, the data name after combining is "Data1-Data2-merge", and the item names of the items included in the data after combining are "Year", "Sales", and "Genre". And “Name”. The fields corresponding to the item names “Year” present in each of Data 1 and Data 2 are integrated in the combined data to form one column. Therefore, only one item name "Year" is displayed on the combined result display unit 72.

  FIG. 7 is a flowchart showing a flow of machine learning processing executed by the data processing device according to the embodiment of the present invention. First, based on an instruction from the user terminal 20, the acquisition unit 12 acquires a plurality of data for performing machine learning (S20). At this time, the user may cause the data processing apparatus 10 to acquire, for example, two pieces of data Data1 and Data2 as data for performing machine learning. When the data is acquired, the data processing apparatus 10 causes the display unit of the user terminal 20 to display a data combination screen 400 as shown in FIG. 6, and the user determines how to combine the acquired data to perform learning. Accept instructions from Thereafter, the input unit 13 receives the selection (S21) of the item name of the item to which the combination condition is given. At this time, the user may select, for example, an item of the item name “Year” from Data 1 and Data 2 as an item for providing the combining condition. Further, the input unit 13 receives a selection of items to be combined (S22). At this time, the user selects items of item names “Year”, “Sales” and “Genre” from Data 1 as items to be combined, and selects items of item names “Year” and “Name” from Data 2 May be

  The combining unit 16 compares fields of items to be combined selected from a plurality of data, integrates records including common fields, and includes fields of items to be combined in one data. , Combine a plurality of data (S23). In this example, the combining unit 16 compares the fields of the item name “Year” included in each of Data 1 and Data 2 and integrates the records of the common year, and combines the item names “Year”, “Sales”, “ Combine Data1 and Data2 so that the fields of "genre" and "Name" are included in one data. Thereafter, the data processing apparatus 10 performs machine learning processing based on the combined data (S24). Thus, the processing by the data processing device 10 is completed. The number of data to be combined is not limited to two, and any number of data can be combined.

  According to this embodiment, a plurality of separately prepared data can be combined into one data without including items which do not need to be combined in the combined data, and the learning processing and verification processing of the learning model It is easy to generate a set of data required for

  FIG. 8 is a diagram showing an example of acceptance of missing value complementation by the data processing apparatus 10 according to the embodiment of the present invention. When data including a missing value is used for machine learning processing, there is a possibility that the learning processing can not be appropriately performed, for example, the performance of the generated learned model may be degraded. In addition, when data including a missing value is used for verification processing, verification processing may not be appropriately performed, for example, the performance of a learned model can not be accurately verified. Therefore, in order to appropriately perform machine learning processing or verification processing using data that does not include missing values, missing values need to be complemented. The figure shows a data editing screen 500 displayed on the display unit of the user terminal 20. In the data editing screen 500, a column editing button 45 is displayed for each column. The user can select a column to be edited by selecting one of the column editing buttons 45 from among the plurality of column editing buttons 45 provided for each column. When the column editing button 45 is selected, a column editing menu 80 is displayed. The column editing menu 80 displays a plurality of options. The column editing menu 80 shown in FIG. 8 displays the options “missing value”, “change data type”, “replace field”, “change item name”, “delete column” and “graph display”. There is.

  When the option "missing value" is selected in the column editing menu 80, a missing value processing menu 81 is displayed, and processing can be performed on the missing value included in the data. In the missing value processing menu 81, the options “deletion of line” and “complement of missing value” are displayed. If "Delete Row" is selected, the record containing the missing value can be deleted and the column can be made free of the missing value.

  In addition, when the option “complement of missing value” is selected, the complementing menu 82 is displayed, and the field including the missing value can be complemented with the value displayed in the complementing menu 82. In the complement menu 82, the options "mode", "median", "average", "maximum", "minimum", "longitudinal", and "arbitrary" are displayed. When the option "Mode" is selected, interpolation is performed with the value most contained in the column. If the option "central value" is selected, interpolation is performed with the value located at the center when the values included in the column are arranged from the smaller value. When the option "average value" is selected, interpolation is performed using the sum of the values included in the column divided by the number of values included in the column. When the option "maximum value" is selected, interpolation is performed with the largest value among the values included in the column. When the option "minimum value" is selected, interpolation is performed with the smallest value among the values included in the column. When the option “back and forth value” is selected, in a column including a missing value, completion is performed with the values of fields included in records before and after the record including the missing value. When the option "arbitrary value" is selected, the user can input an arbitrary value, and the interpolation is performed with the input value.

  If the data type of the field constituting the column including the missing value is a character type, the complement menu 82 displays the option “most frequent character”. If the option "Most characters" is selected, missing values are complemented with the most frequently contained characters in the column. A complementation method suitable for the data type of the field constituting the column is displayed in the complementation menu 82.

  According to the present embodiment, even when the data is used as learning data or verification data, an appropriate learning process or verification process can not be performed because the record includes an empty field. By complementing the missing value, it is possible to appropriately perform learning processing and verification processing without deleting a record including the missing value. In addition, empty fields can be complemented by appropriate statistical values according to data types, and data division is performed so that unintended data content deviation does not occur between learning data and verification data. be able to.

  When the option “change data type” is selected in the column editing menu 80, the data types of the fields constituting the column can be changed. The change of the data type will be described later with reference to FIG.

  When the option "replace field" is selected in the column editing menu 80, a replacement rule can be set, and fields constituting the column can be replaced with other fields according to the replacement rule. For example, when the column is configured by the sales amount field expressed in Japanese yen units, the conversion rate between Japanese yen and US dollars is set as the substitution rule, and substitution of the field is performed to display in US dollar units. It is possible to generate a field for the sales amount that has been sold. In addition, for example, when the same object is described using a plurality of names, it is possible to perform name identification and eliminate the writing fluctuation. According to the present embodiment, unit conversion and writing fluctuation can be reduced by replacing numerical values and characters of the field according to a predetermined substitution rule, and learning processing and verification processing of a learning model can be made more efficient. It can be carried out.

  When the option “change item name” is selected in the column editing menu 80, the item name can be changed to any name. For example, the item name “genre” shown in FIG. 2 can be changed to “Genre”. If an error occurs because the item name is represented by double-byte characters such as kana or kanji, the error can be avoided by changing the item name to single-byte characters such as alphabets or numerical values. .

  According to the present embodiment, the item name can be changed to any item name, and if an error occurs due to the character code of the character used for the item name, the appropriate character code can be used. Errors can be avoided by changing to characters. Therefore, the learning process of the learning model can be efficiently performed using the prepared data, and the performance of the learned model can be properly verified in the verification process.

  When the option “delete column” is selected in the column editing menu 80, a column can be deleted from the data. In FIG. 8, since the column of the item name "Sales" is selected, if "Delete column" is temporarily selected in the column editing menu 80, the column of the item name "Sales" is deleted.

  When the option “graph display” is selected in the column editing menu 80, statistical information of the selected column and data contained in the column can be displayed as a graph. The graph display will be described later with reference to FIG.

  FIG. 9 is a diagram showing an example of acceptance of data type change by the data processing apparatus according to the embodiment of the present invention. The figure shows a data type change screen 600 displayed on the display unit of the user terminal 20. When the option “change data type” is selected from the column editing menu 80, a data type selection menu 83 is displayed. In the data type selection menu 83, "character type", "integer type", "floating point type", "true / false value type" and "time stamp type" are displayed as data type options after the change. For example, when the character type is selected in the data type selection menu 83 of FIG. 9, the data type of the field constituting the column of the item name “Year” is changed from integer type to character type. In the data type selection menu 83, only changeable data types may be displayed.

  For example, when data is input to a learning model, if an error occurs because a column without data type (empty field) is included in a column whose data type is integer type, the data You can eliminate the error by changing the type from integer to character and replacing the field without numbers with a field that contains zero characters (ie no entry for characters).

  According to the present embodiment, when an error occurs due to the inappropriate data type, the error can be avoided by changing to the appropriate data type. In addition, by selecting a data type from among changeable data types, it is possible to prevent an erroneous change to the data type.

  FIG. 10 is a flowchart showing the flow of machine learning processing executed by the data processing apparatus according to the embodiment of the present invention. FIG. 10 shows a flow of processing of complementing missing values, changing of data types, replacing of fields, changing of item names, and machine learning processing performed by the data processing apparatus 10. First, information on the missing value is displayed on the display unit of the user terminal 20 (S30). For example, as shown in the missing value column 43 of FIG. 8, the number and ratio of fields including the missing value may be displayed on the display unit of the user terminal 20. After that, when the input unit 13 receives the column selection and the missing value complementation method (S31: Yes), the field of the selected column is statistics, and a value for complementing the missing value is obtained (S32). For example, the data processing apparatus 10 causes the display unit of the user terminal 20 to display the data editing screen 500 as shown in FIG. 8, and accepts an instruction from the user as to how to compensate for the missing value. It is also good. As shown in FIG. 8, when the column of the item name “Sales” is selected, and the option “mode” is selected from the complement menu 82, the editor 17 statistics the field of the column of the item name “Sales”. Then, after finding the mode, the missing value is complemented by the mode.

  When the complementation of the missing value is completed or when the input unit 13 does not receive the instruction of the missing value complementation (S31: NO), the input unit 13 determines whether the column selection and the data type selection are accepted. (S34). When the input unit 13 receives column selection and data type selection (S34: Yes), the editing unit 17 changes the data type of the selected column to the selected data type (S35). For example, the data processing apparatus 10 causes the display unit of the user terminal 20 to display a data type change screen 600 as shown in FIG. 9 and accepts an instruction from the user as to what data type to change to. Good. As shown in FIG. 9, when the column of the item name "Year" is selected and the option "character type" is selected from the data type selection menu 83, the editing unit 17 selects the data type of the column of the item name "Year". Change from "integer type" to "character type".

  When the data type change is completed or when the input unit 13 does not receive the data type change instruction (S34: NO), the input unit 13 determines whether the column selection and the field replacement rule input are received. It judges (S36). If the input unit 13 receives column selection and field substitution rule input (S36: Yes), the editing unit 17 substitutes the field of the selected column based on the substitution rule (S37). For example, when the option "replacement of field" is selected in the column editing menu 80 as shown in FIG. 8, the data processing apparatus 10 displays a setting screen of a replacement rule on the display unit of the user terminal. Accept the settings. In this example, when the Japanese yen and US dollar conversion rates are set as the replacement rule, the editing unit 17 converts the sales amount in Japanese yen units included in the column of the item name "Sales" into US dollar units based on the conversion rates. Replace with the sales amount of.

  When the field replacement is completed or when the input unit 13 does not receive the field replacement instruction (S36: No), the input unit 13 determines whether the column selection and the item name input are received (S38). ). When the input unit 13 receives column selection and item name input (S38: Yes), the editing unit 17 changes the item name of the selected column to the input item name (S39). For example, when the option “change item name” is selected in the column editing menu 80 as shown in FIG. 8, the data processing apparatus 10 displays an input screen of a new item name on the display unit of the user terminal, Accept input of item name. In the present example, when “Genre” is input as a new item name, the editing unit 17 changes the item name “genre” to “Genre”. When the change of the item name is completed or when the instruction of the item name change is not received (S38: No), the data processing device 10 performs a machine learning process based on the edited data (S40). Thus, the processing by the data processing device 10 is completed. Note that various data processing performed by the data processing apparatus 10, such as complementation of missing values, change of data type, replacement of fields, and change of item names, may not be performed in the order described using FIG. May be performed in the order of

  FIG. 11 is a diagram showing an example of the statistical information section 700 and the data graph 800 created by the data processing apparatus 10 according to the embodiment of the present invention. After being transmitted to the user terminal 20, the statistical information section 700 and the data graph 800 created by the data processing apparatus 10 may be displayed on the display unit of the user terminal 20. The figure shows a statistical information field 700 and a data graph 800 displayed on the display unit of the user terminal 20. The statistical information column 700 and the data graph 800 may be displayed, for example, when the user selects the option “graph display” in the column editing menu 80 shown in FIG. An item selected to display a graph the relationship between the item selected to display statistical information by the user or another item is referred to as a target item, and in this example, the item "Sales" is selected as the target item.

  The statistical information column 700 displays the result of the statistics of the field corresponding to the selected target item. The statistical information column 700 shown in FIG. 11 displays the result of the statistics of the field corresponding to the target item "Sales". Although “mode”, “average value”, “maximum value” and “minimum value” are displayed as statistical information in the statistical information column 700, the displayed statistical information is not limited to these, and the median Arbitrary statistical information, such as and distribution, may be displayed.

  The data graph 800 is a graph representing the relationship between the target item and the horizontal axis item. Data graph 800 includes a horizontal axis 90 and a vertical axis 91. On the vertical axis 91, a scale suitable for the content of the selected target item is inscribed, and the item name of the target item is displayed as a label of the axis. In the vertical axis 91 in this example, in order to indicate the size of sales, numerical values at fixed intervals are inscribed as a scale, and the item name "Sales" of the target item is displayed as a label of the axis. Further, the horizontal axis 90 is also engraved with an appropriate scale similarly to the vertical axis 91, and the item name of the horizontal axis item is displayed as the axis label. The item to be set as the horizontal axis item can be arbitrarily selected by the user. For example, the data processing apparatus 10 may cause the horizontal axis item selection unit 92 to display a list of items included in the data, and allow the user to select an item to be set as the horizontal axis item from the displayed list. In this example, since the item “Year” is selected in the horizontal axis item selection unit 92, the year is inscribed on the horizontal axis 90 as a scale, and the item name “Year” is displayed as a label of the axis. Further, the range selection unit 93 can select the range of the horizontal axis 90 to be displayed graphically. In this example, since the range of the horizontal axis 90 is selected as “2000” to “2017” in the range selection unit 93, sales from 2000 to 2017 are displayed in the data graph 800. Although one item can be set on the horizontal axis in this example, items that can be set on the horizontal axis are not necessarily limited to one, and a mode in which a plurality of items can be set may be adopted. For example, when two items are set as the horizontal axis, a new axis is added in the depth direction of the data graph 800 shown in FIG. 11, and a three-dimensional data graph 800 in which three axes intersect is created. Good. The data graph 800 created by the data processing apparatus 10 is not limited to a bar graph, and may be any graph such as a line graph or a scatter chart.

  The graph output button 94 is a button for outputting the graph created by the data processing device 10. For example, when the graph output button 94 is selected by the user, image data or the like of the data graph 800 is transmitted from the data processing apparatus 10 to the user terminal 20 or the like. The table output button 95 is a button for outputting the table created by the data processing device 10. For example, when the table output button 95 is selected by the user, the data processing apparatus 10 creates a table in which fields corresponding to the target items and fields corresponding to the horizontal axis items are listed, and the CSV data or image of the table Data and the like are transmitted to the user terminal 20 and the like.

  According to this embodiment, the relationship between the selected target item and the horizontal axis item can be visually displayed using a graph. Therefore, it is possible to easily identify the item that affects the target item, and it is possible to appropriately select the item to be included in the learning data. For example, consider the case of preparing learning data for creating a learned model for sales forecasting. At this time, a sales item included in the prepared data is set as the target item on the vertical axis 91, and a sales year item is set as the horizontal axis item on the horizontal axis 90 to create a graph. If the value of the created graph fluctuates from year to year, it is considered that the year of sale is likely to affect the sales, so it is understood that the item of year of sale should be included in the data for learning. On the other hand, when the other item is set as the horizontal axis 90, if the value of the graph hardly changes for each field of the item, the item is considered to be unlikely to affect sales, so the item It can be seen that the need to include in the training data is low compared to the year of sales item.

  The embodiments described above are for the purpose of facilitating the understanding of the present invention, and are not for the purpose of limiting the present invention. The elements included in the embodiment and the arrangement, the material, the conditions, the shape, the size, and the like of the elements are not limited to those illustrated, and can be changed as appropriate. In addition, configurations shown in different embodiments can be partially substituted or combined with each other.

  DESCRIPTION OF SYMBOLS 10 ... Data processing apparatus, 11 ... Communications part, 12 ... Acquisition part, 13 ... Input part, 14 ... Addition part, 15 ... Division part, 16 ... Coupling part, 17 ... Editing part, 20 ... User terminal, 30 ... Database, 40 ... item name column, 41 ... data type column, 42 ... unique value column, 43 ... missing value column, 44 ... column addition button, 45 ... column editing button, 50 ... column addition menu, 51 ... addition rule selection unit, 52 ... item name input unit, 53 ... cancel button, 54 ... execution button, 60 ... reference item selection unit, 61 ... sort condition selection unit, 62 ... division number input unit, 63 ... division data name input unit, 64 ... division ratio input Part 70a: First combination condition selection unit 70b: Second combination condition selection unit 71a: First combination target selection unit 71b: second combination target selection unit 72: Combination result display unit 80: Row edit menu , 81 ... Missing value processing New, 82: Complement menu, 83: Data type selection menu, 90: Horizontal axis, 91: Vertical axis, 92: Horizontal axis item selection section, 93: Range selection section, 94: Graph output button, 95: Table output button, 100 ... data table, 200 ... column addition screen, 300 ... data division screen, 400 ... data combination screen, 500 ... data edit screen, 600 ... data type change screen, 700 ... statistical information column, 800 ... data graph

Claims (11)

An acquisition unit for acquiring data having a plurality of records, each including a plurality of fields corresponding to a plurality of items;
An input unit that receives selection of one of the plurality of items and input of a division ratio of the data;
A field corresponding to the selected item is extracted from each of the plurality of records, and based on the extracted field, the data is divided into a plurality of data including machine learning learning data and verification data at the division ratio. A division unit to be divided,
A learning processing unit that performs machine learning processing based on the learning data, and verifies a learning model generated by the machine learning processing based on the verification data;
A data processing apparatus comprising: The input unit accepts an input of an additional rule of a field,
The system further comprises an addition unit for adding one non-empty field according to the addition rule to each of the plurality of records,
The division unit divides the data into a plurality of data including machine learning learning data and verification data at the division ratio, based on the added field.
The data processing apparatus according to claim 1. The acquisition unit acquires a plurality of the data.
The input unit receives a selection of an item to which a combination condition is given among the plurality of data and a selection of an item to be combined.
And a combining unit configured to combine the plurality of data into one piece of data including a field corresponding to the item to be combined based on a field corresponding to the item giving the combining condition.
The data processing apparatus according to claim 1. The input unit receives selection of a complementation method of the empty field, when the data includes the empty field,
It further comprises an editor for complementing the empty field based on statistics according to the selected complementation scheme regarding the field in which the empty field and the item are common.
The data processing apparatus according to any one of claims 1 to 3. The statistics may include determining a mode, a median, an average, a maximum value, or a minimum value of the common field, when the data type of the empty field is a numeric type. If the data type of is a character type, it is to find the most frequent character of the common field,
The data processing device according to claim 4. The input unit receives item selection and data type selection.
The editing unit changes the data type of the field corresponding to the selected item to the selected data type.
A data processing apparatus according to claim 4 or 5. The input unit receives an input of item selection and replacement rules;
The editing unit replaces the field corresponding to the selected item according to the substitution rule.
A data processing apparatus according to any one of claims 4 to 6. The input unit receives item selection and item name input.
The editing unit changes the name of the selected item to the input item name.
The data processing device according to any one of claims 4 to 7. The input unit receives selection of a target item and a horizontal axis item.
The editing unit sets the target item on the vertical axis, and creates a graph in which the horizontal axis item is set on the horizontal axis.
A data processing apparatus according to any one of claims 4 to 8. Obtaining data having a plurality of records each including the plurality of fields corresponding to a plurality of items;
Accepting selection of one of the plurality of items and input of a division ratio of the data;
A field corresponding to the selected item is extracted from each of the plurality of records, and based on the extracted field, the data is divided into a plurality of data including machine learning learning data and verification data at the division ratio. To divide and
Performing machine learning processing based on the learning data, and verifying a learning model generated by the machine learning processing based on the verification data;
Data processing method including: Computer,
An acquisition unit for acquiring data having a plurality of records, each including the plurality of fields corresponding to a plurality of items;
An input unit that receives selection of one of the plurality of items and input of a division ratio of the data;
A field corresponding to the selected item is extracted from each of the plurality of records, and based on the extracted field, the data is divided into a plurality of data including machine learning learning data and verification data at the division ratio. Division part to divide,
A learning processing unit that performs machine learning processing based on the learning data, and verifies a learning model generated by the machine learning processing based on the verification data;
Data processing program to function as.