JP2020149693A - Generation device, generation method, and generation program - Google Patents

Abstract

To provide a generation device, a generation method, and generation program, which improve accuracy of AI models.SOLUTION: An information provision device 10 provided herein comprises an acquisition unit 41 for acquiring learning data used for learning a model, and an index generation unit 42 configured to generate a generation index for the model based on statistical features of the learning data.SELECTED DRAWING: Figure 36

Description

The present invention relates to a generator, a generator and a generator.

Artificial intelligence models have been developed in various prior art initiatives. Specifically, data has been acquired and models have been generated using machine learning. Conventional techniques for generating artificial intelligence models have required considerable manual work (eg, human intervention), including data acquisition and data model testing and evaluation.

In recent years, a technique has been proposed in which various models such as SVM (Support vector machine) and DNN (Deep Neural Network) are trained to learn the characteristics of the training data so that the models can perform various predictions and classifications. As an example of such a learning method, a technique has been proposed in which a learning mode of learning data is dynamically changed according to a hyperparameter value or the like.

JP-A-2019-164793

Conventional methods have various problems and drawbacks. For example, but not limited to, manual work related to model generation is to provide access to entities such as developers, programmers, analysts, and testers so that they can access personal data. Become. Sensitive and / or personal information such as purchasing, consumption habits, etc. can be accessed throughout testing and evaluation, training in model generation or other aspects. Therefore, end users can be at risk as a result of potentially exposing sensitive and / or personal information. In addition, other entities, such as distributors and resellers, may be at risk due to data breaches or security breaches, or possible access to sensitive corporate information.

Moreover, once the prior art artificial intelligence models are generated, resizing these models requires extremely large amounts of capacity, such as computer processing power and storage capacity. The reason for the difficulty of this prior art is that the inputs and parameters associated with the artificial intelligence model are static and cannot be modified or optimized in an efficient way. For example, optimizing an artificial intelligence model involves manual intervention. This requires additional time and resources, which can be used for other tasks. Further, the manual optimization method of the prior art cannot optimize toward the overall optimum point, and it may be difficult to use the manual optimizer.

Therefore, one or more of the aforementioned prior art problems and / or shortcomings need to be addressed, but remain unaddressed.

Further, in the above-mentioned technique, there is room for improving the accuracy of the model. For example, in the above-mentioned example, the learning data to be learned of the feature is only dynamically changed according to the value of the hyperparameter or the like. Therefore, if the values of hyperparameters are not appropriate, it may not be possible to improve the accuracy of the model.

Here, it is known that the accuracy of the model changes depending on what value the training data is, what features the training data has, which features are trained, and the like. ing. In addition, the accuracy of the model also changes depending on the learning mode when the training data is trained by the model, that is, the learning mode indicated by the hyperparameters. From such many elements, it is not easy to select the most suitable element for learning a model according to the purpose of the user.

The generation device according to the present application is characterized by having an acquisition unit for acquiring learning data used for learning a model, and an index generation unit for generating an index for generating the model based on the characteristics of the training data. ..

According to one aspect of the embodiment, the accuracy of the model can be improved.

It is a schematic diagram of the implementation example. It is the schematic of the implementation example in the context of TensorFlow Extended. It is a figure of the process of the framework of artificial intelligence by the implementation example. It is the figure of the whole architecture of the implementation example. It is a figure of the implementation example of the feature function selection algorithm. It is a figure of the deep framework architecture by the implementation example. It is a figure of the work related to the deep framework by the implementation example. It is a figure of the model file by the implementation example. It is a figure of API by the implementation example. It is a figure of the implementation example explaining the mapping of different data types so that it can be mapped to various data density determinations, and the related feature functions that can be implemented. It is a figure of the user experience example related to the implementation example. It is a figure of another user experience example. It is a figure of another implementation example of a user experience. It is a figure which shows the comparison between the model which executes the implementation example with respect to the operating system as data. It is a figure which shows the comparison between the models which execute the implementation example for the age group as data. It is a figure of the user interface example. It is the figure (1) of the output of the implementation example. It is the figure (2) of the output of the implementation example. It is the figure (3) of the output of the implementation example. It is the figure (4) of the output of the implementation example. It is a figure of the implementation example related to the feature function bundle. It is a figure of the state which is the overlearning state measured by the implementation example. It is a figure of the state in the under-learning state measured by the implementation example. It is a figure of the solution space showing the result of a plurality of specific regions and the result of the global maximum. It is a figure of the framework of the tuner by the implementation example. It is a figure of the algorithm by the implementation example. It is a figure (1) of the result related to the work of the implementation example. It is a figure (2) of the result related to the work of the implementation example. It is a figure (3) of the result related to the work of the implementation example. FIG. 5 is a diagram of an example computing environment with examples of computer devices suitable for use in some implementation examples. It is a figure of the environment example suitable for some implementation examples. It is a figure of the graphical image of the difference between the approach by the prior art and the implementation example. It is a figure which shows an example of the information provision system which concerns on embodiment. It is a figure which shows the order which the information providing apparatus which concerns on embodiment optimize an index. It is a figure explaining an example of the flow of model generation using the information providing apparatus in embodiment. It is a figure which shows the configuration example of the information providing apparatus which concerns on embodiment. It is a figure which shows an example of the information registered in the learning database which concerns on embodiment. It is a figure which shows an example of the information registered in the generation condition database which concerns on embodiment. It is a flowchart which shows an example of the flow of the generation processing which concerns on embodiment. It is a figure which shows an example of a hardware configuration.

Aspects of the implementation relate to methods, systems, and user experiences related to generating and optimizing artificial intelligence models while simultaneously minimizing manual intervention.

According to the embodiment of the implementation example, a computer implementation method is provided for generating and optimizing an artificial intelligence model. This method includes steps to receive input data and labels, to perform data validation to generate a configuration file, to split the data to generate split data for training and evaluation, and to generate split data. It includes the steps of performing training and evaluation to measure the error level, and performing actions based on the error level. This action includes modifying the configuration file, automatically tuning the artificial intelligence model, generating the artificial intelligence model based on training, evaluation and tuning, and serving the model in production. At least one of is included.

According to another aspect, the training involves automatically optimizing one or more pre-filled identities associated with the populated data, automatically optimizing the hyperparameters associated with the generated artificial intelligence model. It includes steps to, and automatically generate an update model based on one or more optimized input qualities and optimized hyperparameters.

According to another aspect, one or more input features are optimized by a general algorithm, optimizing a combination of one or more input features, and a list of the optimized input features. To generate.

According to still one aspect, the step of automatically optimizing hyperparameters involves applying at least one of Bayesian algorithms and random algorithms that optimize based on hyperparameters.

According to still one aspect, the step of automatically optimizing one or more input features is performed in a first iteration loop that is executed until the first specified number of iterations is met, and is hyper. The steps of automatically optimizing the parameters and automatically generating the update model are performed in a second iteration loop that is executed until the second specified number of iterations is met.

According to another aspect, the first iterative loop and the second iterative loop are iteratively executed until the third defined number of iterations is met.

According to another aspect, performing training and evaluation involves performing one or more feature functions based on the data type of the data, the density of the data, and the amount of data.

Implementation examples can also include non-transitory computer-readable media with storage devices and processors capable of executing instructions that generate and optimize artificial intelligence models.

The following detailed description presents the details of the charts and implementation examples of the present application. Reference numbers and descriptions of overlapping elements between figures and tables are omitted for clarity. The words used throughout the description are presented as examples and are not limited to them.

Implementation examples cover methods and systems for creating artificial intelligence models while minimizing manual human intervention required by prior art approaches. More specifically, implementation examples include data frameworks, deep frameworks, and tuner frameworks. The data framework includes data validation, generation of configuration files required for deep frameworks, and / or organization of training, evaluation and testing data. A deep learning framework (eg, a deep learning framework) builds a production deep learning model without the need to generate separate code. The tuner framework optimizes one or more hyperparameters and their combinations for the data framework, as well as the combination of input features, feature types and model formats. For example, but not limited to this, this implementation example is Python 2.7 or Python 3. It can be performed by using X and using TensorFlow 1.12.0 or later, and other implementations, as will be appreciated by those skilled in the art, without departing from the scope of the invention. Can be used instead of.

FIG. 1 is a schematic view 100 of an implementation example. According to schematic 100, data 101 and label 103 are provided as inputs. For example, but not limited to this, the data 101 may be in TSV TFRecord or HDFS format and the label 103 may be provided as a string. A data framework, a deep framework, and a tuner framework are displayed on the 105. As the output 107, a model such as a model served by TensorFlow is provided in production. As a single command, the implementation example shown in 105 of the present specification can be executed by a user, for example.

In the context of TensorFlow Extended, this implementation example can optionally be integrated as follows: More details are as shown in FIG. 2, 200. TensorFlow Extended provides an integrated front end 201 for job management, monitoring, debugging, and data / model / evaluation visualization, as well as the shared configuration framework shown in 203, and job organization. In this implementation example, the tuner framework 205 can be integrated therein. In addition, data framework 207 provides data analysis, data transformation, and data validation, and deep framework 209 provides trainers, model evaluation and validation, and serving. Further, implementation examples can integrate aspects such as the shared utility of unwanted data collection and data access control shown in 211, and pipeline storage shown in 213 with TensorFlow. Therefore, the artificial intelligence model can be created for production with only the configuration file and the initial data.

As described herein, implementation examples include automatic optimization. For example, but not limited to, optimizations can be performed on input feature combinations, input feature column formats, input cross features, and input embed sizes. In addition, optimizations can be performed on model selection, model architecture, model network / connectivity, model hyperparameters, and model size.

For pipelines with implementation examples, the artificial intelligence framework, which can be integrated with TensorFlow Extended, has a variety of steps. FIG. 3 shows the process of the artificial intelligence framework 300 according to the implementation example. For example, a process can include, but is not limited to, job / resource management 301, monitoring 303, visualization 305, and execution 307 (eg, Kubernetes), followed by data framework 309, deep framework 311, and. The tuner framework 313 follows, followed by rollout and serving, logging, training hardware, and inference hardware. For example, but not limited to, the data framework 309 can include data analysis, data transformation, data validation, and data partitioning (after the data acquisition process). Deep framework 311 can include, for example, trainers, model building, model validation, training on scales, interfaces with training hardware, rollouts, logging and serving interfaces with interfering hardware.

According to the architectural example, tab-delimited values or TSV format data configuration files and input data are provided to the data framework. The data framework performs data validation and generates configuration files for deep frameworks, including schema, identity, model and cross-feature files, as well as validation reports. The data framework also divides the data for training, evaluation, and optionally testing and / or prediction.

The output of the data framework is provided to the deep framework. The deep framework outputs a model analysis report at the same time as the output to the production model, and performs training, evaluation and testing, export of serving model, model analysis and serving.

The configuration file is also provided to the tuner framework, which uses the optimizer configuration file to provide input feature and hyperparameter optimization, automatic model selection and automatic machine learning.

Regarding the execution of the above-mentioned architecture, the work is performed as follows. First, the input data is prepared, and the input data is provided in the TSV format without a header or the TSV format without a header and a schema configuration file. Next, data validation is performed and the configuration file for the deep framework is exported. In addition, the data is divided by the data framework for training, evaluation and testing. A confirmation is then provided as to whether the training can be performed by the deep framework. In addition, the tuner framework can perform hyperparameter optimizations and optimization of combinations of entered features, feature typologies, and model formats. The model serving (eg, providing predictions or using the probability of output by applying the model) and inference can then be performed by the deep framework.

FIG. 4 is a diagram of the entire architecture 400 of the implementation example. As mentioned above, the data 401 and the label 403 are provided as inputs, and the input data configuration file 405 may optionally be provided. Data validation 409 and data division 411 are performed in the data framework 407. As a result of data validation 409, validation report 413 and configuration file 415 for deep frameworks are generated. In the data division 411, the data is divided for training, evaluation, and optionally for testing and prediction, as shown in 417.

The output of the data framework to the deep framework 419 is the configuration file 415 and the split file 417. In deep framework 419, training, evaluation and testing, serving model export, model analysis and serving are performed. In addition, the tuner framework 421 interfaces with the deep framework 419, as described in more detail below. The tuner framework automatically optimizes the entered features and hyperparameters and provides automatic model selection. Optionally, an optimizer configuration file 423 can be provided. As output, the tuner framework 421 provides the most preferred configuration file 424 for model optimization, as well as report 425. The deep framework 419 provides the serving model 427 in production as its output, and at the same time provides the model analysis report 429.

The implementation examples described above can be performed through a non-transitory computer-readable medium containing instructions for executing the methods and systems described herein. For example, but not limited to, an instruction can be executed on a single processor in a single machine, on multiple processors in a single machine, and / or on multiple processors in multiple machines. it can. For example, in a single server with one CPU and one GPU, the implementation example can execute instructions on that GPU, and in a single server with multiple GPUs, processing can be done on some GPUs or all. It can be executed in parallel using the GPU of. In a multi-CPU, multi-server environment, load distribution technology is used in a way that optimizes efficiency. According to one implementation example, the kukai system developed by Yahoo Japan Corporation may be adopted.

For the data framework, as previously disclosed, data validation, generation of configuration files for deep frameworks, and division of training and evaluation and test data are performed. Implementation examples related to these schemes are described in more detail below.

For example, but not limited to, Deep Framework 1.7.1 or later can be used with the data framework according to the implementation examples. However, other approaches or schemes can be submitted for that purpose within the implementation without departing from the scope of the invention. In addition, a data file is provided as input to the data framework. In this implementation example, the data format can support TSVs, with favorable specifications on the first line of the data file, or on the Deep Framework schema. Provided in yaml. Furthermore, the data structure file is DATA. Provided as yaml.

According to the framework data validation, the data validation is performed as described below. Data validation has a function that specifies which columns to ignore. If you specify columns to ignore, these columns will not be exported to the configuration file. Optionally, the columns can be weighted. More specifically, weight columns can improve the performance of the model if the numbers of the label classes in the data are not uniform. More specifically, the weight column can be multiplied by the loss of the sample. Semantically, the weight column may be a string or numeric column representing the weight, and this weight column is used to reduce the weight or increase the sample during training. The column value can be multiplied by the loss associated with the sample. If the value is a string type, use it as a key to derive the weight tensor from the features. If the column value is a number, pull out the RAW tensor, then apply the normalizer and apply the weighted tensor. Furthermore, the maximum number of records to be read can be specified.

In addition, a density ratio threshold can be specified, distinguishing between dense and sparse densities and implementing this function as described below. Similarly, you can specify a maximum threshold to distinguish between small and large amounts of dense data, as well as a unique total threshold to distinguish between small and large amounts of sparse data. Can be provided. In addition, the column name of the user ID can be specified, and the relationship between the recount and the user can be reported.

As part of data validation, it is possible to provide a unique total threshold to distinguish between small and large amounts of data, as well as a total threshold to distinguish between large and very large values. it can. You may optionally specify the number of buckets. There are two types of partition output related to the bucketing function, as described below. The first section defines the distinction between the maximum value and the minimum value according to the specified number. The second compartment is defined to be classified into buckets of approximately the same size. The actual number of buckets calculated may be less than or greater than the number of requests. These partitions can be used to optimize the model optimizer's feature functions, as described below.

In addition, the data framework divides the data into training data, evaluation and test data. For example, but not limited to this, the ratio of each data file can be specified so that the total value is 1.0. Alternatively, the ratio can be calculated automatically based on the data size. In addition, optionally, data export of records to each data file can be performed based on its value being set to "true". Further, the dataset can be divided into each user ID at a specified ratio based on the column name of the user ID, and the dataset is based on the time stamp by specifying the column name of the time stamp. It can be divided after classification.

According to the embodiment, the work of the data framework is performed as follows. First, the task of validating the data and the deep framework configuration file is performed by referring to the above implementation example of the data validation function. After the data validation work has been performed, reports and histogram files can be generated and reviewed. For example, but not limited to, validation reports can provide information about data errors or warnings about data problems. In addition, a report log can be generated to provide information such as density.

After inspecting the validation report and histogram files, you can validate the deep framework configuration file. In addition, the task of dividing the data can be performed, followed by a comparison of the training data with the evaluation data. The result of the comparison can also be verified.

According to the implementation example, the feature function selection algorithm is provided as follows. FIG. 5 is a diagram of an implementation example of the feature function selection algorithm 500. For integer type data, the density is determined in 501 based on the ratio of the unique total number to the maximum value + 1. If at 503 it is determined that the density is greater than or equal to the threshold, at 505 the data is characterized as dense. If the density is determined to be below the threshold in 507, the data is characterized as sparse in 509. For data characterized as dense in 505, it is determined whether the maximum value is greater than or equal to the small amount constant. If it is greater than or equal to a small number constant, dense data is characterized as large at 511, and at 513 a classification column with an ID is executed, as well as embedding. On the other hand, if it is determined that the maximum value is less than the small amount constant, the data is characterized as dense and small in 515 and executed using the classification column with the ID in 517.

For the sparse data determined in 509, the unique total number of data is compared to the threshold. If 509 determines that the total number of uniques is greater than or equal to the threshold, then 519 the data is characterized as large and sparse, a classification column with a hash bucket is provided, and an embedded column is executed at 521. To. In contrast, if the unique total number is determined to be less than the threshold, the data is characterized as small and sparse at 523, a classification column with a hash bucket is provided and executed at 525. ..

For the string type data determined in 527, the unique total number is compared with the small constant. If it is determined in 529 that the total number of uniques is less than a small amount constant, in 531 it is determined that the string data is small and a classification column with a vocabulary list is provided and in 533 a classification column with a vocabulary file Is executed. If it is determined in 535 that the unique total number is less than a large number constant, it is determined in 537 that the string data is large, a classification column with a vocabulary file is provided, and an embedded column is executed in 539. If the total number of uniques in 541 is greater than or equal to a large number of constants, it is determined in 543 that the string data is very large, a classification column with a hash bucket is provided, and an embedded column is executed in 545.

For floating point data determined at 547, the data is characterized as either a bucketed column executed at 549 or a numeric column executed at 551.

Regarding the acquisition of data for a data framework, a user can provide information such as an ID, a time stamp, and a position from information related to a user device such as a mobile phone. Similarly, operating system information can be obtained from IP addresses, MAC addresses, or other available information associated with user equipment that has access to the system. Demographic information such as gender, age and occupation can be obtained from user profile data with the consent of the user. In addition, it should be noted that the user ID and additional information are encrypted and the developer cannot identify the user based on one or more types of information about the user.

Regarding the division of data, the training data and the evaluation data must be prepared separately for the machine learning method, and the data needs to be trained and evaluated. As mentioned above, the data framework provides training data and evaluation data. According to the implementation example, the training data and the evaluation data may be duplicated. In addition, the tests can be performed iteratively and the data may be shuffled at each iteration to provide optimal data test performance.

As explained below, the deep framework provides for data training, which is performed automatically without the need for the user or developer to provide code. Also, as described herein, mechanisms or methods are provided to detect data characteristics such as string, integer, and floating point, small or large quantities of data, as well as density-related information. ..

Therefore, the output of the data framework provides the model, schema, features, cross-features, and the data itself divided for training and evaluation testing, and optionally predictions. Based on this information, the deep framework is implemented as described below.

As shown in FIG. 6, the deep framework architecture 600 goes through the deep framework 611 having the interface 613 and, as described above, the configuration files (eg, model, schema, feature, and cross-feature configuration files). 601 to 607) and receiving data 609. Further, the deep framework 611 provides an estimator 615, a core 619 interfaced with the tuner framework 621, a production model 623 and a report 625, which will be described a little further below.

More specifically, as shown in FIG. 7, a series of work 700 related to the deep framework is provided. The data framework prepares the data in 701 and creates the configuration file in 703. The deep framework has training 705 and rating 707 based on the configuration file received from the data framework. With a high training error rate, feedback to the data framework provides a larger model, longer training, and / or a new model architecture, or performs automatic tuning by the tuner framework, as shown in 709. Will be done. If the evaluation error rate is high, the feedback to the data framework will provide a modified configuration file that incorporates more data, normalization and / or a new model architecture, or a tuner as shown in 711. It will provide the framework to perform automatic tuning. Upon completion of training and evaluation by the deep network, a test phase is performed at 713, a model export is performed at 715, and a serving is performed at 717.

As mentioned above, the input data file is optionally provided in TSV format, no header, and TFRecord. Optionally, the implementation example converts TSVs and TFRecords by using a conversion function, specifying the number of export records to convert, and specifying the schema file if the input TSV file does not contain a header. The method of doing so can be included.

The configuration file is provided as having a schema file containing the column ID and column name, the array matches the input data file, and the column name is sensitive. Deep frameworks can convert configuration files into functions such as the TensorFlow function. More specifically, by using the column name as a key, the configuration file is transformed into a function, but the parameter name and function name can be retained. In addition, some can be omitted or set to default.

Once generated and given a numeric reference, the function can be used for automatic optimization of feature functions associated with the model optimizer, and you can specify as many values as you need. This has been mentioned above in relation to buckets for feature function algorithms and data frameworks.

One or more basic feature functions can be provided. These feature functions can be selected for use based on the feature function algorithm, as described above in relation to the data framework. For example, a classification column with an ID and a classification column function with an ID and an embedded column can be used when the input is an integer in the range from zero to the number of buckets. The feature function of a classification column with a hash bucket and a classification column with a hash bucket and an embedded column can be used when the features are sparse and the ID is set by the use of a hash. A classification column with a vocabulary list and a classification column identity function with a vocabulary list and an embedded column are provided when the input is in string or integer format and an in-memory vocabulary that maps each value to an integer ID is provided. Can be used for.

A classification column with a vocabulary file and a classification column identity function with a vocabulary file and an embedded column are provided when the input is in string or integer format and a vocabulary file is provided that maps each value to an integer ID. Can be used. Numeric column feature functions are provided, and the data show the numeric features that have values. In addition, the feature function of the bucketed column is provided for the data showing the discretized and dense input. Further, in relation to the one or more feature functions described above, a continuous feature function can be provided to handle consecutive values.

As shown in FIG. 8, for the model file, the model 800 may be linear, such as wide model 801 or deep model 803, or a combination of wide and deep models. The model setup can have one or more classifier classes and one or more regression classes. In the context of personalized recommender systems, user information 805 such as user IDs, demographics, operating systems, and / or user devices or user devices can be provided. Similarly, product information 807 such as product ID, name, tag, category, issue date, and supplier can be provided.

The work of the feature function can be performed as described above. For sparse features, work is performed at 809 as appropriate, and wide model 801 or deep model 803, or a combination thereof, can be performed based on the result of the work of the feature function. In 811 for dense embedding, another task can be performed as shown as hidden layer 813, based on the determination of the feature function as described above for the implementation of the deep model 803. Further, the output unit 815 is provided for serving models and the like.

In summary, user information and product information are provided to the data framework to determine the sparseness of features. If the features are dense enough, dense embedding is performed and deep generalization is performed to produce output through the hidden layer, as described above in relation to the feature function model. Alternatively, in the absence of dense embedding, wide memorialization is performed to produce the output. The output unit can provide a probability result for any or all commodities.

One or more APIs can be provided to support deep frameworks. FIG. 9 is a diagram showing an API 900 according to an implementation example. For example, but not limited to this, the API can be provided in REST at 901, including a client input 905 to a serving server 907 such as a TensorFlow serving container, the serving server 907 generating a model replica and servicing it. Synchronize with model 911. In addition, the API provides training 913 through model building, which includes experimentation, idea generation, and modification of configuration files based on the generated ideas. In addition, a Python API (eg, gRPC) can be provided in 903, which is similar to the REST API associated with elements 907, 911, and 913. Further, the Python API can be provided with information from a client, middleware including a preprocessing logic circuit 917 and a postprocessing logic circuit 919, and an API interface 915 with a gRPC client 921.

FIG. 10 is a diagram of an implementation example showing mapping 1000 representing various data types that can be mapped to various data density determinations, and related feature functions that can be implemented. For example, an integer data model is shown in 1001, which includes identifiers such as user IDs and product IDs, numerical values such as age, year, month, and day, and categories such as device, gender, and OS. Furthermore, the Boolean data type is shown in 1003 as a flag type such as click, the character string data including tags and queries is shown in 1005 as a lexical type, and the floating point data is the actual temperature, weight, height, price, etc. It is shown in 1007 as a numerical type.

If it is determined in 1017 that the data is dense and contains a small amount of data, the feature function of the classification column with the ID is applied in 1027. If the data is determined to be dense and large in 1015, the feature function and embedding of the classification column with the ID is applied in 1029. If the data is determined to be sparse and small in 1013, a classification column feature function with a hash bucket is applied in 1031. If the data is determined to be sparse and large, a feature function and embedding of a classification column with a hash bucket is performed in 1033. If it is determined in 1009 that the data has been bucketed, then in 1035 the data is evaluated as a bucketed column as a feature function. If none of the above data verdicts are met, the data is characterized as a numeric column at 1037.

Further, regarding the data type of the character string value, when it is determined in 1019 that the amount of data is small, the identity function of the classification column having the vocabulary list and the classification column having the vocabulary file is applied in 1039. If it is determined in 1021 that the data is large, the feature function and embedding of the classification column with the vocabulary file is applied in 1041. If it is determined in 1023 that the data is very large, the feature function and embedding of the classification column with the hash bucket is applied in 1043.

For the data type determined to be a floating point type in 1007, the data is determined to be bucketed in 1025, and the feature function of the bucketed column is applied in 1045. Alternatively, the feature function of numeric column 1037 is applied to floating-point data.

For example, but not limited to, a reference classifier can be provided, which omits the feature values and establishes the simple criteria provided to predict the average value of each label. For single-label problems, the reference classifier can predict the distribution of probabilities for a class as well as labels. For multi-label problems, the reference classifier can predict the percentage of samples that are positive for each class.

In addition, a linear classifier can be provided to train a linear model to classify an instance into one of several possible classes. For example, but not limited to this, if the number of possible classes is 2, this is a binary classification. In addition, a DNN classifier can be provided to train the DNN model to classify the instance into one of multiple possible classes, thereby if the number of possible classes is two. , This is a binary classification. It is also possible to provide a combination of a linear classifier and a DNN classifier, thereby combining the above-mentioned linear classifier model and a DNN classifier model. In addition, models such as AdaNet can be provided or combined with the classifier. Train a recurrent neural network model to instantiate an instance into one or another classifier of multiple classes (eg, DNN with a residual network, or automatic nature interaction learning using a self-attention neural network). The TensorFlow RNN model to classify will be understood by those skilled in the art.

Similarly, the regressor can be provided to the above-mentioned classifier, the feature value can be omitted, the average value can be predicted, and an estimate can be provided.

The model can have one or more functions. For example, but not limited to, one or more functions can have a stop function, which does not decrease the metric within a given maximum step, does not increase within a given maximum step, and thresholds. Stop training under certain conditions, such as above or below the value. Other functions may be included, not limited to the cases described above, as will be appreciated by those skilled in the art.

As mentioned above, datasets and configuration files can be used to perform training and evaluation. This training and evaluation runs on a single machine with one CPU or GPU, which can be used automatically when available. In addition, the process can be parallelized for multiple devices and a specified number of GPUs or CPUs can be specified. The process can be executed in the background, the console log is displayed, and the process can be stopped at will.

According to the implementation example, the test model works, the predictive model works, followed by the model analyzer. The export is then performed on the cervical model, the model server is started, and the interface with the REST and Python APIs continues to operate as described above.

Optionally, TensorBoard can be used to visualize the deep framework. For example, the TensorBoard can be viewed when run, displaying training and evaluation data graphically, as well as providing a graph of the work and further displaying the data.

For example, FIG. 11 is a diagram showing a user experience example related to an implementation example adopting TensorBoard. In 1101, the user selects "scalars". In 1103, the comparison between the training data and the evaluation data is displayed in a graph format. In 1105 and 1107, the curves of the training data and the evaluation data, respectively, are shown to represent the loss.

FIG. 12 is a diagram showing another user experience example. More specifically, in 1200, an image of the trace structure is shown, in which the user has selected "graphs" at 1201. In 1203, the relationships between the components are displayed in a graph.

FIG. 13 is 1300, which provides another implementation of the user experience. More specifically, in 1301 the user selects "projector" and in 1303 the user selects the kernel. As a result, the data display is shown at 1305.

The deep framework has a model analyzer. More specifically, the model analyzer will export and generate accurate reports for each column associated with the entered data. For example, but not limited to, if the entered data includes a user ID, operating system, and agent address, an accurate report will be generated for each of these columns. More specifically, it is possible to determine whether the user has high accuracy or low accuracy for a predetermined model. Similarly, judgments can be made on the types of data that may need to be improved in model accuracy and on the missing data.

According to one implementation example, accuracy is determined for each of the two models, android and iOS. The output of the model analyzer is provided with an accuracy score between 0.0 and 1.0 for each model, for android and iOS respectively. As you can see, android was more accurate than iOS on both models. In addition, the total number of data is provided for both android and iOS to validate the amount of data. In addition, the output demonstrated that the second model has high accuracy for both android and iOS operating systems.

For example, as shown in FIG. 14, comparisons 1400 between models are provided for android and iOS operating systems, respectively. As shown in 1401, for both model A and model B, android shows higher accuracy in comparison to iOS. Moreover, between model A and model B, model B shows better accuracy in comparison with model A. In addition, 1403 indicates the number of data for the operating system.

The age of the user was provided as an input to the model analyzer, and the accuracy of each age group was determined for each model. It will be appreciated that the second model provides high accuracy for most age groups. In addition, the number of data is also provided to age groups.

For example, as shown in FIG. 15, for each of model A and model B, comparison 1500 is provided across age groups. As shown in 1501, model B has higher accuracy in most age groups when compared to model A. In addition, 1503 indicates the number of data for the age group.

In addition, the implementation example provides a tool called "what if", which allows you to scrutinize the model in detail without coding. When you run this tool, you can enter data, model information, and model model. For example, FIG. 16 is a diagram showing such a user interface example 1600. Entering this information can produce output, such as displaying the data visually. In addition, it provides a data point editor to modify feature values to activate updated inferences, set ground truth feature criteria, compare fair metrics, or review performance, page, user ID, timestamp. It is possible to visualize the display of numerical features for various entered features such as stamps. For example, such outputs are shown in FIGS. 17-20.

In addition to automatic tuning, manual tuning options can be provided, as described below for the tuner framework. More specifically, in some artificial intelligence models, the results of the tuner framework may not fully meet the developer's customization requirements. In such situations, and optionally in situations that require user consent, the model can be customized after the output of the tuner framework. To fully automate the process, you can optionally disable or disable the manual tuning option. This can prevent manual access to potentially sensitive information. Further, it is possible to provide a hybrid approach that combines some aspects of automatic tuning described herein in an implementation example, and a related manual tuning approach.

The deep framework implementation example described above can be run on a sample dataset. In addition, multiclassification, binary classification, and regression can be performed on one or more sample datasets. FIG. 21 is a diagram showing an implementation example related to the handling of the feature function. More specifically, as shown in 2100, we show a plurality of scenarios related to the work of performing feature functions, transforming functions and classification. In 2101, the feature functions of the classification column with the hash bucket, the classification column with the vocabulary list, the classification column with the vocabulary file, and the classification column with the ID are executed to provide the output. The function can be executed on the output and a linear classifier and linear regressor can be executed based on the determination that the data is sparse. On the other hand, in the case of determining that the data is dense, a classification function can be further executed as shown in 2101. Similarly, when the embedding is performed, the schema is shown in 2103. On the other hand, when the feature function is a numerical column, the determination of whether or not the data is dense is performed at 2105, and the classification is executed as shown therein. For bucketed columns, a numeric column is defined in 2107, it is determined that the data is dense, and various classifications are performed.

According to the implementation example, it is possible to identify an overfitting scenario in which the loss of evaluation data exceeds the loss of training data. In such a situation, as shown in FIG. 22 (for example, a case where there is a large difference in loss between the evaluation data and the training data), the configuration file is made by requesting more data and regularization. Judgment may be made to modify or provide a new model architecture.

As shown in FIG. 23, when the loss of training data and evaluation data is large, there is a possibility of underfitting. In such situations, the configuration file may be modified to provide a larger model, train longer, or adopt a new model architecture.

According to the implementation example, the deep framework exports data-related statistics based on the result and data type of the feature function to work with the data and provide recommendations for the best choice. ..

Implementation examples using the deep framework provided herein allow automatic selection of features using density for ranges. For example, for various data types, including, but not limited to, whether the data density is sparse, dense, or dense is taken into account. Embedding can be performed on sparse data with a large sample size. In addition, for dense or very dense data, a decision may be made to see how much data is present, based on whether the threshold is met. Embedding is performed. IDs can be used to classify data if such thresholds are not met or if lower thresholds are provided.

If the data is not dense enough, it may not be possible to classify. Furthermore, if the data is sparse, hashing can be performed to avoid displaying the ID. Using this embedded model, the implementation example determines if the threshold is met. In this way, model optimization can be provided, and model selection can be made, for example, either randomly or based on Bayesian optimization, as described in connection with the tuner framework. Can be executed.

As mentioned above, a tuner framework is provided to automatically optimize hyperparameters, combinations of input data, and model selection. In some environments, it is not possible to manually obtain the optimum value for the model. For example, as shown in 2400 of FIG. 24, in the solution space 2401 sandwiched between the first hyperparameter 2403, the second hyperparameter 2405, and the object 2407, the result 2409 of a specific region is obtained by manual optimization. Can be provided. However, the global maximum result 2411 cannot be obtained by manual optimization alone. In addition, manual optimization may allow the operator to view user and / or product data in a non-privacy-protected format.

Therefore, this example provides random search algorithms and Bayesian optimization algorithms and is provided in the context of deep and data frameworks. More specifically, as shown in FIG. 25, the tuner framework 2500 includes a deep framework configuration file 2501, an optimizer file 2503, and, for example, the input data 2505 in the TSV format described above.

More specifically, the model optimizer 2507 received the generated configuration file 2509, used the optimizer in 2511 to evaluate the model with the generated configuration file, analyzed the results in 2513, and output in 2515. The report is provided in report format 2519 for the deep framework configuration 2517.

As mentioned above, the configuration file may be generated by the data framework and provided directly to the tuner framework with or without editing. In the configuration file, specify metric tags such as average_loss (MINIMIZE) for the regressor model and accuracy (MAXIMIZE) for the classifier model. In addition, you need to specify either a random search or Bayesian optimization algorithm, and the maximum number of model parameters allowed.

According to the implementation example, the random search algorithm can be executed as follows, as shown in 2600 of FIG. In the first task 2601, the input features are optimized, and this operation is repeated as long as the number of trials is less than the number of trials of the input features. In the second operation 2607, hyperparameter optimization 2603 and model automatic selection 2605 are executed. These tasks are performed as long as the number of trials is less than the number of model trials. At 2609, as long as the first and second operations are performed in loops and the number of loops is less than the number of loops required to perform a random search.

When the random search execution model is executed, if the result is "false", the first work is executed, and then the second work is executed, as shown in 2611. On the other hand, when the result is "true", this work is canceled, the second work is executed, and then the first work is executed, as shown in 2613.

In relation to the first work, this is the setting of the work of the input features, but in order to automatically extract the optimum configuration of the input features, the implementation example is executed as follows. Once the number of trials has been proven and optimization is possible by setting the feature column function type to "true" to generate the feature function as described above, the function performing the random search is based on the best results. Judgment is made as to whether or not "true" is set for the input feature. In this case, a general algorithm is used to optimize the combination of input functions. Optionally, it can provide a list of entered features used in each task during the optimization process, as well as the number of iterations and results per trial that will be passed on to the next hyperparameter optimization process. Can be provided. As a result, automatic optimization of the entered features is performed.

For the second task, hyperparameter optimization is provided. For example, the value of a certain parameter can be optimized based on the number of trials setting and the algorithm set in "Bayesian optimization". In addition, the number of trials and iterations can be set. The model optimizer configuration file is then inspected and edited as needed. The previous results of the tuner framework are removed, followed by the tuner framework with the dataset running using the configuration file and the data provided by the data framework.

The tuner framework can run on a single device on a single machine with a CPU or GPU, or on multiple devices on a single machine with a GPU that is automatically used across CPUs. it can. The processing of the tuner framework runs in the background.

Multiple devices on one machine can be used to provide parallelization of several CPUs or GPUs. In addition, the tuner framework can be run on multiple devices on multiple machines. Optionally, the tuner framework can generally be chosen to automatically use the GPU across CPUs on a given machine or machines. More specifically, running the tuner framework would modify the server list file and run the tuner framework using multiple devices on multiple machines.

As described herein, the tuner framework automatically tunes and creates artificial intelligence models. It uses a deep framework as a library, and based on the execution of that deep framework, the tuner framework provides an updated model, or recommended changes to the model. Optionally, a report may be provided to the user, which contains the display of incorrect, missed, or inappropriate data that needs to be modified and gives the user the opportunity to modify such data. provide. This option can be used to provide an opportunity for feedback and further refine the model. Performance can also be further improved by removing data that should not be included in the deep and tuner frameworks.

The implementation examples described herein have an input optimizer and a hyperparameter optimizer, which are implemented in the tuner framework to provide optimal model determination. The input optimizer provides optimization according to the RAW data provided by the user and determines the optimal combination of provided RAW data.

According to the implementation example, the tuner framework optimizer provides input optimization. In contrast, prior art techniques do not provide input optimization. Instead, prior art techniques attempt to aggregate all information, including all data, into a model, but do not provide input optimization after partitioning the data. Instead, prior art techniques seek to maximize the input data. However, in the implementation example, the tuner framework determines and selects the optimal combination of features, thereby selecting important information and parameters and removing noise. For example, the genetic algorithms described herein can be employed to optimize input. In addition, as also described herein, one or both of the random model and the Bayesian model are employed for hyperparameter optimization.

In addition, the implementation example provides an iterative approach. As described herein, iterative methods are provided in relation to input optimization, and independently, iterative methods are also provided in relation to hyperparameter optimization. In addition, input optimization and hyperparameter optimization are contained within an iterative loop. The inventor concludes that adding input optimizations and hyperparameter optimizations to the iterative loop can provide important and unexpected results.

When the execution of the tuner framework is completed, the processing result is confirmed. More specifically, it is possible to provide the highest numerical result of a given, in real time or based on a stop point, thereby revisiting the ranked result. Instead, a display tool such as TensorBoard can be used to display all the results, for example, the exact loss or the average loss.

The final result can be confirmed, and the log of the result can be exported together with the input feature information. The final results can be used to train the model with the best results, which allows the configuration file to be modified and the training to work again.

According to the examples, as shown in FIG. 27, the model optimizer can be operated on the data related to the problem type "map life magazine" of multiple classifications. As you can see from 2701 and 2703, two different processing models are used, with precision and recall increasing before and after optimization, respectively. In addition, features and models will be understood to be optimized before and after optimization.

As shown in FIG. 28, the 2801 uses an alternative hardware configuration, which substantially increases the processing speed. This can be indicated by the number of hours required to calculate the accuracy. This new version can show that the performance of parallel distributed processing is substantially improved in terms of processing time, as shown in FIG.

Output can be provided based on accuracy or possibility. In an example implementation of a user who is devoted to online search, such as searching to purchase a product or service, the search results can be ranked or sorted based on the probability that the product will be purchased by the user. Since the above implementation example provides services automatically, the operator does not have to manually review the information related to the user. Thus, this implementation example provides a privacy-protected approach and can use artificial intelligence technology to provide, for example, ranking output in online searches.

For example, according to one e-commerce model, the input data is user information including a user ID, demographic information, an operating system, a device used for searching, and the like. In addition, the entered data includes product information such as product ID, name, model, metadata, category, issue date, supplier, company name, etc., and the above data includes data frameworks, deep networks, and tuner frameworks. Can be used as an input to. The output of the model is the probability of occurrence of user- and product-related events such as purchases. As mentioned earlier, embedding is used to vectorize the data and evaluate the similarity between the data.

In addition, the implementation examples described above also provide candidate features. For example, but not limited to, export candidates and function types can be provided in automatic form, in addition to statistics and candidate features. It provides the best results from the best functions for the model and generates the parameters and inputs to use. The implementation example receives the base model and information extracted from the log such as model size, metric, and average loss. As a result, the user can understand the optimal model based on the information provided by the data framework, the deep framework, and the tuner framework.

The model can be used, for example, to predict the likelihood that a user will purchase a product, and based on the ranking of that likelihood, the product list or recommendations in the area prioritized for the user requesting a search. Can be provided. Instead, for a given product, the seller of the product can be provided with a ranking of users who may purchase the product. For example, for a website that offers a wide variety of products, optionally categorized, this implementation example may be a classified and ranked user output of the product based on the likelihood of purchase, or the user may product. Can provide user-ranked distributor output based on the likelihood of purchasing. Therefore, recommendations are automatically personalized to the user performing the search. The model automatically learns user preferences and user characteristics and applies the learned information to calculate the likelihood that a user will purchase one or more products.

Therefore, the implementation examples provide at least one or more benefits for the benefits associated with privacy protection. For example, implementation examples can be performed such that the data is provided and processed and output without any human intervention to access, review or analyze the data. In addition, implementations can also provide restrictions so that no one can access the data through processing.

Optionally, in combination with implementation examples, anonymization, pseudo-anonymization, hashes or other privacy protection techniques can provide additional security to the user data. As long as external access to the model is required, such access is only possible through the API as described above. In such situations, only the user, or the service concerned, may have access to the final result, not privacy-related information related to the data.

As will be appreciated by those skilled in the art, the data is considered to be arbitrary data, but according to some implementation examples, the data can have user behavior data. For example, but not limited to, user behavior data may include information about the user's demographics, which may be combined with other data entered into the data framework.

More specifically, training and inference can be performed to generate predictions in connection with artificial intelligence models, especially deep frameworks. The example implementations described above are intended for inferences used to generate predictions of user behavior associated with the product, depending on the outcome of the training described above.

FIG. 30 is a diagram showing an example computing environment 3000 including a computer device 3005 suitable for use in some implementation examples. The computing device 3005 in the computing environment 3000 includes one or more processors, cores, or processors 3010, memory 3015 (eg, RAM, ROM, etc.), internal storage 3020 (eg, magnetic, optical, semiconductor, etc.). Storage and / or organisms), and / or I / O interfaces 3025, all of which combine with a communication mechanism or bus 3030 to convey information, or into a computing device 3005. Can be incorporated.

The computing device 3005 can be communicably coupled with the input / interface 3035 and the output device / interface 3040. Either one or both of the input / interface 3035 and the output device / interface 3040 can be wired or wireless interfaces and can be removable. The input / interface 3035 may include any device, component, sensor, or physical or virtual interface, which can be used to provide input (eg, buttons, touch screen interface, keyboard, etc.). Pointer / cursor control, microphone, camera, Braille, human sensor, optical reader, etc.).

The output device / interface 3040 may include a display, a television, a monitor, a printer, a speaker, Braille, and the like. In some implementations, the input / interface 3035 (eg, user interface) and output device / interface 3040 can be integrated into or physically coupled to the computing device 3005. In other embodiments, other computing devices can function as or provide functionality as input / interface 3035 and output device / interface 3040 for computing device 3005.

Examples of the computing device 3005 are, but are not limited to, advanced mobile devices (eg, devices mounted on smartphones, cars or other machines, devices carried by humans or animals), mobile devices (eg, tablets). , Laptops, laptops, personal computers, portable TVs, radios, etc.), and devices not designed for mobility (eg desktop computers, server devices, other computers, information kiosks, one or more It may include a television, radio, etc. with or combined with multiple processors.

The computing device 3005 includes one or more computing devices of the same or different configurations and with external storage 3045 and network 3050 to communicate with any number of networked components, devices, and systems. It can be communicably coupled (eg, via the I / O interface 3025). Compute Device 3005 or any connected compute device acts as a server, client, thin server, general machine, purpose-built machine, or another indicator, or provides services for them. Or they can be called them. For example, but not limited to, network 3050 can include blockchain networks and / or clouds.

The I / O interface 3025 is limited to, but is not limited to, any communication, I / O protocol, or communication standard (eg, Ethernet, 802.1xs, universal system bus (USB), WiMAX, modem, mobile phone network protocol, etc. ) Has a wired and / or wireless interface capable of communicating information with at least all connected components, devices, and networks in a computing environment 3000. The network 3050 can be any network or any combination of networks (eg, internet, local area network, wide area network, telephone network, mobile phone network, satellite network, etc.).

The computing device 3005 includes temporary and non-temporary media, and can use and / or communicate with computer-enabled and computer-readable media. Temporary media include transmission media (eg, metal cables, optical fibers), signals, carrier waves, and the like. Non-temporary media include magnetic media (eg, disks and tapes), optical media (eg, CD ROMs, digital video discs, Blu-ray discs), semiconductor media (eg, RAM, ROM, flash memory, semiconductor storage), and others. Includes non-temporary storage or memory.

The computing device 3005 can be used to implement a technology, method, application, process, or computer executable instruction in some example computing environment. Computer-executable instructions can be received from temporary media, stored in non-temporary media, and retrieved from that non-temporary medium. Executable instructions can be created from one or more of any programming language, scripting language, and machine language (eg, C, C ++, C #, Java, Visual Basic, Python, Perl, Javascript, etc.). ..

The processor (s) 3010 can run under any operating system (OS) (not shown) in a native or virtual environment. One or more applications can be placed, logic unit 3055, application programming interface (API) unit 3060, input unit 3065, output unit 3070, data processing unit 3075, deep learning modeling unit 3080, automatic tuning unit 3085, And various parts include a partial communication mechanism 3095 for communicating with each other using the OS and using other applications (not shown).

For example, the data processing unit 3075, the deep learning modeling unit 3080, and the automatic tuning unit 3085 can implement one or more of the processes previously shown in connection with the above structure. The described parts and parts need not be uniform in design, function, configuration, or mounting, and are not limited to those described.

In some implementations, when information or execution instructions are received from API unit 3060, one or more other parts (eg, logic unit 3055, input unit 3065, data processing unit 3075, deep learning modeling unit 3080, And can communicate with the automatic tuning unit 3085).

For example, the data processing unit 3075 can receive and process input information, perform data analysis, conversion, and verification, and divide the data. The output of the data processing unit 3075 can provide a configuration file and data divided into test, evaluation, training, and the like. The data is provided to the deep learning modeling unit 3080, which performs training to build a model, verifying the performance of the model and scale training, followed by the final serving of the actual model. .. Further, the automatic tuning unit 3085 can provide automatic optimization of inputs and hyperparameters based on the information acquired from the data processing unit 3075 and the deep learning modeling unit 3080.

In some cases, the logic unit 3055 controls the information flow between the parts in some of the implementation examples described above, the API unit 3060, the input unit 3065, the data processing unit 3075, the deep learning modeling unit 3080, and the automatic tuning unit. It can be configured to manage the services provided by 3085. For example, the flow of one or more processes or implementations can be controlled by the logic unit 3055 alone or in cooperation with the API unit 3060.

FIG. 31 is a diagram showing an environment example suitable for some implementation examples. Environment 3100 includes devices 3105 to 3145, each of which is communicably connected to at least one other device via, for example, a network 3160 (eg, wired and / or wireless connection). Some devices may be communicably connected to one or more storage devices 3130 and 3145.

An example of one or more devices 3105 to 3145 may be the computing device 3005 shown in FIG. 30, respectively. Devices 3105-3145 are, but are not limited to, a computer 3105 (eg, a laptop computing device), a mobile device 3110 (eg, a smartphone or tablet), a television 3115, a vehicle-related device having a monitor and associated webcam. 3120, server computer 3125, computing device 3135-3140, storage devices 3130 and 3145 may be included.

In some implementations, devices 3105-3120 can be thought of as user devices associated with a user who can receive broadcasts remotely and can provide the user with settings and interfaces. Devices 3125-3145 may be devices associated with the service provider (eg, used to store and process information associated with document templates, third party applications, etc.).

The above implementation examples can provide various benefits and benefits to various entities.

In the implementation example, the end user can provide the information to the service. The service, on the other hand, can provide recommendations to the user. In the prior art, due to the manual intervention of computer programmers, data analysts, etc., the user's privacy information may be exposed to those subjects in order to perform model optimization. However, as described herein, implementation examples provide an automated approach that does not require the intervention of such an intermediary or subject. Therefore, the personal and privacy information of the user can be restricted from other users, developers and the like. Therefore, the benefit of privacy protection is in the implementation example.

Distributors who adopt this implementation example may not need to provide the platform with sensitive or personal information of the customer in order to realize the benefits of the artificial intelligence technique. Instead, the automation techniques described herein can be used to enable distributors, such as service providers, to obtain optimized model information while at the same time protecting user privacy. .. In addition, end-user privacy is further protected when input optimization provides the determination that less data is needed.

Similarly, the platform or developer can realize a variety of benefits and benefits. For example, but not limited to, the model can be optimized without the need for additional manual coding or input of information. The requirements imposed on the platform or developer can be limited to choosing the options to implement. If the developer needs to manually review or modify the model with the user's permission and wants to understand the parameters and change the entered data, the above-mentioned what if tool allows the user to do so. Allows you to take. For example, with the permission of the user, the developer can modify the input data and make it easier to get the result, but the input data is a model based on inference and optimization. Modified based on the tuner framework output associated with.

In addition, mobile device manufacturers, server manufacturers, or user equipment manufacturers such as data storage and processing entities can also realize a variety of benefits and benefits. As mentioned earlier, end-user data is treated in a privacy-friendly way, the tuner framework provides optimizations, and this optimization reduces the amount of information that needs to be provided by the device. Alternatively, data such as parameters can be restricted. For example, in some cases, if a GPS-based user location is determined to be a non-optimal input or parameter, the updated model may not request or collect information from such end-user devices. is there. As a result, the information that is acquired, sensed, collected, and potentially stored within the end-user device can be protected from use by the model. In addition, automation of data frameworks, deep frameworks, and tuner frameworks allows actors, developers, analysts, distributors, or equivalents on the platform to access potentially sensitive and personal information about users. No need. Therefore, devices associated with these entities do not need to be accessed by the user and can further gain privacy protection.

In one implementation, online retail-related entities such as online retailers, manufacturers, distributors, etc. can use the implementation to determine how to promote products and / or services. .. In such situations, implementation examples use tools, technologies, systems, and techniques described herein to retail online recommendations for advertisements that are most likely to drive users to purchase products. Can be provided to vendors. Conversely, when a user visits, browses, searches, or shop online on an online website, the implementation examples provide the user with recommendations based on what the user is most likely to need. can do. In addition, implementation examples can be associated with services such as those related to financial forecasting, promotion of sales of various services and products, recommendations for what to buy, and whether or not they bought it.

The above implementation examples can have various benefits and advantages. As shown herein, accuracy and relative operating characteristics can be substantially improved over conventional techniques by using implementation examples.

As shown in FIG. 32, a graphic representation 3200 is provided to show the differences between prior art methods and implementation examples in relation to the binary classification of financial models. More specifically, in 3201, the method of the prior art is shown by a broken line, and in 3203, the method according to the implementation example is shown. According to this implementation example, for exactly the same data, using the implementation example in comparison with the prior art, the accuracy is increased by 7.62% and the relative operating characteristics are increased by 2.78%. Will be understood.

In addition, implementation examples can be used to reduce unnecessary input data / parameters, dramatically reducing computational costs. The implementation example approach can provide additional benefits in that it can substantially increase processing speed and reduce the time it takes to process data on the model through optimization. Therefore, the model requires less processing and is beneficial for hardware systems as compared to prior art methods, without sacrificing accuracy.

Another advantage or benefit of this implementation is that the framework makes it easy to scale. For example, but not limited to, the tuner framework provides optimizations that can reduce the amount of data, inputs, parameters, etc., as described above. The result of this optimization is the ability to add scale without increasing the amount of computation, storage, communication, or other required resources compared to prior art techniques.

Further, according to the implementation example, as mentioned above, the tuner software provides optimization of the artificial intelligence model. For example, but not limited to, the model can be optimized for individual work types and provided as templates according to behavior type (eg, commercial). For example, but not limited to this, the difference between online grocery purchases, online car purchases, and online new mortgage loans is quite large, so various models are provided as templates based on prior optimization. can do. In contrast, prior art techniques do not provide templates for such models. This is because it creates a model but does not include implementation example optimizations as provided by the tuner framework.

Yet another benefit and advantage is that developers can experience ease of use. For example, but not limited to, users of the frameworks described in these implementations do not have to write any code for their own work, and at most developers will revisit their feedback, options, etc. You just have to select. As a result of this approach of providing automatic tuning, privacy is protected, as described above.

Although new implementation examples have been shown and described, these implementation examples are provided to convey the subject matter described herein to those familiar with the art. It will be appreciated that the subject matter described herein can be implemented in various forms without limitation to the implementation examples described herein. The subject matter described herein can be practiced without the use of specific definitions or items, or with other parts or items not described. Those familiar with the art will understand that changes can be made in these implementations without departing from the subject matter described herein, as defined in the accompanying claims and their equivalents. Will be done.

[Example of Embodiment]
Hereinafter, an example of a generation device, a generation method, and a generation program that realizes the above-mentioned various processes will be described.

The information providing device according to the following embodiment executes the following generation processing. First, the information providing device acquires learning data used for learning the model. Then, the information providing device generates a model generation index based on the characteristics of the learning data. For example, the information providing device generates an index for generating a model, that is, a generation index that serves as a recipe for the model to be generated, based on the statistical characteristics of the training data.

Hereinafter, a generator, a generation method, and a mode for carrying out the generation program according to the present application (hereinafter, referred to as “the embodiment”) will be described in detail with reference to the drawings. It should be noted that this embodiment does not limit the generator, the generation method, and the generation program according to the present application. Further, in each of the following embodiments, the same parts are designated by the same reference numerals, and duplicate description is omitted.

[1. Information provision system configuration]
First, the configuration of the information providing system including the information providing device 10 which is an example of the generating device will be described with reference to FIG. 33. FIG. 33 is a diagram showing an example of the information providing system according to the embodiment. As shown in FIG. 33, the information providing system 1 includes an information providing device 10, a model generation server 2, and a terminal device 3. The distribution system 1 may have a plurality of model generation servers 2 and a plurality of terminal devices 3. Further, the information providing device 10 and the model generation server 2 may be realized by the same server device, cloud system, or the like. Here, the information providing device 10, the model generation server 2, and the terminal device 3 are communicably connected via a network N (see, for example, FIG. 36) by wire or wirelessly.

The information providing device 10 executes an index generation process for generating a generation index which is an index (that is, a model recipe) in model generation, and a model generation process for generating a model according to the generation index, and the generated generation index and the generated index It is an information processing device that provides a model, and is realized by, for example, a server device or a cloud system.

The model generation server 2 is a generation device that generates a model in which the features of the training data are learned, and is realized by, for example, a server device or a cloud system. For example, when the model generation server 2 receives a config file such as the type and behavior of the model to be generated and how to train the characteristics of the training data as a model generation index, the model is automatically generated according to the received config file. I do. The model generation server 2 may learn the model by using an arbitrary model learning method. Further, for example, the model generation server 2 may be various existing services such as AutoML.

The terminal device 3 is a terminal device used by the user U, and is realized by, for example, a PC (Personal Computer), a server device, or the like. For example, the terminal device 3 generates a model generation index through communication with the information providing device 10, and acquires a model generated by the model generation server 2 according to the generated generation index.

[2. Outline of processing executed by the information providing device 10]
First, an outline of the processing executed by the information providing device 10 will be described. First, the information providing device 10 receives the indication of the learning data that causes the model to learn the features from the terminal device 3 (step S1). For example, the information providing device 10 stores various learning data used for learning in a predetermined storage device, and receives an indication of the learning data designated by the user U as the learning data. The information providing device 10 may acquire learning data used for learning from, for example, the terminal device 3 or various external servers.

Here, any data can be adopted as the learning data. For example, the information providing device 10 uses various information related to the user as learning data, such as a history of the position of each user, a history of web contents browsed by each user, a history of purchases by each user, and a history of search queries. May be good. Further, the information providing device 10 may use the user's demographic attributes, psychographic attributes, and the like as learning data. Further, the information providing device 10 may use the types and contents of various web contents to be distributed, the metadata of the creator, and the like as learning data.

In such a case, the information providing device 10 generates a candidate for the generation index based on the statistical information of the learning data used for learning (step S2). For example, the information providing device 10 generates candidates for a generation index indicating what kind of model and what kind of learning method should be used for learning based on the characteristics of values included in the learning data. In other words, the information providing device 10 generates a model capable of accurately learning the features of the training data and a learning method for causing the model to learn the features accurately as a generation index. That is, the information providing device 10 optimizes the learning method. It should be noted that what kind of learning data is selected and what kind of content generation index is generated will be described later.

Subsequently, the information providing device 10 provides a candidate for the generation index to the terminal device 3 (step S3). In such a case, the user U modifies the candidate of the generation index according to the preference, the rule of thumb, and the like (step S4). Then, the candidate of each generation index of the information providing device 10 and the learning data are provided to the model generation server 2 (step S5).

On the other hand, the model generation server 2 generates a model for each generation index (step S6). For example, the model generation server 2 causes the model having the structure indicated by the generation index to learn the features of the learning data by the learning method indicated by the generation index. Then, the model generation server 2 provides the generated model to the information providing device 10 (step S7).

Here, it is considered that each model generated by the model generation server 2 has a difference in accuracy due to a difference in the generation index. Therefore, the information providing device 10 generates a new generation index by a genetic algorithm based on the accuracy of each model (step S8), and repeatedly executes the generation of the model using the newly generated generation index (step S8). S9).

For example, the information providing device 10 is a model in which the learning data is divided into evaluation data and learning data, and the features of the learning data are trained, and a plurality of models generated according to different generation indexes are generated. get. For example, the information providing device 10 generates 10 generation indexes, and generates 10 models by using the generated 10 generation indexes and learning data. In such a case, the information providing device 10 measures the accuracy of each of the 10 models using the evaluation data.

Subsequently, the information providing device 10 selects a predetermined number of models (for example, 5) in order from the one with the highest accuracy among the 10 models. Then, the information providing device 10 generates a new generation index from the generation index adopted when the five selected models are generated. For example, the information providing device 10 regards each generation index as an individual of a genetic algorithm, and the model type, model structure, and various learning methods (that is, various indexes indicated by the generation indexes) indicated by each generation index. Is regarded as a gene in the genetic algorithm. Then, the information providing device 10 newly generates 10 next-generation generation indexes by selecting an individual to cross genes and crossing genes. The information providing device 10 may consider mutation when crossing genes. Further, the information providing device 10 may perform two-point crossing, multi-point crossing, uniform crossing, and random selection of genes to be crossed. Further, the information providing device 10 may adjust the crossover rate at the time of crossing so that, for example, the gene of an individual with higher model accuracy is inherited by the next generation individual.

In addition, the information providing device 10 generates 10 new models again using the next-generation generation index. Then, the information providing device 10 generates a new generation index by the above-mentioned genetic algorithm based on the accuracy of the new 10 models. By repeatedly executing such processing, the information providing device 10 can bring the generation index closer to the generation index according to the characteristics of the learning data, that is, the optimized generation index.

Further, the information providing device 10 satisfies a predetermined condition such as when a new generation index is generated a predetermined number of times, or when the maximum value, the average value, or the minimum value of the accuracy of the model exceeds a predetermined threshold value. If so, select the model with the highest accuracy as the provision target. Then, the information providing device 10 provides the terminal device 3 with the corresponding generation index together with the selected model (step S10). As a result of such processing, the information providing device 10 can generate an appropriate model generation index and provide a model according to the generated generation index only by selecting learning data from the user.

In the above-mentioned example, the information providing device 10 has realized the stepwise optimization of the generation index by using the genetic algorithm, but the embodiment is not limited to this. As will be clarified in the explanation described later, the accuracy of the model is not only the characteristics of the model itself such as the type and structure of the model, but also what kind of training data is input to the model and what kind of hyperparameters are used. It changes greatly depending on the index when the model is generated (that is, when the features of the training data are trained), such as whether the model is trained by using it.

Therefore, the information providing device 10 does not have to perform optimization using a genetic algorithm as long as it generates a generation index presumed to be optimal according to the learning data. For example, the information providing device 10 presents to the user a generation index generated according to whether or not the learning data satisfies various conditions generated according to the empirical rule, and follows the presented generation index. You may generate a model. Further, when the information providing device 10 accepts the modification of the presented generation index, the model is generated according to the received modified generation index, the accuracy of the generated model is presented to the user, and the generation index is generated again. You may accept the correction of. That is, the information providing device 10 may make the user U try and error the optimum generation index.

[3. About generation of generation index]
Hereinafter, an example of what kind of generation index is generated for what kind of learning data will be described. The following example is just an example, and any process can be adopted as long as the generation index is generated according to the characteristics of the learning data.

[3-1. About the generation index]
First, an example of the information indicated by the generation index will be described. For example, when the model is trained with the features of the training data, the model in which the mode when the training data is input to the model, the mode of the model, and the training mode of the model (that is, the features indicated by the hyperparameters) are finally obtained. It is thought that it contributes to the accuracy of. Therefore, the information providing device 10 improves the accuracy of the model by generating a generation index that optimizes each aspect according to the characteristics of the learning data.

For example, it is considered that the learning data includes data with various labels, that is, data showing various characteristics. However, if the training data is data showing features that are not useful when classifying the data, the accuracy of the finally obtained model may deteriorate. Therefore, the information providing device 10 determines the characteristics of the input learning data as an mode when the learning data is input to the model. For example, the information providing device 10 determines which label-assigned data (that is, data indicating which feature) is input among the training data. In other words, the information providing device 10 optimizes the combination of input features.

Further, it is considered that the learning data includes columns of various formats such as data containing only numerical values and data containing character strings. When inputting such training data into the model, it is considered that the accuracy of the model changes depending on whether the data is input as it is or converted into data in another format. For example, when inputting multiple types of training data (learning data showing different characteristics) and inputting the training data of a character string as the training data of a numerical value, the case where the character string and the numerical value are input as they are and the case where the character is input It is considered that the accuracy of the model changes depending on whether the column is converted to a numerical value and only the numerical value is input, or the numerical value is regarded as a character string and input. Therefore, the information providing device 10 determines the format of the learning data to be input to the model. For example, the information providing device 10 determines whether the learning data to be input to the model is a numerical value or a character string. In other words, the information providing device 10 optimizes the column type of the feature to be input.

In addition, when learning data showing different features exists, it is considered that the accuracy of the model changes depending on which combination of features is input at the same time. That is, when learning data showing different features exists, it is considered that the accuracy of the model changes depending on which feature combination of features (that is, the relationship between the combinations of a plurality of features) is trained. For example, when there are learning data showing the first feature (for example, gender), learning data showing the second feature (for example, address), and learning data showing the third feature (for example, purchase history), the first The accuracy of the model is higher when the learning data showing one feature and the learning data showing the second feature are input at the same time and when the learning data showing the first feature and the learning data showing the third feature are input at the same time. It is expected to change. Therefore, the information providing device 10 optimizes the combination of features (cross-future) that causes the model to learn the relationship.

Here, various models project the input data into a space of a predetermined dimension divided by a predetermined hyperplane, and the input data depends on which space the projected position belongs to in the divided space. Will be classified. Therefore, if the number of dimensions of the space on which the input data is projected is lower than the optimum number of dimensions, the classification ability of the input data deteriorates, and as a result, the accuracy of the model deteriorates. In addition, when the number of dimensions of the space on which the input data is projected is higher than the optimum number of dimensions, the inner product value with the hyperplane changes, and as a result, data different from the data used at the time of training can be appropriately classified. It may disappear. Therefore, the information providing device 10 optimizes the number of dimensions of the input data input to the model. For example, the information providing device 10 optimizes the number of dimensions of the input data by controlling the number of nodes in the input layer of the model. In other words, the information providing device 10 optimizes the number of dimensions of the space in which the input data is embedded.

Further, in the model, in addition to the SVM, there is a neural network or the like having a plurality of intermediate layers (hidden layers). Further, such a neural network includes a feed-forward type DNN in which information is transmitted in one direction from the input layer to the output layer, and a convolutional neural network (CNN) in which information is convoluted in the intermediate layer. Various neural networks such as a recurrent neural network (RNN) having a closed path and a Boltzmann machine are known. Further, such various neural networks include LSTM (Long short-term memory) and various other neural networks.

As described above, when the types of models for learning various features of the training data are different, the accuracy of the model is considered to change. Therefore, the information providing device 10 selects the type of model that is presumed to learn the features of the training data with high accuracy. For example, the information providing device 10 selects the type of the model according to what kind of label is given as the label of the training data. To give a more specific example, the information providing device 10 is considered to be able to better learn the characteristics of the history when the data with the term related to "history" is present as a label. If there is data with a term related to "image" as a label, select CNN, which is considered to be able to better learn the features of the image. In addition to these, the information providing device 10 determines whether or not the label is a term designated in advance or a term similar to the term, and a model of a type previously associated with the term determined to be the same or similar. You just have to select.

Further, when the number of intermediate layers of the model or the number of nodes included in one intermediate layer changes, it is considered that the learning accuracy of the model changes. For example, if the number of intermediate layers of the model is large (the model is deep), it may be possible to realize classification according to more abstract features, while local error in backpropagation is the input layer. As a result of difficulty in propagating to, there is a risk that learning cannot be performed properly. Further, if the number of nodes included in the mesosphere is small, a higher level of abstraction can be performed, but if the number of nodes is too small, the information necessary for classification is likely to be lost. Therefore, the information providing device 10 optimizes the number of intermediate layers and the number of nodes included in the intermediate layer. That is, the information providing device 10 optimizes the architecture of the model.

In addition, it is considered that the accuracy of the nodes changes depending on the presence or absence of attention and the case where the nodes included in the model have autoregressive or not, depending on which node is connected. Therefore, the information providing device 10 optimizes the network such as whether or not it has autoregression and which nodes are connected to each other.

When learning a model, a model optimization method (algorithm used at the time of learning), a dropout rate, a node activation function, the number of units, and the like are set as hyperparameters. It is considered that the accuracy of the model also changes when such hyperparameters change. Therefore, the information providing device 10 optimizes the learning mode when learning the model, that is, the hyperparameters.

The accuracy of the model also changes when the size of the model (the number of input layers, intermediate layers, output layers, and the number of nodes) changes. Therefore, the information providing device 10 also optimizes the size of the model.

In this way, the information providing device 10 optimizes the indexes when generating the various models described above. For example, the information providing device 10 holds in advance the conditions corresponding to each index. It should be noted that such a condition is set by, for example, an empirical rule such as the accuracy of various models generated from the past learning model. Then, the information providing device 10 determines whether or not the learning data satisfies each condition, and adopts an index previously associated with the condition that the learning data satisfies or does not satisfy as a generation index (or a candidate thereof). As a result, the information providing device 10 can generate a generation index capable of accurately learning the features of the learning data.

As described above, when the generation index is automatically generated from the training data and the process of creating the model according to the generation index is automatically performed, the user refers to the inside of the training data and what kind of It is not necessary to judge whether the distribution data exists. As a result, the information providing device 10 can reduce the time and effort required for the data scientist or the like to recognize the learning data when creating the model, and can prevent the privacy from being damaged due to the recognition of the learning data.

[3-2. Generation index according to data type]
Hereinafter, an example of the conditions for generating the generation index will be described. First, an example of conditions according to what kind of data is adopted as learning data will be described.

For example, the learning data used for learning includes integers, floating point numbers, character strings, and the like as data. Therefore, if an appropriate model is selected for the format of the input data, it is estimated that the learning accuracy of the model will be higher. Therefore, the information providing device 10 generates a generation index based on whether the learning data is an integer, a floating point number, or a character string.

For example, when the learning data is an integer, the information providing device 10 generates a generation index based on the continuity of the learning data. For example, when the density of the learning data exceeds a predetermined first threshold value, the information providing device 10 considers the learning data to be continuous data, and the maximum value of the learning data sets the predetermined second threshold value. Generate an index based on whether or not it exceeds. Further, when the density of the learning data is lower than the predetermined first threshold value, the information providing device 10 considers the learning data to be sparse learning data, and determines the number of unique values included in the learning data. The generation index is generated based on whether or not the third threshold value is exceeded.

A more specific example will be described. In the following example, an example of a process of selecting a feature function from the configuration files to be transmitted to the model generation server 2 that automatically generates a model by AutoML as a generation index will be described. .. For example, when the learning data is an integer, the information providing device 10 determines whether or not the density exceeds a predetermined first threshold value. For example, the information providing device 10 calculates the number of unique values included in the learning data divided by the value obtained by adding 1 to the maximum value of the learning data as the density.

Subsequently, when the density exceeds a predetermined first threshold value, the information providing device 10 determines that the learning data is continuous learning data, and the value obtained by adding 1 to the maximum value of the learning data is the second value. Determine if the threshold is exceeded. Then, when the value obtained by adding 1 to the maximum value of the learning data exceeds the second threshold value, the information providing device 10 selects "Categorical_colum_with_identity & embedding_column" as the characteristic function. On the other hand, the information providing device 10 selects "Categorical_column_with_identity" as the characteristic function when the value obtained by adding 1 to the maximum value of the training data is lower than the second threshold value.

On the other hand, when the density is lower than the predetermined first threshold value, the information providing device 10 determines that the training data is sparse, and whether or not the number of unique values included in the training data exceeds the predetermined third threshold value. Is determined. Then, when the number of unique values included in the training data exceeds a predetermined third threshold value, the information providing device 10 selects "Categorical_column_with_hash_bucket & embedding_column" as the feature function and sets the unique values included in the training data. If the number is below a predetermined third threshold, select "Categorical_column_with_hash_bucket" as the feature function.

Further, when the learning data is a character string, the information providing device 10 generates a generation index based on the number of types of the character string included in the learning data. For example, the information providing device 10 counts the number of unique character strings (number of unique data) included in the training data, and when the counted number falls below a predetermined fourth threshold value, the feature function is "categorical_column_with_vocabulary_list". Or / and select "categorical_column_with_vocabulary_file". Further, the information providing device 10 selects "categorical_column_with_vocabulary_file & embedding_column" as the characteristic function when the counted number is less than the fifth threshold value larger than the predetermined fourth threshold value. In addition, the information providing device 10 selects "categorical_column_with_hash_bucket & embedding_column" as the characteristic function when the counted number exceeds the fifth threshold value larger than the predetermined fourth threshold value.

Further, when the training data is floating point, the information providing device 10 generates a conversion index for input data for inputting the training data to the model as a model generation index. For example, the information providing device 10 selects "bucketized_column" or "numeric_colum" as the characteristic function. That is, the information providing device 10 buckets (groups) the learning data and selects whether to input the bucket number or the numerical value as it is. The information providing device 10 may, for example, bucket the learning data so that the range of numerical values associated with each bucket is about the same. For example, the learning data classified into each bucket may be bucketd. A range of numbers may be associated with each bucket so that the numbers are comparable. Further, the information providing device 10 may select the number of buckets or the range of numerical values associated with the buckets as the generation index.

Further, the information providing device 10 acquires learning data indicating a plurality of features, and generates a generation index indicating a feature to be trained by the model among the features of the training data as a model generation index. For example, the information providing device 10 determines which label of training data to be input to the model, and generates a generation index indicating the determined label. Further, the information providing device 10 generates, as a model generation index, a generation index indicating a plurality of types of training data for learning the correlation with respect to the model. For example, the information providing device 10 determines a combination of labels to be input to the model at the same time, and generates a generation index indicating the determined combination.

Further, the information providing device 10 generates a generation index indicating the number of dimensions of the learning data input to the model as a model generation index. For example, the information providing device 10 is in the input layer of the model according to the number of unique data included in the training data, the number of labels to be input to the model, the combination of the number of labels to be input to the model, the number of buckets, and the like. You may decide the number of nodes.

Further, the information providing device 10 generates a generation index indicating the type of the model for learning the features of the training data as the model generation index. For example, the information providing device 10 determines and determines the type of model to be generated according to the density and sparseness of training data that has been learned in the past, the content of labels, the number of labels, the number of combinations of labels, and the like. Generates a generation index that indicates the type of label. For example, the information providing device 10 has "BaselineClassifier", "LinearClassifier", "DNNClassifier", "DNNLinearCombinedClassifier", "BoostedTreesClassifier", "AdaNetClassifier", "RNNClassifier", "DNNResNetClassifier", "AutoIntClassifier", etc. as model classes in AutoML. Generates a generation index that indicates.

The information providing device 10 may generate a generation index indicating various independent variables of the models of each of these classes. For example, the information providing device 10 may generate a generation index indicating the number of intermediate layers of the model or the number of nodes included in each layer as a model generation index. Further, the information providing device 10 may generate a generation index indicating the connection mode between the nodes of the model and a generation index indicating the size of the model as the model generation index. These independent variables will be appropriately selected depending on whether or not the various statistical features of the training data satisfy a predetermined condition.

Further, the information providing device 10 may generate a learning mode in which the model learns the features of the learning data, that is, a generation index indicating hyperparameters, as a model generation index. For example, the information providing device 10 may generate a generation index indicating "stop_if_no_decrease_hook", "stop_if_no_increase_hook", "stop_if_higher_hook", or "stop_if_lower_hook" in setting the learning mode in AutoML.

That is, the information providing device 10 causes the model to learn the characteristics of the training data to be trained by the model, the mode of the model to be generated, and the characteristics of the training data based on the label of the training data used for learning and the characteristics of the data itself. A generation index indicating the learning mode is generated. More specifically, the information providing device 10 generates a config file for controlling the generation of the model in AutoML.

[3-3. About the order of determining the generation index]
Here, the information providing device 10 may optimize the various indexes described above in parallel, or may execute them in an appropriate order. Further, the information providing device 10 may be able to change the order of optimizing each index. That is, the information providing device 10 accepts from the user the designation of the order of determining the characteristics of the learning data to be trained by the model, the mode of the model to be generated, and the learning mode when the model is trained with the features of the training data. Each index may be determined in the order received.

For example, FIG. 34 is a diagram showing an order in which the information providing device according to the embodiment optimizes the index. For example, in the example shown in FIG. 34, when the information providing device 10 starts generating the generation index, it optimizes the input characteristics such as the characteristics of the learning data to be input and the mode in which the learning data is input. Then, the input cross element of which combination of features is to be learned is optimized. Subsequently, the information providing device 10 selects a model and optimizes the model structure. After that, the information providing device 10 optimizes the hyperparameters and finishes the generation of the generation index.

Here, the information providing device 10 selects and corrects various input qualities such as the characteristics and input modes of the input learning data in the input habit optimization, and selects a new input trait using a genetic algorithm. The input characteristics may be repeatedly optimized. Similarly, in the input cross feature optimization, the information providing device 10 may repeatedly optimize the input cross feature, or may repeatedly execute model selection and model structure optimization. Further, the information providing device 10 may repeatedly execute the optimization of the hyper parameter. Further, the information providing device 10 repeatedly executes a series of processes such as input feature optimization, input cross feature optimization, model selection, model structure optimization, and hyperparameter optimization to optimize each index. May be good.

Further, for example, the information providing device 10 may perform model selection and model structure optimization after optimizing hyperparameters, and after model selection and model structure optimization, optimization of input characteristics and input. Cross-based optimization may be performed. Further, the information providing device 10 repeatedly executes, for example, input feature optimization, and then repeatedly performs input cross feature optimization. After that, the information providing device 10 may repeatedly execute the input feature optimization and the input cross feature optimization. In this way, any setting can be adopted for which index is optimized in which order and which optimization process is repeatedly executed in the optimization.

[3-4. About the flow of model generation realized by the information providing device]
Subsequently, an example of the flow of model generation using the information providing device 10 will be described with reference to FIG. 35. FIG. 35 is a diagram illustrating an example of a flow of model generation using the information providing device in the embodiment. For example, the information providing device 10 receives the learning data and the label of each learning data. The information providing device 10 may accept the label together with the designation of the learning data.

In such a case, the information providing device 10 analyzes the data and divides the data according to the analysis result. For example, the information providing device 10 divides the training data into training data used for learning the model and evaluation data used for evaluating the model (that is, measuring accuracy). The information providing device 10 may further divide the data for various tests. In addition, various known techniques can be adopted for the process of dividing such learning data into training data and evaluation data.

Further, the information providing device 10 uses the learning data to generate the various generation indexes described above. For example, the information providing device 10 generates a model generated in AutoML and a config file that defines the learning of the model. In such a configuration file, various functions used in AutoML are stored as they are as information indicating a generation index. Then, the information providing device 10 generates a model by providing the training data and the generation index to the model generation server 2.

Here, the information providing device 10 may realize the optimization of the generation index, and eventually the optimization of the model, by repeatedly evaluating the model by the user and automatically generating the model. For example, the information providing device 10 optimizes the input features (optimization of input features and input cross features), optimizes hyperparameters, and optimizes the model to be generated, and automatically follows the optimized generation index. Model is generated in. Then, the information providing device 10 provides the generated model to the user.

On the other hand, the user trains, evaluates, and tests the automatically generated model, and analyzes and provides the model. Then, the user modifies the generated generation index to automatically generate a new model again, and performs evaluation, testing, and the like. By repeatedly executing such a process, it is possible to realize a process of improving the accuracy of the model through trial and error without executing a complicated process.

[4. Configuration of information providing device]
Next, an example of the functional configuration of the information providing device 10 according to the embodiment will be described with reference to FIG. FIG. 36 is a diagram showing a configuration example of the information providing device according to the embodiment. As shown in FIG. 36, the information providing device 10 includes a communication unit 20, a storage unit 30, and a control unit 40.

The communication unit 20 is realized by, for example, a NIC (Network Interface Card) or the like. Then, the communication unit 11 is connected to the network N by wire or wirelessly, and transmits / receives information to / from the model generation server 2 and the terminal device 3.

The storage unit 30 is realized by, for example, a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory (Flash Memory), or a storage device such as a hard disk or an optical disk. Further, the storage unit 30 has a learning data database 31 and a generation condition database 32.

The learning data is registered in the learning data database 31. For example, FIG. 37 is a diagram showing an example of information registered in the learning database according to the embodiment. In the example shown in FIG. 37, the learning data ID (Identifier) and the learning data are registered in association with each other in the learning data database 31. Here, the learning data ID is an identifier that identifies a plurality of data groups that serve as learning data. The learning data is data used for learning.

For example, in the example shown in FIG. 37, in the training data database 31, the pair of “label # 1-1” and “data # 1-1” and “label # 1” for “training data # 1” The pair of "-2" and "data # 1-2" is registered in association with each other. Such information is given "data # 1-1" to which "label # 1-1" is attached and "label # 1-2" as training data indicated by "learning data # 1". Indicates that "Data # 1-2" is registered. In addition, a plurality of data showing the same characteristics may be registered in each label. Further, in the example shown in FIG. 37, conceptual values such as "training data # 1", "label # 1-1", and "data # 1-1" are described, but in reality, the training data is identified. Character strings and numerical values for this purpose, character strings to be labels, various integers to be data, floating points, character strings, etc. will be registered.

Returning to FIG. 36, in the generation condition database 38, the generation conditions in which various conditions related to the learning data are associated with the generation indexes or various indexes determined as candidates thereof when the training data is the condition are registered. There is. For example, FIG. 38 is a diagram showing an example of information registered in the generation condition database according to the embodiment. In the example shown in FIG. 38, the condition ID, the condition content, and the index candidate are registered in association with each other in the generation condition data database 32.

Here, the condition ID is an identifier that identifies the generation condition. Further, the condition content is a condition to be determined whether or not the learning data is satisfied, and depends on various conditions such as a content condition which is a condition related to the content of the learning data and a tendency condition related to the tendency of the learning data. It is composed. Further, the index candidates are various indexes included in the generated index when each condition included in the associated condition content is satisfied.

For example, the condition ID "condition ID # 1", the content condition "integer", the tendency condition "density <threshold value", and the index candidate "generation index # 1" are registered in the generation condition database 38 in association with each other. Such information can be obtained by setting the condition ID "condition ID # 1" as the index candidate "generation index # 1" when the learning data is the content condition "integer" and the tendency condition "density <threshold" is satisfied. Indicates that it is determined as a generation index.

In the example shown in FIG. 38, a conceptual value such as "generation index # 1" is described, but in reality, information adopted as various generation indexes is registered. For example, various functions described in the config file in AutoML are registered as index candidates in the generation condition database 38. In the generation condition database 38, for example, a plurality of generation indexes may be registered for one condition.

As described above, it is possible to arbitrarily set what kind of generation index is generated when what kind of condition is satisfied. For example, the generation condition database 38 contains various generation indexes for models that have been generated in the past and whose accuracy exceeds a predetermined threshold value, and features and tendencies of training data used for learning the model. It suffices if the generated generation condition is registered.

Returning to FIG. 36, the description will be continued. The control unit 40 is realized by, for example, using a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or the like to execute various programs stored in a storage device inside the information providing device 10 using the RAM as a work area. Will be done. Further, the control unit 40 is realized by, for example, an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). As shown in FIG. 36, the control unit 40 includes an acquisition unit 41, an index generation unit 42, a presentation unit 43, a reception unit 44, a model generation unit 45, and a provision unit 46.

The acquisition unit 41 acquires learning data used for learning the model. For example, when the acquisition unit 41 receives various data to be used as training data and labels attached to the various data from the terminal device 3, the received data and the label are registered in the training data database 3 as training data. The acquisition unit 41 may accept the indication of the learning data ID and the label of the learning data used for learning the model among the data registered in the learning data database 31 in advance.

The index generation unit 42 generates a model generation index based on the characteristics of the learning data. For example, the index generation unit 42 generates a generation index based on the statistical characteristics of the learning data. For example, the index generation unit 42 acquires learning data from the acquisition unit 41. Then, the index generation unit 42 generates a generation index according to whether or not the acquired learning data satisfies the generation condition registered in the generation condition database 32.

For example, the index generation unit 42 may generate a generation index based on whether the training data is an integer, a floating point number, or a character string. To give a more specific example, when the training data is an integer, the index generation unit 42 may generate a generation index based on the continuity of the training data. For example, the index generation unit 42 calculates the density of the learning data, and when the calculated density exceeds a predetermined first threshold value, the index generation unit 42 generates an index based on whether or not the maximum value of the learning data exceeds a predetermined second threshold value. May be generated. That is, the index generation unit 42 may generate different generation indexes depending on whether or not the maximum value exceeds the second threshold value. Further, when the density of the training data is lower than the predetermined first threshold value, the index generation unit 42 generates the generation index based on whether or not the number of unique values included in the training data exceeds the predetermined third threshold value. You may.

The index generation unit 42 may generate different generation indexes according to conditional branching such as whether or not the density and maximum value of the learning data exceed various threshold values. For example, the density and maximum value of the training data and the like may be generated. A generation index corresponding to the value of itself may be generated. For example, the index generation unit 42 calculates the values of parameters that serve as various generation indexes, such as the number of model nodes and the number of intermediate layers, based on statistical values such as the number of training data, density, and maximum value. May be good. That is, if the index generation unit 42 generates different generation indexes based on the characteristics of the learning data, the index generation unit 42 may generate the generation indexes based on arbitrary conditions.

Further, when the learning data is a character string, the index generation unit 42 generates a generation index based on the number of types of the character string included in the learning data. That is, the index generation unit 42 generates different generation indexes according to the number of unique character string types. Further, when the training data is a floating point number, the index generation unit 42 generates a conversion index for input data for inputting the training data to the model as a model generation index. For example, the index generation unit 42 determines whether or not to bucket the floating point number, which range of values should be classified into which bucket, and the like based on the statistical information of the training data. To give a more specific example, whether or not the index generation unit 42 buckets according to the range of values included in the floating-point training data and the content of the label attached to the training data. You may decide which range of values to classify into which bucket. Further, the index generation unit 42 determines whether or not the range of values corresponding to each bucket is constant based on the characteristics of the learning data, and the number of learning data to be classified into each bucket is constant (or a predetermined distribution). ) May be decided.

In addition, the index generation unit 42 generates, as a model generation index, a generation index indicating a feature to be trained by the model among the features of the training data. For example, the index generation unit 42 determines the label of the data to be trained by the model based on the characteristics of the training data. Further, the index generation unit 42 generates, as a model generation index, a generation index indicating a plurality of types of training data for learning the correlation with respect to the model.

It should be noted that the determination of the features (labels) to be learned and the relationships between the features is determined according to the purpose such as what kind of model the user desires, such as the label of the data output by the model. May be done. Regarding which feature to adopt and which feature combination to learn, for example, in the above-mentioned genetic algorithm, a bit indicating whether to adopt each feature or feature combination is regarded as a gene. It may be determined by discovering features and combinations of features that further improve the accuracy of the model by generating Cenozoic generation indicators.

In addition, the index generation unit 42 generates a generation index indicating the number of dimensions of the learning data input to the model as a model generation index. Further, the index generation unit 42 generates, as a model generation index, a generation index indicating the type of the model for learning the features of the training data. Further, the index generation unit 42 generates a generation index indicating the number of intermediate layers of the model or the number of nodes included in each layer as a model generation index. In addition, the index generation unit 42 generates a generation index indicating the connection mode between the nodes of the model as a model generation index. In addition, the index generation unit 42 generates a generation index indicating the size of the model as a model generation index. For example, the index generation unit 42 determines the dimension of the training data input to the model according to the number of unique training data, the number of features adopted or combinations thereof, the number of numerical values to be the training data, the number of bits of the character string, and the like. A generation index indicating a number may be generated, and for example, various structures of the model may be determined.

In addition, the index generation unit 42 generates, as a model generation index, a generation index indicating a learning mode when the model learns the features of the training data. For example, the index generation unit 42 may determine the contents of the hyperparameters based on the characteristics of the learning data and the various generation indexes described above. In this way, the index generation unit 42 generates a generation index indicating the characteristics of the learning data to be trained by the model, the mode of the model to be generated, and the learning mode when the model is trained with the features of the training data. It is not necessary for the index generation unit 42 to determine and generate all the above-mentioned generation indexes, and only the generation indexes of any type need to be determined and generated.

The presentation unit 43 presents the index generated by the index generation unit 42 to the user. For example, the presentation unit 43 transmits the AutoML config file generated as the generation index to the terminal device 3.

The reception unit 44 accepts the modification of the generation index presented to the user. In addition, the reception unit 44 receives from the user the designation of the order of determining the characteristics of the learning data to be trained by the model, the mode of the model to be generated, and the learning mode when the model is trained with the features of the training data. In such a case, the index generation unit 42 makes the model learn the features of the training data to be trained by the user, the mode of the model to be generated, and the features of the training data in the order specified by the user. To determine. That is, the index generation unit 42 regenerates various generation indexes in the order specified by the user.

The model generation unit 45 generates a model in which the features of the training data are trained according to the generation index. For example, the model generation unit 45 divides the training data into training data and evaluation data, and transmits the training data and the generation index to the model generation server 2. Then, the model generation unit 45 acquires the model generated from the training data by the model generation server 2 according to the generation index. In such a case, the model generation unit 45 calculates the accuracy of the acquired model using the evaluation data.

The index generation unit 42 generates a plurality of different generation indexes. In such a case, the index generation unit 42 generates a different model for each generation index and calculates the accuracy of each model. The index generation unit 42 may generate different training data and evaluation data for each model, or may adopt the same training data and evaluation data.

When a plurality of models are generated in this way, the index generation unit 42 newly generates a model generation index based on the accuracy of the generated model. For example, the index generation unit 42 generates a new generation index from a plurality of generation indexes by using a genetic algorithm that genetically considers whether or not to use each learning data and which generation index is adopted. To do. Then, the model generation unit 45 generates a new model based on the new generation index. By repeatedly executing such trial and error a predetermined number of times or until the accuracy of the model exceeds a predetermined threshold value, the information providing device 10 can realize the generation of a generation index that improves the accuracy of the model.

The index generation unit 42 may also optimize the order in which the generation indexes are determined as the target of the genetic algorithm. Further, the presentation unit 43 may present the generation index to the user each time the generation index is generated. For example, the presentation unit 43 presents only the generation index corresponding to the model whose accuracy exceeds a predetermined threshold value to the user. You may.

The providing unit 46 provides the generated model to the user. For example, when the accuracy of the model generated by the model generation unit 45 exceeds a predetermined threshold value, the providing unit 46 transmits the model and the generation index corresponding to the model to the terminal device 3. As a result, the user can evaluate and try the model and modify the generation index.

[5. Processing flow of information providing device 10]
Next, the procedure of the process executed by the information providing device 10 will be described with reference to FIG. 39. FIG. 39 is a flowchart showing an example of the flow of the generation process according to the embodiment.

For example, the information providing device 10 accepts the designation of the learning data (step S101). In such a case, the information providing device 10 identifies the statistical features of the designated learning data (step S102). Then, the information providing device 10 creates a candidate of the model generation index based on the statistical characteristics (step S103).

Subsequently, the information providing device 10 determines whether or not the correction of the created generation index has been accepted (step S104), and if it has received the correction (step S104: Yes), the correction is performed according to the instruction (step S105). On the other hand, if the information providing device 10 does not accept the correction, the information providing device 10 skips the execution of step S105. Then, the information providing device 10 generates a model according to the generation index (step S106), provides the generated model (step S107), and ends the process.

[6. Modification example]
In the above, an example of the generation process has been described. However, the embodiments are not limited to this. Hereinafter, a modified example of the providing process will be described.

[6-1. Device configuration〕
In the above embodiment, an example in which the distribution system 1 has an information providing device 10 for generating a generation index and a model generation server 2 for generating a model according to the generation index has been described, but the embodiment is limited to this. It's not something. For example, the information providing device 10 may have a function of the model generation server 2. Further, the function exhibited by the information providing device 10 may be included in the terminal device 3. In such a case, the terminal device 3 automatically generates the generation index and automatically generates the model using the model generation server 2.

[6-2. Others]
Further, among the processes described in the above-described embodiment, all or a part of the processes described as being automatically performed can be manually performed, or the processes described as being manually performed can be performed. All or part of it can be done automatically by a known method. In addition, the processing procedure, specific name, and information including various data and parameters shown in the above document and drawings can be arbitrarily changed unless otherwise specified. For example, the various information shown in each figure is not limited to the illustrated information.

Further, each component of each of the illustrated devices is a functional concept, and does not necessarily have to be physically configured as shown in the figure. That is, the specific form of distribution / integration of each device is not limited to the one shown in the figure, and all or part of the device is functionally or physically distributed / physically in arbitrary units according to various loads and usage conditions. It can be integrated and configured.

In addition, the above-described embodiments can be appropriately combined as long as the processing contents do not contradict each other.

[6-3. program〕
Further, the information providing device 10 according to the above-described embodiment is realized by, for example, a computer 1000 having a configuration as shown in FIG. 40. FIG. 40 is a diagram showing an example of a hardware configuration. The computer 1000 is connected to the output device 1010 and the input device 1020, and the arithmetic unit 1030, the primary storage device 1040, the secondary storage device 1050, the output IF (Interface) 1060, the input IF 1070, and the network IF 1080 are connected by the bus 1090. Has.

The arithmetic unit 1030 operates based on a program stored in the primary storage device 1040 or the secondary storage device 1050, a program read from the input device 1020, or the like, and executes various processes. The primary storage device 1040 is a memory device that temporarily stores data used by the arithmetic unit 1030 for various calculations, such as a RAM. Further, the secondary storage device 1050 is a storage device in which data used by the calculation device 1030 for various calculations and various databases are registered, and is realized by a ROM (Read Only Memory), an HDD, a flash memory, or the like.

The output IF 1060 is an interface for transmitting information to be output to an output device 1010 that outputs various information such as a monitor and a printer. For example, USB (Universal Serial Bus), DVI (Digital Visual Interface), and the like. It is realized by a standard connector such as HDMI (registered trademark) (High Definition Multimedia Interface). Further, the input IF 1070 is an interface for receiving information from various input devices 1020 such as a mouse, a keyboard, and a scanner, and is realized by, for example, USB.

The input device 1020 is, for example, an optical recording medium such as a CD (Compact Disc), a DVD (Digital Versatile Disc), a PD (Phase change rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), or a tape. It may be a device that reads information from a medium, a magnetic recording medium, a semiconductor memory, or the like. Further, the input device 1020 may be an external storage medium such as a USB memory.

The network IF1080 receives data from another device via the network N and sends it to the arithmetic unit 1030, and also transmits the data generated by the arithmetic unit 1030 to the other device via the network N.

The arithmetic unit 1030 controls the output device 1010 and the input device 1020 via the output IF 1060 and the input IF 1070. For example, the arithmetic unit 1030 loads a program from the input device 1020 or the secondary storage device 1050 onto the primary storage device 1040, and executes the loaded program.

For example, when the computer 1000 functions as the information providing device 10, the arithmetic unit 1030 of the computer 1000 realizes the function of the control unit 40 by executing the program loaded on the primary storage device 1040.

[7. effect〕
As described above, the information providing device 10 acquires the learning data used for learning the model, and generates a model generation index based on the characteristics of the learning data. For example, the information providing device 10 generates a generation index based on the statistical characteristics of the learning data. As a result of such processing, the information providing device 10 can provide a generation index for generating a model in which accuracy is expected, without the user making complicated settings.

For example, the information providing device 10 generates a generation index based on whether the training data is an integer, a floating point number, or a character string. Further, when the learning data is an integer, the information providing device 10 generates a generation index based on the continuity of the learning data. To give a more specific example, the information providing device 10 generates an index based on whether or not the maximum value of the learning data exceeds the predetermined second threshold value when the density of the learning data exceeds the predetermined first threshold value. To generate. Further, when the density of the training data is lower than the predetermined first threshold value, the information providing device 10 generates a generation index based on whether or not the number of unique values included in the training data exceeds the predetermined third threshold value. To do.

Further, when the learning data is a character string, the information providing device 10 generates a generation index based on the number of types of the character string included in the learning data. Further, when the training data is floating point, the information providing device 10 generates a conversion index for input data for inputting the training data to the model as a model generation index. Further, the information providing device 10 acquires learning data indicating a plurality of features, and generates a generation index indicating a feature to be trained by the model among the features of the training data as a model generation index.

Further, the information providing device 10 acquires learning data showing the characteristics of a plurality of types, and generates a generation index indicating a plurality of types of the training data types for learning the correlation with respect to the model as a model generation index. .. Further, the information providing device 10 generates a generation index indicating the number of dimensions of the learning data input to the model as a model generation index. Further, the information providing device 10 generates a generation index indicating the type of the model for learning the features of the training data as the model generation index.

Further, the information providing device 10 generates a generation index indicating the number of intermediate layers of the model or the number of nodes included in each layer as a model generation index. Further, the information providing device 10 generates a generation index indicating the connection mode between the nodes of the model as a model generation index. Further, the information providing device 10 generates, as a model generation index, a generation index indicating a learning mode when the model learns the features of the learning data. Further, the information providing device 10 generates a generation index indicating the size of the model as a model generation index. Further, the information providing device 10 generates a generation index indicating the characteristics of the learning data to be trained by the model, the mode of the model to be generated, and the learning mode when the model is trained with the features of the training data.

In this way, the information providing device 10 automatically generates various generation indexes when generating the model. As a result, the information providing device 10 can save the trouble of creating the generation index by the user and can more easily generate the model. Further, the information providing device 10 can recognize the content of the learning data and save the trouble of generating a model according to the recognition result. As a result, when various information of the user is used as the learning data, the privacy is damaged. Can be prevented.

Further, the information providing device 10 receives from the user the designation of the order of determining the characteristics of the learning data to be trained by the model, the mode of the model to be generated, and the learning mode when the model is trained with the features of the training data. Then, the information providing device 10 determines the characteristics of the learning data to be trained by the model, the mode of the model to be generated, and the learning mode when the model is trained with the features of the training data, in the order specified by the user. .. As a result of such processing, the information providing device 10 can further improve the accuracy of the model.

Further, the information providing device 10 generates a model in which the features of the learning data are learned according to the generation index. Further, the information providing device 10 newly generates a model generation index based on the accuracy of the model generated by the model generation unit, and newly generates a model according to the generation index newly generated by the index generation unit. For example, the information providing device 10 uses a genetic algorithm to generate a new generation index from a plurality of generation indexes. As a result of such processing, the information providing device 10 can generate a generation index for generating a model with higher accuracy.

Although some of the embodiments of the present application have been described in detail with reference to the drawings, these are examples, and various modifications are made based on the knowledge of those skilled in the art, including the embodiments described in the disclosure column of the invention. It is possible to carry out the present invention in other improved forms.

In addition, the above-mentioned "section, module, unit" can be read as "means" or "circuit". For example, the distribution unit can be read as a distribution means or a distribution circuit.

[Additional Notes]
In addition to the above description of the embodiment, the following additional notes will be further disclosed.
(Appendix 1)
A computer implementation method for generating and optimizing an artificial intelligence model.
The steps of receiving the entered data and labels, performing data validation to generate a configuration file, and splitting the data to generate split data for training and evaluation.
A step of performing training and evaluation of the divided data to measure an error level and performing an action based on the error level, wherein the action modifies the configuration file and the artificial intelligence model. Have at least one of the steps to tune automatically,
A step of generating the artificial intelligence model based on the training, the evaluation, and the tuning.
The step of serving the model in production and
Computer implementation methods, including.
(Appendix 2)
The tuning
A step that automatically optimizes one or more entered features related to the entered data, and
The steps to automatically optimize the hyperparameters associated with the generated artificial intelligence model,
Steps to automatically generate an updated model based on one or more optimized input features and the optimized hyperparameters.
The computer mounting method according to Appendix 1, which comprises.
(Appendix 3)
The one or more input features are optimized by a general algorithm, the combination of the one or more input features is optimized, and a list of the optimized input features is generated, Appendix 2. Computer implementation method described in.
(Appendix 4)
The computer implementation method according to Appendix 2, wherein the step of automatically optimizing the hyperparameters comprises applying at least one of a Bayesian algorithm and a random algorithm to optimize based on the hyperparameters.
(Appendix 5)
The step of automatically optimizing the one or more input features is performed in a first iteration loop that is executed until the first defined number of iterations is met, and the hyperparameters are automatically said. 2. The computer implementation method according to Appendix 2, wherein the step of optimizing the above and the step of automatically generating the updated model are executed in a second iteration loop until the second specified number of iterations is satisfied.
(Appendix 6)
The computer implementation method according to Appendix 5, wherein the first iteration loop and the second iteration loop are iteratively executed until a third defined number of iterations is met.
(Appendix 7)
Addendum 1. The step of performing the training and the evaluation comprises performing one or more feature functions based on the data type of the data, the density of the data, and the amount of the data. Computer implementation method.
(Appendix 8)
A non-transitory computer-readable medium configured to execute machine-readable instructions stored in storage in order to generate and optimize an artificial intelligence model.
The steps of receiving the entered data and labels, performing data validation to generate a configuration file, and splitting the data to generate split data for training and evaluation.
A step of performing training and evaluation of the divided data to measure an error level and performing an action based on the error level, wherein the action modifies the configuration file and the artificial intelligence model. Have at least one of the steps to tune automatically,
A step of generating the artificial intelligence model based on the training, the evaluation, and the tuning.
The step of serving the model in production and
Non-transitory computer-readable media, including.
(Appendix 9)
The tuning
A step that automatically optimizes one or more entered features related to the entered data, and
The steps to automatically optimize the hyperparameters associated with the generated artificial intelligence model,
Steps to automatically generate an updated model based on one or more optimized input features and the optimized hyperparameters.
The non-transitory computer-readable medium according to Appendix 8, comprising.
(Appendix 10)
The one or more input features are optimized by a general algorithm, the combination of the one or more input features is optimized, and a list of the optimized input features is generated. Non-temporary computer-readable media described in.
(Appendix 11)
The non-temporary computer-readable medium according to Appendix 9, wherein the step of automatically optimizing the hyperparameters comprises applying at least one of a Bayesian algorithm and a random algorithm to optimize based on the hyperparameters.
(Appendix 12)
The step of automatically optimizing the one or more input features is performed in a first iteration loop that is executed until the first defined number of iterations is met, and the hyperparameters are automatically said. The non-transient computer according to Appendix 9, wherein the step of optimizing the above and the step of automatically generating the updated model are executed in the second iteration loop until the second specified number of iterations is satisfied. Readable medium.
(Appendix 13)
The non-transitory computer-readable medium according to Appendix 12, wherein the first iteration loop and the second iteration loop are iteratively executed until a third defined iteration count is met.
(Appendix 14)
8. Addendum 8, wherein the step of performing the training and the evaluation comprises performing one or more feature functions based on the data type of the data, the density of the data, and the amount of the data. Non-temporary computer-readable medium.
(Appendix 15)
A system that generates and optimizes an artificial intelligence model.
A data framework configured to receive input data and labels, perform data validation to generate a configuration file, and split the data to generate split data for training and evaluation.
Training and evaluation of the divided data is performed to measure the error level, actions are performed based on the error level, and the artificial intelligence model is generated based on the training, the evaluation, and the tuning. With a deep framework configured to serve the model in production,
A tuner framework configured to perform the action, the action including at least one of modifying the configuration file and automatically tuning the artificial intelligence model. Work and
Including the system.
(Appendix 16)
The tuner framework automatically optimizes one or more input elements related to the input data, and automatically optimizes and optimizes the hyperparameters related to the generated artificial intelligence model. The system according to Appendix 15, which is configured to automatically generate an updated model based on one or more entered inputs and the optimized hyperparameters.
(Appendix 17)
The tuner framework automatically optimizes the one or more input features by applying a general algorithm, optimizes the combination of the one or more input features, and optimizes the optimized features. The system according to Appendix 16, which generates a list of entered features of.
(Appendix 18)
The system according to Appendix 16, wherein the tuner framework automatically optimizes the hyperparameters by applying at least one of a Bayesian algorithm and a random algorithm and optimizing based on the hyperparameters.
(Appendix 19)
The tuner framework performs a step of automatically optimizing the one or more input features in the first iteration loop until the first specified number of iterations is met, and in the second iteration loop. 16. The system of Appendix 16, wherein the steps of automatically optimizing the hyperparameters and automatically generating the updated model are performed until the second defined number of iterations is met.
(Appendix 20)
19. The system of Appendix 19, wherein the tuner framework iteratively executes the first iterative loop and the second iterative loop until a third defined number of iterations is met.

2 Model generation server 3 Terminal device 10 Information providing device 20 Communication unit 30 Storage unit 40 Control unit 41 Acquisition unit 42 Index generation unit 43 Presentation unit 44 Reception unit 45 Model generation unit 46 Providing unit

Claims (23)

An acquisition unit that acquires training data used for model training,
A generation device including an index generation unit that generates a generation index of the model based on the characteristics of the training data. The generator according to claim 1, wherein the index generation unit generates the generation index based on the statistical characteristics of the learning data. The index generation unit according to claim 1 or 2, wherein the index generation unit generates the generation index based on whether the training data is an integer, a floating point number, or a character string. Generator. The generation device according to claim 3, wherein the index generation unit generates the generation index based on the continuity of the learning data when the learning data is an integer. The index generation unit is characterized in that when the density of the learning data exceeds a predetermined first threshold value, the index generation unit generates the generation index based on whether or not the maximum value of the learning data exceeds a predetermined second threshold value. The generator according to claim 4. When the density of the training data is lower than the predetermined first threshold value, the index generation unit generates the generation index based on whether or not the number of unique values included in the training data exceeds the predetermined third threshold value. The generator according to claim 4 or 5, wherein the generator is generated. Of claims 3 to 6, the index generation unit is particularly good at generating the generation index based on the number of types of character strings included in the training data when the learning data is a character string. The generator according to any one. The index generation unit is characterized in that, when the training data is a floating point number, a conversion index for inputting the training data into the input data is generated as a generation index of the model. The generator according to any one of 7. The acquisition unit acquires learning data showing a plurality of features and obtains learning data.
The index generation unit is any one of claims 1 to 8 characterized in that, as a generation index of the model, a generation index indicating a feature to be trained by the model is generated among the features of the training data. The generator described in. The acquisition unit acquires learning data indicating a plurality of types of features, and obtains learning data.
The index generation unit according to claim 1 to 9, wherein, as a generation index of the model, a generation index indicating a plurality of types of the training data types for learning the correlation with respect to the model is generated. The generator according to any one of them. The index generation unit according to any one of claims 1 to 10, wherein the index generation unit generates a generation index indicating the number of dimensions of the learning data input to the model as the generation index of the model. Generator. The index generation unit according to any one of claims 1 to 11, wherein the index generation unit generates a generation index indicating the type of the model for learning the characteristics of the training data as the generation index of the model. Generator. The index generation unit is any one of claims 1 to 12, characterized in that, as a generation index of the model, a generation index indicating the number of intermediate layers possessed by the model or the number of nodes included in each layer is generated. The generator according to one. The generation device according to any one of claims 1 to 13, wherein the index generation unit generates a generation index indicating a connection mode between nodes of the model as a generation index of the model. .. Any one of claims 1 to 14, wherein the index generation unit generates, as a generation index of the model, a generation index indicating a learning mode when the model is trained with the features of the training data. The generator according to one. The generator according to any one of claims 1 to 15, wherein the index generation unit generates a generation index indicating the size of the model as a generation index of the model. The index generation unit is characterized in that it generates a generation index indicating the characteristics of the learning data to be trained by the model, the mode of the model to be generated, and the learning mode when the model is trained with the features of the training data. The generator according to any one of claims 1 to 16. It has a reception unit that accepts from the user the specifications of the learning data to be trained by the model, the mode of the model to be generated, and the order of determining the learning mode when the model is trained with the features of the training data.
The index generation unit determines the characteristics of the training data to be trained by the model, the mode of the model to be generated, and the learning mode when the model is trained with the features of the training data, in the order specified by the user. The generator according to any one of claims 1 to 17, wherein the generator. The generator according to any one of claims 1 to 18, further comprising a model generator that generates a model in which the features of the training data are trained according to the generation index. The index generation unit newly generates a generation index of the model based on the accuracy of the model generated by the model generation unit.
The generation device according to claim 19, wherein the model generation unit newly generates the model according to a generation index newly generated by the index generation unit. The index generation unit generates a plurality of generation indexes,
The model generation unit generates the model for each generation index, and generates the model.
The generation device according to claim 20, wherein the index generation unit generates a new generation index from the plurality of generation indexes by using a genetic algorithm. It is a generation method executed by the generation device.
The acquisition process to acquire the training data used for model training,
A generation method including an index generation step of generating a generation index of the model based on the characteristics of the training data. Acquisition procedure to acquire training data used for model training,
A generation program for causing a computer to execute an index generation procedure for generating a generation index of the model based on the characteristics of the training data.