WO2022044233A1 - Estimation device, estimation method, and program - Google Patents

Estimation device, estimation method, and program Download PDF

Info

Publication number
WO2022044233A1
WO2022044233A1 PCT/JP2020/032485 JP2020032485W WO2022044233A1 WO 2022044233 A1 WO2022044233 A1 WO 2022044233A1 JP 2020032485 W JP2020032485 W JP 2020032485W WO 2022044233 A1 WO2022044233 A1 WO 2022044233A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
machine learning
inference result
data set
inference
Prior art date
Application number
PCT/JP2020/032485
Other languages
French (fr)
Japanese (ja)
Inventor
洋介 高橋
Original Assignee
日本電信電話株式会社
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 日本電信電話株式会社 filed Critical 日本電信電話株式会社
Priority to PCT/JP2020/032485 priority Critical patent/WO2022044233A1/en
Publication of WO2022044233A1 publication Critical patent/WO2022044233A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning

Definitions

  • the present invention relates to an estimation device, an estimation method and a program.
  • a feature quantity x of a feature quantity vector (x 1 , x 2 , x 3 , x 4 ).
  • a new feature vector (x 1 , x 2 ⁇ x 3 , x 4 ) can be created.
  • Non-Patent Documents 1 to 3 describe technologies called LIMITE (Local Interpretable Model-agnostic Expansions) and SHAP (SHapley Additive exPlanations), respectively, and create a plurality of data in which noise is added to input data, and they are described. Is input to the machine learning model, and the contribution of the feature to a specific inference result is estimated by observing the output result and performing a linear approximation.
  • LIMITE Local Interpretable Model-agnostic Expansions
  • SHAP SHapley Additive exPlanations
  • Non-Patent Document 3 the inference result when a specific feature amount of the input data is randomly rearranged is observed, and the degree of deterioration of the inference accuracy due to the rearrangement is evaluated as the importance of the feature amount. There is.
  • feature amount synthesis function the function used for feature amount synthesis
  • One embodiment of the present invention has been made in view of the above points, and an object thereof is to estimate a feature amount synthesis function.
  • the estimation device provides a machine learning service and a communication network that perform feature quantity synthesis on data containing one or more feature quantities and provide inference results for a predetermined task.
  • An estimation device connected via a device that transmits a first data set composed of data including one or more feature quantities to the machine learning service, and for each data constituting the first data set.
  • the first acquisition unit that acquires the first inference result data indicating the inference result
  • the generation unit that generates the candidate function of the feature quantity synthesis function used for the feature quantity synthesis of the machine learning service, and the candidate function.
  • To create a second data set composed of the converted data by performing a predetermined conversion on each data constituting the first data set based on the above, and the second data set.
  • a second acquisition unit that transmits a data set to the machine learning service and acquires a second inference result data indicating an inference result for each data constituting the second data set, and the first inference result data. It is characterized by having an estimation unit for calculating the similarity between the data and the second inference result data and estimating whether or not the candidate function is used for feature quantity synthesis of the machine learning service.
  • the feature composition function can be estimated.
  • the feature quantity synthesis function estimation system 1 capable of estimating the feature quantity synthesis function used will be described.
  • the feature amount synthesis function is a function used for feature amount synthesis, which is one of the preprocessing of data.
  • FIG. 1 is a diagram showing an example of the overall configuration of the feature quantity synthesis function estimation system 1 according to the present embodiment.
  • the feature quantity synthesis function estimation system 1 includes an estimation device 10, a user terminal 20, and a machine learning service providing device 30. These are communicably connected via any communication network, including the Internet N.
  • the estimation device 10 is a computer or a computer system that estimates the feature amount synthesis function used in the machine learning service.
  • the user terminal 20 is various terminals (for example, a PC (personal computer), a smartphone, a tablet terminal, etc.) used by the user of the machine learning service.
  • the user operates the user terminal 20 to send a set of training data for creating a machine learning model (hereinafter, also referred to as a “learning data set”) to the machine learning service providing device 30, or has already learned.
  • a set of data for obtaining an inference result (hereinafter, also referred to as "inference data set”) can be transmitted to the machine learning service providing device 30 by the machine learning model of.
  • the learning data set is composed of one or more learning data (training data), and each learning data includes a data ID, one or more feature amount (Feature), and an objective variable (Target). ..
  • the inference data set is composed of one or more inference data, and each inference data includes a data ID and one or more feature quantities.
  • the objective variable is a variable indicating the purpose of the machine learning model, and the feature quantity is a numerical value that characterizes the objective variable.
  • the machine learning model infers the value of the objective variable from the feature quantity contained in the inference data by learning the relationship between the feature quantity contained in the training data and the objective variable.
  • the machine learning service providing device 30 is a server managed by an external machine learning vendor or the like that provides the machine learning service.
  • the machine learning service has a learning phase in which a machine learning model is created using a learning data set and an inference phase in which an inference result is obtained from a machine learning model that has been trained using an inference data set.
  • the machine learning service providing device 30 performs predetermined preprocessing including feature quantity synthesis on each learning data constituting the learning data set transmitted from the user terminal 20, and then preprocesses.
  • a machine learning model is trained using each of the later training data.
  • the machine learning service providing device 30 performs preprocessing including feature quantity synthesis on each inference data constituting the inference data set transmitted from the user terminal 20, and then preprocesses.
  • An inference result is obtained by a machine learning model that has been trained using each inference data after processing, and is returned to the user terminal 20.
  • the machine learning service is, for example, an AutoML service or a service in which an external machine learning vendor creates a machine learning model
  • the user is a feature quantity synthesis function used in the machine learning service. It is not possible to know.
  • the configuration of the feature quantity synthesis function estimation system 1 shown in FIG. 1 is an example, and may be another configuration.
  • the estimation device 10 may be included in the user terminal 20.
  • FIG. 2 is a diagram showing an example of the hardware configuration of the estimation device 10 according to the present embodiment.
  • the estimation device 10 is realized by hardware of a general computer or computer system, and includes an input device 11, a display device 12, an external I / F13, and a communication I / F14. It has a processor 15 and a memory device 16. Each of these hardware is connected so as to be communicable via the bus 17.
  • the input device 11 is, for example, a keyboard, a mouse, a touch panel, or the like.
  • the display device 12 is, for example, a display or the like.
  • the estimation device 10 does not have to have at least one of the input device 11 and the display device 12.
  • the external I / F13 is an interface with an external device.
  • the external device includes a recording medium 13a and the like.
  • the estimation device 10 can read or write the recording medium 13a via the external I / F 13.
  • the recording medium 13a includes, for example, a CD (Compact Disc), a DVD (Digital Versatile Disk), an SD memory card (Secure Digital memory card), a USB (Universal Serial Bus) memory card, and the like.
  • the communication I / F 14 is an interface for connecting the estimation device 10 to the communication network.
  • the processor 15 is, for example, various arithmetic units such as a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit).
  • the memory device 16 is, for example, various storage devices such as an HDD (Hard Disk Drive), an SSD (Solid State Drive), a RAM (Random Access Memory), a ROM (Read Only Memory), and a flash memory.
  • the estimation device 10 By having the hardware configuration shown in FIG. 2, the estimation device 10 according to the present embodiment can realize the estimation process described later.
  • the hardware configuration shown in FIG. 2 is an example, and the estimation device 10 may have another hardware configuration.
  • the estimation device 10 may have a plurality of processors 15 or a plurality of memory devices 16.
  • FIG. 3 is a diagram showing an example of the functional configuration of the estimation device 10 according to the present embodiment.
  • the estimation device 10 includes an inference result acquisition unit 101, a function generation unit 102, a conversion unit 103, a distance calculation unit 104, and a determination unit 105.
  • Each of these parts is realized, for example, by a process of causing the processor 15 to execute one or more programs installed in the estimation device 10.
  • the estimation device 10 has a storage unit 106.
  • the storage unit 106 is realized by, for example, a memory device 16.
  • the storage unit 106 may be realized by, for example, a database server connected to the estimation device 10 via a communication network or the like.
  • the storage unit 106 stores the inference data set D used when estimating the feature amount synthesis function used in the machine learning service.
  • one or more inference data sets D may be stored in the storage unit 106.
  • the inference result acquisition unit 101 transmits the inference data set D stored in the storage unit 106 to the machine learning service providing device 30, and as a reply, the inference result of the trained machine learning model for the inference data set D. Acquire the data R. Further, the inference result acquisition unit 101 transmits the inference data set D'created by the conversion unit 103, which will be described later, to the machine learning service providing device 30, and as a reply, the learned machine for the inference data set D'. Acquire the inference result data R'of the training model.
  • the function generation unit 102 generates candidates for the feature amount synthesis function used in the machine learning service.
  • the feature amount synthesis function that is, the true feature amount synthesis function
  • trans t the candidate of the feature amount synthesis function created by the function generation unit 102
  • trans the candidate of the feature amount synthesis function created by the function generation unit 102
  • the distance calculation unit 104 calculates the similarity distance (R, R') between the inference result data R and the inference result data R'using a predetermined distance function distance.
  • the determination unit 105 determines whether or not the similarity distance (R, R') calculated by the distance calculation unit 104 is equal to or greater than a predetermined threshold value ⁇ . Then, when distance (R, R') ⁇ ⁇ , the determination unit 105 determines that the function trans created by the function generation unit 102 may be used in the machine learning service. On the other hand, when distance (R, R')> ⁇ , the determination unit 105 determines that the function trans created by the function generation unit 102 is not used in the machine learning service. As a result, the feature composition function used in the machine learning service is estimated.
  • FIG. 4 is a flowchart showing an example of the estimation process according to the present embodiment. It is assumed that the trained machine learning model has been created by the machine learning service.
  • the inference result acquisition unit 101 transmits the inference data set D stored in the storage unit 106 to the machine learning service providing device 30, and as a reply, of the trained machine learning model for the inference data set D.
  • the inference result data R is acquired (step S101).
  • the inference result of the trained machine learning model for the inference data d is represented by a scalar value
  • the inference result data R is represented by a one-dimensional vector. Therefore, for example, when the inference data set D is composed of n inference data d, the inference result data R is represented by a one-dimensional vector composed of n elements (inference result).
  • the function generation unit 102 generates candidate transs of the feature amount synthesis function used in the machine learning service (step S102).
  • the function generation unit 102 may randomly create a function trans from a combination of four rules of operation of each feature amount included in the inference data d constituting the inference data set D.
  • the inference data d is represented by a four-dimensional feature vector (x 1 , x 2 , x 3 , x 4 ) having four features x 1 , x 2 , x 3 , and x 4 .
  • the function generation unit 102 calculates a four-rule operation of two arbitrary features and generates a function that outputs a three-dimensional feature vector as a candidate trans of the feature composition function. Specifically, it is conceivable to generate the following function as a trans.
  • a function for calculating an arbitrary operation for example, logical operation, logarithmic conversion, exponential conversion, triangular function, exponentiation, root, etc.
  • a function for calculating an arbitrary operation for example, logical operation, logarithmic conversion, exponential conversion, triangular function, exponentiation, root, etc.
  • the user has some knowledge about inference data or a machine learning model (for example, knowledge that feature quantity synthesis is performed between two feature quantities), this knowledge is also included.
  • Candidate trans of the feature amount synthesis function may be generated by using.
  • a candidate trans of a certain feature amount synthesis function is generated in step S102 above.
  • the input of the candidate trans of the feature quantity synthesis function is also referred to as a "feature quantity vector”
  • the output is also referred to as a "composite feature quantity vector”.
  • the conversion unit 103 is composed of these inference data d'by converting each inference data d so as not to change the composite feature amount vector and creating new inference data d', respectively.
  • the conversion method in which the feature amount vector of the inference data d and the composite feature amount vector are the same differs depending on the candidate trans of the feature amount synthesis function.
  • the inference data d is represented by a four-dimensional feature vector (x 1 , x 2 , x 3 , x 4 )
  • FIGS. 5 to 5 an example of conversion when trans performs a simple four arithmetic operation is shown in FIGS. 5 to 5. It is shown in FIG.
  • the vector (x 1 / x 2 , 1, x 3 , x 4 ) representing each product) may be used as the feature quantity vector of the inference data d'having the same data ID as the inference data d.
  • trans (D) trans (D') and D ⁇ D'is established.
  • a vector (1, x 3, 1 / x 3, 1) representing the Hadamard product of the feature vector (x 1 , x 2 , x 3 , x 4 ) of the inference data d and the conversion vector (1, x 3 , 1 / x 3 , 1).
  • x 1 , x 2 x x 3 , 1, x 4 ) may be set as a feature vector of the inference data d'having the same data ID as the inference data d.
  • trans (D) trans (D') and D ⁇ D'is established.
  • x 2 + x 3 , 0, x 4 ) may be set as a feature vector of the inference data d'having the same data ID as the inference data d.
  • trans (D) trans (D') and D ⁇ D'is established.
  • X 2 -x 3 , 0, x 4 may be set as a feature vector of the inference data d'having the same data ID as the inference data d.
  • trans (D) trans (D') and D ⁇ D'is established.
  • the inference result acquisition unit 101 transmits the inference data set D'created in step S103 to the machine learning service providing device 30, and as a reply to the inference data set D'. Acquire the inference result data R'of the trained machine learning model. (Step S104).
  • the inference result data R' is also represented by a one-dimensional vector like the inference result data R.
  • the distance calculation unit 104 calculates the similarity distance (R, R') between the inference result data R and the inference result data R'using a predetermined distance function distance (step S105).
  • a distance function distance for example, a root mean square error (RMSE: Root Mean Square Error), a mean absolute error rate (MAPE: Mean Absolute Percentage Error), or the like may be used.
  • RMSE Root Mean Square Error
  • MPE Mean Absolute Percentage Error
  • the determination unit 105 determines whether or not the similarity distance (R, R') calculated in step S105 is equal to or greater than a predetermined threshold value ⁇ . Then, when the determination (R, R') ⁇ ⁇ , the determination unit 105 determines that the function trans generated in step S102 may be used as a feature composition function in the machine learning service. , Disstance (R, R')> ⁇ , it is determined that the function trans is not used as a feature composition function in the machine learning service (step S106). As a result, when distance (R, R') ⁇ ⁇ , the feature composition function used in the machine learning service is estimated.
  • the estimation device 10 changes the output information (inference result data R) of the inference data set D input to the external machine learning service and the inference data set D.
  • the output information (inference result data R') of By analyzing the output information (inference result data R') of, the feature quantity synthesis function used in the machine learning service can be estimated. This makes it possible to clarify the feature amount synthesis function used in the machine learning service (that is, to clarify the feature amount engineering process) and improve the explainability to the inference result of the machine learning model.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Software Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Medical Informatics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Artificial Intelligence (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

An estimation device according to one embodiment is an estimation device to be connected, through a communication network, to a machine learning service for providing an inference result of a predetermined task by performing feature amount combining on data including one or more feature amounts, the estimation device comprising: a first acquisition unit for transmitting a first data set made up of data including one or more feature amounts to the machine learning service, and acquiring first inference result data indicating an inference result for each piece of data forming the first data set; a generation unit for generating a candidate function for a feature amount combining function used for feature amount combining in the machine learning service; a creation unit for performing, on the basis of the candidate function, predetermined conversion on each piece of data forming the first data set to create a second data set made up of the converted data; a second acquisition unit for transmitting the second data set to the machine learning service and acquiring second inference result data indicating an inference result for each piece of data forming the second data set; and an estimation unit for calculating a degree of similarity between the first inference result data and the second inference result data and estimating whether the candidate function is used for feature amount combining in the machine learning service.

Description

推定装置、推定方法及びプログラムEstimator, estimation method and program
　本発明は、推定装置、推定方法及びプログラムに関する。 The present invention relates to an estimation device, an estimation method and a program.
　訓練データから何等かの特徴やパターンを学習することで、予測や分類といったタスクを実現する機械学習モデルに関する研究が行われている。従来、機械学習モデルを作成する際には専門的なスキルを有するモデル作成者が試行錯誤を重ねることで適切なモデルを作成し、タスクの精度を向上させてきた。しかしながら、機械学習モデルの作成には高度なスキルを必要とするため、近年では、必要となる訓練データを外部の機械学習ベンダに提供し、モデル作成を委託するケースが増加している。例えば、ユーザが訓練データをクラウド上にアップロードし、タスク内容を指定するだけで、自動で機械学習モデルの作成を実施するサービスが様々な機械学習ベンダから提供されている。このようなサービスは、一般に、ＡｕｔｏＭＬ（Automated Machine Learning）サービスとも呼ばれている。 Research is being conducted on machine learning models that realize tasks such as prediction and classification by learning some features and patterns from training data. Conventionally, when creating a machine learning model, a model creator with specialized skills has repeated trial and error to create an appropriate model and improve the accuracy of the task. However, since creating a machine learning model requires advanced skills, in recent years, there have been an increasing number of cases where necessary training data is provided to an external machine learning vendor and the model creation is outsourced. For example, various machine learning vendors provide services that automatically create machine learning models by simply uploading training data to the cloud and specifying task content. Such a service is also generally referred to as an AutoML (Automated Machine Learning) service.
　ＡｕｔｏＭＬサービスでは、一般に、（１）ユーザからアップロードされた複数の訓練データを読み込み、（２）各訓練データに対して前処理を実施した後、（３）複数の機械学習モデルに対して様々なパラメータを使って、前処理後の訓練データにより同時並行で学習させ、（４）それらの中で精度の高かった機械学習モデルをユーザに提供する、といったことが行われることが多い。また、上記の（２）の前処理としては、データの正規化や特徴量合成といった処理が行われることが多い。これらの前処理の中でも特徴量合成は機械学習モデルの推論結果の精度を左右するという点で重要な部分である。なお、特徴量合成とは特徴量ベクトルを変換して新たな特徴量ベクトルを作成する処理のことであり、例えば、特徴量ベクトル（ｘ１，ｘ２，ｘ３，ｘ４）の特徴量ｘ２とｘ３を乗算することで、新たな特徴量ベクトル（ｘ１，ｘ２×ｘ３，ｘ４）を作成すること等が挙げられる。 In the AutoML service, in general, (1) a plurality of training data uploaded from a user are read, (2) each training data is preprocessed, and then (3) various machine learning models are subjected to various methods. In many cases, parameters are used to train the training data after preprocessing in parallel, and (4) a machine learning model with high accuracy is provided to the user. Further, as the preprocessing of (2) above, processing such as data normalization and feature amount synthesis is often performed. Among these preprocessing, feature synthesis is an important part in that it affects the accuracy of the inference result of the machine learning model. Note that feature quantity synthesis is a process of converting a feature quantity vector to create a new feature quantity vector. For example, a feature quantity x of a feature quantity vector (x 1 , x 2 , x 3 , x 4 ). By multiplying 2 and x 3 , a new feature vector (x 1 , x 2 × x 3 , x 4 ) can be created.
　一方で、機械学習モデルの実社会への適用が進むにつれて、その説明可能性が重要な課題になりつつある。つまり、機械学習モデルが出力した推論結果（予測結果や分類結果等）に対して、その判断基準、すなわちデータ内のどの特徴量が推論結果にどの程度の影響を及ぼしたのかを提示することが求められている。 On the other hand, as the application of machine learning models to the real world progresses, the possibility of explaining them is becoming an important issue. In other words, for the inference results (prediction results, classification results, etc.) output by the machine learning model, it is possible to present the judgment criteria, that is, which features in the data have an effect on the inference results. It has been demanded.
　なお、機械学習モデルの説明可能性に関する従来技術としては、非特許文献１～３に記載されている技術が知られている。非特許文献１及び２にはそれぞれＬＩＭＥ（Local Interpretable Model-agnostic Explanations）及びＳＨＡＰ（SHapley Additive exPlanations）と呼ばれる技術が記載されており、入力データに対してノイズを加えたデータを複数作成し、それらを機械学習モデルに入力し、その出力結果を観測して線形近似することで特定の推論結果に対する特徴量の寄与度を推定している。また、非特許文献３には入力データの特定の特徴量をランダムに並び替えた際の推論結果を観測し、その並び替えによって推論精度が低下した度合いを当該特徴量の重要度として評価している。 As a conventional technique relating to the explainability of a machine learning model, the techniques described in Non-Patent Documents 1 to 3 are known. Non-Patent Documents 1 and 2 describe technologies called LIMITE (Local Interpretable Model-agnostic Expansions) and SHAP (SHapley Additive exPlanations), respectively, and create a plurality of data in which noise is added to input data, and they are described. Is input to the machine learning model, and the contribution of the feature to a specific inference result is estimated by observing the output result and performing a linear approximation. Further, in Non-Patent Document 3, the inference result when a specific feature amount of the input data is randomly rearranged is observed, and the degree of deterioration of the inference accuracy due to the rearrangement is evaluated as the importance of the feature amount. There is.
　ＡｕｔｏＭＬサービスや外部の機械学習ベンダを利用する場合、機械学習モデルの学習の際にどのような関数を用いて特徴量合成が行われているのかを後から知ることは困難である。一方で、特徴量合成に用いられている関数（以下、「特徴量合成関数」ともいう。）を知ることができれば、機械学習モデルの推論結果に対する説明可能性の向上に繋がると考えられる。しかしながら、機械学習モデルの説明可能性に関する従来技術では特徴量合成の影響については考慮されていなかった。 When using the AutoML service or an external machine learning vendor, it is difficult to know later what function is used for feature quantity synthesis when learning a machine learning model. On the other hand, if the function used for feature amount synthesis (hereinafter, also referred to as "feature amount synthesis function") can be known, it is considered that the possibility of explaining the inference result of the machine learning model will be improved. However, in the prior art regarding the explainability of machine learning models, the influence of feature synthesis was not taken into consideration.
　本発明の一実施形態は、上記の点に鑑みてなされたもので、特徴量合成関数を推定することを目的とする。 One embodiment of the present invention has been made in view of the above points, and an object thereof is to estimate a feature amount synthesis function.
　上記目的を達成するため、一実施形態に係る推定装置は、１以上の特徴量が含まれるデータに対して特徴量合成を行って所定のタスクの推論結果を提供する機械学習サービスと通信ネットワークを介して接続される推定装置であって、１以上の特徴量が含まれるデータで構成される第１のデータセットを前記機械学習サービスに送信し、前記第１のデータセットを構成する各データに対する推論結果を示す第１の推論結果データを取得する第１の取得部と、前記機械学習サービスの特徴量合成に利用されている特徴量合成関数の候補関数を生成する生成部と、前記候補関数に基づいて、前記第１のデータセットを構成する各データに対して所定の変換を行って、前記変換後のデータで構成される第２のデータセットを作成する作成部と、前記第２のデータセットを前記機械学習サービスに送信し、前記第２のデータセットを構成する各データに対する推論結果を示す第２の推論結果データを取得する第２の取得部と、前記第１の推論結果データと前記第２の推論結果データとの類似度を計算し、前記候補関数が前記機械学習サービスの特徴量合成に利用されているか否かを推定する推定部と、を有することを特徴とする。 In order to achieve the above object, the estimation device according to the embodiment provides a machine learning service and a communication network that perform feature quantity synthesis on data containing one or more feature quantities and provide inference results for a predetermined task. An estimation device connected via a device that transmits a first data set composed of data including one or more feature quantities to the machine learning service, and for each data constituting the first data set. The first acquisition unit that acquires the first inference result data indicating the inference result, the generation unit that generates the candidate function of the feature quantity synthesis function used for the feature quantity synthesis of the machine learning service, and the candidate function. To create a second data set composed of the converted data by performing a predetermined conversion on each data constituting the first data set based on the above, and the second data set. A second acquisition unit that transmits a data set to the machine learning service and acquires a second inference result data indicating an inference result for each data constituting the second data set, and the first inference result data. It is characterized by having an estimation unit for calculating the similarity between the data and the second inference result data and estimating whether or not the candidate function is used for feature quantity synthesis of the machine learning service.
　特徴量合成関数を推定することができる。 The feature composition function can be estimated.
本実施形態に係る特徴量合成関数推定システムの全体構成の一例を示す図である。It is a figure which shows an example of the whole structure of the feature quantity synthesis function estimation system which concerns on this embodiment. 本実施形態に係る推定装置のハードウェア構成の一例を示す図である。It is a figure which shows an example of the hardware composition of the estimation apparatus which concerns on this embodiment. 本実施形態に係る推定装置の機能構成の一例を示す図である。It is a figure which shows an example of the functional structure of the estimation apparatus which concerns on this embodiment. 本実施形態に係る推定処理の一例を示すフローチャートである。It is a flowchart which shows an example of the estimation process which concerns on this embodiment. 推論用データの変換の一例を説明するための図（その１）である。It is a figure (the 1) for demonstrating an example of conversion of inference data. 推論用データの変換の一例を説明するための図（その２）である。It is a figure (the 2) for demonstrating an example of conversion of inference data. 推論用データの変換の一例を説明するための図（その３）である。It is a figure (3) for demonstrating an example of conversion of inference data. 推論用データの変換の一例を説明するための図（その４）である。It is a figure (4) for demonstrating an example of conversion of inference data.
　以下、本発明の一実施形態について説明する。本実施形態では、外部の機械学習サービス（例えば、ＡｕｔｏＭＬサービス、外部の機械学習ベンダが機械学習モデルを作成するサービス等）で所定のタスクの機械学習モデルを作成した際に、この機械学習サービスで利用されている特徴量合成関数を推定することが可能な特徴量合成関数推定システム１について説明する。なお、特徴量合成関数とは、データの前処理の１つである特徴量合成に用いられる関数のことである。 Hereinafter, an embodiment of the present invention will be described. In the present embodiment, when a machine learning model of a predetermined task is created by an external machine learning service (for example, AutoML service, a service in which an external machine learning vendor creates a machine learning model, etc.), this machine learning service is used. The feature quantity synthesis function estimation system 1 capable of estimating the feature quantity synthesis function used will be described. The feature amount synthesis function is a function used for feature amount synthesis, which is one of the preprocessing of data.
　＜特徴量合成関数推定システム１の全体構成＞
　まず、本実施形態に係る特徴量合成関数推定システム１の全体構成について、図１を参照しながら説明する。図１は、本実施形態に係る特徴量合成関数推定システム１の全体構成の一例を示す図である。 <Overall configuration of feature amount synthesis function estimation system 1>
First, the overall configuration of the feature quantity synthesis function estimation system 1 according to the present embodiment will be described with reference to FIG. FIG. 1 is a diagram showing an example of the overall configuration of the feature quantity synthesis function estimation system 1 according to the present embodiment.
　図１に示すように、本実施形態に係る特徴量合成関数推定システム１には、推定装置１０と、ユーザ端末２０と、機械学習サービス提供装置３０とが含まれる。これらは、インターネットＮを含む任意の通信ネットワークを介して通信可能に接続される。 As shown in FIG. 1, the feature quantity synthesis function estimation system 1 according to the present embodiment includes an estimation device 10, a user terminal 20, and a machine learning service providing device 30. These are communicably connected via any communication network, including the Internet N.
　推定装置１０は、機械学習サービスで利用されている特徴量合成関数を推定するコンピュータ又はコンピュータシステムである。 The estimation device 10 is a computer or a computer system that estimates the feature amount synthesis function used in the machine learning service.
　ユーザ端末２０は、機械学習サービスのユーザが利用する各種端末（例えば、ＰＣ（パーソナルコンピュータ）やスマートフォン、タブレット端末等）である。ユーザは、ユーザ端末２０を操作して、機械学習モデルを作成するための訓練データの集合（以下、「学習用データセット」ともいう。）を機械学習サービス提供装置３０に送信したり、学習済みの機械学習モデルで推論結果を得るためのデータの集合（以下、「推論用データセット」ともいう。）を機械学習サービス提供装置３０に送信したりすることができる。 The user terminal 20 is various terminals (for example, a PC (personal computer), a smartphone, a tablet terminal, etc.) used by the user of the machine learning service. The user operates the user terminal 20 to send a set of training data for creating a machine learning model (hereinafter, also referred to as a “learning data set”) to the machine learning service providing device 30, or has already learned. A set of data for obtaining an inference result (hereinafter, also referred to as "inference data set") can be transmitted to the machine learning service providing device 30 by the machine learning model of.
　なお、学習用データセットは１以上の学習用データ（訓練データ）で構成されており、各学習用データにはデータＩＤと１以上の特徴量（Feature）と目的変数（Target）とが含まれる。一方で、推論用データセットは１以上の推論用データで構成されており、各推論用データにはデータＩＤと１以上の特徴量とが含まれる。目的変数は機械学習モデルの目的を示す変数であり、特徴量は目的変数を特徴づける数値である。機械学習モデルは学習用データに含まれる特徴量と目的変数の関係を学習することで、推論用データに含まれる特徴量から目的変数の値を推測する。 The learning data set is composed of one or more learning data (training data), and each learning data includes a data ID, one or more feature amount (Feature), and an objective variable (Target). .. On the other hand, the inference data set is composed of one or more inference data, and each inference data includes a data ID and one or more feature quantities. The objective variable is a variable indicating the purpose of the machine learning model, and the feature quantity is a numerical value that characterizes the objective variable. The machine learning model infers the value of the objective variable from the feature quantity contained in the inference data by learning the relationship between the feature quantity contained in the training data and the objective variable.
　機械学習サービス提供装置３０は、機械学習サービスを提供する外部の機械学習ベンダ等が管理するサーバである。 The machine learning service providing device 30 is a server managed by an external machine learning vendor or the like that provides the machine learning service.
　ここで、機械学習サービスには、学習用データセットを用いて機械学習モデルを作成する学習フェーズと、推論用データセットを用いて学習済みの機械学習モデルにより推論結果を得る推論フェーズとがある。学習フェーズでは、機械学習サービス提供装置３０は、ユーザ端末２０から送信された学習用データセットを構成する各学習用データに対して特徴量合成を含む所定の前処理を行った上で、前処理後の各学習用データを用いて機械学習モデルを学習させる。一方で、推論フェーズでは、機械学習サービス提供装置３０は、ユーザ端末２０から送信された推論用データセットを構成する各推論用データに対して特徴量合成を含む前処理を行った上で、前処理後の各推論用データを用いて学習済みの機械学習モデルにより推論結果を得て当該ユーザ端末２０に返信する。 Here, the machine learning service has a learning phase in which a machine learning model is created using a learning data set and an inference phase in which an inference result is obtained from a machine learning model that has been trained using an inference data set. In the learning phase, the machine learning service providing device 30 performs predetermined preprocessing including feature quantity synthesis on each learning data constituting the learning data set transmitted from the user terminal 20, and then preprocesses. A machine learning model is trained using each of the later training data. On the other hand, in the inference phase, the machine learning service providing device 30 performs preprocessing including feature quantity synthesis on each inference data constituting the inference data set transmitted from the user terminal 20, and then preprocesses. An inference result is obtained by a machine learning model that has been trained using each inference data after processing, and is returned to the user terminal 20.
　なお、上述したように、機械学習サービスは、例えば、ＡｕｔｏＭＬサービスや外部の機械学習ベンダが機械学習モデルを作成するサービス等であり、ユーザはその機械学習サービス内で利用されている特徴量合成関数を知ることはできないものとする。また、図１に示す特徴量合成関数推定システム１の構成は一例であって、他の構成であってもよい。例えば、推定装置１０がユーザ端末２０に含まれる構成であってもよい。 As described above, the machine learning service is, for example, an AutoML service or a service in which an external machine learning vendor creates a machine learning model, and the user is a feature quantity synthesis function used in the machine learning service. It is not possible to know. Further, the configuration of the feature quantity synthesis function estimation system 1 shown in FIG. 1 is an example, and may be another configuration. For example, the estimation device 10 may be included in the user terminal 20.
　＜推定装置１０のハードウェア構成＞
　次に、本実施形態に係る推定装置１０のハードウェア構成について、図２を参照しながら説明する。図２は、本実施形態に係る推定装置１０のハードウェア構成の一例を示す図である。 <Hardware configuration of estimation device 10>
Next, the hardware configuration of the estimation device 10 according to the present embodiment will be described with reference to FIG. FIG. 2 is a diagram showing an example of the hardware configuration of the estimation device 10 according to the present embodiment.
　図２に示すように、本実施形態に係る推定装置１０は一般的なコンピュータ又はコンピュータシステムのハードウェアで実現され、入力装置１１と、表示装置１２と、外部Ｉ／Ｆ１３と、通信Ｉ／Ｆ１４と、プロセッサ１５と、メモリ装置１６とを有する。これらの各ハードウェアは、それぞれがバス１７を介して通信可能に接続されている。 As shown in FIG. 2, the estimation device 10 according to the present embodiment is realized by hardware of a general computer or computer system, and includes an input device 11, a display device 12, an external I / F13, and a communication I / F14. It has a processor 15 and a memory device 16. Each of these hardware is connected so as to be communicable via the bus 17.
　入力装置１１は、例えば、キーボードやマウス、タッチパネル等である。表示装置１２は、例えば、ディスプレイ等である。なお、推定装置１０は、入力装置１１及び表示装置１２のうちの少なくとも一方を有していなくてもよい。 The input device 11 is, for example, a keyboard, a mouse, a touch panel, or the like. The display device 12 is, for example, a display or the like. The estimation device 10 does not have to have at least one of the input device 11 and the display device 12.
　外部Ｉ／Ｆ１３は、外部装置とのインタフェースである。外部装置には、記録媒体１３ａ等がある。推定装置１０は、外部Ｉ／Ｆ１３を介して、記録媒体１３ａの読み取りや書き込み等を行うことができる。なお、記録媒体１３ａには、例えば、ＣＤ（Compact Disc）、ＤＶＤ（Digital Versatile Disk）、ＳＤメモリカード（Secure Digital memory card）、ＵＳＢ（Universal Serial Bus）メモリカード等がある。 The external I / F13 is an interface with an external device. The external device includes a recording medium 13a and the like. The estimation device 10 can read or write the recording medium 13a via the external I / F 13. The recording medium 13a includes, for example, a CD (Compact Disc), a DVD (Digital Versatile Disk), an SD memory card (Secure Digital memory card), a USB (Universal Serial Bus) memory card, and the like.
　通信Ｉ／Ｆ１４は、推定装置１０を通信ネットワークに接続するためのインタフェースである。プロセッサ１５は、例えば、ＣＰＵ（Central Processing Unit）やＧＰＵ（Graphics Processing Unit）等の各種演算装置である。メモリ装置１６は、例えば、ＨＤＤ（Hard Disk Drive）やＳＳＤ（Solid State Drive）、ＲＡＭ（Random Access Memory）、ＲＯＭ（Read Only Memory）、フラッシュメモリ等の各種記憶装置である。 The communication I / F 14 is an interface for connecting the estimation device 10 to the communication network. The processor 15 is, for example, various arithmetic units such as a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). The memory device 16 is, for example, various storage devices such as an HDD (Hard Disk Drive), an SSD (Solid State Drive), a RAM (Random Access Memory), a ROM (Read Only Memory), and a flash memory.
　本実施形態に係る推定装置１０は、図２に示すハードウェア構成を有することにより、後述する推定処理を実現することができる。なお、図２に示すハードウェア構成は一例であって、推定装置１０は、他のハードウェア構成を有していてもよい。例えば、推定装置１０は、複数のプロセッサ１５を有していてもよいし、複数のメモリ装置１６を有していてもよい。 By having the hardware configuration shown in FIG. 2, the estimation device 10 according to the present embodiment can realize the estimation process described later. The hardware configuration shown in FIG. 2 is an example, and the estimation device 10 may have another hardware configuration. For example, the estimation device 10 may have a plurality of processors 15 or a plurality of memory devices 16.
　＜推定装置１０の機能構成＞
　次に、本実施形態に係る推定装置１０の機能構成について、図３を参照しながら説明する。図３は、本実施形態に係る推定装置１０の機能構成の一例を示す図である。 <Functional configuration of estimation device 10>
Next, the functional configuration of the estimation device 10 according to the present embodiment will be described with reference to FIG. FIG. 3 is a diagram showing an example of the functional configuration of the estimation device 10 according to the present embodiment.
　図３に示すように、本実施形態に係る推定装置１０は、推論結果取得部１０１と、関数生成部１０２と、変換部１０３と、距離計算部１０４と、判定部１０５とを有する。これら各部は、例えば、推定装置１０にインストールされた１以上のプログラムがプロセッサ１５に実行させる処理により実現される。 As shown in FIG. 3, the estimation device 10 according to the present embodiment includes an inference result acquisition unit 101, a function generation unit 102, a conversion unit 103, a distance calculation unit 104, and a determination unit 105. Each of these parts is realized, for example, by a process of causing the processor 15 to execute one or more programs installed in the estimation device 10.
　また、本実施形態に係る推定装置１０は、記憶部１０６を有する。記憶部１０６は、例えば、メモリ装置１６により実現される。ただし、記憶部１０６は、例えば、推定装置１０と通信ネットワークを介して接続されるデータベースサーバ等により実現されてもよい。 Further, the estimation device 10 according to the present embodiment has a storage unit 106. The storage unit 106 is realized by, for example, a memory device 16. However, the storage unit 106 may be realized by, for example, a database server connected to the estimation device 10 via a communication network or the like.
　記憶部１０６には、機械学習サービスで利用されている特徴量合成関数を推定する際に用いられる推論用データセットＤが記憶されている。なお、記憶部１０６には、１以上の推論用データセットＤが記憶されていてもよい。 The storage unit 106 stores the inference data set D used when estimating the feature amount synthesis function used in the machine learning service. In addition, one or more inference data sets D may be stored in the storage unit 106.
　推論結果取得部１０１は、記憶部１０６に記憶されている推論用データセットＤを機械学習サービス提供装置３０に送信し、その返信として、当該推論用データセットＤに対する学習済み機械学習モデルの推論結果データＲを取得する。また、推論結果取得部１０１は、後述する変換部１０３で作成された推論用データセットＤ'を機械学習サービス提供装置３０に送信し、その返信として、当該推論用データセットＤ'に対する学習済み機械学習モデルの推論結果データＲ'を取得する。 The inference result acquisition unit 101 transmits the inference data set D stored in the storage unit 106 to the machine learning service providing device 30, and as a reply, the inference result of the trained machine learning model for the inference data set D. Acquire the data R. Further, the inference result acquisition unit 101 transmits the inference data set D'created by the conversion unit 103, which will be described later, to the machine learning service providing device 30, and as a reply, the learned machine for the inference data set D'. Acquire the inference result data R'of the training model.
　関数生成部１０２は、機械学習サービスで利用されている特徴量合成関数の候補を生成する。以降では、機械学習サービスで利用されている特徴量合成関数（つまり、真の特徴量合成関数）をｔｒａｎｓｔ、関数生成部１０２により作成された特徴量合成関数の候補をｔｒａｎｓと表記する。 The function generation unit 102 generates candidates for the feature amount synthesis function used in the machine learning service. Hereinafter, the feature amount synthesis function (that is, the true feature amount synthesis function) used in the machine learning service is referred to as trans t , and the candidate of the feature amount synthesis function created by the function generation unit 102 is referred to as trans.
　変換部１０３は、推論用データセットＤを変換した新たな推論用データセットＤ'を作成する。このとき、変換部１０３は、推論用データセットＤに含まれる各推論用データｄに対してｔｒａｎｓ（ｄ）＝ｔｒａｎｓ（ｄ'）かつｄ≠ｄ'となる推論用データｄ'をそれぞれ作成し、これらの推論用データｄ'で構成される推論用データセットＤ'を作成する。なお、以降では、全ての推論用データｄと、当該推論用データｄから作成された推論用データｄ'との間でｔｒａｎｓ（ｄ）＝ｔｒａｎｓ（ｄ'）かつｄ≠ｄ'を満たす場合、推論用データセットＤと推論用データセットＤ'との間でｔｒａｎｓ（Ｄ）＝ｔｒａｎｓ（Ｄ'）かつＤ≠Ｄ'を満たすという。 The conversion unit 103 creates a new inference data set D'that is a conversion of the inference data set D. At this time, the conversion unit 103 creates inference data d'in which trans (d) = trans (d') and d ≠ d'for each inference data d included in the inference data set D. , Create an inference data set D'consisting of these inference data d'. In the following, when trans (d) = trans (d') and d ≠ d'are satisfied between all the inference data d and the inference data d'created from the inference data d. It is said that trans (D) = trans (D') and D ≠ D'are satisfied between the inference data set D and the inference data set D'.
　距離計算部１０４は、所定の距離関数ｄｉｓｔａｎｃｅを用いて、推論結果データＲと推論結果データＲ'との間の類似度ｄｉｓｔａｎｃｅ（Ｒ，Ｒ'）を計算する。 The distance calculation unit 104 calculates the similarity distance (R, R') between the inference result data R and the inference result data R'using a predetermined distance function distance.
　判定部１０５は、距離計算部１０４で計算された類似度ｄｉｓｔａｎｃｅ（Ｒ，Ｒ'）が所定の閾値σ以上であるか否かを判定する。そして、ｄｉｓｔａｎｃｅ（Ｒ，Ｒ'）≦σである場合は、判定部１０５は、関数生成部１０２で作成した関数ｔｒａｎｓが機械学習サービスで利用されている可能性があると判定する。一方で、ｄｉｓｔａｎｃｅ（Ｒ，Ｒ'）＞σである場合は、判定部１０５は、関数生成部１０２で作成した関数ｔｒａｎｓは機械学習サービスで利用されていないと判定する。これにより、機械学習サービスで利用されている特徴量合成関数が推定される。これは、推論用データセットＤを構成する各推論用データｄに対してｔｒａｎｓ（ｄ）＝ｔｒａｎｓ（ｄ'）となるように推論用データｄ'が作成されているため、特徴量合成関数の候補ｔｒａｎｓが真の特徴量合成関数ｔｒａｎｓｔと同一である場合には推論結果データＲと推論結果データＲ'が同一となるためである。 The determination unit 105 determines whether or not the similarity distance (R, R') calculated by the distance calculation unit 104 is equal to or greater than a predetermined threshold value σ. Then, when distance (R, R') ≦ σ, the determination unit 105 determines that the function trans created by the function generation unit 102 may be used in the machine learning service. On the other hand, when distance (R, R')> σ, the determination unit 105 determines that the function trans created by the function generation unit 102 is not used in the machine learning service. As a result, the feature composition function used in the machine learning service is estimated. This is because the inference data d'is created so that trans (d) = trans (d') for each inference data d constituting the inference data set D, so that the feature amount synthesis function is used. This is because the inference result data R and the inference result data R'are the same when the candidate trans is the same as the true feature amount synthesis function trans t .
　＜推定処理＞
　次に、本実施形態に係る推定装置１０により機械学習サービスで利用されている特徴量合成関数を推定する処理の流れについて、図４を参照しながら説明する。図４は、本実施形態に係る推定処理の一例を示すフローチャートである。なお、機械学習サービスによって学習済み機械学習モデルが作成されているものとする。 <Estimation processing>
Next, the flow of the process of estimating the feature amount synthesis function used in the machine learning service by the estimation device 10 according to the present embodiment will be described with reference to FIG. FIG. 4 is a flowchart showing an example of the estimation process according to the present embodiment. It is assumed that the trained machine learning model has been created by the machine learning service.
　まず、推論結果取得部１０１は、記憶部１０６に記憶されている推論用データセットＤを機械学習サービス提供装置３０に送信し、その返信として、当該推論用データセットＤに対する学習済み機械学習モデルの推論結果データＲを取得する（ステップＳ１０１）。 First, the inference result acquisition unit 101 transmits the inference data set D stored in the storage unit 106 to the machine learning service providing device 30, and as a reply, of the trained machine learning model for the inference data set D. The inference result data R is acquired (step S101).
　以降では、簡単のため、推論用データｄに対する学習済み機械学習モデルの推論結果はスカラー値で表され、推論結果データＲは１次元のベクトルで表されるものとする。したがって、例えば、推論用データセットＤがｎ個の推論用データｄで構成されている場合、推論結果データＲはｎ個の要素（推論結果）で構成される１次元のベクトルで表される。 From now on, for the sake of simplicity, the inference result of the trained machine learning model for the inference data d is represented by a scalar value, and the inference result data R is represented by a one-dimensional vector. Therefore, for example, when the inference data set D is composed of n inference data d, the inference result data R is represented by a one-dimensional vector composed of n elements (inference result).
　次に、関数生成部１０２は、機械学習サービスで利用されている特徴量合成関数の候補ｔｒａｎｓを生成する（ステップＳ１０２）。関数生成部１０２は、推論用データセットＤを構成する推論用データｄに含まれる各特徴量の四則演算の組み合わせ等からランダムに関数ｔｒａｎｓを作成すればよい。 Next, the function generation unit 102 generates candidate transs of the feature amount synthesis function used in the machine learning service (step S102). The function generation unit 102 may randomly create a function trans from a combination of four rules of operation of each feature amount included in the inference data d constituting the inference data set D.
　例えば、推論用データｄが４つの特徴量ｘ１，ｘ２，ｘ３，ｘ４を持つ４次元の特徴量ベクトル（ｘ１，ｘ２，ｘ３，ｘ４）で表されるとする。この場合、関数生成部１０２は、任意の２つの特徴量の四則演算を計算して３次元の特徴量ベクトルを出力する関数を特徴量合成関数の候補ｔｒａｎｓとして生成することが考えられる。具体的には、以下のような関数をｔｒａｎｓとして生成することが考えられる。 For example, suppose that the inference data d is represented by a four-dimensional feature vector (x 1 , x 2 , x 3 , x 4 ) having four features x 1 , x 2 , x 3 , and x 4 . In this case, it is conceivable that the function generation unit 102 calculates a four-rule operation of two arbitrary features and generates a function that outputs a three-dimensional feature vector as a candidate trans of the feature composition function. Specifically, it is conceivable to generate the following function as a trans.
　・ｔｒａｎｓ（ｘ１，ｘ２，ｘ３，ｘ４）＝（ｘ１／ｘ２，ｘ３，ｘ４）
　・ｔｒａｎｓ（ｘ１，ｘ２，ｘ３，ｘ４）＝（ｘ１，ｘ２×ｘ３，ｘ４）
　・ｔｒａｎｓ（ｘ１，ｘ２，ｘ３，ｘ４）＝（ｘ１，ｘ２＋ｘ３，ｘ４）
　・ｔｒａｎｓ（ｘ１，ｘ２，ｘ３，ｘ４）＝（ｘ１，ｘ２－ｘ３，ｘ４）
　また、例えば、任意の３つの特徴量の四則演算の組み合わせを計算して２次元の特徴量ベクトルを出力する関数を特徴量合成関数の候補ｔｒａｎｓとして生成する場合は、以下のような関数をｔｒａｎｓとして生成することが考えられる。 -Trans (x 1 , x 2 , x 3 , x 4 ) = (x 1 / x 2 , x 3 , x 4 )
-Trans (x 1 , x 2 , x 3 , x 4 ) = (x 1 , x 2 x x 3 , x 4 )
-Trans (x 1 , x 2 , x 3 , x 4 ) = (x 1 , x 2 + x 3 , x 4 )
-Trans (x 1 , x 2 , x 3 , x 4 ) = (x 1 , x 2 -x 3 , x 4 )
Further, for example, when a function that calculates a combination of four arithmetic operations of any three features and outputs a two-dimensional feature vector is generated as a candidate trans of a feature composition function, the following function is trans. It is conceivable to generate as.
　・ｔｒａｎｓ（ｘ１，ｘ２，ｘ３，ｘ４）＝（ｘ１，ｘ２／ｘ３＋ｘ４）
　・ｔｒａｎｓ（ｘ１，ｘ２，ｘ３，ｘ４）＝（ｘ１，ｘ２×（ｘ３－ｘ４））
　・ｔｒａｎｓ（ｘ１，ｘ２，ｘ３，ｘ４）＝（ｘ１＋ｘ２＋ｘ３，ｘ４）
　・ｔｒａｎｓ（ｘ１，ｘ２，ｘ３，ｘ４）＝（ｘ１×（ｘ２＋ｘ３），ｘ４）
　ただし、これらの関数は一例であって、推論用データｄに含まれる任意の個数の特徴量の任意の四則演算又はその組み合わせを計算して任意の次元の特徴量ベクトルを出力する関数をｔｒａｎｓとして生成することが可能である。また、四則演算以外にも任意の演算（例えば、論理演算、対数変換、指数変換、三角関数、冪乗、冪根等）を計算する関数がｔｒａｎｓとして生成されてもよい。更に、ユーザが推論用データや機械学習モデルに関する何等かの知識（例えば、或る２つの特徴量間で特徴量合成が行われているとの知識等）を有する場合には、これらの知識も用いて特徴量合成関数の候補ｔｒａｎｓが生成されてもよい。 -Trans (x 1 , x 2 , x 3 , x 4 ) = (x 1 , x 2 / x 3 + x 4 )
-Trans (x 1 , x 2 , x 3 , x 4 ) = (x 1 , x 2 x (x 3 -x 4 ))
-Trans (x 1 , x 2 , x 3 , x 4 ) = (x 1 + x 2 + x 3 , x 4 )
-Trans (x 1 , x 2 , x 3 , x 4 ) = (x 1 x (x 2 + x 3 ), x 4 )
However, these functions are an example, and a function that calculates an arbitrary four arithmetic operations of an arbitrary number of features included in the inference data d or a combination thereof and outputs a feature vector of an arbitrary dimension is defined as trans. It is possible to generate. In addition to the four-rule operation, a function for calculating an arbitrary operation (for example, logical operation, logarithmic conversion, exponential conversion, triangular function, exponentiation, root, etc.) may be generated as trans. Further, if the user has some knowledge about inference data or a machine learning model (for example, knowledge that feature quantity synthesis is performed between two feature quantities), this knowledge is also included. Candidate trans of the feature amount synthesis function may be generated by using.
　なお、以降では、上記のステップＳ１０２で或る１つの特徴量合成関数の候補ｔｒａｎｓが生成されたものとする。また、特徴量合成関数の候補ｔｒａｎｓの入力を「特徴量ベクトル」、出力を「合成特徴量ベクトル」ともいう。 In the following, it is assumed that a candidate trans of a certain feature amount synthesis function is generated in step S102 above. Further, the input of the candidate trans of the feature quantity synthesis function is also referred to as a "feature quantity vector", and the output is also referred to as a "composite feature quantity vector".
　次に、変換部１０３は、合成特徴量ベクトルを変化させないように各推論用データｄを変換し、新たな推論用データｄ'をそれぞれ作成することで、これらの推論用データｄ'で構成される推論用データセットＤ'を作成する（ステップＳ１０３）。すなわち、変換部１０３は、ｔｒａｎｓ（Ｄ）＝ｔｒａｎｓ（Ｄ'）かつＤ≠Ｄ'となる推論用データセットＤ'を作成する。 Next, the conversion unit 103 is composed of these inference data d'by converting each inference data d so as not to change the composite feature amount vector and creating new inference data d', respectively. Inference data set D'is created (step S103). That is, the conversion unit 103 creates an inference data set D'in which trans (D) = trans (D') and D ≠ D'.
　ここで、推論用データｄの特徴量ベクトルと合成特徴量ベクトルとが同一となるような変換方法は特徴量合成関数の候補ｔｒａｎｓによって異なる。以下、推論用データｄが４次元の特徴量ベクトル（ｘ１，ｘ２，ｘ３，ｘ４）で表されるものとして、ｔｒａｎｓが単純な四則演算を行う場合の変換の例を図５～図８に示す。 Here, the conversion method in which the feature amount vector of the inference data d and the composite feature amount vector are the same differs depending on the candidate trans of the feature amount synthesis function. Hereinafter, assuming that the inference data d is represented by a four-dimensional feature vector (x 1 , x 2 , x 3 , x 4 ), an example of conversion when trans performs a simple four arithmetic operation is shown in FIGS. 5 to 5. It is shown in FIG.
　図５は、ｔｒａｎｓ（ｘ１，ｘ２，ｘ３，ｘ４）＝（ｘ１／ｘ２，ｘ３，ｘ４）である場合の例を示している。この場合、推論用データｄの特徴量ベクトル（ｘ１，ｘ２，ｘ３，ｘ４）と、（１／ｘ２，１／ｘ２，１，１）という変換ベクトルとのアダマール積（要素毎の積）を表すベクトル（ｘ１／ｘ２，１，ｘ３，ｘ４）を、当該推論用データｄと同一データＩＤを持つ推論用データｄ'の特徴量ベクトルとすればよい。これにより、ｔｒａｎｓ（Ｄ）＝ｔｒａｎｓ（Ｄ'）かつＤ≠Ｄ'が成立する。 FIG. 5 shows an example in the case where trans (x 1 , x 2 , x 3 , x 4 ) = (x 1 / x 2 , x 3 , x 4 ). In this case, the Hadamard product (element) of the feature vector (x 1 , x 2 , x 3 , x 4 ) of the inference data d and the transformation vector (1 / x 2 , 1 / x 2 , 1, 1). The vector (x 1 / x 2 , 1, x 3 , x 4 ) representing each product) may be used as the feature quantity vector of the inference data d'having the same data ID as the inference data d. As a result, trans (D) = trans (D') and D ≠ D'is established.
　図６は、ｔｒａｎｓ（ｘ１，ｘ２，ｘ３，ｘ４）＝（ｘ１，ｘ２×ｘ３，ｘ４）である場合の例を示している。この場合、推論用データｄの特徴量ベクトル（ｘ１，ｘ２，ｘ３，ｘ４）と、（１，ｘ３，１／ｘ３，１）という変換ベクトルとのアダマール積を表すベクトル（ｘ１，ｘ２×ｘ３，１，ｘ４）を、当該推論用データｄと同一データＩＤを持つ推論用データｄ'の特徴量ベクトルとすればよい。これにより、ｔｒａｎｓ（Ｄ）＝ｔｒａｎｓ（Ｄ'）かつＤ≠Ｄ'が成立する。 FIG. 6 shows an example in the case where trans (x 1 , x 2 , x 3 , x 4 ) = (x 1 , x 2 × x 3 , x 4 ). In this case, a vector (1, x 3, 1 / x 3, 1) representing the Hadamard product of the feature vector (x 1 , x 2 , x 3 , x 4 ) of the inference data d and the conversion vector (1, x 3 , 1 / x 3 , 1). x 1 , x 2 x x 3 , 1, x 4 ) may be set as a feature vector of the inference data d'having the same data ID as the inference data d. As a result, trans (D) = trans (D') and D ≠ D'is established.
　図７は、ｔｒａｎｓ（ｘ１，ｘ２，ｘ３，ｘ４）＝（ｘ１，ｘ２＋ｘ３，ｘ４）である場合の例を示している。この場合、推論用データｄの特徴量ベクトル（ｘ１，ｘ２，ｘ３，ｘ４）に対して（０，ｘ３，－ｘ３，０）という変換ベクトルを加えたベクトル（ｘ１，ｘ２＋ｘ３，０，ｘ４）を、当該推論用データｄと同一データＩＤを持つ推論用データｄ'の特徴量ベクトルとすればよい。これにより、ｔｒａｎｓ（Ｄ）＝ｔｒａｎｓ（Ｄ'）かつＤ≠Ｄ'が成立する。 FIG. 7 shows an example in the case where trans (x 1 , x 2 , x 3 , x 4 ) = (x 1 , x 2 + x 3 , x 4 ). In this case, the vector (x 1 , 0) to which the conversion vector (0, x 3 , -x 3 , 0) is added to the feature vector (x 1 , x 2 , x 3 , x 4 ) of the inference data d. x 2 + x 3 , 0, x 4 ) may be set as a feature vector of the inference data d'having the same data ID as the inference data d. As a result, trans (D) = trans (D') and D ≠ D'is established.
　図８は、ｔｒａｎｓ（ｘ１，ｘ２，ｘ３，ｘ４）＝（ｘ１，ｘ２－ｘ３，ｘ４）である場合の例を示している。この場合、推論用データｄの特徴量ベクトル（ｘ１，ｘ２，ｘ３，ｘ４）に対して（０，－ｘ３，－ｘ３，０）という変換ベクトルを加えたベクトル（ｘ１，ｘ２－ｘ３，０，ｘ４）を、当該推論用データｄと同一データＩＤを持つ推論用データｄ'の特徴量ベクトルとすればよい。これにより、ｔｒａｎｓ（Ｄ）＝ｔｒａｎｓ（Ｄ'）かつＤ≠Ｄ'が成立する。 FIG. 8 shows an example in the case where trans (x 1 , x 2 , x 3 , x 4 ) = (x 1 , x 2 -x 3 , x 4 ). In this case, a vector (x 1 ) obtained by adding a conversion vector (0, -x 3 , -x 3 , 0) to the feature vector (x 1 , x 2 , x 3 , x 4 ) of the inference data d. , X 2 -x 3 , 0, x 4 ) may be set as a feature vector of the inference data d'having the same data ID as the inference data d. As a result, trans (D) = trans (D') and D ≠ D'is established.
　図４に戻る。ステップＳ１０３に続いて、推論結果取得部１０１は、上記のステップＳ１０３で作成された推論用データセットＤ'を機械学習サービス提供装置３０に送信し、その返信として、当該推論用データセットＤ' に対する学習済み機械学習モデルの推論結果データＲ'を取得する。（ステップＳ１０４）。なお、推論結果データＲ'も推論結果データＲと同様に、１次元のベクトルで表される。 Return to Fig. 4. Following step S103, the inference result acquisition unit 101 transmits the inference data set D'created in step S103 to the machine learning service providing device 30, and as a reply to the inference data set D'. Acquire the inference result data R'of the trained machine learning model. (Step S104). The inference result data R'is also represented by a one-dimensional vector like the inference result data R.
　以降では、推論用データセットＤはデータＩＤ「１」～「ｎ」までのｎ個の推論用データｄで構成されているものとして、Ｒ＝（ｒ１，ｒ２，・・・，ｒｎ）と表されているものとする。ここで、ｒｋはデータＩＤ「ｋ」（ただし、ｋ＝１，・・・，ｎ）の推論用データｄに対する学習済み機械学習モデルの推論結果である。同様に、データＩＤ「ｋ」（ただし、ｋ＝１，・・・，ｎ）の推論用データｄ'に対する学習済み機械学習モデルの推論結果をｒｉ'として、Ｒ'＝（ｒ１'，ｒ２'，・・・，ｒｎ'）と表されているものとする。 Hereinafter, it is assumed that the inference data set D is composed of n inference data d from the data IDs “1” to “n”, and R = (r 1 , r 2 , ..., R n) . ). Here, r k is the inference result of the trained machine learning model for the inference data d of the data ID “k” (where k = 1, ..., N). Similarly, the inference result of the trained machine learning model for the inference data d'of the data ID "k" (however, k = 1, ..., N) is set as r i ', and R'= (r 1 ', It shall be expressed as r 2 ', ..., r n ').
　次に、距離計算部１０４は、所定の距離関数ｄｉｓｔａｎｃｅを用いて、推論結果データＲと推論結果データＲ'との間の類似度ｄｉｓｔａｎｃｅ（Ｒ，Ｒ'）を計算する（ステップＳ１０５）。距離関数ｄｉｓｔａｎｃｅとしては、例えば、平均平方二乗誤差（ＲＭＳＥ：Root Mean Square Error）や平均絶対誤差率（ＭＡＰＥ：Mean Absolute Percentage Error）等を用いればよい。例えば、平均平方二乗誤差を用いる場合、距離関数ｄｉｓｔａｎｃｅ（Ｒ，Ｒ'）は以下で計算することができる。 Next, the distance calculation unit 104 calculates the similarity distance (R, R') between the inference result data R and the inference result data R'using a predetermined distance function distance (step S105). As the distance function distance, for example, a root mean square error (RMSE: Root Mean Square Error), a mean absolute error rate (MAPE: Mean Absolute Percentage Error), or the like may be used. For example, when using the root mean square error, the distance function distance (R, R') can be calculated as follows.
Figure JPOXMLDOC01-appb-M000001
 　次に、判定部１０５は、上記のステップＳ１０５で計算された類似度ｄｉｓｔａｎｃｅ（Ｒ，Ｒ'）が所定の閾値σ以上であるか否かを判定する。そして、判定部１０５は、ｄｉｓｔａｎｃｅ（Ｒ，Ｒ'）≦σである場合は上記のステップＳ１０２で生成した関数ｔｒａｎｓが機械学習サービスで特徴量合成関数として利用されている可能性があると判定し、ｄｉｓｔａｎｃｅ（Ｒ，Ｒ'）＞σである場合は当該関数ｔｒａｎｓが機械学習サービスで特徴量合成関数としては利用されていないと判定する（ステップＳ１０６）。これにより、ｄｉｓｔａｎｃｅ（Ｒ，Ｒ'）≦σである場合は機械学習サービスで利用されている特徴量合成関数が推定される。これは、ｔｒａｎｓ（Ｄ）＝ｔｒａｎｓ（Ｄ'）かつＤ≠Ｄ'となるように推論用データセットＤ'が作成されているため、ｔｒａｎｓ＝ｔｒａｎｓｔである場合にはＲ＝Ｒ'でありｄｉｓｔａｎｃｅ（Ｒ，Ｒ'）＝０となる一方で、ｔｒａｎｓ≠ｔｒａｎｓｔである場合にはＲ≠Ｒ'であり或る閾値σに対してｄｉｓｔａｎｃｅ（Ｒ，Ｒ'）＞σとなるためである。
Figure JPOXMLDOC01-appb-M000001
 Next, the determination unit 105 determines whether or not the similarity distance (R, R') calculated in step S105 is equal to or greater than a predetermined threshold value σ. Then, when the determination (R, R') ≤ σ, the determination unit 105 determines that the function trans generated in step S102 may be used as a feature composition function in the machine learning service. , Disstance (R, R')> σ, it is determined that the function trans is not used as a feature composition function in the machine learning service (step S106). As a result, when distance (R, R') ≤ σ, the feature composition function used in the machine learning service is estimated. This is because the inference data set D'is created so that trans (D) = trans (D') and D ≠ D', so when trans = trans t , R = R'. This is because while distance (R, R') = 0, when trans ≠ trans t , R ≠ R'and distance (R, R')> σ for a certain threshold value σ. ..
　以上のように、本実施形態に係る推定装置１０は、外部の機械学習サービスに入力される推論用データセットＤの出力情報（推論結果データＲ）と当該推論用データセットＤを変化させた場合の出力情報（推論結果データＲ'）とを分析することで、当該機械学習サービスで利用されている特徴量合成関数を推定することができる。これにより、機械学習サービスで利用されている特徴量合成関数を明らかにし（つまり、特徴量エンジニアリング処理を明らかにし）、機械学習モデルの推論結果に対する説明可能性を向上させることが可能となる。 As described above, the estimation device 10 according to the present embodiment changes the output information (inference result data R) of the inference data set D input to the external machine learning service and the inference data set D. By analyzing the output information (inference result data R') of, the feature quantity synthesis function used in the machine learning service can be estimated. This makes it possible to clarify the feature amount synthesis function used in the machine learning service (that is, to clarify the feature amount engineering process) and improve the explainability to the inference result of the machine learning model.
　本発明は、具体的に開示された上記の実施形態に限定されるものではなく、請求の範囲の記載から逸脱することなく、種々の変形や変更、既知の技術との組み合わせ等が可能である。 The present invention is not limited to the above-described embodiment specifically disclosed, and various modifications and modifications, combinations with known techniques, and the like are possible without departing from the description of the claims. ..
　１　　　　特徴量合成関数推定システム
　１０　　　推定装置
　１１　　　入力装置
　１２　　　表示装置
　１３　　　外部Ｉ／Ｆ
　１３ａ　　記録媒体
　１４　　　通信Ｉ／Ｆ
　１５　　　プロセッサ
　１６　　　メモリ装置
　１７　　　バス
　２０　　　ユーザ端末
　３０　　　機械学習サービス提供装置
　１０１　　推論結果取得部
　１０２　　関数生成部
　１０３　　変換部
　１０４　　距離計算部
　１０５　　判定部
　１０６　　記憶部
　Ｎ　　　　インターネット 1 Feature composition function estimation system 10 Estimator 11 Input device 12 Display device 13 External I / F
13a Recording medium 14 Communication I / F
15 Processor 16 Memory device 17 Bus 20 User terminal 30 Machine learning service provider 101 Inference result acquisition unit 102 Function generation unit 103 Conversion unit 104 Distance calculation unit 105 Judgment unit 106 Storage unit N Internet

Claims (6)

  1. 　１以上の特徴量が含まれるデータに対して特徴量合成を行って所定のタスクの推論結果を提供する機械学習サービスと通信ネットワークを介して接続される推定装置であって、
    　１以上の特徴量が含まれるデータで構成される第１のデータセットを前記機械学習サービスに送信し、前記第１のデータセットを構成する各データに対する推論結果を示す第１の推論結果データを取得する第１の取得部と、
    　前記機械学習サービスの特徴量合成に利用されている特徴量合成関数の候補関数を生成する生成部と、
    　前記候補関数に基づいて、前記第１のデータセットを構成する各データに対して所定の変換を行って、前記変換後のデータで構成される第２のデータセットを作成する作成部と、
    　前記第２のデータセットを前記機械学習サービスに送信し、前記第２のデータセットを構成する各データに対する推論結果を示す第２の推論結果データを取得する第２の取得部と、
    　前記第１の推論結果データと前記第２の推論結果データとの類似度を計算し、前記候補関数が前記機械学習サービスの特徴量合成に利用されているか否かを推定する推定部と、
    　を有することを特徴とする推定装置。     An estimation device connected via a communication network to a machine learning service that synthesizes features for data containing one or more features and provides inference results for a predetermined task.
    A first data set composed of data including one or more features is transmitted to the machine learning service, and a first inference result data showing an inference result for each data constituting the first data set is transmitted. The first acquisition unit to acquire and
    A generator that generates a candidate function for the feature synthesis function used for feature synthesis of the machine learning service, and a generator.
    A creation unit that creates a second data set composed of the converted data by performing a predetermined conversion on each data constituting the first data set based on the candidate function.
    A second acquisition unit that transmits the second data set to the machine learning service and acquires the second inference result data indicating the inference result for each data constituting the second data set.
    An estimation unit that calculates the similarity between the first inference result data and the second inference result data and estimates whether or not the candidate function is used for feature quantity synthesis of the machine learning service.
    An estimation device characterized by having.
  2. 　前記作成部は、
    　前記第１のデータセットを構成する各データに対する前記候補関数の出力を変化させないように、前記第１のデータセットを構成する各データに対して所定の変換を行う、ことを特徴とする請求項１に記載の推定装置。     The creation part
    The claim is characterized in that a predetermined conversion is performed for each data constituting the first data set so as not to change the output of the candidate function for each data constituting the first data set. The estimation device according to 1.
  3. 　前記推定部は、
    　所定の距離関数を用いて前記第１の推論結果データと前記第２の推論結果データとの類似度を計算し、
    　前記類似度が所定の閾値以下である場合は、前記候補関数が前記機械学習サービスの特徴量合成に利用されていると推定し、
    　前記類似度が所定の閾値より大きい場合は、前記候補関数は前記機械学習サービスの特徴量合成に利用されていないと推定する、ことを特徴とする請求項１又は２に記載の推定装置。     The estimation unit
    Using a predetermined distance function, the similarity between the first inference result data and the second inference result data is calculated.
    When the similarity is equal to or less than a predetermined threshold value, it is estimated that the candidate function is used for feature quantity synthesis of the machine learning service.
    The estimation device according to claim 1 or 2, wherein when the similarity is larger than a predetermined threshold value, it is estimated that the candidate function is not used for feature quantity synthesis of the machine learning service.
  4. 　前記生成部は、
    　前記１以上の特徴量間で所定の演算を行う関数を前記候補関数としてランダムに生成する、ことを特徴とする請求項１乃至３の何れか一項に記載の推定装置。     The generator is
    The estimation device according to any one of claims 1 to 3, wherein a function that performs a predetermined operation between the one or more feature quantities is randomly generated as the candidate function.
  5. 　１以上の特徴量が含まれるデータに対して特徴量合成を行って所定のタスクの推論結果を提供する機械学習サービスと通信ネットワークを介して接続される推定装置が、
    　１以上の特徴量が含まれるデータで構成される第１のデータセットを前記機械学習サービスに送信し、前記第１のデータセットを構成する各データに対する推論結果を示す第１の推論結果データを取得する第１の取得手順と、
    　前記機械学習サービスの特徴量合成に利用されている特徴量合成関数の候補関数を生成する生成手順と、
    　前記候補関数に基づいて、前記第１のデータセットを構成する各データに対して所定の変換を行って、前記変換後のデータで構成される第２のデータセットを作成する作成手順と、
    　前記第２のデータセットを前記機械学習サービスに送信し、前記第２のデータセットを構成する各データに対する推論結果を示す第２の推論結果データを取得する第２の取得手順と、
    　前記第１の推論結果データと前記第２の推論結果データとの類似度を計算し、前記候補関数が前記機械学習サービスの特徴量合成に利用されているか否かを推定する推定手順と、
    　を実行することを特徴とする推定方法。     An estimation device connected via a communication network to a machine learning service that synthesizes features for data containing one or more features and provides inference results for a predetermined task.
    A first data set composed of data including one or more features is transmitted to the machine learning service, and a first inference result data showing an inference result for each data constituting the first data set is transmitted. The first acquisition procedure to acquire and
    A generation procedure for generating a candidate function of the feature amount synthesis function used for feature amount synthesis of the machine learning service, and a generation procedure.
    Based on the candidate function, a predetermined conversion is performed on each data constituting the first data set to create a second data set composed of the converted data, and a creation procedure.
    The second acquisition procedure of transmitting the second data set to the machine learning service and acquiring the second inference result data indicating the inference result for each data constituting the second data set, and the second acquisition procedure.
    An estimation procedure for calculating the similarity between the first inference result data and the second inference result data and estimating whether or not the candidate function is used for feature quantity synthesis of the machine learning service, and an estimation procedure.
    An estimation method characterized by performing.
  6. 　コンピュータを、請求項１乃至４の何れか一項に記載の推定装置として機能させるプログラム。 A program that causes a computer to function as the estimation device according to any one of claims 1 to 4.
PCT/JP2020/032485 2020-08-27 2020-08-27 Estimation device, estimation method, and program WO2022044233A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
PCT/JP2020/032485 WO2022044233A1 (en) 2020-08-27 2020-08-27 Estimation device, estimation method, and program

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2020/032485 WO2022044233A1 (en) 2020-08-27 2020-08-27 Estimation device, estimation method, and program

Publications (1)

Publication Number Publication Date
WO2022044233A1 true WO2022044233A1 (en) 2022-03-03

Family

ID=80352918

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2020/032485 WO2022044233A1 (en) 2020-08-27 2020-08-27 Estimation device, estimation method, and program

Country Status (1)

Country Link
WO (1) WO2022044233A1 (en)

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2017529583A (en) * 2014-06-30 2017-10-05 アマゾン・テクノロジーズ・インコーポレーテッド Feature processing trade-off management
US10510022B1 (en) * 2018-12-03 2019-12-17 Sas Institute Inc. Machine learning model feature contribution analytic system

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2017529583A (en) * 2014-06-30 2017-10-05 アマゾン・テクノロジーズ・インコーポレーテッド Feature processing trade-off management
US10510022B1 (en) * 2018-12-03 2019-12-17 Sas Institute Inc. Machine learning model feature contribution analytic system

Similar Documents

Publication Publication Date Title
Pal et al. Practical time series analysis: master time series data processing, visualization, and modeling using python
Loper et al. SMPL: A skinned multi-person linear model
Desislavov et al. Compute and energy consumption trends in deep learning inference
Talagala et al. Meta-learning how to forecast time series
US11373117B1 (en) Artificial intelligence service for scalable classification using features of unlabeled data and class descriptors
Ajdari et al. An adaptive exploration-exploitation algorithm for constructing metamodels in random simulation using a novel sequential experimental design
Dong et al. Adaptive neural network-based approximation to accelerate eulerian fluid simulation
CN111401700A (en) Data analysis method, device, computer system and readable storage medium
KR102192949B1 (en) Apparatus and method for evaluating start-up companies using artifical neural network
US20230040564A1 (en) Learning Causal Relationships
US20210151128A1 (en) Learning Method, Mixing Ratio Prediction Method, and Prediction Device
CN111652453A (en) Intelligent workflow advisor for part design, simulation and manufacturing
Wheeler Bayesian additive adaptive basis tensor product models for modeling high dimensional surfaces: an application to high-throughput toxicity testing
CN115545114A (en) Training method of multi-task model, content recommendation method and device
WO2022044233A1 (en) Estimation device, estimation method, and program
CN116662538A (en) Text abstract generation method, device, equipment and medium based on multitask learning
US20230139396A1 (en) Using learned physical knowledge to guide feature engineering
Liang et al. Long-term RNN: Predicting hazard function for proactive maintenance of water mains
Leung et al. Theoretical and practical data science and analytics: challenges and solutions
Zhu et al. A hybrid model for nonlinear regression with missing data using quasilinear kernel
JP7544274B2 (en) Accumulation calculation device, accumulation calculation method, and program
Eisenhauer The approximate solution of finite‐horizon discrete‐choice dynamic programming models
Humphries et al. Machine Learning and ‘The Cloud’for Natural Resource Applications: Autonomous Online Robots Driving Sustainable Conservation Management Worldwide?
JP6142878B2 (en) Information system performance evaluation apparatus, method and program
Kokko et al. PYLFIRE: Python implementation of likelihood-free inference by ratio estimation

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20951485

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20951485

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: JP