CN111191795A - Method, device and system for training machine learning model - Google Patents
Method, device and system for training machine learning model Download PDFInfo
- Publication number
- CN111191795A CN111191795A CN201911412450.0A CN201911412450A CN111191795A CN 111191795 A CN111191795 A CN 111191795A CN 201911412450 A CN201911412450 A CN 201911412450A CN 111191795 A CN111191795 A CN 111191795A
- Authority
- CN
- China
- Prior art keywords
- setting
- parameter
- machine learning
- learning model
- parameters
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Medical Informatics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Artificial Intelligence (AREA)
- Feedback Control In General (AREA)
Abstract
The invention provides a method, a device and a system for training a machine learning model, wherein the method comprises the following steps: acquiring preset setting parameters of a machine learning model, parameter values of the setting parameters and setting indexes for evaluating the effect of the machine learning model; acquiring an index value of a set index obtained by training a machine learning model according to the parameter value of the set parameter; recording parameter values of set parameters and index values of the set indexes of the machine learning model in at least one training process, and generating a chart reflecting the corresponding relation between the parameter values of the set parameters and the index values of the set indexes.
Description
Technical Field
The present invention relates to the field of machine learning technologies, and more particularly, to a method of training a machine learning model, an apparatus for training a machine learning model, a system including at least one computing apparatus and at least one storage apparatus, and a readable storage medium.
Background
Machine learning is the core of artificial intelligence, is the fundamental approach for computers to have intelligence, and is applied to all fields of artificial intelligence.
In the training process of the traditional machine learning model, the machine learning model is usually trained for multiple times by adjusting the parameter value of at least one parameter of the machine learning model, and information such as the parameter values of all parameters, the index values of all indexes, the training samples and the like in each training process is recorded in a corresponding training log.
If the user needs to adjust the parameter values according to the historical parameter values and the corresponding index values in the training process, the training logs obtained by training the machine learning model each time need to be searched, and the required parameter values and the required index values are screened out from a large amount of information recorded in the training logs, so that the searching process is very complicated and time-consuming.
Disclosure of Invention
An object of the present invention is to provide a new technical solution for training a machine learning model.
According to a first aspect of the invention, there is provided a method of training a machine learning model, comprising:
acquiring preset setting parameters of a machine learning model, parameter values of the setting parameters and setting indexes for evaluating the effect of the machine learning model;
acquiring an index value of the set index obtained by training the machine learning model according to the parameter value of the set parameter;
recording the parameter values of the set parameters and the index values of the set indexes in at least one training process of the machine learning model, and generating a chart reflecting the corresponding relation between the parameter values of the set parameters and the index values of the set indexes.
Optionally, the obtaining preset parameters of the machine learning model, the parameter values of the preset parameters, and the setting indexes for evaluating the effect of the machine learning model includes:
providing a first interface for acquiring the setting parameters, the parameter values of the setting parameters and the setting indexes;
and acquiring the setting parameters, the parameter values of the setting parameters and the setting indexes through the first interface.
Optionally, the obtaining the setting parameter, the parameter value of the setting parameter, and the setting index through the first interface includes:
acquiring an algorithm code of the machine learning model through the first interface;
identifying all parameters related in the algorithm codes or preset reference parameters as the set parameters; identifying all indexes related in the algorithm codes or preset reference indexes as the set indexes;
and acquiring the parameter values of the set parameters in the algorithm codes.
Optionally, the providing a first interface for obtaining the setting parameter, the parameter value of the setting parameter, and the setting index includes:
and providing a graphical interface of a data processing flow chart of the machine learning model, wherein the graphical interface comprises the first interface.
Optionally, the graphical interface further includes a second interface for acquiring a logic code, where the logic code is configured to configure a manner of acquiring the set index according to the parameter value of the set parameter;
the method further comprises the following steps:
acquiring the logic code through the second interface;
and obtaining the index value of the set index according to the logic code and the parameter value of the set parameter.
Optionally, the providing a first interface for obtaining the setting parameter, the parameter value of the setting parameter, and the setting index includes:
the acquiring the setting parameter, the parameter value of the setting parameter, and the setting index through the first interface includes:
acquiring a specified command line input through the first interface; the specified command line is used for calling a recording service, and the specified command line comprises the setting parameters, the parameter values of the setting parameters and the setting indexes;
and acquiring the setting parameters, the parameter values of the setting parameters and the setting indexes according to the specified command line.
Optionally, the obtaining of the index value of the set index obtained by training the machine learning model according to the parameter value of the set parameter includes:
responding to a request for reading the parameter values of the set parameters in the training process of the machine learning model, providing the parameter values of the set parameters, and performing training of the machine learning model;
and obtaining the index value of the set index obtained by training the machine learning model according to the parameter value of the set parameter.
Optionally, the method further includes:
recording time information for each training of the machine learning model.
Optionally, the recording the parameter value of the setting parameter and the index value of the setting index in at least one training process of the machine learning model includes:
and converting the setting parameters, the current parameter values of the setting parameters, the setting indexes, the current index values of the setting indexes and the time information acquired in each training process of the machine learning model into json files according to a preset format, and storing the json files in a database.
Optionally, the number of the setting parameters is multiple, the number of the setting indexes is multiple,
the method further comprises the following steps:
providing a third interface for selecting setting parameters and a fourth interface for selecting setting indexes;
acquiring target setting parameters selected by a user through the third interface and target setting indexes selected through the fourth interface;
the generating of the map reflecting the correspondence between the parameter value of the setting parameter and the index value of the setting index includes:
and generating a relation graph reflecting the corresponding relation between the parameter value of the target setting parameter and the index value of the target setting index.
Optionally, the method further includes:
responding to the operation of adjusting the parameter value of the target setting parameter by a user, and acquiring the adjusted parameter value;
obtaining a predicted value of the target setting index obtained according to the adjusted parameter value;
and displaying the predicted value of the target setting index corresponding to the parameter value after the target setting parameter is adjusted in the relation graph.
Optionally, the method further includes:
responding to the operation of training the machine learning model according to the adjusted parameter value, and acquiring an actual value of the target setting index obtained by retraining the machine learning model according to the parameter value adjusted by the target setting parameter;
and displaying the actual value of the target setting index corresponding to the parameter value after the target setting parameter is adjusted in a relation graph.
Optionally, the number of the target setting parameters is one, and the relationship graph is a scatter diagram; or, the target setting parameters are two, and the relation graph is a thermodynamic diagram.
According to a second aspect of the present invention, there is provided an apparatus for training a machine learning model, comprising:
the first acquisition module is used for acquiring preset setting parameters of a machine learning model, parameter values of the setting parameters and setting indexes for evaluating the effect of the machine learning model;
the second acquisition module is used for acquiring the index value of the set index obtained by training the machine learning model according to the parameter value of the set parameter;
and the chart generation module is used for recording the parameter values of the set parameters and the index values of the set indexes in at least one training process of the machine learning model and generating a chart reflecting the corresponding relation between the parameter values of the set parameters and the index values of the set indexes.
Optionally, the first obtaining module is specifically configured to:
providing a first interface for acquiring the setting parameters, the parameter values of the setting parameters and the setting indexes;
and acquiring the setting parameters, the parameter values of the setting parameters and the setting indexes through the first interface.
Optionally, the obtaining the setting parameter, the parameter value of the setting parameter, and the setting index through the first interface includes:
acquiring an algorithm code of the machine learning model through the first interface;
identifying all parameters related in the algorithm codes or preset reference parameters as the set parameters; identifying all indexes related in the algorithm codes or preset reference indexes as the set indexes;
and acquiring the parameter values of the set parameters in the algorithm codes.
Optionally, the providing a first interface for obtaining the setting parameter, the parameter value of the setting parameter, and the setting index includes:
and providing a graphical interface of a data processing flow chart of the machine learning model, wherein the graphical interface comprises the first interface.
Optionally, the graphical interface further includes a second interface for acquiring a logic code, where the logic code is configured to configure a manner of acquiring the set index according to the parameter value of the set parameter;
the device further comprises:
means for obtaining the logic code through the second interface;
and the module is used for obtaining the index value of the set index according to the logic code and the parameter value of the set parameter.
Optionally, the providing a first interface for obtaining the setting parameter, the parameter value of the setting parameter, and the setting index includes:
the acquiring the setting parameter, the parameter value of the setting parameter, and the setting index through the first interface includes:
acquiring a specified command line input through the first interface; the specified command line is used for calling a recording service, and the specified command line comprises the setting parameters, the parameter values of the setting parameters and the setting indexes;
and acquiring the setting parameters, the parameter values of the setting parameters and the setting indexes according to the specified command line.
Optionally, the second obtaining module is configured to:
responding to a request for reading the parameter values of the set parameters in the training process of the machine learning model, providing the parameter values of the set parameters, and performing training of the machine learning model;
and obtaining the index value of the set index obtained by training the machine learning model according to the parameter value of the set parameter.
Optionally, the method further includes:
means for recording time information for each training of the machine learning model.
Optionally, the recording the parameter value of the setting parameter and the index value of the setting index in at least one training process of the machine learning model includes:
and converting the setting parameters, the current parameter values of the setting parameters, the setting indexes, the current index values of the setting indexes and the time information acquired in each training process of the machine learning model into json files according to a preset format, and storing the json files in a database.
Optionally, the number of the setting parameters is multiple, the number of the setting indexes is multiple,
the device further comprises:
a module for providing a third interface for selecting setting parameters and a fourth interface for selecting setting indexes;
a module for obtaining a target setting parameter selected by a user through the third interface and a target setting index selected through the fourth interface;
the generating of the map reflecting the correspondence between the parameter value of the setting parameter and the index value of the setting index includes:
and generating a relation graph reflecting the corresponding relation between the parameter value of the target setting parameter and the index value of the target setting index.
Optionally, the method further includes:
a module for acquiring an adjusted parameter value in response to an operation of adjusting the parameter value of the target setting parameter by a user;
a module for obtaining a predicted value of the target setting index obtained according to the adjusted parameter value;
and the module is used for displaying the predicted value of the target setting index corresponding to the parameter value after the target setting parameter is adjusted in the relation graph.
Optionally, the method further includes:
a module for obtaining an actual value of the target setting index obtained by retraining the machine learning model according to the parameter value adjusted by the target setting parameter in response to an operation of training the machine learning model according to the adjusted parameter value;
and the module is used for displaying the actual value of the target setting index corresponding to the parameter value after the target setting parameter is adjusted in a relation graph.
Optionally, the number of the target setting parameters is one, and the relationship graph is a scatter diagram; or, the target setting parameters are two, and the relation graph is a thermodynamic diagram.
According to a third aspect of the present invention there is provided a system comprising at least one computing device and at least one storage device, wherein the at least one storage device is arranged to store instructions for controlling the at least one computing device to perform the method according to the first aspect of the present invention.
According to a fourth aspect of the present invention, there is provided a computer readable storage medium having stored thereon a computer program which, when executed by a processor, performs the method according to the first aspect of the present invention.
The method has the advantages that the corresponding relation between the parameter value of the set parameter and the index value of the set index in each training process is displayed through the visual chart, so that a user can conveniently check the training result of the machine learning model, and the parameter value of the set parameter can be adjusted according to the training result.
Other features of the present invention and advantages thereof will become apparent from the following detailed description of exemplary embodiments thereof, which proceeds with reference to the accompanying drawings.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description, serve to explain the principles of the invention.
Fig. 1 is a block diagram of one example of a hardware configuration of an electronic device that can be used to implement an embodiment of the present invention.
FIG. 2 is a flow diagram of a method of training a machine learning model according to an embodiment of the invention;
FIGS. 3 and 4 are codes associated with a logistic regression algorithm according to an embodiment of the present invention;
FIG. 5 is a schematic view of a graphical interface according to an embodiment of the invention;
FIG. 6 is a schematic diagram of a second interface according to an embodiment of the invention;
FIG. 7 is a table diagram according to an embodiment of the invention;
FIG. 8 is a schematic diagram of a scatter plot according to an embodiment of the invention;
FIG. 9 is a schematic illustration of a thermodynamic diagram according to an embodiment of the invention;
FIG. 10 is a block schematic diagram of an apparatus for training a machine learning model according to an embodiment of the present invention;
fig. 11 is a block schematic diagram of a system according to an embodiment of the invention.
Detailed Description
Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that: the relative arrangement of the components and steps, the numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present invention unless specifically stated otherwise.
The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.
Techniques, methods, and apparatus known to those of ordinary skill in the relevant art may not be discussed in detail but are intended to be part of the specification where appropriate.
In all examples shown and discussed herein, any particular value should be construed as merely illustrative, and not limiting. Thus, other examples of the exemplary embodiments may have different values.
It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, further discussion thereof is not required in subsequent figures.
Various embodiments and examples according to embodiments of the present invention are described below with reference to the accompanying drawings.
< hardware configuration >
Fig. 1 is a block diagram showing a hardware configuration of an electronic apparatus 1000 that can implement an embodiment of the present invention.
The electronic device 1000 may be a laptop, desktop, cell phone, tablet, etc. As shown in fig. 1, the electronic device 1000 may include a processor 1100, a memory 1200, an interface device 1300, a communication device 1400, a display device 1500, an input device 1600, a speaker 1700, a microphone 1800, and the like. The processor 1100 may be a central processing unit CPU, a microprocessor MCU, or the like. The memory 1200 includes, for example, a ROM (read only memory), a RAM (random access memory), a nonvolatile memory such as a hard disk, and the like. The interface device 1300 includes, for example, a USB interface, a headphone interface, and the like. The communication device 1400 is capable of wired or wireless communication, for example, and may specifically include Wifi communication, bluetooth communication, 2G/3G/4G/5G communication, and the like. The display device 1500 is, for example, a liquid crystal display panel, a touch panel, or the like. The input device 1600 may include, for example, a touch screen, a keyboard, a somatosensory input, and the like. A user can input/output voice information through the speaker 1700 and the microphone 1800.
The electronic device shown in fig. 1 is merely illustrative and is in no way meant to limit the invention, its application, or uses. In an embodiment of the present invention, the memory 1200 of the electronic device 1000 is used for storing instructions for controlling the processor 1100 to operate so as to execute any one of the methods for training a machine learning model provided by the embodiment of the present invention. It will be appreciated by those skilled in the art that although a plurality of means are shown for the electronic device 1000 in fig. 1, the present invention may relate to only some of the means therein, e.g. the electronic device 1000 relates to only the processor 1100 and the storage means 1200. The skilled person can design the instructions according to the disclosed solution. How the instructions control the operation of the processor is well known in the art and will not be described in detail herein.
< method examples >
In the present embodiment, a method of training a machine learning model is provided. The method of training a machine learning model may be implemented by an electronic device. The electronic device may be the electronic device 1000 as shown in fig. 1.
According to fig. 2, the method for training a machine learning model of the present embodiment may include the following steps S2100 to S2300:
step S2100 obtains preset setting parameters of the machine learning model, parameter values of the setting parameters, and setting indexes for evaluating the effect of the machine learning model.
The setting parameter may be a parameter for training the machine learning model, and the setting parameter may be, for example, a hyper-parameter, the number of training rounds, a step size, or the like of the machine learning model.
The setting index may be an index for evaluating the effect of the machine learning model. Specifically, the setting index may be an Area Under the receiver operating characteristic Curve (ROC Curve) and a coordinate Axis (AUC), accuracy, recall rate, or the like.
In an embodiment of the present invention, the step of obtaining the preset setting parameters of the machine learning model, the parameter values of the setting parameters, and the setting index for evaluating the effect of the machine learning model may include steps S2110 to S2120 as follows:
step S2110, providing a first interface for acquiring the setting parameter, the parameter value of the setting parameter, and the setting index.
Step S2120, obtaining the setting parameter, the parameter value of the setting parameter, and the setting index through the first interface.
In the first embodiment of the present invention, the obtaining of the setting parameter, the parameter value of the setting parameter, and the setting index through the first interface may include:
acquiring a designated command line input through a first interface; the specified command line is used for calling recording service, and comprises setting parameters, parameter values of the setting parameters and setting indexes;
and acquiring a setting parameter, a parameter value of the setting parameter and a setting index according to the specified command line.
Correspondingly, in the present embodiment, the first interface may be a window for inputting a specified command line, and the window may be provided in response to a specified operation by the user. For example, the first interface may be a shell under the linux operating system. Shell is the user interface of the system, and provides an interface for the user to interact with the kernel.
In the present embodiment, a designation command line for calling a service of recording setting parameters, setting parameter values, and a designation command line for calling index values of recording setting indexes and setting indexes may be input through the first interface, respectively.
Taking Python code as an example, for example, there is a lib named sage, and the service for calling the record setting parameter and the parameter value of the setting parameter is log _ parameter, then the specified command line for calling the service for recording the record setting parameter and the parameter value of the setting parameter may be:
sage.log_parameter(key＝“metrics_1”,value＝metrics_1)
wherein the parameters of the service log _ parameter include a name (key) specifying the parameter and a source (value) of a parameter value specifying the parameter.
Taking Python code as an example, for example, there is a lid named sage, and the service for calling record setting index and setting index value of the index is log _ metric, then the specified command line for calling the service for calling record setting index and setting index value of the index may be:
sage.log_metric(key＝“AUC”,value＝auc)
wherein the parameters of the service log _ metric include a name (key) of the specified metric and a source (value) of the metric value of the specified metric.
In the second embodiment of the present invention, the acquiring the setting parameter, the parameter value of the setting parameter, and the setting index through the first interface may include steps S2121 to S2123 as follows:
step S2121, obtaining an algorithm code of the machine learning model through the first interface.
Correspondingly, the first interface may be a region of the interface into which a code file may be dragged or algorithm code may be entered, and the user may place a file of algorithm code of the machine learning model into the first interface or enter algorithm code of the machine learning model into the first interface.
Step S2122, identifying all parameters related in the algorithm code or preset reference parameters as set parameters; and identifying all indexes related in the algorithm codes or preset reference indexes as set indexes.
In an embodiment of the present invention, the preset reference parameter and the preset reference index may be preset according to an application scenario or specific requirements, and may not be set only for the algorithm code of the machine learning model obtained through the first interface. Therefore, the reference parameter may include parameters not related to the algorithm code, and the reference index may also include indexes not related to the algorithm code.
In an embodiment of the present invention, for some commonly used machine learning packages such as sklern, xgboost, keras, etc., in API interfaces of these machine learning packages, recording modes of setting parameters and setting indexes may be fixed.
Fig. 3 and 4 are examples of sklern logistic regression algorithms. In the example shown in fig. 3, part of the box is the relevant code defining the parameter values of the set parameters. In the example shown in fig. 4, the part in the box is the correlation code for calculating the accure _ score setting index.
For the algorithm codes of the tool kit, the setting parameters and the setting indexes can be directly identified from the algorithm codes.
For example, a method of auto _ log _ parameter and auto _ log _ metric may be added, and all parameters and metrics may be automatically recorded according to the related codes for setting parameters and setting metrics in the algorithm codes of this type of toolkit.
Step S2123, obtaining parameter values of the set parameters in the algorithm codes.
In this embodiment, the parameter values of the setting parameters may be defined in the algorithm code in advance. Therefore, the parameter values of the setting parameters can be acquired from the algorithm code.
In a third embodiment of the present invention, providing a first interface for acquiring a setting parameter, a parameter value of the setting parameter, and a setting index may include:
and providing a graphical interface of a data processing flow chart of the machine learning model, wherein the graphical interface comprises a first interface.
The data processing flow diagram may be used to represent the flow of data processing performed by the various nodes in the diagram. The node may be an operator or a model for performing data processing, may be data for processing, and may be a sample for training the operator or the model.
In one embodiment of the invention, the data processing flow diagram is a directed acyclic graph. A directed acyclic graph, called DAG graph for short, refers to a directed graph without loops.
The window in the graphical interface including the first interface may be, as shown in fig. 5, a user may input the setting parameters of the machine learning model by clicking a button for adding the setting parameters. And responding to the operation of clicking the input box of the set parameters by the user, and providing all the parameters related to the machine learning model for the user to select the set parameters.
Accordingly, the user can input the setting index for evaluating the machine learning model by clicking the button for adding the setting index. And providing all indexes related to the machine learning model in response to the operation of clicking the input box of the set index by the user so that the user can select the set index.
In one embodiment of the present invention, the setting parameter, the parameter value of the setting parameter, and the setting index may be directly obtained from the first interface.
In an embodiment of the present invention, the setting parameter and the setting index may be obtained through the first interface, and then the parameter value of the setting parameter in the algorithm code is obtained.
Step S2200 is to obtain an index value of the set index obtained by training the machine learning model according to the parameter value of the set parameter.
In another embodiment of the present invention, the graphical interface may further include a second interface for acquiring a logic code, where the logic code is configured to configure a manner of acquiring the setting index according to the parameter value of the setting parameter; then, the manner of obtaining the index value of the specified index may further include:
acquiring a logic code through a second interface;
and obtaining an index value of the set index according to the logic code and the parameter value of the set parameter.
As partially shown in the block of fig. 6, the second interface may be a logic code input window, and a user may input a logic code for configuring a manner of obtaining the setting index according to the parameter value of the setting parameter through the second interface.
Further, the index value of the set index may be obtained from the logic code and a machine learning model trained from the parameter value of the set parameter.
In the first and third embodiments, the parameter values of the setting parameters may not be preset in the algorithm code, but may be set by the user through a command line or a graphical interface. Then, the manner of obtaining the index value of the setting index may include steps S2210 to S2220 shown below:
step S2210, in response to a request for reading a parameter value of the setting parameter in the training process of the machine learning model, providing the parameter value of the setting parameter, and performing training of the machine learning model.
In an embodiment of the present invention, a user may set a parameter value of a setting parameter in a preset recording function module, and correspondingly, the method of the embodiment of the present invention may be executed by the recording function module.
Specifically, the recording function module may provide the parameter values of the setting parameters to the machine learning model in response to a request that the machine learning model reads the parameter values of the setting parameters in the training process, so as to train the machine learning model.
Step S2220, obtaining the index value of the set index obtained by training the machine learning model according to the parameter value of the set parameter.
Specifically, the recording function module may acquire the index value of the setting index after the machine learning model performs training according to the parameter value of the setting parameter to obtain the index value of the setting index.
Step S2300, recording parameter values of the set parameters and index values of the set indexes in at least one training process of the machine learning model, and generating a chart reflecting the corresponding relation between the parameter values of the set parameters and the index values of the set indexes.
In the embodiment of the invention, the corresponding relation between the parameter value of the set parameter and the index value of the set index in each training process is displayed through an intuitive chart, so that a user can conveniently check the training result of the machine learning model, and further, the parameter value of the set parameter can be adjusted according to the training result.
In one embodiment of the present invention, the method may further comprise:
and recording the time information of the machine learning model trained each time, and displaying the time information of the machine learning model trained each time in the chart. For example as shown in fig. 7.
In one embodiment of the present invention, the manner of recording the parameter values of the set parameters and the index values of the set indexes in at least one training process of the machine learning model may include:
and converting the set parameters, the current parameter values of the set parameters, the set indexes, the current index values of the set indexes and the time information of the machine learning model trained this time into json files according to a preset format, and storing the json files in a database.
Thus, a graph reflecting the correspondence between the parameter value of the setting parameter and the index value of the setting index can be generated from the json file obtained by training the machine learning model each time.
In one embodiment of the present invention, the chart may be a table, and the table may be as shown in fig. 7, and represents parameter values of the setting parameters and index values of the setting indexes in each training process of the machine learning model. In the table shown in fig. 7, it may be that the parameter values and index values of each row correspond to the same training process.
In one embodiment of the present invention, in the case where the number of setting indexes is 1 and the number of setting parameters is 1 or 2, the graph may be a relational graph reflecting the correspondence between the parameter values of the setting parameters and the index values of the setting indexes.
Specifically, when the number of setting indexes is 1 and the number of setting parameters is 1, the relationship map may be a scatter map, for example, as shown in fig. 8. When the number of setting indexes is 1 and the number of setting parameters is 2, the relational map may be a thermodynamic map, for example, as shown in fig. 9.
In an embodiment of the present invention, if a plurality of parameters are set and a plurality of indexes are set, the method may further include:
providing a third interface for selecting setting parameters and a fourth interface for selecting setting indexes; and acquiring target setting parameters selected by a user through a third interface and target setting indexes selected through a fourth interface.
On the basis, generating a graph reflecting the correspondence between the parameter value of the setting parameter and the index value of the setting index may include:
and generating a relational graph reflecting the corresponding relation between the parameter value of the target setting parameter and the index value of the target setting index.
In one embodiment of the present invention, in response to the user clicking the third interface, a list of setting parameters is provided, which may include all the setting parameters, for the user to select at most two target setting parameters from the list. Correspondingly, in response to the operation of clicking the fourth interface by the user, a list of setting indexes is provided, and all the setting indexes can be included in the list, so that the user can select at most one target setting index from the list.
In the case where the target setting parameter is one, the relationship diagram may be a scatter diagram; in the case where the target setting parameter is two, the relationship diagram is a thermodynamic diagram.
In another embodiment of the present invention, the third interface may be an input box for inputting a target setting parameter, and the fourth interface may be an input box for inputting a target setting index. For example as shown in fig. 8 and 9.
In the example shown in fig. 8, the third interface may be an input box selecting an abscissa, and the fourth interface may be an input box selecting an ordinate. And generating a scatter diagram reflecting the corresponding relation between the parameter value of the target setting parameter and the index value of the target setting index in response to the operation of clicking a button for generating the scatter diagram by the user.
In the example shown in fig. 9, the third interface may be an input box selecting the abscissa and an input box selecting the ordinate, and the fourth interface may be an input box selecting the thermal item. And generating a thermodynamic diagram reflecting the corresponding relation between the parameter value of the target setting parameter and the index value of the target setting index in response to the operation of clicking a button for generating the thermodynamic diagram by the user.
In one embodiment of the present invention, the method may further comprise:
responding to the operation of adjusting the parameter value of the target setting parameter by a user, and acquiring the adjusted parameter value; obtaining a predicted value of a set index obtained according to the adjusted parameter value; and displaying the predicted value of the setting index corresponding to the parameter value after the target setting parameter is adjusted in the relation graph.
Specifically, the user may re-input the parameter value of the target setting parameter through the first interface, so as to achieve the effect of adjusting the parameter value of the target setting parameter.
In an embodiment of the present invention, the button may be a button for providing a performance of a prediction index in a graphical interface, as shown in fig. 8, a user clicks the button for performing the performance of the prediction index to trigger obtaining a predicted value of a setting index obtained according to an adjusted parameter value; and displaying the operation of the predicted value of the setting index corresponding to the parameter value after the target setting parameter is adjusted in the relation graph.
The predicted value of the target setting index is not obtained by training the machine learning model based on the parameter value adjusted by the target setting parameter, and may be the same as or different from the actual index value of the target setting index obtained by training the machine learning model based on the parameter value adjusted by the target setting parameter.
In an embodiment of the present invention, the predicted value of the target setting index corresponding to the parameter value after the target setting parameter adjustment may be obtained according to the historical relative relationship between the target setting parameter and the target setting index.
In one embodiment of the present invention, the method may further comprise:
responding to the operation of training the machine learning model according to the adjusted parameter value, and acquiring an actual value of a target setting index obtained by retraining the machine learning model according to the parameter value adjusted by the target setting parameter; and displaying the actual value of the target setting index corresponding to the parameter value after the target setting parameter is adjusted in the relation graph.
In an embodiment of the present invention, a button executed by using a new parameter may be provided in the graphical interface, as shown in fig. 8, the user triggers an operation of training the machine learning model according to the adjusted parameter value by clicking the button expressed by the prediction index, and displays an actual value of the target setting index corresponding to the adjusted parameter value in the relationship graph, or replaces a predicted value of the target setting index corresponding to the adjusted parameter value in the relationship graph with the actual index value.
In an embodiment of the present invention, in the relationship diagram, the predicted value of the target setting index corresponding to the parameter value after the target setting parameter is adjusted may be different from the actual value of the target setting index obtained by the training model, so as to distinguish the predicted value from the actual value of the target setting index.
< apparatus embodiment >
In the present embodiment, an apparatus 5000 for training a machine learning model is provided, as shown in fig. 10, including a first obtaining module 5100, a second obtaining module 5200 and a chart generating module 5300. The first obtaining module 5100 is configured to obtain preset setting parameters of the machine learning model, parameter values of the setting parameters, and setting indexes for evaluating an effect of the machine learning model; the second obtaining module 5200 is configured to obtain an index value of a set index obtained by training a machine learning model according to a parameter value of the set parameter; the graph generation module 5300 is configured to record parameter values of the set parameters and index values of the set indexes in at least one training process of the machine learning model, and generate a graph reflecting a correspondence between the parameter values of the set parameters and the index values of the set indexes.
In an embodiment of the present invention, the first obtaining module 5100 may specifically be configured to:
providing a first interface for acquiring a set parameter, a parameter value of the set parameter and a set index;
and acquiring the setting parameters, the parameter values of the setting parameters and the setting indexes through the first interface.
In one embodiment of the present invention, the obtaining of the setting parameter, the parameter value of the setting parameter, and the setting index through the first interface includes:
acquiring an algorithm code of a machine learning model through a first interface;
identifying all parameters related in the algorithm codes or preset reference parameters as set parameters; identifying all indexes related in the algorithm codes or preset reference indexes as set indexes;
and acquiring parameter values of the set parameters in the algorithm codes.
In one embodiment of the present invention, providing a first interface for acquiring a setting parameter, a parameter value of the setting parameter, and a setting index includes:
and providing a graphical interface of a data processing flow chart of the machine learning model, wherein the graphical interface comprises a first interface.
In an embodiment of the present invention, the graphical interface further includes a second interface for acquiring a logic code, where the logic code is configured to configure a manner for acquiring the setting index according to the parameter value of the setting parameter;
the apparatus 5000 for training a machine learning model may further include:
a module for obtaining the logic code through the second interface;
and the module is used for obtaining an index value of the set index according to the logic code and the parameter value of the set parameter.
In one embodiment of the present invention, providing a first interface for acquiring a setting parameter, a parameter value of the setting parameter, and a setting index includes:
the step of obtaining the setting parameters, the parameter values of the setting parameters and the setting indexes through the first interface comprises the following steps:
acquiring a designated command line input through a first interface; the specified command line is used for calling recording service, and comprises setting parameters, parameter values of the setting parameters and setting indexes;
and acquiring a setting parameter, a parameter value of the setting parameter and a setting index according to the specified command line.
In one embodiment of the present invention, the second obtaining module 5200 may further be configured to:
responding to a request for reading parameter values of set parameters in the training process of the machine learning model, providing the parameter values of the set parameters, and training the machine learning model;
and obtaining the index value of the set index obtained by training the machine learning model according to the parameter value of the set parameter.
In an embodiment of the present invention, the apparatus 5000 for training a machine learning model may further include:
and a module for recording time information for each training of the machine learning model.
In one embodiment of the present invention, recording the parameter values of the setting parameters and the index values of the setting indexes of the machine learning model in at least one training process comprises:
and converting the setting parameters, the current parameter values of the setting parameters, the setting indexes, the current index values of the setting indexes and the time information acquired in each training process of the machine learning model into json files according to a preset format, and storing the json files in a database.
In an embodiment of the present invention, the setting parameter is a plurality of parameters, and the setting index is a plurality of indexes, and the apparatus 5000 for training a machine learning model may further include:
a module for providing a third interface for selecting setting parameters and a fourth interface for selecting setting indexes;
a module for obtaining a target setting parameter selected by a user through a third interface and a target setting index selected through a fourth interface;
generating a map reflecting a correspondence between a parameter value of the setting parameter and an index value of the setting index includes:
and generating a relational graph reflecting the corresponding relation between the parameter value of the target setting parameter and the index value of the target setting index.
In an embodiment of the present invention, the apparatus 5000 for training a machine learning model may further include:
a module for acquiring an adjusted parameter value in response to an operation of a user to adjust a parameter value of a target setting parameter;
a module for obtaining a predicted value of the target setting index obtained according to the adjusted parameter value;
and the module is used for displaying the predicted value of the target setting index corresponding to the parameter value after the target setting parameter is adjusted in the relation graph.
In an embodiment of the present invention, the apparatus 5000 for training a machine learning model may further include:
a module for obtaining an actual value of a target setting index obtained by retraining the machine learning model according to the parameter value adjusted according to the target setting parameter in response to an operation of training the machine learning model according to the adjusted parameter value;
and the module is used for displaying the actual value of the target setting index corresponding to the parameter value after the target setting parameter is adjusted in the relation graph.
In one embodiment of the invention, the target setting parameter is one, and the relation graph is a scatter diagram; alternatively, the target setting parameters are two, and the relational graph is a thermodynamic diagram.
Those skilled in the art will appreciate that the means 5000 for training the machine learning model may be implemented in a variety of ways. The apparatus 5000 for training a machine learning model may be implemented, for example, by instructing a configuration processor. For example, the apparatus 5000 for training a machine learning model may be implemented by storing instructions in ROM and reading the instructions from ROM into a programmable device when the device is started. For example, the means 5000 for training the machine learning model may be incorporated into a dedicated device (e.g., an ASIC). The means 5000 for training the machine learning model may be divided into separate units or they may be implemented by being combined together. The means 5000 for training the machine learning model may be implemented by one of the various implementations described above, or may be implemented by a combination of two or more of the various implementations described above.
In this embodiment, the apparatus 5000 for training the machine learning model may have various implementation forms, for example, the apparatus 5000 for training the machine learning model may be any functional module running in a software product or an application program providing the model training service, or a peripheral insert, a plug-in, a patch, etc. of the software product or the application program, and may also be the software product or the application program itself.
< System embodiment >
In this embodiment, as shown in fig. 11, a system 6000 of at least one computing device 6100 and at least one storage device 6200 is also provided. The at least one storage device 6200 is to store executable instructions; the instructions are for controlling at least one computing device 6100 to perform a method of training a machine learning model according to any embodiment of the present invention.
In this embodiment, the system 6000 may be a device such as a mobile phone, a tablet computer, a palm computer, a desktop computer, a notebook computer, a workstation, a game machine, or a distributed system formed by a plurality of devices.
< computer-readable storage Medium >
In this embodiment, there is also provided a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements a method of training a machine learning model according to any of the embodiments of the invention.
The present invention may be an apparatus, method and/or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions embodied therewith for causing a processor to implement various aspects of the present invention.
The computer readable storage medium may be a tangible device that can hold and store the instructions for use by the instruction execution device. The computer readable storage medium may be, for example, but not limited to, an electronic memory device, a magnetic memory device, an optical memory device, an electromagnetic memory device, a semiconductor memory device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a Static Random Access Memory (SRAM), a portable compact disc read-only memory (CD-ROM), a Digital Versatile Disc (DVD), a memory stick, a floppy disk, a mechanical coding device, such as punch cards or in-groove projection structures having instructions stored thereon, and any suitable combination of the foregoing. Computer-readable storage media as used herein is not to be construed as transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., optical pulses through a fiber optic cable), or electrical signals transmitted through electrical wires.
The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to a respective computing/processing device, or to an external computer or external storage device via a network, such as the internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers. The network adapter card or network interface in each computing/processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in the respective computing/processing device.
The computer program instructions for carrying out operations of the present invention may be assembler instructions, Instruction Set Architecture (ISA) instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider). In some embodiments, aspects of the present invention are implemented by personalizing an electronic circuit, such as a programmable logic circuit, a Field Programmable Gate Array (FPGA), or a Programmable Logic Array (PLA), with state information of computer-readable program instructions, which can execute the computer-readable program instructions.
Aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer-readable program instructions.
These computer-readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer-readable program instructions may also be stored in a computer-readable storage medium that can direct a computer, programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer-readable medium storing the instructions comprises an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.
The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer, other programmable apparatus or other devices implement the functions/acts specified in the flowchart and/or block diagram block or blocks.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions. It is well known to those skilled in the art that implementation by hardware, by software, and by a combination of software and hardware are equivalent.
Having described embodiments of the present invention, the foregoing description is intended to be exemplary, not exhaustive, and not limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen in order to best explain the principles of the embodiments, the practical application, or improvements made to the technology in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein. The scope of the invention is defined by the appended claims.
Claims (10)
1. A method of training a machine learning model, comprising:
acquiring preset setting parameters of a machine learning model, parameter values of the setting parameters and setting indexes for evaluating the effect of the machine learning model;
acquiring an index value of the set index obtained by training the machine learning model according to the parameter value of the set parameter;
recording the parameter values of the set parameters and the index values of the set indexes in at least one training process of the machine learning model, and generating a chart reflecting the corresponding relation between the parameter values of the set parameters and the index values of the set indexes.
2. The method of claim 1, wherein the obtaining preset setting parameters of the machine learning model, parameter values of the setting parameters, and setting indexes for evaluating the effect of the machine learning model comprises:
providing a first interface for acquiring the setting parameters, the parameter values of the setting parameters and the setting indexes;
and acquiring the setting parameters, the parameter values of the setting parameters and the setting indexes through the first interface.
3. The method of claim 2, the obtaining the setting parameter, the parameter value of the setting parameter, and the setting indicator through the first interface comprising:
acquiring an algorithm code of the machine learning model through the first interface;
identifying all parameters related in the algorithm codes or preset reference parameters as the set parameters; identifying all indexes related in the algorithm codes or preset reference indexes as the set indexes;
and acquiring the parameter values of the set parameters in the algorithm codes.
4. The method of claim 2, the providing a first interface for obtaining the setting parameter, the parameter value of the setting parameter, and the setting indicator comprising:
and providing a graphical interface of a data processing flow chart of the machine learning model, wherein the graphical interface comprises the first interface.
5. The method of claim 4, wherein the graphical interface further comprises a second interface for obtaining logic code, wherein the logic code is configured to configure a manner of obtaining the set index according to a parameter value of the set parameter;
the method further comprises the following steps:
acquiring the logic code through the second interface;
and obtaining the index value of the set index according to the logic code and the parameter value of the set parameter.
6. The method of claim 2, the providing a first interface for obtaining the setting parameter, the parameter value of the setting parameter, and the setting indicator comprising:
the acquiring the setting parameter, the parameter value of the setting parameter, and the setting index through the first interface includes:
acquiring a specified command line input through the first interface; the specified command line is used for calling a recording service, and the specified command line comprises the setting parameters, the parameter values of the setting parameters and the setting indexes;
and acquiring the setting parameters, the parameter values of the setting parameters and the setting indexes according to the specified command line.
7. The method of claim 1, wherein the obtaining the index value of the set index obtained by the machine learning model trained according to the parameter value of the set parameter comprises:
responding to a request for reading the parameter values of the set parameters in the training process of the machine learning model, providing the parameter values of the set parameters, and performing training of the machine learning model;
and obtaining the index value of the set index obtained by training the machine learning model according to the parameter value of the set parameter.
8. An apparatus to train a machine learning model, comprising:
the first acquisition module is used for acquiring preset setting parameters of a machine learning model, parameter values of the setting parameters and setting indexes for evaluating the effect of the machine learning model;
the second acquisition module is used for acquiring the index value of the set index obtained by training the machine learning model according to the parameter value of the set parameter;
and the chart generation module is used for recording the parameter values of the set parameters and the index values of the set indexes in at least one training process of the machine learning model and generating a chart reflecting the corresponding relation between the parameter values of the set parameters and the index values of the set indexes.
9. A system comprising at least one computing device and at least one storage device, wherein the at least one storage device is to store instructions for controlling the at least one computing device to perform the method of any of claims 1 to 7.
10. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911412450.0A CN111191795B (en) | 2019-12-31 | 2019-12-31 | Method, device and system for training machine learning model |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911412450.0A CN111191795B (en) | 2019-12-31 | 2019-12-31 | Method, device and system for training machine learning model |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111191795A true CN111191795A (en) | 2020-05-22 |
| CN111191795B CN111191795B (en) | 2023-10-20 |
Family
ID=70707949
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911412450.0A Active CN111191795B (en) | 2019-12-31 | 2019-12-31 | Method, device and system for training machine learning model |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111191795B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112256537A (en) * | 2020-11-12 | 2021-01-22 | 腾讯科技（深圳）有限公司 | Model running state display method and device, computer equipment and storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6920608B1 (en) * | 1999-05-21 | 2005-07-19 | E Numerate Solutions, Inc. | Chart view for reusable data markup language |
| US20110088011A1 (en) * | 2009-10-14 | 2011-04-14 | Vermeg Sarl | Automated Enterprise Software Development |
| CN107844837A (en) * | 2017-10-31 | 2018-03-27 | 第四范式（北京）技术有限公司 | The method and system of algorithm parameter tuning are carried out for machine learning algorithm |
| US20190034767A1 (en) * | 2017-07-31 | 2019-01-31 | Splunk Inc. | Automated data preprocessing for machine learning |
| US20190102693A1 (en) * | 2017-09-29 | 2019-04-04 | Facebook, Inc. | Optimizing parameters for machine learning models |
| CN109711548A (en) * | 2018-12-26 | 2019-05-03 | 歌尔股份有限公司 | Choosing method, application method, device and the electronic equipment of hyper parameter |
-
2019
- 2019-12-31 CN CN201911412450.0A patent/CN111191795B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6920608B1 (en) * | 1999-05-21 | 2005-07-19 | E Numerate Solutions, Inc. | Chart view for reusable data markup language |
| US20110088011A1 (en) * | 2009-10-14 | 2011-04-14 | Vermeg Sarl | Automated Enterprise Software Development |
| US20190034767A1 (en) * | 2017-07-31 | 2019-01-31 | Splunk Inc. | Automated data preprocessing for machine learning |
| US20190102693A1 (en) * | 2017-09-29 | 2019-04-04 | Facebook, Inc. | Optimizing parameters for machine learning models |
| CN107844837A (en) * | 2017-10-31 | 2018-03-27 | 第四范式（北京）技术有限公司 | The method and system of algorithm parameter tuning are carried out for machine learning algorithm |
| CN109711548A (en) * | 2018-12-26 | 2019-05-03 | 歌尔股份有限公司 | Choosing method, application method, device and the electronic equipment of hyper parameter |
Non-Patent Citations (1)
| Title |
|---|
| 奚雪峰;周国栋;: "面向自然语言处理的深度学习研究", 自动化学报, no. 10, pages 1445 - 1465 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112256537A (en) * | 2020-11-12 | 2021-01-22 | 腾讯科技（深圳）有限公司 | Model running state display method and device, computer equipment and storage medium |
| CN112256537B (en) * | 2020-11-12 | 2024-03-29 | 腾讯科技（深圳）有限公司 | Model running state display method and device, computer equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111191795B (en) | 2023-10-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111190718A (en) | Method, device and system for realizing task scheduling | |
| JP2017513138A (en) | Predictive analysis for scalable business process intelligence and distributed architecture | |
| US10976904B2 (en) | Graphical chatbot interface facilitating user-chatbot interaction | |
| WO2012006352A2 (en) | Visualizing expressions for dynamic analytics | |
| CN108153848B (en) | Method and device for searching light application data and electronic device | |
| CN108459792B (en) | Flow switching method and device and computer equipment | |
| US11315039B1 (en) | Systems and methods for model management | |
| US20180129544A1 (en) | Suggesting Application Programming Interfaces Based on Feature and Context Analysis | |
| US10168990B1 (en) | Automatic replacement of a floating-point function to facilitate fixed-point program code generation | |
| JP2015230582A (en) | Program visualization device, program visualization method, and program visualization program | |
| US10699450B2 (en) | Interactive tool for causal graph construction | |
| KR20230006601A (en) | Alignment methods, training methods for alignment models, devices, electronic devices and media | |
| CN111191795B (en) | Method, device and system for training machine learning model | |
| US10606580B2 (en) | Cognitive identification of related code changes | |
| US11341197B2 (en) | Recommendation system based on adjustable virtual indicium | |
| CN111767316A (en) | Target task processing method and device and electronic equipment | |
| US20180357046A1 (en) | Locating features in a layered software application | |
| US20180300029A1 (en) | Smart Bookmarks For Viewing Content | |
| CN111427738A (en) | Display method, application monitoring module, bytecode enhancement module and display system | |
| US11093041B2 (en) | Computer system gesture-based graphical user interface control | |
| CN114548407A (en) | Hierarchical target oriented cause and effect discovery method and device and electronic equipment | |
| US10997214B2 (en) | User interaction during ground truth curation in a cognitive system | |
| US11138273B2 (en) | Onboarding services | |
| CN112749235A (en) | Method and device for analyzing classification result and electronic equipment | |
| KR20170072281A (en) | Hit-test to determine enablement of direct manipulations in response to user actions |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |