KR20220026647A - System for sharing or selling machine learning model and operating method thereof - Google Patents

Abstract

The present invention relates to an operation method of a server for sharing or selling machine learning models. According to the present invention, the operation method comprises the following steps: receiving registration information related to a machine learning model from a provider terminal; receiving the machine learning model, a test code for the machine learning model, and a test data set for the machine learning model from the provider terminal; applying the test data set to the test code to acquire a test result; approving the machine learning model on the basis of the test result; performing watermarking when the machine learning model is approved to acquire watermarking result information; and storing the registration information related to the machine learning model, the machine learning model, the test code, the test data set, and the watermarking result information in a database.

Description

SYSTEM FOR SHARING OR SELLING MACHINE LEARNING MODEL AND OPERATING METHOD THEREOF

The present disclosure relates to a system for sharing or selling a machine learning model and an operating method of the system.

Recently, the social demand for artificial intelligence deep learning has increased significantly. As technology advances rapidly, many companies, institutions, or individuals are interested in deep learning technology and are increasingly trying to apply it to their respective fields of application. Despite this explosion of social demand, there are difficulties in technology including data collection and data purification, so companies, institutions, or individuals have difficulties in applying deep learning technology. These difficulties are more problematic for small and medium-sized enterprises and institutions that cannot have their own research team.

Companies, institutions, or individuals can use deep learning technology by utilizing a variety of easy-to-use software packages developed by large global companies. However, developing an appropriate model for a specialized field requires a deep understanding of deep learning, so many difficulties may arise.

First, the acquisition, purification, and labeling of hundreds to millions of data is labor-intensive and time-consuming. It's not something a small company can handle with a handful of people. In addition, developing and training a suitable deep learning model based on secondarily refined data requires a lot of computing support and is a very time-consuming task that takes several to tens of days even with the latest computing devices. Even if you use a cloud service, which has recently been increasingly used, a large amount of cloud usage time and cost is required to search for an optimal model. This labor-intensive, high time and cost is a very big factor that prevents small and medium-sized enterprises from adopting artificial intelligence.

The operating method of a server for sharing or selling a machine learning model of the present disclosure includes the steps of receiving registration information related to a machine learning model from a supplier terminal, a machine learning model from a supplier terminal, a test code for the machine learning model, and machine learning Receiving a test data set for the model, applying the test data set to test code to obtain test results, approving the machine learning model based on the test results, watermarking when the machine learning model is approved Obtaining watermarking result information by performing, and storing registration information related to the machine learning model, machine learning model, test code, test data set, and watermarking result information in a database.

The step of obtaining the watermarking result information of the operating method of the server for sharing or selling the machine learning model of the present disclosure is the step of obtaining a watermark image and a label for the watermark image, the machine learning model is a watermark image and a water A step of additionally learning based on the label for the mark image, and the step of applying the watermark image to a machine learning model to obtain watermarking result information.

The method of operation of a server for sharing or selling a machine learning model of the present disclosure includes transmitting registration information related to the machine learning model to the consumer's terminal at the request of the consumer's terminal, and identifying information about the machine learning model selected from the consumer's terminal. receiving, transmitting the selected machine learning model to the consumer terminal, transmitting a watermark image for the selected machine learning model and watermarking result information to the consumer terminal, and a watermark for the machine learning model selected in the consumer terminal and generating watermarking confirmation information by applying the image to the selected machine learning model, and receiving information about whether the watermarking result information and the watermarking confirmation information for the selected machine learning model match from the consumer terminal.

The method of operating a server for sharing or selling a machine learning model of the present disclosure includes the steps of receiving tuning request information for a selected machine learning model from a consumer terminal, information on whether tuning is possible for the selected machine learning model is selected machine learning When it indicates that the model is tunable, requesting a data set from the consumer terminal, receiving a data set including a training data set and a test data set for tuning from the consumer terminal, based on the selected machine learning model on the training data set to generate a tuned machine learning model by re-learning, and obtaining accuracy for the tuned machine learning model by applying a test data set for tuning to the tuned machine learning model.

The operating method of a server for sharing or selling a machine learning model of the present disclosure includes the steps of performing watermarking on a tuned machine learning model to obtain tuned watermarking result information, and registration information related to the tuned machine learning model; Storing the tuned machine learning model, the tuned test code, the test data set for tuning, and the tuned watermarking result information in the database.

The method of operating a server for sharing or selling a machine learning model of the present disclosure includes receiving a plurality of analysis target data and labeling request information from a consumer terminal, and a labeling project for a plurality of analysis target data in response to the labeling request information generating, and receiving a plurality of pieces of label information respectively corresponding to a plurality of analysis target data from terminals of at least one project participant.

The method of operating a server for sharing or selling a machine learning model of the present disclosure includes the steps of receiving tuning request information for a selected machine learning model from a consumer terminal, information on whether tuning is possible for the selected machine learning model is selected machine learning When it indicates that the model is tunable, requesting a data set from a consumer terminal, receiving a data set including at least one of a training data set and a test data set for tuning from the consumer terminal, training data for the selected machine learning model Generating a tuned machine learning model by re-learning based on the set, performing watermarking on the tuned machine learning model to obtain tuned watermarking result information, and registration information related to the tuned machine learning model , and storing the tuned machine learning model, the tuned test code, the test data set for tuning, and the tuned watermarking result information in the database.

The method of operating a server for sharing or selling a machine learning model of the present disclosure includes receiving a plurality of analysis target data and labeling request information from the consumer terminal, and corresponding to the labeling request information, the plurality of analysis target data. Generating a labeling project, and receiving a plurality of pieces of label information respectively corresponding to the plurality of analysis target data from the terminals of at least one project participant.

The method of operating a server for sharing or selling a machine learning model of the present disclosure includes the steps of receiving registration information related to a machine learning model from a supplier terminal, the machine learning model from the supplier terminal, a test code for the machine learning model, and receiving a test data set for the machine learning model, applying the test data set to the test code to obtain a test result, approving the machine learning model based on the test result, consumer terminal Receiving a plurality of analysis target data and labeling request information for use in the machine learning model from, generating a labeling project for the plurality of analysis target data in response to the labeling request information, and at least one project and receiving a plurality of pieces of label information respectively corresponding to the plurality of analysis target data from the participant's terminal.

A server for sharing or selling a machine learning model of the present disclosure includes a processor and a memory, wherein the processor receives registration information related to the machine learning model from a provider terminal based on an instruction stored in the memory, and the provider terminal receive the machine learning model, the test code for the machine learning model, and the test data set for the machine learning model from Approving the machine learning model based on, when the machine learning model is approved, performing watermarking to obtain watermarking result information, registration information related to the machine learning model, the machine learning model, the test code, The test data set and the watermarking result information are stored in a database.

In addition, a program for implementing the operating method of the server for sharing or selling the machine learning model as described above may be recorded in a computer-readable recording medium.

1 is a block diagram of a provider terminal 310 or a server 330 according to an embodiment of the present disclosure.
2 is a diagram illustrating a provider terminal 310 or a server 330 according to an embodiment of the present disclosure.
3 is a diagram schematically illustrating a machine learning model sharing or sales system according to an embodiment of the present disclosure.
4 is a flowchart illustrating an operation method of a server for sharing a machine learning model according to an embodiment of the present disclosure.
5 is a flowchart illustrating an operation of a server according to an embodiment of the present disclosure.
6 is a flowchart illustrating an operation of a server according to an embodiment of the present disclosure.
7 is a diagram illustrating a screen that may be displayed in a consumer terminal according to an embodiment of the present disclosure.
8 is a diagram illustrating a screen that may be displayed in a consumer terminal according to an embodiment of the present disclosure.
9 is a flowchart illustrating an operation of a server according to an embodiment of the present disclosure.
10 is a diagram illustrating a screen that may be displayed in a consumer terminal according to an embodiment of the present disclosure.
11 is a flowchart illustrating an operation of a server according to an embodiment of the present disclosure.
12 is a diagram illustrating a screen that may be displayed in a participant terminal according to an embodiment of the present disclosure.

Advantages and features of the disclosed embodiments, and methods of achieving them, will become apparent with reference to the embodiments described below in conjunction with the accompanying drawings. However, the present disclosure is not limited to the embodiments disclosed below, but may be implemented in various different forms, and only the present embodiments allow the present disclosure to be complete, and those of ordinary skill in the art to which the present disclosure pertains. It is only provided to fully inform the person of the scope of the invention.

Terms used in this specification will be briefly described, and the disclosed embodiments will be described in detail.

Terms used in this specification have been selected as currently widely used general terms as possible while considering the functions in the present disclosure, but these may vary depending on the intention or precedent of a person skilled in the art, the emergence of new technology, and the like. In addition, in a specific case, there is a term arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the corresponding invention. Therefore, the terms used in the present disclosure should be defined based on the meaning of the term and the contents of the present disclosure, rather than the simple name of the term.

References in the singular herein include plural expressions unless the context clearly dictates the singular. Also, the plural expression includes the singular expression unless the context clearly dictates the plural.

In the entire specification, when a part "includes" a certain element, this means that other elements may be further included, rather than excluding other elements, unless otherwise stated.

Also, as used herein, the term “unit” refers to a software or hardware component, and “unit” performs certain roles. However, "part" is not meant to be limited to software or hardware. A “unit” may be configured to reside on an addressable storage medium and may be configured to refresh one or more processors. Thus, by way of example, “part” includes components such as software components, object-oriented software components, class components and task components, processes, functions, properties, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays and variables. The functionality provided within components and “parts” may be combined into a smaller number of components and “parts” or further divided into additional components and “parts”.

According to an embodiment of the present disclosure, “unit” may be implemented with a processor and a memory. The term “processor” should be interpreted broadly to include general purpose processors, central processing units (CPUs), microprocessors, digital signal processors (DSPs), controllers, microcontrollers, state machines, and the like. In some circumstances, a “processor” may refer to an application specific semiconductor (ASIC), a programmable logic device (PLD), a field programmable gate array (FPGA), or the like. The term "processor" refers to a combination of processing devices, such as, for example, a combination of a DSP and a microprocessor, a combination of a plurality of microprocessors, a combination of one or more microprocessors in combination with a DSP core, or any other such configurations. may refer to.

The term “memory” should be interpreted broadly to include any electronic component capable of storing electronic information. The term memory includes random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), programmable read-only memory (PROM), erase-programmable read-only memory (EPROM), electrical may refer to various types of processor-readable media, such as erasable PROM (EEPROM), flash memory, magnetic or optical data storage, registers, and the like. A memory is said to be in electronic communication with the processor if the processor is capable of reading information from and/or writing information to the memory. A memory integrated in the processor is in electronic communication with the processor.

Hereinafter, with reference to the accompanying drawings, embodiments will be described in detail so that those of ordinary skill in the art to which the present disclosure pertains can easily implement them. And in order to clearly describe the present disclosure in the drawings, parts not related to the description will be omitted.

1 is a block diagram of a provider terminal 310 or a server 330 according to an embodiment of the present disclosure.

Referring to FIG. 1 , a provider terminal 310 or a server 330 according to an embodiment may include at least one of a data learning unit 110 and a data recognition unit 120 . The provider terminal 310 or server 330 as described above may include a processor and a memory.

The data learning unit 110 may learn a machine learning model for performing a target task by using the training data set. The data learning unit 110 may receive analysis target data included in the training data set and label information corresponding to the analysis target data. The data to be analyzed may be data to be analyzed, and the label information may be ground truth information about the data to be analyzed. If the target task is to recognize the name of an object included in the photo from the photo, the data to be analyzed will be a plurality of photos, and the label information will be the name of an object corresponding to each of the plurality of photos.

The data learning unit 110 may acquire a machine learning model by performing machine learning on the relationship between the analysis target data and the label information. The machine learning model obtained by the data learning unit 110 may be a model for generating label information by using the analysis target data.

The data recognition unit 120 may receive and store the machine learning model of the data learning unit 110 . The data recognition unit 120 may output label information by applying a machine learning model to input data. Also, the data recognition unit 120 may use input data, label information, and a result output by the machine learning model to update the machine learning model.

At least one of the data learning unit 110 and the data recognition unit 120 may be manufactured in the form of at least one hardware chip and mounted in an electronic device. For example, at least one of the data learning unit 110 and the data recognition unit 120 may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI), or a conventional general-purpose processor (eg, CPU Alternatively, it may be manufactured as a part of an application processor) or a graphics-only processor (eg, GPU) and mounted on various electronic devices described above.

Also, the data learning unit 110 and the data recognition unit 120 may be respectively mounted on separate electronic devices. For example, one of the data learning unit 110 and the data recognition unit 120 may be included in the electronic device, and the other may be included in the server. In addition, the data learning unit 110 and the data recognition unit 120 may provide the machine learning model information built by the data learning unit 110 to the data recognition unit 120 through wire or wireless, and recognize data. Data input to the unit 120 may be provided to the data learning unit 110 as additional learning data.

Meanwhile, at least one of the data learning unit 110 and the data recognition unit 120 may be implemented as a software module. When at least one of the data learning unit 110 and the data recognition unit 120 is implemented as a software module (or a program module including instructions), the software module is a memory or computer-readable ratio It may be stored in a non-transitory computer readable media. Also, in this case, at least one software module may be provided by an operating system (OS) or may be provided by a predetermined application. Alternatively, a part of the at least one software module may be provided by an operating system (OS), and the other part may be provided by a predetermined application.

The data learning unit 110 according to an embodiment of the present disclosure includes a data acquisition unit 111 , a preprocessing unit 112 , a training data selection unit 113 , a model learning unit 114 , and a model evaluation unit 115 . may include

The data acquisition unit 111 may acquire data necessary for machine learning. Since a lot of data is required for learning, the data acquisition unit 111 may receive a learning data set including a plurality of data.

Label information may be allocated to each of the plurality of data. The label information may be information describing each of a plurality of data. The label information may be information that a target task wants to derive. The label information may be obtained from a user input, from a memory, or from a result of a machine learning model. For example, if the target task is to predict the content of an image from an image, a plurality of data used for machine learning will be an image, and label information will be the content of the image.

The preprocessor 112 may preprocess the obtained data so that the received data can be used for machine learning. The preprocessor 112 may process the acquired training data set into a preset format so that the model learning unit 114 to be described later can use it.

The learning data selection unit 113 may select data necessary for learning from among the preprocessed data. The selected data may be provided to the model learning unit 114 . The learning data selection unit 113 may select data necessary for learning from preprocessed data according to a preset criterion. In addition, the training data selection unit 113 may select data according to a preset criterion by learning by the model learning unit 114 to be described later.

The model learning unit 114 may learn a criterion regarding which label information to output based on the training data set. Also, the model learning unit 114 may perform machine learning by using the analysis target data and label information as training data. Also, the model learning unit 114 may perform machine learning by additionally using the previously acquired machine learning model. In this case, the previously acquired machine learning model may be a pre-built model. For example, the machine learning model may be a model built in advance by receiving basic learning data.

The machine learning model may be constructed in consideration of the field of application of the learning model, the purpose of learning, or the computer performance of the device. The machine learning model may be, for example, a model based on a neural network. For example, models such as Deep Neural Network (DNN), Recurrent Neural Network (RNN), Long Short-Term Memory models (LSTM), BRDNN (Bidirectional Recurrent Deep Neural Network), Convolutional Neural Networks (CNN) can be used as machine learning models. However, the present invention is not limited thereto.

According to various embodiments, the model learning unit 114 may determine, as a machine learning model to learn, a machine learning model having a high correlation between the input learning data and the basic learning data when there are a plurality of pre-built machine learning models. there is. In this case, the basic learning data may be pre-classified for each type of data, and the machine learning model may be previously built for each type of data. For example, the basic training data may be pre-classified according to various criteria such as a place where the training data is generated, a time when the training data is generated, the size of the training data, a generator of the training data, and the type of object in the training data.

Also, the model learning unit 114 may train the machine learning model using, for example, a learning algorithm including error back-propagation or gradient descent.

Also, the model learning unit 114 may learn the machine learning model through supervised learning using, for example, learning data as an input value. In addition, the model learning unit 114, for example, by self-learning the type of data required for the target task without any guidance, unsupervised learning (unsupervised learning) to find a standard for the target task Through this, a machine learning model can be obtained. Also, the model learning unit 114 may learn the machine learning model through, for example, reinforcement learning using feedback on whether a result of a target task according to learning is correct.

In addition, when the machine learning model is learned, the model learning unit 114 may store the learned machine learning model. In this case, the model learning unit 114 may store the learned machine learning model in the memory of the electronic device including the data recognition unit 120 . Alternatively, the model learning unit 114 may store the learned machine learning model in a memory of a server connected to the electronic device through a wired or wireless network.

The memory in which the learned machine learning model is stored may also store, for example, commands or data related to at least one other component of the electronic device. The memory may also store software and/or programs. A program may include, for example, a kernel, middleware, an application programming interface (API) and/or an application program (or "application"), and the like.

The model evaluation unit 115 inputs evaluation data to the machine learning model, and when a result output from the evaluation data does not satisfy a predetermined criterion, the model learning unit 114 may learn again. In this case, the evaluation data may be preset data for evaluating the machine learning model.

For example, the model evaluation unit 115, among the results of the machine learning model learned for the evaluation data, does not satisfy a predetermined criterion when the number or ratio of evaluation data for which the recognition result is not accurate exceeds a preset threshold. can be evaluated as For example, when the predetermined criterion is defined as a ratio of 2%, when the learned machine learning model outputs an erroneous recognition result for more than 20 evaluation data out of a total of 1000 evaluation data, the model evaluation unit 115 learns It can be evaluated that the machine learning model is not suitable.

On the other hand, when there are a plurality of learned machine learning models, the model evaluation unit 115 evaluates whether a predetermined criterion is satisfied with respect to each learned machine learning model, and the model that satisfies the predetermined criterion is used as the final machine learning model. can decide In this case, when there are a plurality of models satisfying the predetermined criteria, the model evaluation unit 115 may determine one or a predetermined number of models preset in the order of the highest evaluation score as the final machine learning model.

Meanwhile, at least one of the data acquisition unit 111 , the preprocessor 112 , the training data selection unit 113 , the model learning unit 114 , and the model evaluation unit 115 in the data learning unit 110 is at least one It may be manufactured in the form of a hardware chip of For example, at least one of the data acquisition unit 111 , the preprocessor 112 , the training data selection unit 113 , the model learning unit 114 , and the model evaluation unit 115 is artificial intelligence (AI). It may be manufactured in the form of a dedicated hardware chip for this purpose, or it may be manufactured as a part of an existing general-purpose processor (eg, CPU or application processor) or graphics-only processor (eg, GPU) and mounted on the various electronic devices described above.

In addition, the data acquisition unit 111 , the preprocessor 112 , the training data selection unit 113 , the model learning unit 114 , and the model evaluation unit 115 may be mounted in one electronic device, or separate Each of the electronic devices may be mounted. For example, some of the data acquisition unit 111 , the preprocessor 112 , the training data selection unit 113 , the model learning unit 114 , and the model evaluation unit 115 are included in the electronic device, and the rest are It can be included in the server.

In addition, at least one of the data acquisition unit 111 , the preprocessor 112 , the training data selection unit 113 , the model learning unit 114 , and the model evaluation unit 115 may be implemented as a software module. A program in which at least one of the data acquisition unit 111, the preprocessor 112, the learning data selection unit 113, the model learning unit 114, and the model evaluation unit 115 includes a software module (or instructions) module), the software module may be stored in a computer-readable non-transitory computer readable medium. Also, in this case, at least one software module may be provided by an operating system (OS) or may be provided by a predetermined application. Alternatively, a part of the at least one software module may be provided by an operating system (OS), and the other part may be provided by a predetermined application.

The data recognition unit 120 according to an embodiment of the present disclosure includes a data obtaining unit 121 , a preprocessing unit 122 , a recognition data selecting unit 123 , a recognition result providing unit 124 , and a model updating unit 125 . may include

The data acquisition unit 121 may receive input data. The preprocessor 122 may preprocess the acquired input data so that the acquired input data can be used by the recognition data selection unit 123 or the recognition result providing unit 124 .

The recognition data selection unit 123 may select necessary data from the pre-processed data. The selected data may be provided to the recognition result providing unit 124 . The recognition data selection unit 123 may select some or all of the preprocessed data according to a preset criterion. Also, the recognition data selection unit 123 may select data according to a preset criterion by learning by the model learning unit 114 .

The recognition result providing unit 124 may obtain result data by applying the selected data to the machine learning model. The machine learning model may be a machine learning model generated by the model learning unit 114 . The recognition result providing unit 124 may output result data.

The model updating unit 125 may update the machine learning model based on the evaluation of the recognition result provided by the recognition result providing unit 124 . For example, the model update unit 125 may provide the model learning unit 114 with the recognition result provided by the recognition result providing unit 124 so that the model learning unit 114 updates the machine learning model. there is.

Meanwhile, at least one of the data acquisition unit 121 , the preprocessor 122 , the recognition data selection unit 123 , the recognition result providing unit 124 , and the model update unit 125 in the data recognition unit 120 is at least It may be manufactured in the form of a single hardware chip and mounted in an electronic device. For example, at least one of the data acquisition unit 121 , the preprocessor 122 , the recognition data selection unit 123 , the recognition result providing unit 124 , and the model update unit 125 is artificial intelligence (AI). ), or may be manufactured as a part of an existing general-purpose processor (eg, CPU or application processor) or graphics-only processor (eg, GPU) and mounted on the various electronic devices described above.

In addition, the data acquisition unit 121 , the preprocessor 122 , the recognition data selection unit 123 , the recognition result providing unit 124 , and the model update unit 125 may be mounted in one electronic device or separately. It may be mounted on each of the electronic devices of For example, some of the data acquiring unit 121 , the preprocessing unit 122 , the recognition data selection unit 123 , the recognition result providing unit 124 , and the model updating unit 125 are included in the electronic device, and the remaining parts are included in the electronic device. may be included in the server.

In addition, at least one of the data acquisition unit 121 , the preprocessor 122 , the recognition data selection unit 123 , the recognition result providing unit 124 , and the model update unit 125 may be implemented as a software module. At least one of the data acquisition unit 121, the preprocessor 122, the recognition data selection unit 123, the recognition result providing unit 124, and the model update unit 125 includes a software module (or instructions) When implemented as a program module), the software module may be stored in a computer-readable non-transitory computer readable medium. Also, in this case, at least one software module may be provided by an operating system (OS) or may be provided by a predetermined application. Alternatively, a part of the at least one software module may be provided by an operating system (OS), and the other part may be provided by a predetermined application.

Hereinafter, a method and apparatus for the data acquisition unit 111 , the preprocessor 112 , and the training data selection unit 113 of the data learning unit 110 to receive and process the training data will be described in more detail.

2 is a diagram illustrating a provider terminal 310 or a server 330 according to an embodiment of the present disclosure.

The provider terminal 310 or the server 330 may include a processor 210 and a memory 220 . The processor 210 may execute instructions stored in the memory 220 .

As described above, the provider terminal 310 or the server 330 may include the data learning unit 110 or the data recognition unit 120 . The data learning unit 110 or the data recognition unit 120 may be implemented by the processor 210 and the memory 220 .

1 and 2 have been described with respect to the provider terminal 310 or the server 330, but is not limited thereto. The consumer terminal 320 to be described below may also include the same configuration as the provider terminal 310 or the server 330 .

3 is a diagram schematically illustrating a machine learning model sharing or sales system according to an embodiment of the present disclosure.

A system for sharing and selling a machine learning model may include a supplier terminal 310 , a consumer terminal 320 , or a server 330 . The provider terminal 310 is a terminal used by the provider, and may include a smartphone, a PC, or a tablet. The consumer terminal 320 is a terminal used by the consumer, and may include a smartphone, a PC, or a tablet. The server 330 may communicate with the provider terminal 310 or the consumer terminal 320 by wire or wireless. The server 330 may provide a service for distributing the machine learning model. The server 330 may provide a payment service to a consumer or a provider. That is, the server 330 may provide a profit to the supplier according to the sale of the machine learning model. In addition, the server 330 may charge the consumer according to the purchase of the machine learning model. In addition, the server 330 may provide various services related to machine learning model distribution.

The provider terminal 310 may transmit the machine learning model to the server 330 . The server 330 may store the received machine learning model. In addition, the server 330 may store the machine learning model in a separate cloud. The server 330 may provide a model tuning service for tuning the machine learning model received from the provider terminal 310 . By the model tuning service, the consumer can change the already created machine learning model to suit his or her taste. When there is a model tuning request, the server 330, the provider terminal 310, or the cloud may re-learn the machine learning model based on new data. In addition, the server 330 or the cloud may store the retrained machine learning model.

The server 330 may provide a data refinement service. Data refinement service refers to labeling raw data or processing the data so that it can be applied to a machine learning model. Data refinement services may use crowdsourcing. For example, according to the data revision service, the server 330 may create a project requesting labeling for the raw data provided by the consumer terminal 320 . At least one participant can participate in the project and label the raw data. The server 330 may provide points to the participants of the project as a reward. Points can be cashed out.

Hereinafter, operations of the provider terminal 310 , the consumer terminal 320 , and the server 330 will be described in more detail.

4 is a flowchart illustrating an operation method of a server for sharing a machine learning model according to an embodiment of the present disclosure.

The server 330 may perform step 410 of receiving registration information related to the machine learning model from the provider terminal 310 . The registration information related to the machine learning model is the title of the machine learning model, the description of the machine learning model, the price of the machine learning model, the framework, the purpose of the machine learning model, the field of the machine learning model, the class of the machine learning model, the machine learning model It may include at least one of a watermark image of , GitHub (code), or accuracy.

Here, the framework is a tool that provides an environment in which users can easily implement machine learning models. Frameworks can include TensorFlow or PyTorch.

The class of the machine learning model may indicate the kind that the machine learning model can classify. For example, if the machine learning model can distinguish whether an object in an image is a cat or a dog, the class could be a cat or a dog.

The watermark image of the machine learning model is a configuration to check whether the supplier's machine learning model matches the consumer's machine learning model or whether the supplier's machine learning model has been stolen by others. Explain. The watermark image may be provided by the provider terminal 310 or may be generated by the server 330 .

The accuracy of the machine learning model may be the accuracy of the machine learning model derived from the model evaluator 115 or the model updater 125 of FIG. 1 . Accuracy may be derived based on whether a predicted value (prediction label) is derived by applying a machine learning model to a test data set, and whether the predicted value is the same as a ground truth value (actual label).

The server 330 may perform step 420 of receiving the machine learning model, the test code for the machine learning model, and the test data set for the machine learning model from the provider terminal 310 . The machine learning model may consist of at least one file. In addition, the machine learning model can be executed by the test code. The test code may be implemented in at least one programming language. Also, the test data set may be data for testing the machine learning model. The test data set may include data to be analyzed. In addition, the test data set may further include actual label information corresponding to the data to be analyzed. Consumers can use the machine learning model for their own purposes based on the machine learning model and test code.

The server 330 may perform step 430 of obtaining a test result by applying the test data set to the test code. The server 330 may execute the machine learning model using the test code. In addition, the server 330 may obtain a test result by applying the analysis target data among the test data sets to the machine learning model. The test result may be predicted label information for the data to be analyzed. However, the present invention is not limited thereto, and the manager of the server 330 may apply the analysis target data included in the test data set to the machine learning model by using the test code, and may input the test result to the server 330 .

The server 330 may perform step 440 of approving the machine learning model based on the test result. The server 330 may compare the test result with actual label information of the test data. Based on the comparison, the server 330 may obtain the accuracy of the test result. When the value obtained by subtracting the accuracy of the test result from the accuracy obtained in step 410 is equal to or less than the threshold value, the server 330 may approve the machine learning model. The threshold value is a positive number and may be a predetermined value.

If the machine learning model is not approved, the server 330 may request the machine learning model or test data set again from the provider terminal 310 . The server 330 may further perform steps 430 and 440 based on the retransmitted machine learning model or test data set. In addition, if the value obtained by subtracting the accuracy of the test result from the accuracy obtained in step 410 exceeds the threshold, the server 330 replaces the accuracy of the accuracy test result obtained in step 410 with the accuracy of the test result, and the machine learning model can approve

When the machine learning model is approved, the server 330 may perform watermarking to obtain watermarking result information ( 450 ). Watermarking may be a procedure for confirming whether the machine learning model received by the server 330 or the consumer terminal 320 is genuine. In addition, watermarking may be a procedure for the server 330 to determine the theft of the machine learning model. Watermarking will be described in detail with reference to FIG. 5 .

The server 330 may perform step 460 of storing registration information related to the machine learning model, machine learning model, test code, test data set, and watermarking result information in a database. The server 330 may transmit registration information related to the machine learning model to the consumer terminal 320 based on the request of the consumer terminal 320 . The consumer can check the registration information related to the machine learning model and purchase the machine learning model. The consumer terminal 320 may transmit payment information to the server 330 , and the server 330 may transmit a machine learning model or the like to the consumer terminal 320 after confirming the payment information. Here, the payment information may include information necessary for remittance to the server 330, such as information related to a card for payment or information related to a bank account.

5 is a flowchart illustrating an operation of a server according to an embodiment of the present disclosure.

The server 330 may perform the following operation in order to perform the step 450 of obtaining the watermarking result information. The server 330 may perform an operation 510 of obtaining at least one of a watermark image and label information for the watermark image. The server 330 may receive at least one of a watermark image and label information for the watermark image from the provider terminal 310 . In addition, the server 330 may generate at least one of the watermark image and label information for the watermark image based on the information on the provider.

According to an embodiment of the present disclosure, the server 330 may determine whether there is a watermark image received from the provider terminal 310 . When there is a watermark image received from the provider terminal 310 , the server 330 may use the received watermark image. When there is no watermark image received from the provider terminal 310, the server 330 may generate a watermark image based on information on the provider. The information on the provider may include identification information of the provider and identification information of the provider terminal. Information on the provider may be generated using at least one of ID, name, gender, address, email, purification number, model name of the provider terminal, IP address of the provider terminal, or MAC address of the provider. Also, the server 330 may transmit the generated watermark image to the provider terminal 310 .

According to another embodiment of the present disclosure, the server 330 may generate a watermark image based on information about the provider, without determining whether there is a watermark image received from the provider terminal 310 . In addition, the server 330 may transmit the watermark image to the provider terminal 310 .

The label information for the watermark image may be a label corresponding to the watermark image. Label information for the watermark image may be arbitrarily determined. Label information for the watermark image may be used as ground truth label information.

The server 330 or the provider terminal 310 may perform an operation 520 of additionally learning the machine learning model based on the watermark image and label information for the watermark image.

According to an embodiment of the present disclosure, when the server 330 or the provider terminal 310 additionally learns the machine learning model, it may mean performing forward propagation. The process of performing forward propagation may be performed by the recognition result providing unit 124 included in the server 330 or the provider terminal 310 . Forward propagation means applying the watermark image to the machine learning model. When data for performing step 520 is in the server 330 , the provider terminal 310 may receive data from the server 330 . Conversely, when data for performing step 520 is in the provider terminal 310 , the server 330 may receive data from the provider terminal 310 . The server 330 or the provider terminal 310 may obtain watermarking result information corresponding to the watermark image by applying the watermark image to the machine learning model.

According to another embodiment of the present disclosure, that the server 330 or the provider terminal 310 additionally learns the machine learning model may mean updating the machine learning model by performing forward propagation and back propagation. The process of performing forward propagation and back propagation may be performed by the server 330 or the model learning unit 114 of the provider terminal 310 . More specifically, the server 330 or the provider terminal 310 may obtain prediction data by applying the watermark image to the machine learning model. This is called forward propagation. In addition, the server 330 or the provider terminal 310 may perform a process of updating the weight included in the machine learning model so that the error is reduced based on the error of the label for the prediction data and the watermark image. This is called backpropagation. Through this, the machine learning model can be updated.

The server 330 or the provider terminal 310 may perform a step 530 of obtaining watermarking result information by applying a watermark image to the machine learning model. Step 530 may be performed by the server 330 or the data recognition unit 120 of the provider terminal 310 . Here, the machine learning model may be a machine learning model updated by the watermark image as described above, or a machine learning model that is not updated. The server 330 or the provider terminal 310 may obtain watermarking result information by applying the watermark image to the machine learning model. That is, forward propagation may be performed in step 530 . The watermarking result information may be the final result value of the machine learning model. The final result value may be an output of the final layer (n) included in the hidden layer. However, the present invention is not limited thereto, and may be a value included in the hidden layer of the machine learning model. For example, the watermarking result information may be an output of the immediately preceding layer (n-1) of the final layer (n). The watermarking result information may be the same as or different from the label for the watermark image. The server 330 may obtain the watermarking result information or may receive it from the provider terminal 310 . In step 460, the server 330 may store the watermarking result information.

6 is a flowchart illustrating an operation of a server according to an embodiment of the present disclosure.

Before FIG. 6 is performed, the server 330 may generate an application programming interface (API) based on the machine learning model received from the provider terminal 310 . Here, the API may refer to an interface that helps a user to easily implement software using a machine learning model. The server 330 may store the API.

The consumer terminal 320 may request registration information related to the machine learning model. The server 330 may perform a step 610 of transmitting registration information related to the machine learning model to the consumer terminal according to the request of the consumer terminal. The consumer terminal 320 may display a screen as shown in FIG. 7 based on the received information.

7 is a diagram illustrating a screen that may be displayed in a consumer terminal according to an embodiment of the present disclosure.

A thumbnail may be displayed in the first area 710 of the screen of the consumer terminal 320 . An image or an image related to the machine learning model may be displayed in the first area 710 . For example, in the first area 710 , information on advantages of the machine learning model or information on a method of using the machine learning model may be displayed as an image.

A description may be displayed in the second area 720 of the screen of the consumer terminal 320 . Text related to the machine learning model may be displayed in the second area 720 . For example, in the second area 720 , information on advantages of the machine learning model or information on a method of using the machine learning model may be displayed as text.

Registration information related to the machine learning model may be displayed in the third area 730 of the screen of the consumer terminal 320 . For example, in the third area 730, the title of the machine learning model, the description of the machine learning model, the price of the machine learning model, the framework, the purpose of the machine learning model, the field of the machine learning model, the class of the machine learning model, the machine At least one of a watermark image of the learning model, GitHub (code), or accuracy may be displayed.

A button for selecting a machine learning model may be displayed in the fourth area 740 of the screen of the consumer terminal 320 . The button in the fourth area 740 may be a button for purchasing or downloading the machine learning model. When the user clicks or touches the button of the fourth area 740 , the consumer terminal 320 may transmit identification information for the selected machine learning model to the server 330 . The server 330 may acquire the machine learning model selected by the consumer terminal 320 from the storage unit based on the identification information.

A button for testing the machine learning model may be displayed in the fifth area 750 of the screen of the consumer terminal 320 . When the user clicks or touches the button of the fifth area 750 , the consumer terminal 320 may display a screen for testing the machine learning model.

Referring back to FIG. 6 , as already described, the consumer terminal 320 may transmit identification information for the selected machine learning model to the server 330 . In addition, the server 330 may perform a step 620 of receiving identification information for the selected machine learning model from the consumer terminal 320 . The server 330 may receive information about the request contents of the consumer terminal 320 together with identification information on the machine learning model. The request content may include either a purchase or a test.

When the request is a purchase, the server 330 and the consumer terminal 320 may transmit and receive payment information. The server 330 may request payment, and the consumer terminal 320 may transmit information necessary for remittance to the server 330 . When the payment is normally made from the consumer terminal 320 to the server 330, the following operations may be further performed.

The server 330 may perform step 630 of transmitting the selected machine learning model to the consumer terminal. The transmitted machine learning model may be the same as the machine learning model received from the provider terminal 310 in step 420 . The server 330 may transmit an API related to the selected machine learning model to the consumer terminal. Consumers can easily implement software based on the selected machine learning model by using the API. With the API, the consumer can reduce the effort to implement the software using the machine learning model. In addition, the server 330 may charge differently depending on whether the selected machine learning model is transmitted to the consumer terminal or an API related to the selected machine learning model is transmitted. For example, when transmitting the selected machine learning model, the server 330 may charge cheaper than when transmitting the API. Conversely, when the server 330 transmits the selected machine learning model, it may charge more expensively than when transmitting the API. However, it is not limited thereto and the charge may not be different.

In addition, when the API for the selected machine learning model is not implemented, the consumer terminal 320 may request the server 330 to implement the API. The server 330 may request the consumer terminal 320 for the cost required to implement the API. When the consumer terminal 320 remits the corresponding cost, the server 330 may generate an API and provide it to the consumer terminal 320 .

The server 330 may perform an operation 640 of transmitting at least one of a watermark image for the selected machine learning model and watermarking result information to the consumer terminal. The consumer terminal 320 may apply the received watermark image to the machine learning model. The consumer terminal 320 may generate watermarking confirmation information by applying a watermark image for the selected machine learning model to the selected machine learning model.

According to an embodiment of the present disclosure, the consumer terminal 320 may compare the watermarking confirmation information with the watermarking result information to determine whether it is the same. The consumer terminal 320 may transmit information on whether the watermarking result information and the watermarking confirmation information match to the server 330 . In addition, the server 330 may perform a step 650 of receiving from the consumer terminal information on whether the watermarking result information and the watermarking confirmation information for the selected machine learning model match.

If the watermarking confirmation information is not the same as the watermarking result information, the consumer terminal 320 may request the machine learning model from the server 330 again. If not identical, the consumer terminal 320 may request the machine learning model from the server 330 again. In addition, the consumer terminal 320 may request a refund from the server 330 . If the watermarking confirmation information is the same as the watermarking result information, the consumer terminal 320 may transmit the transaction completion information to the server 330 .

According to another embodiment of the present disclosure, the consumer terminal 320 may transmit the watermarking confirmation information to the server 330 . The server 330 may compare the watermarking confirmation information with the watermarking result information to determine whether it is the same. If the watermarking confirmation information is not the same as the watermarking result information, the server 330 may retransmit the machine learning model to the consumer terminal 320 . In addition, the consumer terminal 320 may request a refund from the server 330 . If the watermarking confirmation information is the same as the watermarking result information, the server 330 may generate transaction completion information.

Through the process of FIG. 6 as described above, the consumer can confirm that the machine learning model in the server 330 and the machine learning model in the consumer terminal 320 are the same. Also, when the watermarking confirmation information does not match the watermarking result information, the server 330 may transmit the information to the provider so that the provider can take action. In addition, if the same watermark image is input to machine learning models registered by different suppliers and the obtained watermarking confirmation information is the same, it can be considered that one of the different suppliers has violated the copyright. Therefore, the server 330 can effectively protect the copyright.

8 is a diagram illustrating a screen that may be displayed in a consumer terminal according to an embodiment of the present disclosure.

In the case of clicking or touching the model test of the fifth region 750 from the consumer terminal 320 to the consumer terminal 320 in FIG. 7 , the consumer terminal 320 may display a screen as shown in FIG. 8 . The consumer terminal 320 may transmit the input data (input1, input2, ...) less than a predetermined maximum number to the server 330 . The predetermined maximum number may be stored in the memory of the consumer terminal 320 or the server 330 . Referring to FIG. 8 , the predetermined maximum number may be three. The server 330 may receive input data and apply it to the selected machine learning model. In addition, the server 330 may generate prediction data corresponding to the input data as an output of the machine learning model. The server 330 may transmit the prediction data to the consumer terminal 320 . The consumer terminal 320 may display the received prediction data (output1, output2, ...) on the screen.

The consumer may pay the amount charged by the server 330 for an additional test. The server 330 may receive payment information from the consumer terminal 320 . That is, the server 330 may receive a predetermined amount from the consumer terminal 320 . The server 330 may receive the input data exceeding the predetermined maximum number and transmit the prediction data to the consumer terminal 320 . Consumers can test the performance of the machine learning model before purchasing it.

9 is a flowchart illustrating an operation of a server according to an embodiment of the present disclosure.

The server 330 may perform an operation 910 of receiving tuning request information for the machine learning model selected from the consumer terminal 320 . The server 330 may receive payment information from the consumer terminal 320 . The server 330 may proceed to step 920 when the consumer terminal 320 has paid the tuning cost. The tuning cost may be determined based on the size of the data set provided by the consumer terminal 320 . More specifically, the size and cost of the data set may be proportional.

The server 330 may determine whether the information on whether the selected machine learning model is tunable indicates that the selected machine learning model is tunable. Information on whether tuning is possible may be stored in a database of the server 330 . Information on whether tuning is possible may be received from the provider terminal in step 410 and may be stored in a database in step 460 .

When the information on whether the selected machine learning model is tunable or not indicates that the selected machine learning model is tunable, the server 330 may perform the step of requesting the data set from the consumer terminal 320 . The consumer terminal 320 may transmit the data set to the server 330 . The data set may be a set of data used to tune a machine learning model including at least one of a training data set and a test data set for tuning. The training data set may include analysis target data and label information corresponding to the analysis target data. Here, the label information may be ground truth information. For example, the data to be analyzed may be a photo, and the label information may be a name of an object included in the photo. The test data set for tuning may include data to be analyzed. The test dataset for tuning can be used to test the tuned machine learning model. The tuning test data set may further include label information corresponding to the analysis target data.

The server 330 may perform an operation 920 of receiving a data set including at least one of a training data set and a test data set for tuning from the consumer terminal. The server 330 may store information on the minimum number of data required for tuning. The minimum number of data may be predetermined in the server 330 or may be received from the provider terminal 310 . When the number of data included in the data set is smaller than the minimum number of data, the server 330 may request to add the data set to the consumer terminal 320 . Since the server 330 performs machine learning based on data equal to or greater than the minimum number of data, it is possible to generate a machine learning model with high accuracy.

The server 330 may perform step 930 of generating a tuned machine learning model by re-learning the selected machine learning model based on the training data set. However, the present invention is not limited thereto, and the server 330 may transmit the training data set to the provider terminal 310 . The provider terminal 310 may generate a machine learning model tuned based on the training data set and transmit it to the server 330 . The server 330 may receive the tuned machine learning model from the provider terminal 310 .

The server 330 or the provider terminal 310 may perform a step 950 of obtaining accuracy for the tuned machine learning model by applying a test data set for tuning to the tuned machine learning model. The server 330 or the provider terminal 310 may compare the accuracy before tuning with the accuracy after tuning. When the value obtained by subtracting the accuracy after tuning from the accuracy before tuning is less than the threshold value, the server 330 may transmit information indicating that the accuracy is low to the provider terminal 310 . The threshold may be negative. The threshold value may be a predetermined value or a value received from the consumer terminal 320 . In addition, when the value obtained by subtracting the accuracy after tuning from the accuracy before tuning is smaller than the threshold value, the server 330 may request an additional data set from the consumer terminal 320 .

The consumer terminal 320 may abandon the purchase when the accuracy of the tuned machine learning model is not sufficient.

The server 330 may perform watermarking on the tuned machine learning model to obtain the tuned watermarking result information. The process of watermarking has been described with reference to FIG. 5, and overlapping descriptions will be omitted.

The server 330 may perform the step of storing registration information related to the tuned machine learning model, the tuned machine learning model, the tuned test code, test data for tuning, and the tuned watermarking result information in the database. The tuned test code may be the same as the test code of FIG. 4 . However, the present invention is not limited thereto, and the tuned test code is a code specific to the tuned machine learning model, and may be different from the test code of FIG. 4 .

10 is a diagram illustrating a screen that may be displayed in a consumer terminal according to an embodiment of the present disclosure.

The screen of FIG. 10 may be displayed on the consumer terminal 320 before the process of FIG. 9 . The consumer terminal 320 may display registration information related to the machine learning model in the sixth area 1010 based on the information received from the server 330 . The consumer can identify the machine learning model to purchase based on the registration information related to the machine learning model.

A model purchase button may be displayed in the seventh area 1020 of the consumer terminal 320 . When the consumer clicks or touches the model purchase button, the consumer terminal 320 may transmit information requesting purchase to the server 330, and the purchase procedure may be started.

A model tuning request button may be displayed in the eighth area 1030 of the consumer terminal 320 . When the consumer clicks or touches the model tuning request button, the consumer terminal 320 may transmit tuning request information for the selected machine learning model. In response, the server 330 may perform step 910 .

11 is a flowchart illustrating an operation of a server according to an embodiment of the present disclosure.

The server 330 may perform an operation 1110 of receiving a plurality of analysis target data and labeling request information from the consumer terminal. The plurality of analysis target data may be data to be used in a machine learning model. For example, a plurality of analysis target data may be an analysis target of the machine learning model. The plurality of analysis target data may be data input to the data learning unit 110 or the data recognition unit 120 .

Also, the server 330 may perform an operation 1120 of generating a labeling project for a plurality of analysis target data in response to the labeling request information. Creating a project means creating a working space for lyrics. Project participants can perform labeling on a plurality of data to be analyzed by accessing a virtual workspace with a participant terminal. The labeling project may include a plurality of analysis target data obtained from the consumer. In addition, the labeling project may include work guide information received from the consumer terminal 320 . The participant may input label information based on the work guide information using the participant terminal. The participant terminal may transmit label information for a plurality of analysis target data included in the project to the server 330 . A labeling project may contain information about the total points assigned to the project. The total point information may be received from the consumer terminal 320 .

The server 330 may perform an operation 1130 of receiving a plurality of pieces of label information respectively corresponding to a plurality of analysis target data from the terminals of at least one project participant. The server 330 may inspect a plurality of analysis target data received from the participant's terminal. For example, the server 330 may receive label information for one piece of analysis target data from a plurality of participant terminals. The plurality of participant terminals may be an odd number. The server 330 may determine the more selected label information from the participant terminal as the actual label information. In addition, the server 330 may determine the label information selected above a threshold ratio among the participant terminals as the actual label information. The threshold ratio may be a predetermined value. When the server 330 cannot determine the actual label information, it may request a plurality of other participant terminals to attach label information for the corresponding analysis target data.

The server 330 may pay points based on a ratio of the number of pieces of label information transmitted by the project participant terminal among the number of a plurality of analysis target data. For example, points may be paid to the participant terminal according to the following formula.

P = T * (n-a)/M

Here, P may be the points to be paid to project participants, T may be the total points, n may be the number of pieces of label information transmitted by the participant terminals, and M may be the number of pieces of data to be analyzed. Also, a may be the number of label information that is not determined as actual label information among a plurality of label information transmitted by the participant terminal. Points can have branches proportional to cash. The server 330 may pay cash to the participant based on the amount of points.

12 is a diagram illustrating a screen that may be displayed in a participant terminal according to an embodiment of the present disclosure.

An image may be displayed in the ninth area 1211 of the screen 1210 of the participant terminal. The image may be received from the server 330 . The image may be data provided by the consumer terminal 320 to the server 330 . The image displayed in the ninth area 1211 may be included in the analysis target data.

In addition, selectable label information may be included in the tenth area 1212 of the participant terminal. The selectable label information may be class information. The selectable label information may be received from the server 330 . The selectable label information may be data provided by the consumer terminal 320 to the server 330 . The participant terminal may select one of selectable label information based on the participant's input. The label information may be a type of an object indicated in an image.

An image may be displayed in the eleventh area 1221 of the screen 1220 of the participant terminal. In addition, selectable label information may be included in the twelfth area 1222 of the participant terminal. The label information may be a description of the motion of the object shown in the image.

An image may be displayed in the thirteenth area 1231 of the participant terminal. Participants can also specify the location of specific objects. For example, the participant terminal may determine that a specific object is located in the 14th area 1232 and the 15th area 1233 based on the participant's input.

The server 330 may store label information for the data to be analyzed in the information received from the participant terminal. Also, the server 330 may provide points to the participant terminals. In addition, the consumer terminal 320 may re-learn the machine learning model or operate the machine learning model based on the plurality of analysis target data and the plurality of label information.

So far, various embodiments have been mainly looked at. Those of ordinary skill in the art to which the present invention pertains will understand that the present invention can be implemented in a modified form without departing from the essential characteristics of the present invention. Therefore, the disclosed embodiments are to be considered in an illustrative rather than a restrictive sense. The scope of the present invention is indicated in the claims rather than the foregoing description, and all differences within the scope equivalent thereto should be construed as being included in the present invention.

Meanwhile, the above-described embodiments of the present invention can be written as a program that can be executed on a computer, and can be implemented in a general-purpose digital computer that operates the program using a computer-readable recording medium. The computer-readable recording medium includes a storage medium such as a magnetic storage medium (eg, a ROM, a floppy disk, a hard disk, etc.) and an optically readable medium (eg, a CD-ROM, a DVD, etc.).

Claims (10)

Receiving registration information related to the machine learning model from the provider terminal;
receiving the machine learning model, a test code for the machine learning model, and a test data set for the machine learning model from the provider terminal;
applying the test data set to the test code to obtain a test result;
approving the machine learning model based on the test result;
when the machine learning model is approved, performing watermarking to obtain watermarking result information; and
and storing registration information related to the machine learning model, the machine learning model, the test code, the test data set, and the watermarking result information in a database.
The method of claim 1,
The step of obtaining the watermarking result information comprises:
obtaining a watermark image and a label for the watermark image;
further learning the machine learning model based on the watermark image and a label for the watermark image; and
and obtaining watermarking result information by applying the watermark image to the machine learning model.
3. The method of claim 2,
transmitting registration information related to the machine learning model to the consumer terminal at the request of the consumer terminal;
receiving identification information on the selected machine learning model from the consumer terminal;
transmitting the selected machine learning model to the consumer terminal;
transmitting a watermark image and watermarking result information for the selected machine learning model to the consumer terminal; and
The consumer terminal applies the watermark image for the selected machine learning model to the selected machine learning model to generate watermarking confirmation information, and the watermarking result information for the selected machine learning model and the watermarking confirmation information match Method of operation of a server comprising the step of receiving information on whether or not to do so from the consumer terminal.
4. The method of claim 3,
receiving tuning request information for the selected machine learning model from the consumer terminal;
requesting a data set from the consumer terminal when the information on whether the selected machine learning model is tunable indicates that the selected machine learning model is tunable;
receiving a data set including a training data set and a tuning test data set from the consumer terminal;
generating a tuned machine learning model by re-learning the selected machine learning model based on the training data set; and
and acquiring accuracy for the tuned machine learning model by applying the tuning test data set to the tuned machine learning model.
5. The method of claim 4,
performing watermarking on the tuned machine learning model to obtain tuned watermarking result information; and
Registering information related to the tuned machine learning model, the tuned machine learning model, the tuned test code, the test data set for tuning, and the step of storing the tuned watermarking result information in a database Operation method of a server .
6. The method of claim 5,
receiving a plurality of analysis target data and labeling request information from the consumer terminal;
generating a labeling project for the plurality of analysis target data in response to the labeling request information; and
Method of operating a server comprising the step of receiving a plurality of label information respectively corresponding to the plurality of analysis target data from the terminals of at least one project participant.
Receiving tuning request information for the selected machine learning model from the consumer terminal;
requesting a data set from the consumer terminal when the information on whether the selected machine learning model is tunable indicates that the selected machine learning model is tunable;
receiving a data set including at least one of a training data set and a tuning test data set from the consumer terminal;
generating a tuned machine learning model by re-learning the selected machine learning model based on the training data set;
performing watermarking on the tuned machine learning model to obtain tuned watermarking result information; and
Registering information related to the tuned machine learning model, the tuned machine learning model, the tuned test code, the test data set for tuning, and the step of storing the tuned watermarking result information in a database Operation method of a server .
8. The method of claim 7,
receiving a plurality of analysis target data and labeling request information from the consumer terminal;
generating a labeling project for the plurality of analysis target data in response to the labeling request information; and
Method of operating a server comprising the step of receiving a plurality of label information respectively corresponding to the plurality of analysis target data from the terminals of at least one project participant.
Receiving registration information related to the machine learning model from the provider terminal;
receiving the machine learning model, a test code for the machine learning model, and a test data set for the machine learning model from the provider terminal;
applying the test data set to the test code to obtain a test result;
approving the machine learning model based on the test result;
receiving a plurality of analysis target data and labeling request information for use in the machine learning model from a consumer terminal;
generating a labeling project for the plurality of analysis target data in response to the labeling request information; and
Method of operating a server comprising the step of receiving a plurality of label information respectively corresponding to the plurality of analysis target data from the terminals of at least one project participant.
A server for sharing a machine learning model includes a processor and memory,
The processor based on the instructions stored in the memory,
Receive registration information related to a machine learning model from a supplier terminal, receive the machine learning model, a test code for the machine learning model, and a test data set for the machine learning model from the supplier terminal, and the test data set is applied to the test code to obtain a test result, and the machine learning model is approved based on the test result, and when the machine learning model is approved, watermarking is performed to obtain watermarking result information, and the A server for storing registration information related to a machine learning model, the machine learning model, the test code, the test data set, and the watermarking result information in a database.

Images