KR102662498B1 - Method for dynamically switching deep learning models according to inference response time to user queries - Google Patents

Abstract

The present invention relates to a method of dynamically switching a deep learning model according to the inference response time to a user query. When performing inference using a deep learning model according to inference requests by multiple users, various parameter sizes are used. This is about a method of securing optimal inference response speed by dynamically switching a deep learning model with .

Description

How to dynamically switch deep learning models according to the inference response time to user queries {METHOD FOR DYNAMICALLY SWITCHING DEEP LEARNING MODELS ACCORDING TO INFERENCE RESPONSE TIME TO USER QUERIES}

The present invention relates to a method of dynamically switching a deep learning model according to the inference response time to a user inquiry. More specifically, when performing inference using a deep learning model according to inference requests by multiple users, This is about a method to secure optimal inference response speed by dynamically switching deep learning models with multiple parameter sizes.

Because deep learning models require a lot of complex calculations, performing inference using a GPU rather than a CPU helps reduce inference time.

For example, in the case of an image recognition model, using a GPU, inference can be performed in less than one-tenth of the time it takes with a CPU. GPU is a piece of hardware designed for graphic design, specialized in parallel computation, capable of reducing time, high power efficiency, and low cost.

Deep learning inference using GPUs is actively used in various fields such as autonomous driving, image analysis, and natural language processing.

However, for multiple deep learning models, it is very frequent that multiple users request inference at the same time, which can cause a problem of slow response speed due to insufficient GPU resources required for inference.

Therefore, in the present invention, in order to secure the optimal inference response speed according to requests by multiple users, the response speed can be improved by dynamically switching multiple parameter deep learning models to prevent insufficient resources of the GPU required for inference work. I would like to suggest a solution.

Next, we will briefly describe the prior inventions existing in the technical field of the present invention, and then describe the technical details that the present invention seeks to achieve differently compared to the prior inventions.

Korean Patent No. 10-2374530 (2022.03.16.) extracts answer content through a plurality of inference machines that derive intent according to different standards for the query message, sorts it by priority according to the inference rate, and provides information to the user. This is a prior invention regarding an optimal question answering system and method that can provide answers that are more in line with the user's intent.

However, the present invention secures the inference response speed to user requests by dynamically switching multiple parameter deep learning models, and derives optimal responses to various types of user inquiries such as machine learning-based interactive messenger programs. When compared to the Korean Patent No. 10-2374530 (2022.03.16.) to which we reply, there are significant differences in composition.

The present invention was created to solve the above problems, and when performing inference by multiple users' queries (i.e., inference requests), deep learning models with various parameter sizes are dynamically switched to provide optimal inference. The purpose is to provide a method to secure response speed.

Another purpose of the present invention is to provide a method for preventing a decrease in response speed due to a lack of GPU resources required for inference.

In addition, the present invention provides a method for dynamically switching to a deep learning model with an optimized inference response speed by utilizing the inference response time using the currently applied deep learning model and the deep learning model map for each preset response time. Another purpose is to provide

In addition, another purpose of the present invention is to provide a method for implementing similar performance for each deep learning model by learning with a data set refined according to domain characteristics for each deep learning model.

However, the technical challenges that this embodiment aims to achieve are not limited to the technical challenges described above, and other technical challenges may exist.

The method of dynamically switching a deep learning model according to the inference response time to a user query according to an embodiment of the present invention is performed in a system that dynamically switches the deep learning model, and receives inference requests from a plurality of user terminals. A step of receiving an inference request; an inference performing step of inferring by inputting the received inference request into a deep learning model currently being applied; An inference response that measures the inference response time required for inference for each user performed in the currently applied deep learning model, averages the inference response time required for inference for each user measured, and measures the average inference response time. time measurement step; And in order to prevent a decrease in response speed due to lack of GPU resources required for inference, any one that optimizes the inference response time of the currently applied deep learning model among a plurality of deep learning models with reference to the measured inference response time. A deep learning model dynamic conversion step of converting to a deep learning model, wherein the deep learning model dynamic conversion step includes, when the average inference response time is measured in the inference response time measurement step, a deep learning model map for each preset response time. Inquiry step; Comparing the parameter size of the currently applied deep learning model with the parameter size matched to the same inference response time as the average inference response time confirmed in the deep learning model map for each response time; determining whether the comparison results meet preset dynamic conversion conditions; If the dynamic switching condition is not satisfied as a result of the determination, performing inference according to the inference request of the user terminal by using the currently applied deep learning model as is; And if the dynamic switching condition is satisfied as a result of the determination, the currently applied deep learning model is a deep learning model having a parameter size matched to the same inference response time as the average inference response time confirmed in the deep learning model map for each response time. Characterized in that it includes a step of converting to;

delete

delete

In addition, in the dynamic switching condition, the parameter size of the currently applied deep learning model is different from the parameter size matched to the same inference response time as the average inference response time confirmed in the deep learning model map for each response time, and the parameter size is different. It is characterized by a condition in which the size of is different and occurs continuously more than a preset number of times.

In addition, the inference performance step converts the currently applied deep learning model into a deep learning model with an inference response time equal to the average inference response time confirmed in the deep learning model map for each response time through the deep learning model dynamic conversion step. The method further includes performing inference on the inference request of the user terminal using the currently applied deep learning model until the conversion is completed.

In addition, the deep learning model map by response time is set through testing in advance, and is characterized by matching deep learning models with different parameter sizes according to the minimum and maximum inference response times inferred from a given GPU.

In addition, the deep learning model is characterized in that each deep learning model is created by learning from a data set refined according to the domain characteristics, thereby preventing differences in the quality of the inference response depending on the parameter size.

In addition, the domain is a characteristic of the data learned by the deep learning model, and can be divided into a data domain including images, text, voice, and video, and a task domain including classification, regression, generation, recommendation, and robot control. Do it as

As described above, according to the method of dynamically switching a deep learning model according to the inference response time to a user query of the present invention, the optimal deep learning model is dynamically switched in response to inference requests by multiple users. By securing the inference response speed, there is an effect of preventing the response speed from deteriorating due to a lack of GPU resources required for inference.

However, the effects of the present invention are not limited to the effects described above, and effects not mentioned can be clearly understood by those skilled in the art from this specification and the attached drawings.

Figure 1 is a diagram schematically showing the entire configuration including a system for dynamically switching a deep learning model according to the inference response time to a user inquiry according to an embodiment of the present invention.
Figure 2 is a block diagram showing in more detail the configuration of a system for dynamically switching a deep learning model according to an embodiment of the present invention.
Figure 3 is a diagram showing an example of a deep learning model map for each response time applied to the present invention.
Figure 4 is a diagram showing the hardware structure of a system for dynamically switching deep learning models according to an embodiment of the present invention.
Figure 5 is a flowchart showing in detail the operation process of a method for dynamically switching a deep learning model according to the inference response time to a user inquiry according to an embodiment of the present invention.

Hereinafter, specific embodiments of the present invention will be described in detail with reference to the drawings. However, the spirit of the present invention is not limited to the presented embodiments, and a person skilled in the art who understands the spirit of the present invention may add, change, or delete other components within the scope of the same spirit, thereby creating other degenerative inventions or the present invention. Other embodiments that are included within the scope of the invention can be easily proposed, but this will also be said to be included within the scope of the invention of the present application.

In addition, components having the same function within the scope of the same idea shown in the drawings of each embodiment will be described using the same reference numerals.

Figure 1 is a diagram schematically showing the entire configuration including a system for dynamically switching a deep learning model according to the inference response time to a user inquiry according to an embodiment of the present invention.

As shown in Figure 1, the present invention is a system for dynamically switching a deep learning model according to the inference response time to a user inquiry (100, hereinafter referred to as a system for dynamically switching a deep learning model), and a plurality of user terminals. (200), database (300), etc.

The system 100 for dynamically switching the deep learning model is composed of a server computer, a platform, etc., and receives inference requests using a deep learning model from a plurality of user terminals 200 connected to communication through a network, and the user A response is provided according to the inference request from the terminal 200.

At this time, inference requests from the user terminal 200 are often requested simultaneously, and as a result, GPU resources required for inference responses are insufficient, which may result in a slowdown in responses to inference requests.

To solve this problem, the system 100 for dynamically switching the deep learning model dynamically switches a deep learning model with several parameter sizes when performing inference on inference requests received from a plurality of user terminals 200. Switch to to ensure optimal inference response speed.

In particular, the system 100 for dynamically switching the deep learning model utilizes the average inference response time for inference requests from multiple users using the currently applied deep learning model and the deep learning model map for each preset response time to optimize the By performing a dynamic transition to a deep learning model with an inference response speed, it prevents a decrease in response speed due to a lack of GPU resources required for inference and provides inference results to users at an optimal response speed.

At this time, the deep learning model map by response time is information matching deep learning models with different parameter sizes according to the minimum inference response time and maximum inference response time (see Figure 3).

In addition, it is necessary to ensure that there is no difference in response quality depending on parameter size by learning and creating each deep learning model matched on the deep learning model map for each response time with a data set refined according to domain characteristics. .

Meanwhile, the parameter size of the deep learning model represents the number of weights and biases that the neural network must learn. The weights serve to connect input and output, the bias is a value added to the output of the neuron, and the The larger the variable size, the more complex relationships the neural network can learn.

As an example, GPT-3 is a language model with 175 billion parameters, and the size of these parameters means that it can learn the complex relationships needed to understand and create meaning in text.

In addition, the number of required parameters varies depending on the type of model such as CNN, RNN, LSTM, etc. The more layered the model is, the more required parameters are, and the larger the data size, the more required parameters are.

In this way, the parameter size of the deep learning model affects the performance of the model, and it is very important to appropriately select the parameter size when designing a deep learning model.

The larger the parameter size, the better the model's performance, but if the parameter size is too large, the model may become difficult to train and the risk of overfitting may increase.

The user terminal 200 may be a digital device equipped with a processor and memory and computing power, such as a laptop computer, notebook computer, desktop computer, web pad, or mobile phone owned by each individual. Additionally, various functions provided by the server or system can be executed through web-based or separate software/applications.

The user terminal 200 connects to the system 100 for dynamically switching the deep learning model through the network and requests inference using the deep learning model. In other words, it performs inquiries about what the user is curious about.

The network may be a core network integrated with a wired public network, a wireless mobile communication network, or a mobile Internet, and may include the TCP/IP protocol and various services existing at its upper layer, such as HTTP (Hyper Text Transfer Protocol) and HTTPS (Hyper Text Transfer Protocol). Secure), Telnet, FTP (File Transfer Protocol), etc., it can refer to a global open computer network structure, and is not limited to these examples, but comprehensively refers to a data communication network that can transmit and receive data in various forms. .

Additionally, the applicant terminal 200 receives inference results for inference requests from the system 100 that dynamically switches the deep learning model.

The database 300 stores and manages various operation programs used in the system 100 for dynamically switching the deep learning model, as well as deep learning model maps for each response time and various deep learning models used for inference.

Additionally, the database 300 may store and manage information on inference requests requested by each user and inference results provided to the user by performing a deep learning model according to the inference request.

Figure 2 is a block diagram showing in more detail the configuration of a system for dynamically switching a deep learning model according to an embodiment of the present invention.

As shown in Figure 2, the system 100 for dynamically switching the deep learning model includes an inference request receiver 110, an inference performance unit 120, an inference response time measurement unit 130, and a deep learning model dynamic switch. It is comprised of a unit 140, an inference result providing unit 150, etc.

In addition, the system 100 for dynamically switching the deep learning model, although not shown in FIG. 2, generates, updates and processes deep learning models including open interactive language models such as the GPT (Generative Pre-trained Transformer) series. Additional configurations can be configured to perform management.

The inference request receiving unit 110 receives an inference request from the user terminal 200 connected to communication through a network, and transmits the received inference request to the inference performing unit 120.

The inference performance unit 120 inputs the inference request for each user received through the inference request receiver 110 into the deep learning model currently being applied to perform inference. At this time, the inference result performed by the inference performance unit 120 is sent to the corresponding user terminal 200 through the inference result provider 150, regardless of the deep learning model dynamic conversion in the deep learning model dynamic conversion unit 140. It can be provided as .

In addition, if the inference performance unit 120 determines to use the currently applied deep learning model as is because the dynamic conversion condition is not satisfied as a result of the determination by the deep learning model dynamic conversion unit 140, the currently applied deep learning model is not satisfied. Inference can be performed according to the inference request of the user terminal 200 by using the learning model as is.

In addition, even if the inference performance unit 120 satisfies the dynamic conversion conditions as determined by the deep learning model dynamic conversion unit 140 and dynamic conversion to another deep learning model is made, the other deep learning model to be converted is Until it is finally loaded into the system, inference according to the inference request of the user terminal 200 can be performed using the deep learning model currently being applied.

The inference response time measurement unit 130 performs a function of measuring the inference response time required for inference performed by the inference performance unit 120.

At this time, the inference response time measurement unit 130 measures the inference response time required for inference for each user performed in the currently applied deep learning model, and then averages the inference response time required for inference for each user measured. Thus, the average inference response time is measured, and the measured average inference response time is transmitted to the deep learning model dynamic conversion unit 140.

The deep learning model dynamic switching unit 140 refers to the inference response time measured by the inference response time measurement unit 130 and determines whether to apply the currently applied deep learning model as is or to infer among a plurality of deep learning models. Decide whether to switch to one different deep learning model that optimizes response time.

At this time, the deep learning model dynamic conversion unit 140 is composed of a query unit 141, a parameter size comparison unit 142, a dynamic conversion condition determination unit 143, and a dynamic conversion processing unit 144.

When the average inference response time is delivered from the inference response time measurement unit 130, the search unit 141 searches the deep learning model map for each response time stored in the database 300. The deep learning model map for each response time will be described in detail in FIG. 3.

The parameter size comparison unit 142 checks the parameter size of the deep learning model currently being applied.

In addition, the parameter size comparison unit 142 refers to the deep learning model map for each response time searched through the inquiry unit 141, and requests inference for each user measured by the inference response time measurement unit 130. Check the parameter size of the deep learning model that has an inference response time equal to the average inference response time for .

Subsequently, the parameter size comparison unit 142 determines the parameter size of the two deep learning models (i.e., the parameter size of the deep learning model currently being applied and the average of the inference requests for each user confirmed in the deep learning model map for each response time). The inference response time and the parameter size of the deep learning model having the same inference response time) are compared, and the compared result is transmitted to the dynamic switching condition determination unit 143.

The dynamic conversion condition determination unit 143 determines whether a preset dynamic conversion condition is met based on the result compared by the parameter size comparison unit 142, and the determined result is sent to the dynamic conversion processor 144. Pass it to

At this time, the dynamic switching condition is the parameter size of the deep learning model currently being applied and the parameter size of the deep learning model with the same inference response time as the average inference response time for each user's inference request confirmed in the deep learning model map by response time. refers to a condition in which the sizes of two parameters are different and occur consecutively more than a preset number of times (e.g., three times).

At this time, in the present invention, it is preferable to set the number of times to 3, but it is not limited to this, and it can be set to reduce or increase depending on the usage environment.

If the dynamic switching condition is satisfied as determined by the dynamic switching condition determination unit 143, the dynamic switching processing unit 144 selects the currently applied deep learning model as equal to the average inference response time confirmed in the deep learning model map for each response time. It performs the function of converting to a deep learning model with inference response time.

In other words, if the dynamic conversion condition is satisfied, the currently applied deep learning model is dynamically converted to another deep learning model with a response speed optimized for inference at the current time.

The inference result providing unit 150 provides the inference result performed by the inference performing unit 120 to the user terminal 200.

The inference results generated at this time can be generated as text data, or in graphic or table format according to a predetermined template. Additionally, inference results can be generated in several file formats, and can be converted to a desired file format and provided upon user request.

Figure 3 is a diagram showing an example of a deep learning model map for each response time applied to the present invention.

As shown in FIG. 3, the system 100 that dynamically switches the deep learning model can build, store, and manage a deep learning model map for each response time in advance in the database 300.

The deep learning model map by response time is table data set by matching deep learning models with different parameter sizes according to the minimum inference response time and maximum response time inferred through a given GPU, and can be set through testing in advance. .

For example, if the response time inferred from a given GPU is 60 to 120 seconds, a deep learning model with a parameter size of 1.3B (1.3 billion) is matched, and if the response time is 30 to 60 seconds, the parameter size is matched. Match a deep learning model with a parameter size of 3B (3 billion), and if the response time is between 5 and 30 seconds, match a deep learning model with a parameter size of 13B (13 billion). If the response time is between 0 and 5 seconds, match the deep learning model with a parameter size of 13B (13 billion). By matching a deep learning model with a parameter size of 40B (40 billion), it is possible to set an appropriate model map for each response time. Here, the response time or parameter size can be arbitrarily changed based on test results.

At this time, the deep learning model must be created by learning from a data set refined according to domain characteristics in order to prevent differences in the quality of the inference response depending on the parameter size.

In addition, the domain refers to the characteristics of the data learned by the deep learning model, and can be divided into a data domain including images, text, voice, and video, and a task domain including classification, regression, generation, recommendation, and robot control. .

Here, the data domain includes resolution, color, scene, lighting, camera type, etc. for images, language, grammar, style, topic, etc. for text, and sound quality, direction, background noise, etc. for voice. In the case of video, it may include frame rate, resolution, color, scene, etc.

Additionally, the task domain includes images, text, voice, video, etc. for classification, numerical prediction, time series prediction, etc. for regression, images, text, voice, video, etc. for generation, and recommendation. In this case, it includes products, movies, music, etc., and in the case of robot control, it can include object recognition, movement, interaction, etc.

Meanwhile, deep learning models with hundreds of millions or billions of parameters take a long time because they must be trained using large amounts of data and computational resources, but they can solve complex problems by learning complex patterns using numerous parameters. It can provide more accurate and efficient results than existing models, and can be used in a wide variety of fields such as natural language processing, computer vision, speech recognition, and natural language generation.

As an example, natural language processing deep learning models are models used to understand and generate text such as question answering, translation, summarization, and generation, and include GPT-3, RoBERTa, LaMDA, Megatron-Turing NLG, and Jurassic-1 Jumbo. . In addition, computer vision deep learning models are models used to understand and process images and videos for object recognition, face recognition, and natural image classification, and include ResNet, VGG, YOLO, SSD, Swin Transformer, and ConvMixer. In addition, voice recognition deep learning models are models used to understand and process voices such as voice commands and voice synthesis, and include Transformer, WaveNet, DNN, Megatron-Turing NLG, and WuDao 2.0. In addition, natural language generation deep learning models are models used to generate text such as news articles, web pages, poems, and codes, and include GPT-3, T5, Megatron-Turing NLG, Jurassic-1 Jumbo, and WuDao 2.0. In addition, there are deep learning models for games such as AlphaGo, AlphaZero, Dota 2 AI, StarCraft II AI, and WuDao 2.0.

Figure 4 is a diagram showing the hardware structure of a system for dynamically switching deep learning models according to an embodiment of the present invention.

As shown in FIG. 4, the hardware structure of the system 100 for dynamically switching the deep learning model includes a central processing unit 1000, a memory 2000, a user interface 3000, a database interface 4000, It consists of a network interface (5000), a web server (6000), a graphics processing unit (7000), etc.

The user interface 3000 provides an input and output interface to the user by using a graphical user interface (GUI).

The database interface 4000 provides an interface between a database and a hardware structure. The network interface 5000 provides network connections between devices owned by users.

The web server 6000 provides a means for users to access the hardware structure through a network. Most users can use the system 100 to connect to a web server remotely and dynamically switch the deep learning model.

Each step of the above-described configuration or method may be implemented as computer-readable code on a computer-readable recording medium or transmitted through a transmission medium. A computer-readable recording medium is a data storage device that can store data that can be read by a computer system.

Examples of computer-readable recording media include, but are not limited to, databases, ROM, RAM, CD-ROM, DVD, magnetic tape, floppy disk, and optical data storage devices. Transmission media may include carrier waves transmitted through the Internet or various types of communication channels. Additionally, the computer-readable recording medium may be distributed through a network-coupled computer system such that the computer-readable code is stored and executed in a distributed manner.

In addition, at least one or more components applied to the present invention may include or be implemented by a processor such as a central processing unit (CPU) or microprocessor that performs each function, and two or more of the components may be implemented as a single It can be combined into components and perform all operations or functions of two or more components combined. Additionally, part of at least one or more components applied to the present invention may be performed by other components among these components. Additionally, communication between the components may be performed through a bus (not shown).

Next, an embodiment of a method for dynamically switching a deep learning model according to the inference response time to a user query according to the present invention configured as described above will be described in detail with reference to FIG. 5. At this time, the order of each step according to the method of the present invention may be changed depending on the usage environment or a person skilled in the art.

Figure 5 is a flowchart showing in detail the operation process of a method for dynamically switching a deep learning model according to the inference response time to a user inquiry according to an embodiment of the present invention.

As shown in FIG. 5, the system 100 for dynamically switching the deep learning model receives an inference request (i.e., query) from the user terminal 200 connected through the network (S100).

At this time, the system 100 that dynamically switches the deep learning model may provide a user interface such as a web page or app screen to receive the inference request.

Additionally, the inference request received from the user terminal 200 may be a query such as an interactive question, command, or directive sentence used in daily life.

When an inference request is received from the user terminal 200 through step S100, the system 100, which dynamically switches the deep learning model, performs inference on the inference request through the deep learning model currently being applied ( S200).

Afterwards, the system 100 for dynamically switching the deep learning model measures the inference response time required for inference (S300). In other words, the average inference response time is measured by measuring the inference response time required for inference for each user performed in the currently applied deep learning model, and then averaging the total inference response time required for inference for each user measured above. will be.

Subsequently, the system 100 for dynamically switching the deep learning model performs the task of converting the deep learning model currently being applied to another deep learning model that optimizes the inference response time based on the inference response time measured in step S300. do.

To be more specific, the system 100 for dynamically switching the deep learning model measures the average inference response time through step S300, and then stores and manages the deep learning model map for each response time in the database 300. Search (S400).

And the parameter size of the deep learning model currently being applied and the parameter size of the deep learning model having the same inference response time as the average inference response time measured in step S300 confirmed in the deep learning model map for each response time viewed in step S400. is compared, and based on the comparison result, it is determined whether the dynamic conversion condition is met (S500).

At this time, the dynamic switching condition is that the parameter size of the currently applied deep learning model and the parameter size of the specific deep learning model confirmed in the deep learning model map for each response time based on the average inference response time measured in step S300 are different from each other, and As explained above, the condition in which variables with different sizes occur three or more times in succession is as described above.

If the dynamic switching condition is satisfied as determined through step S500, the system 100 for dynamically switching the deep learning model measures the deep learning model currently being applied in step S300 confirmed through the deep learning model map for each response time. Switch to a deep learning model with an inference response time equal to the average inference response time (S600).

At this time, the system 100 for dynamically converting the deep learning model uses the currently applied deep learning model as is until conversion of the currently applied deep learning model to another deep learning model is completed through step S600. Performs inference on inference requests.

In addition, the system 100 for dynamically switching the deep learning model, if the dynamic switching condition is not satisfied as a result of the determination through step S500, uses the currently applied deep learning model as is to make inferences according to the inference request of the user terminal. Perform (S700).

As such, since the present invention secures optimal inference response speed by dynamically switching multiple parameter deep learning models in response to inference requests by multiple users, the response speed is reduced due to lack of GPU resources required for inference. can be prevented.

In the attached drawings, in order to more clearly express the technical idea of the present invention, components that are unrelated or less relevant to the technical idea of the present invention are briefly expressed or omitted.

In the above, the configuration and features of the present invention have been described based on the embodiments according to the present invention, but the present invention is not limited thereto, and various changes or modifications may be made within the spirit and scope of the present invention. It is obvious to those skilled in the art, and therefore, it is stated that such changes or modifications fall within the scope of the appended patent claims.

100: A system that dynamically switches deep learning models according to the inference response time to user queries
110: Inference request receiver
120: Inference execution unit
130: Inference response time measurement unit
140: Deep learning model dynamic conversion unit
141: Inquiry section
142: Parameter size comparison unit
143: Dynamic conversion condition determination unit
144: Dynamic conversion processing unit
150: Inference result provision unit
200: user terminal
300: database

Claims (8)

This is performed in a system that dynamically switches deep learning models,
An inference request receiving step of receiving inference requests from a plurality of user terminals;
an inference performing step of inferring by inputting the received inference request into a deep learning model currently being applied;
An inference response that measures the inference response time required for inference for each user performed in the currently applied deep learning model, averages the inference response time required for inference for each user measured, and measures the average inference response time. time measurement step; and
In order to prevent a decrease in response speed due to lack of GPU resources required for inference, with reference to the measured inference response time, the currently applied deep learning model is selected from one of a plurality of deep learning models that optimizes the inference response time. It includes a deep learning model dynamic conversion step of converting to a learning model,
The deep learning model dynamic conversion step is,
When the average inference response time is measured in the inference response time measurement step, searching a deep learning model map for each preset response time;
Comparing the parameter size of the currently applied deep learning model with the parameter size matched to the same inference response time as the average inference response time confirmed in the deep learning model map for each response time;
determining whether the comparison results meet preset dynamic conversion conditions;
If the dynamic switching condition is not satisfied as a result of the determination, performing inference according to the inference request of the user terminal by using the currently applied deep learning model as is; and
If the dynamic conversion condition is satisfied as a result of the determination, the currently applied deep learning model is converted into a deep learning model with a parameter size matched to the inference response time equal to the average inference response time confirmed in the deep learning model map for each response time. A method of dynamically switching a deep learning model, comprising the step of switching.
delete delete In claim 1,
The dynamic switching conditions are:
The parameter size of the currently applied deep learning model and the parameter size matched to the same inference response time as the average inference response time confirmed in the deep learning model map for each response time are different, and the sizes of the parameters are different continuously. A method of dynamically switching a deep learning model characterized by a condition that occurs more than a set number of times.
delete In claim 1,
The deep learning model map for each response time is,
A method of dynamically switching a deep learning model, which is set through testing in advance and is set by matching deep learning models with different parameter sizes according to the minimum and maximum inference response times inferred from a given GPU. delete delete