KR20210086490A - Method for optimizing program using reinforcement learning - Google Patents

Abstract

The present disclosure relates to a method for automatically optimizing a program based on reinforcement learning. The method for automatically optimizing the program based on the reinforcement learning comprises: (a) a step of receiving input for a source program comprising a fixed parameter and a variable parameter; (b) a step of generating the source program based on the received input; (c) a step of converting the source program into an object program; (d) a step of measuring a performance of the executed object program by executing the converted object program; (e) a step of inputting the variable parameter and the measured performance to a machine learning model, and outputting the variable parameter conversion amount; and (f) a step of regenerating the source program reflecting the variable parameter conversion amount. Therefore, the present invention is capable of efficiently searching a larger number of search ranges in a given time.

Description

Reinforcement learning-based program optimization method {METHOD FOR OPTIMIZING PROGRAM USING REINFORCEMENT LEARNING}

The present invention relates to a program optimization method based on reinforcement learning. Specifically, it relates to a method for automatically optimizing a program based on reinforcement learning and a computer program for continuously improving program performance.

It can be important to optimize a computer program to perform various program executions or tasks with limited resources, energy, resources, and the like. Here, program optimization may refer to a series of processes of changing a program so that a given program outputs the same result with higher performance for a given input.

In general, program optimization can be performed by modifying, compiling, and executing the program several times by an experienced programmer who simultaneously understands the hardware system and the structure of the program. However, such a series of processes requires a cost for input of an experienced programmer and may take a long time.

The present disclosure provides a method and a computer program for automatically optimizing a program based on reinforcement learning to solve the above problems.

The present disclosure may be embodied in various ways, including a method, a system, an apparatus, or a computer-readable storage medium storing instructions, a computer program.

According to an embodiment of the present disclosure, a method for automatically optimizing a program based on reinforcement learning comprises the steps of: (a) receiving an input for a source program comprising fixed parameters and variable parameters; (b) the received input generating a source program based on (c) converting the source program into an object program, (d) measuring the performance of the executed object program by executing the converted object program, and (e) variable parameters and inputting the measured performance into the machine learning model, outputting a variable parameter change amount, and (f) regenerating a source program reflecting the variable parameter change amount.

According to an embodiment of the present disclosure, step (f) includes applying a variable parameter change amount to a variable parameter, and outputting a regenerated source program using the variable parameter and the fixed parameter to which the variable parameter change amount is applied. .

According to an embodiment of the present disclosure, after step (f), (g) setting the regenerated source program as the source program, further comprising the step of performing steps (c) to (f) again.

According to an embodiment of the present disclosure, the method further includes training the machine learning model by repeatedly performing step (g) until a predetermined batch size is reached.

According to an embodiment of the present disclosure, executing steps (c) to (f), repeatedly executing at least some of steps (c) to (f) until the target performance is reached, and the target performance If reached, the method further includes determining a source program corresponding to the highest performance as an optimized source program.

According to an embodiment of the present disclosure, step (e) includes inputting the variable parameter, the measured performance, and the source program into the machine learning model to output the variable parameter change amount.

According to an embodiment of the present disclosure, the machine learning model includes a deep learning model. The step of inputting the variable parameter, the measured performance and the source program into the machine learning model and outputting the variable parameter variation includes: generating an input vector representing the variable parameter, the measured performance, and the source program; and deep learning the generated input vector. and generating an output vector representing the variable parameter variation through the deep learning model by input to the model.

According to an embodiment of the present disclosure, step (d) includes measuring a time taken to output a result value through the target program when the target program is executed, and determining the measured time as an index of performance. .

According to an embodiment of the present disclosure, the step of measuring the time it takes to output a result value through the target program when the target program is executed includes: inputting a test set for the target program and a result value corresponding to the test set and determining the time taken to be output through the target program as an indicator of performance.

A computer program stored in a computer-readable recording medium is provided for executing a method for automatically optimizing a program based on the above-described reinforcement learning according to an embodiment of the present disclosure on a computer.

According to some embodiments of the present disclosure, the processor automatically changes the variable parameters to generate an optimized source program from a given source program, without the programmer of the program changing or modifying the variable parameters directly each time for optimizing the performance of the program. can do.

According to some embodiments of the present disclosure, since the processor can replace the role of an experienced programmer, the program optimization can be completed in a shorter time than the programmer directly optimizing, or a larger number of search ranges can be efficiently searched at a given time. can be explored with

According to some embodiments of the present disclosure, when program optimization is performed, an amount of human resource input required for program optimization can be minimized to generate profits, and service quality can be improved by using a program with improved performance.

According to some embodiments of the present disclosure, without the developer of the program directly optimizing the program by changing variable parameters, it is possible to automatically generate the optimized program by repeating the performance measurement until the processor reaches the target performance. have.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Embodiments of the present disclosure will be described with reference to the accompanying drawings described below, wherein like reference numerals denote like elements, but are not limited thereto.
1 is a diagram illustrating an example in which a processor generates an optimized source program according to an embodiment of the present disclosure.
2 is a block diagram illustrating an internal configuration of a processor according to an embodiment of the present disclosure.
3 is a flowchart illustrating an example of a method for automatically optimizing a program based on reinforcement learning according to an embodiment of the present disclosure.
4 is a block diagram illustrating an example in which a process for determining an optimized program is repeatedly performed according to an embodiment of the present disclosure.
5 is a diagram illustrating an example of a machine learning model that outputs a variable parameter variation based on reinforcement learning according to an embodiment of the present disclosure.
6 is a diagram illustrating an example of a machine learning model that outputs a variable parameter variation based on reinforcement learning according to another embodiment of the present disclosure.
7 is a flowchart illustrating an example of a method for training a reinforcement learning-based machine learning model according to an embodiment of the present disclosure.
8 is a flowchart illustrating an example of an inference method through a machine learning model according to an embodiment of the present disclosure.
9 is a diagram illustrating an example of performing reinforcement learning on a machine learning model according to an embodiment of the present disclosure.

Hereinafter, specific contents for carrying out the present disclosure will be described in detail with reference to the accompanying drawings. However, in the following description, if there is a risk of unnecessarily obscuring the subject matter of the present disclosure, detailed descriptions of well-known functions or configurations will be omitted.

In the accompanying drawings, identical or corresponding components are assigned the same reference numerals. In addition, in the description of the embodiments below, overlapping description of the same or corresponding components may be omitted. However, even if description regarding components is omitted, it is not intended that such components are not included in any embodiment.

Advantages and features of the disclosed embodiments, and methods of achieving them, will become apparent with reference to the embodiments described below in conjunction with the accompanying drawings. However, the present disclosure is not limited to the embodiments disclosed below, but may be implemented in various different forms, and only the present embodiments allow the present disclosure to be complete, and the present disclosure provides those skilled in the art with the scope of the invention. It is provided for complete information only.

Terms used in this specification will be briefly described, and the disclosed embodiments will be described in detail. The terms used in this specification are selected as currently widely used general terms as possible while considering the functions in the present disclosure, but may vary depending on the intention or precedent of a person skilled in the art, the emergence of new technology, and the like. In addition, in a specific case, there is a term arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the corresponding invention. Therefore, the terms used in the present disclosure should be defined based on the meaning of the term and the contents of the present disclosure, rather than the simple name of the term.

References in the singular herein include plural expressions unless the context clearly dictates that the singular is singular. Also, the plural expression includes the singular expression unless the context clearly dictates the plural. When a part includes a certain component throughout the specification, this means that other components may be further included, rather than excluding other components, unless otherwise stated.

In addition, the term 'module' or 'unit' used in the specification means a software or hardware component, and 'module' or 'unit' performs certain roles. However, 'module' or 'unit' is not meant to be limited to software or hardware. A 'module' or 'unit' may be configured to reside on an addressable storage medium or may be configured to refresh one or more processors. Thus, as an example, a 'module' or 'part' refers to components such as software components, object-oriented software components, class components, and task components, and processes, functions, and properties. , procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, database, data structures, tables, arrays or at least one of variables. Components and 'modules' or 'units' are the functions provided therein that are combined into a smaller number of components and 'modules' or 'units' or additional components and 'modules' or 'units' can be further separated.

According to an embodiment of the present disclosure, a 'module' or a 'unit' may be implemented with a processor and a memory. 'Processor' should be construed broadly to include general purpose processors, central processing units (CPUs), microprocessors, digital signal processors (DSPs), controllers, microcontrollers, state machines, and the like. In some contexts, a 'processor' may refer to an application specific semiconductor (ASIC), a programmable logic device (PLD), a field programmable gate array (FPGA), or the like. 'Processor' refers to a combination of processing devices, such as, for example, a combination of a DSP and a microprocessor, a combination of a plurality of microprocessors, a combination of one or more microprocessors in combination with a DSP core, or any other such configurations. You may. Also, 'memory' should be construed broadly to include any electronic component capable of storing electronic information. 'Memory' means random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), programmable read-only memory (PROM), erase-programmable read-only memory (EPROM); may refer to various types of processor-readable media, such as electrically erasable PROM (EEPROM), flash memory, magnetic or optical data storage, registers, and the like. A memory is said to be in electronic communication with the processor if the processor is capable of reading information from and/or writing information to the memory. A memory integrated in the processor is in electronic communication with the processor.

In the present disclosure, 'automatically optimizing a program' or 'optimizing a program' may refer to a series of processes of changing a program to output the same result with higher performance for a given input of a given program.

In the present disclosure, 'performance' may be variously defined according to the type of program and the execution environment. For example, execution time, energy consumption, resource usage, etc. may be used as an index for determining the performance of the program. For example, performance may be determined based on the time it takes to respond to a test input of a given program and output a result value corresponding to the test input through the program.

In this disclosure, a 'program' means a general program (eg, C program, Java program, Python program, etc.) executed on a CPU, a program (eg, Open CL, CUDA, etc.) executed on a GPU and/or other than CPU and GPU It may refer to all software executed on the hardware of

In the present disclosure, 'fixed parameters' and 'variable parameters' may include some or all information necessary to configure one program, and may be embodied in various forms depending on a method of generating or implementing a program. According to an embodiment, the fixed parameter may refer to a predefined basic (skeletal) code required to construct a given program, and the variable parameter may represent a numerical value of a part of a source program that can be implemented differently. have. For example, a skeleton code using a large number of C macro variables may be defined as a fixed parameter, and a set of C macro variable declarations may be defined as a variable parameter. As another example, a variable parameter may be defined for each operation in a program. In the case of a deep learning program, the variable parameter may refer to a parameter determined according to how to implement when defining a kernel. As another example, the variable parameter may refer to a number indicating how many input values (eg, pixels) are calculated per one thread of the GPU, how many input values are calculated at one time in a loop, and the like.

1 is a diagram illustrating an example in which a processor 100 generates an optimized source program 120 according to an embodiment of the present disclosure. As illustrated, the processor 100 may generate the optimized source program 120 by using the source program 110 . Here, the source program 110 may refer to a source code of a program (eg, a GPU program) operable in a multi-core CPU, GPU, FPGA, or the like. Also, the optimized source program 120 may refer to a source program having better performance than other source programs (eg, a program having the best performance).

According to an embodiment, the processor 100 may receive an input for a source program including a fixed parameter and a variable parameter. For example, the fixed parameter may indicate a predetermined parameter in which the number and type of parameters are fixed, and the variable parameter may indicate a parameter in which the number and type of parameters may be variably changed. When an input to the source program 110 is received, the processor 100 may generate the source program 110 based on the received input.

Then, the processor 100 may convert the source program 110 into an object program. For example, the processor 100 may convert the source program 110 into an object program using an interpreter such as a compiler, an interpreter, an assembler, or a preprocessor. Here, the target program is a language that can be immediately executed by a computer, and may refer to a program (eg, binary code, etc.) generated by translating the source program 110 .

The processor 100 may measure the performance of the executed object program by executing the converted object program. For example, the performance of the target program may be measured based on execution time required to output the same result, energy consumption, resource usage, etc., but is not limited thereto, and any performance capable of measuring the performance of the program Indicators may be used.

According to an embodiment, the processor 100 may input the variable parameter and the measured performance to the machine learning model, and output the variable parameter change amount. Here, the machine learning model may refer to an agent of reinforcement learning being trained or learning by reinforcement learning, for example, and may include a deep learning model such as a deep neural network. . That is, the processor 100 may output the variable parameter change amount for the given program from the machine learning model trained to output the variable parameter change amount capable of improving the performance of the target program.

When the variable parameter change amount is output, the processor 100 may regenerate the source program reflecting the variable parameter change amount. That is, the processor 100 may change the number, type, etc. of the variable parameter by reflecting the variable parameter change amount in the received variable parameter. Then, the processor 100 may generate a new source program by using the changed variable parameter and the fixed parameter. In addition, the processor 100 may repeatedly perform the process of converting a new source program into an object program, measuring the performance of the object program, and outputting the changed variable parameter and the amount of change in the variable parameter based on the measured performance.

As described above, the processor 100 measures the performance of the program, outputs the variable parameter change amount, and repeatedly performs the process of changing the variable parameter according to the output variable parameter change amount, so that the performance is higher than that of other source programs. A good optimized source program 120 may be generated and/or determined. With such a configuration, the processor 100 automatically changes the variable parameters without requiring the programmer of the program to directly change or modify the variable parameters each time for optimizing the performance of the program, and thus the optimized source from the given source program 110 . The program 120 may be created.

2 is a block diagram illustrating an internal configuration of the processor 100 according to an embodiment of the present disclosure. As shown, the processor 100 may include a program generation unit 210 , a translator 220 , a program execution unit 230 , a performance measurer 240 , a machine learning processing unit 250 , and the like. In addition, the processor 100 may communicate with the database 260 and exchange information and/or data necessary for program performance optimization.

As described above, the processor 100 may receive an input for a source program including fixed parameters and variable parameters. For example, the fixed parameter and the variable parameter may include all information necessary to generate one program. According to an embodiment, the processor 100 may receive fixed parameters and variable parameters from a developer or user of a program, or may extract fixed parameters and variable parameters corresponding to a specific program from the database 260 .

According to an embodiment, the program generator 210 may generate a source program by using the received or extracted fixed parameters and variable parameters. Here, the fixed parameter and the variable parameter may be numerical values and/or codes expressed or implemented in a form for the program generating unit 210 to generate a source program. That is, the fixed parameter and the variable parameter may be embodied in various forms according to the implementation method of the program generator 210 .

The source program generated by the program generator 210 may be transmitted to the translator 220 . As described above, the translator 220 may include, but is not limited to, a compiler, an interpreter, an assembler, a preprocessor, and the like. Such a translator 220 may convert a source program into an object program executable in a computer.

The target program converted or generated by the translator 220 may be provided to the program execution unit 230 . The program execution unit 230 may receive an object program and a test set for executing the object program. For example, a test set may include a test input and a test output (result data) corresponding to the test input. That is, the program execution unit 230 may be configured to input a test input to the target program and output a result value. Also, the program execution unit 230 may compare the output result value with the test output to determine whether the output result value corresponds to a correct answer (ie, test output).

According to the execution result of the target program, the performance meter 240 may measure the performance of the target program. For example, when the result of the target program corresponds to the correct answer, the performance meter 240 may measure the performance of the target program. In another example, when the result of the target program is output, the performance measurer 240 may measure the performance of the target program. Here, the performance may be measured based on the execution time of the target program, energy consumption, resource usage, and the like, but is not limited thereto.

The performance of the target program measured in this way may be provided to the machine learning processing unit 250 . The machine learning processing unit 250 may output a variable parameter change amount using the machine learning model. According to an embodiment, the machine learning processing unit 250 extracts a machine learning model corresponding to the corresponding target program from the database 260, and inputs variable parameters and measured performance to the extracted machine learning model, so that the variable parameter change amount can be printed out. That is, the machine learning processing unit 250 may output the variable parameter change amount so that the performance of the target program can be improved. Then, the processor 100 may regenerate the source program reflecting the output variable parameter change amount.

The processor 100 generates a source program, converts the source program into an object program, measures the performance of the object program, outputs a variable parameter change amount, and repeatedly performs the process of regenerating the source program, Performance can be improved gradually. This can be seen as a program optimization process based on reinforcement learning. Then, when the performance of the program reaches the target performance, the source program corresponding to the highest performance among a plurality of programs generated until the target performance is reached may be determined as the optimized source program.

According to an embodiment, the database 260 may be included in one device and directly connected to the processor 100 . According to another embodiment, the database 260 may be arranged to be communicable by the processor 100 (eg, a cloud system).

In FIG. 2 , the configuration of the processor 100 has been described separately for each function, but it does not necessarily mean that they are physically separated. For example, the program execution unit 230 may be configured to include the translator 220 and/or the performance meter 240 . In addition, although it is illustrated that one database 260 exists in FIG. 2 , the present invention is not limited thereto, and for example, a database in which fixed parameters and/or variable parameters are stored and a database in which one or more machine learning models are stored may exist. . With such a configuration, since the processor 100 can replace the role of an experienced programmer, the program optimization can be completed in a shorter time than the programmer directly optimizing, or a larger number of search ranges (that is, a larger number of search ranges at a given time) , part or all included in the program) can be searched efficiently.

3 is a flowchart illustrating an example of a method 300 for automatically optimizing a program based on reinforcement learning according to an embodiment of the present disclosure. According to an embodiment, the reinforcement learning-based program optimization method 300 may be performed by at least one processor (eg, the processor 100) for performing program optimization. The method 300 for automatically optimizing a program based on reinforcement learning may be initiated by a processor receiving an input for a source program including fixed parameters and variable parameters (S310). For example, the source program may refer to a program written in a source language.

The processor may generate a source program based on the received input (S320). Also, the processor may convert the source program into the target program (S330). For example, the target program may refer to a program written in an object language or a target language, and described to be executed by a computer. For example, the processor may generate the target program using a translator corresponding to the source program.

The processor may measure the performance of the executed object program by executing the converted object program (S340). According to an embodiment, the processor may measure a time taken to output a result value through the target program when the target program is executed, and determine the measured time as an index of performance. In this case, the processor may input a test set for the target program, and determine a time taken for a result value corresponding to the test set to be output through the target program as an index of performance of the target program.

Then, the processor may input the variable parameter and the measured performance to the machine learning model, and output the variable parameter change amount (S350). Additionally or alternatively, the processor may input the variable parameter, the measured performance, and the source program to the machine learning model, and output the variable parameter variation. According to an embodiment, the machine learning model may include a deep learning model such as a deep neural network. When the machine learning model is a deep learning model, the processor generates an input vector representing variable parameters, performance, and source program, and inputs the generated input vector to the deep learning model to obtain an output vector representing variable parameter variation through the deep learning model. can create

The processor may regenerate the source program reflecting the variable parameter change amount (S360). That is, the processor may apply the variable parameter change amount to the variable parameter and output the regenerated source program using the variable parameter and the fixed parameter to which the variable parameter change amount is applied.

4 is a block diagram illustrating an example in which a process for determining an optimized program is repeatedly performed according to an embodiment of the present disclosure. As illustrated, the program generator 210 may generate a source program using fixed parameters and variable parameters. For example, the fixed parameter and the variable parameter may include at least some information necessary to generate one program.

The translator 220 may convert the source program generated by the program generator 210 into an object program. In this way, the target program converted by the translator 220 may be provided to the program execution unit 230 . In this case, the program execution unit 230 may check a result output by executing the target program. Also, the performance meter 240 may measure the performance of the target program according to the execution of the target program. In addition, the machine learning processing unit 250 may input the variable parameter and the measured performance to the machine learning model, and output the variable parameter change amount. In this way, the process/method of generating the source program, converting the source program into the object program, executing the object program, measuring the performance of the object program, and outputting the variable parameter change amount is performed the same/similar to that described above. can be

According to an embodiment, in terms of learning the machine learning model, the above-described process may be repeatedly performed until a predetermined batch size is reached. Here, the predetermined batch size may indicate the size or number of data sets required to train the entire machine learning model. That is, a plurality of variable parameter variations, a plurality of performances, source codes of a plurality of programs, etc. generated by repeating the above-described process may be stored in a specific memory and/or a database, and the stored data may be stored in a plurality of variable parameter variations, a plurality of When the performance of , the source code of a plurality of programs, etc. reach a predetermined batch size, the stored data and/or information may be used as training data for training the machine learning model. That is, the result obtained by inferring the machine learning model may be used to train the machine learning model again. For example, a result obtained by inferring a machine learning model may be generated in a plurality of local inference environments and transmitted to a global machine learning model (eg, a DNN model) to be used for reinforcement learning.

In terms of generation (inference) of the optimized source program, the above-described process can be repeatedly executed until the target performance is reached. According to an embodiment, the target performance may refer to a performance when the improved performance is no longer measured when the above-described process is repeatedly performed. For example, after a performance-related numerical value is output in 1.6 seconds, if only a performance value of 1.6 seconds or less is output a predetermined number of times (for example, once or multiple times) in the repeated process, the processor sets the target performance can be considered to have been reached. When it is determined that the target performance is reached, the processor may determine a source program corresponding to the highest performance as an optimized source program. In the above-described example, the source program generated based on the variable parameter and the fixed parameter when the performance value is 1.6 seconds may be determined as the optimized source program. When program optimization is performed by such a configuration, the amount of human resource input required for program optimization can be minimized to generate profits, and service quality can be improved by using a program with improved performance.

5 is a diagram illustrating an example of a machine learning model 500 that outputs a variable parameter variation based on reinforcement learning according to an embodiment of the present disclosure. According to an embodiment, the machine learning model 500 may correspond to an artificial neural network. For example, the artificial neural network may be configured as a deep neural network (DNN). Here, the artificial neural network may refer to a statistical learning algorithm implemented based on the structure of a biological neural network or a structure for executing the algorithm in machine learning technology and cognitive science. In other words, in an artificial neural network, as in a biological neural network, an error between a correct output corresponding to a specific input and an inferred output by repeatedly adjusting the weight of a synapse by nodes, which are artificial neurons that form a network by combining synapses, as in a biological neural network. By learning to reduce , represents the machine learning model 500 having problem-solving ability.

In general, an artificial neural network is implemented as a multilayer perceptron (MLP) consisting of multiple layers of nodes and connections between them. The artificial neural network according to the present embodiment may be implemented using one of various artificial neural network structures including MLP. An artificial neural network is located between an input layer that receives an input signal or data from the outside, an output layer that outputs an output signal or data corresponding to input data, and an input layer and an output layer. It consists of n hidden layers. Here, the output layer receives a signal from the hidden layer and outputs it to the outside.

The processor may input the variable parameters and the performance of the program to the machine learning model 500 to output the variable parameter variation. According to an embodiment, the processor generates an input vector representing the variable parameter and the measured performance, and inputs the generated input vector to the machine learning model 500 to output the variable parameter change amount through the machine learning model 500 . You can create vectors. Here, the variable parameter change amount may be an arbitrary value output to improve the performance of a given program. That is, the processor may apply the variable parameter change amount to the existing variable parameter and output a new source program using the variable parameter and the fixed parameter to which the variable parameter change amount is applied.

6 is a diagram illustrating an example of a machine learning model 600 for outputting a variable parameter variation based on reinforcement learning according to another embodiment of the present disclosure. According to an embodiment, the machine learning model 600 may correspond to an artificial neural network. For example, the machine learning model 600 may include an input layer, a hidden layer, an output layer, etc. implemented as a multi-layer perceptron, and may include, for example, a deep neural network (DNN).

According to an embodiment, the machine learning model 600 may be a model trained (eg, reinforcement learning) to output a variable parameter change amount using a variable parameter, program performance, a source program, and the like. In this case, the variable parameter variation output by the machine learning model 600 from the variable parameters, program performance, source program, etc. may be used as new learning data for learning the machine learning model 600 .

According to an embodiment, the processor may input the variable parameter, the performance of the program, and the source program to the machine learning model 600 to output the variable parameter change amount. For example, the processor generates an input vector representing the variable parameter, the measured performance, and the source program, and inputs the generated input vector to the machine learning model 600 to represent the variable parameter variation through the machine learning model 600 . You can create an output vector. That is, the processor may apply the output vector indicating the amount of change of the variable parameter to the existing variable parameter, and output a new source program by using the variable parameter and the fixed parameter to which the output vector indicating the amount of change of the variable parameter is applied.

7 is a flowchart illustrating an example of a method 700 for training a reinforcement learning-based machine learning model according to an embodiment of the present disclosure. According to an embodiment, the method 700 for training a reinforcement learning-based machine learning model may be performed by at least one processor (eg, the processor 100 ) for performing program optimization or generating training data. can The method 700 may be initiated when the processor converts a source program into an object program ( S710 ). The source program may be generated using fixed parameters and variable parameters including all information necessary for generating the corresponding program. Here, the variable parameter may be received from a programmer or a database. Alternatively, the variable parameter may refer to a parameter in which a variable parameter variation amount is applied to an existing variable parameter.

The processor may measure the performance of the executed object program by executing the converted object program (S720). According to an embodiment, the processor may receive a predefined test set (test input and test output) according to a target program. Then, the processor may input a test input to the target program and check the output result value. That is, the processor may check whether the output result value and the test output are the same, and measure the performance of the target program. For example, when the output result value and the test output are not the same, since the modified source program is different from the existing source program, the processor may initialize the variable parameter to a default value.

The processor may input the variable parameter and the measured performance to the machine learning model and output the variable parameter change amount (S730). Also, the processor may regenerate the source program reflecting the variable parameter change amount ( S740 ). For example, the processor may output the variable parameter change amount so that the performance of the program can be improved in consideration of the degree to which the variable parameter affects the performance of the program. Then, the source program may be regenerated using the variable parameter and the fixed parameter to which the variable parameter change amount is reflected.

The processor may set the regenerated source program as the source program and perform the above-described process/step again (S750). According to an embodiment, the processor may repeatedly perform the above-described steps until the cumulative size or the number of executions of the data used for learning reaches a predetermined batch size ( S760 ). That is, the machine learning model for outputting the variable parameter variation can be continuously learned, and the training data required for learning can be generated or updated by repeatedly performing the above-described steps until a predetermined batch size is reached.

According to an embodiment, all intermediate results and data generated by repeating the above-described process may be stored in an arbitrary memory or in an arbitrary external storage device. In other words, variable parameter variation, variable parameter, fixed parameter, source program, target program (eg, binary code), test input, test output, The resulting values and/or performance may be stored in any memory or external storage device. The data and/or information stored or collected in this way may be used as training data for reinforcement learning (eg, global DNN) of other machine learning models.

8 is a flowchart illustrating an example of an inference method 800 through a machine learning model according to an embodiment of the present disclosure. According to an embodiment, the inference method 800 of the machine learning model may be performed by at least one processor (eg, the processor 100 ) for performing program optimization. The inference method 800 of the machine learning model may be started when the processor receives an input for a source program including a variable parameter and a fixed parameter ( S810 ). For example, the variable parameters and the fixed parameters may be directly received from a program developer or may be extracted from a database in which the variable parameters and the fixed parameters are stored.

The processor may generate a source program based on the received input (S820). Also, the processor may convert the source program into the target program (S830). Here, the process of generating the source program based on the fixed parameter and the variable parameter or converting the generated source program into the target program may be performed in the same manner as/in the same manner as described above.

The processor may measure the performance of the executed object program by executing the converted object program (S840). For example, the execution time, energy consumption, and/or resource usage required for the target program to output a result value from input may be used as a performance indicator for measuring the performance of the program. That is, the shorter the execution time, the lower the energy consumption, and/or the lower the resource usage, the better the performance of the program may be determined.

The processor may determine whether the measured performance of the program is higher than the previous best performance (S850). In other words, the processor may determine whether the performance of the program has reached a target performance. For example, the processor may determine that the performance of the corresponding program has reached the target performance when the measured program performance is no longer improved while at least a part of the above-described process is repeatedly performed.

According to an embodiment, when it is determined that the measured performance is higher than the previous best performance, the processor may input the variable parameter and the measured performance to the machine learning model, and output the variable parameter change amount ( S860 ). In addition, the processor may regenerate the source program reflecting the variable parameter change amount (S870). That is, the processor may continuously change the variable parameter and measure the change in program performance according to the change of the variable parameter.

According to an embodiment, when the performance of the program reaches the target performance, the processor may determine the source program corresponding to the highest performance as the optimized source program ( S880 ). That is, even if all or part of the above-described process is repeatedly performed, if the performance of the program does not improve any more than the previous best performance, the processor returns the source program corresponding to the highest performance with the best performance so far. A program having variable parameters corresponding to performance) may be determined as an optimized source program. With such a configuration, it is possible to automatically generate an optimized program by repeating performance measurement until the processor reaches the target performance without optimizing the program by directly changing variable parameters by the developer of the program. In FIG. 8 , if the performance of the program is not improved once from the previous best performance through step S850, a program corresponding to the previous best performance is generated as an optimized program, but the present invention is not limited thereto, and a plurality of predetermined When the performance of the program measured through the repeated process of S830 to S850 by the number of times does not exceed the previous best performance, the program corresponding to the previous best performance may be determined as the optimized program.

9 is a diagram illustrating an example of performing reinforcement learning on the machine learning models 910_1 and 910_2 according to an embodiment of the present disclosure. For example, FIG. 9 may show a structure for learning the global DNN 940 and the machine learning models 910_1 and 910_2 using the reinforcement learning technique of the Asynchronous Advantage Actor-Critic (A3C) structure. Here, reinforcement learning is a process in which a DNN-based agent intervenes in the external environment (Action) to receive a result (State) and a reward (Reward), and repeats the process of learning the DNN from the result and reward again. It can refer to the method of execution. According to an embodiment, there may be one or more machine learning models 910_1 and 910_2 that measure the performance of a program and output a change amount of a variable parameter.

As described above, the machine learning models 910_1 and 910_2 repeat the process until a predetermined batch size is reached, and are generated variable parameter variation, variable parameter, fixed parameter, source program, and target program (eg, program binary), test inputs, test outputs, result values, performance, etc. can be stored in any memory or external storage device. As such, data and/or information stored in an arbitrary memory or external storage device may be provided to the global DNN manager 930 as inference results 920_1 and 920_2 . For example, the inference results 920_1 and 920_2 may be asynchronously provided to the global DNN manager 930 .

The global DNN manager 930 receiving the inference results 920_1 and 920_2 may generate training data for learning the global DNN using the inference results 920_1 and 920_2 . Then, the global DNN manager 930 may transmit the generated training data to the global DNN 940 . That is, the global DNN 940 may perform reinforcement learning by using all of the inference results 920_1 and 920_2 generated by one or more machine learning models 910_1 and 910_2 .

The global DNN 940 may generate DNN parameters 950_1 and 950_2 as a result of reinforcement learning after performing reinforcement learning using training data. As such, the generated DNN parameters 950_1 and 950_2 may be provided to one or more machine learning models 910_1 and 910_2. In this case, parameters of the machine learning models 910_1 and 910_2 may be substituted with the received DNN parameters 950_1 and 950_2, and accordingly, reinforcement learning of the machine learning models 910_1 and 910_2 may be performed.

Although it is illustrated that two machine learning models 910_1 and 910_2 exist in FIG. 9 , the present invention is not limited thereto, and three or more machine learning models may exist. In addition, although it has been described above that reinforcement learning is performed using the A3C model in FIG. 9 , the present invention is not limited thereto, and any Artor-Critic model, Deep Q Networks (DQN) model, or the like may be used.

The above-described method may be provided as a computer program stored in a computer-readable recording medium for execution by a computer. The medium may be to continuously store a computer executable program, or to temporarily store it for execution or download. In addition, the medium may be various recording means or storage means in the form of a single or several hardware combined, it is not limited to a medium directly connected to any computer system, and may exist distributed on a network. Examples of the medium include a hard disk, a magnetic medium such as a floppy disk and a magnetic tape, an optical recording medium such as CD-ROM and DVD, a magneto-optical medium such as a floppy disk, and those configured to store program instructions, including ROM, RAM, flash memory, and the like. In addition, examples of other media may include recording media or storage media managed by an app store that distributes applications, sites that supply or distribute various other software, and servers.

The method, operation, or techniques of this disclosure may be implemented by various means. For example, these techniques may be implemented in hardware, firmware, software, or a combination thereof. Those of ordinary skill in the art will appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design requirements imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementations should not be interpreted as causing a departure from the scope of the present disclosure.

In a hardware implementation, the processing units used to perform the techniques include one or more ASICs, DSPs, digital signal processing devices (DSPDs), programmable logic devices (PLDs). ), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, electronic devices, other electronic units designed to perform the functions described in this disclosure. , a computer, or a combination thereof.

Accordingly, the various illustrative logic blocks, modules, and circuits described in connection with this disclosure are suitable for use in general-purpose processors, DSPs, ASICs, FPGAs or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or the present disclosure. It may be implemented or performed in any combination of those designed to perform the functions described in A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, eg, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other configuration.

In firmware and/or software implementations, the techniques may include random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), PROM ( on computer readable media such as programmable read-only memory), erasable programmable read-only memory (EPROM), electrically erasable PROM (EEPROM), flash memory, compact disc (CD), magnetic or optical data storage devices, etc. It may be implemented with stored instructions. The instructions may be executable by one or more processors and may cause the processor(s) to perform certain aspects of the functionality described in this disclosure.

Although the embodiments described above have been described utilizing aspects of the presently disclosed subject matter in one or more standalone computer systems, the present disclosure is not so limited and may be implemented in connection with any computing environment, such as a network or distributed computing environment. . Still further, aspects of the subject matter in this disclosure may be implemented in a plurality of processing chips or devices, and storage may be similarly affected across the plurality of devices. Such devices may include PCs, network servers, and portable devices.

Although the present disclosure has been described in connection with some embodiments herein, various modifications and changes may be made without departing from the scope of the present disclosure that can be understood by those skilled in the art to which the present disclosure pertains. Further, such modifications and variations are intended to fall within the scope of the claims appended hereto.

100: processor 110: source program
120: optimized source program

Claims (10)

A method for automatically optimizing a program based on reinforcement learning, the method comprising:
(a) receiving an input for a source program comprising fixed parameters and variable parameters;
(b) generating the source program based on the received input;
(c) converting the source program into an object program;
(d) measuring the performance of the executed object program by executing the converted object program; and
(e) inputting the variable parameter and the measured performance into a machine learning model, and outputting a variable parameter change amount; and
(f) regenerating the source program reflecting the variable parameter change amount
A method for automatically optimizing a program based on reinforcement learning, comprising:
According to claim 1,
The step (f) is,
applying the variable parameter change amount to the variable parameter; and
outputting the regenerated source program using the variable parameter and the fixed parameter to which the variable parameter change amount is applied;
A method for automatically optimizing a program based on reinforcement learning, comprising:
3. The method of claim 2,
After step (f),
(g) setting the regenerated source program as the source program, and performing the steps (c) to (f) again. The method of automatically optimizing a program based on reinforcement learning.
4. The method of claim 3,
and training the machine learning model by repeatedly performing step (g) until a predetermined batch size is reached.
4. The method of claim 3,
executing steps (c) to (f), but repeatedly executing at least a portion of steps (c) to (f) until a target performance is reached; and
determining a source program corresponding to the highest performance as an optimized source program when the target performance is reached
A method for automatically optimizing a program based on reinforcement learning, further comprising:
According to claim 1,
Step (e) is,
and outputting the variable parameter variation by inputting the variable parameter, the measured performance, and the source program to the machine learning model.
7. The method of claim 6,
The machine learning model includes a deep learning model,
The step of inputting the variable parameter, the measured performance, and the source program to the machine learning model and outputting the variable parameter change amount,
generating an input vector representing the variable parameter, the measured performance and the source program; and
inputting the generated input vector to the deep learning model to generate an output vector representing the variable parameter change amount through the deep learning model
A method for automatically optimizing a program based on reinforcement learning, comprising:
According to claim 1,
Step (d) is,
measuring a time taken to output a result value through the target program when the target program is executed; and
determining the measured time as an indicator of the performance;
A method for automatically optimizing a program based on reinforcement learning, comprising:
9. The method of claim 8,
Measuring the time it takes to output a result value through the target program when the target program is executed includes:
inputting a test set for the target program; and
determining, as an indicator of the performance, a time taken for a result value corresponding to the test set to be output through the target program;
A method for automatically optimizing a program based on reinforcement learning, comprising:
A computer program stored in a computer-readable recording medium for executing on a computer the method for automatically optimizing a program based on reinforcement learning according to any one of claims 1 to 9.

Images