CN112926629A - Hyper-parameter determination method, device, deep reinforcement learning framework, medium and equipment - Google Patents
Hyper-parameter determination method, device, deep reinforcement learning framework, medium and equipment Download PDFInfo
- Publication number
- CN112926629A CN112926629A CN202110129788.6A CN202110129788A CN112926629A CN 112926629 A CN112926629 A CN 112926629A CN 202110129788 A CN202110129788 A CN 202110129788A CN 112926629 A CN112926629 A CN 112926629A
- Authority
- CN
- China
- Prior art keywords
- value
- target
- parameter
- hyper
- sampling
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 64
- 230000002787 reinforcement Effects 0.000 title claims abstract description 30
- 238000005070 sampling Methods 0.000 claims abstract description 145
- 230000002452 interceptive effect Effects 0.000 claims abstract description 33
- 230000003993 interaction Effects 0.000 claims description 38
- 230000009471 action Effects 0.000 claims description 34
- 238000005457 optimization Methods 0.000 claims description 26
- 238000012549 training Methods 0.000 claims description 21
- 230000008569 process Effects 0.000 claims description 17
- 238000004590 computer program Methods 0.000 claims description 16
- 238000012545 processing Methods 0.000 claims description 15
- 238000009826 distribution Methods 0.000 claims description 11
- 230000006870 function Effects 0.000 description 21
- 238000010586 diagram Methods 0.000 description 9
- 238000004891 communication Methods 0.000 description 6
- 230000003287 optical effect Effects 0.000 description 6
- 238000013473 artificial intelligence Methods 0.000 description 4
- 230000001186 cumulative effect Effects 0.000 description 3
- 238000013528 artificial neural network Methods 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 239000003795 chemical substances by application Substances 0.000 description 2
- 238000013135 deep learning Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000007613 environmental effect Effects 0.000 description 2
- 238000000605 extraction Methods 0.000 description 2
- 239000013307 optical fiber Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 241001465754 Metazoa Species 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000003062 neural network model Methods 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 238000009827 uniform distribution Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Software Systems (AREA)
- Artificial Intelligence (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Computation (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Computing Systems (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Evolutionary Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Medical Informatics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Image Analysis (AREA)
Abstract
The disclosure relates to a hyper-parameter determination method, a hyper-parameter determination device, a deep reinforcement learning framework, a medium and equipment, wherein the method comprises the following steps: acquiring a sampling sample corresponding to a sampling value under the sampling value of a target hyper-parameter of a target model; generating an interactive sample corresponding to the target hyper-parameter according to the sampling sample, wherein the interactive sample comprises the sampling value and an optimized characteristic parameter corresponding to the target model; updating the state value corresponding to the target hyper-parameter according to the interactive sample, wherein the parameter space of the target hyper-parameter is discretized into a plurality of value areas; determining a target area from the plurality of value areas according to the updated state value corresponding to the target hyper-parameter; and determining the target value of the target hyper-parameter according to the target area. Therefore, the value of the hyper-parameter of the model can be accurately set, and the phenomenon that the target model cannot be converged or the convergence speed is too low due to the fact that the set value of the hyper-parameter is not appropriate due to the limitation of human experience is avoided.
Description
Technical Field
The disclosure relates to the field of computers, in particular to a hyper-parameter determination method, a hyper-parameter determination device, a deep reinforcement learning framework, a medium and equipment.
Background
With the development of random computer technology, various large models and complex machine learning models are gradually applied. A large number of parameters are needed to be calculated in the model, so that the model can meet the requirements of users. Some parameters in the model can be optimized through the training of the model, such as weights in the neural network model, and some parameters cannot be optimized through the training of the model, and such parameters are hyper-parameters of the model, such as the number of hidden layers in the neural network. The hyper-parameters are used for adjusting the training process of the model, usually set manually by workers based on experience, do not directly participate in the training process of the model, and are not updated in the training process of the model. And the setting of the hyper-parameters has great influence on the iteration times, the convergence efficiency and the like of model training.
Disclosure of Invention
This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.
In a first aspect, the present disclosure provides a method for hyper-parameter determination, the method comprising:
acquiring a sampling sample corresponding to a sampling value under the sampling value of a target hyper-parameter of a target model;
generating an interactive sample corresponding to the target hyper-parameter according to the sampling sample, wherein the interactive sample comprises the sampling value and an optimized characteristic parameter corresponding to the target model;
updating the state value corresponding to the target hyper-parameter according to the interactive sample, wherein the parameter space of the target hyper-parameter is discretized into a plurality of value areas;
determining a target area from the plurality of value areas according to the updated state value corresponding to the target hyper-parameter;
and determining the target value of the target hyper-parameter according to the target area.
In a second aspect, the present disclosure provides a hyper-parameter determination apparatus, the apparatus comprising:
the acquisition module is used for acquiring a sampling sample corresponding to a sampling value under the sampling value of a target hyper-parameter of a target model;
the generating module is used for generating an interactive sample corresponding to the target hyper-parameter according to the sampling sample, wherein the interactive sample comprises the sampling value and an optimized characteristic parameter corresponding to the target model;
the updating module is used for updating the state value corresponding to the target hyper-parameter according to the interaction sample, wherein the parameter space of the target hyper-parameter is discretized into a plurality of value areas;
the first determining module is used for determining a target area from the plurality of value areas according to the updated state value corresponding to the target hyper-parameter;
and the second determination module is used for determining the target value of the target hyper-parameter according to the target area.
In a third aspect, a deep reinforcement learning framework is provided, and a value of a hyper-parameter in the deep reinforcement learning framework is determined based on the hyper-parameter determination method in the first aspect.
In a fourth aspect, a computer-readable medium is provided, on which a computer program is stored which, when being executed by a processing device, carries out the steps of the method of the first aspect.
In a fifth aspect, an electronic device is provided, comprising:
a storage device having a computer program stored thereon;
processing means for executing the computer program in the storage means to carry out the steps of the method of the first aspect.
Therefore, in the above technical solution, the parameter space of the target hyper-parameter is discretized into a plurality of value areas, so that the value of the target hyper-parameter can be determined based on the optimized characteristic parameter of the target model based on the sampling sample corresponding to the target model using the target hyper-parameter. Therefore, by means of the technical scheme, on one hand, the value of the hyper-parameter of the model can be accurately set, and the phenomenon that the target model cannot be converged or the convergence speed is too low due to the fact that the hyper-parameter setting value is not appropriate due to the limitation of human experience is avoided. On the other hand, the value of the target hyperparameter is determined through the sampling sample of the target model and the optimization characteristic parameter, so that the matching degree of the target hyperparameter and the actual application scene of the target model is improved while the value of the target hyperparameter is ensured to be accurate, meanwhile, the target value is determined based on the optimization characteristic parameter of the target model, and a target area is determined based on the condition that the optimization characteristic parameter is better when the target value is determined, so that the training efficiency of the target model can be effectively improved, the iteration times of training of the target model are reduced to a certain extent, and the convergence efficiency of the target model is improved.
Additional features and advantages of the disclosure will be set forth in the detailed description which follows.
Drawings
The above and other features, advantages and aspects of various embodiments of the present disclosure will become more apparent by referring to the following detailed description when taken in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numbers refer to the same or similar elements. It should be understood that the drawings are schematic and that elements and features are not necessarily drawn to scale. In the drawings:
FIG. 1 is a flow chart of a method of hyperparameter determination provided in accordance with an embodiment of the present disclosure;
FIG. 2 is a flow diagram of an exemplary implementation of updating a state value corresponding to a target hyper-parameter based on an interaction sample provided in one embodiment of the present disclosure;
FIG. 3 is a flow diagram of an exemplary implementation of determining a target region from a plurality of value regions according to an updated state value corresponding to a target hyper-parameter provided in an embodiment of the present disclosure;
FIG. 4 is a block diagram of a hyper-parameter determination apparatus provided in accordance with an embodiment of the present disclosure;
FIG. 5 illustrates a schematic diagram of an electronic device suitable for use in implementing embodiments of the present disclosure.
Detailed Description
Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While certain embodiments of the present disclosure are shown in the drawings, it is to be understood that the present disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein, but rather are provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the disclosure are for illustration purposes only and are not intended to limit the scope of the disclosure.
It should be understood that the various steps recited in the method embodiments of the present disclosure may be performed in a different order, and/or performed in parallel. Moreover, method embodiments may include additional steps and/or omit performing the illustrated steps. The scope of the present disclosure is not limited in this respect.
The term "include" and variations thereof as used herein are open-ended, i.e., "including but not limited to". The term "based on" is "based, at least in part, on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions for other terms will be given in the following description.
It should be noted that the terms "first", "second", and the like in the present disclosure are only used for distinguishing different devices, modules or units, and are not used for limiting the order or interdependence relationship of the functions performed by the devices, modules or units.
It is noted that references to "a", "an", and "the" modifications in this disclosure are intended to be illustrative rather than limiting, and that those skilled in the art will recognize that "one or more" may be used unless the context clearly dictates otherwise.
The names of messages or information exchanged between devices in the embodiments of the present disclosure are for illustrative purposes only, and are not intended to limit the scope of the messages or information.
Fig. 1 is a flowchart of a method for determining a hyper-parameter according to an embodiment of the present disclosure, and as shown in fig. 1, the method may include:
in step 11, a sampling sample corresponding to the sampling value is obtained under the sampling value of the target hyper-parameter of the target model.
Illustratively, the target model may be a deep reinforcement learning model. The deep reinforcement learning model combines the perception capability of deep learning and the decision capability of reinforcement learning, obtains a high-dimensional observation through interaction between an agent (agent) and the environment at each moment, and perceives the observation by using a deep learning method to obtain specific state characteristic representation of the observation; then, a value function (state value function) of each state and a value function (action value function) of a state-action pair can be evaluated based on expected returns, and a decision strategy is promoted based on the two value functions, wherein the decision strategy is used for mapping the current state into a corresponding decision action; the environment will react to this decision-making action and get the next observation. The above process is continuously cycled to obtain the optimal strategy for achieving the goal, illustratively, the goal is to maximize the accumulated return.
Therefore, in the application scenario, the sampling sample is an interaction sequence obtained by sampling in a process of interacting a virtual object with a virtual environment, where the virtual object is controlled based on the deep reinforcement learning model, the interaction sequence includes a plurality of sampling data under the sampling value, and each sampling data includes an environment state of the virtual environment, a decision action performed by the virtual object in the environment state determined by the deep reinforcement learning model, and a return value corresponding to the decision action.
The virtual environment may be a virtual scene environment generated by a computer, for example, the virtual environment may be a game scene, and illustratively, multimedia data for interacting with a user is rendered so that the multimedia data can be rendered and displayed as the game scene, the virtual environment provides a multimedia virtual world, and the user can control actions of virtual objects through controls on an operation interface, or directly control virtual objects operable in the virtual environment and observe objects, characters, scenery and the like in the virtual environment from the perspective of the virtual objects, and interact through the virtual objects and other virtual objects and the like in the virtual environment. As another example, the virtual environment may also include other virtual objects in the scene, and the like. The virtual object may be an avatar in a virtual environment for simulating a user, which may be a human or other animal avatar, or the like.
As an example, the virtual object may perform a decision-making action in a first state of the virtual environment, and after the virtual object performs the decision-making action, the virtual environment may react to the decision-making action to obtain a second state of the virtual environment and a return value corresponding to the decision-making action. When sampling is performed during the interaction between the virtual object and the virtual environment, the first state, the decision action, the second state, and the return value may be used as sampling data corresponding to the sampling time, and if not otherwise stated, the environment state in the sampling data in the embodiment of the present disclosure is the first state in the sampling data. In a complete interactive process, the sampling data according to the sequence of the sampling time is formed into an interactive sequence. When sampling is performed, an image of the virtual environment corresponding to the sampling time can be acquired, so that feature extraction can be performed on the image to obtain the first state. After the virtual object executes the decision-making action, sampling to acquire an image of the virtual environment and performing feature extraction on the image to acquire a second state. The reward value may be a change of a score value corresponding to the virtual object after the decision action is executed, or a change of a virtual life bar, and the reward value may be set according to an actual usage scenario, which is not limited by the present disclosure.
In step 12, an interactive sample corresponding to the target hyper-parameter is generated according to the sampling sample, and the interactive sample includes the sampling value and the optimized characteristic parameter corresponding to the target model.
The optimized characteristic parameter may be a cumulative reward corresponding to the interaction sequence. As an example, the accumulated reward may be a sum of reward values corresponding to each decision-making action included in the interaction sequence. As another example, the more distant a decision-making action has an effect on the current decision-making action, the less the accumulated reward is, the accumulated reward may be the accumulated sum of the return value of each decision-making action in the interaction sequence multiplied by the attenuation coefficient corresponding to the decision-making action, wherein the attenuation coefficients corresponding to the decision-making actions decrease in the order of the decision-making actions, for example:
Gt＝Rt+1+γRt+2+γ2Rt+3+…+γn-1Rt+n
＝Rt+1+γ(Rt+2+γRt+3+…+γn-2Rt+n)
＝Rt+1+γGt+1
wherein R isiA return value for the decision action at time i, γ for the attenuation factor, and n for the number of samples in the interaction sequence after time t to the end of the interaction.
Thus, in another embodiment, the accumulated reward may be obtained from the last decision-making action of the interaction sequence by multiplying its reward value by the decay value and adding the reward value of the previous decision-making action until the reward value of the first decision-making action in the interaction sequence is added. Wherein the attenuation value can be set according to the actual usage scenario.
In step 13, the state value corresponding to the target hyper-parameter is updated according to the interaction sample, wherein the parameter space of the target hyper-parameter is discretized into a plurality of value areas.
In the field, when a model is known, an expected accumulated return brought by an arbitrary policy pi can be estimated, a state value function is generally adopted to evaluate the value of a certain state, the value of a certain state can be expressed by the values of all actions in the state, that is, the expected value of the accumulated return based on a state s can be obtained, under the policy, the accumulated return obeys a distribution, and the expected value of the accumulated return at the state is defined as a state value function v(s):
Vπ(s)＝Eπ[Gt|St＝s]
i.e. representing the state S at time t under strategy pitWhen s is the value, G is accumulatedtThe expected value at s. Then in this embodiment the state value at all value areas of the state s can be evaluated based on the state value function. The determination of the accumulated reward is described in detail above, and is not described herein again.
In the deep reinforcement learning model, the calculation of the state value function can be realized through a neural network. Therefore, the environmental state in the sample data can be input into the state value function network, so that the output value of the state value function network, that is, the state value of the state value function corresponding to the environmental state can be obtained.
As an example, the parameter space of the target hyper-parameter is discretized into a plurality of value areas, and the state value corresponding to the target hyper-parameter can be used to represent the accumulated return brought by further selecting the value of the target hyper-parameter from each value area based on a policy in a state where the value of the target hyper-parameter is a sampling value. For example, in the present disclosure, the state value corresponding to the target hyper-parameter may be determined in an iterative update manner, that is, the state value corresponding to the target hyper-parameter is iteratively updated according to the sampling value corresponding to the interaction sample.
As an example, the number of value areas corresponding to the parameter space may be predetermined, and then the parameter space of the target hyper-parameter may be uniformly divided to obtain a plurality of value areas. If the parameter space of the target hyper-parameter is [0,9], the parameter space is divided into 9 value areas, the value area corresponding to the value area a1 is [0,1 ], the value area corresponding to the value area a2 is [1,2 ], and so on, and the description thereof is omitted here. The state value corresponding to the target hyper-parameter can be represented by a vector, that is, the state values corresponding to the 9 value-taking areas are respectively a dimension value in the vector.
In step 14, a target area is determined from the plurality of value areas according to the updated state value corresponding to the target hyper-parameter.
In this step, the accumulated return of each value area selected by the target hyper-parameter can be accurately evaluated by determining the state value corresponding to the target hyper-parameter according to the sampling sample of the target model, so that the target area for determining the value of the target hyper-parameter can be selected according to the evaluation result, and the accuracy of the value of the target hyper-parameter and the consistency of the value of the target hyper-parameter and the actual application process of the target model are ensured.
In step 15, a target value of the target hyper-parameter is determined according to the target area.
As an embodiment, the uniform distribution sampling is performed in the value range corresponding to the target area, and the value obtained by sampling is determined as the target value.
Therefore, in the above technical solution, the parameter space of the target hyper-parameter is discretized into a plurality of value areas, so that the value of the target hyper-parameter can be determined based on the optimized characteristic parameter of the target model based on the sampling sample corresponding to the target model using the target hyper-parameter. Therefore, by means of the technical scheme, on one hand, the value of the hyper-parameter of the model can be accurately set, and the phenomenon that the target model cannot be converged or the convergence speed is too low due to the fact that the hyper-parameter setting value is not appropriate due to the limitation of human experience is avoided. On the other hand, the value of the target hyperparameter is determined through the sampling sample of the target model and the optimization characteristic parameter, so that the matching degree of the target hyperparameter and the actual application scene of the target model is improved while the value of the target hyperparameter is ensured to be accurate, meanwhile, the target value is determined based on the optimization characteristic parameter of the target model, and a target area is determined based on the condition that the optimization characteristic parameter is better when the target value is determined, so that the training efficiency of the target model can be effectively improved, the iteration times of training of the target model are reduced to a certain extent, and the convergence efficiency of the target model is improved.
As described above, the virtual environment may be a game environment, and then sampling may be performed in the process of interacting the virtual object with the virtual environment to obtain interaction data, and the value of the hyper-parameter in the deep reinforcement learning model may be determined based on the above manner in the training process of the deep reinforcement learning model, so that a greater return can be obtained when the deep reinforcement learning model determines the decision-making action of the virtual object, the accuracy of the decision-making action of the virtual object is ensured, the accuracy of virtual character control is improved, and meanwhile, the amount of data and manpower required in the training process can be reduced.
In one possible embodiment, the method may further comprise:
and taking the target value as a new sampling value, and re-executing the step of obtaining the interactive sample corresponding to the target hyper-parameter under the sampling value of the target hyper-parameter of the target model to the step of determining the target value of the target hyper-parameter according to the target area until the training of the target model is finished.
In this embodiment, after the target value of the target hyper-parameter is determined, the value of the target hyper-parameter in the target model may be updated to the target value, so that steps 11 to 15 may be performed again, to further determine whether the target value is accurate for an interactive sample of the target hyper-parameter of the target model under the target value, thereby achieving dynamic adjustment of the value of the target hyper-parameter, and simultaneously optimizing an optimized characteristic parameter corresponding to the target model, while further improving the accuracy of the value of the target hyper-parameter, ensuring convergence and accuracy of the target model, and improving training efficiency of the target model.
In a possible embodiment, in step 13, according to the interaction sample, an exemplary implementation manner of updating the state value corresponding to the target hyper-parameter is as follows, as shown in fig. 2, and this step may include:
in step 21, a value area to which the sample value belongs is determined according to the sample value.
For example, the value area to which the sampling value belongs may be determined based on the range length corresponding to each value area, and as described above, the subscript i of the value area to which the sampling value belongs may be determined by the following formula:
i＝(min(max(x,l),r)-l)//acc
wherein x is used to represent the sampling value; l is used to represent the left boundary of the parameter space; r is used to represent the right boundary of the parameter space; // for integer division symbols; the acc is used for representing the range length of the value area.
In step 22, a value area to be updated is determined according to the value area to which the sampling value belongs.
As an example, the value area to which the sampling value belongs may be directly determined as the value area to be updated; as another example, the value area to which the sampling value belongs and the value area within a preset range adjacent to the value area to which the sampling value belongs may be determined as the value area to be updated. The preset range can be quantity information of the value area, and the adjacent preset ranges can be used for representing the sum of the preset range on the left side of the current value area and the preset range on the right side of the current value area. For example, if the value area to which the determined sampling value belongs is a2, and the preset range is one value area, the value area a2, one value area a1 on the left side of the value area a2, and one value area A3 on the right side of the value area may be determined as the value area to be updated.
In step 23, the state value of the value area to be updated is updated according to the optimized characteristic parameters.
In this embodiment, a value area to be updated corresponding to the sampling value can be determined by a relationship between the sampling value and the parameter space, so that a state value of the value area to be updated can be updated, and for other value areas except the value area to be updated, the corresponding state value does not need to be updated, so that accuracy of the state value corresponding to the target hyper-parameter can be ensured, and data support is provided for subsequently and accurately selecting the target area.
In a possible embodiment, the value area to which the sampling value belongs is determined as the value area to be updated, and then in step 23, the state value of the value area to be updated can be updated according to the optimized characteristic parameter by the following formula:
wherein T is used to represent the optimization feature parameter, and if the optimization feature parameter is cumulative reward, that is, the cumulative reward is optimized in the direction of increasing during optimization, T may be GtIf the optimization characteristic parameter is error rate of the target model, namely the error rate is optimized in the direction of reducing during optimization, T can be-error; k(s) is used to indicate the number of hits of the value area s to be updated, that is, the number of times that the value of the target hyper-parameter corresponding to the sampling sample belongs to the value area s to be updated, V(s) is used to indicate the current state value of the value area s to be updated, and V'(s) is used to indicate the state value of the value area s to be updated after updating.
In another possible embodiment, as in the above-described embodiments, the value area to which the sampling value belongs and the value area within the preset range adjacent to the value area to which the sampling value belongs are determined as the value area to be updated. Correspondingly, in step 23, according to the optimized feature parameter, an exemplary implementation manner of updating the state value of the value area to be updated is as follows, and the step may include:
and updating the state value of each value area to be updated according to the optimization characteristic parameters and the state values of the value areas to be updated.
For example, the updated state value V of the jth value-to-be-updated area can be determined by the following formulaj’：
Wherein, width is used for representing the number of value areas in a preset range; vjThe state value of the jth value area to be updated is represented; lr is used to indicate a learning rate for updating the state value; i is used to indicate the subscript of the value area to which the sample value belongs. As in the above example, width is 1, the value area to which the determined sampling value belongs is a2, that is, i is 2, j may be 1,2, or 3, and the state values of the value areas a1, a2, and A3 may be updated according to the state values of the value areas a1, a2, and A3, respectively, so as to obtain V1’、V2’、V3’。
In this embodiment, when the state value corresponding to the target hyper-parameter is updated, the mutual influence between the adjacent value areas is considered, and the state values of the value area to which the first value belongs and the adjacent value areas can be updated at the same time, so that an error caused by inaccurate calculation of a single value area can be effectively avoided, and the influence on optimization of the target model is avoided.
In a possible implementation manner, in step 14, according to the updated state value corresponding to the target hyper-parameter, an exemplary implementation manner of determining the target area from the multiple value areas is as follows, as shown in fig. 3, and this step may include:
in step 31, a target score of each value area is determined according to the updated state value corresponding to the target hyper-parameter, wherein the target score is used for representing the reliability degree of selecting the value area.
In a possible embodiment, an exemplary implementation manner of determining the target score of each value area according to the updated state value corresponding to the target hyper-parameter is as follows, and the step may include:
in the updated state value corresponding to the target hyper-parameter, for each value area, determining the average value of the state value of the value area and the state value of the value area in a preset range adjacent to the value area as the value score of the value area, and then determining the value score as the target score.
For example, as described above, in step 13, the state value of the value area to be updated may be updated to update the state value corresponding to the target hyper-parameter. Illustratively, the state value corresponding to the updated target hyper-parameter is represented as { V }1’,V2’,…,VN', wherein N is the total number of the value area.
Accordingly, for a value range i (i ═ 1,2, …, N), the value score S of the value range can be determined by the following formulai：
In order to improve efficiency of the value determination of the hyper-parameter, when the number of the interaction samples reaches a preset threshold, for each interaction sample, a step of updating a state value corresponding to the target hyper-parameter according to the interaction sample may be performed, where values of the target hyper-parameter corresponding to each interaction sample are taken in different value areas. That is, the state value corresponding to the target hyper-parameter may be updated based on a plurality of interaction samples, and the updating manner based on each interaction sample is the same as that described above, and is not described herein again.
In this embodiment, after the state value corresponding to the target hyper-parameter is updated, the score corresponding to each value area may be recalculated, and meanwhile, when the target score of a value area is determined, the value score of the value area may be determined in combination with the state value of an adjacent value area, so that the accuracy of the score of each value area may be ensured, and meanwhile, the smooth correlation among the scores of multiple value areas may be ensured, thereby providing accurate data support for determining the target area.
In another possible embodiment, an exemplary implementation manner of determining the target score of each value area according to the updated state value corresponding to the target hyper-parameter is as follows, and the step may include:
and in the updated state value corresponding to the target hyper-parameter, determining the average value of the state value of the value area and the state value of the value area in a preset range adjacent to the value area as the value score of the value area aiming at each value area. The manner in which the value score is determined is described in detail above.
And then, aiming at each value area, determining a target score of the value area according to the value score of the value area and the hit frequency of the value area.
Illustratively, the target Score of the value area i may be determined by the following formulai：
Wherein c is a preset constant and is used for adjusting the influence of the hit times on the target score, and Mi is the hit times of the value area i.
As described above, in the embodiment of the present disclosure, the state value of the value area may be determined in the iterative update manner, the state value of each value area is initially 0, and for each value area, after the value area to be updated is determined according to the sampling value of the target hyper-parameter, the state value of the corresponding value area to be updated is updated, and the state values of other value areas except the value area to be updated are kept unchanged. Therefore, in this embodiment, in order to improve the diversity of target region selection in the initial training process, the number of hits in the value region needs to be considered when determining the score of the value region, so as to reduce the influence degree of the values hit historically on the selection of the target region. Therefore, in the process, as the number of interactive samples increases, the state value corresponding to the target hyper-parameter is more accurate, and as the number of hits increases, the influence of the number of hits on the target score is gradually reduced, so that the diversity and the exploration space of target area selection can be improved in the initial learning stage, the accuracy of the determined target value can be improved to a certain extent, the excessive influence of random samples in the initial state is avoided, and when the state value is accurate, the influence of the number of hits on the target area selection is reduced, so that the forward optimization adjustment of the target area selection on the optimization characteristic parameters is ensured.
In another embodiment, the target Score of the value area i may be determined by the following formulai：
Wherein, mu ({ V)j'}j＝1,2,...,(r-l)//acc) And σ ({ V)j'}j＝1,2,...,(r-l)//acc) And respectively representing the mean value and the standard deviation corresponding to the updated state value of each value area, namely normalizing the value score of each value area through the formula so as to balance the influence of the value score and the hit frequency on the target score.
Turning back to fig. 3, in step 32, a target region is determined from the plurality of value regions based on the target score for each value region.
In a possible embodiment, the step of determining a target area from the plurality of value areas according to the target score of each value area may include:
and determining the value area with the maximum target score as the target area.
In the embodiment of the disclosure, the value area with the largest target score can be directly selected as the target area, so that effective adjustment of the target value determined from the target area to the optimization of the target model can be effectively ensured, and the efficiency of the optimization of the target model is improved.
In another possible embodiment, the step of determining a target area from the plurality of value areas according to the target score of each value area may include:
and performing softmax processing on the target scores corresponding to the plurality of value areas to obtain probability distribution formed by probability information of the plurality of value areas, sampling the plurality of value areas according to the probability distribution, and determining the value areas obtained by sampling as the target areas.
In this embodiment, in order to further improve the diversity of the target hyper-parameter value exploration, the state values of the value areas may be mapped based on a softmax function, so as to map the state values into values in the range of 0 to 1, which are used as probability information of the value areas, so as to obtain probability distributions of the value areas. When sampling is carried out based on probability distribution, the sampling possibility is also realized in the value area with smaller probability information, so that the possibility that a plurality of value areas are sampled can be ensured to a certain extent, the problem that the determined target area is in the local optimal parameter for the characteristic optimization parameter is avoided, the training of the target model is prevented from being stopped due to the fact that the training reaches the local optimal parameter, and the accuracy and the robustness of the training of the target model can be ensured.
The disclosure also provides a deep reinforcement learning framework, and the value of the hyper-parameter in the deep reinforcement learning framework is determined based on the hyper-parameter determination method. By way of example, the game artificial intelligence can be trained based on the deep reinforcement learning framework, and then through the technical scheme, the accuracy of the determined game artificial intelligence decision can be guaranteed, the decision-making capability of the game artificial intelligence is improved when the game artificial intelligence interacts with a user, and the user interaction experience is improved.
The present disclosure also provides a hyper-parameter determination apparatus, as shown in fig. 4, the apparatus 10 comprising:
an obtaining module 100, configured to obtain a sampling sample corresponding to a sampling value of a target hyper-parameter of a target model;
a generating module 200, configured to generate an interactive sample corresponding to the target hyper-parameter according to the sampling sample, where the interactive sample includes the sampling value and an optimized feature parameter corresponding to the target model;
an updating module 300, configured to update a state value corresponding to the target hyper-parameter according to the interaction sample, where a parameter space of the target hyper-parameter is discretized into a plurality of value areas;
a first determining module 400, configured to determine a target region from the multiple value regions according to the updated state value corresponding to the target hyper-parameter;
a second determining module 500, configured to determine a target value of the target hyper-parameter according to the target area.
Optionally, the apparatus further comprises:
the trigger module is used for taking the target value as a new sampling value, triggering the acquisition module to acquire a sampling sample corresponding to the sampling value under the sampling value of a target hyper-parameter of a target model, generating an interactive sample corresponding to the target hyper-parameter according to the sampling sample, updating a state value corresponding to the target hyper-parameter by the updating module according to the interactive sample, determining a target area from the plurality of value areas according to the updated state value corresponding to the target hyper-parameter by the first determination module, and determining the target value of the target hyper-parameter according to the target area by the second determination module until the training of the target model is completed.
Optionally, the update module includes:
the first determining submodule is used for determining a value area to which the sampling value belongs according to the sampling value;
the second determination submodule is used for determining a value area to be updated according to the value area to which the sampling value belongs;
and the updating submodule is used for updating the state value of the value area to be updated according to the optimization characteristic parameter.
Optionally, the second determining sub-module includes:
a third determining submodule, configured to determine, as the to-be-updated value area, a value area to which the sample value belongs and a value area within a preset range adjacent to the value area to which the sample value belongs;
and the updating submodule is used for respectively updating the state value of each value area to be updated according to the optimization characteristic parameter and the state value of each value area to be updated.
Optionally, the first determining module includes:
the fourth determining submodule is used for determining the target score of each value area according to the updated state value corresponding to the target hyper-parameter;
and the fifth determining submodule is used for determining a target area from the plurality of value areas according to the target score of each value area.
Optionally, the fourth determining sub-module includes:
a sixth determining submodule, configured to determine, in the updated state value corresponding to the target hyper-parameter, for each value area, an average value of the state value of the value area and the state value of the value area within a preset range adjacent to the value area as a value score of the value area;
and the seventh determining submodule is used for determining the target score of the value area according to the value score of the value area and the hit times of the value area aiming at each value area.
Optionally, the fifth determining sub-module includes:
an eighth determining submodule, configured to determine a value area with a largest target score as the target area; or
And the ninth determining submodule is used for performing softmax processing on the target scores of the value areas to obtain probability distribution formed by probability information of each value area, sampling the value areas according to the probability distribution, and determining the value areas obtained by sampling as the target areas.
Optionally, the target model is a deep reinforcement learning model, the sampling sample is an interaction sequence obtained by sampling in a process of interacting a virtual object with a virtual environment, wherein the virtual object is controlled based on the deep reinforcement learning model, the interaction sequence includes a plurality of sampling data under the sampling value, each sampling data includes an environment state of the virtual environment, a decision action performed by the virtual object under the environment state and determined by the deep reinforcement learning model, and a return value corresponding to the decision action, and the optimized feature parameter is an accumulated return corresponding to the interaction sequence.
Referring now to FIG. 5, a block diagram of an electronic device 600 suitable for use in implementing embodiments of the present disclosure is shown. The terminal device in the embodiments of the present disclosure may include, but is not limited to, a mobile terminal such as a mobile phone, a notebook computer, a digital broadcast receiver, a PDA (personal digital assistant), a PAD (tablet computer), a PMP (portable multimedia player), a vehicle terminal (e.g., a car navigation terminal), and the like, and a stationary terminal such as a digital TV, a desktop computer, and the like. The electronic device shown in fig. 5 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present disclosure.
As shown in fig. 5, electronic device 600 may include a processing means (e.g., central processing unit, graphics processor, etc.) 601 that may perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM)602 or a program loaded from a storage means 608 into a Random Access Memory (RAM) 603. In the RAM 603, various programs and data necessary for the operation of the electronic apparatus 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input/output (I/O) interface 605 is also connected to bus 604.
Generally, the following devices may be connected to the I/O interface 605: input devices 606 including, for example, a touch screen, touch pad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 607 including, for example, a Liquid Crystal Display (LCD), a speaker, a vibrator, and the like; storage 608 including, for example, tape, hard disk, etc.; and a communication device 609. The communication means 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. While fig. 5 illustrates an electronic device 600 having various means, it is to be understood that not all illustrated means are required to be implemented or provided. More or fewer devices may alternatively be implemented or provided.
In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a non-transitory computer readable medium, the computer program containing program code for performing the method illustrated by the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network via the communication means 609, or may be installed from the storage means 608, or may be installed from the ROM 602. The computer program, when executed by the processing device 601, performs the above-described functions defined in the methods of the embodiments of the present disclosure.
It should be noted that the computer readable medium in the present disclosure can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In contrast, in the present disclosure, a computer readable signal medium may comprise a propagated data signal with computer readable program code embodied therein, either in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, optical cables, RF (radio frequency), etc., or any suitable combination of the foregoing.
In some embodiments, the clients, servers may communicate using any currently known or future developed network Protocol, such as HTTP (HyperText Transfer Protocol), and may interconnect with any form or medium of digital data communication (e.g., a communications network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), the Internet (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed network.
The computer readable medium may be embodied in the electronic device; or may exist separately without being assembled into the electronic device.
The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to: acquiring a sampling sample corresponding to a sampling value under the sampling value of a target hyper-parameter of a target model; generating an interactive sample corresponding to the target hyper-parameter according to the sampling sample, wherein the interactive sample comprises the sampling value and an optimized characteristic parameter corresponding to the target model; updating the state value corresponding to the target hyper-parameter according to the interactive sample, wherein the parameter space of the target hyper-parameter is discretized into a plurality of value areas; determining a target area from the plurality of value areas according to the updated state value corresponding to the target hyper-parameter; and determining the target value of the target hyper-parameter according to the target area.
Computer program code for carrying out operations for the present disclosure may be written in any combination of one or more programming languages, including but not limited to an object oriented programming language such as Java, Smalltalk, C + +, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The modules described in the embodiments of the present disclosure may be implemented by software or hardware. For example, the obtaining module may be further described as a module that obtains a sample corresponding to a sampling value of a target hyper-parameter of the target model under the sampling value.
The functions described herein above may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), systems on a chip (SOCs), Complex Programmable Logic Devices (CPLDs), and the like.
In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
Example 1 provides a hyper-parameter determination method, according to one or more embodiments of the present disclosure, wherein the method comprises:
acquiring a sampling sample corresponding to a sampling value under the sampling value of a target hyper-parameter of a target model;
generating an interactive sample corresponding to the target hyper-parameter according to the sampling sample, wherein the interactive sample comprises the sampling value and an optimized characteristic parameter corresponding to the target model;
updating the state value corresponding to the target hyper-parameter according to the interactive sample, wherein the parameter space of the target hyper-parameter is discretized into a plurality of value areas;
determining a target area from the plurality of value areas according to the updated state value corresponding to the target hyper-parameter;
and determining the target value of the target hyper-parameter according to the target area.
Example 2 provides the method of example 1, wherein the method further comprises:
and taking the target value as a new sampling value, and re-executing the step of obtaining the interactive sample corresponding to the target hyper-parameter under the sampling value of the target hyper-parameter of the target model to the step of determining the target value of the target hyper-parameter according to the target area until the training of the target model is finished.
Example 3 provides the method of example 1, wherein the updating, according to the interaction sample, the state value corresponding to the target hyper-parameter includes:
determining a value area to which the sampling value belongs according to the sampling value;
determining a value area to be updated according to the value area to which the sampling value belongs;
and updating the state value of the value area to be updated according to the optimization characteristic parameters.
According to one or more embodiments of the present disclosure, example 4 provides the method of example 3, wherein the determining a value region to be updated according to the value region to which the sampling value belongs includes:
determining the value area to which the sampling value belongs and the value area in a preset range adjacent to the value area to which the sampling value belongs as the value area to be updated;
the updating the state value of the value area to be updated according to the optimization characteristic parameters comprises the following steps:
and updating the state value of each value area to be updated according to the optimization characteristic parameters and the state values of the value areas to be updated.
Example 5 provides the method of example 1, wherein determining a target region from the plurality of value regions according to the updated state value corresponding to the target hyper-parameter includes:
determining a target score of each value area according to the updated state value corresponding to the target hyper-parameter;
and determining a target area from the plurality of value areas according to the target score of each value area.
Example 6 provides the method of example 5, wherein the determining a target score for each of the value areas according to the updated state value corresponding to the target hyper-parameter includes:
in the updated state value corresponding to the target hyper-parameter, determining the average value of the state value of the value area and the state value of the value area in a preset range adjacent to the value area as the value score of the value area aiming at each value area;
and aiming at each value area, determining a target score of the value area according to the value score of the value area and the hit frequency of the value area.
Example 7 provides the method of example 5, wherein determining a target region from the plurality of value regions according to the target score of each value region includes:
determining the value area with the maximum target score as the target area;
or
And performing softmax processing on the target scores of the value areas to obtain probability distribution formed by probability information of each value area, sampling the value areas according to the probability distribution, and determining the value areas obtained by sampling as the target areas.
Example 8 provides the method of example 1, where the target model is a deep reinforcement learning model, the sampling samples are interaction sequences obtained by sampling in a process of interacting a virtual object with a virtual environment, the virtual object is controlled based on the deep reinforcement learning model, the interaction sequences include a plurality of sampling data under the sampling values, each sampling data includes an environment state of the virtual environment, a decision action performed by the virtual object under the environment state, which is determined by the deep reinforcement learning model, and a reward value corresponding to the decision action, and the optimized feature parameter is an accumulated reward corresponding to the interaction sequences.
Example 9 provides a hyper-parameter determination apparatus, in accordance with one or more embodiments of the present disclosure, wherein the apparatus comprises:
the acquisition module is used for acquiring a sampling sample corresponding to a sampling value under the sampling value of a target hyper-parameter of a target model;
the generating module is used for generating an interactive sample corresponding to the target hyper-parameter according to the sampling sample, wherein the interactive sample comprises the sampling value and an optimized characteristic parameter corresponding to the target model;
the updating module is used for updating the state value corresponding to the target hyper-parameter according to the interaction sample, wherein the parameter space of the target hyper-parameter is discretized into a plurality of value areas;
the first determining module is used for determining a target area from the plurality of value areas according to the updated state value corresponding to the target hyper-parameter;
and the second determination module is used for determining the target value of the target hyper-parameter according to the target area.
Example 10 provides a deep reinforcement learning framework in which values of hyper-parameters are determined based on the hyper-parameter determination method of any one of examples 1 to 8, according to one or more embodiments of the present disclosure.
Example 11 provides a computer-readable medium having stored thereon a computer program that, when executed by a processing apparatus, performs the steps of the method of any of examples 1-8, in accordance with one or more embodiments of the present disclosure.
Example 12 provides, in accordance with one or more embodiments of the present disclosure, an electronic device, comprising:
a storage device having a computer program stored thereon;
processing means for executing the computer program in the storage means to carry out the steps of the method of any of examples 1-8.
The foregoing description is only exemplary of the preferred embodiments of the disclosure and is illustrative of the principles of the technology employed. It will be appreciated by those skilled in the art that the scope of the disclosure herein is not limited to the particular combination of features described above, but also encompasses other embodiments in which any combination of the features described above or their equivalents does not depart from the spirit of the disclosure. For example, the above features and (but not limited to) the features disclosed in this disclosure having similar functions are replaced with each other to form the technical solution.
Further, while operations are depicted in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Likewise, while several specific implementation details are included in the above discussion, these should not be construed as limitations on the scope of the disclosure. Certain features that are described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination.
Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims. With regard to the apparatus in the above-described embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.
Claims (12)
1. A method for hyper-parameter determination, the method comprising:
acquiring a sampling sample corresponding to a sampling value under the sampling value of a target hyper-parameter of a target model;
generating an interactive sample corresponding to the target hyper-parameter according to the sampling sample, wherein the interactive sample comprises the sampling value and an optimized characteristic parameter corresponding to the target model;
updating the state value corresponding to the target hyper-parameter according to the interactive sample, wherein the parameter space of the target hyper-parameter is discretized into a plurality of value areas;
determining a target area from the plurality of value areas according to the updated state value corresponding to the target hyper-parameter;
and determining the target value of the target hyper-parameter according to the target area.
2. The method of claim 1, further comprising:
and taking the target value as a new sampling value, and re-executing the step of obtaining a sampling sample corresponding to the sampling value under the sampling value of the target hyper-parameter of the target model to the step of determining the target value of the target hyper-parameter according to the target area until the training of the target model is finished.
3. The method according to claim 1, wherein the updating the state value corresponding to the target hyper-parameter according to the interaction sample comprises:
determining a value area to which the sampling value belongs according to the sampling value;
determining a value area to be updated according to the value area to which the sampling value belongs;
and updating the state value of the value area to be updated according to the optimization characteristic parameters.
4. The method of claim 3, wherein the determining a value area to be updated according to the value area to which the sampling value belongs comprises:
determining the value area to which the sampling value belongs and the value area in a preset range adjacent to the value area to which the sampling value belongs as the value area to be updated;
the updating the state value of the value area to be updated according to the optimization characteristic parameters comprises the following steps:
and updating the state value of each value area to be updated according to the optimization characteristic parameters and the state values of the value areas to be updated.
5. The method of claim 1, wherein determining a target region from the plurality of value regions according to the updated state value corresponding to the target hyper-parameter comprises:
determining a target score of each value area according to the updated state value corresponding to the target hyper-parameter;
and determining a target area from the plurality of value areas according to the target score of each value area.
6. The method of claim 5, wherein the determining the target score for each value area according to the updated state value corresponding to the target hyper-parameter comprises:
in the updated state value corresponding to the target hyper-parameter, determining the average value of the state value of the value area and the state value of the value area in a preset range adjacent to the value area as the value score of the value area aiming at each value area;
and aiming at each value area, determining a target score of the value area according to the value score of the value area and the hit frequency of the value area.
7. The method of claim 5, wherein determining a target region from the plurality of value regions based on the target score for each of the value regions comprises:
determining the value area with the maximum target score as the target area;
or
And performing softmax processing on the target scores of the value areas to obtain probability distribution formed by probability information of each value area, sampling the value areas according to the probability distribution, and determining the value areas obtained by sampling as the target areas.
8. The method according to claim 1, wherein the target model is a deep reinforcement learning model, the sampling samples are interaction sequences obtained by sampling in a process of interacting a virtual object with a virtual environment, wherein the virtual object is controlled based on the deep reinforcement learning model, the interaction sequences include a plurality of sampling data under the sampling values, each sampling data includes an environment state of the virtual environment, a decision action performed by the virtual object under the environment state, which is determined by the deep reinforcement learning model, and a return value corresponding to the decision action, and the optimized feature parameter is an accumulated return corresponding to the interaction sequences.
9. A hyper-parameter determination apparatus, the apparatus comprising:
the acquisition module is used for acquiring a sampling sample corresponding to a sampling value under the sampling value of a target hyper-parameter of a target model;
the generating module is used for generating an interactive sample corresponding to the target hyper-parameter according to the sampling sample, wherein the interactive sample comprises the sampling value and an optimized characteristic parameter corresponding to the target model;
the updating module is used for updating the state value corresponding to the target hyper-parameter according to the interaction sample, wherein the parameter space of the target hyper-parameter is discretized into a plurality of value areas;
the first determining module is used for determining a target area from the plurality of value areas according to the updated state value corresponding to the target hyper-parameter;
and the second determination module is used for determining the target value of the target hyper-parameter according to the target area.
10. A deep reinforcement learning framework, wherein values of hyper-parameters in the deep reinforcement learning framework are determined based on the hyper-parameter determination method of any one of claims 1 to 8.
11. A computer-readable medium, on which a computer program is stored, characterized in that the program, when being executed by processing means, carries out the steps of the method of any one of claims 1 to 8.
12. An electronic device, comprising:
a storage device having a computer program stored thereon;
processing means for executing the computer program in the storage means to carry out the steps of the method according to any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110129788.6A CN112926629B (en) | 2021-01-29 | 2021-01-29 | Super-parameter determination method, device, deep reinforcement learning framework, medium and equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110129788.6A CN112926629B (en) | 2021-01-29 | 2021-01-29 | Super-parameter determination method, device, deep reinforcement learning framework, medium and equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112926629A true CN112926629A (en) | 2021-06-08 |
| CN112926629B CN112926629B (en) | 2024-04-02 |
Family
ID=76168816
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110129788.6A Active CN112926629B (en) | 2021-01-29 | 2021-01-29 | Super-parameter determination method, device, deep reinforcement learning framework, medium and equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112926629B (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109711548A (en) * | 2018-12-26 | 2019-05-03 | 歌尔股份有限公司 | Choosing method, application method, device and the electronic equipment of hyper parameter |
| JP2019087096A (en) * | 2017-11-08 | 2019-06-06 | 本田技研工業株式会社 | Action determination system and automatic driving control device |
| CN110659738A (en) * | 2019-09-12 | 2020-01-07 | 苏州浪潮智能科技有限公司 | Method and device for adjusting hyper-parameters and computer readable storage medium |
| US20200076857A1 (en) * | 2018-08-31 | 2020-03-05 | Microsoft Technology Licensing, Llc | Secure exploration for reinforcement learning |
| CN111260762A (en) * | 2020-01-19 | 2020-06-09 | 腾讯科技（深圳）有限公司 | Animation implementation method and device, electronic equipment and storage medium |
| KR20200105365A (en) * | 2019-06-05 | 2020-09-07 | 아이덴티파이 주식회사 | Method for reinforcement learning using virtual environment generated by deep learning |
-
2021
- 2021-01-29 CN CN202110129788.6A patent/CN112926629B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2019087096A (en) * | 2017-11-08 | 2019-06-06 | 本田技研工業株式会社 | Action determination system and automatic driving control device |
| US20200076857A1 (en) * | 2018-08-31 | 2020-03-05 | Microsoft Technology Licensing, Llc | Secure exploration for reinforcement learning |
| CN109711548A (en) * | 2018-12-26 | 2019-05-03 | 歌尔股份有限公司 | Choosing method, application method, device and the electronic equipment of hyper parameter |
| KR20200105365A (en) * | 2019-06-05 | 2020-09-07 | 아이덴티파이 주식회사 | Method for reinforcement learning using virtual environment generated by deep learning |
| CN110659738A (en) * | 2019-09-12 | 2020-01-07 | 苏州浪潮智能科技有限公司 | Method and device for adjusting hyper-parameters and computer readable storage medium |
| CN111260762A (en) * | 2020-01-19 | 2020-06-09 | 腾讯科技（深圳）有限公司 | Animation implementation method and device, electronic equipment and storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 王?儒;李俊;: "采用双经验回放池的噪声流双延迟深度确定性策略梯度算法", 武汉科技大学学报, no. 02 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112926629B (en) | 2024-04-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112766497A (en) | Deep reinforcement learning model training method, device, medium and equipment | |
| US20220176248A1 (en) | Information processing method and apparatus, computer readable storage medium, and electronic device | |
| EP3926554A1 (en) | Method and apparatus for optimizing recommendation system, device and computer storage medium | |
| EP4113394A2 (en) | Federated learning method, apparatus and system, electronic device, and storage medium | |
| CN110795569A (en) | Method, device and equipment for generating vector representation of knowledge graph | |
| CN111414543B (en) | Method, device, electronic equipment and medium for generating comment information sequence | |
| CN113568727A (en) | Mobile edge calculation task allocation method based on deep reinforcement learning | |
| CN112528160B (en) | Intelligent recommendation method, intelligent recommendation model training method, intelligent recommendation device, intelligent recommendation model training device, electronic equipment and storage medium | |
| CN111695698A (en) | Method, device, electronic equipment and readable storage medium for model distillation | |
| CN113052253A (en) | Hyper-parameter determination method, device, deep reinforcement learning framework, medium and equipment | |
| CN112926628A (en) | Action value determination method, device, learning framework, medium and equipment | |
| CN114240506A (en) | Modeling method of multi-task model, promotion content processing method and related device | |
| CN112949850B (en) | Super-parameter determination method, device, deep reinforcement learning framework, medium and equipment | |
| CN111510473B (en) | Access request processing method and device, electronic equipment and computer readable medium | |
| CN112926629B (en) | Super-parameter determination method, device, deep reinforcement learning framework, medium and equipment | |
| CN113052252A (en) | Hyper-parameter determination method, device, deep reinforcement learning framework, medium and equipment | |
| CN116306981A (en) | Policy determination method, device, medium and electronic equipment | |
| CN113052312A (en) | Deep reinforcement learning model training method and device, medium and electronic equipment | |
| CN113052248A (en) | Hyper-parameter determination method, device, deep reinforcement learning framework, medium and equipment | |
| CN112926735A (en) | Method, device, framework, medium and equipment for updating deep reinforcement learning model | |
| CN111443806A (en) | Interactive task control method and device, electronic equipment and storage medium | |
| CN111330269B (en) | Application difficulty adjustment and strategy determination method, device, system, equipment and medium | |
| CN112465717B (en) | Face image processing model training method, device, electronic equipment and medium | |
| CN117829222A (en) | Model quantization method, apparatus, electronic device, and computer-readable storage medium | |
| CN112187945A (en) | Information pushing method and device and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |