CN113052248A

CN113052248A - Hyper-parameter determination method, device, deep reinforcement learning framework, medium and equipment

Info

Publication number: CN113052248A
Application number: CN202110349555.7A
Authority: CN
Inventors: 范嘉骏
Original assignee: Beijing ByteDance Network Technology Co Ltd
Current assignee: Beijing ByteDance Network Technology Co Ltd
Priority date: 2021-03-31
Filing date: 2021-03-31
Publication date: 2021-06-29
Anticipated expiration: 2041-03-31
Also published as: CN113052248B

Abstract

The disclosure relates to a hyper-parameter determination method, a hyper-parameter determination device, a deep reinforcement learning framework, a medium and equipment, wherein the method comprises the following steps: acquiring a sampling sample corresponding to a sampling value combination under the sampling value combination corresponding to a target hyper-parameter combination of a target model; generating an interactive sample corresponding to the target hyper-parameter combination according to the sampling sample, wherein the interactive sample comprises the sampling value combination and an optimized characteristic parameter corresponding to the target model; updating the state value corresponding to the target hyper-parameter combination according to the interaction sample; determining a target space from the plurality of value spaces according to the updated state value corresponding to the target hyper-parameter combination; and determining a target value combination corresponding to the target hyper-parameter combination according to the target space. Therefore, values of a plurality of hyper-parameters with incidence relation of the model can be accurately set.

Description

Hyper-parameter determination method, device, deep reinforcement learning framework, medium and equipment

Technical Field

The disclosure relates to the field of computers, in particular to a hyper-parameter determination method, a hyper-parameter determination device, a deep reinforcement learning framework, a medium and equipment.

Background

With the development of random computer technology, various large models and complex machine learning models are gradually applied. A large number of parameters are needed to be calculated in the model, so that the model can meet the requirements of users. Some parameters in the model can be optimized through the training of the model, such as weights in the neural network model, and some parameters cannot be optimized through the training of the model, and such parameters are hyper-parameters of the model, such as the number of hidden layers in the neural network. The hyper-parameters are used for adjusting the training process of the model, usually set manually by workers based on experience, do not directly participate in the training process of the model, and are not updated in the training process of the model. And the setting of the hyper-parameters has great influence on the iteration times, the convergence efficiency and the like of model training.

Disclosure of Invention

This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

In a first aspect, the present disclosure provides a method for hyper-parameter determination, the method comprising:

acquiring a sampling sample corresponding to a sampling value combination under the sampling value combination corresponding to a target hyper-parameter combination of a target model, wherein the target hyper-parameter combination comprises a plurality of target hyper-parameters with an association relation, and the sampling value combination comprises a sampling value corresponding to each target hyper-parameter;

generating an interactive sample corresponding to the target hyper-parameter combination according to the sampling sample, wherein the interactive sample comprises the sampling value combination and an optimized characteristic parameter corresponding to the target model;

updating the state value corresponding to the target hyper-parameter combination according to the interaction sample, wherein the target hyper-parameter corresponds to the dimension in the target hyper-space corresponding to the target hyper-parameter combination one by one, and the parameter space of each target hyper-parameter is discretized into a plurality of value intervals under the dimension corresponding to the target hyper-parameter, so that the target hyper-space is discretized into a plurality of value spaces;

determining a target space from the plurality of value spaces according to the updated state value corresponding to the target hyper-parameter combination;

and determining a target value combination corresponding to the target hyper-parameter combination according to the target space.

In a second aspect, the present disclosure provides a hyper-parameter determination apparatus, the apparatus comprising:

the system comprises an acquisition module, a processing module and a processing module, wherein the acquisition module is used for acquiring a sampling sample corresponding to a sampling value combination under the sampling value combination corresponding to the target hyper-parameter combination of a target model, the target hyper-parameter combination comprises a plurality of target hyper-parameters with incidence relation, and the sampling value combination comprises a sampling value corresponding to each target hyper-parameter;

a generating module, configured to generate an interaction sample corresponding to the target hyper-parameter combination according to the sampling sample, where the interaction sample includes the sampling value combination and an optimized feature parameter corresponding to the target model;

an updating module, configured to update a state value corresponding to the target hyper-parameter combination according to the interaction sample, where the target hyper-parameter corresponds to a dimension in a target hyper-space corresponding to the target hyper-parameter combination, and a parameter space of each target hyper-parameter is discretized into a plurality of value intervals in the dimension corresponding to the target hyper-parameter, so that the target hyper-space is discretized into a plurality of value spaces;

the first determining module is used for determining a target space from the plurality of value spaces according to the updated state value corresponding to the target hyper-parameter combination;

and the second determining module is used for determining a target value combination corresponding to the target hyper-parameter combination according to the target space.

In a third aspect, a deep reinforcement learning framework is provided, and values of a plurality of hyper-parameters having an association relationship in the deep reinforcement learning framework are determined based on the hyper-parameter determination method in the first aspect.

In a fourth aspect, a computer-readable medium is provided, on which a computer program is stored which, when being executed by a processing device, carries out the steps of the method of the first aspect.

In a fifth aspect, an electronic device is provided, comprising:

a storage device having a computer program stored thereon;

processing means for executing the computer program in the storage means to carry out the steps of the method of the first aspect.

Therefore, in the above technical solution, for a plurality of target hyper-parameters having an association relationship, a parameter space of each target hyper-parameter can be simultaneously represented through a target hyper-space, so that the parameter space of the target hyper-parameter in each dimension can be discretized into a plurality of value spaces, and the value of the target hyper-parameter combination can be determined based on the optimized characteristic parameter of the target model based on the sampling sample corresponding to the target model using the target hyper-parameter combination. Therefore, by means of the technical scheme, on one hand, the value of the hyper-parameter of the model can be accurately set, and the phenomenon that the target model cannot be converged or the convergence speed is too low due to the fact that the hyper-parameter setting value is not appropriate due to the limitation of human experience is avoided. On the other hand, by uniformly determining the values of the multiple hyper-parameters with the incidence relation, the accuracy of the value of each target hyper-parameter can be ensured, meanwhile, the matching degree of the values of the multiple hyper-parameters and the whole target model can be ensured, and the problem that each hyper-parameter falls into a saddle point when being respectively optimized is avoided. Meanwhile, a target value combination is determined based on the optimization characteristic parameters of the target model, and a target space is determined based on a condition that the optimization characteristic parameters are better when the target value combination is determined, so that the training efficiency of the target model can be effectively improved, the iteration times of training of the target model are reduced to a certain extent, and the convergence efficiency of the target model is improved.

Additional features and advantages of the disclosure will be set forth in the detailed description which follows.

Drawings

The above and other features, advantages and aspects of various embodiments of the present disclosure will become more apparent by referring to the following detailed description when taken in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numbers refer to the same or similar elements. It should be understood that the drawings are schematic and that elements and features are not necessarily drawn to scale. In the drawings:

FIG. 1 is a flow chart of a method of hyperparameter determination provided in accordance with an embodiment of the present disclosure;

FIG. 2 is a schematic illustration of a target hyperspace provided in accordance with an embodiment of the present disclosure;

FIG. 3 is a schematic diagram of a target hyperspace and its corresponding reference hyperspace provided in accordance with an embodiment of the present disclosure;

FIG. 4 is a flow diagram of an exemplary implementation for determining a target space from the plurality of value spaces according to an updated state value corresponding to the target hyper-parameter combination according to an embodiment of the present disclosure;

FIG. 5 is a block diagram of a hyper-parameter determination apparatus provided in accordance with an embodiment of the present disclosure;

FIG. 6 shows a schematic structural diagram of an electronic device suitable for use in implementing embodiments of the present disclosure.

Detailed Description

Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While certain embodiments of the present disclosure are shown in the drawings, it is to be understood that the present disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein, but rather are provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the disclosure are for illustration purposes only and are not intended to limit the scope of the disclosure.

It should be understood that the various steps recited in the method embodiments of the present disclosure may be performed in a different order, and/or performed in parallel. Moreover, method embodiments may include additional steps and/or omit performing the illustrated steps. The scope of the present disclosure is not limited in this respect.

The term "include" and variations thereof as used herein are open-ended, i.e., "including but not limited to". The term "based on" is "based, at least in part, on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions for other terms will be given in the following description.

It should be noted that the terms "first", "second", and the like in the present disclosure are only used for distinguishing different devices, modules or units, and are not used for limiting the order or interdependence relationship of the functions performed by the devices, modules or units.

It is noted that references to "a", "an", and "the" modifications in this disclosure are intended to be illustrative rather than limiting, and that those skilled in the art will recognize that "one or more" may be used unless the context clearly dictates otherwise.

The names of messages or information exchanged between devices in the embodiments of the present disclosure are for illustrative purposes only, and are not intended to limit the scope of the messages or information.

Fig. 1 is a flowchart of a method for determining a hyper-parameter according to an embodiment of the present disclosure, and as shown in fig. 1, the method may include:

in step 11, a sampling sample corresponding to a sampling value combination is obtained under the sampling value combination corresponding to a target hyper-parameter combination of a target model, wherein the target hyper-parameter combination includes a plurality of target hyper-parameters having an association relationship, and the sampling value combination includes a sampling value corresponding to each target hyper-parameter.

Illustratively, the target model may be a deep reinforcement learning model. The deep reinforcement learning model combines the perception capability of deep learning and the decision capability of reinforcement learning, obtains a high-dimensional observation through interaction between an agent (agent) and the environment at each moment, and perceives the observation by using a deep learning method to obtain specific state characteristic representation of the observation; then, a value function (state value function) of each state and a value function (action value function) of a state-action pair can be evaluated based on expected returns, and a decision strategy is promoted based on the two value functions, wherein the decision strategy is used for mapping the current state into a corresponding decision action; the environment will react to this decision-making action and get the next observation. The above process is continuously cycled to obtain the optimal strategy for achieving the goal, illustratively, the goal is to maximize the accumulated return.

For example, taking a deep reinforcement learning model as an example, a search space of the strategy pi may be represented by 2 target hyper-parameters in the deep reinforcement model, and the target hyper-parameter combination α may be represented as: α ═ t (τ)₁,τ₂) The sampling value combination can include the target hyper-parameter tau respectively₁And τ₂The sampling value of (2).

Therefore, in the application scenario, the sampling sample is an interaction sequence obtained by sampling in a process of interacting a virtual object with a virtual environment, where the virtual object is controlled based on the deep reinforcement learning model, the interaction sequence includes a plurality of sampling data under a sampling value combination corresponding to the target hyper-parameter combination, and each sampling data includes an environment state of the virtual environment, a decision action performed by the virtual object in the environment state, which is determined by the deep reinforcement learning model, and a return value corresponding to the decision action.

The virtual environment may be a virtual scene environment generated by a computer, for example, the virtual environment may be a game scene, and illustratively, multimedia data for interacting with a user is rendered so that the multimedia data can be rendered and displayed as the game scene, the virtual environment provides a multimedia virtual world, and the user can control actions of virtual objects through controls on an operation interface, or directly control virtual objects operable in the virtual environment and observe objects, characters, scenery and the like in the virtual environment from the perspective of the virtual objects, and interact through the virtual objects and other virtual objects and the like in the virtual environment. As another example, the virtual environment may also include other virtual objects in the scene, and the like. The virtual object may be an avatar in a virtual environment for simulating a user, which may be a human or other animal avatar, or the like.

The application scene may be a scene in which the virtual object senses the environment in which the virtual object is located and acts according to the sensed environment state. The application scene may include a virtual object and a plurality of environment objects included in an environment where the virtual object is located, and in the scene, the virtual object may fuse an environment state of the environment where the virtual object is located, and input the fused environment state into the target model, so as to obtain a target action to be executed by the virtual object. The virtual object may be any agent that can interact with the environment and act according to the environment state of the environment.

Illustratively, the virtual object is game Artificial Intelligence (AI), the deep reinforcement learning model is used for training the game artificial intelligence, the sampling sample is an interaction sequence obtained by sampling the game artificial intelligence in a game play of a target game, and the virtual environment is a training environment in which the game artificial intelligence is located in the target game.

As an example, the target game is a gunfight type game, the virtual object may be a game fight AI, and the corresponding decision action may be to control game fight AI character attack, movement, stop, and the like. As another example, the target game is a driving game, the virtual object may be an automatic driving game vehicle AI, and the corresponding decision-making action may be to control the vehicle to turn, go straight, brake, etc. As another example, the target game may be an assembly game, the virtual object may be a robot AI, and the corresponding decision-making action may be to control the robot AI to move, grab and put down an object to be assembled, and so on.

For example, when the game artificial intelligence is sampled in the game of the target game to obtain the interaction sequence, the environment where the game artificial intelligence is located can be sensed by the game artificial intelligence to obtain the multi-modal environment state of the training environment where the game artificial intelligence is located. The environment state may include an environment image and object information of each environment object in the environment image, where the object information includes specific parameters corresponding to the environment object. For example, when the virtual object is a game match AI in a gun-battle game, the virtual environment may be a training environment in which the game match AI is located in the gun-battle game, the environment image may be a game map in which the game match AI is located, the environment object may be an enemy unit, a road, a building, or the like in the game map, and the object information may include information such as a numerical parameter (for example, blood volume, offensive power, skill) of the enemy unit, a name, a location, and the like. When the virtual object is a game vehicle AI of a driving game, the virtual environment may be a training environment in which the game vehicle AI is located in the driving game, the environment image may be a captured image of the surroundings of the vehicle, the environment object may be another vehicle, an obstacle, a road, and the like around the vehicle, and the object information may include information such as a vehicle speed, a traveling direction, and a size of the other vehicle. When the virtual object is a robot AI in the assembly game, the virtual environment may be a training environment in which the robot AI is located in the assembly game, the environment image may be a photographed image of an area where the object to be assembled is located, the environment object may be the object to be assembled, and the object information may include information such as the size, shape, and position of the object to be assembled. The object information for each environmental object may then be preprocessed to obtain an object feature vector for each environmental object. For example, the object information of each environmental object may be input into a pre-trained deep learning network to convert the object information of each environmental object into an object feature vector of the environmental object.

As an example, the virtual object may perform a decision-making action in a first state of the virtual environment, and after the virtual object performs the decision-making action, the virtual environment may react to the decision-making action to obtain a second state of the virtual environment and a return value corresponding to the decision-making action. When sampling is performed during the interaction between the virtual object and the virtual environment, the first state, the decision action, the second state, and the return value may be used as sampling data corresponding to the sampling time, and if not otherwise stated, the environment state in the sampling data in the embodiment of the present disclosure is the first state in the sampling data. In a complete interactive process, the sampling data according to the sequence of the sampling time is formed into an interactive sequence. When sampling is performed, an image of the virtual environment corresponding to the sampling time can be acquired, so that feature extraction can be performed on the image to obtain the first state. After the virtual object executes the decision-making action, sampling to acquire an image of the virtual environment and performing feature extraction on the image to acquire a second state. The reward value may be a change of a score value corresponding to the virtual object after the decision action is executed, or a change of a virtual life bar, and the reward value may be set according to an actual usage scenario, which is not limited by the present disclosure.

In step 12, an interactive sample corresponding to the target hyper-parameter combination is generated according to the sampling sample, and the interactive sample includes the sampling value combination and the optimized characteristic parameter corresponding to the target model.

The optimized characteristic parameter may be a cumulative reward corresponding to the interaction sequence. As an example, the accumulated reward may be a sum of reward values corresponding to each decision-making action included in the interaction sequence. As another example, the more distant a decision-making action has an effect on the current decision-making action, the less the accumulated reward is, the accumulated reward may be the accumulated sum of the return value of each decision-making action in the interaction sequence multiplied by the attenuation coefficient corresponding to the decision-making action, wherein the attenuation coefficients corresponding to the decision-making actions decrease in the order of the decision-making actions, for example:

G_t＝R_t+1+γR_t+2+γ²R_t+3+…+γ^n-1R_t+n

＝R_t+1+γ(R_t+2+γR_t+3+…+γ^n-2R_t+n)

＝R_t+1+γG_t+1

wherein R is_iFor representingThe return value of the decision action at time i, γ, is used to represent the attenuation coefficient, and n may be used to represent the number of sample data in the interaction sequence after time t until the end of the interaction.

Thus, in another embodiment, the accumulated reward may be obtained from the last decision-making action of the interaction sequence by multiplying its reward value by the decay value and adding the reward value of the previous decision-making action until the reward value of the first decision-making action in the interaction sequence is added. Wherein the attenuation value can be set according to the actual usage scenario.

In step 13, according to an interaction sample, updating a state value corresponding to a target hyper-parameter combination, where the target hyper-parameter corresponds to a dimension in a target hyper-space corresponding to the target hyper-parameter combination, and a parameter space of each target hyper-parameter is discretized into a plurality of value intervals in the dimension corresponding to the target hyper-parameter, so that the target hyper-space is discretized into a plurality of value spaces.

In the above example, the target hyper-parameter combination includes two target hyper-parameters, each of which is τ₁And τ₂Then the target hyper-space corresponding to the target hyper-parameter combination is a two-dimensional space, as shown in FIG. 2, wherein the X-axis dimension corresponds to the target hyper-parameter τ₁The Y-axis dimension corresponds to the target hyper-parameter τ₂Target hyperparameter τ₁And τ₂The parameter space that corresponds respectively can carry out the discretization in its corresponding dimension, and wherein the interval that different dimensions carried out the discretization can be the same, also can be different, and the user can set up based on the in-service use scene, and this disclosure does not restrict this.

As shown in FIG. 2, the object hyper-parameter τ is measured in the X-axis dimension₁Is discretized at discrete intervals H1 to obtain the target hyper-parameter tau in the Y-axis dimension₂The parameter space of (2) is discretized at discretization intervals H2, the target hyperspace is discretized into 12 value spaces (C00-C23) as shown in fig. 2. The state value corresponding to the target hyper-parameter combination can be represented by a vector, namely the state value corresponding to each of the 12 value spacesThe values are each a dimension value in the vector.

In the field, when a model is known, an expected accumulated return brought by an arbitrary policy pi can be estimated, a state value function is generally adopted to evaluate the value of a certain state, the value of a certain state can be expressed by the values of all actions in the state, that is, the expected value of the accumulated return based on a state s can be obtained, under the policy, the accumulated return obeys a distribution, and the expected value of the accumulated return at the state is defined as a state value function v(s):

V_π(s)＝E_π[G_t|S_t＝s]

V_π(S) means the state S at time t under strategy π_tWhen s is the value, G is accumulated_tThe expected value at s. Then in this embodiment, the state value at all value intervals of the state s can be evaluated based on the state value function. The determination of the accumulated reward is described in detail above, and is not described herein again.

In the deep reinforcement learning model, the calculation of the state value function can be realized through a neural network. Therefore, the environmental state in the sample data can be input into the state value function network, so that the output value of the state value function network, that is, the state value of the state value function corresponding to the environmental state can be obtained.

As an example, a parameter space of each target hyper-parameter in the target hyper-parameter combination is discretized into a plurality of value intervals, so that the target hyper-space corresponding to the target hyper-parameter combination is discretized into a plurality of value spaces, and then the state value corresponding to the target hyper-parameter combination can be used to represent the accumulated return brought by selecting each target hyper-parameter value in the target hyper-parameter combination from each value space based on a policy in a state where the value of the target hyper-parameter in the target hyper-parameter combination is the corresponding sampling value in the sampling value combination. For example, in the present disclosure, the state value corresponding to the target hyper-parameter combination may be determined in an iterative update manner, that is, the state value corresponding to the target hyper-parameter combination is iteratively updated according to the sampling value combination corresponding to the interactive sample.

As an example, when discretizing the parameter space in each dimension in the target hyper-space, for each dimension, the number of value intervals corresponding to the parameter space may be predetermined, and then the parameter space of the target hyper-parameter corresponding to the dimension may be uniformly divided to obtain a plurality of value intervals in the dimension. If the parameter space of the target hyper-parameter is [0,9], the parameter space is divided into 9 value ranges, the value range corresponding to the value range a1 is [0,1 ], the value range corresponding to the value range a2 is [1,2 ], and so on, and the description thereof is omitted here.

In step 14, a target space is determined from the plurality of value spaces according to the updated state value corresponding to the target hyper-parameter combination.

In this step, the accumulated return of the values of the target hyper-parameter combination from each value space can be accurately evaluated by determining the state value corresponding to the target hyper-parameter combination according to the sampling sample of the target model, so that the target space for determining the value corresponding to the target hyper-parameter combination can be selected according to the evaluation result, and the accuracy of the value corresponding to the target hyper-parameter combination and the consistency of the value corresponding to the target hyper-parameter combination and the actual application process of the target model can be ensured.

In step 15, a target value combination corresponding to the target hyper-parameter combination is determined according to the target space.

As an embodiment, the uniform distribution sampling may be performed in a value range corresponding to the target space, and a value corresponding to a point obtained by sampling in each dimension is determined as a target value of the target hyper-parameter corresponding to the dimension. As shown in fig. 2, if the determined target space is C13, and the sampling point obtained by sampling from the target space is P1, the value Px of P1 corresponding to the X-axis dimension can be determined as the target hyper-parameter τ₁Determining a value Py corresponding to P1 in the Y-axis dimension as a target hyper-parameter tau₂To obtain a targetAnd (4) value combination.

As described above, the virtual environment may be a game environment, and then sampling may be performed in the process of interacting the virtual object with the virtual environment to obtain interaction data, and the value of the hyper-parameter in the deep reinforcement learning model may be determined based on the above manner in the training process of the deep reinforcement learning model, so that a greater return can be obtained when the deep reinforcement learning model determines the decision-making action of the virtual object, the accuracy of the decision-making action of the virtual object is ensured, the accuracy of virtual character control is improved, and meanwhile, the amount of data and manpower required in the training process can be reduced.

In one possible embodiment, the method may further comprise:

and taking the target value combination as a new sampling value combination corresponding to the target hyper-parameter combination, and re-executing the steps from the step of obtaining the sampling sample corresponding to the sampling value combination under the sampling value combination corresponding to the target hyper-parameter combination of the target model to the step of determining the target value combination corresponding to the target hyper-parameter combination according to the target space until the training of the target model is completed.

The sampling value combination corresponding to the target hyper-parameter combination can be obtained based on model sampling of random initialization when the sampling value combination is initial. In this embodiment, after determining the target value combination corresponding to the target hyper-parameter combination, the value of the corresponding target hyper-parameter in the target model may be updated to the corresponding target value, so that steps 11 to 15 may be executed again, to further determine whether the target value combination is accurate for the interactive sample of the target hyper-parameter combination of the target model under the target value combination, thereby implementing dynamic adjustment of the value corresponding to the target hyper-parameter combination, and simultaneously optimizing the optimized feature parameter corresponding to the target model, while further improving the accuracy of the value corresponding to the target hyper-parameter combination, ensuring the convergence and accuracy of the target model, and improving the training efficiency of the target model.

In a possible embodiment, in step 13, according to the interaction sample, an exemplary implementation manner of updating the state value corresponding to the target hyper-parameter combination is as follows, and the step may include:

and determining the value space to which the sampling value combination belongs as the value space to be updated according to the sampling value combination.

For example, the identifier of the sample value under the dimension corresponding to the sample value may be respectively determined according to each sample value in the sample value combination, for example, the value interval to which the sample value belongs may be determined based on the range length corresponding to the value interval of the sample value, and the subscript i of the value interval to which the sample value belongs may be determined by the following formula:

i＝(min(max(x,l),r)-l)//acc

wherein x is used to represent the sampling value; l is used to represent the left boundary of the parameter space; r is used to represent the right boundary of the parameter space; // for integer division symbols; acc is used to indicate the range length of the value interval.

Therefore, when the subscript of the sampling interval corresponding to each sampling value is determined based on the above manner, the value space to be updated is determined based on the dimension of each sampling value. For example, if the subscript determined by the X-axis dimension is 2 and the subscript determined by the Y-axis dimension is 1, the value space to which the sampling value combination belongs is C21.

And then, updating the state value of the value space to be updated according to the optimization characteristic parameters.

In this embodiment, a to-be-updated value space corresponding to the sampling value can be determined by a relationship between the sampling value and the parameter space, so that a state value of the to-be-updated value space can be updated, and for other value spaces except the to-be-updated value space, the corresponding state value does not need to be updated, so that accuracy of the state value corresponding to the target hyper-parameter combination can be ensured, and data support is provided for subsequently and accurately selecting the target space.

In this embodiment, the value space to which the sampling value combination belongs is determined as the value space to be updated, and the state value of the value space to be updated can be updated according to the optimized characteristic parameter by the following formula:

wherein T is used to represent the optimization feature parameter, and if the optimization feature parameter is cumulative reward, that is, the cumulative reward is optimized in the direction of increasing during optimization, T may be G_tIf the optimization characteristic parameter is the error rate error of the target model, i.e. the error rate is optimizedOptimizing towards the direction of reduction, then T can be-error; k(s) is used to represent the number of hits of the value space s to be updated, that is, the number of times that the value of the target hyper-parameter combination corresponding to the sampling sample belongs to the value space s to be updated, V(s) is used to represent the current state value of the value space s to be updated, and V'(s) is used to represent the state value of the value space s to be updated after updating.

For another example, the state value of the value space to be updated can be updated according to the following formula and the optimized characteristic parameter by the following formula:

V'(s)＝V(s)+lr*(T-V(s))

here, lr represents a learning rate for updating the state value.

Therefore, by the technical scheme, the state value corresponding to the target hyper-parameter space can be updated based on the interactive sample, so that the state value corresponds to the actual optimization characteristic parameter in the target model, the accuracy of the subsequently determined target value combination can be ensured, and the training efficiency of the target model is optimized.

In a possible embodiment, the target hyperspace corresponds to a plurality of reference hyperspaces, a discrete interval corresponding to a value space in each reference hyperspace is greater than a discrete interval corresponding to a value space in the target hyperspace, and an origin corresponding to each reference hyperspace is different.

As shown in fig. 3, where the a space is the target hyperspace shown in fig. 2, the B1 space and the B2 space are reference hyperspaces corresponding to the target hyperspace shown in fig. 2, and for convenience of representation, the B1 space and the B2 space are represented as shown in fig. 3, the positions of the origins of the B1 space and the B2 space are offset from the positions of the origins of the target hyperspaces, but positions corresponding to the a space exist in both the B1 space and the B2 space.

As shown in fig. 3, the discrete interval corresponding to the reference hyperspace B1 and the discrete interval corresponding to the reference hyperspace B2 are both greater than the discrete interval corresponding to the target hyperspace, for example, the discrete intervals corresponding to the reference hyperspaces may be set according to the discrete intervals corresponding to each dimension in the target hyperspace, which may be the same or different, and may be set according to the actual use requirement.

Accordingly, according to the interaction sample, an exemplary implementation manner of updating the state value corresponding to the target hyper-parameter combination is as follows, and this step may include:

and determining a reference value space to which the sampling value combination belongs in each reference hyperspace and a value space to be updated to which the sampling value combination belongs in the target hyperspace according to the sampling value combination.

And updating the state value of each reference value space according to the optimization characteristic parameters.

The method for determining the value space to which the sampling value combination belongs in each hyperspace and updating the state value of the value space is described in detail above, and is not described herein again.

And then, updating the state value of the value space to be updated according to the updated state value corresponding to each reference hyperspace.

According to the technical scheme, the state value corresponding to the target hyperspace can be updated based on the multiple reference hyperspaces, and the discrete interval corresponding to the value space in each reference hyperspace is larger than the discrete interval corresponding to the value space in the target hyperspace, so that the calculation efficiency for determining the state value corresponding to the reference value space in the reference hyperspace is higher, on one hand, the updating efficiency of the state value corresponding to the target hyperspace can be improved to a certain degree, on the other hand, the accuracy of the state value corresponding to the determined target hyperparameter combination can be improved to a certain degree based on fusion expression of the multiple reference hyperspaces corresponding to the target hyperspace, and accurate data support is provided for subsequently determining the target value combination.

In a possible embodiment, an exemplary implementation manner of updating the state value of the value space to be updated according to the updated state value corresponding to each reference hyperspace is as follows, and the step may include:

and determining a mapping value space corresponding to the value space to be updated in each reference hyperspace.

Wherein the mapping value space may be determined based on an offset between an origin of each of the reference hyperspaces and an origin of the target hyperspace. The coordinate systems corresponding to the target hyperspace and the reference hyperspace may be converted to be determined, and the mapping manner of the coordinate system conversion may adopt any conversion manner in the art, which is not described herein again.

For example, as shown in fig. 3, if the value space to be updated in the target hyperspace a is C01, the reference value space corresponding to the target hyperspace a in the reference hyperspace B1 is M1, and the reference value spaces corresponding to the target hyperspace B2 are M2 and M3.

And determining the state value of the value space to be updated according to the state value of each mapping value space.

For example, the average value of the state value of each mapping value space may be used to determine the state value of the value space to be updated.

Therefore, by the technical scheme, the state value corresponding to the target hyper-parameter combination can be updated based on the interactive sample instead of directly updating the state value corresponding to the multiple reference hyper-spaces based on the interactive sample, so that the target hyper-spaces can be fused and characterized through the multiple reference hyper-spaces based on the mapping relation between the target hyper-spaces and the reference hyper-spaces, the accuracy of the determined state value corresponding to the target hyper-parameter combination can be improved, and the accuracy of the subsequently determined value of the hyper-parameters is further improved.

In order to improve efficiency of the value determination of the hyper-parameter, when the number of the interaction samples reaches a preset threshold, for each interaction sample, a step of updating a state value corresponding to the target hyper-parameter combination according to the interaction sample may be performed, where values of the target hyper-parameter combination corresponding to each interaction sample are taken under different value spaces. That is, the state value corresponding to the target hyper-parameter combination may be updated based on a plurality of interaction samples, and the updating manner based on each interaction sample is the same as that described above, and is not described herein again.

In this embodiment, after the state value corresponding to the target hyper-parameter combination is updated, the score corresponding to each value space may be recalculated, so that the accuracy of the score of each value space may be ensured, and accurate data support may be provided for determining the target space.

In a possible embodiment, an exemplary implementation manner of determining a target space from the multiple value spaces according to the updated state value corresponding to the target hyper-parameter combination is as follows, as shown in fig. 4, and the step may include:

in step 41, a target score of each value space is determined according to the updated state value corresponding to the target hyper-parameter combination, wherein the target score is used for representing the reliability degree of selecting the value space.

As an example, an exemplary implementation manner of determining the target score of each value space according to the updated state value corresponding to the target hyper-parameter combination is as follows, and the step may include:

and in the updated state value corresponding to the target hyper-parameter combination, aiming at each value space, determining the result obtained after normalization processing is carried out on the state value of the value space as the value score of the value space.

For example, the value score S of the ith value space can be determined by the following formula_i：

Wherein, V_i' is used for representing the latest state value corresponding to the current value space, and mu (V ') and sigma (V ') respectively represent the mean value and standard deviation corresponding to the updated state value of each value space, that is, the value fraction of each value space can be normalized by the formula.

Then, for each of said value spaces, the rootAnd determining the target score of the value space according to the value score of the value space and the hit times of the value space. Exemplarily, a target Score of the ith value space is determined_iThe formula of (a):

wherein c is a preset constant and is used for adjusting the influence of the hit times on the target score, Mi is the hit times of the value space i, and j is used for representing subscripts of each value space.

As described above, in the embodiment of the present disclosure, the state value of the value space may be determined in the iterative update manner, the state value of each value space is initially 0, and for each value space, after the value space to be updated is determined according to the sampling value combination of the target hyper-parameter combination, the state value of the corresponding value space to be updated may be updated, and the state values of other value spaces except the value space to be updated may remain unchanged. Therefore, in this embodiment, in order to improve the diversity of target space selection in the initial training process, the number of hits in the value space needs to be considered when determining the score of the value space, so as to reduce the influence degree of the values of historical hits on the selection of the target space. Therefore, in the process, as the number of interactive samples increases, the state value corresponding to the target hyper-parameter combination is more accurate, and as the number of hits increases, the influence of the number of hits on the target score is gradually reduced, so that the diversity and the exploration space of target space selection can be improved in the initial learning stage, the accuracy of the determined target value combination can be improved to a certain extent, the overlarge influence of random samples in the initial state is avoided, and when the state value is accurate, the influence of the number of hits on the target space selection is reduced, so that the forward optimization adjustment of the target space selection on the optimized characteristic parameters is ensured.

Turning back to fig. 4, in step 42, a target space is determined from the plurality of value spaces based on the target score for each value space.

In a possible embodiment, the step of determining a target space from the plurality of value spaces according to the target score of each value space may include:

and determining the value space with the maximum target score as the target space.

In the embodiment of the disclosure, the value space with the largest target score can be directly selected as the target space, so that effective adjustment of the target value combination determined from the target space to the optimization of the target model can be effectively ensured, and the efficiency of the optimization of the target model is improved.

In another possible embodiment, the step of determining a target space from the plurality of value spaces according to the target score of each value space may include:

and performing softmax processing on the target scores corresponding to the plurality of value spaces to obtain probability distribution formed by probability information of the plurality of value spaces, sampling the plurality of value spaces according to the probability distribution, and determining the value spaces obtained by sampling as the target spaces.

In this embodiment, in order to further improve the diversity of the target hyper-parameter combination value exploration, the state values of the value spaces may be mapped based on the softmax function, so as to map the state values into values in the range of 0 to 1, which are used as probability information of the value space, so as to obtain probability distributions of the value spaces. When sampling is carried out based on probability distribution, the probability that the value space with smaller probability information is sampled can be achieved, so that the possibility that a plurality of value intervals are sampled can be guaranteed to a certain extent, the problem that the determined target space is used for enabling the characteristic optimization parameters to be in the locally optimal parameters is solved, training of the target model is prevented from being stopped due to the fact that the training reaches the locally optimal parameters, and the accuracy and the robustness of the training of the target model can be guaranteed.

The disclosure also provides a deep reinforcement learning framework, and the value of the hyper-parameter in the deep reinforcement learning framework is determined based on the hyper-parameter determination method. By way of example, the game artificial intelligence can be trained based on the deep reinforcement learning framework, and then through the technical scheme, the accuracy of the determined game artificial intelligence decision can be guaranteed, the decision-making capability of the game artificial intelligence is improved when the game artificial intelligence interacts with a user, and the user interaction experience is improved.

The present disclosure also provides a hyper-parameter determination apparatus, as shown in fig. 5, the apparatus 10 includes:

an obtaining module 100, configured to obtain a sample corresponding to a target hyper-parameter combination of a target model under the sample value combination corresponding to the target hyper-parameter combination, where the target hyper-parameter combination includes a plurality of target hyper-parameters having an association relationship, and the sample value combination includes a sample value corresponding to each target hyper-parameter;

a generating module 200, configured to generate an interaction sample corresponding to the target hyper-parameter combination according to the sampling sample, where the interaction sample includes the sampling value combination and an optimized feature parameter corresponding to the target model;

an updating module 300, configured to update a state value corresponding to the target hyper-parameter combination according to the interaction sample, where the target hyper-parameter corresponds to a dimension in a target hyper-space corresponding to the target hyper-parameter combination, and a parameter space of each target hyper-parameter is discretized into a plurality of value intervals in the dimension corresponding to the target hyper-parameter, so that the target hyper-space is discretized into a plurality of value spaces;

a first determining module 400, configured to determine a target space from the multiple value spaces according to the updated state value corresponding to the target hyper-parameter combination;

a second determining module 500, configured to determine, according to the target space, a target value combination corresponding to the target hyper-parameter combination.

Optionally, the apparatus further comprises:

and the third determining module is used for taking the target value combination as a new sampling value combination corresponding to the target hyper-parameter combination, and triggering the acquiring module to execute the acquisition of the sampling sample corresponding to the sampling value combination under the sampling value combination corresponding to the target hyper-parameter combination of the target model to the second determining module to determine the target value combination corresponding to the target hyper-parameter combination according to the target space until the training of the target model is completed.

Optionally, the update module includes:

the first determining submodule is used for determining a value space to which the sampling value combination belongs as a value space to be updated according to the sampling value combination;

and the first updating submodule is used for updating the state value of the value space to be updated according to the optimization characteristic parameter.

Optionally, the target hyperspace corresponds to a plurality of reference hyperspaces, a discrete interval corresponding to a value space in each reference hyperspace is greater than a discrete interval corresponding to a value space in the target hyperspace, and an origin corresponding to each reference hyperspace is different;

the update module includes:

a second determining submodule, configured to determine, according to the sampling value combination, a reference value space to which the sampling value combination belongs in each of the reference hyperspaces, and a value space to be updated to which the sampling value combination belongs in the target hyperspace;

the second updating submodule is used for updating the state value of each reference value space according to the optimization characteristic parameters;

and the third updating submodule is used for updating the state value of the value space to be updated according to the updated state value corresponding to each reference hyperspace.

Optionally, the third update sub-module includes:

a third determining submodule, configured to determine a mapping value space corresponding to the value space to be updated in each reference hyperspace;

and the fourth determining submodule is used for determining the state value of the value space to be updated according to the state value of each mapping value space.

Optionally, the first determining module includes:

a fifth determining submodule, configured to determine a target score of each value space according to the updated state value corresponding to the target hyper-parameter combination;

and the sixth determining submodule is used for determining a target space from the plurality of value spaces according to the target fraction of each value space.

Optionally, the fifth determining sub-module includes:

a seventh determining submodule, configured to determine, in the updated state value corresponding to the target hyper-parameter combination, a value score of each value space by performing normalization processing on the state value of the value space for each value space;

and the eighth determining submodule is used for determining the target score of the value space according to the value score of the value space and the hit times of the value space aiming at each value space.

Optionally, the sixth determining sub-module includes:

a ninth determining submodule, configured to determine a value space with a largest target score as the target space; or

And the tenth determining submodule is used for performing softmax processing on the target scores of the plurality of value spaces to obtain probability distribution formed by probability information of each value space, sampling the plurality of value spaces according to the probability distribution, and determining the value space obtained by sampling as the target space.

Optionally, the target model is a deep reinforcement learning model, the sampling samples are interaction sequences obtained by sampling in a process of interacting a virtual object with a virtual environment, where the virtual object is controlled based on the deep reinforcement learning model, the interaction sequences include a plurality of sampling data under the sampling value combination corresponding to the target hyper-parameter combination, each sampling data includes an environment state of the virtual environment, a decision action performed by the virtual object in the environment state determined by the deep reinforcement learning model, and a return value corresponding to the decision action, and the optimized feature parameter is an accumulated return corresponding to the interaction sequences.

Optionally, the virtual object is game artificial intelligence, the deep reinforcement learning model is used for training the game artificial intelligence, the sampling sample is an interaction sequence obtained by sampling the game artificial intelligence in a game play of a target game, and the virtual environment is a training environment in which the game artificial intelligence is located in the target game.

Referring now to FIG. 6, a block diagram of an electronic device 600 suitable for use in implementing embodiments of the present disclosure is shown. The terminal device in the embodiments of the present disclosure may include, but is not limited to, a mobile terminal such as a mobile phone, a notebook computer, a digital broadcast receiver, a PDA (personal digital assistant), a PAD (tablet computer), a PMP (portable multimedia player), a vehicle terminal (e.g., a car navigation terminal), and the like, and a stationary terminal such as a digital TV, a desktop computer, and the like. The electronic device shown in fig. 6 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present disclosure.

As shown in fig. 6, electronic device 600 may include a processing means (e.g., central processing unit, graphics processor, etc.) 601 that may perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM)602 or a program loaded from a storage means 608 into a Random Access Memory (RAM) 603. In the RAM 603, various programs and data necessary for the operation of the electronic apparatus 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input/output (I/O) interface 605 is also connected to bus 604.

Generally, the following devices may be connected to the I/O interface 605: input devices 606 including, for example, a touch screen, touch pad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 607 including, for example, a Liquid Crystal Display (LCD), a speaker, a vibrator, and the like; storage 608 including, for example, tape, hard disk, etc.; and a communication device 609. The communication means 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. While fig. 6 illustrates an electronic device 600 having various means, it is to be understood that not all illustrated means are required to be implemented or provided. More or fewer devices may alternatively be implemented or provided.

In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a non-transitory computer readable medium, the computer program containing program code for performing the method illustrated by the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network via the communication means 609, or may be installed from the storage means 608, or may be installed from the ROM 602. The computer program, when executed by the processing device 601, performs the above-described functions defined in the methods of the embodiments of the present disclosure.

It should be noted that the computer readable medium in the present disclosure can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In contrast, in the present disclosure, a computer readable signal medium may comprise a propagated data signal with computer readable program code embodied therein, either in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, optical cables, RF (radio frequency), etc., or any suitable combination of the foregoing.

In some embodiments, the clients, servers may communicate using any currently known or future developed network Protocol, such as HTTP (HyperText Transfer Protocol), and may interconnect with any form or medium of digital data communication (e.g., a communications network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), the Internet (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed network.

The computer readable medium may be embodied in the electronic device; or may exist separately without being assembled into the electronic device.

The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to: acquiring a sampling sample corresponding to a sampling value combination under the sampling value combination corresponding to a target hyper-parameter combination of a target model, wherein the target hyper-parameter combination comprises a plurality of target hyper-parameters with an association relation, and the sampling value combination comprises a sampling value corresponding to each target hyper-parameter; generating an interactive sample corresponding to the target hyper-parameter combination according to the sampling sample, wherein the interactive sample comprises the sampling value combination and an optimized characteristic parameter corresponding to the target model; updating the state value corresponding to the target hyper-parameter combination according to the interaction sample, wherein the target hyper-parameter corresponds to the dimension in the target hyper-space corresponding to the target hyper-parameter combination one by one, and the parameter space of each target hyper-parameter is discretized into a plurality of value intervals under the dimension corresponding to the target hyper-parameter, so that the target hyper-space is discretized into a plurality of value spaces; determining a target space from the plurality of value spaces according to the updated state value corresponding to the target hyper-parameter combination; and determining a target value combination corresponding to the target hyper-parameter combination according to the target space.

Computer program code for carrying out operations for the present disclosure may be written in any combination of one or more programming languages, including but not limited to an object oriented programming language such as Java, Smalltalk, C + +, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The modules described in the embodiments of the present disclosure may be implemented by software or hardware. For example, the obtaining module may be further described as a module that obtains a sample corresponding to a sample value combination under the sample value combination corresponding to the target hyper-parameter combination of the target model.

The functions described herein above may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), systems on a chip (SOCs), Complex Programmable Logic Devices (CPLDs), and the like.

In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

Example 1 provides a hyper-parameter determination method, according to one or more embodiments of the present disclosure, wherein the method comprises:

Example 2 provides the method of example 1, wherein the method further comprises:

Example 3 provides the method of example 1, wherein the updating, according to the interaction sample, the state value corresponding to the target hyper-parameter combination includes:

determining a value space to which the sampling value combination belongs as a value space to be updated according to the sampling value combination;

and updating the state value of the value space to be updated according to the optimization characteristic parameters.

Example 4 provides the method of example 1, wherein the target hyperspace corresponds to a plurality of reference hyperspaces, a discrete interval corresponding to a value space in each of the reference hyperspaces is greater than a discrete interval corresponding to a value space in the target hyperspace, and an origin corresponding to each of the reference hyperspaces is different;

the updating the state value corresponding to the target hyper-parameter combination according to the interaction sample comprises:

according to the sampling value combination, determining a reference value space to which the sampling value combination belongs in each reference hyperspace and a value space to be updated to which the sampling value combination belongs in the target hyperspace;

updating the state value of each reference value space according to the optimization characteristic parameters;

and updating the state value of the value space to be updated according to the updated state value corresponding to each reference hyperspace.

Example 5 provides the method of example 4, wherein updating the state value of the value space to be updated according to the updated state value corresponding to each reference hyperspace includes:

determining a mapping value space corresponding to the value space to be updated in each reference hyperspace;

Example 6 provides the method of example 1, wherein determining a target space from the plurality of value spaces according to the updated state value corresponding to the target hyper-parameter combination includes:

determining a target score of each value space according to the updated state value corresponding to the target hyper-parameter combination;

and determining a target space from the plurality of value spaces according to the target fraction of each value space.

Example 7 provides the method of example 6, wherein the determining a target score for each value space according to the updated state value corresponding to the target hyper-parameter combination includes:

in the updated state value corresponding to the target hyper-parameter combination, for each value space, determining the result obtained after normalization processing of the state value of the value space as the value score of the value space;

and aiming at each value space, determining a target score of the value space according to the value score of the value space and the hit times of the value space.

Example 8 provides the method of example 6, wherein the determining a target space from the plurality of value spaces according to the target score of each of the value spaces, includes:

determining the value space with the maximum target score as the target space;

or

And performing softmax processing on the target fractions of the value spaces to obtain probability distribution formed by probability information of each value space, sampling the value spaces according to the probability distribution, and determining the value space obtained by sampling as the target space.

Example 9 provides the method of example 1, where the target model is a deep reinforcement learning model, the sampling samples are interaction sequences obtained by sampling in a process of interacting a virtual object with a virtual environment, the virtual object is controlled based on the deep reinforcement learning model, the interaction sequences include a plurality of sampling data under the sampling value combination corresponding to the target hyper-parameter combination, each sampling data includes an environmental state of the virtual environment, a decision action performed by the virtual object in the environmental state determined by the deep reinforcement learning model, and a return value corresponding to the decision action, and the optimized feature parameter is an accumulated return corresponding to the interaction sequences.

Example 10 provides the method of example 9, wherein the virtual object is game artificial intelligence, the deep reinforcement learning model is used for training the game artificial intelligence, the sample is an interaction sequence obtained by sampling the game artificial intelligence in a game play of a target game, and the virtual environment is a training environment in which the game artificial intelligence is located in the target game.

Example 11 provides, in accordance with one or more embodiments of the present disclosure, a hyper-parameter determination apparatus, the apparatus comprising:

Example 12 provides a deep reinforcement learning framework in which values of a plurality of hyper-parameters having an association relationship are determined based on the hyper-parameter determination method in any one of examples 1 to 10, according to one or more embodiments of the present disclosure.

Example 13 provides a computer-readable medium having stored thereon a computer program that, when executed by a processing apparatus, performs the steps of the method of any of examples 1-10, in accordance with one or more embodiments of the present disclosure.

Example 14 provides, in accordance with one or more embodiments of the present disclosure, an electronic device, comprising:

a storage device having a computer program stored thereon;

processing means for executing the computer program in the storage means to carry out the steps of the method of any of examples 1-10.

The foregoing description is only exemplary of the preferred embodiments of the disclosure and is illustrative of the principles of the technology employed. It will be appreciated by those skilled in the art that the scope of the disclosure herein is not limited to the particular combination of features described above, but also encompasses other embodiments in which any combination of the features described above or their equivalents does not depart from the spirit of the disclosure. For example, the above features and (but not limited to) the features disclosed in this disclosure having similar functions are replaced with each other to form the technical solution.

Further, while operations are depicted in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Likewise, while several specific implementation details are included in the above discussion, these should not be construed as limitations on the scope of the disclosure. Certain features that are described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination.

Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims. With regard to the apparatus in the above-described embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.

Claims

1. A method for hyper-parameter determination, the method comprising:

2. The method of claim 1, further comprising:

3. The method of claim 1, wherein updating the state value corresponding to the target hyper-parameter combination according to the interaction sample comprises:

4. The method according to claim 1, wherein the target hyperspace corresponds to a plurality of reference hyperspaces, a discrete interval corresponding to a value space in each reference hyperspace is larger than a discrete interval corresponding to a value space in the target hyperspace, and an origin corresponding to each reference hyperspace is different;

5. The method according to claim 4, wherein the updating the state value of the value space to be updated according to the updated state value corresponding to each reference hyperspace comprises:

6. The method of claim 1, wherein determining a target space from the plurality of value spaces according to the updated state value corresponding to the target hyper-parameter combination comprises:

7. The method of claim 6, wherein the determining the target score for each value space according to the updated state value corresponding to the target hyper-parameter combination comprises:

8. The method of claim 6, wherein determining a target space from the plurality of value spaces based on the target score for each of the value spaces comprises:

determining the value space with the maximum target score as the target space;

or

9. The method according to claim 1, wherein the target model is a deep reinforcement learning model, the sampling samples are interaction sequences obtained by sampling in a process of interacting a virtual object with a virtual environment, wherein the virtual object is controlled based on the deep reinforcement learning model, the interaction sequences include a plurality of sampling data under the sampling value combinations corresponding to the target hyper-parameter combinations, each of the sampling data includes an environment state of the virtual environment, a decision action performed by the virtual object in the environment state determined by the deep reinforcement learning model, and a return value corresponding to the decision action, and the optimized feature parameter is an accumulated return corresponding to the interaction sequences.

10. The method of claim 9, wherein the virtual object is game artificial intelligence, the deep reinforcement learning model is used for training the game artificial intelligence, the sampling sample is an interaction sequence obtained by sampling the game artificial intelligence in a game play of a target game, and the virtual environment is a training environment in which the game artificial intelligence is located in the target game.

11. A hyper-parameter determination apparatus, the apparatus comprising:

12. A deep reinforcement learning framework, wherein values of a plurality of hyper-parameters having an association relationship in the deep reinforcement learning framework are determined based on the hyper-parameter determination method according to any one of claims 1 to 10.

13. A computer-readable medium, on which a computer program is stored, characterized in that the program, when being executed by processing means, carries out the steps of the method of any one of claims 1-10.

14. An electronic device, comprising:

a storage device having a computer program stored thereon;

processing means for executing the computer program in the storage means to carry out the steps of the method according to any one of claims 1 to 10.