TWI705387B - Reinforcement learning system and its servo device and reinforcement learning method - Google Patents

Abstract

A reinforcement learning system includes at least one agent device and a server device. According to a set condition, the at least one proxy device transmits a data state set related to the state of the environment, receives a data action set for performing an action, and transmits a data feedback message generated after interacting with the environment to the server device through the network . The server device allocates a predetermined ratio of memory space as at least one workstation according to the setting condition, and selects a training model to be temporarily stored in the at least one workstation, and the at least one workstation uses the current state set, action set and feedback message as parameters , After importing the training model, perform reinforcement learning and generate the next action set to achieve the goal. In this way, the present invention uses hardware resources located in the cloud and configured to perform training. Not only can learning efficiency be improved, but also the modular design can make the reinforcement learning architecture easier to reuse and update, and apply different Acting device to enhance the scope of application and practicality.

Description

Reinforcement learning system and its servo device and reinforcement learning method

The invention relates to a reinforcement learning system, in particular to a reinforcement learning system for reinforcement learning training and its servo device and reinforcement learning method.

"Reinforcement learning (RL)" can be said to be the hottest direction in the field of artificial intelligence. The reason for its popularity is related to the great success of the DeepMind team on AlphaGo and AlphaZero.

Reinforcement learning is a way close to human learning, emphasizing how to act based on the environment to maximize the expected benefits. Take a white mouse in the laboratory how to learn how to operate a lever to obtain food as an example: the white mouse is an agent that makes decisions, its initial state, because there is no opportunity to explore the environment , Their behavior is random and non-target-oriented at first, until they inadvertently touch the lever in the specially set environment, and from the action of pulling the lever, they accidentally obtain food, which is the reward for this behavior. (Reward), driven by the brain’s reward mechanism, the white mice began to adopt a goal-oriented learning method. In order to get more food rewards, they may gather next to the lever and keep trying until they learn to pull the lever correctly. So far.

Since reinforcement learning must act based on the environment, all architectures are set on the agent device. Not only is the learning efficiency limited by the resources and performance of the agent itself, but also importantly, for each agent As far as the device (agent) is concerned, all hardware architectures and software resources related to reinforcement learning must be configured, and once the learning objectives are different, the original reinforcement learning architecture cannot be reused or updated, but in the scope of application and practicality In terms of sex, there is still room for substantial improvement.

Therefore, the purpose of the present invention is to provide a reinforcement learning system and its servo device and reinforcement learning method that can improve learning efficiency, and improve the scope and practicability.

Thus, the reinforcement learning system and its servo device and reinforcement learning method of the present invention.

The effect of the present invention is that the present invention uses hardware resources located in the cloud and configured for training, which can not only improve learning efficiency, but also has a modular design that makes it easier to reuse and update the reinforcement learning architecture. Applicable to different agent devices to improve the scope of application and practicality.

1 and FIG. 2, an embodiment of the reinforcement learning system of the present invention includes a number agent device 1 and a server device 2.

According to a set condition 10, each agent device 1 transmits a data state set 11 related to the state of the environment, receives a data action set 12 for performing an action, and transmits the data after interacting with the environment. A feedback message 13 is generated, and a stop signal S used to stop the action. The setting condition 10 is used to achieve a goal. The feedback message 13 includes the reward value R (reward) for the aforementioned action.

It should be noted that the number of the proxy devices 1 is not limited to multiple, and may be one in other variations of this embodiment.

The servo device 2 includes a connection unit 21, a storage unit 22, and a processing unit 23.

In this embodiment, the connection unit 21 communicates with the proxy devices 1 through communication technology, and is used to receive the setting conditions 10, the state sets 11, and the feedback messages 13 of each proxy device 1. And send the action sets 12 to each agent device 1.

In this embodiment, the storage unit 22 is a memory or storage medium, and includes a training model pool 221 for storing a trained model (Trained Model), and an estimate for storing a data estimation model (Untrained Model). The measurement model pool 222, an action noise model pool 223 for storing an action noise module, and a buffer data model pool 224 for storing data buffer data models. In this embodiment, these training models include but are not limited to DQN module, DDPG module, A3C module, PPO module, Q-learning module. Each estimation model is one of the training models that has completed training and can achieve the goal. The motion noise model is used to increase the exploration of the environment by the at least one agent device 1, including but not limited to ε greedy module and Uhlenbeck module. The buffer data model is used to determine the way of data access and temporary storage, including but not limited to replay memory module and simple buffer module.

The processing unit 23 includes a calculation module 231 and a processing module 232 connected to the connection unit 21, the storage unit 22 and the calculation module 231.

In this embodiment, the computing module 231 may be composed of more than one central processing unit (Central Processing Unit), or composed of more than one graphics processing unit (Graphics Processing Unit), or composed of more than one It is composed of central processing unit and graphics processor. The calculation module 231 also has a memory space 233 and a number operation core 234.

In this embodiment, the processing module 232 is composed of more than one central processing unit (Central Processing Unit), and will allocate a predetermined ratio of memory space 233 and a predetermined number according to the setting condition 10 of each agent device 1. The computing core 234 of is more than one workstation 20 (worker).

Refer to Figure 2 and Figure 3, for the convenience of description below, the numbers 1a, 2a, 3a-1, 3a-2, 3a-3 to distinguish the proxy devices 1 correspond to the numbers 1a, 2a, 3a-1, 3a- 2. The environment of 3a-3, and the workstations 20 numbered 1a, 2a, 3a-1, 3a-2, and 3a-3. Taking the environment of number 1a as a carpole 3 as an example, how to pass Move the pendulum bike 3 to the left or right, so that a pendulum 31 on the pendulum bike 3 can maintain standing without falling down. The setting conditions 10 are shown in Table 1.

Table 1:

Referring to Figure 4, Figure 1 and Figure 2, the reinforcement learning method of this embodiment implements the following steps through the servo device 2:

Step 401: Start.

Step 402: Establish a connection with the proxy device 1 numbered 1a through the connection unit 21, so that the server device 2 and the proxy device 1 numbered 1a communicate with each other.

Step 403: The processing module 232 allocates a predetermined ratio of memory space 233 and a predetermined number of computing cores 234 to more than one workstation 20 according to the setting condition 10 of the agent device 1 numbered 1a.

It should be noted that the setting condition 10 is not limited to using 10% of the memory space 233 as shown in Table 1, and one computing core 234 and one workstation 20. In other variations of this embodiment, 20% can also be used. ~100% memory space 233, and P computing cores 234, K workstations 20, each workstation 20 uses 20~100%/k of memory space 233, and P≦K.

Step 404: The processing module 232 generates a unique identification code U according to the connected proxy device 1, and the identification code U and the selected training model in the setting condition 10, the selected buffer data model and the The motion noise model is temporarily stored in the corresponding workstation 20.

It is worth noting that the identification code U includes a universally unique identifier (UUID) U1 and a primary number (room_id) U2. Different main number strings U1 represent different agent devices 1 in different environments. Different sub-digit strings U2 represent different agent devices 1 performing different actions in the same environment. If there is only one agent device 1 performing one type of action in the same environment, the sub-digit strings U2 are omitted.

As shown in Figure 2, the environment numbered 1a is the pendulum bike 3, and the identification code U of the agent device 1 corresponding to the environment numbered 1a is /ikSYiGCPfUVSNbs5NmhQHe/, and the environment numbered 2a is a brick-and-mortar game. The identification code U of the number 2a environment and the number 2a proxy device 1 is /F6bdVeRR86gsm6rJtLz3af/, the number 3-1a environment, the number 3-2a environment, and the number 3-3a environment are the same obstacle avoidance space, corresponding to number 3. The identification code U of the agent device 1 in the environment of -1a and number 3-1a is /CjTYQXbCbdKQ9ssi79Z7C7/01/, and the identification code U of the agent device 1 corresponding to the environment of number 3-2a and number 3-2a is /CjTYQXbCbdKQ9ssi79Z7C7/02/ , The identification code U of the proxy device 1 corresponding to the environment numbered 3-3a and numbered 3-3a is /CjTYQXbCbdKQ9ssi79Z7C7/03/.

Step 405: The processing module 232 sends the identification code U to the proxy device 1 with the number 1a through the connection unit 21, so that the status set 11, the action set 12, and the feedback message 13 are stored in the same The identification code U is transmitted between the proxy device 1 and the corresponding workstation 20, that is, between the proxy device 1 with the number 1a and the workstation 20 with the number 1a.

Step 406: The processing module 232 judges whether it is a reward mode according to the setting condition 10? If yes, go to step 500, if not, go to step 407.

Step 407: The processing module 232 determines whether it is the estimation mode according to the setting condition 10? If yes, proceed to step 600, if not, proceed to step 408.

Step 408: End.

Referring to FIG. 5, FIG. 2, FIG. 6 and FIG. 7, where step 500 includes:

Step 501: Start.

Step 502: Receive the initial state set 11 from the proxy device 1 numbered 1a through the connection unit 21. The state set 11 is obtained by the agent device 1 observing the environment.

Step 503: The workstation 20 numbered 1a temporarily stores the state set 11 through the memory space 233, and uses the initial state set 11 as a parameter through the computing core 234 to generate the initial action set 12 after importing the selected training model.

Step 504: The workstation 20 of number 1a transmits the action set 12 to the agent device 1 of number 1a through the connection unit 21, so that the agent device 1 of number 1a interacts with the environment of number 1a according to the action set 12, thereby changing the number 1a As shown in Figure 3, the agent device 1 numbered 1a changes the position, speed, and speed of the pendulum 3 by pushing the pendulum 3 one step to the left or one step to the right. The angle, speed and other status of the camera.

It is worth noting that the position and speed of the pendulum 3, the angle and speed of the pendulum 31, etc. are detected and observed by the sensor (not shown) of the agent device 1 After the position of the pendulum 31, the speed of the pendulum car 3 and the pendulum 31 are calculated. The aforementioned state-obtained technology has been disclosed in the prior art of reinforcement learning, and is not a technical feature of the application in this case. Since a person with ordinary knowledge in the field can infer the details of the expansion based on the above description, no further description is given.

Step 505: The workstation 20 numbered 1a judges whether to receive the stop signal S through the connection unit 21? If yes, proceed to step 506, which indicates that the angle of the simple pendulum 31 has exceeded ±12° and may have fallen down, or indicates that the The position of the pendulum bike 3 exceeds ±2.4m (meters), and it is out of the detectable range, or it means that the number of steps in this round has exceeded 200 steps. If not, continue to judge.

Step 506: The workstation 20 with the number 1a receives the feedback message 13 and the next state set 11 sent by the proxy device 1 with the number 1a through the connection unit 21, and obtains the reward value R (reward) for the aforementioned action. The aforementioned next state set 11 is generated by the agent device 1 numbered 1a interacting with the environment numbered 1a with the initial action set 12 in step 504, for example, the position or speed of the pendulum bike 3 changes, or the pendulum 31 The angle or speed changes.

Step 507: The workstation 20 numbered 1a temporarily stores the current action set 12, the feedback message 13 generated corresponding to the current action set 12, and the next state set 11 through the memory space 233, and uses the current action set through the computing core 234 Set 12, corresponding to the feedback message 13 generated by the current action set 12 and the next state set 11 as parameters, import the training model for reinforcement learning, and generate the next action set 12.

Step 508: Workstation 20 numbered 1a, through the computing core 234, judge whether the pendulum bike 3 has more than 200 steps for 10 consecutive rounds, and the goal is reached? If yes, go to step 509, if not, go back to step 504.

Step 509: End.

Refer to Figure 2, Figure 8 and Figure 9, where step 600 includes:

Step 601: Start.

Step 602: Receive the state set 11 from the proxy device 1 numbered 1a through the connection unit 21.

Step 603: The workstation 20 numbered 1a temporarily stores the state set 11 through the memory space 233, and uses the state set 11 as a parameter through the computing core 234 to import the selected estimation model to generate the action set 12.

Step 604: The workstation 20 of number 1a transmits the action set 12 to the agent device 1 of number 1a through the connection unit 21, so that the agent device 1 of number 1a interacts with the environment of number 1a according to the action set 12, and then changes the number 1a The state of the environment.

Step 605: Workstation 20 numbered 1a, through the computing core 234, judge whether the pendulum bike 3 has more than 200 steps for 10 consecutive rounds, and the goal is reached? If yes, go to step 606, if not, go back to step 602.

Step 606: End.

10 and FIG. 11, in addition, it is worth noting that in step 404, the training model is not limited to be selected by the setting condition 10. In other variations of this embodiment, the processing module 232 uses the number of connected agent devices 1 and the number of computing cores 234 as parameters, and is determined by a sampling calculation model, and automatically finds the best training model. In this embodiment, the sampling calculation model is the Thompson Sampling algorithm, taking P arithmetic cores 234 and K workstations 20 as an example, and the description is as follows:

Assuming K=4, P=1, when automatically searching for the best training model and suitable workstation 20, the processing module 232 selects only one workstation 20 and one computing core 234 for sampling at a time to obtain an expected value, and then Select the workstation 20 with the highest expectation and its training model.

Assuming K=4, P=2, when automatically searching for the best training model and suitable workstation 20, the processing module 232 selects any two workstations 20 and two computing cores 234 for sampling each time, and selects the expected value One of the highest workstations 20 is compared with other workstations 20, and then the workstation 20 with the highest expected value and its training model are selected.

In this way, since the final selection of the sampling calculation model is the workstation 20 with a higher expected value, a training model that is more suitable for the environment can be selected, thereby improving the learning efficiency.

It is worth noting that the aforementioned Thompson Sampling algorithm is a widely used algorithm at present. Since those with ordinary knowledge in the field can infer the details of the expansion based on the above description, no further explanation is given.

Based on the above description, the advantages of the foregoing embodiments can be summarized as follows:

1. The present invention divides the reinforcement learning system into an agent device 1 (agent) that only needs to be able to observe the environment and interact with the environment, and a server device 2 that is located in the cloud and configured hardware resources for training. The hardware architecture and software resources of each agent device 1 are simplified, and the learning efficiency can be greatly improved through the server device 2.

2. The present invention can modularize the workstation 20, training model, and motion noise model in the cloud through the aforementioned special design, so that the aforementioned modular workstation 20, training model, and motion noise model can be repeated Being used or updated, the server device 2 of the present invention can be applied to different agent devices 1 and effectively improve the use efficiency.

3. The present invention can use the aforementioned modular design to directly update the trained training model to an estimation model, perform final verification, and further improve learning efficiency.

4. In addition, in the present invention, the processing module 232 can use the number of connected agent devices 1 and the number of computing cores 234 as parameters to determine the sampling calculation model and automatically find the best training model. , And a suitable workstation 20 to further improve learning efficiency.

However, the above are only examples of the present invention. When the scope of implementation of the present invention cannot be limited by this, all simple equivalent changes and modifications made in accordance with the scope of the patent application of the present invention and the content of the patent specification still belong to Within the scope of the patent for the present invention.

1: Agent device 10: Setting conditions 11: State set 12: Action set 13: Feedback message 2: Servo device 20: Workstation 21: Connection unit 22: Storage unit 221: Training model pool 222: Estimated model pool 223: Action Noise model pool 224: buffer data model pool 23: processing unit 231: calculation module 232: processing module 233: memory space 234: computing core 401~408: steps 501~509: steps 601~606: step R: Reward value S: suspension signal

Other features and effects of the present invention will be clearly presented in the implementation with reference to the drawings, in which: Figure 1 is a block diagram illustrating an embodiment of the reinforcement learning system of the present invention; Figure 2 is a block diagram , It is explained that the number state set, the number action set and the number feedback message in this embodiment can be transmitted between the agent device and the corresponding workstation storing the same identification code; Figure 3 is a schematic diagram of a pendulum car; Figure 4 is a flow chart illustrating the step flow of the reinforcement learning method of the present invention combined with this embodiment; Figure 5 is a flow chart illustrating the step flow of a reward mode in this embodiment; Figure 6 is a block diagram, Explain the relationship between the reward mode and time in this embodiment; Figure 7 is a schematic diagram illustrating the modularized buffer data model, training model, estimation model, and action noise model in this embodiment; Figure 8 is a picture A flowchart illustrating the flow of steps in an estimation mode in this embodiment; Fig. 9 is a block diagram illustrating the relationship between the estimation mode and time in this embodiment; Fig. 10 is a schematic diagram illustrating the procedure in this embodiment Take one computing core as an example to automatically find the best training model; and FIG. 11 is a schematic diagram illustrating that two computing cores are used as an example in this embodiment to automatically find the best training model.

1: proxy device

10: Setting conditions

11: Status set

12: Action set

13: Feedback

2: Servo device

223: Motion Noise Model Pool

224: buffer data model pool

23: processing unit

231: Computing Module

232: Processing module

233: memory space

20: Workstation

21: Connection unit

22: storage unit

221: training model pool

234: Computing Core

R: reward value

S: Stop signal

Claims (12)

A server device of a reinforcement learning system is connected to at least one agent device. The at least one agent device transmits a number state set related to the state of the environment, receives a number action set for performing an action, and transmits an execution action according to a set condition The data feedback message generated after interacting with the environment during the process. The setting condition is used to achieve a goal. The server device includes: a connection unit that communicates with the at least one proxy device through a network to receive the at least The setting conditions of an agent device, the state sets and the feedback messages, and sending the action sets to the at least one agent device; a storage unit storing a number of training models; and a processing unit including a calculation module, And a processing module connected to the connection unit, the storage unit, and the calculation module. The calculation module has a memory space and a number operation core. The processing module configures a predetermined ratio of The memory space and the predetermined number of computing cores are at least one workstation, and after entering a reward mode, one of the training models is selected to be temporarily stored in the at least one workstation, and the at least one workstation is imported into the at least one workstation with the initial state set as a parameter After a training model, an initial action set is generated. Then, using the current action set, the feedback message corresponding to the current action set and the next state set as parameters, import one of the training models for reinforcement learning, and then generate the next one Action set, to achieve the goal, the training model selected by the processing module is determined by the processing module through a sampling calculation model according to the number of connected proxy devices and the number of computing cores. The server device of the reinforcement learning system according to claim 1 is connected to a data agent device, wherein the processing module is configured with a predetermined proportion of memory space as a data workstation, and when the connection unit is connected to each agent device, the The processing module generates a unique identification code based on each of the connected agent devices, and transmits each identification code to each agent device through the connection unit, and temporarily stores the identification code in the corresponding workstation. The code is used to enable the state set, the action set and the feedback message to be transmitted between each agent device storing the same identification code and the corresponding workstation. The identification code includes a primary digit string and a primary digit string, Different primary numeric strings represent different agent devices in different environments. The processing module selects different training models according to different agent devices. Different secondary numeric strings represent different agent devices in the same environment. The processing module According to different agent devices performing different actions in the same environment, different training models are selected. The server device of the reinforcement learning system according to claim 2, wherein the storage unit includes a training model pool and an estimation model pool, the training model pool is used to store the training models, and each workstation also Further, after the corresponding agent device completes the target, one of the training models that have completed the target is updated as an estimated model and stored in the estimated model pool. The servo device of the reinforcement learning system according to claim 3, wherein the storage unit further stores at least one action noise module for increasing environmental exploration, and the processing module further stores at least one action noise module. The motion noise model is temporarily stored in the at least one workstation, and the at least one workstation further uses each action set as a parameter to import the motion noise model to optimize each action set. The servo device of the reinforcement learning system according to claim 4, wherein the storage unit further stores a number of buffer data models, and the buffer data models are used to determine data access and temporary storage methods, and the processing module further One of the buffer data models is temporarily stored in the at least one workstation, so that the selected training model, the selected buffer data model, and the motion noise model are temporarily stored in the at least one workstation. The server device of the reinforcement learning system according to claim 3, wherein each workstation further stores the identification code and the estimation model in the estimation model pool after the corresponding agent device completes the target, and generates According to the corresponding relationship, the processing module further enters an estimation mode according to the set conditions, and selects the estimation model corresponding to the same identification code according to the identification code of each agent device, and temporarily stores it in the corresponding workstation, and At least one workstation further uses the current state set as a parameter to generate an initial action set through the estimation model, and then uses the next state set as a parameter to generate a next action set to achieve the goal. The servo device of the reinforcement learning system according to claim 1, wherein the sampling calculation model is a Thompson Sampling algorithm. A reinforcement learning method for enabling at least one agent device to implement the following steps: (a) communicate with a server device via a network according to a set condition, the set condition is used to achieve a goal; (b) the set condition Send to the server device, so that the server device and the at least one agent device enter a reward mode according to the set condition; (c) send a state set related to the state of the environment to the server device (D) receiving an action set for performing actions, the action set is generated by at least one workstation of the servo device using the state set of step (c) as a parameter, after importing a training model; (e) according to the current The action set interacts with the environment, and sends a feedback message and the next state set to the at least one workstation; (f) receives the next action set for executing the action, and the next action set is performed by the at least one workstation in steps ( The action set of e), the feedback message of step (e) and the state set of step (e) are parameters, which are generated after importing the training model for reinforcement learning; (g) repeat steps (e) ~ step (f), to Achieve that goal. The reinforcement learning method as described in claim 8, further including step (h) to step (l) after step (g), and step (b) includes: (b-1) sending the setting condition to the servo Device; (b-2) Determine whether it is the reward mode, if yes, proceed to step (c), if not, proceed to step (h); (h) send the state set to the servo device; (i) receive the action The action set is generated by the at least one workstation after importing an estimation model with the state set of step (h) as a parameter; (j) interacting with the environment according to the current action set, and sending the next state set to the At least one workstation; (k) receiving the next action set, which is generated by the at least one workstation after importing the estimation model with the state set of step (j) as a parameter; (l) Repeat step (j) ~ step (k) to achieve the goal. An enhanced learning method that implements the following steps through a server device: (a) communicating with at least one agent device via a network; (b) receiving a setting condition from the at least one agent device, and the setting condition is used to achieve a Objective; (c) According to the setting condition, allocate a predetermined proportion of memory space as at least one workstation; (d) According to the setting condition, enter a reward mode, and select a training model to temporarily store in the at least one workstation; (e) ) Receive an initial state set from the at least one agent device and related to the state of the environment; (f) The at least one workstation uses the initial state set as a parameter to import the training model to generate an initial action set; ( g) Send an action set to the at least one agent device, so that the at least one agent device interacts with the environment according to the action set; (h) Receive a feedback message and the next state. The feedback message and the next state are determined by the at least one The current action set of the agent device is generated after interacting with the environment; (i) The at least one workstation uses the current action set, the feedback message corresponding to the current action set, and the next state set as parameters, and imports the training model for reinforcement learning , And generate the next action set; (j) Repeat step (g) ~ step (i) until the goal is reached; (k) the at least one workstation updates the training model that has completed the goal to an estimation model, so that The identification code has a corresponding relationship with the estimation model. The reinforcement learning method according to claim 10, wherein, in step (a), the server device communicates with the number agent device, and in step (c), the server device configures a predetermined ratio according to the setting condition The memory space of is a number of workstations, and a unique identification code is generated according to each of the agent devices connected, and each identification code is transmitted to each agent device, and temporarily stored in the corresponding workstation, the identification code It is used to transfer the status set, the action set and the feedback message between each agent device storing the same identification code and the corresponding workstation. The identification code includes a primary digit string and a primary digit string. The main number string of represents different agent devices in different environments. In step (d), the server selects different training models according to different agent devices, and different sub-digit strings represent different agents in the same environment. Device, the processing module selects different training models according to different agent devices that perform different actions in the same environment. As described in claim 11, the reinforcement learning method further includes step (l) to step (r) after step (k), and step (d) includes: (d-1) determining whether it is based on the set condition For the reward mode, if yes, proceed to step (d-2), if not, proceed to step (l); (d-2) select a training model to temporarily store in the at least one workstation, and then proceed to step (e); l) According to the setting condition, judge whether it is an estimation mode, if yes, proceed to step (m), if not, proceed to step (r); (m) According to the identification code of the at least one proxy device, select the corresponding identification The training model of the code is temporarily stored in the at least one workstation; (n) receiving the state related to the environment from the at least one agent device State set; (o) the at least one workstation uses the state set as a parameter to generate the action set after importing the estimation model; (p) transmits the action set to the corresponding agent device, so that the corresponding agent device is based on the The action set interacts with the environment; (q) Repeat step (n) ~ step (p) until the goal is reached; (r) end.

Images