KR102590791B1 - Method and apparatus of uncertainty-conditioned deep reinforcement learning - Google Patents

Abstract

One embodiment of the present disclosure provides a deep reinforcement learning device and method, which inputs a state (s_t) and outputs a feature vector (h_t) for the state (s_t), estimates an epistemological uncertainty (u) for the state (s_t), and estimates an action (a_t) based on the feature vector (h_t) and the epistemological uncertainty (u), wherein the deep reinforcement learning device learns based on a reward (r_t) for the action (a_t). Accordingly, the present invention determines whether an input state is well-trained in a deep reinforcement learning model, and determines an appropriate action by considering this.

Description

Uncertainty conditional deep reinforcement learning method and processing device thereof {METHOD AND APPARATUS OF UNCERTAINTY-CONDITIONED DEEP REINFORCEMENT LEARNING}

This disclosure relates to an uncertainty conditional deep reinforcement learning method and a processing device thereof. More specifically, the present disclosure relates to a method and processing device for determining an action by considering uncertainty as well as the current state in a deep reinforcement learning model.

Reinforcement learning is learning which action is optimal in the current state. Whenever an action is taken, a reward is given from the external environment, and this reward is maximized. Learning progresses in this direction.

A deep reinforcement learning (DRL) model using a deep neural network solves tasks by learning a policy expressed as a function that creates an action when the current state is given as input. Learn how to do it. However, existing deep reinforcement learning models have the problem of overconfidence in the selected action even when the input state is far from the distribution of learned states. For example, taking overconfident actions in an untrained situation in an autonomous vehicle can lead to incorrect decisions, which can have fatal consequences for surrounding vehicles, pedestrians, and the autonomous vehicle itself. To solve this problem, it is necessary to check whether the input state has been well trained in the deep reinforcement learning model and decide on an action by considering this.

In the end, according to the existing method, there was a problem in that it could not provide a method of checking whether the input state was well trained in the deep reinforcement learning model and determining an action by considering this, and the present invention This is to solve this problem.

One embodiment of the present disclosure is to check whether the input state has been well trained in the deep reinforcement learning model and determine an appropriate action by considering this.

Additionally, an embodiment of the present disclosure is intended to provide a deep reinforcement learning model that determines an appropriate action by considering the uncertainty of the input state.

An embodiment of the present disclosure seeks to provide an uncertainty conditional deep reinforcement learning method and a processing device thereof.

An embodiment of the present disclosure includes a feature extraction unit that receives a state (s _t ) as input and outputs a feature vector (h _t ) for the state (s _t ); an uncertainty estimation unit that estimates epistemological uncertainty (u) for the state (s _t ); And an action estimation unit that estimates the action (a _t ) based on the feature vector (h _t ) and the epistemological uncertainty (u), wherein the deep reinforcement learning device provides a reward for the action (a _t ). The aim is to provide a deep reinforcement learning device that learns based on r _t ).

In one embodiment, the feature extractor may include an encoder that outputs a feature vector (h _t ) for the state (s _t ).

In one embodiment, the uncertainty estimation unit may estimate uncertainty about the state (s _t ) using a Monte-Carlo dropout method.

In one embodiment, the uncertainty estimation unit applies Monte-Carlo dropout to the encoder, using the variance of N Monte Carlo samples as an uncertainty indicator ( ) may include an uncertainty calculation unit that derives.

In one embodiment, the uncertainty estimation unit sets the uncertainty indicator, which is a vector having the same size as the feature vector that is the output of the encoder, to the uncertainty indicator ( ) for each element ( ) upper limit ( ), it may include a scalar mapping unit that performs a weighted average and outputs a scalar value (u).

In one embodiment, the behavior estimation unit estimates the mean and variance of the behavior (a _t ) based on the characteristic vector (h _t ) and the epistemic uncertainty (u), and has the mean and the variance. Actions for the current state can be provided by sampling from the distribution of the actions (a _t ).

In one embodiment, the action estimation unit determines the mutual information between the epistemic uncertainty (u) and the action (a _t ) so that the action (a _t ) varies depending on the degree of the epistemic uncertainty (u). It can be learned so that the lower limit is the maximum.

In one embodiment, the action estimator may improve soft policy and learn to maximize the sum of the lower bound of the mutual information of the epistemic uncertainty (u) and the action (a _t ).

In one embodiment, the lower bound of the mutual information of the epistemic uncertainty (u) and the action (a _t ) may be calculated using data sampled from a replay buffer.

One embodiment of the present disclosure includes a feature extraction step of inputting a state (s _t ) and outputting a feature vector (h _t ) for the state (s _t ); An uncertainty estimation step of estimating epistemological uncertainty (u) for the state (s _t ); And a behavior estimation step of estimating an action (a _t ) based on the feature vector (h _t ) and the epistemological uncertainty (u), wherein the feature extraction step and the uncertainty estimation step are performed in parallel. , we aim to provide a deep reinforcement learning method.

In one embodiment, in the feature extraction step, a feature vector (h _t ) for the state (s _t ) may be output using an encoder.

In one embodiment, the uncertainty about the state (s _t ) may be estimated using a Monte-Carlo dropout method in the uncertainty estimation step.

In one embodiment, the uncertainty estimation step applies Monte-Carlo dropout to the encoder, and the uncertainty index ( ) can be derived.

In one embodiment, the uncertainty estimation step is to use the uncertainty indicator, which is a vector having the same size as the feature vector that is the output of the encoder, as the uncertainty indicator ( ) for each element ( ) upper limit ( After normalizing with ), a scalar value (u) can be output by weighted average.

In one embodiment, the action estimation step estimates the mean and variance of the action (a t) based on the feature vector (h _t ) and the epistemic uncertainty ( _u ), and calculates the mean and the variance. A branch can provide an action for the current state by sampling from the distribution of the action (a _t ).

In one embodiment, in the action estimation step, mutual information between the epistemic uncertainty (u) and the action (a _t ) is used so that the action (a _t ) varies depending on the degree of the epistemic uncertainty (u). It can be learned so that the lower limit of is the maximum.

In one embodiment, in the action estimation step, a soft policy can be improved and learned such that the sum of the lower bound of the mutual information of the epistemic uncertainty (u) and the action (a _t ) is maximized.

In one embodiment, the lower bound of the mutual information of the epistemic uncertainty (u) and the action (a _t ) may be calculated using data sampled from a replay buffer.

One embodiment of the present disclosure includes a program stored in a recording medium to execute the method according to the embodiment of the present disclosure on a computer.

An embodiment of the present disclosure includes a computer-readable recording medium on which a program for executing a method according to an embodiment of the present disclosure on a computer is recorded.

An embodiment of the present disclosure includes a computer-readable recording medium that records a database used in an embodiment of the present disclosure.

According to an embodiment of the present disclosure, an uncertainty conditional deep reinforcement learning method and a processing device thereof may be provided.

In addition, according to an embodiment of the present disclosure, a deep reinforcement learning model is provided that checks whether the input state is well trained in the deep reinforcement learning model and determines an appropriate action by considering this. You can.

Additionally, according to an embodiment of the present disclosure, it is possible to provide a deep reinforcement learning model that determines an appropriate action by considering the uncertainty of the input state.

Additionally, according to an embodiment of the present disclosure, a deep reinforcement learning model that takes appropriate action for unlearned situations can be provided.

Figure 1 is a graph illustrating the difference between a deep reinforcement learning model and a model that does not consider uncertainty according to an embodiment of the present disclosure.
Figure 2 is a conceptual diagram illustrating a deep reinforcement learning model considering uncertainty according to an embodiment of the present disclosure.
Figure 3 is a conceptual diagram illustrating a deep reinforcement learning model considering uncertainty according to an embodiment of the present disclosure.
Figure 4 is a flow chart illustrating an uncertainty conditional deep reinforcement learning method according to an embodiment of the present disclosure.
Figure 5 is a conceptual diagram illustrating an experimental environment for evaluating the performance of an embodiment of the present disclosure.
6 to 9 are graphs showing the results of evaluating the performance of an embodiment of the present disclosure using the MuJoCo simulator.

In order to clarify the technical idea of the present disclosure, embodiments of the present disclosure will be described in detail with reference to the attached drawings. In describing the present disclosure, if it is determined that a detailed description of a related known function or component may unnecessarily obscure the gist of the present disclosure, the detailed description will be omitted. Components having substantially the same functional configuration among the drawings are given the same reference numbers and symbols as much as possible, even if they are shown in different drawings. For convenience of explanation, if necessary, the device and method should be described together. Each operation of the present disclosure does not necessarily have to be performed in the order described, and may be performed in parallel, selectively, or individually.

The terms used in the embodiments of the present disclosure have selected general terms that are currently widely used as much as possible while considering the function of the present disclosure, but this may vary depending on the intention or precedent of a person working in the art, the emergence of new technology, etc. . In addition, in certain cases, there are terms arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the relevant embodiment. Therefore, the terms used in this specification should not be defined simply as the names of the terms, but should be defined based on the meaning of the term and the overall content of the present disclosure.

Throughout this disclosure, singular expressions may include plural expressions, unless the context clearly dictates otherwise. Terms such as "include" or "have" are intended to designate the presence of a feature, number, step, operation, component, part, or combination thereof, but not one or more other features, numbers, steps, operations, or composition. It should be understood that this does not exclude in advance the possibility of the presence or addition of elements, parts, or combinations thereof. In other words, when it is said that a part "includes" a certain element throughout the present disclosure, this means that other elements may be further included rather than excluding other elements, unless specifically stated to the contrary.

An expression such as "at least one" modifies the entire list of elements, not the elements of the list individually. For example, “at least one of A, B, and C” and “at least one of A, B, or C” means only A, only B, only C, both A and B, both B and C, and A and C All refers to all of A, B, and C, or a combination thereof.

In addition, terms such as “..unit” and “..module” described in the present disclosure refer to a unit that processes at least one function or operation, which may be implemented as hardware or software or through a combination of hardware and software. You can.

Throughout the present disclosure, when a part is said to be “connected” to another part, this includes not only the case where it is “directly connected,” but also the case where it is “electrically connected” with another element in between. do. Additionally, when a part "includes" a certain component, this means that it may further include other components rather than excluding other components, unless specifically stated to the contrary.

The expression “configured to” used throughout the present disclosure may be used, depending on the context, for example, “suitable for,” “having the capacity to.” )", "designed to", "adapted to", "made to", or "capable of". . The term “configured (or set to)” may not necessarily mean “specifically designed to” in hardware. Instead, in some contexts, the expression “system configured to” may mean that the system is “capable of” in conjunction with other devices or components. For example, the phrase "processor configured (or set) to perform A, B, and C" refers to a processor dedicated to performing the operations (e.g., an embedded processor), or by executing one or more software programs stored in memory. It may refer to a general-purpose processor (e.g., CPU or application processor) that can perform the corresponding operations.

Functions related to artificial intelligence according to the present disclosure may be operated through a processor and memory. The processor may include a general-purpose processor such as a CPU, AP, or DSP (Digital Signal Processor), a graphics-specific processor such as a GPU or VPU (Vision Processing Unit), or an artificial intelligence-specific processor such as an NPU. Additionally, the processor can control input data to be processed according to predefined operation rules or artificial intelligence models stored in memory. Additionally, an artificial intelligence-specific processor may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

Predefined operation rules or artificial intelligence models are characterized by being created through learning. Here, being created through learning may mean that a basic artificial intelligence model is learned using a plurality of learning data by a learning algorithm, thereby creating a predefined operation rule or artificial intelligence model set to perform the desired purpose. . This learning may be accomplished in the device itself that performs the artificial intelligence according to the present disclosure, or may be accomplished through a separate server and/or system.

An artificial intelligence model may be composed of multiple neural network layers. Each of the plurality of neural network layers has a plurality of weight values, and a neural network calculation can be performed through calculation between the calculation result of the previous layer and the plurality of weights. Multiple weights of multiple neural network layers can be optimized by the learning results of the artificial intelligence model. For example, a plurality of weights may be updated so that loss or cost values obtained from the artificial intelligence model are reduced or minimized during the learning process. Artificial neural networks are, for example, Convolutional Neural Network (CNN), Deep Neural Network (DNN), Recurrent Neural Network (RNN), Restricted Boltzmann Machine (RBM), Deep Belief Network (DBN), and Bidirectional Recurrent Deep Neural Network (BRDNN). Neural Network) or Deep Q-Networks, etc., but are not limited thereto.

This disclosure relates to an uncertainty conditional deep reinforcement learning method and a processing device thereof. More specifically, the present disclosure relates to a method and processing device for determining an action by considering uncertainty as well as the current state in a deep reinforcement learning model.

Figure 1 is a graph illustrating the difference between a deep reinforcement learning model and a model that does not consider uncertainty according to an embodiment of the present disclosure.

Existing deep reinforcement learning models only consider the current state and do not consider the uncertainty of the learning model for that state. The uncertainty considered in this disclosure is model uncertainty or epistemic uncertainty resulting from insufficient knowledge of the model representing the data, and data uncertainty or aleatoric uncertainty arising from the randomness inherent in the input. uncertainty) does not apply to this. Therefore, the uncertainty mentioned in this disclosure means epistemological uncertainty.

In an existing deep reinforcement learning model, once the current state is determined, the corresponding action is determined regardless of the degree of uncertainty. Referring to Figure 1(a), the current state ( ) is determined, the action (a _t ) is determined regardless of uncertainty u, so if uncertainty (u) is high, the selected action (a _t ) may be an overconfident action. In other words, it may be an overconfident action selected when the learning model has not sufficiently learned the corresponding state.

The deep reinforcement learning model according to an embodiment of the present disclosure receives the current state and the degree of uncertainty as input, and determines an action for the current state according to the degree of uncertainty. Referring to Figure 1(b), the same state ( ), the behavior (a _t ) may vary depending on the degree of uncertainty (u). For example, the current state ( ) is a driving situation on a narrow road that has never been driven before, and if uncertainty (u) is determined to be high because the learning model for autonomous driving has not been sufficiently learned for this situation, the control behavior of the autonomous vehicle (a _t ) You can drive cautiously by selecting a smaller value than when uncertainty is low.

Figure 2 is a conceptual diagram illustrating a deep reinforcement learning model considering uncertainty according to an embodiment of the present disclosure.

The deep reinforcement learning model according to an embodiment of the present disclosure may include a feature extraction unit 100, an uncertainty estimation unit 200, and a behavior estimation unit 300. In one embodiment, the current state (s _t ) is input to the feature extraction unit 100 and the uncertainty estimation unit 200, and the behavior estimation unit 300 inputs the feature extraction unit 100 and the uncertainty estimation unit 200. It receives the output of as input and outputs an action (a _t ) corresponding to the current state (s _t ) and uncertainty (u).

In one embodiment, the feature extractor 100 may receive a state (s _t ) as input and output a feature vector (h _t ) for the state. For example, the feature extractor 100 may include an encoder network.

In one embodiment, the uncertainty estimation unit 200 can estimate the epistemic uncertainty of the reinforcement learning model for the current state (s _t ) using the Monte-Carlo dropout method. there is. For example, the uncertainty estimation unit 200 normalizes each element of the uncertainty vector having the same dimension as the feature vector (h _t ), which is the output of the feature extraction unit 100, and then performs a weighted arithmetic average. The degree of uncertainty (u) can be output as a scalar value.

In one embodiment, the behavior estimation unit 300 receives the feature vector (h _t ), which is the output of the feature extraction unit 100, and the degree of uncertainty (u), the output of the uncertainty estimation unit 200, and generates the corresponding Action (a _t ) can be output. For example, the feature vector (ht) and the degree of uncertainty (u) may be concatenated into one vector and input to the behavior estimation unit 300, and the behavior estimation unit 300 may perform the corresponding action ( The mean and variance of a _t ) can be derived, and the final behavior (a _t ) can be estimated from the distribution of this behavior (a _t ).

Figure 3 is a conceptual diagram illustrating a deep reinforcement learning model considering uncertainty according to an embodiment of the present disclosure.

According to an embodiment of the present disclosure, the feature extractor 100 is an encoder 110 of a policy network. The encoder network 110 is learned to output a feature vector (h _t ) from the input state (s _t ), It can be expressed as

According to an embodiment of the present disclosure, the uncertainty estimation unit 200 may include an uncertainty calculation unit 210 and a scalar mapping unit 220. In one embodiment, the uncertainty calculation unit 210 may estimate epistemological uncertainty by applying dropout, which omits part of the neural network, to the encoder 110 of the learned policy network. For example, by applying Monte-Carlo Dropout to a neural network, the variance of N Monte Carlo samples is used as an uncertainty indicator as shown in the formula below. can be calculated (related paper [Y. Gal and Z. Ghahramani, "Dropout as a Bayesian approximation: Representing model uncertainty in deep learning," in Proc. 33rd Int. Conf. Mach. Learn., 2016, vol. 48, pp. 1050-1059.] are incorporated herein by reference).

(One)

here, is a neural network with dropout applied to input x is the ith output of silver is the average of

In one embodiment, the uncertainty calculation unit 210 is the encoder 110 of the learned policy network. By applying Monte-Carlo Dropout to uncertainty as shown in Equation (2) below: can be estimated.

(2)

here, is the i-th output of the encoder 110 to which dropout is applied, silver is the average of

In one embodiment, the encoder network 110 outputs a feature vector (h _t ), so the output of the uncertainty calculation unit 210 is also a vector value (U) having the same size as the feature vector (h _t ), and scalar mapping Unit 220 maps the output of uncertainty calculation unit 210 to a scalar value (u). In other words, what the uncertainty estimation unit 200 wants to estimate is the value for each element of the given state (s _t ) corresponding to each element of the vector value (U) that is the output of the uncertainty calculation unit 210. Since it is not uncertainty, but uncertainty about the given state (s _t ) itself, each element of the vector value (U) is weighted and mapped to a scalar value (u).

For example, the scalar mapping unit 220 can generalize the scalar value (u) representing the degree of uncertainty to a range between [0, 1], and for this purpose, the upper limit for each element of the uncertainty vector (U) is used. By normalizing and calculating the weighted average, the uncertainty scalar value (u) can be calculated. For this purpose, during learning of the encoder network 110, the minimum and maximum outputs for each element (h ⁱ ) of the feature vector (h _t ) are and can be collected as follows by updating each time a new maximum or minimum is found (in the formula below, h ⁱ refers to the ith element of vector h _t ).

(3)

(4)

collected and Using , the upper limit of uncertainty for each element (h ⁱ ) of the feature vector (h _t ) ( ) can be calculated as the variance of the uniform distribution as follows, assuming that the output follows a uniform distribution.

(5)

Upper limit of uncertainty according to equation (5) ( ) does not interfere with the learning process because the neural network model converges early in training when there is high uncertainty for all data. In one embodiment, the gradient for optimizing the encoder network 110 may not be calculated during uncertainty calculation.

Finally, the scalar value (u), which represents the degree of uncertainty, is the uncertainty value of each element ( ) is the upper limit of uncertainty ( ) and can be calculated as follows by taking the weighted average for each element.

(6)

here, , and d is It is the size of . Additionally, since a weighted average is used in Equation (6), it is possible to prevent elements with high uncertainty from being offset by elements with low uncertainty.

According to an embodiment of the present disclosure, the behavior estimation unit 300 inputs a feature vector (h _t ), which is the output of the feature extraction unit 100, and an uncertainty scalar value (u _t ), which is the output of the uncertainty estimation unit 200. Then, estimate the action (a _t ) using deep reinforcement learning. In one embodiment, the feature extraction unit 100 and the uncertainty estimation unit 200 calculate a feature vector (h _t ) and an uncertainty scalar value (u _t ) in parallel, respectively, and input them to the behavior estimation unit 300. It can be provided as a value.

In one embodiment, the action estimation unit 300 is a Q-function, which is an action value function for reinforcement learning. , a policy that takes uncertainty into account. and a neural information measure. can be used. For example, neural information measures can be defined as the lower limit of mutual information, which represents the degree of interdependence between uncertainty and action.

(7)

In one embodiment, the behavior estimation unit 300 concatenates the encoded state feature vector (h _t ), which is the output of the feature extraction unit 100, and a scalar value (u _t ) indicating the degree of epistemological uncertainty. input, the action estimation unit 300 can provide the mean and variance of the action distribution using reinforcement learning, and finally sample the output action (a _t ) from this action distribution. .

In one embodiment, the Q-function of the behavior estimation unit 300 can estimate the Q-value, which is the action value, in a given situation (s _t , u _t , a _t ). Additionally, the Q-function can be trained using soft policy evaluation in the same way as the reinforcement learning algorithm SAC (Soft Actor-Critic).

In one embodiment, the policy function of the behavior estimation unit 300 To maximize soft policy improvement, which is the basic goal, while classifying policies according to the degree of uncertainty. trained to maximize For example, the policy function To train, the extended objective function can be constructed as follows.

(8)

here, is calculated as follows using the data sampled from the replay buffer ((s ⁽¹⁾ , u ⁽¹⁾ ), ...,(s ^(b) , u ^(b) ))~D. You can.

(9)

here, is approximately , b is the number of sampled data, is the randomly sampled degree of uncertainty approximating the sampled data from the marginal distribution P _U .

By maximizing the objective function of Equation (8), the action estimation unit 300 improves the policy to provide an action that maximizes the expected cumulative reward, while improving the policy's action according to the degree of uncertainty. can be learned to distinguish between.

Figure 4 is a flow chart illustrating an uncertainty conditional deep reinforcement learning method according to an embodiment of the present disclosure.

According to an embodiment of the present disclosure, the method for self-balancing an online dataset includes an initialization step (S100), a feature extraction step (S200), an uncertainty estimation step (S300), an action estimation step (S400), and an action execution and reward confirmation step. (S500), a replay buffer storage step (S600), and a parameter update step (S700).

According to one embodiment, the parameters in the initialization step (S100) and initialize the replay buffer D.

According to one embodiment, in the feature extraction step (S200), the current state (s _t ) may be input and a feature vector (h _t ) for the state may be output. For example, the feature extraction step (S200) may be performed using an encoder network.

According to one embodiment, in the uncertainty estimation step (S300), the epistemic uncertainty of the reinforcement learning model for the current state (s _t ) is estimated using the Monte-Carlo dropout method. You can. For example, the uncertainty estimation step (S300) normalizes each element of the uncertainty vector having the same dimension as the feature vector (h _t ) and then performs a weighted arithmetic average to calculate the degree of uncertainty (u) as a scalar ( It can be output as a scalar) value.

According to one embodiment, the feature extraction step (S200) and the uncertainty estimation step (S300) may be performed in parallel. Additionally, in the feature extraction step (S200) and the uncertainty estimation step (S300), the gradient for optimizing the encoder network 110 may not be calculated.

According to one embodiment, in the action estimator step (S400), the feature vector (h _t ) derived in the feature extraction step (S200) and the degree of uncertainty (u) derived in the uncertainty estimation step (S300) are input, The corresponding action (a _t ) can be output. For example, the feature vector (h _t ) and the degree of uncertainty (u) are concatenated into one vector and input into a neural network trained by reinforcement learning, and the mean and variance of the high action (a _t ) distribution of the reward are is derived, and the final action (a _t ) can be estimated from this action (a _t ) distribution.

According to one embodiment, the action execution and reward confirmation step (S500) executes the action (a _t ) output in the action estimation step (S400), and the corresponding reward (r _t ) and new state (s _t+1 ) This can be confirmed by observing.

According to one embodiment, the replay buffer storage step (S600) stores the state (s _t ), uncertainty (u _t ), action (a _t ), reward (r _t ), and new state (s _t+1 ) in the replay buffer ( It can be stored in replay buffer D.

According to one embodiment, in the parameter update step (S700), data is sampled from the replay buffer, a gradient for parameter update is calculated, and the parameter update is used. can be updated. In one embodiment, the parameter update step (S700) may be performed repeatedly for each sampled data.

Figure 5 is a conceptual diagram illustrating an experimental environment for evaluating the performance of an embodiment of the present disclosure.

To evaluate and compare the performance of this disclosure with existing methods, the MuJoCo physics simulator and the CARLA simulator, an open source simulator for autonomous driving research, were used.

Referring to Figure 5(a), the MuJoCo simulator uses the original task set for each environment in four environments (Ant, Hopper, Humanoid, and Walker2D) without modifying the state or reward in the training phase, but multivariate normal in the testing phase. Random noise sampled from a distribution (multi-variate normal distribution) was added to the state. Here, it is assumed that the added noise increases the epistemological uncertainty in the experiment.

Referring to Figures 5(b) to 5(e), in the CARLA environment, there is a central roundabout scenario that agents must pass through (Figure 5(b)), and a cyclist scenario where all vehicles are replaced by cyclists (Figure 5 (c)), contains a randomly generated scenario (Figure 5(d)) generated from a location the agent has never seen before, and a bird's eye view semantic image (Figure 5(e)) is given as an observation. The bird's eye view semantic image given in the CARLA environment is encoded into a latent vector to represent the state of the environment. When the latent vector is given as a state, steering is performed when the entire path is given. It was set to ensure minimum driving ability and calculated based on acceleration, deceleration, and action.

6 to 9 are graphs showing the results of evaluating the performance of an embodiment of the present disclosure using the MuJoCo simulator.

In the MuJoCo simulator, SAC is a deep reinforcement learning method that does not consider uncertainty, SAC-U with a degree of uncertainty u as an additional input, a method (UNICON) according to an embodiment of the present disclosure, and a mapping function in an embodiment of the present disclosure. The model (UNICON-V) excluding the scalar mapping unit 220 corresponding to was compared.

Referring to Figures 6 to 9, it can be seen that an embodiment of the present disclosure has little overall performance degradation and a high average return value.

In addition, in the CARLA environment, SAC, which is a deep reinforcement learning method that does not consider uncertainty, SAC-U, where the degree of uncertainty is u as an additional input, and a method according to an embodiment of the present disclosure (represented by UNICON) are used to achieve success rate and collision rate. were compared as performance indicators.

Referring to Tables 1 to 3 below, it can be seen that for all CARLA scenarios, the method according to an embodiment of the present disclosure has the highest success rate and the lowest collision rate.

<Table 1>

<Table 2>

<Table 3>

In addition, the paper [C. Kim, J. K. Cho, H. S. Yoon, S. W. Seo, and S. W. Kim, “UNICON: Uncertainty-Conditioned Policy for Robust Behavior in Unfamiliar Scenarios,” IEEE Robotics and Automation Letters, Vol. 7, No. 4, pp. 9099-9106.] are incorporated into this disclosure by reference.

An embodiment of the present disclosure may also be implemented in the form of a recording medium containing instructions executable by a computer, such as program modules executed by a computer. Computer-readable media can be any available media that can be accessed by a computer and includes both volatile and non-volatile media, removable and non-removable media. Additionally, computer-readable media may include both computer storage media and communication media. Computer storage media includes both volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data. Communication media typically includes computer-readable instructions, data structures, or program modules and includes any information delivery medium.

The foregoing description of the present disclosure is for illustrative purposes, and a person skilled in the art to which the present disclosure pertains will understand that the present invention can be easily modified into other specific forms without changing its technical idea or essential features. will be.

Therefore, the embodiments described above should be understood in all respects as illustrative and not restrictive. For example, each component described as unitary may be implemented in a distributed manner, and similarly, components described as distributed may also be implemented in a combined form.

The scope of the present disclosure is indicated by the claims described below rather than the detailed description above, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be construed as being included in the scope of the present disclosure. do.

Claims (19)

In the deep reinforcement learning device,
a feature extraction unit that receives a state (s _t ) as input and outputs a feature vector (h _t ) for the state (s _t );
an uncertainty estimation unit that estimates epistemological uncertainty (u) for the state (s _t ); and
A behavior estimation unit that estimates the behavior (a _t ) based on the characteristic vector (h _t ) and the epistemological uncertainty (u);
The deep reinforcement learning device is learned based on the reward (r _t ) for the action (a _t ),
The feature extraction unit includes an encoder that outputs a feature vector (h _t ) for the state (s _t ),
Deep reinforcement learning device. delete The method of claim 1, wherein the uncertainty estimation unit,
Estimating uncertainty about the state (s _t ) using the Monte-Carlo dropout method,
Deep reinforcement learning device. The method of claim 1, wherein the uncertainty estimation unit,
By applying Monte-Carlo dropout to the encoder, the uncertainty index ( ), including an uncertainty calculation unit that derives
Deep reinforcement learning device. The method of claim 4, wherein the uncertainty estimation unit,
The uncertainty indicator, which is a vector with the same size as the feature vector output from the encoder, is used as the uncertainty indicator ( ) for each element ( )'s upper limit ( ), and then includes a scalar mapping unit that outputs a scalar value (u) by weighted average,
Deep reinforcement learning device. The method of claim 1, wherein the behavior estimation unit,
Based on the feature vector (h _t ) and the epistemic uncertainty (u), estimate the mean and variance of the behavior (a _t ), and sample from the distribution of the behavior (a _t ) with the mean and the variance. to provide action on the current state,
Deep reinforcement learning device. The method of claim 1, wherein the behavior estimation unit,
In order for the action (a _t ) to vary depending on the degree of the epistemic uncertainty (u), learning so that the lower limit of the mutual information between the epistemological uncertainty (u) and the action (a _t ) is maximized,
Deep reinforcement learning device. The method of claim 1, wherein the behavior estimation unit,
Soft policy improvement and learning such that the sum of the lower bound of the epistemic uncertainty (u) and the mutual information of the action (a _t ) is maximized,
Deep reinforcement learning device. In clause 7,
The lower bound of the mutual information of the epistemic uncertainty (u) and the action (a _t ) is,
Calculated using data sampled from the replay buffer,
Deep reinforcement learning device. In the deep reinforcement learning method,
A feature extraction step of inputting a state (s _t ) and outputting a feature vector (h _t ) for the state (s _t );
An uncertainty estimation step of estimating epistemological uncertainty (u) for the state (s _t ); and
An action estimation step of estimating an action (a _t ) based on the feature vector (h _t ) and epistemological uncertainty (u),
The feature extraction step and the uncertainty estimation step are performed in parallel,
Outputting a feature vector (h _t ) for the state (s _t ) using an encoder in the feature extraction step,
Deep reinforcement learning methods. delete The method of claim 10, wherein in the uncertainty estimation step,
Estimating uncertainty about the state (s _t ) using the Monte-Carlo dropout method,
Deep reinforcement learning methods. The method of claim 10, wherein the uncertainty estimation step is,
By applying Monte-Carlo dropout to the encoder, the uncertainty index ( ), which derives
Deep reinforcement learning methods. The method of claim 13, wherein the uncertainty estimation step is,
The uncertainty indicator, which is a vector with the same size as the feature vector output from the encoder, is used as the uncertainty indicator ( ) for each element ( ) upper limit ( ), then output a scalar value (u) by weighted average,
Deep reinforcement learning methods. The method of claim 10, wherein the action estimation step is,
Based on the feature vector (h _t ) and the epistemic uncertainty (u), estimate the mean and variance of the behavior (a _t ), and sample from the distribution of the behavior (a _t ) with the mean and the variance. to provide action on the current state,
Deep reinforcement learning methods. The method of claim 10, wherein in the action estimation step,
In order for the action (a _t ) to vary depending on the degree of the epistemic uncertainty (u), learning so that the lower limit of the mutual information between the epistemological uncertainty (u) and the action (a _t ) is maximized,
Deep reinforcement learning methods. The method of claim 10, wherein in the action estimation step,
Soft policy improvement and learning such that the sum of the lower bound of the epistemic uncertainty (u) and the mutual information of the action (a _t ) is maximized,
Deep reinforcement learning methods. According to clause 16,
The lower bound of the mutual information of the epistemic uncertainty (u) and the action (a _t ) is,
Calculated using data sampled from the replay buffer,
Deep reinforcement learning methods. A program stored in a computer-readable recording medium to execute the method of any one of claims 10 and 12 to 18 on a computer.