KR20210048969A - Method and system for optimizing reinforcement learning based navigation to human preference - Google Patents

Abstract

Disclosed are a method and a system for optimizing reinforcement learning based autonomous driving in accordance with user preference. The method for optimizing autonomous driving includes steps of: learning robot autonomous driving by assigning different autonomous driving parameters to a plurality of robot agents in simulation through automatic setting by a system or direct setting by an administrator; and optimizing the autonomous driving parameter by using preference data for the autonomous driving parameter.

Description

Autonomous driving optimization method and system based on reinforcement learning according to user preference {METHOD AND SYSTEM FOR OPTIMIZING REINFORCEMENT LEARNING BASED NAVIGATION TO HUMAN PREFERENCE}

The description below relates to the robot's autonomous driving technology.

Autonomous robot is a technology to which a robot widely used in the industrial field is applied.For example, the movement distance and direction from the previous position to the next position after acquiring speed information and azimuth information using the Odometry method. You can recognize your own location and direction by calculating the information about.

For example, Korean Patent Registration No. 10-1771643 (registration date August 21, 2017) discloses an autonomous driving robot capable of automatically moving to a destination by recognizing absolute coordinates, and a navigation method thereof.

It provides a technology for optimizing autonomous driving based on reinforcement learning according to user preference.

It provides a new in-depth reinforcement learning-based autonomous driving technology that can adapt to various parameters and rewards without retraining.

It provides a technology that can find out an autonomous driving parameter that fits a use-case using a small number of preference data.

In the autonomous driving learning method executed in a computer system, the computer system comprises at least one processor configured to execute computer readable instructions contained in a memory, and the autonomous driving learning method comprises: by the at least one processor , Provides an autonomous driving learning method comprising the step of learning robot autonomous driving by assigning different autonomous driving parameters to a plurality of robot agents in the simulation through automatic setting by a system or direct setting by a manager.

According to an aspect, in the learning step, reinforcement learning may be performed simultaneously by inputting a randomly sampled autonomous driving parameter for the plurality of robot agents.

According to another aspect, in the learning step, autonomous driving of the plurality of robot agents may be simultaneously learned using a neural network composed of a fully-connected layer and gated recurrent units (GRU).

According to another aspect, the learning may include using a sensor value obtained in real time from the robot as an input of a neural network for learning of the robot autonomous driving and an autonomous driving parameter that is randomly assigned in relation to the autonomous driving policy. It may include.

According to another aspect, the autonomous driving learning method may further include, by the at least one processor, optimizing the autonomous driving parameter using preference data for the autonomous driving parameter. .

According to another aspect, in the optimizing step, the autonomous driving parameter may be optimized by reflecting feedback on a driving image of a robot in which the autonomous driving parameter is set differently from each other.

According to another aspect, the step of optimizing may include evaluating a preference for the autonomous driving parameter through pairwise comparisons of the autonomous driving parameter.

According to another aspect, the step of optimizing may include modeling a preference for the autonomous driving parameter using a Bayesian neural network model.

According to another aspect, the step of optimizing may include generating a query for pairwise comparison of the autonomous driving parameters based on an uncertainty of a preference model.

There is provided a computer program stored in a non-transitory computer-readable recording medium for executing the autonomous driving learning method on the computer system.

A non-transitory computer-readable recording medium in which a program for executing the autonomous driving learning method is recorded on a computer is provided.

A computer system comprising at least one processor configured to execute computer-readable instructions contained in a memory, wherein the at least one processor is configured automatically by a system or directly by an administrator in a plurality of robot agents in a simulation. A learning unit for learning robot autonomous driving by assigning different autonomous driving parameters through; And an optimization unit for optimizing the autonomous driving parameter by using the preference data for the autonomous driving parameter.

According to embodiments of the present invention, it is possible to achieve various and unpredictable real-world learning effects by simultaneously performing reinforcement learning in various environments, and to implement an adaptive autonomous driving algorithm without increasing data. have.

According to embodiments of the present invention, after modeling a preference indicating whether a driving image of a robot is appropriate as a use case, an autonomous driving parameter may be optimized using a small number of preference data based on the uncertainty of the model.

1 is a block diagram illustrating an example of an internal configuration of a computer system according to an embodiment of the present invention.
2 is a diagram illustrating an example of components that may be included in a processor of a computer system according to an embodiment of the present invention.
3 is a flow chart showing an example of an autonomous driving learning method that can be performed by a computer system according to an embodiment of the present invention.
4 shows an example of an adaptive autonomous driving policy learning algorithm according to an embodiment of the present invention.
5 shows an example of a neural network for learning an adaptive autonomous driving policy according to an embodiment of the present invention.
6 shows an example of a neural network for learning a utility function according to an embodiment of the present invention.
7 shows an example of an autonomous driving parameter optimization algorithm using preference data in an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

Embodiments of the present invention relate to autonomous driving technology of a robot.

Embodiments including those specifically disclosed in this specification can provide a new deep reinforcement learning-based autonomous driving technology that can adapt to various parameters and compensate for a variety of parameters without a retraining process, and use a small number of preference data. Autonomous driving parameters that fit the case can be found.

1 is a block diagram showing an example of a computer system according to an embodiment of the present invention. For example, the autonomous driving learning system according to embodiments of the present invention may be implemented by the computer system 100 illustrated in FIG. 1.

As shown in FIG. 1, the computer system 100 is a component for executing the autonomous driving learning method according to embodiments of the present invention, and includes a memory 110, a processor 120, a communication interface 130, and an input/output. It may include an interface 140.

The memory 110 is a computer-readable recording medium and may include a permanent mass storage device such as a random access memory (RAM), a read only memory (ROM), and a disk drive. Here, a non-destructive large-capacity recording device such as a ROM and a disk drive may be included in the computer system 100 as a separate permanent storage device separated from the memory 110. In addition, an operating system and at least one program code may be stored in the memory 110. These software components may be loaded into the memory 110 from a computer-readable recording medium separate from the memory 110. Such a separate computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD/CD-ROM drive, and a memory card. In another embodiment, software components may be loaded into the memory 110 through a communication interface 130 other than a computer-readable recording medium. For example, software components may be loaded into the memory 110 of the computer system 100 based on a computer program installed by files received through the network 160.

The processor 120 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input/output operations. Instructions may be provided to the processor 120 by the memory 110 or the communication interface 130. For example, the processor 120 may be configured to execute a command received according to a program code stored in a recording device such as the memory 110.

The communication interface 130 may provide a function for the computer system 100 to communicate with other devices through the network 160. For example, requests, commands, data, files, etc., generated by the processor 120 of the computer system 100 according to a program code stored in a recording device such as the memory 110, are transmitted to the network ( 160) can be delivered to other devices. Conversely, signals, commands, data, files, etc. from other devices may be received by the computer system 100 through the communication interface 130 of the computer system 100 via the network 160. Signals, commands, data, etc. received through the communication interface 130 may be transmitted to the processor 120 or the memory 110, and files, etc. may be further included in the computer system 100 (as described above). Permanent storage).

The communication method is not limited, and not only a communication method using a communication network (for example, a mobile communication network, wired Internet, wireless Internet, broadcasting network) that the network 160 may include, but also short-range wired/wireless communication between devices may be included. have. For example, the network 160 is a personal area network (PAN), a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN), and a broadband network (BBN). And any one or more of networks such as the Internet. In addition, the network 160 may include any one or more of a network topology including a bus network, a star network, a ring network, a mesh network, a star-bus network, a tree or a hierarchical network, etc. Not limited.

The input/output interface 140 may be a means for an interface with the input/output device 150. For example, the input device may include a device such as a microphone, a keyboard, a camera, or a mouse, and the output device may include a device such as a display or a speaker. As another example, the input/output interface 140 may be a means for interfacing with a device in which input and output functions are integrated into one, such as a touch screen. The input/output device 150 may be configured with the computer system 100 and one device.

Further, in other embodiments, the computer system 100 may include fewer or more components than the components of FIG. 1. However, there is no need to clearly show most of the prior art components. For example, the computer system 100 may be implemented to include at least some of the input/output devices 150 described above, or may further include other components such as a transceiver, a camera, various sensors, and a database.

In recent years, in-depth reinforcement learning methods for autonomous driving have been actively studied, and the autonomous driving technology of robots using reinforcement learning shows higher performance than autonomous driving based on path planning.

However, the existing reinforcement learning method learns by using fixed values for parameters such as weights representing the trade-off between the robot's maximum speed and compensation components (e.g. Keeping a large safe distance from pouring).

Desirable robot behavior can be a problem in real world scenarios as it depends on the use case. Robots deployed in hospital wards, for example, need to be careful to avoid collisions with sophisticated equipment and not scare patients, while warehouse robots' top priority is to reach their targets as quickly as possible. Robots trained with fixed parameters cannot meet various requirements and require retraining to fine-tune them for each scenario. In addition, the desirable behavior of robots interacting with humans often depends on human preferences, which requires a lot of effort and cost to collect such preference data.

Therefore, there is a need for an agent capable of adapting to various parameters as well as a method capable of quickly and accurately predicting a parameter close to optimal from a small number of human preference data.

FIG. 2 is a diagram showing an example of components that can be included in a processor of a computer system according to an embodiment of the present invention, and FIG. 3 is an autonomous driving that can be performed by a computer system according to an embodiment of the present invention. It is a flow chart showing an example of a learning method.

As shown in FIG. 2, the processor 120 may include a learning unit 201 and an optimization unit 202. Components of the processor 120 may be expressions of different functions performed by the processor 120 according to a control command provided by at least one program code. For example, the learning unit 201 may be used as a functional expression that operates to control the computer system 100 so that the processor 120 learns autonomous driving of a robot based on deep reinforcement learning.

The processor 120 and components of the processor 120 may perform steps S310 to S320 included in the autonomous driving learning method of FIG. 3. For example, the processor 120 and the components of the processor 120 may be implemented to execute an instruction according to the code of the operating system included in the memory 110 and the at least one program code described above. Here, at least one program code may correspond to a code of a program implemented to process the autonomous driving learning method.

The autonomous driving learning method may not occur in the illustrated order, and some of the steps may be omitted or an additional process may be further included.

The processor 120 may load the program code stored in the program file for the autonomous driving learning method into the memory 110. For example, a program file for the autonomous driving learning method may be stored in a permanent storage device separate from the memory 110, and the processor 120 stores program codes from a program file stored in a permanent storage device through a bus. Computer system 100 can be controlled to be loaded onto 110. At this time, each of the processor 120 and the learning unit 201 and the optimization unit 202 included in the processor 120 executes a command of a corresponding part of the program code loaded in the memory 110 and performs subsequent steps ( It may be different functional expressions of the processor 120 for executing S310 to S320. In order to execute the steps S310 to S320, the processor 120 and components of the processor 120 may directly process an operation according to a control command or control the computer system 100.

First, the self-driving problem based on reinforcement learning is formulated as follows.

In this embodiment, a path-following autonomous driving task is considered, and an agent (ie, a robot) moves a path to a destination, and the path may be expressed as a series of stops. When the agent reaches the last stop (destination), it is given a new target and waypoint, and the task is modeled as a Markov decision process (S, A, Ω, r, p _trans , p _{obs ).} S is the states, A is the actions, Ω is the observations, r is the reward function, p _trans is the conditional state-transition, p _obs is the observation probability ( observation probabilities).

인 범용 설정을 적용한다.As an autonomous driving robot, a differential two-wheeled mobile platform model is used, and the discount factor is

Apply the universal setting.

(1) Navigation parameters

를 고려한다.Many parameters affect the behavior of the reinforcement learning-based autonomous driving agent. For example, an autonomous driving parameter consisting of 7 parameters

Consider.

[Equation 1]

은 충돌 또는 비상 정지 시 보상,

는 다른 에이전트와 충돌할 수 있는 최소 예상 시간,

은

을 위반한 것에 대한 보상,

는 최대 선형 속도(maximum linear speed),

는 선형 가속도(linear acceleration),

는 각 속도(angular speed),

는 각 가속도(angular acceleration)를 의미한다.here,

Compensation in case of collision or emergency stop,

Is the minimum estimated time to collide with other agents,

silver

Compensation for violating

Is the maximum linear speed,

Is the linear acceleration,

Is the angular speed,

Means angular acceleration.

에 적응할 수 있고 주어진 사용 사례에 적합한 파라미터

를 효율적으로 찾는 에이전트를 훈련시키는 것이다.The aim of the present invention is a variety of parameters

Parameters that are adaptable to and suitable for a given use case

It is to train an agent that finds efficiently.

(2) observations

The agent's observation format is shown in Equation 2.

[Equation 2]

는 라이더(lidar)와 같은 거리 센서의 스캔 데이터로 구성된다. -180°에서 180°까지의 데이터를 20° 간격으로 임시 저장하여 각 빈(bin)에서 최소값을 취한다. 에이전트가 지각할 수 있는 최대 거리는 3m이다.here,

Consists of scan data from a distance sensor such as a lidar. Data from -180° to 180° are temporarily stored at 20° intervals, and the minimum value is taken from each bin. The maximum distance the agent can perceive is 3m.

는 현재 선형 및 각 속도로 구성되며, 이전 단계에서 위치와 관련된 로봇 위치의 변화로서 수학식 3과 같이 같다.

Is composed of the current linear and angular velocity, as shown in Equation 3 as the change of the robot position related to the position in the previous step.

[Equation 3]

,

,

는 로봇의 x, y 위치 변화량과 방향(heading) 변화량을 의미하고,

는 한 단계(timestep)의 지속 시간을 의미한다.At this time,

,

,

Denotes the amount of change in the x, y position of the robot and the amount of change in the heading,

Means the duration of one step (timestep).

은

와 같고, 여기서

는 로봇의 좌표계에서 다음 경유지에 대한 상대적 각도를 의미한다.Finally,

silver

Same as, where

Denotes the relative angle to the next stop in the robot's coordinate system.

(3) actions

에서의 벡터로서 간격

로 정규화된 로봇의 원하는 선형 속력을 나타내고, 각속도는

로 정규화된다. 로봇이 액션을 실행하면

의 각 가속도가 적용되며, 속도를 증가시킬 때는 선형 가속도가

이고 감소할 때는

이다.The agent's action is

Spacing as a vector at

Represents the desired linear speed of the robot normalized to, and the angular speed is

Is normalized to When the robot executes an action

The angular acceleration of is applied, and when increasing the speed, the linear acceleration is

And when it decreases

to be.

(4) reward function

는 수학식 4와 같이 5가지 구성요소의 합을 의미한다.Reward function

Denotes the sum of the five components as shown in Equation 4.

[Equation 4]

은 에이전트가 최소 시간 내에 경유지에 도달하도록 권장하기 위해 모든 단계에서 주어진다.reward

Is given at every step to encourage the agent to reach the waypoint within the minimum amount of time.

와 같이 설정되고, 이때

이고,

는 단계 t에서 경유지까지의 유클리드 거리(Euclidean distance),

는 단계의 지속 시간이다. 충돌 회피에 필요한 최단 경로에서 작은 편차에 대한 패널티를 줄이기 위해 제곱근을 사용한다. 에이전트와 현재 경유지 사이의 거리가 1m 미만일 경우

의 보상이 있고 경유지가 업데이트 된다.

Is set as, at this time

ego,

Is the Euclidean distance from step t to the waypoint,

Is the duration of the step. The square root is used to reduce the penalty for small deviations in the shortest path required for collision avoidance. When the distance between the agent and the current stop is less than 1m

There is a reward and the waypoint is updated.

의 보상이 주어지는 경우 선형 속도를 0m/s로 설정하여 로봇을 정지시킨다. 예상 충돌 시간은 현재 동작에서 주어진 목표 속도로 계산되며,

으로 대표되는 장애 지점을 활용하여 0.5m 변의 정사각형으로 로봇을 모델링한다.When the estimated time to collide with an obstacle or other object is less than 1 second, or when a collision occurs, to ensure that the robot maintains a minimum safe distance in the simulation and real environment,

If the compensation of is given, the robot is stopped by setting the linear speed to 0m/s. The expected collision time is calculated at the target speed given in the current motion,

The robot is modeled as a 0.5m square using the point of failure represented by.

초보다 작을 때 보상

이 주어진다. 예상 충돌 시간은 스캔 데이터 대신 3m 범위 내의 다른 에이전트의 위치를 사용하는 것을 제외하고

에 대해 계산된다. 관측에 다른 에이전트의 위치를 포함시키지 않기 때문에 로봇은 스캔 데이터의 순서를 활용하여 다른 에이전트의 정적 장애물을 구별한다.The expected crash time for other agents is

Reward when less than seconds

Is given. The expected crash time is, except for using the location of another agent within a range of 3m instead of the scan data.

Is calculated for Since the observation does not include the location of other agents, the robot uses the order of the scan data to differentiate between static obstacles from other agents.

Referring to Figure 3, an example of the autonomous driving learning method according to the present invention includes the following two steps.

In step S310, the learning unit 201 randomly assigns autonomous driving parameters to several robots in a simulation environment in order to learn an autonomous driving policy that can adapt to a wide range of autonomous driving parameters without retraining, and simultaneously performs learning.

The learning unit 201 may use sensor data and autonomous driving parameters as inputs of a neural network for autonomous driving learning. The sensor data is a sensor value acquired in real time from the robot, and may include, for example, a time of flight (ToF) sensor value, a current speed, an odometry, a driving direction, an obstacle position, and the like. Autonomous driving parameters are randomly assigned setting values, and can be automatically set by the system or set directly by an administrator. For example, autonomous driving parameters may include compensation for collision, compensation for safety distance and safety distance necessary for collision avoidance, maximum speed (linear speed, rotational speed), maximum acceleration (linear acceleration, rotational acceleration), and the like. Assuming that the parameter range is from 1 to 10, simulations can be performed using a total of 10 robots from a robot with a parameter value of 1 to a robot with a parameter value of 10. In this case, the autonomous driving parameter may be designated based on a preference to be described below.

The learning unit 201 learns several robots at the same time by assigning randomly sampled parameters to each robot in the simulation, so that autonomous driving is possible according to various parameters without re-learning and generalizes to new parameters that were not used for existing learning. (generalization) is possible.

As an example, a decentralized multi-agent training method may be applied as summarized in the algorithm of FIG. 4. For each episode, multiple agents are placed in a shared environment. In order to adjust the policy to various autonomous driving parameters, the autonomous driving parameters of each agent are randomly sampled from the distribution at the start of each episode. In the case of reinforcement learning algorithms, parameter sampling is efficient, stable, and produces policies with better performance.

5 and 6 illustrate an example of a neural network structure for autonomous driving learning according to an embodiment of the present invention.

과

는 공유 가중치를 이용하여 계산된다.The neural network architecture for self-driving learning according to the present invention uses an adaptive policy learning structure (FIG. 5) and a utility function learning (FIG. 6) structure. FC represents a fully-connected layer, BayesianFC represents a Bayesian fully-connected layer, and a merged branch represents concatenation. Utility function

and

Is calculated using the shared weight.

As shown in FIG. 5, the agent's autonomous driving parameters are provided as additional inputs to the network. In order to model the time dynamics of the agent and the agent environment, gated recurrent units (GRUs) are used that require less computation and provide competitive performance compared to Long Short-Term Memory models (LSTM).

In the present embodiments, by simultaneously learning a robot having various settings in a simulation, reinforcement learning in multiple inputs can be performed at the same time, thereby obtaining various and unpredictable learning effects in the real world. Even if several randomly sampled parameters are used as a setting for autonomous driving learning, the total amount of data required for learning is the same or similar to the case of using one fixed parameter, so it is possible to create an adaptive algorithm with less data.

In FIG. 3 again, in step S320, the optimizer 202 may optimize the autonomous driving parameter by using the preference data for the driving image of the simulation robot. When a person views the driving image of the robot and gives feedback, the optimizer 202 may reflect the feedback value and learn the autonomous driving parameter in a manner that people prefer, thereby optimizing the autonomous driving parameter for user preference.

가 되고, 출력은 소프트맥스(softmax) 계산에 따른 스코어로서 유틸리티 함수

가 된다. 즉, 사용자 피드백에 따라 소프트맥스를 1 또는 0으로 학습하고 스코어가 가장 높게 나오는 파라미터를 찾는 것이다.The optimizer 202 may utilize a neural network that receives and reflects human feedback on a driving image of a robot in which autonomous driving parameters are set differently from each other. 6, the input of the neural network is an autonomous driving parameter

And the output is a utility function as the score according to the softmax calculation

Becomes. In other words, the softmax is learned as 1 or 0 according to user feedback, and the parameter with the highest score is found.

Even if there are agents that can adapt to a wide range of autonomous driving parameters, the problem of finding the optimal autonomous driving parameters for a given use case remains. Therefore, we propose a new Bayesian approach to optimizing autonomous driving parameters using preference data. In this embodiment, preference may be evaluated through pairwise comparisons that can be easily deduced.

가

보다 선호될 확률은 수학식 5와 같다.For example, the Bradley-Terry model can be used for model preference. Autonomous driving parameters

end

The probability of becoming more preferred is shown in Equation 5.

[Equation 5]

과

는

과

를 사용하여 수집한 로봇 궤적이고,

는

이

보다 선호되는 것을 나타내고,

은 유틸리티 함수이다. 정확한 선호도 평가를 위해서는 동일한 환경 및 경유지를 이용하여 궤적

과

를 수집한다. 선호 데이터에 유틸리티 함수

를 맞추고 이를 사용하여 새 자율주행 파라미터에 대한 환경 설정을 예측한다.here,

and

Is

and

Is the robot trajectory collected using

Is

this

Indicate that it is more preferred,

Is a utility function. For accurate evaluation of preference, trajectory is used using the same environment and waypoint.

and

To collect. Utility function on preferred data

Fit and use it to predict the environment settings for the new autonomous driving parameters.

이 있는 베이지안 신경망에서 유틸리티 함수

을 학습한다. 특히, 능동적으로 쿼리를 생성하기 위해 예측 불확실성에 대한 추정치를 사용함으로써 쿼리의 수를 최소화할 수 있다.Parameters for active learning of the preference model

Utility functions in Bayesian neural networks with

To learn. In particular, the number of queries can be minimized by actively using estimates of prediction uncertainty to generate queries.

As shown in the algorithm of FIG. 7, the neural network (FIG. 6) is trained in a direction that minimizes the negative log-likelihood (Equation 6) of the preference model.

[Equation 6]

를 시작으로

단계씩 네트워크를 훈련시킨다. 일례로, 수학식 7과 같이 설정하여 새로운 쿼리를 적극적으로 샘플링하기 위해 변경된 UCB(upper-confidence bounds)를 사용할 수 있다.Parameters from previous step in each iteration

Starting with

Train the network step by step. For example, it is possible to use a modified upper-confidence bounds (UCB) in order to actively sample a new query by setting it as shown in Equation 7.

[Equation 7]

와

는 네트워크의

포워드 패스(forward pass)로 계산된

의 평균과 표준 편차를 의미한다. 시뮬레이션 환경에서 일반적으로

앞에 나타나는

계수는 생략한다.here,

Wow

Of the network

Calculated as a forward pass

Mean the mean and standard deviation of. Typically in a simulation environment

Appearing in front

The coefficient is omitted.

균일하게 샘플링된 자율주행 파라미터 중

가 가장 높은

자율주행 파라미터를 사용하여 로봇의 궤적을 생성한다. 그런 다음,

의 새로운 선호도 쿼리를 능동적으로 생성한다. 이를 위해, 모든 자율주행 파라미터의 집합인 모든

에 대해

과

을 계산한다.

을

에서 가장 높은

의

로 하고,

를

에서 가장 높은

의

로 하여 샘플 집합이라 하자. 각각의 선호도 쿼리는

과

에서

과

가 균일하게 샘플링되는 자율주행 파라미터 쌍

으로 구성된다.

Among the uniformly sampled autonomous driving parameters

The highest

The robot's trajectory is created using the autonomous driving parameters. after that,

It actively creates a new preference query for. To this end, all autonomous driving parameters are set.

About

and

Calculate

of

Highest in

of

And

To

Highest in

of

Let it be a sample set. Each preference query is

and

in

and

Pairs of autonomous driving parameters that are uniformly sampled

It consists of.

In other words, the optimizer 202 shows two video clips of the robot driven with different parameters to users, investigates the preference of which video is more suitable for the use case, and then models the preference based on the uncertainty of the model. As new clips are created, parameters with high satisfaction can be found with a small number of preference data. Each time the connection strength of the neural network is calculated, it is sampled from a constant distribution, and in the process of actively generating queries using Bayesian neural networks, the number of queries required for overall training is effectively reduced by inducing learning with inputs with high uncertainty in the prediction results I can.

As described above, according to embodiments of the present invention, reinforcement learning in various environments can be simultaneously performed to achieve various and unpredictable learning effects in the real world, and an adaptive autonomous driving algorithm can be implemented without increasing data. Moreover, according to embodiments of the present invention, after modeling the preference indicating whether the driving image of the robot is appropriate as a use case, it is possible to optimize the autonomous driving parameter using a small number of preference data based on the uncertainty of the model. .

The apparatus described above may be implemented as a hardware component, a software component, and/or a combination of a hardware component and a software component. For example, the devices and components described in the embodiments include a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), and a programmable gate array (PLU). It may be implemented using one or more general purpose or special purpose computers, such as a logic unit), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications executed on the operating system. Further, the processing device may access, store, manipulate, process, and generate data in response to the execution of software. For the convenience of understanding, although it is sometimes described that one processing device is used, one of ordinary skill in the art, the processing device is a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it may include. For example, the processing device may include a plurality of processors or one processor and one controller. In addition, other processing configurations are possible, such as a parallel processor.

The software may include a computer program, code, instructions, or a combination of one or more of these, configuring the processing unit to operate as desired or processed independently or collectively. You can command the device. Software and/or data may be embodyed in any type of machine, component, physical device, computer storage medium or device to be interpreted by the processing device or to provide instructions or data to the processing device. have. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. In this case, the medium may be one that continuously stores a program executable by a computer, or temporarily stores a program for execution or download. Further, the medium may be a variety of recording means or storage means in a form in which a single or several pieces of hardware are combined, but is not limited to a medium directly connected to a computer system, but may be distributed on a network. Examples of media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, And ROM, RAM, flash memory, and the like may be configured to store program instructions. In addition, examples of other media include an app store that distributes applications, a site that supplies or distributes various software, and a recording medium or a storage medium managed by a server.

As described above, although the embodiments have been described by the limited embodiments and drawings, various modifications and variations are possible from the above description to those of ordinary skill in the art. For example, the described techniques are performed in a different order from the described method, and/or components such as systems, structures, devices, circuits, etc. described are combined or combined in a form different from the described method, or other components Alternatively, even if substituted or substituted by an equivalent, an appropriate result can be achieved.

Therefore, other implementations, other embodiments, and those equivalent to the claims also fall within the scope of the claims to be described later.

Claims (20)

In the autonomous driving learning method executed in a computer system,
The computer system includes at least one processor configured to execute computer readable instructions contained in a memory,
The autonomous driving learning method,
Learning robot autonomous driving by assigning different autonomous driving parameters to a plurality of robot agents in the simulation by the at least one processor through automatic setting by the system or direct setting by the manager
Self-driving learning method comprising a. The method of claim 1,
The learning step,
Simultaneously performing reinforcement learning by inputting randomly sampled autonomous driving parameters for the plurality of robot agents
Self-driving learning method, characterized in that. The method of claim 1,
The learning step,
Simultaneous learning of autonomous driving of the plurality of robot agents using a neural network composed of a fully-connected layer and gated recurrent units (GRU)
Self-driving learning method, characterized in that. The method of claim 1,
The learning step,
Using a sensor value obtained in real time from the robot as an input of a neural network for learning of the robot autonomous driving and an autonomous driving parameter that is randomly assigned in relation to the autonomous driving policy.
Self-driving learning method comprising a. The method of claim 1,
The autonomous driving learning method,
Optimizing, by the at least one processor, the autonomous driving parameter using preference data for the autonomous driving parameter
Self-driving learning method further comprising. The method of claim 5,
The step of optimizing,
Optimizing the autonomous driving parameters by reflecting feedback on a driving image of a robot in which the autonomous driving parameters are set differently from each other
Self-driving learning method, characterized in that. The method of claim 5,
The step of optimizing,
Evaluating a preference for the autonomous driving parameter through pairwise comparisons of the autonomous driving parameter
Self-driving learning method comprising a. The method of claim 5,
The step of optimizing,
Modeling the preference for the autonomous driving parameter using a Bayesian neural network model
Self-driving learning method comprising a. The method of claim 8,
The step of optimizing,
Generating a query for pairwise comparison of the autonomous driving parameters based on the uncertainty of the preference model
Self-driving learning method comprising a. A computer program stored in a non-transitory computer-readable recording medium for executing the autonomous driving learning method of any one of claims 1 to 9 on the computer system. A non-transitory computer-readable recording medium in which a program for executing the autonomous driving learning method of any one of claims 1 to 9 is recorded on a computer. In a computer system,
At least one processor configured to execute computer readable instructions contained in memory
Including,
The at least one processor,
A learning unit that learns robot autonomous driving by assigning different autonomous driving parameters to a plurality of robot agents in the simulation through automatic setting by the system or through direct setting by the administrator.
Computer system comprising a. The method of claim 12,
The learning unit,
Simultaneously performing reinforcement learning by inputting randomly sampled autonomous driving parameters for the plurality of robot agents
Computer system, characterized in that. The method of claim 12,
The learning unit,
Learning the autonomous driving of the plurality of robot agents at the same time using a neural network composed of a fully connected layer and GRU
Computer system, characterized in that. The method of claim 12,
The learning unit,
Using a sensor value acquired in real time from a robot as an input to a neural network for learning of the robot's autonomous driving and an autonomous driving parameter that is randomly assigned in relation to the autonomous driving policy.
Computer system, characterized in that. The method of claim 12,
The at least one processor,
Optimizer for optimizing the autonomous driving parameter using preference data for the autonomous driving parameter
Computer system further comprising a. The method of claim 16,
The optimization unit,
Optimizing the autonomous driving parameters by reflecting feedback on a driving image of a robot in which the autonomous driving parameters are set differently from each other
Computer system, characterized in that. The method of claim 16,
The optimization unit,
Evaluating the preference for the autonomous driving parameters through pairwise comparison of the autonomous driving parameters
Computer system, characterized in that. The method of claim 16,
The optimization unit,
Modeling the preference for the autonomous driving parameter using a Bayesian neural network model
Computer system, characterized in that. The method of claim 19,
The optimization unit,
Generating a query for pairwise comparison of the autonomous driving parameters based on the uncertainty of the preference model
Computer system, characterized in that.

Images