KR20230045826A - Method, computer system, and computer program for reinforcement learning-based navigation adaptable to sensor configuration and robot shape - Google Patents

Abstract

A reinforcement learning-based autonomous driving method, a computer system, and a computer program adaptable to a sensor configuration and a robot shape are disclosed. The autonomous driving learning method may include a step of learning robot autonomous driving by randomly assigning robot setting parameters, including a robot shape and a sensor configuration, as autonomous driving parameters to at least one robot agent in a simulation. Thus, the present invention provides a reinforcement learning algorithm that allows robots with various shapes and sensor configurations to drive autonomously.

Description

Reinforcement learning-based autonomous driving method adaptable to sensor configuration and robot shape, computer system, and computer program

The description below relates to the autonomous driving technology of robots.

The self-driving robot is a technology applied to robots widely used in the industrial field. For example, after obtaining speed information and azimuth information by using an odometry method, the moving distance and direction from the previous position to the next position You can recognize your location and direction by calculating the information on .

For example, Korean Patent Registration No. 10-1771643 (registered on August 21, 2017) discloses an autonomous robot capable of automatically moving to a destination by recognizing absolute coordinates and a navigation method thereof.

It provides a reinforcement learning algorithm that allows robots with various shapes and sensor configurations to drive autonomously.

An algorithm for learning by randomizing the robot shape and sensor configuration in the simulation is provided.

An autonomous driving learning method executed in a computer system, wherein the computer system includes at least one processor configured to execute computer readable instructions contained in a memory, and the autonomous driving learning method is performed by the at least one processor. , Learning autonomous driving by randomly assigning robot setting parameters including a robot shape and sensor configuration as autonomous driving parameters to at least one robot agent in the simulation to learn autonomous driving. provides a way

According to one aspect, in the learning step, reinforcement learning may be performed for the robot agent using randomly sampled robot setting parameters as an input.

According to another aspect, the robot setting parameter corresponding to the sensor configuration may include at least one of a field of view (FOV) and a maximum sensing range of the sensor system.

According to another aspect, the learning may include sampling the robot configuration parameters based on hindsight domain randomization.

According to another aspect, in the learning step, autonomous navigation of the robot agent may be learned using an asymmetric actor-critic (AC) model-based neural network.

According to another aspect, the learning may include using sensor values obtained in real time from a robot as inputs of a neural network for learning robot autonomous driving and the robot setting parameters randomly given in relation to an autonomous driving policy. steps may be included.

According to another aspect, the self-driving learning method may further include performing, by the at least one processor, fine-tuning on the robot agent learned with the robot setting parameters. .

According to another aspect, the learning may include learning autonomous driving of the robot agent using an asymmetric AC (Actor-Critic) model-based neural network, and the performing may include a normalization loss in the loss of the asymmetric AC model. It may include adding a term (regularization loss term).

According to another aspect, the autonomous driving learning method may further include optimizing, by the at least one processor, the robot setting parameters using preference data for the robot setting parameters. .

According to another aspect, the optimizing may include optimizing the robot setting parameters by reflecting feedback on a driving image of the robot for which the robot setting parameters are set.

A computer program stored in a computer readable recording medium is provided to execute the self-driving learning method on the computer system.

A computer system comprising at least one processor configured to execute computer readable instructions contained in a memory, wherein the at least one processor provides a robot shape and sensor configuration as autonomous driving parameters to at least one robot agent in a simulation. Provided is a computer system including a learning unit for learning robot self-driving by randomly assigning robot setting parameters.

According to embodiments of the present invention, it is possible to provide a generalized reinforcement learning agent that can be used for various sensor configurations and robot types.

According to embodiments of the present invention, it is possible to use a hindsight domain randomization method to overcome the vast search space of sensor settings and an asymmetric AC architecture to achieve better performance without requiring expensive context data in a real environment. there is.

According to embodiments of the present invention, the learned agent can be used as a backbone agent, which can be fine-tuned with a small amount of data to be competitive and perform better than the baseline given a specific robot shape and sensor settings.

1 is a block diagram for explaining an example of an internal configuration of a computer system according to an embodiment of the present invention.
2 is a diagram showing an example of components that may be included in a processor of a computer system according to an embodiment of the present invention.
3 is a flowchart illustrating an example of an autonomous driving learning method that can be performed by a computer system according to an embodiment of the present invention.
Figure 4 shows an example of a robot setting parameter boundary in one embodiment of the present invention.
5 illustrates an example of a neural network for adaptive self-driving policy learning according to an embodiment of the present invention.
6 illustrates an example of a neural network for learning a utility function according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

Embodiments of the present invention relate to autonomous driving technology for robots.

Embodiments including those specifically disclosed in this specification can provide a new deep reinforcement learning-based autonomous driving technology that can be applied to various robot types and sensor configurations.

1 is a block diagram illustrating an example of a computer system according to one embodiment of the present invention. For example, an autonomous driving learning system according to embodiments of the present invention may be implemented by the computer system 100 shown in FIG. 1 .

As shown in FIG. 1, the computer system 100 is a component for executing the autonomous driving learning method according to embodiments of the present invention, and includes a memory 110, a processor 120, a communication interface 130, and input/output. Interface 140 may be included.

The memory 110 is a computer-readable recording medium and may include a random access memory (RAM), a read only memory (ROM), and a permanent mass storage device such as a disk drive. Here, a non-perishable mass storage device such as a ROM and a disk drive may be included in the computer system 100 as a separate permanent storage device separate from the memory 110 . Also, an operating system and at least one program code may be stored in the memory 110 . These software components may be loaded into the memory 110 from a recording medium readable by a separate computer from the memory 110 . The separate computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD/CD-ROM drive, and a memory card. In another embodiment, software components may be loaded into the memory 110 through the communication interface 130 rather than a computer-readable recording medium. For example, software components may be loaded into memory 110 of computer system 100 based on a computer program installed by files received over network 160 .

The processor 120 may be configured to process commands of a computer program by performing basic arithmetic, logic, and input/output operations. Instructions may be provided to processor 120 by memory 110 or communication interface 130 . For example, processor 120 may be configured to execute received instructions according to program codes stored in a recording device such as memory 110 .

Communication interface 130 may provide functionality for computer system 100 to communicate with other devices via network 160 . For example, a request, command, data, file, etc. generated according to a program code stored in a recording device such as the memory 110 by the processor 120 of the computer system 100 is transferred to a network ( 160) to other devices. Conversely, signals, commands, data, files, etc. from other devices may be received into the computer system 100 via the communication interface 130 of the computer system 100 via the network 160 . Signals, commands, data, etc. received through the communication interface 130 may be transmitted to the processor 120 or memory 110, and files, etc. may be stored in the computer system 100. permanent storage).

The communication method is not limited, and may include not only a communication method utilizing a communication network (eg, a mobile communication network, wired Internet, wireless Internet, and broadcasting network) that the network 160 may include, but also short-distance wired/wireless communication between devices. there is. For example, the network 160 may include a personal area network (PAN), a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN), and a broadband network (BBN). , one or more arbitrary networks such as the Internet. In addition, the network 160 may include any one or more of network topologies including a bus network, a star network, a ring network, a mesh network, a star-bus network, a tree or a hierarchical network, and the like. Not limited.

The input/output interface 140 may be a means for interface with the input/output device 150 . For example, the input device may include devices such as a microphone, keyboard, camera, or mouse, and the output device may include devices such as a display and a speaker. As another example, the input/output interface 140 may be a means for interface with a device in which functions for input and output are integrated into one, such as a touch screen. The input/output device 150 may be configured as one device with the computer system 100 .

Also, in other embodiments, computer system 100 may include fewer or more elements than those of FIG. 1 . However, there is no need to clearly show most of the prior art components. For example, the computer system 100 may be implemented to include at least some of the aforementioned input/output devices 150 or may further include other components such as a transceiver, a camera, various sensors, and a database.

First, the related studies are as follows.

<Deep Reinforcement Learning-based Mobile Robot Autonomous Driving>

Recently, deep reinforcement learning methods for autonomous driving have been actively studied, and robot autonomous driving technology using reinforcement learning shows higher performance than path planning-based autonomous driving.

Reinforcement learning-based autonomous driving technology has better performance and robustness than path planning-based autonomous driving technology in terms of collision avoidance with complex obstacles or moving obstacles, but it can be applied to robots with different shapes or sensor configurations from the learned settings. .

These embodiments make it possible to generalize reinforcement learning agents with various settings for sensor configurations and robot types.

<Domain Randomization for Deep Reinforcement Learning>

Domain randomization is widely used in robot reinforcement learning to improve robustness and performance. Although autonomous driving parameters such as maximum speed, acceleration, and safety distance from obstacles of the robot can be randomized, the effect of randomization decreases as the search space for autonomous driving parameters increases.

The present embodiments may apply hindsight domain randomization to improve generalization to various sensor configurations.

<Relationship with meta-reinforcement learning>

Meta-reinforcement learning is a framework for building reinforcement learning agents, which can quickly adapt to new tasks through learning on many tasks sampled from batches of reinforcement learning agents.

The present embodiments use context (robot setting) information for quick adaptation in the setting distribution and learn how to form situation variables from observation data. Unlike the meta reinforcement learning method, situation variable information can be directly provided to the agent. can

<Asymmetric AC (Actor-Critic) Architecture>

The AC method is a reinforcement learning algorithm composed of an actor model that learns a value function and a critic model that learns a policy that maximizes the learned value function.

The AC method is used to learn the vision-based robot grasping task, and there are studies that provide information of other agents to the critic model in a multi-agent reinforcement learning setting.

A recent study uses an asymmetric AC architecture for the navigation of a mobile robot equipped with a sensor with a limited field of view.

In this embodiment, we use an approach similar to the asymmetric AC architecture to improve the agent's performance under extreme randomization of sensor configuration without requiring expensive information.

2 is a diagram showing an example of components that may be included in a processor of a computer system according to an embodiment of the present invention, and FIG. 3 is an autonomous driving that a computer system according to an embodiment of the present invention may perform. It is a flow chart showing an example of a learning method.

As shown in FIG. 2 , the processor 120 may include a learning unit 210 and a fine adjustment unit 220 . Components of the processor 120 may represent different functions performed by the processor 120 according to a control command provided by at least one program code. For example, the learning unit 210 may be used as a functional expression that operates to control the computer system 100 so that the processor 120 learns autonomous navigation of the robot based on deep reinforcement learning.

The processor 120 and components of the processor 120 may perform steps S310 to S320 included in the autonomous driving learning method of FIG. 3 . For example, the processor 120 and components of the processor 120 may be implemented to execute an instruction according to an operating system code included in the memory 110 and at least one program code described above. Here, at least one program code may correspond to a code of a program implemented to process an autonomous driving learning method.

The self-driving learning method may not occur in the shown order, and some of the steps may be omitted or additional processes may be included.

The processor 120 may load a program code stored in a program file for an autonomous driving learning method into the memory 110 . For example, a program file for an autonomous driving learning method may be stored in a permanent storage device separate from the memory 110, and the processor 120 may store a program code from a program file stored in the permanent storage device through a bus. The computer system 100 may be controlled to be loaded into 110. At this time, each of the processor 120, the learning unit 210, and the fine-tuning unit 220 included in the processor 120 executes a command of a corresponding part among the program codes loaded in the memory 110 to perform subsequent steps ( There may be different functional representations of the processor 120 for executing S310 to S320. For the execution of steps S310 to S320, the processor 120 and components of the processor 120 may directly process an operation according to a control command or control the computer system 100.

The formulation of the reinforcement learning-based autonomous driving problem is as follows.

The problem setting in the present invention is the autonomous driving of a wheeled mobile robot in which the robot must sequentially pass through waypoints without colliding with obstacles.

An agent (i.e., a robot) travels a route to a destination, and at this time, the route can be expressed as a series of waypoints. When the agent reaches its last waypoint (destination), it is given a new goal and waypoint, modeling the task as a partially observed Markov decision process (S, A, Ω, r, p _trans , po _obs ) do. At this time, S is states, A is actions, Ω is observations, r is reward function, p _trans is conditional state-transition, and p _obs is observation It means observation probabilities.

(1) Robot setting parameters

The robot setting parameter w relates to navigation parameters that affect the operation of the reinforcement learning-based autonomous navigation agent. In this embodiment, based on the characteristics of the robot to be considered for navigation performance, a robot setting parameter w composed of nine elements as shown in Equation 1 is considered.

[Equation 1]

where w _maxL is the maximum linear speed, w _minL is the minimum linear velocity, w _maxA is the maximum angular speed, w _accL is the linear acceleration, and w _dccL is For linear deceleration, w _accA means angular acceleration.

And, as a robot setting parameter related to the sensor configuration, w _enabled means the field of view (FOV) of the sensor system, and w _maxR means the maximum sensing range of the sensor system.

). 일례로, 로봇 형태를 볼록 팔각형(convex octagon)으로 모델링 가능하고 로봇 형태와 센서 구성(w_enabled, w_maxR)이 좌우 대칭인 것으로 가정한다.Finally, w _shape as a robot setting parameter for the robot shape means the robot shape (

). As an example, it is assumed that the robot shape can be modeled as a convex octagon and the robot shape and sensor configuration (w _enabled , w _maxR ) are symmetrical.

The robot shape can be expressed as a vector as shown in Equation 2.

[Equation 2]

Each vertex of an octagon is (x ₁ , y ₁ ), (x ₂ , y ₂ ), (-x ₃ , y ₃ ), (-x ₄ , y ₄ ), (-x ₃ , -y ₃ ), ( x ₂ , -y ₂ ) and (x ₁ , -y ₁ ). For all coordinates and angles, use the robot's local coordinate frame unless otherwise specified.

In addition, the size of the robot can be defined as in Equation 3.

[Equation 3]

(2) states and observations

We model the state of the agent as the position of all waypoints, the position and dynamics of all obstacles in the environment and the robot itself (speed, acceleration, policy, etc.).

A real agent can only observe some information using sensors, and it can observe the position of obstacles using range sensors, waypoint positions, and its own speed.

을 정의한다. o_full의 i번째 요소는 로봇의 x축에서 2(i-1) 내지 2i도 내에 위치한 가장 가까운 장애물까지의 거리를 나타낸다.To define range sensor observations, first full range observation

define o The i-th element of _full represents the distance to the nearest obstacle located within 2(i-1) to 2i degrees from the robot's x-axis.

과 최대값을 설명하는

을 추가로 정의하며, 이는 각 bin의 최대 탐지 범위를 나타낸다.Range sensor observations in various sensor configurations, such as limited FOV or sparse time of flight (TOF) sensors. o A binary vector indicating whether each element can observe its angle to model _range .

and describes the maximum

is further defined, which represents the maximum detection range of each bin.

In other words, the range sensor observation o _range can be defined as in Equation 4.

[Equation 4]

Here, the subscript denotes the i-th element of the vector.

을 정의하며, 여기서 o_full,i가 다른 에이전트에서 온다면 o_agent,i는 1이고 그렇지 않다면 0이다.additionally,

, where o agent _{,i is 1 if o full ,i} comes from another agent and 0 otherwise.

는 수학식 5와 같이 정의된다.route information

Is defined as in Equation 5.

[Equation 5]

Here, d _i is the relative distance to the next ith waypoint cut to [0.01, 100] m (meters), and θ _i means the relative angle to the next ith waypoint.

의 관측은 수학식 6과 같이 정의된다.Finally, the speed of the agent itself

The observation of is defined as in Equation 6.

[Equation 6]

Here, l _c is the current linear velocity of the agent, a _c is the current angular velocity of the agent, l _a is the desired linear velocity of the agent in the previous action, and a _a is the desired angular velocity of the agent in the previous action.

(3) actions

을 사용한다.A normalized two-dimensional vector as an action representing the desired linear velocity and desired angular velocity of the robot.

Use

The desired linear velocity l _a and the desired angular velocity a _a of the robot are defined as in Equation 7.

[Equation 7]

Here, w _maxL is the maximum linear velocity, w _minL is the minimum linear velocity, and w _maxA is the maximum angular velocity. u ₀ means the first element of the action, and u ₁ means the second element of the action.

The desired linear velocity l _a and the desired angular velocity _{a a} calculated from the action are transmitted to the robot, and the robot updates the movement velocity based on Equation 8.

[Equation 8]

Here, l' _c means the updated linear velocity, and _a'c means the updated angular velocity. l _c is the current linear velocity, a _c is the current angular velocity, l _a is the desired linear velocity calculated from the action, and a _a is the desired angular velocity calculated from the action. As robot setting parameters related to acceleration, w _accL means linear acceleration, w _dccL means linear deceleration, and w _accA means angular acceleration. Δt denotes the time scale.

(4) Compensation

In this embodiment, a compensation function may be applied that recommends that the agent follow waypoints as efficiently as possible while avoiding collisions in a multi-agent environment and complying with correct passing rules. In addition, the reward is designed as simply as possible so that the agent can learn a good policy even if the settings such as sensor configuration and robot shape are extremely randomized.

Referring to FIG. 3 , an example of the autonomous driving learning method according to the present invention includes the following two steps.

In step S310, the learning unit 210 randomly assigns robot setting parameters including sensor configurations and robot shapes to several robots in a simulation environment in order to learn autonomous driving policies that can adapt to various shapes and sensor configurations. Robots can learn autonomous driving.

In this embodiment, the hindsight domain randomization method is used to learn a navigation agent that can be generalized to various settings such as sensor configuration and robot type.

The learning unit 210 may use sensor data and robot setting parameters as inputs of a neural network for autonomous driving learning. The sensor data is a sensor value obtained in real time from the robot, and may include, for example, a time of flight (ToF) sensor value, current speed, odometry, driving direction, obstacle position, and the like. The robot setting parameter is a value given by random setting, and can be automatically set by the system or directly set by an administrator. In particular, the robot setting parameters may include a robot shape (w _shape ) and sensor configuration (w _enabled , w _maxR ).

In other words, the learning unit 210 may learn an adaptive self-driving policy by arranging several agents having randomly assigned robot setting parameters in a simulation environment.

When a simulation episode starts, the learning unit 210 samples w _maxL , w _minL , w _maxA , w _accL , w _dccL , and w _accA of each agent from the uniform distribution presented in the table of FIG. 4 .

For w _shape , extract x ₂ , y ₁ , x ₃ , y ₄ from uniform distribution U(0,0.35)m and then extract x ₁ , y ₂ , x ₄ , y ₃ from uniform distribution U(0.15,0.35)m. sample Thereafter, the learning unit 210 calculates x ₁ = max(x ₁ , x ₂ ), y ₂ = max(y ₁ , y ₂ ), x ₄ = max(x ₃ , x ₄ ), y to fit the convex assumption. Set ₃ = max(y ₃ , y ₄ ).

Then, the learning unit 210 samples w _enabled using block dropout and bin dropout. Divide the first half of w _enabled into 9 blocks. Here, the ith block is composed of w _{enabled, 10(i-1):10i} . The elements in the first block are always set to a value of 1 because they are difficult to navigate without a sensor in front of them. For other blocks, set the element value to 0 with probability 0.5, otherwise set to 1. For each active block, we sample the bin dropout probability p _dropout at U(0.1, 0.9) and set each element of the block to zero using the p _dropout probability. In the first half of w _maxR , each element is sampled at U(0.5,5)m.

Finally, in order to comply with the horizontal symmetry assumption, the sensor configuration (w _enabled , w _maxR ) is set as shown in Equation 9.

[Equation 9]

The learner 210 not only randomizes the sensor configurations at the beginning of an episode, but also resamples the sensor configurations during each training iteration. To do this, for each data point in the mini-batch, we store the full range of observations o _full in the replay buffer, samples w _enabled and w _maxR , then use the new sensor configuration before each training iteration. to calculate O _range . Hindsight domain randomization can improve agent performance in extreme randomization by reusing trajectories collected in different settings.

5 and 6 show an example of a neural network structure for self-driving learning in one embodiment of the present invention.

과

는 공유 가중치를 이용하여 계산된다.The neural network architecture for self-driving learning according to the present invention uses an adaptive policy learning structure (FIG. 5) and a utility function learning (FIG. 6) structure. FC represents a fully-connected layer, BayesianFC represents a Bayesian fully-connected layer, and merged branches represent concatenation. utility function

class

is calculated using shared weights.

As shown in FIG. 5, the agent's autonomous driving parameters (robot setting parameters including sensor configuration and robot shape) are provided as additional inputs to the network. To model the temporal dynamics of agents and agent environments, we use gated recurrent units (GRUs), which require less computation compared to LSTMs (Long Short-Term Memory models) while providing competitive performance.

In the present embodiments, by training robots in various settings in a simulation, reinforcement learning can be simultaneously performed on various inputs, so that various and unpredictable learning effects in the real world can be obtained. Even if multiple randomly sampled parameters are used as settings for self-driving learning, the total amount of data required for learning is the same or similar to the case of using one fixed parameter, so an adaptive algorithm can be created with less data.

In this embodiment, a soft actor-critic (SAC) algorithm can be used as an example of a reinforcement learning algorithm composed of an actor model that learns a value function and a critic model that learns a policy that maximizes the value function learned through the actor model. Since it can be difficult to learn an accurate value function with only the information provided to the agent due to sensor configuration randomization, an asymmetric AC architecture can be used to provide more information to the critic model.

In particular, the actor network uses o _range as the single-channel data as input to the convolutional layer. On the other hand, the critic network can use 5-channel data that is a connection of o _range , o _full , o _enabled , o _maxR , and o _agent . Since the actor network does not require expensive information such as o _full or o _agent , agents can be deployed in the real environment without expensive data. In addition, w _enabled and w _maxR are not provided to the actor network, but p _anchor is provided only to the critic network.

Referring back to FIG. 3 , in step S320, the fine-tuning unit 220 may perform fine-tuning for the learned agent with robot setting parameters of random settings. The fine-tuning unit 220 may fine-tune the learned agent with random settings to improve sample efficiency and performance when specific robot settings are completed through reinforcement learning.

The fine adjustment unit 220 adds a regularization loss term defined as shown in FIG. 10 to both the actor loss and the critic loss.

[Equation 10]

Here, θ is the current parameter of the model, θ _pre is the parameter of the pretraining model, and λ is the coefficient to balance the reinforcement learning loss and regularization loss.

In addition, the processor 120 may optimize robot setting parameters using preference data for driving images of the simulation robot. The processor 120 may optimize robot setting parameters for user preference by learning robot setting parameters in a manner preferred by people by reflecting feedback values when a person gives feedback after viewing a driving image of the robot.

가 되고, 출력은 소프트맥스(softmax) 계산에 따른 스코어로서 유틸리티 함수

가 된다. 즉, 사용자 피드백에 따라 소프트맥스를 1 또는 0으로 학습하고 스코어가 가장 높게 나오는 파라미터를 찾는 것이다.The processor 120 may utilize a neural network that receives and reflects human feedback on driving images of robots in which robot setting parameters are set differently. Referring to FIG. 6, the input of the neural network is a robot setting parameter.

, and the output is a utility function as a score according to softmax calculation.

becomes That is, according to user feedback, softmax is learned as 1 or 0 and the parameter with the highest score is found.

In this embodiment, preference can be evaluated through pairwise comparisons that can be easily derived. In other words, the processor 120 shows two video clips of a robot traveling with different parameters to users, investigates their preference for which video is more appropriate for the use case, and then models the preference, based on the uncertainty of the model. As new clips are created, parameters with high satisfaction can be found with a small number of preference data. Each time the connection strength of the neural network is calculated, it is sampled from a certain distribution and, in particular, in the process of actively generating queries using Bayesian neural networks, learning is induced with inputs with high uncertainty in prediction results, effectively reducing the number of queries required for overall learning. can

As such, according to the embodiments of the present invention, it is possible to provide a generalized reinforcement learning agent that can be used for various sensor configurations and robot types. In particular, according to embodiments of the present invention, a hindsight domain randomization method for overcoming the vast search space of sensor settings and an asymmetric AC architecture for achieving better performance without requiring expensive context data in a real environment are developed. can be used Since the agent learned by the method according to the present invention can be used as a backbone agent, it can be fine-tuned with a small amount of data to be competitive and perform better than the reference value given a specific robot shape and sensor settings.

The devices described above may be implemented as hardware components, software components, and/or a combination of hardware components and software components. For example, devices and components described in the embodiments include a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), and a programmable PLU (programmable logic unit). logic unit), microprocessor, or any other device capable of executing and responding to instructions. The processing device may run an operating system (OS) and one or more software applications running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of software. For convenience of understanding, there are cases in which one processing device is used, but those skilled in the art will understand that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it can include. For example, a processing device may include a plurality of processors or a processor and a controller. Other processing configurations are also possible, such as parallel processors.

Software may include a computer program, code, instructions, or a combination of one or more of the foregoing, which configures a processing device to operate as desired or processes independently or collectively. You can command the device. The software and/or data may be embodied in any tangible machine, component, physical device, computer storage medium or device to be interpreted by or to provide instructions or data to a processing device. there is. Software may be distributed on networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer readable media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. In this case, the medium may continuously store a program executable by a computer or temporarily store the program for execution or download. In addition, the medium may be various recording means or storage means in the form of a single or combined hardware, but is not limited to a medium directly connected to a certain computer system, and may be distributed on a network. Examples of the medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROM and DVD, magneto-optical media such as floptical disks, and ROM, RAM, flash memory, etc. configured to store program instructions. In addition, examples of other media include recording media or storage media managed by an app store that distributes applications, a site that supplies or distributes various other software, and a server.

As described above, although the embodiments have been described with limited examples and drawings, those skilled in the art can make various modifications and variations from the above description. For example, the described techniques may be performed in an order different from the method described, and/or components of the described system, structure, device, circuit, etc. may be combined or combined in a different form than the method described, or other components may be used. Or even if it is replaced or substituted by equivalents, appropriate results can be achieved.

Therefore, other implementations, other embodiments, and equivalents of the claims are within the scope of the following claims.

Claims (20)

In the self-driving learning method executed in a computer system,
The computer system includes at least one processor configured to execute computer readable instructions contained in a memory;
The self-driving learning method,
The at least one processor randomly assigns robot setting parameters including a robot shape and sensor configuration as autonomous driving parameters to at least one robot agent in simulation to learn robot autonomous driving step
Self-driving learning method comprising a. According to claim 1,
The learning step is
Performing reinforcement learning for the robot agent with randomly sampled robot setting parameters as input
Characterized by an autonomous driving learning method. According to claim 1,
The robot setting parameter corresponding to the sensor configuration includes at least one of a field of view (FOV) and a maximum sensing range of the sensor system.
Characterized by an autonomous driving learning method. According to claim 1,
The learning step is
sampling the robot setup parameters based on hindsight domain randomization.
Self-driving learning method comprising a. According to claim 1,
The learning step is
Learning autonomous driving of the robot agent using an asymmetric AC (Actor-Critic) model-based neural network
Characterized by an autonomous driving learning method. According to claim 1,
The learning step is
Using sensor values obtained in real time from the robot and the robot setting parameters randomly assigned in relation to autonomous driving policies as inputs of the neural network for learning the autonomous driving of the robot.
Self-driving learning method comprising a. According to claim 1,
The self-driving learning method,
Performing, by the at least one processor, fine-tuning on the robot agent learned with the robot setting parameters.
Self-driving learning method further comprising. According to claim 7,
The learning step is
Learning autonomous driving of the robot agent using an asymmetric AC (Actor-Critic) model-based neural network,
The above steps are
Adding a regularization loss term to the loss of the asymmetric AC model.
Self-driving learning method comprising a. According to claim 1,
The self-driving learning method,
Optimizing, by the at least one processor, the robot setting parameters using preference data for the robot setting parameters.
Self-driving learning method further comprising. According to claim 9,
The optimizing step is
Optimizing the robot setting parameters by reflecting feedback on the driving image of the robot in which the robot setting parameters are set
Characterized by an autonomous driving learning method. A computer program stored in a computer readable recording medium in order to execute the autonomous driving learning method according to any one of claims 1 to 10 in the computer system. In a computer system,
at least one processor configured to execute computer readable instructions contained in memory;
including,
The at least one processor,
A learning unit that learns robot autonomous driving by randomly assigning robot setting parameters including robot shape and sensor configuration as autonomous driving parameters to at least one robot agent in the simulation.
A computer system comprising a. According to claim 12,
The learning unit,
Performing reinforcement learning with randomly sampled robot setting parameters as input for the robot agent
Characterized by a computer system. According to claim 12,
The robot setting parameter corresponding to the sensor configuration includes at least one of the FOV and maximum detection range of the sensor system.
Characterized by a computer system. According to claim 12,
The learning unit,
sampling the robot configuration parameters based on hindsight domain randomization.
Characterized by a computer system. According to claim 12,
The learning unit,
Learning autonomous driving of the robot agent using an asymmetric AC (Actor-Critic) model-based neural network
Characterized by a computer system. According to claim 12,
The learning unit,
Using sensor values obtained in real time from the robot and the robot setting parameters randomly assigned in relation to autonomous driving policies as inputs of the neural network for learning the autonomous driving of the robot.
Characterized by a computer system. According to claim 12,
The at least one processor,
Fine-tuning unit for performing fine-tuning on the robot agent learned as the robot setting parameter
A computer system comprising a. According to claim 18,
The learning unit,
Learning autonomous driving of the robot agent using an asymmetric AC (Actor-Critic) model-based neural network,
The fine adjustment unit,
Adding a normalized loss term to the loss of the asymmetric AC model
Characterized by a computer system. According to claim 12,
The at least one processor,
Optimizing the robot setting parameters using preference data for the robot setting parameters
Characterized by a computer system.

Images