JP7248053B2 - Control device and control method - Google Patents

Description

The present invention relates to a control device and control method.

In general, when manufacturing and selling vehicles such as ordinary automobiles, the fuel consumption and exhaust gas are measured when the vehicle is driven in a specific driving pattern (hereinafter referred to as "mode") stipulated in the country or region. It is required to conduct a test and display the results of this test.
The mode can be represented, for example, by a graph of the relationship between the time from the start of running and the vehicle speed to be reached.
The vehicle speed to be reached is sometimes referred to as command vehicle speed in terms of a command given to the vehicle regarding the speed to be achieved.
A test for measuring fuel consumption and exhaust gas is performed by placing a vehicle on a chassis dynamometer and driving the vehicle according to a mode by an autopilot robot (Drive Robot (registered trademark)) installed in the vehicle.
A permissible error range is defined for the commanded vehicle speed, and the test is invalidated when the vehicle speed falls outside the permissible error range.
Therefore, the control of the autopilot robot requires high followability to the commanded vehicle speed, and the autopilot robot is controlled by a learning model learned by reinforcement learning.

Patent Literature 1, which is an example of conventional technology, discloses a control device and a control method for an autopilot robot controlled by a learning model learned by reinforcement learning.

Patent Literature 2, which is an example of conventional technology, discloses a learning system and a learning method that are learned by reinforcement learning of an operation inference learning model that creates a vehicle model and controls an autopilot robot.

In reinforcement learning, a controlled object is controlled by trial and error, and a control method, that is, a policy, is learned so as to increase an evaluation value called a reward.
In continuous value control learning, trial and error is generally expressed as adding random perturbations to the control value according to the current learning state.
In the learning progress of reinforcement learning, it is important to have an experience that increases the value of the reward to be obtained (hereinafter referred to as "good experience" in this specification.), and random perturbation It is expected that the explorability of

JP 2020-56737 A JP 2020-148593 A

However, according to the conventional technology described above, a good experience in the early stage of learning occurs by chance, and it is assumed that a good experience cannot be obtained.
There is a problem that if a good experience cannot be obtained, learning does not progress, or erroneous learning such as using only the accelerator or brake pedal may progress.
Incorrect learning may result in damage to the real vehicle being controlled.

The present invention has been made in view of the above, and an object of the present invention is to make it easier to have a good experience during learning and to streamline the progress of learning in the early stages of learning.

One aspect of the present invention that solves the above-described problems and achieves the object is a control device that controls the controlled object so that the state of the controlled object matches a command, wherein the controlled object is controlled based on a reinforcement learning algorithm. and a sub-operation content inference unit that outputs a control command based on control theory. After exiting the initial stage, a determination is made to adopt the control command output by the operation content inference unit, a control method determination unit that outputs the adopted control command, and the operation of the controlled object is controlled by the adopted control command. and a control device.

Alternatively, in one aspect of the present invention that solves the above-described problems and achieves the object, an autopilot robot mounted on a vehicle that drives the vehicle is controlled so that the vehicle runs according to a prescribed command vehicle speed. A control device for the autopilot robot, comprising: an operation content inference unit for outputting a control command for operating the vehicle based on a reinforcement learning algorithm; a secondary operation content inference unit for outputting a control command based on control theory; Initially, the control command output by the secondary operation content inference unit is adopted, and after the initial stage of learning is passed, a determination is made to adopt the control command output by the operation content inference unit, and the adopted control command is output. A control device for an autopilot robot, comprising: a method determination unit; and a vehicle operation control unit that controls operation of the vehicle according to the adopted control command.

In the control device for the autopilot robot configured as described above, it is determined based on the correlation of the operation content sequence whether it is in the initial stage of learning or out of the initial stage of learning. and the behavior of the sub-operation content inference unit are similar to each other.

In the control device for the autopilot robot configured as described above, it is preferable that pre-learning using a simulation using a test device model is performed before learning under the environment of the actual test device.

Alternatively, in one aspect of the present invention that solves the above-described problems and achieves the object, an autopilot robot mounted on a vehicle that drives the vehicle is controlled so that the vehicle runs according to a prescribed command vehicle speed. A control method for the autopilot robot, comprising: outputting a control command for operating the vehicle based on a reinforcement learning algorithm; outputting a control command based on a control theory; adopting a control command based on the control theory after the initial stage of learning has passed, and outputting the adopted control command; and controlling the operation of the vehicle according to the adopted control command. , is a control method for an autopilot robot.

ADVANTAGE OF THE INVENTION According to this invention, it becomes easy to have a good experience at the time of learning, and learning progress in the early stage of learning can be made efficient.

FIG. 1 is a diagram showing an outline of a test environment using a drive robot, which is an autopilot robot, according to Embodiment 1. FIG. FIG. 2 is a functional block diagram showing the test device according to the first embodiment and the control device for the autopilot robot according to the first embodiment. FIG. 3 is a functional block diagram showing a test device according to the third embodiment and a control device for an autopilot robot according to the third embodiment.

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments for carrying out the present invention will be described with reference to the accompanying drawings.
However, the present invention is not to be construed as limited by the description of the following embodiments.

(Embodiment 1)
FIG. 1 is a diagram showing an outline of a test environment using a drive robot 11, which is an autopilot robot in this embodiment.
FIG. 2 is a functional block diagram showing the test device 1 according to this embodiment and the control device 2 for the autopilot robot according to this embodiment.

A test apparatus 1 includes a drive robot 11 , a vehicle 12 and a chassis dynamometer 13 .
The vehicle 12 is a vehicle under test, the performance of which is measured and placed on the floor of the test environment.
The chassis dynamometer 13 is installed below the floor surface of the test environment and configured to measure the characteristics of the vehicle 12 by running the vehicle 12 on chassis rollers instead of on the road.
Vehicle 12 is arranged such that drive wheels 121 , which are front wheels of vehicle 12 , are positioned above chassis dynamometer 13 .
When the drive wheels 121 rotate, the chassis dynamometer 13 rotates in the direction opposite to the rotation of the drive wheels 121 .

The drive robot 11 is a machine that includes actuators 110a and 110b, is installed in the driver's seat 122 of the vehicle 12 in place of a human driver, and makes the vehicle 12 run.
Actuators 110a and 110b abut vehicle operation pedals 123a and 123b, respectively.
One of the vehicle operation pedals 123a and 123b is an accelerator pedal, and the other is a brake pedal.

Drive robot 11 is controlled by control device 2 .
The control device 2 includes a learning section 20 , a drive robot control section 21 , a secondary operation content inference section 22 and a control method determination section 23 .
The control device 2 controls the actuators 110a and 110b of the drive robot 11 so that the vehicle 12 runs according to the prescribed command vehicle speed, and adjusts the opening degrees of the vehicle operation pedals 123a and 123b.
That is, the control device 2 adjusts the opening degrees of the vehicle operation pedals 123a and 123b of the vehicle 12, thereby controlling the traveling of the vehicle 12 so as to follow the mode, which is the prescribed traveling pattern.
Specifically, the control device 2 controls the traveling of the vehicle 12 so as to follow the commanded vehicle speed, which is the vehicle speed to be reached at each time, as time elapses from the start of traveling.

The vehicle state measuring unit 14 is a measuring unit that measures the state of the vehicle 12 or an externally installed measuring unit.
Here, as the state of the vehicle 12, operation values of the vehicle operation pedals 123a and 123b can be exemplified.
Here, as an externally installed measurement unit, a camera, an infrared sensor, or the like that measures the operation values of the vehicle operation pedals 123a and 123b can be exemplified.

The learning unit 20 includes a command vehicle speed generation unit 200, a reinforcement learning unit 201, a learning data molding unit 202, a learning data storage unit 203, a learning data generation unit 204, and an inference data molding unit 205. Learn a vehicle model for robot control.

The command vehicle speed generator 200 generates a command vehicle speed to be used as input data when inferring drive robot control.
The reinforcement learning unit 201 includes a reward calculation unit 2010, an operation content inference unit 2011, and a state action value inference unit 2012, and performs reinforcement learning for drive robot control.
Reinforcement learning unit 201 is implemented by a computing unit.
The reward calculation unit 2010 calculates a reward for reinforcement learning for the state after action.
The operation content inference unit 2011 has a first learning model, and outputs a control command, which is the operation content of the drive robot 11, in response to state input.
The state-action-value inference unit 2012 has a second learning model and calculates a Q value, which is a time-discounted expected return, for state and action inputs.

The learning data shaping unit 202 performs data shaping by converting the learning data used in the reinforcement learning unit 201 into an appropriate data format.
The learning data storage unit 203 stores learning data used for reinforcement learning in the reinforcement learning unit 201 .
The learning data generation unit 204 generates learning data used for reinforcement learning in the reinforcement learning unit 201 from the data in the learning data storage unit 203 .
The inference data forming unit 205 forms data by converting the initial observation state and command vehicle speed into a data format for inputting to the neural network of the first learning model.

The drive robot control unit 21 includes a drive state acquisition unit 211 and a vehicle operation control unit 212 , and gives control instructions to the drive robot 11 while observing the state of the drive robot 11 .
The drive robot control unit 21 is implemented by a computing unit.
The drive state acquisition unit 211 acquires the state of each component included in the test apparatus 1 , for example, the state of the pedal operation detection value of the drive robot 11 .
The vehicle operation control unit 212 generates a pedal operation command from the input data from the driving state acquisition unit 211 and converts the pedal operation command into a command for the actuators 110a and 110b of the drive robot 11, thereby controlling the operation of the vehicle 12. FIG.

The secondary operation content inference unit 22 outputs a control command, which is the operation content of the drive robot 11, based on control theory.

The control method determination unit 23 determines which of the control command from the operation content inference unit 2011 and the control command from the secondary operation content inference unit 22 is to be adopted, and sends the adopted control command to the drive robot control unit 21. Output.

In addition, in FIG. 2, each configuration connected by an arrow is connected by wire or wirelessly.

In this embodiment, the object is to make learning progress appropriately, to facilitate a good experience, and to streamline the initial progress of learning.
Specifically, in the initial stage of learning, a control method based on control theory such as PI (Proportional-Integral) control is used.
In a control method based on control theory, an accelerator pedal operation value is output when the vehicle speed is insufficient, and a brake pedal operation value is output when the vehicle speed is excessive.
In the control of the drive robot 11, since followability to the commanded vehicle speed is required as a major premise, traveling closer to the commanded vehicle speed is treated as a good experience.
However, the control method based on the control theory does not need to be designed and tuned for highly accurate tracking, and may be roughly adjusted to the extent that some reward for evaluating the tracking performance can be obtained.
The control method based on the control theory should just function as a priming for making the progress of learning more efficient in the initial stage of learning.
In order to apply this control method, policy-off type should be selected as deep reinforcement learning algorithm.
The policy-on or policy-off characteristics are determined by the expression of the algorithm derivation formula, but in policy-on type algorithms, only the experience from the current learning state can be used for learning, Learning can be disrupted.

In this embodiment, efficient learning progress of the first learning model and the second learning model in the reinforcement learning unit 201 is performed.
In the control of the drive robot 11, the first learning model and the second learning model acquire a control law for following a commanded vehicle speed pattern such as WLTC (Worldwide-harmonized Light vehicles Test Cycle) mode.
The operation of the control device 2 is roughly divided into an experience phase and a learning phase, as described below.

<Experience Phase>
The experience phase is a stage of accumulating data used for learning.
The solid line in FIG. 2 represents the flow of the experience phase.

Here, a series of experience cycles such as one WLTC mode run is called an episode.
The first learning model and the second learning model are represented by neural networks and are randomly initialized at the start of learning.
The inference data shaping unit 205 combines the initial observed state obtained from the drive state acquisition unit 211 of the drive robot control unit 21 and the chassis dynamometer 13 and vehicle state measurement unit 14 of the test apparatus 1, and the command obtained from the command vehicle speed generation unit 200. The vehicle speed is converted into a data format for input to the neural network of the first learning model.
The operation content inference unit 2011 uses the converted initial observation state and the command vehicle speed obtained from the command vehicle speed generation unit 200 to generate the following information for the drive robot 11 to operate the accelerator pedal and the brake pedal according to the first learning model. Infer the operation content of
Here, when adding explorability to the experience, random perturbations are added to the inferred operation contents according to the algorithm used in the learning phase.
On the other hand, the secondary operation content inference unit 22 also infers the operation content using the initial observed state and the commanded vehicle speed.
Here, in the secondary operation content inference unit 22, control based on control theory represented by PI control, for example, is implemented.
Thereby, the secondary operation content inference unit 22 outputs the accelerator pedal operation value when the observed vehicle speed is insufficient with respect to the command vehicle speed, and outputs the accelerator pedal operation value when the observed vehicle speed exceeds the command vehicle speed. outputs the brake pedal operation value.
Such a control method does not need to be designed and tuned for highly accurate tracking, but only needs to be roughly adjusted to the extent that some reward for evaluating tracking performance can be obtained.
The operation content output from the operation content inference unit 2011 and the operation content output from the secondary operation content inference unit 22 are sent to the control method determination unit 23 .
The control method determination unit 23 determines to adopt one of the operation content output from the operation content inference unit 2011 and the operation content output from the sub-operation content inference unit 22, and performs control based on the adopted operation content. A command is sent to the vehicle operation control unit 212 .
The sub-operation content inference unit 22 is used to obtain an experience that serves as a prime mover for streamlining the progress of learning in the early stages of learning. The control of the unit 22 is greatly reflected in the learning result.
Therefore, it is preferable that the vehicle operation control unit 212 performs control according to the operation content output from the operation content inference unit 2011 after the learning proceeds and the initial state is exited.
Therefore, the control method determination unit 23 adopts the operation content output from the secondary operation content inference unit 22 in the initial state from the start of learning to a predetermined episode (for example, 10 episodes), and after exiting the initial state, the operation It is preferable that the operation content output from the content inference unit 2011 is adopted.
A control command output from the control method determination unit 23 is sent to the vehicle operation control unit 212 of the drive robot control unit 21 .
Here, the control command from the control method determination unit 23 is converted into a command to the actuators 110a and 110b of the drive robot 11 and transmitted. Matching, upsampling and downsampling of the values of the control commands are performed.
A control command output from the vehicle operation control section 212 of the drive robot control section 21 is sent to the drive robot 11 .
The drive robot 11 operates the accelerator pedal and brake pedal of the vehicle 12 based on the received control commands.
The operation of the drive robot 11 is observed by the drive state acquisition section 211 of the drive robot control section 21 .
The state of the vehicle 12 is observed by a chassis dynamometer 13 and a vehicle state measuring section 14 .
The observed state of the vehicle 12 is sent to the inference data forming section 205 and the secondary operation content inference section 22, and used to generate the control command in the next episode.
Further, the observed state of the vehicle 12 is sent to the learning data molding section 202 together with the reward value obtained by the reward calculation section 2010 of the reinforcement learning section 201 and accumulated in the learning data storage section 203 .
In the control of the drive robot 11, since the followability to the commanded vehicle speed is required as a major premise, it is preferable to design rewards so that traveling closer to the commanded vehicle speed is treated as a good experience.
It should be noted that the learning data storage unit 203 may be set to discard the oldest data when it receives data exceeding the allowable storage amount.

Such a flow of processing is repeated for a predetermined period of time, for example, for one episode, and when a series of experiences is accumulated as learning data, the learning phase is entered.
When the operation content based on the control theory of the sub-operation content inferring unit 22 is used in the early stage of learning, the operation content based on the control theory of the sub-operation content inferring unit 22 is not used. Earn rewards, good experiences are possible.

<Learning phase>
The learning phase is a step of performing reinforcement learning of the first learning model and the second learning model using the learning data accumulated in the experience phase.
The dashed line in FIG. 2 represents the flow of the learning phase.

First, regarding the deep reinforcement learning algorithm, since it is premised on the use of the operation content based on the control theory of the sub-operation content inference unit 22 in the early stage of learning, a policy-off type algorithm is selected.
The policy-on or policy-off characteristic is determined by the expression of the algorithm derivation formula, but in the policy-on type algorithm, only the experience from the current learning state can be used for learning, and the other experiences are not learned. If used for , learning may collapse.
In this embodiment, DDPG (Deep Deterministic Policy Gradient), which is widely known as a policy-off algorithm, is adopted, and the flow of processing will be briefly described.

The learning data generation unit 204 selects data to be used for learning from the learning data storage unit 203 and outputs the data to the operation content inference unit 2011 and the state action value inference unit 2012 .

In the learning of the first learning model of the operation content inference unit 2011, the output when the learning data is input to the first learning model is input to the second learning model together with the learning data, and the output value of the state action value is input to the second learning model. Backpropagate the error using
As a result, a learning gradient for obtaining an output that increases the state-action value can be obtained, and the neural network can be trained by the gradient method-based optimization technique.

In the learning of the second learning model of the state-action-value inference unit 2012, a value called a TD (Temporal Difference) error, which is inferred from the current input data, and the sum of the value inferred from the input data and the current reward. If learning is performed in such a way that the difference becomes small, the value can be appropriately inferred as the learning progresses.

In the experience phase, when experiencing the first learning model that is randomly initialized, the experience with a small reward is the main experience, the learning of the second learning model does not progress, and the learning of the first learning model progresses accordingly. likely not.
If the learning does not progress properly, there is a risk that the learning will progress by mistake as if only the accelerator pedal or the brake pedal is used.
Therefore, as described above, using control commands based on control theory makes it easier to obtain good experience, and the learning of the first learning model and the second learning model progresses efficiently.
After the learning phase ends, the experience phase begins again.
Note that a target network may be used for stabilizing learning when learning the first learning model and the second learning model.
Also, the roles and configurations of the first learning model and the second learning model can be changed depending on the deep reinforcement learning algorithm used.

According to the present embodiment, progress of inappropriate learning can be suppressed, and progress of learning in the initial stage of learning can be made more efficient.

<Embodiment 2>
In the first embodiment, the switching of the operation content in the control method determination unit 23 is performed after a predetermined number of episodes, but the present invention is not limited to this.
In this embodiment, the control method determination unit 23 accumulates the operation contents input from the operation content inference unit 2011 and the secondary operation content inference unit 22, and determines which operation content is selected by correlation of the operation content sequence at the end of the episode. The only difference is that it is determined whether to select, and the other points are the same as those of the first embodiment.
As the correlation between the operation content series increases, learning progresses such that the behavior of the operation content inference unit 2011 consisting of a neural network and the behavior of the sub-operation content inference unit 22 based on control theory are similar. can be determined.
Here, the correlation does not need to be excessively large. For example, when the correlation coefficient reaches +0.3, the operation content of the operation content inference unit 2011 may be set so as to adopt the operation content.

According to this embodiment, in addition to the effect of the first embodiment, there is an effect that it becomes possible to switch the operation content at an appropriate timing.

<Embodiment 3>
In the present embodiment, a form in which pre-learning is performed using simulation by the test apparatus model 1A will be described.
This embodiment differs from the first embodiment only in that pre-learning is performed using a simulation using the test apparatus model 1A, and the rest is the same as the first embodiment.

FIG. 3 is a functional block diagram showing the test device 1 according to this embodiment and the control device 2A for the autopilot robot according to this embodiment.
The control device 2A has a test device model 1A.
The test equipment model 1A is composed of an internal model simulating the test equipment 1, and has a drive robot model 11A, a vehicle model 12A, and a chassis dynamometer model 13A.
The drive robot model 11A simulates the mechanical motion of the drive robot 11 of the test apparatus 1. FIG.
The vehicle model 12A simulates the vehicle 12 of the test device 1 and outputs the behavior of the vehicle 12 with respect to the behavior of the drive robot model 11A.
The chassis dynamometer model 13A simulates the operation and measurement of the chassis dynamometer of the test apparatus 1. FIG.

The experience phase and the learning phase are repeated in the simulation environment by the test equipment model 1A, and after the pre-learning of the first learning model and the second learning model has progressed sufficiently, the learning under the actual test equipment environment using the test equipment 1 is performed. , it is possible to suppress the running time in real time.
In addition, by sufficiently pre-learning the first learning model and the second learning model, it is possible to reduce the risk of malfunction of the test apparatus 1 due to a control command by an immature learning model.
The pre-learning by the test equipment model 1A is performed until sufficient driving performance is obtained by the control of the vehicle model 12A.
For example, when mode driving is assumed, pre-learning by the test device model 1A is performed until a sufficiently small vehicle speed error is obtained by the vehicle model 12A when driving in the target mode.

According to this embodiment, pre-learning is possible, and in addition to the effects of the first embodiment, it is possible to suppress the running time in real time, and it is possible to reduce the risk of malfunction of the test apparatus 1. Effective.

In the present embodiment, the vehicle is the object to be controlled, the vehicle speed is the state of the object to be controlled, and the command is the commanded vehicle speed. Although the control device for the autopilot robot that controls the vehicle to run according to the commanded vehicle speed has been described, the present invention is not limited to this.
That is, a control device that controls the controlled object so that the state of the controlled object matches the command, an operation content inference unit that outputs a control command for operating the controlled object based on a reinforcement learning algorithm; and a control command output by the secondary operation content inference unit at the beginning of learning, and a control command output by the operation content inference unit after the initial stage of learning. The present invention also includes a control device comprising a control method determination unit that determines whether to adopt a control method and outputs the adopted control command, and an operation control unit that controls the operation of the controlled object according to the adopted control command. is.

In addition, the present invention is not limited to the above-described embodiments, and includes various modifications in which components are added, deleted, or converted to the above-described configuration.

Reference Signs List 1 test device 11 drive robot 110a, 110b actuator 12 vehicle 121 drive wheel 122 driver's seat 123a, 123b vehicle operation pedal 13 chassis dynamometer 14 vehicle state measurement unit 1A test device model 11A drive robot model 12A vehicle model 13A chassis dynamometer model 2 , 2A control device 20 learning unit 200 command vehicle speed generation unit 201 reinforcement learning unit 2010 reward calculation unit 2011 operation content inference unit 2012 state action value inference unit 202 learning data molding unit 203 learning data storage unit 204 learning data generation unit 205 inference data molding Section 21 Drive Robot Control Section 211 Drive State Acquisition Section 212 Vehicle Operation Control Section 22 Sub-Operation Content Inference Section 23 Control Method Determination Section

Claims (5)

A control device that controls the controlled object so that the state of the controlled object matches a command,
an operation content inference unit that outputs a control command for operating the controlled object based on a reinforcement learning algorithm;
a sub-operation content inference unit that outputs a control command based on control theory;
In the early stage of learning, which is a predetermined period from the start of learning under the environment of the actual test apparatus, the control command output by the secondary operation content inference unit is adopted, and after the initial learning period ends, the control command output by the operation content inference unit A control method determination unit that determines to adopt and outputs the adopted control command;
and an operation control unit that controls the operation of the controlled object according to the adopted control command. A control device for an autopilot robot that controls an autopilot robot that is mounted on a vehicle and causes the vehicle to travel in accordance with a prescribed command vehicle speed,
an operation content inference unit that outputs a control command for operating the vehicle based on a reinforcement learning algorithm;
a sub-operation content inference unit that outputs a control command based on control theory;
In the early stage of learning, which is a predetermined period from the start of learning under the environment of the actual test apparatus, the control command output by the secondary operation content inference unit is adopted, and after the initial learning period ends, the control command output by the operation content inference unit A control method determination unit that determines to adopt and outputs the adopted control command;
A control device for an autopilot robot, comprising: a vehicle operation control unit that controls operation of the vehicle according to the adopted control command. It is determined based on the correlation of the operation content sequence whether the learning initial stage or the learning initial stage has passed,
3. The method according to claim 2, wherein when the correlation between the operation content sequences increases, it is determined that the learning has progressed such that the behavior of the operation content inference unit and the behavior of the secondary operation content inference unit are similar to each other. autopilot robot controller. 4. The control device for an autopilot robot according to claim 2, wherein pre-learning using a simulation using a test device model is performed before learning under an actual test device environment. A control method for an autopilot robot, comprising: controlling an autopilot robot mounted on a vehicle to drive the vehicle so that the vehicle travels according to a prescribed command vehicle speed,
outputting a control command to operate the vehicle based on a reinforcement learning algorithm;
outputting a control command based on control theory;
Judgment to adopt the control command based on the control theory in the initial stage of learning, which is a predetermined period from the start of learning under the environment of the actual test equipment , and to adopt the control command based on the reinforcement learning algorithm after the initial stage of learning is over. and outputting the adopted control command,
A method of controlling an autopilot robot, comprising: controlling the operation of the vehicle according to the employed control commands.

Images