JP2022135788A - vehicle controller - Google Patents

Abstract

To suppress input of an excessive load into a drive line while securing the running property by individually controlling the torque of each driving wheel.SOLUTION: A vehicle has a mechanism that individually controls the torque of each driving wheel. A controller 100 as a vehicle controller has a storage device 110 and a processing circuit 120. The storage device 110 stores a learned model which determines and outputs an action of operating a control amount of the mechanism when a state variable including road surface information data extracted from an image of a road surface of the front of the vehicle taken by a camera 70 is input. The processing circuit 120 operates the control amount on the basis of the action output by inputting the state variable into the learned model, and controls the mechanism. The learned model is a model learned by reinforcement learning in which a more reward is given as a distance that the vehicle travels without causing slipping of any driving wheels is longer.SELECTED DRAWING: Figure 1

Description

The present invention relates to a vehicle control system.

When the vehicle is traveling on a rough road such as a rocky place, the drive wheels may slip. Then, when the grip of the slipping drive wheels recovers, the rotation is suddenly braked. As a result, a large load is applied to the driveline that transmits the driving force to the drive wheels.

The control device disclosed in Japanese Patent Laid-Open No. 2002-200002 determines whether or not it is estimated that an excessive load will be applied to the driveline when the slip converges when the drive wheels are slipping. Then, when the control device determines that the state is such that an excessive load is expected to act, it changes the gear stage to the high gear ratio side.

By changing the gear stage to the high gear ratio side in this way, the control device lowers the rotation speed of the drive wheels and reduces the load acting on the drive line when the slipping drive wheels grip. ing.

Japanese Patent Application Laid-Open No. 2009-210000

However, if the gear stage is changed to the high gear ratio side as described above, the rotation speed of not only the slipping drive wheels but also all of the drive wheels will decrease. As a result, the speed of the vehicle decreases, and the running performance deteriorates.

Means for solving the above problems and their effects will be described below.
A vehicle control device for solving the above problems controls a vehicle having a mechanism capable of individually controlling the torque of each driving wheel. This vehicle control device stores a learned model that determines and outputs an action for manipulating the control amount of the mechanism when state variables including road surface information data extracted from an image of the road surface in front of the vehicle are input. and a processing circuit for controlling the mechanism by manipulating the control amount based on the behavior output by inputting the state variables to the learned model. The learned model is a model learned by reinforcement learning that gives a greater reward as the vehicle travels a longer distance without slipping any driving wheels.

In the above configuration, actions are determined using state variables including road surface information data extracted from an image of the road surface in front of the vehicle. Therefore, it is possible to reflect the information of the road surface on which the drive wheels will come into contact with the decision of action. As a result, it is possible to control the torque of each driving wheel so as to prevent the occurrence of a slip in consideration of the condition of the road surface on which each driving wheel is grounded. In addition, the learned model is learned by reinforcement learning that gives a greater reward as the distance traveled without any driving wheels slipping increases. Therefore, according to this control device, instead of uniformly suppressing the torque of all drive wheels, the torque of each drive wheel is individually controlled so that the vehicle can continue to move forward while avoiding the occurrence of slip. can be done.

That is, according to the above configuration, it is possible to suppress the input of an excessive load to the driveline while ensuring the running performance.

1 is a schematic diagram showing the relationship between a vehicle control device according to an embodiment and a drive train of a vehicle controlled by the vehicle control device; FIG. The schematic diagram which shows the structure of the multilayer neural network which calculates an action-value function. 4 is a flowchart showing a series of processing flows in learning processing; 4 is a flowchart showing a series of processes in rough road travel processing;

An embodiment of a vehicle control device will be described below with reference to FIGS. 1 to 4. FIG.
FIG. 1 shows a control device 100, which is a vehicle control device, and a drive train of a vehicle in which the control device 100 is mounted.

<Configuration of drive train>
As shown in FIG. 1, this vehicle has four wheels: a right front wheel 50RF, a left front wheel 50LF, a right rear wheel 50RR, and a left rear wheel 50LR.

This vehicle is equipped with an engine 10 as a power source. The engine 10 is housed in an engine compartment at the front of the vehicle. A transmission 20 for changing the speed of rotation of the output shaft of the engine 10 is also housed in the engine compartment. A front differential 21 is housed in the case of the transmission 20 . The front differential 21 transmits the rotation speed-changed by the transmission 20 to the right front drive shaft 51RF and the left front drive shaft 51LF. The right front drive shaft 51RF is connected to the right front wheel 50RF. The right front wheel 50RF is driven by the rotation of the right front drive shaft 51RF. Also, the left front drive shaft 51LF is connected to the left front wheel 50LF. The rotation of the left front drive shaft 51LF drives the left front wheel 50LF. A brake 60RF is provided on the right front wheel 50RF. A left front wheel 50LF is provided with a brake 60LF.

A transfer 30 is provided between the front differential 21 and the right front wheel 50RF in the central portion in the width direction of the vehicle. The transfer 30 is a device for transmitting the rotation transmitted from the front differential 21 to the right front drive shaft 51RF to the rear wheels via the propeller shaft 52 . The transfer 30 has a front disconnect mechanism 31 . The front disconnect mechanism 31 is a mechanism for interrupting transmission of rotation between the front differential 21 and the right front drive shaft 51 RF and the propeller shaft 52 .

Propeller shaft 52 is connected to rear differential 40 . The rear differential 40 is arranged between the right rear wheel 50RR and the left rear wheel 50LR. Rear differential 40 transmits the rotation transmitted through propeller shaft 52 to right rear drive shaft 51RR and left rear drive shaft 51LR. The right rear drive shaft 51RR is connected to the right rear wheel 50RR. The right rear wheel 50RR is driven by the rotation of the right rear drive shaft 51RR. Also, the left rear drive shaft 51LR is connected to the left rear wheel 50LR. The rotation of the left rear drive shaft 51LR drives the left rear wheel 50LR. A brake 60RR is provided on the right rear wheel 50RR. A brake 60LR is provided on the left rear wheel 50LR.

The rear differential 40 is equipped with a rear disconnect mechanism 41, an electronically controlled coupling 42R, and an electronically controlled coupling 42L.
The electronically controlled coupling 42R and the electronically controlled coupling 42L are mechanisms for changing the ratio of transmitting the driving force to the rear drive shaft by operating the built-in electromagnetic clutch. That is, the electronically controlled coupling 42R is a mechanism that can change the ratio of the driving force transmitted to the right rear wheel 50RR by operating the built-in electromagnetic clutch. On the other hand, the electronically controlled coupling 42L is a mechanism that can change the ratio of the driving force transmitted to the left rear wheel 50LR by controlling the built-in electromagnetic clutch.

The rear disconnect mechanism 41 is a mechanism for interrupting transmission of rotation between the propeller shaft 52 and the electronically controlled couplings 42R and 42L.
Thus, the drive train of this vehicle is configured to be able to drive four wheels. That is, in this vehicle, any of the right front wheel 50RF, the left front wheel 50LF, the right rear wheel 50RR, and the left rear wheel 50LR can be driving wheels.

<Configuration of control device 100>
The control device 100 includes a storage device 110 storing programs, and a processing circuit 120 executing the programs stored in the storage device 110 to perform various processes. The control device 100 controls each part of the engine 10 and controls the output of the engine 10 . The control device 100 also controls the transmission 20, the front disconnect mechanism 31, the rear disconnect mechanism 41, the electronically controlled coupling 42R, and the electronically controlled coupling 42L. Further, the control device 100 controls the brakes 60RF, 60LF, 60RR, and 60LR provided on the four wheels. In this vehicle, the braking forces of the brakes 60RF, 60LF, 60RR, and 60LR can be individually controlled.

Control device 100 controls the output of engine 10 and controls the driving force of the vehicle by controlling transmission 20 . Further, in this vehicle, torque distribution to each wheel can be controlled by controlling the front disconnect mechanism 31, the rear disconnect mechanism 41, the electronically controlled coupling 42R, and the electronically controlled coupling 42L.

For example, control device 100 can distribute torque only to right front wheel 50RF and left front wheel 50LF by interrupting transmission of rotation by front disconnect mechanism 31 and rear disconnect mechanism 41 . That is, this vehicle can also run by front-wheel drive without driving the rear wheels. Further, for example, the control device 100 controls the torque distributed to the rear wheels by the electronically controlled couplings 42R and 42L without interrupting the transmission of rotation by the disconnect mechanism, so that the right rear wheel 50RR is controlled. And the torque distribution to the left rear wheel 50LR can be changed.

Furthermore, in this vehicle, the torque of each driving wheel can be individually controlled by controlling the braking force of each driving wheel by the brake 60RF, the brake 60LF, the brake 60RR, and the brake 60LR.

In short, this vehicle controlled by the control device 100 has a mechanism capable of individually controlling the torque of each drive wheel. Specifically, the front disconnect mechanism 31, the rear disconnect mechanism 41, the electronically controlled coupling 42R, the electronically controlled coupling 42L, the brake 60RF, the brake 60LF, the brake 60RR, and the brake 60LR correspond to this mechanism.

The control device 100 is connected to various sensors and devices for collecting information for grasping the state of operation by the driver, the state of the vehicle, and the state of the road surface.
For example, an accelerator position sensor 71 is connected to the control device 100 . The accelerator position sensor 71 detects the accelerator opening, which is the amount of accelerator operation by the driver. A crank position sensor 72 is connected to the control device 100 . Crank position sensor 72 detects a crank angle signal corresponding to the rotation angle of the crankshaft, which is the output shaft of engine 10 . The control device 100 calculates the engine rotation speed, which is the rotation speed of the crankshaft, based on the crank angle signal. A brake sensor 73 is connected to the control device 100 . The brake sensor 73 detects the amount of operation of the brake pedal. A vehicle speed sensor 74 is connected to the control device 100 . Vehicle speed sensor 74 detects the rotation speed of the output shaft of transmission 20 . Control device 100 calculates the speed of the vehicle based on the rotation speed of the output shaft detected by vehicle speed sensor 74 . A steering sensor 75 is connected to the control device 100 . A steering sensor 75 detects the steering angle of the steering wheel. A speed sensor that detects the rotational speed of each drive wheel is connected to the control device 100 . Specifically, the control device 100 is connected to a right front speed sensor 76RF that detects the rotational speed of the right front wheel 50RF and a left front speed sensor 76LF that detects the rotational speed of the left front wheel 50LF. Also connected to the control device 100 are a right rear speed sensor 76RR for detecting the rotational speed of the right rear wheel 50RR and a left rear speed sensor 76LR for detecting the rotational speed of the left rear wheel 50LR. A yaw rate sensor 80 and a linear G sensor 81 are also connected to the control device 100 . A yaw rate sensor 80 detects a yaw rate, which is the angular velocity of the vehicle when it turns. A linear G sensor 81 detects the acceleration of the vehicle.

Further, the control device 100 is connected with a camera 70 that captures the road surface in front of the vehicle. A raindrop sensor 78 is also connected to the controller 100 . A raindrop sensor 78 detects the amount of raindrops adhering to the front window. In addition, a GPS device 77 is installed in this vehicle. The control device 100 acquires vehicle position information from the GPS device 77 . The control device 100 can also calculate the speed of the vehicle based on the acquired position information.

This vehicle is also provided with a driving mode changeover switch 79 . A running mode changeover switch 79 is also connected to the control device 100 . Control device 100 switches the running mode based on a signal from running mode changeover switch 79 . Specifically, in this vehicle, a rough road driving mode with high rough road running performance can be selected by the driving mode selector switch 79 . When the rough road running mode is selected, the control device 100 controls the front disconnect mechanism 31 and the rear disconnect mechanism so that the torque generated by the engine 10 is transmitted to the rear differential 40 via the propeller shaft 52. 41 is fixed. Then, the control device 100 fixes the transmission 20 to a high gear ratio state, and controls the torque of each driving wheel individually to suppress the occurrence of slipping of the driving wheels while causing the vehicle to run.

<Control of Torque of Each Driving Wheel in Rough Road Driving Mode>
The storage device 110 of the control device 100 stores model data used for torque control of each driving wheel when the rough road traveling mode is selected. Note that this model is a learned model that has been learned through reinforcement learning.

The control device 100 controls the output of the engine 10 based on the accelerator opening when the rough road traveling mode is selected. Then, the control device 100 generates information variables including road surface information data extracted from the image of the road surface in front of the vehicle captured by the camera 70, the rotation speed of each drive wheel detected by each speed sensor, the steering angle of the steering wheel, the vehicle speed, and the like. Input to the trained model to determine the behavior that manipulates the control amount of each mechanism of the drive train. By manipulating the control amount of each mechanism according to the determined action, the torque of each driving wheel is controlled so that none of the driving wheels will slip. In other words, the learned model is a model that, when state variables are input, outputs actions that manipulate the control amounts of each mechanism of the drive train.

Specifically, the learned model selects and outputs an action from options of increasing, maintaining, and reducing the torque of each drive wheel. The processing circuit 120 of the control device 100 controls each mechanism of the drive train so as to manipulate the torque of each drive wheel according to the behavior output by the learned model. For example, when increasing the torque of the right rear wheel 50RR, the processing circuit 120 increases the torque distributed to the right rear wheel 50RR by the electronically controlled coupling 42R. Also, for example, the processing circuit 120 increases the braking force of the brake 60LF when reducing the torque of the left front wheel 50LF. In this way, in the rough road running mode, the processing circuit 120 of the control device 100 controls the electronically controlled coupling 42R, the electronically controlled coupling 42L, the brake 60RF, the brake 60LF, the brake 60RR, and the brake 60LR to control the driving wheels. Separate torque control.

<Learning the trained model>
Next, learning of a learned model used for rough road running processing, which is torque control in the rough road running mode, will be described. The trained model stored in the storage device 110 has been trained by reinforcement learning.

A learning system that performs learning causes the control device 100 to determine an action based on state variables and to execute the determined action. Then, if the reward is evaluated according to the state after execution of the action, the action value of the selected action becomes clear. Therefore, the control device 100 of the learning system performs learning by repeatedly acquiring state variables, determining actions according to the acquired state variables, and evaluating rewards obtained by the determined actions.

An agent in reinforcement learning corresponds to a function that selects action a according to a predetermined policy. The environment in reinforcement learning is a function that determines the next state s' based on the agent's selected action a and the current state s, and determines the immediate reward r based on the action a, state s, and state s'. corresponds to

In learning according to this embodiment, the control device 100 of the learning system selects an action a according to a predetermined policy, and repeats the process of updating the state s. Q-learning is employed to calculate Q(s, a).

Here, the action value function Q(s, a) is updated by the following formula (1).

そして、行動価値関数Ｑ（ｓ，ａ）が適正に収束した場合には、その行動価値関数Ｑ（ｓ，ａ）を最大化する行動ａが最適な行動であるとみなされ、この行動ａによって決定される操作量がドライブトレインの機構の最適な操作量であるとみなされる。

Then, when the action-value function Q(s, a) converges appropriately, the action a that maximizes the action-value function Q(s, a) is regarded as the optimal action, and by this action a The determined manipulated variable is considered to be the optimum manipulated variable of the drive train mechanism.

In the above equation (1), the action-value function Q(s, a) is the expected value of future profits obtained when action a is taken in state s. The reward is r. The suffix t in the state s, the action a, and the reward r is a trial number indicating one step in the trial process repeated in chronological order. When the state s changes after action determination, the trial number t is incremented by one. In addition, below, a subscript is described following "_". Therefore, the reward r_t+1 in equation (1) is the reward r obtained when action a_t is selected in state s_t and state s becomes s_t+1. α is the learning rate and γ is the discount rate. Also, a′ is action a that maximizes action value function Q(s_t+1, a_t+1) among actions a_t+1 that can be taken in state s_t+1. And max_(a')Q(s_t+1, a') is the action value function Q maximized by the action a' being selected.

In the reinforcement learning of this embodiment, controlling the torque of each drive wheel corresponds to determining the action a, and information indicating possible actions is recorded in advance in the storage device 110 of the control device 100 .

As described above, the action a can select one of the three actions of increasing, maintaining, and decreasing the torque for each drive wheel. Of course, this is just an example, and the content of action a need not be limited to such content, and action a may have more or fewer options.

In the reinforcement learning of this embodiment, the reward r is set to increase as the distance traveled by the vehicle without slipping any driving wheels increases. Specifically, the state variables include the rotational speed of each drive wheel detected by each speed sensor. If the rotation speed detected by the four speed sensors includes a rotation speed that deviates from the rotation speed detected by the other three speed sensors and is higher than the rotation speed detected by the other three speed sensors, a speed sensor that detects that rotation speed is provided. This means that the drive wheels that are engaged are slipping. Controller 100 thus determines whether slip has occurred on any of the drive wheels, and if so, terminates the learning trial episode at that point. Then, a negative reward r, such as "-10", is given as a reward r for occurrence of a slip. On the other hand, if none of the drive wheels slip, the trial continues. A positive reward r is increased according to the distance traveled by the vehicle while the trial is continued, and the reward r is determined when the trial ends. By doing so, the longer the distance traveled by the vehicle without causing any of the drive wheels to slip, the greater the reward r. Note that the distance traveled by the vehicle can be calculated based on the position information acquired from the GPS device 77, for example.

The next state s' when the action a is adopted in the current state s can be specified by changing the operation amount as the action a, running the vehicle, and acquiring the state variables.
Therefore, in this embodiment, the rotational speed of each drive wheel and the vehicle speed calculated from the positional information obtained from the GPS device 77 are input as state variables. Specifically, the state variables include the rotational speed of the right front wheel 50RF detected by the right front speed sensor 76RF and the rotational speed of the left front wheel 50LF detected by the left front speed sensor 76LF. The state variables also include the rotational speed of the right rear wheel 50RR detected by the right rear speed sensor 76RR and the rotational speed of the left rear wheel 50LR detected by the left rear speed sensor 76LR. In addition, since the vehicle speed is included in the state variables, the traveled distance can be calculated based on the vehicle speed.

State variables also include the degree of accelerator opening and the amount of operation of the brake pedal. The state variables also include the following information in order to determine the action a by referring to the information indicating the road surface condition.

The state variables include road surface information data extracted from the image of the road surface in front of the vehicle captured by the camera 70 . For example, the road surface information data is data obtained by extracting the information of the road surface where each driving wheel is touching by extracting the image corresponding to the location where each driving wheel is touching the ground from the image taken several meters before. be. Specifically, this data may be a vector obtained by extracting a feature amount from the clipped image using a convolutional neural network. Also, this data may be data obtained by analyzing the clipped image and calculating the size and height of unevenness existing on the grounded road surface.

The state variables also include the steering angle detected by the steering sensor 75 to determine the traveling direction of the vehicle and the orientation of the front wheels. The state variables also include the acceleration detected by the linear G sensor 81 in order to grasp the inclination of the vehicle. In addition, when the road surface is wet with rain, slipping is likely to occur, so the detection value of the raindrop sensor 78 is also included in the state variables. It can be seen that the greater the amount of raindrops detected by the raindrop sensor 78, the more likely the vehicle is to slip.

Information indicating variables and functions referred to in the process of learning is stored in the storage device 110 of the learning system. The control device 100 of the learning system observes the state variables, determines the action a according to the observed state variables, and evaluates the reward r obtained by the action a to obtain the action value function Q(s, a ) is adopted. In the control device 100 of the learning system, time-series values of the state variable, the action a, and the reward r are sequentially recorded in the storage device 110 in the process of learning.

This embodiment employs DQN (Deep Q-Network), which is a technique for approximately calculating the action-value function Q(s, a). In DQN, a multilayer neural network is used to estimate the action-value function Q(s,a). This embodiment employs a multi-layer neural network that takes state s as input and outputs the value of action value function Q(s, a) corresponding to the number of selectable actions a.

FIG. 2 is a diagram schematically showing a multilayer neural network that outputs the action-value function Q. As shown in FIG. In FIG. 2, the multi-layer neural network has M states s as state variables as inputs and N values of action-value functions Q as outputs. In FIG. 2, M states s at trial number t are indicated as s_1t to s_Mt.

Note that N is the number of actions a that can be selected, and the output of the multilayer neural network is the value of the action value function Q when a specific action a is selected in the input state s. In FIG. 2, action value functions Q for actions a_1t to a_Nt that can be selected at trial number t are indicated as Q(s_t, a_1t) to Q(s_t, a_Nt). "s_t" written in this action value function Q is a character representing the state s input at the trial number t, that is, the states s_1t to s_Mt. In the example of this embodiment, the number of actions a that can be selected is three for each of the four drive wheels, so there are 12 in total. Therefore N=12.

The multi-layer neural network shown in FIG. 2 performs multiplication of weight w and addition of bias b on the input of the immediately preceding layer at each node of each layer, and performs operations to obtain outputs via activation functions as necessary. It is a fully-connected forward-propagating neural network. In FIG. 2, notation of transmission lines connecting nodes in adjacent layers is omitted.

The structure of a multi-layer neural network is specified by information such as weights w and biases b in each layer, activation functions and layer order. Therefore, in the learning system, parameters for specifying this multilayer neural network are recorded in the storage device 110 . During learning, the weight w and the bias b, which are variable values in the multi-layer neural network, are updated. Below, the parameter of the multi-layer neural network that can change in the course of learning is denoted as θ. By using this θ, the action value functions Q(s_t, a_1t) to Q(s_t, a_Nt) can also be expressed as Q(s_t, a_1t: θ_t) to Q(s_t, a_Nt: θ_t).

Next, the procedure of the learning process will be described with reference to the flowchart shown in FIG. When the learning process is started as shown in FIG. 3, the control device 100 of the learning system acquires state variables in the process of step S100. Next, the control device 100 calculates the action value in the process of step S110. That is, the control device 100 refers to the learning information stored in the storage device 110 to acquire θ. Then, the latest state variables are input to the multilayer neural network indicated by the learning information stored in the storage device 110, and N action value functions Q(s_t, a_1t: θ_t) to Q(s_t, a_Nt: θ_t) are calculated. do. In an initial state immediately after learning is started, θ set as an initial value is stored in storage device 110 as learning information.

The trial number t is 0 at the first execution. If the learning process has not progressed sufficiently, θ indicated by the learning information shown in storage device 110 is not sufficiently optimized. Therefore, the value of the action-value function Q may become an inappropriate value. However, by repeating trials, the action-value function Q is gradually optimized. Also, in the repetition of trials, the state s, the action a, and the reward r are stored in the storage device 110 in association with each trial number t. This allows you to refer to it at any time.

Next, the control device 100 of the learning system selects and executes action a in the process of step S120. In this embodiment, the action a that maximizes the action-value function Q(s, a) is regarded as the optimum action a. Therefore, the control device 100 specifies the maximum value among the N values of the action value functions Q(s_t, a_1t: θ_t) to Q(s_t, a_Nt: θ_t) calculated in step S110.

Then, the control device 100 of the learning system selects the action a that gives the maximum value. For example, if Q(s_t, a_3t: θ_t) is the maximum value among N action value functions Q(s_t, a_1t: θ_t) to Q(s_t, a_Nt: θ_t), action a_3t is selected.

When an action a is selected, the learning system controller 100 controls the manipulated variable of the powertrain mechanism to manipulate the torque according to the action a. For example, when the action a that reduces the torque of the right front wheel 50RF is selected, the processing circuit 120 of the control device 100 increases the braking force of the brake 60RF.

Next, the control device 100 of the learning system acquires state variables in the process of step S130. That is, the control device 100 acquires the state variables by performing the same processing as the processing in step S100. Note that, for example, if the current trial number is t and the selected action a is action a_t, the state s acquired in the process of step S130 is state s_t+1.

Next, the learning system control device 100 evaluates the reward r in the process of step S140. Specifically, control device 100 determines whether a slip has occurred in any of the driving wheels based on the rotation speed detected by each rotation speed sensor as described above. Then, when it is determined that a slip has occurred, a negative reward r is obtained. And finish this learning episode. On the other hand, when it is determined that no slip has occurred, the distance traveled in this trial is calculated based on the vehicle speed calculated based on the position information acquired from the GPS device 77 as described above, and the distance Get a responsive positive reward r. Note that when the current trial number is t, the reward r obtained in step S140 is the reward r_t+1.

Next, in the process of step S150, the control device 100 of the learning system determines whether or not the learning episode ends here. Then, when it is determined that the episode has ended in the process of step S150 (step S150: YES), the control device 100 advances the process to step S200. On the other hand, if it is determined in the process of step S150 that the episode has not ended (step S150: NO), the control device 100 returns the process to step S100 and continues the trial.

In this way, the control device 100 repeats the processing of steps S100 to S150 until it is determined that one of the driving wheels has slipped, and obtains the reward r. Then, when it is determined that one of the driving wheels has slipped, control device 100 obtains negative reward r, terminates the episode at that point, and advances the process to step S200.

In this embodiment, the action-value function Q shown in equation (1) is updated. In order to update the action-value function Q appropriately, a multi-layer neural We have to optimize the network. In order for the multilayer neural network shown in FIG. 2 to properly output the action-value function Q, teacher data that serves as an output target is required. That is, the multi-layer neural network can be optimized by improving θ to minimize the error between the output of the multi-layer neural network and the target.

However, knowledge of the action-value function Q is not sufficient at the stage where learning is not completed. Therefore, it is difficult to identify the target. Therefore, in this embodiment, improvement of θ, which indicates a multi-layer neural network, is performed by the second term of Equation (1), the objective function for minimizing the so-called TD error (Temporal Difference). That is, ((r_t+1)+γmax_(a′)Q(s_t+1, a′:θ_t)) is targeted. Then, θ is learned so as to minimize the error between the target and Q(s_t, a_t: θ_t). However, the target ((r_t+1)+γmax_(a′)Q(s_t+1, a′:θ_t)) includes θ to be learned. Therefore, in this embodiment the target is fixed for a predetermined number of episodes.

In order to perform learning on such a premise, the control device 100 of the learning system calculates an objective function in the process of step S200. That is, the control device 100 calculates an objective function for evaluating the TD error in each episode. The objective function is, for example, a function proportional to the expected value of the square of the TD error or the sum of the squares of the TD error. Note that the TD error is calculated with the target fixed. Therefore, the fixed target is expressed as ((r_t+1)+γmax_(a′)Q(s_t+1, a′:θ_hold)). Then the TD error is ((r_t+1)+γmax_(a′)Q(s_t+1,a′:θ_hold)−Q(s_t,a_t:θ_t)). In this TD error formula, the reward r_t+1 is the reward obtained in step S140 by action a_t.

Also, max_(a′)Q(s_t+1, a′: θ_hold) is obtained when the state s_t+1 obtained in step S130 by the action a_t is input to a multi-layer neural network specified by a fixed θ_hold. The maximum value in the output.

Then, Q(s_t, a_t: θ_t) is the output obtained when the state s_t before action a_t is selected is input to a multi-layer neural network specified by θ_t at the stage of trial number t. , is the value of the output corresponding to the action a_t.

After the objective function is calculated, in the processing of the next step S210, the control device 100 of the learning system determines whether or not the learning is completed. Here, a threshold is set in advance for determining whether the TD error is sufficiently small. Then, when the objective function is equal to or less than the threshold, the control device 100 determines that learning has been completed.

If it is determined that learning has not been completed in the process of step S210 (step S210: NO), control device 100 of the learning system advances the process to step S220. Then, the control device 100 updates the action value in the process of step S220. That is, the control device 100 specifies the change in θ for reducing the objective function based on the partial differentiation of the TD error with respect to θ, and changes θ. Here, θ can be changed by various methods. For example, gradient descent can be employed. Also, adjustment by a learning rate or the like may be appropriately performed. According to such processing, θ can be changed so that the action-value function Q approaches the target.

However, since the target is fixed as described above, the control device 100 of the learning system further determines whether or not to update the target. Specifically, the control device 100 of the learning system determines whether or not the number of episodes is equal to or greater than the predetermined number in the processing of the next step S230. Then, when it is determined in step S230 that the number of episodes is equal to or greater than the predetermined number of times (step S230: YES), the control device 100 advances the process to step S240.

In the process of step S240, the learning system control device 100 updates the target. That is, the control device 100 updates θ referred to when calculating the target to the latest θ. Thereafter, control device 100 returns the process to step S100. Then, the processing after step S100 is repeated.

On the other hand, when it is determined in the process of step S230 that the number of episodes is less than the predetermined number (step S230: NO), the control device 100 skips the process of step S240 and returns the process to step S100. Then, the processing after step S100 is repeated.

Further, when it is determined that the learning is completed in the process of step S210 (step S210: YES), the control device 100 of the learning system advances the process to step S250. Then, in the process of step S250, the control device 100 updates the learning information stored in the storage device 110. FIG. That is, the control device 100 stores θ obtained by learning in the storage device 110 as a learned model. When the learned model including this θ is stored in the storage device 110 of the control device 100 as a vehicle control device, the processing circuit 120 of the control device 100 mounted on the vehicle uses the learned model for each drive. You will be able to control the torque of the wheels.

<Rough road travel processing>
FIG. 4 is a flowchart showing rough road travel processing that is repeatedly executed when the rough road travel mode is selected in the vehicle control device 100 .

When the rough road traveling process is started, the processing circuit 120 of the control device 100, which is the vehicle control device, executes the process of step S300. In the process of step S300, the processing circuit 120 acquires the state variables in the same manner as in the process of step S100 described with reference to FIG.

Next, the processing circuit 120 of the control device 100 determines the control amount of the mechanism of the drive train in the processing of step S310. Specifically, the processing circuit 120 inputs the state variables obtained in step S300 to the trained model stored in the storage device 110 . Then, the processing circuit 120 selects the action a that gives the maximum value in the action value function Q(s, a) that is the output of the trained model. Processing circuitry 120 then determines the amount of control of the drive train mechanism based on the selected action a.

Processing circuitry 120 then advances processing to step S320. Then, in the process of step S320, the processing circuit 120 controls the driving force and braking force of each drive wheel to control the torque of each drive wheel. That is, the processing circuit 120 controls the drive train mechanism based on the control amount determined in step S310. Specifically, the processing circuit 120 controls the electronically controlled coupling 42R and the electronically controlled coupling 42L based on the determined control amount. This controls the driving force distributed to the rear wheels. The processing circuit 120 also controls the brake 60RF, the brake 60LF, the brake 60RR, and the brake 60LR based on the determined control amount. Thereby, the control of the braking force of each driving wheel is performed together. Thereby, the processing circuit 120 controls the driving force and braking force of each drive wheel, and individually controls the torque of each drive wheel. After executing the process of step S320 in this way, the processing circuit 120 once terminates this routine.

<Action>
According to the above configuration, the action a that maximizes the action-value function Q can be selected to control the torque of each drive wheel. The action-value function Q is optimized as a result of repeating many trials through the learning process described above. The learned model is learned by reinforcement learning that gives a larger reward r as the distance traveled without any driving wheels slipping increases. Therefore, according to the control device 100, instead of uniformly suppressing the torque of all drive wheels, the torque of each drive wheel is individually controlled so that the vehicle can continue to move forward while avoiding the occurrence of slip. be able to.

<effect>
(1) In the above configuration, the action a is determined using state variables including road surface information data extracted from the image of the road surface in front of the vehicle. Therefore, it is possible to reflect the information of the road surface on which the drive wheels will touch the ground in the determination of the action a. As a result, it is possible to control the torque of each driving wheel so as to prevent the occurrence of a slip in consideration of the condition of the road surface on which each driving wheel is grounded.

(2) The learned model is learned by reinforcement learning that gives a larger reward r as the distance traveled without any driving wheels slipping increases. Therefore, according to the control device 100, instead of uniformly suppressing the torque of all drive wheels, the torque of each drive wheel is individually controlled so that the vehicle can continue to move forward while avoiding the occurrence of slip. be able to.

(3) According to the above configuration, by realizing the above (1) and (2) at the same time, it is possible to suppress the input of an excessive load to the driveline while ensuring the running performance.
This embodiment can be implemented with the following modifications. This embodiment and the following modified examples can be implemented in combination with each other within a technically consistent range.

・In the above embodiment, an example in which the accelerator opening is included in the state variables is shown, but for example, the output of the engine 10 is automatically controlled so as to maintain a preset constant speed in rough road driving mode. , there is no need to include the accelerator opening in the state variable. That is, the same control using the reinforcement learned model as in the above-described embodiment can be applied to the configuration for executing the rough road running mode in which such speed control is automatically performed.

In the above embodiment, the control device 100 is illustrated as one device, but there are an engine control device that controls the engine 10, a transmission control device that controls the transmission 20, and distribution of driving force to each drive wheel. may be divided into a 4WD control device that controls the , a brake control device that controls each brake, and the like. In this case, the 4WD control device and the brake control device control the driving force distribution and braking force based on the operation amount calculated using the learned model, and cooperate to control the torque of each drive wheel. .

・Since weather information is information that can be used to estimate road conditions, even if a communication device is provided, the weather information for the current location is acquired, and the weather information is input as one of the state variables. good.

- The camera 70 may be a stereo camera. By using a stereo camera, it is possible to more accurately grasp the size of unevenness on the road surface and the distance to the unevenness.
・In addition to the camera 70, a lidar that measures the distance to an object using light, a sonar that detects an object using sound waves, etc. may be used to collect road surface information data as one of the state variables. .

- In the above embodiment, an example is shown in which an electronically controlled coupling is provided as a mechanism for controlling the torque of each driving wheel, which can change the distribution of the driving force to the left and right rear wheels. If the brakes of each drive wheel can be individually controlled, the torque of each drive wheel can be controlled. Therefore, torque control using a model that has undergone reinforcement learning can also be applied to a vehicle that does not have such a mechanism for changing the distribution of driving force, as in the above embodiment.

・Slipping of the drive wheels can be suppressed by individually controlling the torque of the drive wheels. Therefore, torque control using a reinforcement learning model can be applied to a front-wheel drive vehicle and a rear-wheel drive vehicle as in the above embodiment.

- In the above embodiment, the greedy policy for the optimized action-value function Q is selected by optimizing the action-value function Q while trying to select and try the action a with the greedy policy based on the action-value function Q. is considered to be the optimal policy. This process is a so-called value iteration method, but learning may be performed by other methods, such as policy iteration method. Furthermore, various normalizations may be performed on various variables such as state s, action a, and reward r.

・Various methods are adopted as machine learning methods, and trials may be performed by the ε-greedy policy based on the action value function Q. Also, the method of reinforcement learning is not limited to Q-learning as described above, and a method such as SARSA may be used. Alternatively, a technique in which the policy model and the action-value function Q model are separately modeled, such as the Actor-Critic algorithm, may be used.

- The vehicle control device is not limited to one that includes the processing circuit 120 and the storage device 110 and executes software processing. For example, a dedicated hardware circuit (for example, an ASIC, etc.) that performs hardware processing at least part of what is software-processed in the above embodiments may be provided. That is, the vehicle control device may have any one of the following configurations (a) to (c). (a) A processing circuit that executes all processes according to a program and a storage device such as a ROM that stores the program. (b) A processing circuit and a storage device for executing part of the processing according to a program, and a dedicated hardware circuit for executing the remaining processing. (c) provide dedicated hardware circuitry to perform all of the above processing; Here, there may be a plurality of software execution devices provided with processing circuits and storage devices, or a plurality of dedicated hardware circuits.

DESCRIPTION OF SYMBOLS 100... Control device 110... Storage device 120... Processing circuit 10... Engine 20... Transmission 21... Front differential 30... Transfer 31... Front disconnect mechanism 40... Rear differential 41... Rear disconnect mechanism 42R... Electronic control coupling 42L... Electronically controlled coupling 60RF... Brake 60LF... Brake 60RR... Brake 60LR... Brake 70... Camera 76RF... Right front speed sensor 76LF... Left front speed sensor 76RR... Right rear speed sensor 76LR... Left rear speed sensor 77... GPS device

Claims (1)

A vehicle control device for controlling a vehicle equipped with a mechanism capable of individually controlling the torque of each driving wheel,
a storage device storing a learned model that determines and outputs an action for manipulating the control amount of the mechanism when state variables including road surface information data extracted from an image of the road surface in front of the vehicle are input;
a processing circuit that manipulates the control amount based on the behavior output by inputting the state variables to the trained model and controls the mechanism;
The learned model is a model learned by reinforcement learning that gives a greater reward as the vehicle travels a longer distance without slipping any driving wheels.

Images