JP2021067256A - Method of generating vehicle control data, vehicle control device, vehicle control system, and vehicle learning device - Google Patents

Abstract

To provide a method of generating vehicle control data that allows reduction in the number of man-hours necessary for an expert in setting a relationship between a state of a vehicle and an action variable.SOLUTION: In a transient period (S72: NO), a CPU sets a throttle opening degree command value TA* and a retardation amount aop of ignition timing on the basis of time-series data of an accelerator operation amount PA, operates a throttle valve and an ignition device according to them (S32a, S34), and acquires a torque Trq, a torque command value Trq* and an acceleration Gx at that time (S70). If the transient period ends, the CPU updates an action value function Q by rewarding according to whether the torque Trq and the acceleration Gx satisfy criteria (S74). Thus, reinforcement learning is performed only in the transient period.SELECTED DRAWING: Figure 5

Description

The present invention relates to a method for generating control data for a vehicle, a control device for a vehicle, a control system for a vehicle, and a learning device for a vehicle.

For example, Patent Document 1 below describes a control device that operates a throttle valve as an operation unit of an internal combustion engine mounted on a vehicle based on a value obtained by filtering the operation amount of the accelerator pedal.

Japanese Unexamined Patent Publication No. 2016-6327

By the way, in the above filter, it is necessary to set the operation amount of the throttle valve of the internal combustion engine mounted on the vehicle to an appropriate operation amount according to the operation amount of the accelerator pedal. However, it is necessary to spend a lot of man-hours. As described above, conventionally, a skilled person has spent a lot of man-hours to adapt the operation amount of the electronic device in the vehicle according to the state of the vehicle.

Hereinafter, means for solving the above problems and their actions and effects will be described.
1. 1. When the state of the vehicle satisfies a predetermined condition, the first data defining the relationship between the state of the vehicle and the action variable indicating the behavior related to the operation of the electronic device in the vehicle is stored in the storage device. Based on the acquisition process of acquiring the detection value of the sensor that detects the state of the vehicle, the operation process of operating the electronic device, and the detection value acquired by the acquisition process when the predetermined conditions are satisfied, the said A reward calculation process that gives a larger reward than when the characteristics of the vehicle meet the criteria, and when the predetermined condition is satisfied, the state of the vehicle based on the detected value acquired by the acquisition process, the electronic The value of the action variable used for the operation of the device and the reward corresponding to the operation are input to the predetermined update mapping, and the execution device is made to execute the update process for updating the first data. The updated mapping outputs the first data updated so as to increase the expected profit for the reward when the electronic device is operated according to the first data, and the state of the vehicle is predetermined. This is a method of generating vehicle control data in which the relationship between the vehicle state and the action variable is matched and used as the second data regardless of the reward calculation process and the update process when the above conditions are not satisfied.

In the above method, when a predetermined condition is satisfied, by calculating the reward corresponding to the operation of the electronic device, it is possible to grasp what kind of reward can be obtained by the operation. Then, based on the reward, the relationship between the state of the vehicle and the behavior variable can be set by updating the first data by the update mapping according to the reinforcement learning. Therefore, it is possible to reduce the man-hours required for the expert when setting the relationship between the state of the vehicle and the behavior variable. Moreover, when the predetermined condition is satisfied, the first data is updated by reinforcement learning, and when the predetermined condition is not satisfied, the second data is matched without relying on reinforcement learning. By setting the above as a predetermined condition, it becomes possible to use reinforcement learning under a condition in which the effect is remarkable in reducing the man-hours of a skilled person.

2. The predetermined condition is the method for generating vehicle control data according to 1 above, which is a condition for transient driving.
In transient operation, the man-hours of a skilled person tend to be larger in matching the relationship between the vehicle state and the behavior variable than in steady operation. Therefore, in the above method, by setting a predetermined condition as a condition that the driver is in transient operation, reinforcement learning automatically adjusts the relationship between the vehicle state and the behavior variable under the condition that the man-hours of the expert are particularly large. By executing this, the man-hours required for skilled workers can be effectively reduced.

3. 3. The vehicle is equipped with an internal combustion engine, the electronic device includes an operation unit of the internal combustion engine, and the first data is an operation of the operation unit of the internal combustion engine as the state of the vehicle and the action variable. The method for generating vehicle control data according to 1 or 2 above, which defines the relationship with the quantity.

Since an internal combustion engine generally has a large number of operation units and has many required elements such as exhaust characteristics, fuel consumption rate, and drivability, the relationship between the state of the vehicle and the operation amount of the operation unit as an action variable is matched. It tends to require a lot of man-hours for skilled workers. Therefore, in the above method, the man-hours of a skilled person can be effectively reduced by using reinforcement learning to match the relationship between the state of the vehicle and the operation amount of the operation unit as an action variable.

4. For control that inputs the state of the vehicle and outputs the value of the action variable that maximizes the expected profit by associating the state of the vehicle with the value of the action variable that maximizes the expected profit on a one-to-one basis. The vehicle control data generation method according to any one of 1 to 3 above, wherein the execution device executes a process of generating mapping data based on the first data updated by the update process.

In the above method, control mapping data is generated based on the first data learned by reinforcement learning. Therefore, by implementing the control mapping data in the control device, it is possible to easily set the value of the action variable that maximizes the expected return based on the state of the vehicle.

5. The storage device and the execution device according to any one of the above 1 to 3 are provided, the second data is stored in the storage device, and the operation process satisfies the predetermined conditions. In the case, the first operation process of operating the electronic device according to the value of the action variable according to the state of the vehicle acquired by the acquisition process based on the first data, and the first operation process when the predetermined condition is not satisfied, the first operation. 2 This is a vehicle control device including a second operation process for operating the electronic device according to the value of an action variable according to the state of the vehicle acquired by the acquisition process based on the data.

In the above configuration, when a predetermined condition is satisfied, the value of the action variable is set based on the first data learned by reinforcement learning, and the electronic device is operated based on the value, so that the expected return is increased. Can be operated. Moreover, when the predetermined conditions are met, the related regulation data is updated by the reward calculation process, so that the opportunity to update the related regulation data is increased as compared with the case where the user does not update according to the driving scene of the vehicle. Can be made to.

6. The storage device includes an execution device and a storage device, and stores the first data and the second data that define the relationship between the state of the vehicle and the behavior variable which is a variable related to the operation of the electronic device in the vehicle. The executing device has an acquisition process for acquiring a detection value of a sensor that detects the state of the vehicle, an operation process for operating the electronic device, and an acquisition process when the state of the vehicle satisfies a predetermined condition. Based on the detection value acquired by, the reward calculation process that gives a larger reward than the case where the characteristics of the vehicle satisfy the criteria, and the acquisition process when the state of the vehicle satisfies the predetermined condition. The state of the vehicle based on the acquired detected value, the value of the action variable used for the operation of the electronic device, and the reward corresponding to the operation are input to the predetermined update mapping, and the first The update process of updating the data and the update process are executed, and the update mapping is the first data updated so as to increase the expected profit for the reward when the electronic device is operated according to the first data. The operation process operates the electronic device according to the value of the action variable according to the state of the vehicle acquired by the acquisition process based on the first data when the predetermined condition is satisfied. The first operation process to be performed and the second operation to operate the electronic device according to the value of the action variable according to the state of the vehicle acquired by the acquisition process based on the second data when the predetermined condition is not satisfied. A vehicle control device that includes processing.

In the above configuration, when a predetermined condition is satisfied, by calculating the reward corresponding to the operation of the electronic device, it is possible to grasp what kind of reward can be obtained by the operation. Then, based on the reward, the relationship between the state of the vehicle and the behavior variable can be set by updating the first data by the update mapping according to the reinforcement learning. Therefore, when setting the relationship between the state of the vehicle and the behavior variable to an appropriate relationship in the running of the vehicle, the man-hours required for the expert can be reduced. Moreover, since the first data is updated by reinforcement learning when a predetermined condition is satisfied, the effect of reducing the man-hours of the expert is remarkable by setting the condition that the man-hour of the expert is large as the predetermined condition. Reinforcement learning can be used under conditions.

7. The execution device and the storage device according to the above 5 or 6 are provided, and the execution device includes a first execution device mounted on the vehicle and a second execution device different from the in-vehicle device, and the first execution device includes the first execution device. The 1 execution device is a vehicle control system that executes at least the acquisition process and the operation process, and the second execution device executes at least the update process.

In the above configuration, by executing the update process by the second execution device, the calculation load of the first execution device can be reduced as compared with the case where the update process is executed by the first execution device.
The fact that the second executing device is a device different from the in-vehicle device means that the second executing device is not an in-vehicle device.

8. It is a vehicle control device including the first execution device according to the above 7.
9. It is a learning device for a vehicle including the second executing device according to the above 7.

The figure which shows the structure of the drive system and the control device which concerns on 1st Embodiment. The flow chart which shows the procedure of the process executed by the control device which concerns on the same embodiment. The figure which shows the system which generates the map data concerning this embodiment. The flow chart which shows the procedure of the generation process of the steady-state map data which concerns on the same embodiment. The flow chart which shows the procedure of the learning process concerning this embodiment. The flow chart which shows a part of the details of the learning process which concerns on this embodiment. The flow chart which shows the procedure of the generation process of the transient map data which concerns on the same embodiment. The figure which shows the control device and the drive system which concerns on 2nd Embodiment. The flow chart which shows the procedure of the process executed by the control device which concerns on the same embodiment. The figure which shows the structure of the system which concerns on 3rd Embodiment. The flow chart which shows the procedure of the process executed by the control device which concerns on the same embodiment. The flow chart which shows the detailed procedure of a part of the processing executed by the control device which concerns on this embodiment. (A) and (b) are flow charts showing the procedure of processing executed by the system according to the same embodiment.

Hereinafter, embodiments of a vehicle control data generation method, a vehicle control device, a vehicle control system, and a vehicle learning device will be described with reference to the drawings.
<First Embodiment>
FIG. 1 shows the configuration of the drive system and the control device of the vehicle VC1 according to the present embodiment.

As shown in FIG. 1, the intake passage 12 of the internal combustion engine 10 is provided with a throttle valve 14 and a fuel injection valve 16 in this order from the upstream side, and is injected from the air or fuel injection valve 16 sucked into the intake passage 12. The fuel is discharged into the combustion chamber 24 partitioned by the cylinder 20 and the piston 22 as the intake valve 18 is opened. In the combustion chamber 24, the air-fuel mixture is subjected to combustion with the spark discharge of the ignition device 26, and the energy generated by the combustion is converted into the rotational energy of the crankshaft 28 via the piston 22. To. The air-fuel mixture used for combustion is discharged to the exhaust passage 32 as exhaust gas when the exhaust valve 30 is opened. The exhaust passage 32 is provided with a catalyst 34 as an aftertreatment device for purifying the exhaust gas.

The input shaft 52 of the transmission 50 can be mechanically connected to the crankshaft 28 via a torque converter 40 provided with a lockup clutch 42. The transmission 50 is a device that makes the gear ratio variable, which is the ratio of the rotation speed of the input shaft 52 to the rotation speed of the output shaft 54. A drive wheel 60 is mechanically connected to the output shaft 54.

The control device 70 targets the internal combustion engine 10, and in order to control the torque and the exhaust component ratio, which are the controlled amounts thereof, the control device 70 controls the operation unit of the internal combustion engine 10 such as the throttle valve 14, the fuel injection valve 16, and the ignition device 26. Manipulate. Further, the control device 70 controls the torque converter 40 and operates the lockup clutch 42 in order to control the engaged state of the lockup clutch 42. Further, the control device 70 targets the transmission 50 as a control target, and operates the transmission 50 to control the gear ratio as the control amount thereof. Note that FIG. 1 shows the operation signals MS1 to MS5 of the throttle valve 14, the fuel injection valve 16, the ignition device 26, the lockup clutch 42, and the transmission 50, respectively.

The control device 70 controls the intake air amount Ga detected by the air flow meter 80, the opening degree of the throttle valve 14 (throttle opening degree TA) detected by the throttle sensor 82, and the crank angle sensor 84 for controlling the control amount. Refer to the output signal Scr of. Further, the control device 70 uses the detection value Afu of the air-fuel ratio sensor 86 provided on the upstream side of the catalyst 34, the depression amount of the accelerator pedal 88 (accelerator operation amount PA) detected by the accelerator sensor 90, and the acceleration sensor 92. Refer to the detected acceleration Gx in the front-rear direction of the vehicle VC1.

The control device 70 includes a CPU 72, a ROM 74, an electrically rewritable non-volatile memory (storage device 76), and a peripheral circuit 78, which can communicate with each other via the local network 79. Here, the peripheral circuit 78 includes a circuit that generates a clock signal that defines the internal operation, a power supply circuit, a reset circuit, and the like.

The control program 74a is stored in the ROM 74. On the other hand, the storage device 76 stores stationary map data DMs and transient map data DMt. The steady-state map data DMs include map data in which the accelerator operation amount PA and the rotation speed NE are input variables and the command value of the throttle opening degree TA (throttle opening degree command value TA *) is the output variable, and the rotation speed NE and the filling efficiency. It consists of map data with η as the input variable and the reference ignition timing abse as the output variable. The transient map data DMt uses the time-series data of the accelerator operation amount PA as the input variable, the map data with the throttle opening command value TA * as the output variable, and the time-series data of the accelerator operation amount PA as the input variable, and is the reference ignition. It consists of map data whose output variable is the retard angle amount aop with respect to the time abse. The reference ignition timing abse is the timing on the retard side of the MBT ignition timing and the knock limit point. The MBT ignition timing is the ignition timing at which the maximum torque can be obtained (maximum torque ignition timing). The knock limit is an ignition timing advance limit that can keep knocking within an acceptable level under the best possible conditions when using a high octane fuel with a high knock limit.

The map data is a set of data of discrete values of input variables and values of output variables corresponding to the values of the input variables.
FIG. 2 shows a procedure of processing executed by the control device 70 according to the present embodiment. The process shown in FIG. 2 is realized by the CPU 72 repeatedly executing the control program 74a stored in the ROM 74, for example, at a predetermined cycle. In the following, the step number of each process is indicated by a number prefixed with "S".

In the series of processes shown in FIG. 2, the CPU 72 first acquires the rotation speed NE and the filling efficiency η (S10). Here, the rotation speed NE is calculated by the CPU 72 based on the output signal Scr. Further, the filling efficiency η is calculated by the CPU 72 based on the rotation speed NE and the intake air amount Ga. Next, the CPU 72 performs a map calculation of the reference ignition timing abscess using the rotation speed NE and the filling efficiency η as input variables using the map data defined by the stationary map data DMs and using the reference ignition timing abscess as the output variable (S12). .. Here, in the map operation, for example, when the value of the input variable matches any of the values of the input variable of the map data, the value of the output variable of the corresponding map data is used as the operation result, whereas the value does not match. , The value obtained by interpolating the values of a plurality of output variables included in the map data may be used as the calculation result.

Then, the CPU 72 determines whether or not the transient flag F is “1” (S14). When the transient flag F is "1", it indicates that it is in transient operation, and when it is "0", it indicates that it is not in transient operation. When the CPU 72 determines that the transient flag F is "0" (S14: NO), the CPU 72 determines whether or not the absolute value of the change amount ΔPA of the accelerator operation amount PA per unit time is equal to or greater than the predetermined amount ΔPAth. (S16). Here, the change amount ΔPA may be, for example, the difference between the latest accelerator operation amount PA at the execution timing of the process of S16 and the accelerator operation amount PA before the same timing by a unit time.

When the CPU 72 determines that the predetermined amount is ΔPAth or more (S16: YES), the CPU 72 substitutes “1” for the transient flag F (S18).
On the other hand, when the CPU 72 determines that the transient flag F is "1" (S14: YES), the CPU 72 determines whether or not a predetermined period has elapsed since the affirmative determination was made in the process of S16 (S20). Here, the predetermined period is a period until the state in which the absolute value of the change amount ΔPA of the accelerator operation amount PA per unit time is smaller than the predetermined amount ΔPAth and is equal to or less than the specified amount continues for a predetermined time. When the CPU 72 determines that the predetermined period has elapsed (S20: YES), the CPU 72 substitutes “0” for the transient flag F (S22).

The CPU 72 acquires the accelerator operation amount PA when the process of S22 is completed or when a negative determination is made in the process of S16 (S24). Then, the CPU 72 sets the throttle opening command value TA * as the accelerator operation amount PA and the rotation speed NE as the input variables of the map data defined by the steady map data DMs and using the throttle opening command value TA * as the output variable. * Is mapped (S26). Here, in the map operation, for example, when the value of the input variable matches any of the values of the input variable of the map data, the value of the output variable of the corresponding map data is used as the operation result, whereas the value does not match. , The value obtained by interpolating the values of a plurality of output variables included in the map data may be used as the calculation result.

Then, the CPU 72 outputs an operation signal MS1 to the throttle valve 14 to operate the throttle valve 14 based on the throttle opening degree command value TA *, and operates the ignition device 26 to operate the ignition device 26 based on the reference ignition timing abse. The signal MS3 is output (S28).

On the other hand, when the processing of S18 is completed or when a negative determination is made in the processing of S20, the CPU 72 has six sampling values "PA (1), PA (2), ... PA (6) of the accelerator operation amount PA. ) ”(S30). Here, each sampling value constituting the time series data is sampled at different timings from each other. In the present embodiment, time-series data is configured by six sampling values that are adjacent to each other in time series when sampling is performed at a constant sampling cycle.

Then, the CPU 72 performs a map calculation of the throttle opening command value TA * and the retard angle amount aop based on the transient map data DMt (S32). That is, the CPU 72 performs map calculation of the throttle opening degree command value TA * by using the input variable of the map data defined by the transient map data DMt and using the throttle opening degree command value TA * as the output variable as the time series data. Further, the CPU 72 performs a map calculation of the retard angle amount aop by using the input variable of the map data defined by the transient map data DMt and using the retard angle amount aop as the output variable as the time series data.

Then, the CPU 72 outputs the operation signal MS1 to the throttle valve 14 to operate the throttle opening degree TA, and outputs the operation signal MS3 to the ignition device 26 to operate the ignition timing (S34). Here, the CPU 72 sets the ignition timing based on the timing at which the reference ignition timing abse is retarded by the retard angle amount aop. Specifically, when a well-known knocking control (KCS) or the like is performed, the CPU 72 sets the ignition timing as a value obtained by correcting the reference ignition timing abscess with a retard angle amount aop and then feedback-correcting the ignition timing. To do. In this embodiment, the throttle opening degree TA is feedback-controlled to the throttle opening degree command value TA *. Therefore, even if the throttle opening degree command value TA * is the same value, the operation signal MS1 is generated. It can be different signals from each other.

When the processes of S28 and S34 are completed, the CPU 72 temporarily ends the series of processes shown in FIG.
FIG. 3 shows a system for generating the stationary map data DMs and the transient map data DMt.

As shown in FIG. 3, in the present embodiment, the dynamometer 100 is mechanically connected to the crankshaft 28 of the internal combustion engine 10 via the torque converter 40 and the transmission 50. Then, various state variables when the internal combustion engine 10 is operated are detected by the sensor group 102, and the detection results are input to the generator 110, which is a computer that generates steady-state map data DMs and transient map data DMt. The sensor group 102 includes sensors and the like mounted on the vehicle VC1 shown in FIG.

The generator 110 includes a CPU 112, a ROM 114, an electrically rewritable non-volatile memory (storage device 116), and peripheral circuits 118, which are made communicable by the local network 119. Here, the storage device 116 stores the relationship regulation data DR, which is data that defines the relationship between the accelerator operation amount PA, the throttle opening degree command value TA *, and the retard angle amount aop. Further, the ROM 114 stores a learning program 114a for learning the relational regulation data DR by reinforcement learning.

FIG. 4 shows a procedure for generating steady-state map data DMs. The process shown in FIG. 4 is realized by the CPU 112 executing the learning program 114a stored in the ROM 114.

In the series of processes shown in FIG. 4, the CPU 112 sets one reference ignition timing abse according to the rotation speed NE and the filling efficiency η in a state where the internal combustion engine 10 is in steady operation, and the ignition timing is the reference ignition timing abse. The ignition device 26 is operated so as to be (S40). The value set here is one of a plurality of candidates predetermined by an expert. Then, the CPU 72 acquires the torque Trq of the internal combustion engine 10 and the knocking strength by the knocking sensor included in the sensor group 102 (S42). Here, the torque Trq is calculated by the CPU 112 based on the load torque generated by the dynamometer 100 and the gear ratio of the transmission 50. Next, the CPU 112 determines whether or not the reference ignition timing abse set by the process of S40 is the optimum timing based on the torque Trq and the knocking strength (S44). Here, the optimum time is an appropriate time as the time on the retard side of the MBT and the knock limit ignition timing.

When the CPU 112 determines that the timing is not optimal (S44: NO), the CPU 112 returns to the process of S40 and sets the reference ignition timing abscess to another timing. On the other hand, when the CPU 112 determines that the timing is optimal (S44: YES), the CPU 112 determines the ignition timing set by the process of S40 as the reference ignition timing abscess (S46). Then, the CPU 112 determines whether or not the processing of S46 is completed for all the operating points defined by the rotation speed NE and the filling efficiency η and defining the steady-state map data DMs (S48). When the CPU 112 determines that there is an operating point for which the processing of S46 has not been completed (S48: NO), the CPU 112 returns to the processing of S40.

On the other hand, when the CPU 112 determines that the processing of S46 is completed for all the operating points (S48: YES), the throttle according to the accelerator operation amount PA and the rotation speed NE while the internal combustion engine 10 is in steady operation. The opening degree command value TA * is set (S50). The value set here is one of a plurality of candidates preset by an expert. Then, the CPU 112 acquires the torque Trq (S52), and based on the torque Trq, determines whether or not the throttle opening degree command value TA * acquired by the processing of S50 is the optimum opening degree (S54). Here, the CPU 112 may determine the optimum opening degree when the deviation between the torque command value Trq * and the torque Trq according to the accelerator operation amount PA is sufficiently small. When the CPU 112 determines that the opening degree is not the optimum (S54: NO), the CPU 112 returns to the process of S50 and sets another value as the throttle opening degree command value TA *.

On the other hand, when the CPU 112 determines that the opening degree is the optimum (S54: YES), the throttle opening degree command value TA * set in the process of S50 is set according to the accelerator operation amount PA and the rotation speed NE at that time. It is determined as a value (S56). Then, the CPU 112 determines whether or not the processing of S56 is completed for all the sets of the accelerator operation amount PA and the rotation speed NE that define the steady-state map data DMs (S58). Then, when the CPU 112 determines that there is a set in which the processing of S56 has not been performed yet (S58: NO), the CPU 112 returns to the processing of S50.

On the other hand, when the CPU 112 determines that the processing of S56 has been completed for all the sets (S58: YES), the CPU 112 creates steady-state map data DMs (S60).
When the process of S60 is completed, the CPU 112 temporarily ends the process of FIG.

FIG. 5 shows a preprocessing procedure for generating transient map data DMt. The process shown in FIG. 5 is realized by the CPU 112 executing the learning program 114a stored in the ROM 114.

In the series of processes shown in FIG. 5, the CPU 112 first acquires time-series data of the accelerator operation amount PA as the state s in the state where the internal combustion engine 10 is operated (S30). The time-series data here is the same data as that in the process of S30 shown in FIG. However, in the system shown in FIG. 3, the accelerator pedal 88 does not exist. Therefore, it is assumed that the accelerator operation amount PA is simulated by the generator 110 simulating the state of the vehicle VC1, and the pseudo-generated accelerator operation amount PA is regarded as the detected value of the vehicle state. .. Here, the CPU 112 simulates the transient operation state of the internal combustion engine 10 by changing the accelerator operation amount PA.

Next, the CPU 112 sets the action a including the throttle opening degree command value TA * and the retard angle amount aop according to the state s acquired by the process of S30 according to the policy π defined by the relational regulation data DR (S32a).

In the present embodiment, the relational regulation data DR is data that defines the action value function Q and the policy π. In the present embodiment, the action value function Q is a table-type function showing the value of the expected return according to the eight-dimensional independent variables of the state s and the action a. Further, the policy π is predetermined while preferentially selecting the action a (greedy action) that is the largest among the action value functions Q in which the independent variable is given the state s when the state s is given. With the probability ε of, the rule for selecting other actions a is determined.

Specifically, the number of possible values of the independent variable of the action value function Q according to the present embodiment is that a part of all combinations of possible values of the state s and the action a is reduced by human knowledge or the like. is there. That is, for example, in the time series data of the accelerator operation amount PA, one of the two adjacent sampling values becomes the minimum value of the accelerator operation amount PA and the other becomes the maximum value from the operation of the accelerator pedal 88 by a person. The action value function Q is not defined as it cannot occur. In the present embodiment, the possible values of the state s that defines the action value function Q are limited to 10 4 or less, more preferably 10 3 or less, by dimensionality reduction based on human knowledge or the like.

Next, the CPU 112 outputs the operation signals MS1 and MS3 based on the set throttle opening command value TA * and the retard angle amount aop (S34). Next, the CPU 112 acquires the torque Trq of the internal combustion engine 10, the torque command value Trq * for the internal combustion engine 10, and the acceleration Gx (S70). Here, the CPU 112 calculates the torque Trq based on the load torque generated by the dynamometer 100 and the gear ratio of the transmission 50. Further, the CPU 112 sets the torque command value Trq * according to the accelerator operation amount PA. Further, the CPU 112 calculates the acceleration Gx as a value that is assumed to occur in the vehicle if the internal combustion engine 10 or the like is mounted in the vehicle, based on the load torque of the dynamometer 100 or the like. That is, in the present embodiment, the acceleration Gx is also virtual, but this acceleration Gx is also regarded as a detected value of the state of the vehicle.

Next, the CPU 112 determines whether or not the transition period has ended (S72). Here, the CPU 112 sets the absolute value of the change amount ΔPA of the accelerator operation amount PA per unit time to be equal to or greater than the predetermined amount ΔPth, and then sets the absolute value of the change amount ΔPA per unit time to be smaller than the predetermined amount ΔPAth. When this state continues for a predetermined time, it is determined that the transition period has ended. When the CPU 72 determines that the transition period has not yet been completed (S72: NO), the CPU 72 returns to the process of S30.

On the other hand, when the CPU 112 determines that the transition period is completed (S72: YES), the CPU 112 updates the action value function Q by reinforcement learning, assuming that one episode is completed (S74).

FIG. 6 shows the details of the processing of S74.
In the series of processes shown in FIG. 6, the CPU 112 performs time-series data including a set of three sampling values of the torque command value Trq *, the torque Trq, and the acceleration Gx in the most recently completed episode, and the state s and the action a. (S80). FIG. 6 shows that the values in parentheses that differ are the values of the variables at different sampling timings. For example, the torque command value Trq * (1) and the torque command value Trq * (2) have different sampling timings. Further, the time-series data of the action a belonging to the latest episode is defined as the action set Aj, and the time-series data of the state s belonging to the same episode is defined as the state set Sj.

Next, the CPU 112 has a condition (a) that the absolute value of the difference between the arbitrary torque Trq belonging to the latest episode and the torque command value Trq * is the specified amount ΔTrq or less, and the acceleration Gx is the lower limit value GxL or more. It is determined whether or not the logical product with the condition (a) indicating that the upper limit value is GxH or less is true (S82).

Here, the CPU 112 variably sets the specified amount ΔTrq according to the amount of change ΔPA per unit time of the accelerator operation amount PA at the start of the episode. That is, when the absolute value of the change amount ΔPA of the accelerator operation amount PA at the start of the episode is large, the CPU 112 considers that the episode is related to the transient time, and sets the specified amount ΔTrq to a larger value than the episode related to the steady state. Set.

Further, the CPU 112 variably sets the lower limit value GxL according to the change amount ΔPA of the accelerator operation amount PA at the start of the episode. That is, when the CPU 112 is an episode related to the transient time and the amount of change ΔPA is positive, the CPU 112 sets the lower limit value GxL to a larger value than the case of the episode related to the steady state. Further, when the CPU 112 is an episode related to the transient time and the amount of change ΔPA is negative, the CPU 112 sets the lower limit value GxL to a smaller value than the case of the episode related to the steady state.

Further, the CPU 112 variably sets the upper limit value GxH according to the change amount ΔPA per unit time of the accelerator operation amount PA at the start of the episode. That is, when the CPU 112 is an episode related to the transient time and the amount of change ΔPA is positive, the CPU 112 sets the upper limit value GxH to a larger value than the case of the episode related to the steady time. Further, when the CPU 112 is an episode related to the transient time and the amount of change ΔPA is negative, the CPU 112 sets the upper limit value GxH to a smaller value than the case of the episode related to the steady state.

When the CPU 112 determines that the logical product is true (S82: YES), it substitutes "10" for the reward r (S84), while when it determines that the logical product is false (S82: NO), the reward r is "". -10 ”is substituted (S86). When the processing of S84 and S86 is completed, the CPU 112 updates the relational regulation data DR stored in the storage device 116 shown in FIG. In this embodiment, the ε soft policy on-type Monte Carlo method is used.

That is, the CPU 112 adds the reward r to the profit R (Sj, Aj) determined by the set of each state read by the process of S80 and the corresponding action (S88). Here, "R (Sj, Aj)" is a general description of the profit R in which one of the elements of the state set Sj is a state and one of the elements of the action set Aj is an action. Next, each of the revenues R (Sj, Aj) determined by the set of each state read by the process of S80 and the corresponding action is averaged and substituted into the corresponding action value function Q (Sj, Aj) ( S90). Here, the averaging may be a process of dividing the profit R calculated by the process of S88 by the number of times the process of S88 is performed. The initial value of the profit R may be zero.

Next, the CPU 112 describes the throttle opening command value TA * and the retard angle amount aop at the maximum value of the corresponding action value functions Q (Sj, A) for the state read by the process of S80, respectively. The set of actions is substituted into the action Aj * (S92). Here, "A" indicates an arbitrary action that can be taken. The action Aj * has a different value depending on the type of the state read by the process of S80, but here, the notation is simplified and described by the same symbol.

Next, the CPU 72 updates the corresponding policy π (Aj | Sj) for each of the states read by the process of S80 (S94). That is, assuming that the total number of actions is "| A |", the selection probability of the action Aj * selected by S92 is "1-ε + ε / | A |". Further, the selection probabilities of "| A | -1" actions other than the action Aj * are set to "ε / | A |", respectively. Since the process of S94 is a process based on the action value function Q updated by the process of S90, the relationship rule data DR that defines the relationship between the state s and the action a increases the profit R. Will be updated to.

When the process of S94 is completed, the CPU 112 temporarily ends the series of processes shown in FIG.
Returning to FIG. 5, when the processing of S74 is completed, the CPU 112 determines whether or not the action value function Q has converged (S76). Here, it may be determined that the action value function Q has converged when the number of consecutive times when the update amount of the action value function Q is equal to or less than the predetermined value reaches the predetermined number of times for each value of the independent variable. When the CPU 112 determines that the convergence has not occurred (S76: NO), the CPU 112 returns to the process of S30. On the other hand, when the CPU 112 determines that the convergence has occurred (S76: YES), the CPU 112 ends the series of processes shown in FIG.

FIG. 7 shows a procedure of processing for generating transient map data DMt, particularly based on the action value function Q learned by the processing of FIG. 5, among the processes executed by the generation device 110. The process shown in FIG. 7 is realized by the CPU 112 executing the learning program 114a stored in the ROM 114.

In the series of processes shown in FIG. 7, the CPU 112 first selects one state s (S100). Next, the CPU 112 selects the action a that maximizes the value of the action value function Q from the action value functions Q (s, A) corresponding to the state s (S102). That is, here, the action a is selected by the Greedy policy. Next, the CPU 112 stores the set of the state s and the action a in the storage device 116 (S104).

Next, the CPU 112 determines whether or not all the values of the input variables of the transient map data DMt are selected by the processing of S100 (S106). Then, when the CPU 112 determines that there is something that has not been selected (S106: NO), the CPU 112 returns to the process of S100. On the other hand, when it is determined that all are selected (S106: YES), the CPU 112 generates transient map data DMt based on the data stored by the processing of S104 (S108). Here, the value of the output variable corresponding to the state s of the value of the input variable of the transient map data DMt is set as the corresponding action a.

When the process of S108 is completed, the CPU 112 temporarily ends the series of processes shown in FIG. 7.
Here, the operation and effect of this embodiment will be described.

In the system shown in FIG. 3, the CPU 112 conforms to the steady-state map data DMs without relying on reinforcement learning. On the other hand, the CPU 112 generates the transient map data DMt by learning the action value function Q by reinforcement learning. That is, the CPU 112 acquires the time-series data of the accelerator operation amount PA, and sets the action a including the throttle opening degree command value TA * and the retard angle amount aop according to the policy π. Here, the CPU 72 searches for the action a that maximizes the expected return by selecting an action other than the action a that maximizes the expected return with a predetermined probability ε. Then, when the value of the action value function Q converges, the CPU 112 selects an action that maximizes the action value function Q for each of the states that are input variables of the transient map data DMt, and sets the state and the action. It is stored in the storage device 116. Next, the CPU 112 generates the transient map data DMt based on the pair of the state and the action stored in the storage device 116.

Here, when a skilled person fits the transient map data DMt, the work of setting and evaluating the candidate for the value of the output variable by fumbling is repeated, and the man-hours are increased as compared with the steady state. On the other hand, in the present embodiment, the man-hours of a skilled person can be reduced by using reinforcement learning.

According to the present embodiment described above, the effects described below can be further obtained.
(1) The transient map data DMt is stored in the storage device 76 included in the control device 70, not the action value function Q or the like. As a result, the CPU 72 sets the throttle opening command value TA * and the retard angle amount aop based on the map calculation using the transient map data DMt, and thus selects the one having the maximum value among the action value functions Q. The calculation load can be reduced as compared with the case of executing the process.

(2) The time series data of the accelerator operation amount PA was included in the independent variable of the action value function Q. As a result, the value of the action a can be finely adjusted for various changes in the accelerator operation amount PA, as compared with the case where only a single sampling value is used as the independent variable for the accelerator operation amount PA.

(3) The throttle opening command value TA * itself is included in the independent variable of the action value function Q. This makes it easier to increase the degree of freedom of search by reinforcement learning, for example, as compared with the case where the parameters of the model formula that models the behavior of the throttle opening command value TA * are used as independent variables related to the throttle opening. Is.

<Second embodiment>
Hereinafter, the second embodiment will be described with reference to the drawings, focusing on the differences from the first embodiment.

FIG. 8 shows the drive system and the control device of the vehicle VC1 according to the present embodiment. In FIG. 8, the members corresponding to the members shown in FIG. 1 are designated by the same reference numerals for convenience.

As shown in FIG. 8, in the present embodiment, the learning program 74b is stored in the ROM 74 in addition to the control program 74a. Further, although the stationary map data DMs are stored in the storage device 76, the transient map data DMt is not stored. Instead, the relational regulation data DR is stored, and the torque output mapping data DT is stored. It is remembered. Here, the relational regulation data DR is the learned data learned by the process of FIG. Further, the torque output mapping defined by the torque output mapping data DT is data related to a trained model such as a neural network that outputs torque Trq with rotation speed NE, filling efficiency η, and ignition timing as inputs. The torque output mapping data DT may be learned by using the torque Trq acquired by the processing of S70 as the teacher data when executing the processing of FIG. 5, for example.

FIG. 9 shows a procedure of processing executed by the control device 70 according to the present embodiment. The process shown in FIG. 9 is realized by the CPU 72 repeatedly executing the control program 74a and the learning program 74b stored in the ROM 74, for example, at a predetermined cycle. In FIG. 9, the same step numbers are assigned to the processes corresponding to the processes shown in FIGS. 2 and 5 for convenience.

In the series of processes shown in FIG. 9, when the process of S30 is completed, the CPU 72 sequentially executes the processes of S32a, S34, and S70, and temporarily ends the series of processes shown in FIG. Further, when the CPU 72 completes the process of S22, the CPU 72 executes the process of S74. Then, when the CPU 72 completes the process of S74 or determines negative in the process of S16, the CPU 72 executes the processes of S24 to S28 and temporarily ends the series of processes shown in FIG. Incidentally, the processing other than the processing of S74 in the processing of FIG. 9 is realized by the CPU 72 executing the control program 74a, and the processing of S74 is realized by the CPU 72 executing the learning program 74b.

As described above, according to the present embodiment, by mounting the relational regulation data DR and the learning program 74b on the control device 70, the relational regulation data DR can be updated with the actual running of the vehicle VC1, and therefore, the first As compared with the case of the embodiment of the above, the update frequency of the related regulation data DR can be improved.

<Third embodiment>
Hereinafter, the third embodiment will be described with reference to the drawings, focusing on the differences from the second embodiment.

In the present embodiment, the update of the related regulation data DR is executed outside the vehicle VC1.
FIG. 10 shows the configuration of the control system that executes reinforcement learning in the present embodiment. In FIG. 10, the members corresponding to the members shown in FIG. 1 are designated by the same reference numerals for convenience.

The ROM 74 in the control device 70 in the vehicle VC1 shown in FIG. 10 stores the control program 74a, but does not store the learning program 74b. Further, the storage device 76 stores steady-state map data DMs, related regulation data DR, and torque output mapping data DT. The stationary map data DMs according to the present embodiment include the data in which the reference ignition timing abse in the above embodiment is used as the output variable and the data in which the throttle opening degree command value TA * is used as the output variable, and the filling efficiency η. Contains data with the input variable and the base injection amount Qbse as the output variable. Here, in the data in which the filling efficiency η is used as an input variable and the base injection amount Qbse is used as an output variable, the base injection amount Qbse is set so that the air-fuel mixture corresponding to the filling efficiency η is the stoichiometric air-fuel ratio. It is a value obtained by multiplying the filling efficiency η by a predetermined proportional coefficient. Further, in the relational regulation data DR according to the present embodiment, the action variables are the throttle opening degree command value TA *, the retard angle amount aop, and the base injection amount Qbse.

Further, the control device 70 includes a communication device 77. The communication device 77 is a device for communicating with the data analysis center 130 via the network 120 outside the vehicle VC1.
The data analysis center 130 analyzes the data transmitted from the plurality of vehicles VC1, VC2, .... The data analysis center 130 includes a CPU 132, a ROM 134, an electrically rewritable non-volatile memory (storage device 136), peripheral circuits 138, and a communication device 137, which can be communicated by the local network 139. Is. The learning program 74b is stored in the ROM 134, and the related regulation data DR is stored in the storage device 136.

FIG. 11 shows a procedure of processing executed by the control device 70 according to the present embodiment. The process shown in FIG. 11 is realized by the CPU 72 repeatedly executing the control program 74a stored in the ROM 74, for example, at a predetermined cycle. In FIG. 11, the same step numbers are assigned to the processes corresponding to the processes shown in FIG. 9 for convenience.

In the series of processes shown in FIG. 11, the CPU 72 shifts to the process of S24 when the process of S22 is completed or when a negative determination is made in the process of S16. Then, when the processing of S24 is completed, the CPU 72 performs a map calculation of the throttle opening degree command value TA * and the base injection amount Qbse based on the steady-state map data DMs (S26a). Then, in addition to outputting the operation signals MS1 and MS3 in the same manner as the processing of S28, the CPU 72 outputs the operation signal MS2 to the fuel injection valve 16 in order to operate the fuel injection valve 16 based on the base injection amount Qbse. (S28a). Here, the CPU 72 generates the operation signal MS2 based on the value obtained by correcting the base injection amount Qbse by the operation amount for feedback-controlling the detected value Afu to the target value.

On the other hand, when the processing of S18 is completed, the CPU 72 acquires the time-series data of the rotation speed NE and the filling efficiency η in addition to the time-series data of the accelerator operation amount PA as the state s (S30a). In the present embodiment, each time series data of the accelerator operation amount PA, the rotation speed NE, and the filling efficiency η is set to six values sampled at equal intervals. Next, the CPU 72 sets the action a based on the state s acquired in the process of S30a (S32b).

FIG. 12 shows the details of the processing of S32b.
In the present embodiment, the policy π is a multivariate Gaussian distribution that determines the possible probabilities of each manipulated variable that determines the behavior. Here, the average value μ (1) of the multivariate Gaussian distribution indicates the average value of the throttle opening command value TA *, and the average value μ (2) indicates the average value of the retard angle amount aop, and the average value μ. (3) shows the average value of the base injection amount Qbse. Further, in the present embodiment, the covariance matrix of the multivariate Gaussian distribution is a diagonal matrix, and the variance σ (i) corresponding to each average value μ (i) can be a different value.

As shown in FIG. 12, the CPU 72 substitutes the state s acquired by the process of S30a into the input variables x (1) to x (18) of the function approximator for setting the policy π (S110). Specifically, the CPU 72 substitutes the accelerator operation amount PA (i) for the input variable x (i) and substitutes the rotation speed NE (i) for the input variable x (6 + i) with "i = 1 to 6". Substitute the filling efficiency η (i) into the input variable x (12 + i).

Next, the CPU 72 calculates the average value μ (i) and the variance σ (i) for each of “i = 1-3” (S112). In the present embodiment, the average value μ (i) is such that the number of layers of the intermediate layer is “p-1” and the activation functions h1 to hp-1 of each intermediate layer are hyperbolic tangents, and the output layer. The activation function hp of is configured by a neural network of ReLU. Here, ReLU is a function that outputs the input and "0", whichever is not smaller. Further, assuming that m = 1, 2, ..., P-1, the value of each node in the mth intermediate layer inputs the output of the linear map defined by the coefficient w (m) to the activation function hm. Generated by. Here, n1, n2, ..., Np-1 are the number of nodes in the first, second, ..., And p-1 intermediate layers, respectively. For example, the value of each node in the first intermediate layer is the input variables x (1) to x (1) to the linear map defined by the coefficient w (1) ji (j = 0 to n1, i = 0 to 18). It is generated by inputting the output when 18) is input to the activation function h1. Incidentally, w (1) j0 and the like are bias parameters, and the input variable x (0) is defined as “1”.

In the above neural network, the output when the output of the activation function hp is input to the linear map defined by the coefficient w (p) iq (i = 1-3, q = 0 to np-1) is the average value μ ( i).

Further, in the present embodiment, the variance σ (i) is linearly transformed from the input variables x (1) to x (18) by a linear map defined by the coefficient wTik (i = 1-3, k = 1-18). Let each of these values be the value of the function f when it is input to the function f. In this embodiment, ReLU is exemplified as the function f.

Next, the CPU 72 determines the action a based on the measure π defined by the average value μ (i) calculated by the process of S112 and the variance σ (i) (S114). Here, the probability of selecting the average value μ (i) is the highest, and the probability of selecting the average value μ (i) is larger when the variance σ (i) is small than when it is large.

When the CPU 72 completes the process of S114, the CPU 72 completes the process of S32b in FIG. Then, in addition to outputting the operation signals MS1 and MS3 in the same manner as the processing of S34, the CPU 72 feedback-controls the base injection amount Qbse set by the processing of S32b with the detected value Afu as the target value. The operation signal MS2 is output to the fuel injection valve 16 in order to inject the fuel having the value corrected by the operation amount from the fuel injection valve 16 (S34a).

When the CPU 72 completes the processes of S28a and 34a, the CPU 72 temporarily ends the series of processes shown in FIG.
FIG. 13 shows a processing procedure of reinforcement learning according to the present embodiment. The process shown in FIG. 13A is realized by the CPU 72 executing the control program 74a stored in the ROM 74 shown in FIG. Further, the process shown in FIG. 13B is realized by repeatedly executing the learning program 74b stored in the ROM 134 in the execution cycle of the process of S34a by the CPU 132 when the process of S34a is executed. In the following, the process shown in FIG. 13 will be described along with the time series of reinforcement learning.

As shown in FIG. 13A, the CPU 72 acquires the torque command value Trq *, the torque Trq, the acceleration Gx, and the detected value Afu (S120). Next, the CPU 72 operates the communication device 77 to transmit the data acquired by the processing of S120 (S122).

On the other hand, as shown in FIG. 13B, the CPU 132 receives the data transmitted by the processing of S122 (S130). Next, the CPU 132 is a logical product of the above condition (a), the above condition (a), and the condition (c) that the detected value Afu is equal to or more than the rich side upper limit value AfR and is equal to or less than the lean side upper limit value AfL. Is true or not (S132).

Then, when the CPU 132 determines that the logical product is true (S132: YES), it substitutes "1" for the reward r (S134), while when it determines that the logical product is false (S132: NO), the reward r is assigned. Substitute "-1" (S136). When the processing of S134 and S136 is completed, the CPU 132 adds the reward r to the profit R (S138). Then, the CPU 132 determines whether or not the variable t has reached the predetermined time T-1 (S140). When the CPU 132 determines that the predetermined time T-1 has not been reached (S140: NO), the CPU 132 increments the variable t (S142).

On the other hand, when the CPU 132 determines that the predetermined time T-1 is reached (S140: YES), the CPU 132 substitutes the profit R for the profit Ri, initializes the profit R, and further initializes the variable t (S144). ). Next, the CPU 132 determines whether or not the variable i has reached the predetermined value N (S146). Then, when the CPU 132 determines that the predetermined value N has not been reached (S146: NO), the variable i is incremented (S148).

On the other hand, when the CPU 132 determines that the predetermined value N is reached (S146: YES), the CPU 132 updates the variables w (1) to w (p) and the coefficient wT that define the policy π by the policy gradient method (S150). ). In FIG. 13, the variables w (1) to w (p) and the coefficient wT that define the policy π are collectively described as the parameter θ.

Here, a measure in which T sets of states s, action a, and reward r until the variable t becomes 0 to T-1 is a trajectory ht, and the probability pθ (ht) is defined by the parameter θ. Let the probability of becoming a trajectory ht according to π be pθ (ht). Here, the integrated value of “pθ (ht) · Rt” by the trajectory ht is the expected value (expected return J) of the profit R (ht), and the parameter θ is updated so as to maximize this. This can be realized by setting the update amount of each component of the parameter θ to be an amount proportional to the value obtained by partially differentiating the expected return J by the same component.

Here, the probability pθ (ht) is pθ (ht) = p (s0) · p (s1 | s0, a0) · π (when the states s0, s1, ... sT and the actions a0, a1, ... aT are used. a0 | s0) ・ p (s2 | s1, a1) ・ π (a1 | s1)… p (sT | sT-1, aT-1) ・ π (aT-1 | sT-1)
Will be. However, the initial probability p (s0) is the probability of becoming the state s0, and the transition probability p (st + 1 | st, at) is the probability of transitioning from the state st to the state st + 1 at the time of the state st and the action at.

Therefore, the partial differential of the expected return J is given by the following equation (c1).

ここで、確率ｐθ（ｈｔ）については、知ることができないことから、上記の式（ｃ１）における積分を、複数（ここでは、所定値Ｎ個）のトラジェクトリｈｔによる平均値に置き換える。

Here, since the probability pθ (ht) cannot be known, the integral in the above equation (c1) is replaced with an average value of a plurality of (here, N predetermined values) trajectories.

As a result, the partial differential of each component of the parameter θ of the expected profit J is the partial differential coefficient “t = 0 to T-1” of the corresponding component of the logarithmic parameter θ of the policy π (at | ht (i)). The product of the sum and the profit Ri in the above is added to the predetermined value N of the profit Ri and divided by the predetermined value N.

The value obtained by multiplying the partial differential coefficient of the expected return J by each component of the parameter θ by the learning rate α is defined as the update amount of the corresponding component of the parameter θ.
In the processing of S140 to S150, among the learning programs 74b stored in the ROM 134, the states s0, s1, ..., The actions a0, a1, ..., And the reward r are input, and the updated parameter θ is output. It is realized by executing the execution command of the mapping.

When the processing of S150 is completed, the CPU 132 initializes the variables i and the profits R1 to RN (S152). Then, the CPU 132 operates the communication device 137 to transmit the updated parameter θ (S154).

When the processes of S142, S148, and S154 are completed, the CPU 132 temporarily ends the series of processes shown in FIG. 13 (b).
On the other hand, as shown in FIG. 13A, the CPU 72 determines whether or not there is update data (S124). Then, when the CPU 72 determines that there is update data (S124: YES), the CPU 72 receives the update data (S126). Then, the CPU 72 rewrites the coefficients w (1) to w (p) and wT constituting the relational regulation data DR used in the processing of S32b to the data received by the processing of S126 (S128). When the processing of S128 is completed or when a negative determination is made in the processing of S124, the CPU 72 temporarily ends the series of processing shown in FIG. 13A.

By the way, the relational regulation data DR mounted on the control device 70 at the time of shipment of the vehicle VC1 is a trained model generated by executing the processing according to the processing of FIGS. 12 and 13 in the system shown in FIG. is there.

As described above, according to the present embodiment, the calculation load of the CPU 72 can be reduced by executing the update process of the related regulation data DR by the data analysis center 130.
According to the present embodiment described above, the following effects can be further obtained.

(4) By using a function approximator for the relational regulation data DR, even if the state or action is a continuous variable, it can be easily handled.
(5) The base injection amount Qbse was included in the action a. In the transient case, if the base injection amount Qbse as the open loop operation amount is set to a value proportional to the filling efficiency η, the detected value Afu may deviate from the rich side upper limit value AfR and the lean side upper limit value AfL. .. Then, when the method of setting the base injection amount Qbse is performed by repeating trial and error by a skilled person, the number of man-hours required for the skilled person increases. On the other hand, in the present embodiment, the man-hours required for the expert can be effectively reduced by learning the base injection amount Qbse, which is the injection amount of the open loop control at the time of transient, by reinforcement learning.

<Correspondence>
The correspondence between the matters in the above-described embodiment and the matters described in the above-mentioned "means for solving the problem" column is as follows. In the following, the correspondence is shown for each number of the solution means described in the column of "Means for solving the problem".

[1, 2] The execution device corresponds to the CPU 112 and the ROM 114, and the storage device corresponds to the storage device 116. The acquisition process corresponds to the processes of S30 and S70 of FIG. The operation process corresponds to the process of S34. The reward calculation process corresponds to the processes of S82 to S86. The update process corresponds to the processes of S88 to S94. The first data corresponds to the relational regulation data DR, and the second data corresponds to the steady-state map data DMs. [3] The manipulated variable as an action variable corresponds to the throttle opening command value TA * and the retard angle amount aop. [4] The control mapping data corresponds to the transient map data DMt. [5, 6] The execution device corresponds to the CPU 72 and ROM 74 of FIG. 8, and the storage device corresponds to the storage device 76 of FIG. The first operation process corresponds to the process of S34 in FIG. The second operation process corresponds to the process of S28 in FIG. [7-9] The first executing device corresponds to the CPU 72 and the ROM 74, and the second executing device corresponds to the CPU 132 and the ROM 134. The acquisition process corresponds to the processes of S30a and S120, and the update process corresponds to the process of S150.

<Other Embodiments>
In addition, this embodiment can be implemented by changing as follows. The present embodiment and the following modified examples can be implemented in combination with each other within a technically consistent range.

・ "About behavior variables"
In the above embodiment, the throttle opening command value TA * is exemplified as a variable related to the opening degree of the throttle valve as an action variable, but the present invention is not limited to this. For example, the responsiveness of the throttle opening command value TA * to the accelerator operation amount PA is expressed by the wasted time and the second-order lag filter, and the wasted time and the two variables that define the second-order lag filter are a total of three. The variable may be a variable related to the opening degree of the throttle valve. However, in that case, it is desirable that the state variable is a change amount per unit time of the accelerator operation amount PA instead of the time series data of the accelerator operation amount PA.

In the above embodiment, the retard angle amount aop is exemplified as a variable related to the ignition timing as an action variable, but the present invention is not limited to this. For example, it may be the ignition timing itself to be corrected by KCS.

In the above embodiment, as the action variables, a set of variables related to the throttle valve opening degree and the ignition timing, a set of variables related to the throttle valve opening degree, a variable related to the ignition timing, and a set of variables related to the injection amount are illustrated. Not limited to this. For example, regarding the variables related to the throttle valve opening degree, the ignition timing variable, and the injection amount, only the throttle valve opening degree variable and the fuel injection amount are adopted, or the ignition timing variable and the fuel injection amount are adopted. Only the quantity may be adopted. Furthermore, for those three, only one of them may be adopted as the behavioral variable.

Further, as described in the column of "About the internal combustion engine", when the internal combustion engine 10 includes the supercharger and the valve characteristic variable device of the intake valve, the valve characteristic of the intake valve may be included in the action variable. In this case, when the above conditions (a) and (b) are satisfied, the operation of the valve characteristics for enhancing the responsiveness at the time of transition can be learned by reinforcement learning by giving a reward as satisfying a predetermined criterion.

Further, as described in the column of "About the internal combustion engine", in the case of a compression ignition type internal combustion engine, a variable related to the injection amount is used instead of the variable related to the opening degree of the throttle valve, and a variable related to the ignition timing is used instead of the variable related to the ignition timing. You can use the variables related to. In addition to the variables related to the injection timing, the variables related to the number of injections in one combustion cycle and the end timing of one of the two fuel injections adjacent in time series for one cylinder in one combustion cycle and the other It is desirable to add a variable for the time interval between the start timing.

Further, for example, when the transmission 50 is a stepped transmission, the current value of the solenoid valve for adjusting the engaged state of the clutch by flood control may be used as an action variable.
Further, for example, when a hybrid vehicle, an electric vehicle, or a fuel cell vehicle is adopted as the vehicle as described in the column of "About the vehicle" below, the torque and output of the rotary electric machine may be used as an action variable.

・ "About the condition"
In the above embodiment, the time-series data of the accelerator operation amount PA is set to the data consisting of six values sampled at equal intervals, but the present invention is not limited to this. The data may be data consisting of two or more sampling values at different sampling timings, and at this time, it is more desirable that the data consists of three or more sampling values or the sampling intervals are evenly spaced.

In the above embodiment, the time-series data of the rotation speed NE is set to the data consisting of six values sampled at equal intervals, but the present invention is not limited to this. The data may be data consisting of two or more sampling values at different sampling timings, and at this time, it is more desirable that the data consists of three or more sampling values or the sampling intervals are evenly spaced.

In the above embodiment, the time-series data of the filling efficiency η is set to the data consisting of six values sampled at equal intervals, but the present invention is not limited to this. The data may be data consisting of two or more sampling values at different sampling timings, and at this time, it is more desirable that the data consists of three or more sampling values or the sampling intervals are evenly spaced.

Further, for example, as described in the column of "About the action variable", when the current value of the solenoid valve is used as the action variable, the state includes the rotation speed of the input shaft 52 of the transmission, the rotation speed of the output shaft 54, and the solenoid valve. The oil pressure adjusted by may be included. Further, for example, as described in the column of "behavior variable", when the torque or output of the rotary electric machine is used as the action variable, the charge rate or temperature of the battery may be included in the state.

・ "About the first data"
In the above embodiment, the action value function Q is a table-type function, but the present invention is not limited to this. For example, a function approximator may be used.

In the example shown in FIG. 10, the relational regulation data DR implemented at the time of shipment of the vehicle VC1 is set to the data for which reinforcement learning is performed by the system shown in FIG. 3, but the present invention is not limited to this. For example, for the internal combustion engine 10 in which the relational regulation data DR includes the action value function Q and the map data has already been adapted by the conventional method, the action value function is set so that the action corresponding to each state is a greedy action. By obtaining Q from the map data, the relational regulation data DR to be implemented may be generated.

・ "About dimensionality reduction of table format data"
The method for reducing the dimension of the table format data is not limited to the one illustrated in the above embodiment. For example, since it is rare that the accelerator operation amount PA becomes the maximum value, the action value function Q is not defined for the state where the accelerator operation amount PA becomes the specified amount or more, and the accelerator operation amount PA becomes the specified amount or more. In this case, the throttle opening command value TA * and the like may be adapted separately. Further, for example, the dimension may be reduced by excluding the value at which the throttle opening degree command value TA * is equal to or more than the specified value from the values that can be taken.

Further, for example, the value of the independent variable of the action value function Q may be limited to a small number in the process of S32a until an affirmative decision is made in the process of S76 of FIG. In that case, when affirmative judgment is made in the processing of S76, the value near the action a in which the value of the action value function Q becomes large is added to the value that can be taken by the independent variable of the action value function Q, and S30, S32a, S34, The processes of S70 to S72 may be repeated.

However, dimensionality reduction is not essential. For example, in the third embodiment, if reinforcement learning based on the data of a plurality of vehicles is performed and the computing power of the CPU 72 and the storage capacity of the storage device 76 are sufficient, the dimension is reduced before the vehicle is shipped. Although the action value function is learned for only a part, all actions may be executed by exploration after shipment. As a result, it is possible to increase the number of actions that can be taken as a search and find more appropriate actions in view of the fact that sufficient learning data can be secured after shipping as compared with before shipping.

・ "Regarding certain conditions"
The predetermined condition as the execution condition of the reinforcement learning or the usage condition of the control data learned by the reinforcement learning is the predetermined period after the absolute value of the change amount ΔPA of the accelerator operation amount PA becomes the predetermined value ΔPA or more. Not exclusively. For example, it may be a predetermined period after the absolute value of the change amount of the intake air amount Ga per unit time becomes a predetermined amount or more.

However, the predetermined condition is not limited to the condition that the state is in a transient state. For example, a predetermined condition may be that it is during the fail-safe process in which an abnormality occurs in a predetermined electronic component, or that it is not during the fail-safe process.

・ "About updated mapping"
In the processing of S88 to S94, the ε-soft policy on-type Monte Carlo method was illustrated, but the present invention is not limited to this. For example, the policy-off type Monte Carlo method may be used. However, it is not limited to the Monte Carlo method, for example, the policy-off type TD method is used, the policy-on type TD method such as the SARSA method is used, and the eligibility tracing method is used as the policy-on type learning, for example. You may use it.

As described in the column of "About the first data", when the function approximator of the action value function Q is used, the updated map is, for example, the action value function Q according to the parameter defining the action value function Q. It may be configured including a mapping that outputs the update amount of the same parameter based on the partial differential.

In the processing of S150, the revenue Ri is set as a simple average during the time T, but the present invention is not limited to this. For example, it may be a sum using a value discounted by a predetermined discount rate γ as much as the past reward r. This corresponds to exponential moving average processing.

In the processing of S150, instead of the profit Ri, an appropriate baseline function independent of the parameter θ may be subtracted from the profit Ri. Specifically, it is desirable that the baseline function is, for example, a function that minimizes the variance of the partial differential according to the parameter of the expected return J.

Further, only one of the action value function Q and the policy π is not limited to the one directly updated by the reward r. For example, the action value function Q and the policy π may be updated, respectively, as in the actor-critic method. Further, in the actor-critic method, the value function V may be updated instead of the action value function Q, for example.

The “ε” that defines the policy π is not limited to a fixed value, and may be changed according to a predetermined rule according to the degree of learning progress. Further, the learning rate α is not limited to a fixed value, and may be changed according to a predetermined rule according to the degree of progress of learning.

・ "About reward calculation processing"
In the process of FIG. 6, the reward is given according to whether or not the logical product of the condition (a) and the condition (b) is true, but the reward is not limited to this. For example, a process of giving a reward depending on whether or not the condition (a) is satisfied and a process of giving a reward depending on whether or not the condition (b) is satisfied may be executed. Further, for example, with respect to two processes, that is, a process of giving a reward depending on whether or not the condition (a) is satisfied and a process of giving a reward depending on whether or not the condition (b) is satisfied, among them. Only one of the processes may be executed.

In the process of FIG. 13, a reward is given according to whether or not the logical product of the conditions (a) to (c) is true, but the reward is not limited to this. For example, a process of giving a reward depending on whether or not the condition (a) is satisfied, a process of giving a reward depending on whether or not the condition (b) is satisfied, and a process of giving a reward depending on whether or not the condition (c) is satisfied. You may perform a rewarding process. Further, for example, a process of giving a reward depending on whether or not the condition (a) is satisfied, a process of giving a reward depending on whether or not the condition (b) is satisfied, and whether or not the condition (c) is satisfied. With respect to the three processes of rewarding according to the situation, only one of them may be executed.

Further, for example, instead of giving the same reward uniformly when the condition (a) is satisfied, a process may be performed in which a larger reward is given when the absolute value of the difference between the torque Trq and the torque command value Trq * is smaller than when it is large. .. Further, for example, instead of giving the same reward uniformly when the condition (a) is not satisfied, it is also possible to give a smaller reward when the absolute value of the difference between the torque Trq and the torque command value Trq * is large than when it is small. Good.

Further, for example, instead of giving the same reward uniformly when the condition (a) is satisfied, the process may be such that the magnitude of the reward is variable according to the magnitude of the acceleration Gx. Further, for example, instead of giving the same reward uniformly when the condition (a) is not satisfied, the process may be such that the magnitude of the reward is variable according to the magnitude of the acceleration Gx.

Further, for example, instead of giving the same reward uniformly when the condition (c) is satisfied, the process may be such that the size of the reward is variable according to the size of the detected value Afu. Further, for example, instead of giving the same reward uniformly when the condition (c) is not satisfied, the process may be such that the size of the reward is variable according to the size of the detected value Afu.

The drivability standard is not limited to the above-mentioned one, and may be set according to whether or not noise and vibration intensity satisfy the standard, for example. However, the present invention is not limited to this, for example, whether or not the above acceleration meets the standard, whether or not the followability of the torque Trq meets the standard, whether or not the noise meets the standard, and whether or not the vibration intensity meets the standard. It may be any one or more of the ones.

The reward calculation process is not limited to the one in which the reward r is given according to whether or not the criteria for drivability are satisfied and whether or not the exhaust characteristics satisfy the criteria. For example, if the fuel consumption rate meets the criteria, it may be a process that rewards a larger amount than if it does not meet the criteria. In addition, when the drivability standard is met, a larger reward is given than when it is not met, when the fuel consumption rate meets the standard, a larger reward is given than when it is not met, and when the exhaust characteristics meet the standard. May include any two or three of the three processes with the process that rewards greater than if not satisfied.

Further, for example, when the current value of the solenoid valve of the transmission 50 is used as the action variable as described in the column of "behavior", for example, in the reward calculation process, the following three processes (a) to (c) are performed. At least one of these processes may be included.

(A) This is a process in which when the time required for switching the gear ratio by the transmission is within a predetermined time, a larger reward is given than when the predetermined time is exceeded.
(B) This is a process in which when the absolute value of the change speed of the rotation speed of the input shaft 52 of the transmission is equal to or less than the input side predetermined value, a larger reward is given than when the input side predetermined value is exceeded.

(C) This is a process in which when the absolute value of the change speed of the rotation speed of the output shaft 54 of the transmission is equal to or less than the predetermined value on the output side, a larger reward is given than when the predetermined value on the output side is exceeded.
In addition, for example, as described in the column of "About behavior variables", when the torque or output of a rotating electric machine is used as a behavior variable, a process of giving a larger reward than when the battery charge rate is within a predetermined range. Alternatively, it may include a process that rewards a greater amount than if the battery temperature is within a predetermined range.

・ "How to generate control data for vehicles"
In the process of S32a of FIG. 5, the action is determined based on the action value function Q, but the action is not limited to this, and all possible actions may be selected with equal probability.

・ "About operation processing"
For example, as described in the column of "About the first data", when the action value function Q is used as a function approximation device, a set of discrete values for actions that are independent variables of the function of the table type in the above embodiment. For all, the action a that maximizes the action value function Q may be selected by inputting the action value function Q together with the state s.

・ "About control mapping data"
Map as control mapping data that inputs the state of the vehicle and outputs the value of the action variable that maximizes the expected return by associating the state of the vehicle with the value of the action variable that maximizes the expected return on a one-to-one basis. Not limited to data. For example, it may be a function approximator. This can be realized, for example, by using the average value μ after learning as the control mapping data when the policy gradient method illustrated in FIG. 13 is used.

・ "About vehicle control system"
In the example shown in FIG. 11, the process of determining the action based on the policy π (the process of S32b) is executed on the vehicle side, but the present invention is not limited to this. For example, the data acquired by the processing of the vehicle VC1 to S30a may be transmitted, the action a may be determined using the data transmitted at the data analysis center 130, and the determined action may be transmitted to the vehicle VC1.

The vehicle control system is not limited to the one composed of the control device 70 and the data analysis center 130. For example, the user's mobile terminal may be used instead of the data analysis center 130. Further, the vehicle control system may be configured by the control device 70, the data analysis center 130, and the mobile terminal. This can be realized, for example, by executing the processing of S32b by a mobile terminal.

・ "About the execution device"
The execution device is not limited to the one provided with the CPU 72 (112, 132) and the ROM 74 (114, 134) to execute software processing. For example, a dedicated hardware circuit (for example, ASIC or the like) that performs hardware processing on at least a part of what has been software-processed in the above embodiment may be provided. That is, the executing device may have any of the following configurations (a) to (c). (A) A processing device that executes all of the above processing according to a program and a program storage device such as a ROM that stores the program are provided. (B) A processing device and a program storage device that execute a part of the above processing according to a program, and a dedicated hardware circuit that executes the remaining processing are provided. (C) A dedicated hardware circuit for executing all of the above processes is provided. Here, there may be a plurality of software execution devices including a processing device and a program storage device, and a plurality of dedicated hardware circuits.

・ "About storage device"
In the above embodiment, the storage device that stores the related regulation data DR and the storage device (ROM74, 114, 134) that stores the learning programs 74b, 114a and the control program 74a are different storage devices. Not limited to.

・ "About internal combustion engine"
The internal combustion engine is not limited to one having a port injection valve for injecting fuel into the intake passage 12 as a fuel injection valve, and may be provided with an in-cylinder injection valve for directly injecting fuel into the combustion chamber 24. For example, it may include both a port injection valve and an in-cylinder injection valve.

The internal combustion engine may be provided with a valve characteristic variable device for an intake valve or a supercharger.
The internal combustion engine is not limited to the spark ignition type internal combustion engine, and may be, for example, a compression ignition type internal combustion engine that uses light oil or the like as fuel.

・ "About the vehicle"
The vehicle is not limited to a vehicle in which the thrust generator is only an internal combustion engine, and may be, for example, a so-called hybrid vehicle including an internal combustion engine and a rotary electric machine. Further, for example, as the thrust generator, there may be a so-called electric vehicle or a fuel cell vehicle equipped with a rotating electric machine without providing an internal combustion engine.

10 ... Internal engine, 12 ... Intake passage, 14 ... Throttle valve, 16 ... Fuel injection valve, 18 ... Intake valve, 20 ... Cylinder, 22 ... Piston, 24 ... Combustion chamber, 26 ... Ignition device, 28 ... Crank shaft, 30 ... exhaust valve, 32 ... exhaust passage, 34 ... catalyst, 40 ... torque converter, 42 ... lockup clutch, 50 ... transmission, 52 ... input shaft, 54 ... output shaft, 60 ... drive wheel, 70 ... control device, 72 ... CPU, 74 ... ROM, 74a ... Control program, 74b ... Learning program, 76 ... Storage device, 77 ... Communication device, 78 ... Peripheral circuit, 79 ... Local network, 80 ... Airflow meter, 82 ... Throttle sensor, 84 ... Crank Angle sensor, 88 ... Accelerator pedal, 90 ... Accelerator sensor, 92 ... Acceleration sensor, 100 ... Dynamometer, 102 ... Sensor group, 110 ... Generator, 112 ... CPU, 114 ... ROM, 114a ... Learning program, 116 ... Storage device , 118 ... peripheral circuit, 119 ... local network, 120 ... network, 130 ... data analysis center, 132 ... CPU, 134 ... ROM, 136 ... storage device, 137 ... communication device, 138 ... peripheral circuit, 139 ... local network.

Claims (9)

When the state of the vehicle satisfies a predetermined condition, the first data defining the relationship between the state of the vehicle and the action variable indicating the behavior related to the operation of the electronic device in the vehicle is stored in the storage device.
The acquisition process for acquiring the detection value of the sensor that detects the state of the vehicle, and
The operation process for operating the electronic device and
When the predetermined condition is satisfied, a reward calculation process for giving a larger reward than when the characteristic of the vehicle satisfies the standard based on the detected value acquired by the acquisition process, and
When the predetermined condition is satisfied, the state of the vehicle based on the detected value acquired by the acquisition process, the value of the action variable used for the operation of the electronic device, and the reward corresponding to the operation are predetermined. An update process for updating the first data as an input to the updated map, and
To the execution device
The updated map outputs the first data updated so as to increase the expected profit for the reward when the electronic device is operated according to the first data.
When the state of the vehicle does not satisfy a predetermined condition, the control data for the vehicle is used as the second data by matching the relationship between the state of the vehicle and the action variable without depending on the reward calculation process and the update process. Generation method. The method for generating vehicle control data according to claim 1, wherein the predetermined condition is a condition for transient driving. The vehicle is equipped with an internal combustion engine.
The electronic device includes an operation unit of the internal combustion engine.
The method for generating vehicle control data according to claim 1 or 2, wherein the first data defines a relationship between the state of the vehicle and the operation amount of the operation unit of the internal combustion engine as the action variable. For control that inputs the state of the vehicle and outputs the value of the action variable that maximizes the expected profit by associating the state of the vehicle with the value of the action variable that maximizes the expected profit on a one-to-one basis. The method for generating vehicle control data according to any one of claims 1 to 3, wherein the executing device executes a process of generating mapping data based on the first data updated by the update process. The storage device and the execution device according to any one of claims 1 to 3 are provided.
The second data is stored in the storage device.
The operation process is the first operation process of operating the electronic device according to the value of the action variable according to the state of the vehicle acquired by the acquisition process based on the first data when the predetermined condition is satisfied. , A vehicle including a second operation process of operating the electronic device according to the value of the action variable according to the state of the vehicle acquired by the acquisition process based on the second data when the predetermined condition is not satisfied. Control device for. Equipped with an execution device and a storage device
The storage device stores first data and second data that define the relationship between the state of the vehicle and the behavior variable, which is a variable related to the operation of the electronic device in the vehicle.
The executing device is
The acquisition process for acquiring the detection value of the sensor that detects the state of the vehicle, and
The operation process for operating the electronic device and
When the state of the vehicle satisfies a predetermined condition, a reward calculation process of giving a larger reward than when the characteristic of the vehicle satisfies the standard based on the detected value acquired by the acquisition process, and
When the state of the vehicle satisfies a predetermined condition, the state of the vehicle based on the detection value acquired by the acquisition process, the value of the action variable used for the operation of the electronic device, and the said corresponding to the operation. An update process in which the reward is input to a predetermined update map and the first data is updated, and
And
The updated map outputs the first data updated so as to increase the expected profit for the reward when the electronic device is operated according to the first data.
The operation process is the first operation process of operating the electronic device according to the value of the action variable according to the state of the vehicle acquired by the acquisition process based on the first data when the predetermined condition is satisfied. , A vehicle including a second operation process of operating the electronic device according to the value of the action variable according to the state of the vehicle acquired by the acquisition process based on the second data when the predetermined condition is not satisfied. Control device for. The executing device and the storage device according to claim 5 or 6 are provided.
The execution device includes a first execution device mounted on the vehicle and a second execution device different from the in-vehicle device.
The first execution device executes at least the acquisition process and the operation process,
The second execution device is a vehicle control system that executes at least the update process. A vehicle control device including the first execution device according to claim 7. A vehicle learning device including the second execution device according to claim 7.

Images