JP7207289B2 - Vehicle control device, vehicle control system, vehicle learning device, and vehicle learning method - Google Patents

Description

The present invention relates to a vehicle control device, a vehicle control system, a vehicle learning device, and a vehicle learning method.

For example, Patent Literature 1 listed below describes a control device that operates a throttle valve as an operation unit of an internal combustion engine mounted on a vehicle based on a value obtained by filtering an operation amount of an accelerator pedal.

JP 2016-6327 A

By the way, the above filter is required to set the amount of operation of the throttle valve of the internal combustion engine mounted on the vehicle to an appropriate amount of operation according to the amount of operation of the accelerator pedal. requires a lot of man-hours. As described above, conventionally, skilled workers spend a lot of man-hours to adjust the operation amounts of electronic devices in the vehicle according to the state of the vehicle.

Means for solving the above problems and their effects will be described below.
1. an execution device and a storage device, wherein the storage device stores relationship defining data defining a relationship between a vehicle state and an action variable, which is a variable relating to operation of an electronic device mounted on the vehicle; The execution device performs a state acquisition process for acquiring the state of the vehicle each time based on the detected value of the sensor each time, and the behavior variable determined by the state of the vehicle acquired by the state acquisition process and the relationship defining data. operation processing for operating the electronic device based on the value; and remuneration calculation processing for giving a larger reward when the characteristics of the vehicle meet criteria than when they do not, based on the vehicle status acquired by the status acquisition process. and inputting the state of the vehicle obtained by the state obtaining process, the value of the action variable used in the operation of the electronic device, and the reward corresponding to the operation to a predetermined update map, An update process for updating the relationship defining data, a deterioration variable acquisition process for acquiring a deterioration variable that is a variable indicating the degree of deterioration of the vehicle, and a comparison between when the degree of deterioration of the vehicle is equal to or greater than a predetermined value and when the degree of deterioration is less than a predetermined value. and executing a change process for expanding a range in which a value other than a value that maximizes an expected profit for the reward among the values of the action variable is adopted by the operation process, and The update map is a vehicle control device that outputs the relationship-defining data updated so as to increase the expected profit when the electronic device is operated according to the relationship-defining data.

In the above configuration, by calculating the reward associated with the operation of the electronic device, it is possible to grasp what kind of reward is obtained by the operation. Then, based on the reward, the relationship defining data is updated by an update map according to reinforcement learning, so that the relationship between the vehicle state and the behavioral variables can be set to an appropriate relationship during vehicle travel. Therefore, it is possible to reduce the number of man-hours required for an expert when setting the relationship between the vehicle state and the behavioral variable to an appropriate relationship for running the vehicle.

By the way, when the behavior that maximizes the expected profit determined by reinforcement learning converges, it is better to always select the behavior that maximizes the expected profit than to continue searching blindly. It is suitable for controlling. However, as vehicles deteriorate, their expected return-maximizing behavior may change. Therefore, in the above configuration, when the degree of deterioration is greater than or equal to a predetermined value, compared to when the degree of deterioration is less than a predetermined value, by expanding the range in which a value other than the value that maximizes the expected profit is adopted, the deteriorated vehicle can Appropriate behavioral variable values can be found by reinforcement learning.

It is desirable that the state of the vehicle acquired by the state acquisition process includes at least those that change in a shorter time than the value of the deterioration variable.
2. 2. The vehicle control device according to 1 above, wherein the change processing includes processing for expanding a range in which a value other than the value that maximizes the expected profit is adopted from zero to a range greater than zero.

In the above configuration, when the degree of deterioration is less than a predetermined value, the range in which values other than the value that maximizes the expected profit is adopted is set to zero, thereby suppressing unnecessary searches.
3. The deterioration variable is also a variable that subdivides the case where the degree of deterioration is less than a predetermined amount by an amount having a positive correlation with the passage of time. The range in which a value other than the value to be maximized is adopted is a third range from the first range through the second range, and the first range is the second range and the third range. 3, and the third range is a range larger than the second range, and the process of changing the range to an expanding side is performed when the degree of deterioration of the vehicle exceeds a predetermined level. 3. The vehicle control device according to the above 2, wherein the range in which the value other than the value maximizing the expected profit is adopted is changed from the second range to the third range, if the be.

In the above configuration, even if the vehicle deteriorates, it is assumed that the value of the behavioral variable that maximizes the expected profit does not change significantly from the value before deterioration. is also small. This makes it possible to search efficiently because it is more likely that the search will be limited to the values of the behavioral variables that are likely to maximize the expected profit for the degraded vehicle.

4. 4. The execution device according to any one of 1 to 3 above and the storage device, wherein the execution device includes a first execution device mounted on the vehicle and a second execution device separate from the on-vehicle device. , wherein the first execution device executes at least the state acquisition process and the operation process, and the second execution device executes at least the update process.

In the above configuration, by executing the update processing by the second execution unit, the calculation load of the first execution unit can be reduced compared to the case where the update processing is executed by the first execution unit.
Note that the fact that the second execution device is a device different from the in-vehicle device means that the second execution device is not the in-vehicle device.

5. 5. A vehicle control device comprising the first execution device according to 4 above.
6. 5. A vehicle learning device comprising the second execution device according to 4 above.
7. A vehicle learning method for causing a computer to execute the state acquisition process, the operation process, the remuneration calculation process, the update process, the deterioration variable acquisition process, and the change process according to any one of 1 to 3 above.

According to the above method, the same effects as in 1 above can be obtained.

The figure which shows the control apparatus concerning 1st Embodiment, and its drive system. FIG. 4 is a flowchart showing the procedure of processing executed by the control device according to the embodiment; FIG. FIG. 4 is a flowchart showing the procedure of processing executed by the control device according to the embodiment; FIG. FIG. 4 is a flowchart showing a detailed procedure of part of the processing executed by the control device according to the embodiment; FIG. FIG. 4 is a flowchart showing the procedure of processing executed by the control device according to the embodiment; FIG. The figure which shows the structure of the control system for vehicles concerning 2nd Embodiment. 4(a) and 4(b) are flowcharts showing procedures of processing executed by the vehicle control system;

<First Embodiment>
A first embodiment of a vehicle control device will be described below with reference to the drawings.
FIG. 1 shows the configuration of a driving system and a control device of a vehicle VC1 according to this embodiment.

As shown in FIG. 1, an intake passage 12 of an internal combustion engine 10 is provided with a throttle valve 14 and a fuel injection valve 16 in this order from the upstream side. The injected fuel flows into the combustion chamber 24 defined by the cylinder 20 and the piston 22 as the intake valve 18 is opened. In the combustion chamber 24, the mixture of fuel and air is combusted by the spark discharge of the ignition device 26, and the energy generated by the combustion is converted into rotational energy of the crankshaft 28 via the piston 22. be. The combusted air-fuel mixture is discharged as exhaust gas to the exhaust passage 32 as the exhaust valve 30 is opened. The exhaust passage 32 is provided with a catalyst 34 as an aftertreatment device for purifying exhaust gas.

An input shaft 52 of a transmission 50 can be mechanically connected to the crankshaft 28 via a torque converter 40 having a lockup clutch 42 . The transmission 50 is a device that varies a gear ratio, which is the ratio between the rotation speed of the input shaft 52 and the rotation speed of the output shaft 54 . A drive wheel 60 is mechanically connected to the output shaft 54 .

The control device 70 treats the internal combustion engine 10 as a controlled object, and controls the operation units of the internal combustion engine 10 such as the throttle valve 14, the fuel injection valve 16, and the ignition device 26 in order to control the torque, the exhaust component ratio, etc., which are the control amounts. Manipulate. The control device 70 controls the torque converter 40 and operates the lockup clutch 42 to control the engagement state of the lockup clutch 42 . Further, the control device 70 controls the transmission device 50 and operates the transmission device 50 so as to control the gear ratio as its control amount. 1, operation signals MS1 to MS5 for the throttle valve 14, the fuel injection valve 16, the ignition device 26, the lockup clutch 42, and the transmission device 50 are shown.

The control device 70 controls the control amount based on the intake air amount Ga detected by the air flow meter 80, the opening degree of the throttle valve 14 (throttle opening degree TA) detected by the throttle sensor 82, the crank angle sensor 84 The output signal Scr of is referred to. The control device 70 also refers to the depression amount of the accelerator pedal 86 (accelerator operation amount PA) detected by the accelerator sensor 88 and the longitudinal acceleration Gx of the vehicle VC1 detected by the acceleration sensor 90 . The control device 70 also refers to the output signal Sv of the wheel rotation sensor 92 that detects the rotation angle of the driving wheels 60 .

The control device 70 includes a CPU 72 , a ROM 74 , an electrically rewritable nonvolatile memory (storage device 76 ), and a peripheral circuit 78 , which can communicate with each other via a local network 79 . Here, the peripheral circuit 78 includes a circuit that generates a clock signal that defines internal operations, a power supply circuit, a reset circuit, and the like.

The ROM 74 stores a control program 74a and a learning program 74b. On the other hand, the storage device 76 stores relationship defining data DR that defines the relationship between the accelerator operation amount PA, the command value of the throttle opening TA (throttle opening command value TA*), and the retardation amount aop of the ignition device 26. remembered. Here, the retardation amount aop is an amount of retardation with respect to a predetermined reference ignition timing, and the reference ignition timing is the timing on the retard side between the MBT ignition timing and the knock limit point. The MBT ignition timing is the ignition timing at which maximum torque is obtained (maximum torque ignition timing). The knock limit point is the ignition timing advance limit value at which knocking can be kept within an allowable level under the best assumed conditions when using high octane fuel with a high knock limit. The storage device 76 also stores torque output mapping data DT. The torque output map defined by the torque output map data DT is a map that inputs the rotation speed NE of the crankshaft 28, the charging efficiency η, and the ignition timing, and outputs the torque Trq.

FIG. 2 shows the procedure of processing executed by the control device 70 according to this embodiment. The processing shown in FIG. 2 is realized by the CPU 72 repeatedly executing the learning program 74b stored in the ROM 74, for example, at predetermined intervals. In the following description, the step number of each process is indicated by a number prefixed with "S".

In the series of processes shown in FIG. 2, the CPU 72 first acquires the travel distance RL (S10). Here, the running distance RL is calculated by the CPU 72 based on the output signal Sv of the wheel rotation sensor 92 .

Next, the CPU 72 determines whether or not the running distance RL is equal to or less than the convergence determination value RLthL (S12). When the CPU 72 determines that the convergence determination value is equal to or less than the convergence determination value RthL, the CPU 72 substitutes "1" for the deterioration flag Fd (S14). On the other hand, when making a negative determination in the process of S12, the CPU 72 determines whether or not the travel distance RL is greater than the convergence determination value RLthL and less than the deterioration threshold RLthH (S16). If the CPU 72 determines that it is greater than the convergence determination value RLthL and less than the deterioration threshold RLthH (S16: YES), it substitutes "2" for the deterioration flag Fd (S18). When the CPU 72 makes a negative determination in the process of S16, it substitutes "3" for the deterioration flag Fd (S20).

When completing the processes of S14, S18, and S20, the CPU 72 once ends the series of processes shown in FIG.
FIG. 3 shows the procedure of processing executed by the control device 70 according to this embodiment. The processing shown in FIG. 3 is realized by the CPU 72 repeatedly executing the control program 74a and the learning program 74b stored in the ROM 74, for example, at predetermined intervals.

In the series of processes shown in FIG. 3, the CPU 72 first generates time-series data consisting of six sampling values "PA(1), PA(2), . (S30). Here, each sampled value constituting the time-series data is sampled at different timings. In the present embodiment, time-series data is composed of six sampling values that are time-sequentially adjacent to each other when sampled at a constant sampling cycle.

Next, the CPU 72 sets an action a consisting of the throttle opening command value TA* and the retardation amount aop corresponding to the state s obtained by the process of S30, in accordance with the policy π determined by the relationship defining data DR (S32).

In this embodiment, the relationship defining data DR is data that defines the action-value function Q and the policy π. In this embodiment, the action-value function Q is a tabular function that indicates the value of the expected profit according to the eight-dimensional independent variables of the state s and the action a. In addition, when the state s is given, the policy π preferentially selects the action a (greedy action) that maximizes the action value function Q in the state s given the independent variable, while preferentially selecting the action a (greedy action). A rule is established to select the other action a with a probability of .

Specifically, the number of values that the independent variables of the action-value function Q according to this embodiment can take is reduced by human knowledge, etc. It is. That is, for example, one of the two adjacent sampling values of the accelerator operation amount PA time-series data is the minimum value and the other is the maximum value. cannot occur, the action-value function Q is not defined. In this embodiment, the possible values of the state s defining the action-value function Q are limited to 10 4 or less, more preferably 10 3 or less, by dimensionality reduction based on human knowledge or the like.

Next, the CPU 72 outputs an operation signal MS1 to the throttle valve 14 to operate the throttle opening degree TA based on the set throttle opening degree command value TA* and the retardation amount aop, and outputs an operation signal MS3 to the ignition device 26. is output to operate the ignition timing (S34). Here, in the present embodiment, since feedback control of the throttle opening degree TA to the throttle opening degree command value TA* is exemplified, even if the throttle opening degree command value TA* is the same value, the operation signal MS1 can be different signals. Further, when the well-known knocking control (KCS) or the like is performed, the ignition timing is set to a value obtained by retarding the reference ignition timing by the retardation amount aop and feedback corrected by the KCS. Here, the reference ignition timing is variably set by the CPU 72 according to the rotation speed NE of the crankshaft 28 and the charging efficiency η. Note that the rotation speed NE is calculated by the CPU 72 based on the output signal Scr of the crank angle sensor 84 . Further, the charging efficiency η is calculated by the CPU 72 based on the rotation speed NE and the intake air amount Ga.

Next, the CPU 72 acquires the torque Trq of the internal combustion engine 10, the torque command value Trq* for the internal combustion engine 10, and the acceleration Gx (S36). Here, the CPU 112 calculates the torque Trq by inputting the rotational speed NE, the charging efficiency η and the ignition timing into the torque output map. Further, the CPU 72 sets the torque command value Trq* according to the accelerator operation amount PA.

Next, the CPU 72 determines whether or not the transient flag F is "1" (S38). When the transient flag F is "1", it indicates that the operation is in a transient operation, and when it is "0", it indicates that the operation is not in a transient operation. When the CPU 72 determines that the transient flag F is "0" (S38: NO), the CPU 72 determines whether or not the absolute value of the change amount ΔPA of the accelerator operation amount PA per unit time is equal to or greater than a predetermined amount ΔPAth. (S40). Here, the amount of change ΔPA may be, for example, the difference between the latest accelerator operation amount PA at the execution timing of the processing of S40 and the accelerator operation amount PA a unit time before the same timing.

If the CPU 72 determines that it is equal to or greater than the predetermined amount ΔPAth (S40: YES), it substitutes "1" for the transient flag F (S42).
On the other hand, when determining that the transient flag F is "1" (S38: YES), the CPU 72 determines whether or not a predetermined period has elapsed from the execution timing of the process of S42 (S44). Here, the predetermined period is defined as a period until the absolute value of the change amount ΔPA of the accelerator operation amount PA per unit time is equal to or less than a specified amount smaller than the predetermined amount ΔPAth and continues for a predetermined period of time. When determining that the predetermined period has passed (S44: YES), the CPU 72 substitutes "0" for the transient flag F (S46).

When the processes of S42 and S46 are completed, the CPU 72 determines whether or not the deterioration flag Fd is "1" or "3" (S48), assuming that one episode has ended. Then, when determining that the deterioration flag Fd is "1" or "3" (S48: YES), the CPU 72 updates the action value function Q by reinforcement learning (S50).

FIG. 4 shows details of the processing of S50.
In the series of processes shown in FIG. 4, the CPU 72 collects time-series data consisting of a set of three sampled values of torque command value Trq*, torque Trq and acceleration Gx during the most recently completed episode, state s and action a. and the time-series data of (S60). Here, the most recent episode is the period during which the transient flag F was continuously "0" when the process of S60 is performed following the process of S42, and the period of S60 following the process of S46. When the process is performed, it is the period during which the transient flag F is continuously at "1".

In FIG. 4, different numbers in parentheses indicate variable values at different sampling timings. For example, torque command value Trq*(1) and torque command value Trq*(2) have different sampling timings. Also, the time-series data of action a belonging to the latest episode is defined as action set Aj, and the time-series data of state s belonging to the same episode is defined as state set Sj.

Next, the CPU 72 sets the condition (a) that the absolute value of the difference between any torque Trq belonging to the most recent episode and the torque command value Trq* is equal to or less than the prescribed amount ΔTrq, and that the acceleration Gx is equal to or greater than the lower limit value GxL. is equal to or less than the upper limit value GxH (S62).

Here, the CPU 72 variably sets the prescribed amount ΔTrq depending on the change amount ΔPA per unit time of the accelerator operation amount PA at the start of the episode. That is, when the absolute value of the change amount ΔPA is large, the CPU 72 regards it as an episode relating to the transient state, and sets the prescribed amount ΔTrq to a larger value than in the case of the steady state.

Further, the CPU 72 variably sets the lower limit value GxL depending on the change amount ΔPA of the accelerator operation amount PA at the start of the episode. That is, the CPU 72 sets the lower limit value GxL to a larger value when the episode is related to the transient time and the amount of change ΔPA is positive compared to the case of the episode related to the steady state. In addition, when the episode is related to the transient time and the amount of change ΔPA is negative, the CPU 72 sets the lower limit value GxL to a smaller value than in the case of the episode related to the steady state.

Further, the CPU 72 variably sets the upper limit value GxH according to the change amount ΔPA per unit time of the accelerator operation amount PA at the start of the episode. That is, the CPU 72 sets the upper limit value GxH to a larger value when the episode is related to the transient time and the amount of change ΔPA is positive compared to the case of the episode related to the steady state. In addition, when the episode is related to the transient time and the amount of change ΔPA is negative, the CPU 72 sets the upper limit value GxH to a smaller value than in the case of the episode related to the steady state.

If the CPU 72 determines that the logical product is true (S62: YES), it substitutes "10" for the reward r (S64). −10” is substituted (S66). The processing of S62 to S66 is processing to give a larger reward when the criteria for drivability are met than when they are not met. When the processes of S64 and S66 are completed, the CPU 72 updates the relationship defining data DR stored in the storage device 76 shown in FIG. In this embodiment, the ε-soft policy on-type Monte Carlo method is used.

That is, the CPU 72 adds the reward r to each of the revenues R (Sj, Aj) determined by the set of each state and the corresponding action read out in the process of S60 (S68). Here, "R(Sj, Aj)" is a generalized description of the revenue R in which one of the elements of the state set Sj is the state and one of the elements of the action set Aj is the action. Next, each of the returns R (Sj, Aj) determined by the set of each state and the corresponding action read by the processing of S60 is averaged and substituted into the corresponding action value function Q (Sj, Aj) ( S70). Here, the averaging may be a process of dividing the profit R calculated by the process of S68 by the number obtained by adding a predetermined number to the number of times the process of S68 is performed. The initial value of the profit R may be the initial value of the corresponding action value function Q.

Next, the CPU 72 determines the throttle opening degree command value TA* and the retardation amount aop when the corresponding action value function Q(Sj, A) reaches the maximum value for each of the states read by the process of S60. The action that is a pair is substituted for the action Aj* (S72). Here, "A" indicates any possible action. Note that the action Aj* has a different value depending on the type of state read out by the process of S60, but here, the notation is simplified and the same symbol is used.

Next, the CPU 72 updates the corresponding policy π(Aj|Sj) for each of the states read by the process of S60 (S74). That is, if the total number of actions is "|A|", the selection probability of the action Aj* selected in S72 is "(1-ε)+ε/|A|". Also, the selection probabilities of “|A|-1” actions other than action Aj* are assumed to be “ε/|A|”. Since the processing of S74 is processing based on the action-value function Q updated by the processing of S70, the relationship defining data DR that defines the relationship between the state s and the action a is changed so as to increase the revenue R. will be updated to

When the process of S74 is completed, the CPU 72 once terminates the series of processes shown in FIG.
Returning to FIG. 3, the CPU 72 temporarily terminates the series of processes shown in FIG. 3 when the process of S50 is completed or when a negative determination is made in the processes of S40, S44, and S48. The processes of S30 to S46 are realized by the CPU 72 executing the control program 74a, and the processes of S48 and S50 are realized by the CPU 72 executing the learning program 74b. Further, the relational regulation data DR at the time of shipment of the vehicle VC1 is pre-learned data by executing the same processing as the processing shown in FIG. 3 while simulating the running of the vehicle on a test bench.

FIG. 5 shows the procedure of processing executed by the control device 70 . The processing shown in FIG. 5 is realized by the CPU 72 repeatedly executing a learning program 74b stored in the ROM 74, for example, at predetermined intervals.

In the series of processes shown in FIG. 5, the CPU 72 first determines whether or not the deterioration flag Fd is switched from "1" to "2" (S80). When the CPU 72 determines that the deterioration flag Fd has changed from "1" to "2" (S80: YES), the CPU 72 sets the probability of selecting an action other than the greedy action to zero. is substituted with zero (S82).

On the other hand, when the CPU 72 makes a negative determination in the process of S80, it determines whether or not it is time to switch the deterioration flag Fd from "2" to "3" (S83). If the CPU 72 determines that the deterioration flag Fd has switched from "2" to "3" (S83: YES), it selects one of the states s defined as independent variables of the action value function Q. (S84). Next, the CPU 72 substitutes, into the greedy action ag, the action a that maximizes the value of the action value function Q whose independent variable is the state s selected by the process of S84 (S86). Then, the CPU 72 limits the set As of actions a that can be taken in the state s selected in the process of S84 to actions a whose difference from the greedy action ag is equal to or less than a predetermined value δ (S88). Here, the action whose absolute value of the difference from the greedy action ag is equal to or less than the predetermined value δ means that the absolute value of the difference from the throttle opening command value TA* corresponding to the greedy action ag and the retardation amount aop is It is assumed that the throttle opening degree command value TA* and the retardation amount aop are each equal to or smaller than a value obtained by multiplying the range of possible values by a predetermined value δ. That is, if the range of values that the throttle opening degree command value TA* can take is TAmax and "0<δ<1", then the absolute difference between the throttle opening degree command value TA* and the value indicated by the greedy behavior is The value is limited to "δ·TAmax" or less. Further, the CPU 72 sets the absolute value of the difference between the retardation amount aop and the value indicated by the greedy behavior to be less than or equal to "δ·aopmax", where aopmax is the range of possible values of the retardation amount aop. Restrict.

When the process of S88 is completed, the CPU 72 determines whether or not all states s defined as independent variables of the action value function Q have been selected by the process of S84 (S90). When determining that there is a state s that has not been selected yet (S90: NO), the CPU 72 returns to the process of S84.

On the other hand, the CPU 72 temporarily ends the series of processes shown in FIG. .

Here, the action and effect of this embodiment will be described.
The CPU 72 acquires time-series data of the accelerator operation amount PA in accordance with the user's operation of the accelerator pedal 86, and sets the action a consisting of the throttle opening command value TA* and the retardation amount aop according to the policy π. Here, the CPU 72 basically selects the action a that maximizes the expected profit based on the action value function Q defined in the relationship defining data DR. However, the CPU 72 searches for an action a that maximizes the expected profit by selecting an action other than the action a that maximizes the expected profit with a predetermined probability "ε−ε|A|". Accordingly, the relationship defining data DR can be updated by reinforcement learning as the user drives the vehicle VC1. Therefore, the throttle opening degree command value TA* and the retardation amount aop corresponding to the accelerator operation amount PA can be set to appropriate values for traveling of the vehicle VC1 without excessively increasing the number of man-hours required by an expert.

In this manner, after the vehicle VC1 is shipped, the relationship defining data DR is updated as the vehicle VC1 travels until the travel distance RL exceeds the convergence determination value RLthL. Then, when the convergence judgment value RLthL or more is reached, it is assumed that the relationship defining data DR has converged to the optimum value for the running of the vehicle VC1, and by setting "ε" to zero, the policy π is set to take only greedy actions. change.

Here, for example, even if the throttle opening degree TA is the same, if deposits accumulate on the throttle valve 14 and the intake passage 12 due to deterioration of the vehicle VC1, the passage cross-sectional area of the intake passage 12 becomes small. Therefore, the intake air amount Ga becomes smaller. Therefore, when the travel distance RL greatly exceeds the convergence determination value RLthL and the deterioration of the vehicle VC1 progresses, the throttle opening degree command value TA* that maximizes the expected profit according to the time-series data of the accelerator operation amount PA is set. , the travel distance RL may deviate from the value defined by the relationship defining data DR at the time when the travel distance RL reaches the convergence determination value RLthL.

Therefore, when the travel distance RL of the vehicle VC1 is equal to or greater than the deterioration threshold RLthH, the CPU 72 according to the present embodiment sets the probability “ε−ε|A|” of adopting an action other than the greedy action to be greater than zero. However, the CPU 72 imposes a limit that selectable behaviors other than the greedy behavior are less than when the running distance RL is equal to or less than the convergence determination value RLthL. Specifically, the absolute value of the difference from the throttle opening command value TA* indicated by the greedy action becomes equal to or less than ".delta.TAmax" and the absolute value of the difference from the retardation amount aop indicated by the greedy action. is limited to "δ·aopmax" or less. This is because even if the greedy behavior changes from that before deterioration due to deterioration, it is considered that the amount of change from the greedy behavior before deterioration is not so large. By limiting the search range in this way, it is possible to suppress unnecessary searches using actions that cannot be greedy actions.

According to this embodiment described above, the following effects can be obtained.
(1) When the travel distance RL exceeds the convergence judgment value RLthL and is less than the deterioration threshold RLthH, by prohibiting the search, it is possible to avoid the continuation of the unnecessary search and the taking of any action other than the optimum action. .

(2) The travel distance RL is used as a deterioration variable indicating the degree of deterioration of the vehicle VC1, and the search range is changed accordingly. This makes it possible to easily quantify the presence or absence of deterioration of the vehicle VC1.

<Second embodiment>
The second embodiment will be described below with reference to the drawings, focusing on differences from the first embodiment.

In this embodiment, the update of the relationship defining data DR is performed outside the vehicle VC1.
FIG. 6 shows the configuration of a control system that executes reinforcement learning in this embodiment. 6, members corresponding to members shown in FIG. 1 are denoted by the same reference numerals for convenience.

The ROM 74 in the control device 70 in the vehicle VC1 shown in FIG. 6 stores the control program 74a, but does not store the learning program 74b. The control device 70 also includes a communication device 77 . The communication device 77 is a device for communicating with the data analysis center 110 via the network 100 outside the vehicle VC1.

The data analysis center 110 analyzes data transmitted from a plurality of vehicles VC1, VC2, . The data analysis center 110 includes a CPU 112, a ROM 114, an electrically rewritable nonvolatile memory (storage device 116), a peripheral circuit 118, and a communication device 117, which can communicate with each other via a local network 119. be. The ROM 114 stores a learning program 114a, and the storage device 116 stores relationship defining data DR.

FIG. 7 shows a processing procedure of reinforcement learning according to this embodiment. The processing shown in FIG. 7(a) is implemented by the CPU 72 executing a control program 74a stored in the ROM 74 shown in FIG. The processing shown in FIG. 7B is realized by the CPU 112 executing the learning program 114a stored in the ROM 114. FIG. In FIG. 7, the same step numbers are attached to the processes corresponding to the processes shown in FIG. 2 for the sake of convenience. The processing shown in FIG. 7 will be described below along the time series of reinforcement learning.

In the series of processes shown in FIG. 7(a), the CPU 72 executes the processes of S30 to S48, and if the determination in S48 is affirmative, the CPU 72 operates the communication device 77 to update the relationship defining data DR. Send necessary data (S100). Here, the data to be transmitted are the state s set in the process of S30 within the predetermined period, the action a set in the process of S32 within the predetermined period, and the data acquired in the process of S36 within the predetermined period. includes torque command value Trq*, torque Trq, and acceleration Gx.

On the other hand, as shown in FIG. 7B, the CPU 112 receives the transmitted data (S110), and updates the relationship defining data DR based on the received data (S50). Then, the CPU 112 determines whether or not the number of updates of the relationship defining data DR is equal to or greater than a predetermined number of times (S112). The relationship defining data DR is transmitted to the vehicle VC1 that transmitted the data received by the process of S110 (S114). Note that the CPU 112 once ends the series of processes shown in FIG. 7B when completing the process of S114 or when making a negative determination in the process of S112.

On the other hand, as shown in FIG. 7A, the CPU 72 determines whether or not there is update data (S102). Receive (S104). Then, the CPU rewrites the relationship defining data DR used in the process of S32 with the received relationship defining data DR (S106). It should be noted that the CPU 72 temporarily ends the series of processes shown in FIG.

As described above, according to the present embodiment, the processing for updating the relationship defining data DR is performed outside the vehicle VC1, so that the calculation load of the control device 70 can be reduced. Furthermore, for example, in the process of S110, if data from a plurality of vehicles VC1 and VC2 are received and the process of S50 is performed, the number of data used for learning can be easily increased.

<Correspondence relationship>
Correspondence relationships between the items in the above embodiment and the items described in the "Means for Solving the Problems" column are as follows. Below, the corresponding relationship is shown for each number of the means for solving the problem described in the column of "means for solving the problem". [1] The execution device corresponds to the CPU 72 and the ROM 74 , and the storage device corresponds to the storage device 76 . The state acquisition process corresponds to the processes of S30 and S36, the operation process corresponds to the process of S34, the reward calculation process corresponds to the processes of S62 to S66, and the update process corresponds to the processes of S68 to S74. do. The degradation variable acquisition process corresponds to the process of S10, and the change process corresponds to the process of FIG. The updated mapping corresponds to the mapping specified by the instruction for executing the processing of S68-S74 in the learning program 74b. The case where the degree of deterioration is equal to or greater than a predetermined value corresponds to the case where the travel distance RL is equal to or greater than the deterioration threshold RLthH. [2] Corresponds to the processing of S88. [3] The first range corresponds to all actions considered in the process of S86. The second range corresponds to zero. The third range corresponds to the range set by the process of S88. [4-6] The first execution unit corresponds to the CPU 72 and the ROM 74, and the second execution unit corresponds to the CPU 112 and the ROM 114. [7] A computer corresponds to the CPU 72 in FIG. 1 and the CPUs 72 and 112 in FIG.

<Other embodiments>
In addition, this embodiment can be changed and implemented as follows. This embodiment and the following modifications can be implemented in combination with each other within a technically consistent range.

"About deterioration variables"
- The deterioration variable is not limited to the travel distance RL. For example, if an air-fuel ratio sensor is provided, it may be the amount of variation in the detected value. Further, for example, when open-loop control and feedback control are performed on the air-fuel ratio, the magnitude of the feedback correction amount of the fuel injection amount by the fuel injection valve 16 may be used.

When the deterioration variable is a variable that subdivides the degree of deterioration less than a predetermined value by an amount having a positive correlation with the passage of time, for example, the travel distance RL for determining whether the degree of deterioration is a predetermined value or more and a variable indicating whether or not the value of the action-value function Q has converged.

"About change processing"
In the above-described embodiment, only greedy behavior is used and search is prohibited when the travel distance RL is greater than the convergence determination value RLthL and less than the deterioration threshold RLthH, but the present invention is not limited to this. For example, a search within a range narrower than the range defined by the processing of S88 may be permitted.

- For example, as described in the column "Computer", the change process may be configured by prohibiting the search at the time of product shipment and starting the search when the degree of deterioration exceeds a predetermined level.

"About Behavioral Variables"
In the above embodiment, the throttle opening command value TA* was exemplified as a variable relating to the opening of the throttle valve as an action variable, but the present invention is not limited to this. For example, the responsiveness of the throttle opening command value TA* to the accelerator operation amount PA is expressed by a dead time and a secondary lag filter, and the dead time and the two variables that define the secondary lag filter. The variable may be a throttle valve opening variable. However, in that case, it is desirable that the state variable is the amount of change in the accelerator operation amount PA per unit time instead of the time-series data of the accelerator operation amount PA.

In the above embodiment, the retardation amount aop was exemplified as a variable related to ignition timing as a behavioral variable, but the present invention is not limited to this. For example, it may be the ignition timing itself to be corrected by the KCS.

In the above-described embodiment, the behavior variables are the throttle valve opening degree variable and the ignition timing variable, but are not limited to these. For example, the fuel injection amount may be used in addition to the variables relating to the degree of opening of the throttle valve and the variables relating to ignition timing. Further, with respect to these three, only the variable relating to the opening of the throttle valve and the fuel injection amount may be employed as the action variables, or only the variable relating to the ignition timing and the fuel injection amount may be employed. Furthermore, regarding those three, only one of them may be adopted as a behavioral variable.

・As described in the section "Internal Combustion Engine", in the case of a compression ignition type internal combustion engine, a variable related to the injection amount is used instead of a variable related to the opening of the throttle valve, and a variable related to the injection timing is used instead of a variable related to the ignition timing. Variables should be used. In addition to the variables related to the injection timing, the variables related to the number of injections in one combustion cycle, the end timing of one of the two fuel injections adjacent in time series for one cylinder in one combustion cycle, and the other It is desirable to add a variable for the time interval between start timings.

For example, if the transmission 50 is a stepped transmission, the action variable may be a current value of a solenoid valve for adjusting the engagement state of the clutch by hydraulic pressure.
- For example, when a hybrid vehicle, an electric vehicle, or a fuel cell vehicle is adopted as the vehicle as described in the section "Vehicles" below, the torque and output of the rotating electric machine may be used as the action variable. Further, for example, when an in-vehicle air conditioner is equipped with a compressor that is rotated by rotational power of a crankshaft of an internal combustion engine, the load torque of the compressor may be included in the action variable. Moreover, when an electric vehicle-mounted air conditioner is provided, the power consumption of the air conditioner may be included in the behavior variable.

"About the state"
- In the above-described embodiment, the time-series data of the accelerator operation amount PA is data consisting of six values sampled at equal intervals, but the present invention is not limited to this. Data consisting of two or more sampling values at sampling timings different from each other may be used. In this case, data consisting of three or more sampling values or data with equal sampling intervals are more desirable.

・The state variable related to the accelerator operation amount is not limited to the time-series data of the accelerator operation amount PA. may

・For example, when the current value of a solenoid valve is used as an action variable as described in the column "About action variables", the state depends on the rotation speed of the input shaft 52 of the transmission, the rotation speed of the output shaft 54, and the solenoid valve. Include the hydraulic pressure to be adjusted. For example, as described in the column "Behavioral variables", when the torque and output of the rotary electric machine are used as behavioral variables, the state may include the charging rate and temperature of the battery. For example, as described in the column "Behavioral Variables", when the load torque of the compressor and the power consumption of the air conditioner are included in the behavior, the temperature in the passenger compartment may be included in the state.

"About dimensionality reduction of tabular data"
- The dimension reduction method for data in table format is not limited to those exemplified in the above embodiments. For example, since it is rare for the accelerator operation amount PA to reach its maximum value, the action value function Q is not defined for a state where the accelerator operation amount PA is equal to or greater than a specified amount, and the accelerator operation amount PA is equal to or greater than the specified amount. The throttle opening degree command value TA* and the like in this case may be adapted separately. Further, for example, dimensionality reduction may be performed by excluding, from the values that can be taken by actions, those in which the throttle opening degree command value TA* is equal to or greater than a specified value.

"Regarding related regulation data"
- In the above embodiment, the action value function Q is a function in a table format, but it is not limited to this. For example, a function approximator may be used.

・For example, instead of using the action-value function Q, the policy π is expressed by a function approximator with the state s and the action a as independent variables and the probability of taking the action a as the dependent variable, and the parameters that define the function approximator may be updated according to the reward r.

"About operation processing"
・For example, as described in the column "Regarding relational data", when the action value function is a function approximator, a set of discrete values for actions that are independent variables of the table-type function in the above embodiment For all, the action a that maximizes the action-value function Q should be selected by inputting it into the action-value function Q together with the state s.

・For example, as described in the column "Regarding relational data", if policy π is a function approximator with state s and action a as independent variables and the probability of taking action a as a dependent variable, then policy π Action a may be selected based on the indicated probability.

"On update maps"
・In the processing of S68 to S74, the ε-soft policy-on type Monte Carlo method was exemplified, but the present invention is not limited to this. For example, it may be based on off-policy Monte Carlo method. However, it is not limited to the Monte Carlo method. You may use it.

・For example, as described in the column "Regarding relational data", when the policy π is expressed using a function approximator and directly updated based on the reward r, the update mapping is performed using the policy gradient method, etc. should be configured.

- Either one of the action-value function Q and the policy π is not limited to being directly updated with the reward r. For example, the action-value function Q and policy π may be updated, respectively, like the actor-critic method. In addition, in the actor-critic method, the value function V may be updated instead of the action value function Q, for example.

"About Reward Calculation Process"
- In the above-described embodiment, a reward is given according to whether or not the logical product of condition (a) and condition (b) is true, but the present invention is not limited to this. For example, a process of giving a reward depending on whether condition (a) is satisfied and a process of giving a reward depending on whether condition (b) is satisfied may be executed.

- For example, instead of uniformly giving the same reward when the condition (a) is satisfied, processing may be such that when the absolute value of the difference between the torque Trq and the torque command value Trq* is small, a larger reward is given than when the absolute value is large. Alternatively, for example, instead of uniformly giving the same reward when the condition (a) is not satisfied, if the absolute value of the difference between the torque Trq and the torque command value Trq* is large, a smaller reward may be given than if it is small. good.

- For example, instead of uniformly giving the same reward when the condition (a) is satisfied, the processing may be such that the magnitude of the reward is variable according to the magnitude of the acceleration Gx. Further, for example, instead of giving the same reward uniformly when the condition (a) is not satisfied, processing may be performed in which the magnitude of the reward is made variable according to the magnitude of the acceleration Gx.

- The remuneration calculation process is not limited to giving the remuneration r depending on whether or not the standard regarding drivability is satisfied. For example, it may be a process of giving a higher reward when the energy utilization efficiency meets the standard than when it does not, or a process of giving a higher reward when the exhaust characteristic meets the standard than when it does not. A process of giving a higher reward when the criteria for drivability are met than when the criteria are not met, a process of giving a higher reward when the energy utilization efficiency meets the criteria than when the criteria are not met, and a process of giving a greater reward when the criteria are met and the exhaust characteristics satisfy the criteria. It may include two or three of the three treatments with a treatment that rewards more than if it does not meet.

・For example, as described in the section "Behavioral variables", when the current value of the solenoid valve of the transmission 50 is used as the behavioral variable, for example, the following three processes (a) to (c) are performed in the reward calculation process. At least one of the processes may be included.

(a) This is a process of giving a larger reward when the time required for switching the gear ratio by the transmission is within a predetermined period of time than when the predetermined period of time is exceeded.
(b) A process of giving a larger reward when the absolute value of the rate of change of the rotation speed of the input shaft 52 of the transmission is equal to or less than the input side predetermined value than when it exceeds the input side predetermined value.

(c) A process of giving a larger reward when the absolute value of the rate of change of the rotational speed of the output shaft 54 of the transmission is equal to or less than the predetermined value on the output side than when it exceeds the predetermined value on the output side.
・For example, as described in the section "Behavioral variables", if the torque or output of the rotating electric machine is used as the behavioral variable, a process of giving a larger reward when the charging rate of the battery is within a predetermined range than when it is not, or , may include providing a greater reward if the temperature of the battery is within a predetermined range than if it is not. For example, as described in the column "Behavioral variables", when the load torque of the compressor and the power consumption of the air conditioner are included in the behavioral variables, the temperature inside the passenger compartment is larger than when it is not within a predetermined range. You may add the process which gives a reward.

"About Vehicle Control Systems"
- In the process of FIG. 7, all the processes of S50 were performed in the data analysis center 110, but it is not restricted to this. For example, the processing of S62 to S66 of the processing of S50 may be executed on the vehicle VC1 side, and the processing of S100 may be partially modified to transmit the calculation result of the reward r.

- The vehicle control system is not limited to the one configured by the control device 70 and the data analysis center 110 . For example, instead of the data analysis center 110, a mobile terminal owned by the user may be used, and the control device 70 and the mobile terminal may constitute the vehicle control system. Alternatively, for example, it may be configured by the control device 70, the mobile terminal, and the data analysis center 110. FIG. For example, in FIG. 7, this can be realized by the mobile terminal executing the process of S32.

"About Execution Units"
- The execution device is not limited to one that includes the CPU 72 (112) and the ROM 74 (114) and executes software processing. For example, a dedicated hardware circuit such as an ASIC may be provided to perform hardware processing at least part of what is software processed in the above embodiments. That is, the execution device may have any one of the following configurations (a) to (c). (a) A processing device that executes all of the above processes according to a program, and a program storage device such as a ROM that stores the program. (b) A processing device and a program storage device for executing part of the above processing according to a program, and a dedicated hardware circuit for executing the remaining processing. (c) provide dedicated hardware circuitry to perform all of the above processing; Here, there may be a plurality of software execution devices provided with a processing device and a program storage device, or a plurality of dedicated hardware circuits.

"About Computers"
- The computer is not limited to the CPU 72 in FIG. 1 or the CPUs 72 and 112 in FIG. For example, it may be a computer for generating relationship defining data DR before shipment of vehicle VC1 and CPU 72 mounted on vehicle VC1. In that case, the search may be prohibited at the time of shipment, and the search may be permitted when the travel distance RL is equal to or greater than the deterioration threshold RLthH. It is preferable that the range of possible values of the behavioral variables be smaller than the search range in the computer-executed reinforcement learning for generating the relationship defining data DR. Incidentally, in the process of generating the relevant regulation data DR before shipment of the vehicle, the vehicle does not exist, and the running of the vehicle is simulated by running the internal combustion engine 10 or the like on the test bench, thereby simulating the vehicle state. , and the state of the vehicle, which is generated in a simulated manner from sensor detection values, may be grasped and used for reinforcement learning. In that case, the simulated vehicle state is regarded as the vehicle state based on the sensor values.

"About storage devices"
In the above-described embodiment, the storage device storing the relationship defining data DR and the storage device (ROM 74) storing the learning program 74b and the control program 74a are separate storage devices, but the present invention is not limited to this.

"About Internal Combustion Engines"
The internal combustion engine is not limited to the one provided with a port injection valve for injecting fuel into the intake passage 12 as a fuel injection valve, but may be provided with an in-cylinder injection valve for directly injecting fuel into the combustion chamber 24. Further, for example, both a port injection valve and an in-cylinder injection valve may be provided.

- The internal combustion engine is not limited to a spark ignition internal combustion engine, and may be a compression ignition internal combustion engine using light oil or the like as fuel.
"About vehicle"
- The vehicle is not limited to a vehicle having only an internal combustion engine as a thrust generating device, and may be a so-called hybrid vehicle having an internal combustion engine and a rotating electric machine, for example. Further, for example, the thrust generator may be a so-called electric vehicle or a fuel cell vehicle equipped with a rotary electric machine without an internal combustion engine.

DESCRIPTION OF SYMBOLS 10... Internal combustion engine 12... Intake passage 14... Throttle valve 16... Fuel injection valve 18... Intake valve 20... Cylinder 22... Piston 24... Combustion chamber 26... Ignition device 28... Crankshaft 70... Control device 110... Data analysis center

Claims (7)

having an execution unit and a storage unit,
The storage device stores relationship definition data that defines a relationship between a vehicle state and an action variable that is a variable related to operation of an electronic device mounted on the vehicle,
The execution device is
A state acquisition process for acquiring the state of the vehicle each time based on the detected value of the sensor each time;
an operation process of operating the electronic device based on the value of the behavior variable determined by the vehicle state acquired by the state acquisition process and the relationship defining data;
a remuneration calculation process that provides a larger remuneration when the characteristics of the vehicle meet criteria than when they do not, based on the vehicle status acquired by the status acquisition process;
The state of the vehicle obtained by the state obtaining process, the value of the action variable used in the operation of the electronic device, and the reward corresponding to the operation are input to a predetermined update map, and the relationship an update process for updating the stipulated data;
a deterioration variable acquisition process for acquiring a deterioration variable, which is a variable indicating the degree of deterioration of the vehicle;
When the degree of deterioration of the vehicle is greater than or equal to a predetermined value, a value other than the value that maximizes the expected profit for the reward is adopted by the operation processing from among the values of the behavior variables as compared with the case of less than the predetermined degree. change processing to change the range to the side to expand,
and run
The vehicle control device according to claim 1, wherein the updated map outputs the relational data updated so as to increase the expected profit when the electronic device is operated according to the relational data. 2. The vehicle control device according to claim 1, wherein said change processing includes processing for expanding a range in which a value other than the value that maximizes said expected profit is adopted from zero to a range greater than zero. The deterioration variable is also a variable that subdivides the case where the degree of deterioration is less than a predetermined value by an amount that has a positive correlation with the passage of time,
In the change processing, as the time elapses, the range in which the value other than the value that maximizes the expected profit is adopted is changed from the first range to the second range to the third range. can be,
The first range is a range larger than the second range and the third range,
The third range is a range larger than the second range,
In the processing for expanding the range, when the degree of deterioration of the vehicle is a predetermined value or more, the range in which a value other than the value that maximizes the expected profit is adopted is changed from the second range to the third range. 3. The vehicle control device according to claim 2, wherein the processing is a process of changing to an expanding side of the range. comprising the execution device and the storage device according to any one of claims 1 to 3,
The execution device includes a first execution device mounted on the vehicle and a second execution device separate from the on-vehicle device,
The first execution device executes at least the state acquisition process and the operation process,
The vehicle control system, wherein the second execution device executes at least the update process. A vehicle control device comprising the first execution device according to claim 4 . A vehicle learning device comprising the second execution device according to claim 4 . A vehicle learning method for causing a computer to execute the state acquisition process, the operation process, the reward calculation process, the update process, the deterioration variable acquisition process, and the change process according to any one of claims 1 to 3.

Images