JP2021067257A - Vehicle control device, vehicle control system, and vehicle learning device - Google Patents

Abstract

To provide a vehicle control device that allows reduction in the number of man-hours necessary for an expert in setting a relationship between a state of a vehicle and an action variable.SOLUTION: Until one episode ends, a CPU 72 sets a throttle opening degree command value and a retardation amount of ignition timing on the basis of time-series data of an accelerator operation amount PA, operates a throttle valve and an ignition device according to them, and acquires a torque, a torque command value and an acceleration at that time. If the episode ends, the CPU 72 updates relationship definition data DR by rewarding according to whether the torque and the acceleration satisfy criteria. If function recovery treatment of a component in a vehicle is executed, the CPU returns the relationship definition data DR to a value a predetermined period before the execution of the function recovery treatment.SELECTED DRAWING: Figure 1

Description

The present invention relates to a vehicle control device, a vehicle control system, and a vehicle learning device.

For example, Patent Document 1 below describes a control device that operates a throttle valve as an operation unit of an internal combustion engine mounted on a vehicle based on a value obtained by filtering the operation amount of the accelerator pedal.

Japanese Unexamined Patent Publication No. 2016-6327

By the way, in the above filter, it is necessary to set the operation amount of the throttle valve of the internal combustion engine mounted on the vehicle to an appropriate operation amount according to the operation amount of the accelerator pedal. However, it is necessary to spend a lot of man-hours. As described above, conventionally, a skilled person has spent a lot of man-hours to adapt the operation amount of the electronic device in the vehicle according to the state of the vehicle.

Hereinafter, means for solving the above problems and their actions and effects will be described.
1. 1. The storage device includes an execution device and a storage device, and the storage device stores related regulation data that defines a relationship between a vehicle state and an action variable that is a variable related to the operation of an electronic device in the vehicle. Operates the electronic device based on the acquisition process of acquiring the detection value of the sensor that detects the state of the vehicle, and the value of the action variable determined by the detection value acquired by the acquisition process and the relational regulation data. Based on the operation process to be performed, the detection value acquired by the acquisition process, the reward calculation process of giving a larger reward than the case where the characteristics of the vehicle satisfy the criteria, and the said acquisition process. The state of the vehicle based on the detected value, the value of the action variable used for the operation of the electronic device, and the reward corresponding to the operation are input to a predetermined update mapping, and the relevant regulation data is updated. The update process, the detection process for detecting that the function recovery action of the parts in the vehicle that affect the state of the vehicle caused by the operation by the operation process has been performed, and the function recovery action by the detection process. When it is detected that the data has been performed, the related regulation data used for the operation process is switched to the post-treatment data which is the data before the time when the function recovery action is performed, and the data for post-treatment. The updated mapping is a vehicle control device that outputs the relevant regulation data updated so as to increase the expected profit for the reward when the electronic device is operated according to the relevant regulation data.

In the above configuration, by calculating the reward associated with the operation of the electronic device, it is possible to grasp what kind of reward can be obtained by the operation. Then, based on the reward, the relationship between the state of the vehicle and the behavior variable can be set to an appropriate relationship in the running of the vehicle by updating the relational regulation data by the update mapping according to the reinforcement learning. Therefore, it is possible to reduce the man-hours required for the expert when setting the relationship between the state of the vehicle and the behavior variable.

By the way, when the parts in the vehicle are deteriorated, the related regulation data is updated to appropriate data when the deteriorated parts are used by reinforcement learning. Therefore, if functional recovery measures are taken after that, the relevant regulation data may not be appropriate data for increasing the expected return. Therefore, in the above configuration, when the function recovery treatment is performed, the related regulation data used for the operation processing is switched to the post-treatment data by the switching process, thereby suppressing the decrease in the expected profit due to the function recovery treatment. It becomes possible.

2. The switching process is the vehicle control device according to 1 above, which includes a process of using the relevant regulation data updated by the update process as the vehicle travels as post-treatment data.

In the above configuration, the relational regulation data updated by the update process with the running of the vehicle is compared with the initial data before the running of the vehicle is started, and the value of the action variable more appropriate according to the state of the vehicle. Is likely to be specified. Therefore, in the above configuration, the relevant regulation data updated by the update process as the vehicle travels is used as the post-treatment data, and the initial data before the vehicle starts running is used as the post-treatment data. Compared with the case, it is possible to set a more appropriate value of the action variable according to the state of the vehicle after the function recovery treatment based on the relational regulation data after the switching process.

3. 3. The execution device is updated by the update process until the predetermined condition is satisfied, and after the update process is satisfied, the update process is performed separately from the related regulation data which is updated by the update process. The past data maintenance process is executed in which the relevant regulation data that avoids the update due to the above is stored in the storage device, and the switching process is stored in the storage device as the post-treatment data by the past data maintenance process. The vehicle control device according to 1 or 2 above, which includes a process using the relevant regulation data in which the state is maintained.

In the above configuration, by executing the past data maintenance process, it is possible to maintain the relational regulation data having a small degree of update according to the deterioration of the parts in the vehicle as the post-treatment process, and eventually, this is treated. It can be used as later data.

4. When the execution device detects that the function recovery treatment has been performed by the detection process, the execution device transmits a signal requesting the post-treatment data, and the result of the post-treatment data request process and the post-treatment data request process. , The post-treatment data reception process for receiving the transmitted post-treatment data, and the switching process switches the related regulation data used for the operation process to the received post-treatment data. The vehicle control device according to 1 or 2 above, which includes processing.

In the above configuration, by executing the post-treatment data request processing and the post-treatment data reception processing, the post-treatment data can be obtained even when the control device does not hold the post-treatment data. it can.

5. The execution device including the execution device and the storage device according to the above 1 or 2, the execution device includes a first execution device mounted on the vehicle and a second execution device different from the in-vehicle device, and the first execution device. The 2 execution device executes at least the post-treatment data transmission process for transmitting the post-treatment data when it is detected by the detection process that the function recovery treatment has been performed, and the first execution device performs the acquisition. It is a vehicle control system that at least executes post-treatment data reception processing for receiving data transmitted by processing, the operation processing, and the post-treatment data transmission processing.

In the above configuration, the second execution device other than the in-vehicle device executes the post-treatment data transmission process, so that even if the first execution device does not hold the post-treatment data, the post-treatment data is used. Data can be obtained. The fact that the second executing device is a device different from the in-vehicle device means that the second executing device is not an in-vehicle device.

6. The first executing device executes the post-treatment data request processing for transmitting the detection process and the signal requesting the post-treatment data when it is detected that the function recovery treatment has been performed by the detection processing. 5 is a vehicle control system.

In the above configuration, the post-treatment data request processing and the post-treatment data reception processing are executed to obtain the post-treatment data even when the first execution device does not hold the post-treatment data. be able to.

7. The post-treatment data transmission process includes the process of transmitting the relevant regulation data as the post-treatment data in a vehicle in which the degree of deterioration of the parts is lower than that of the vehicle to which the function recovery treatment has been performed. It is a control system for vehicles.

In the above configuration, the relevant regulation data in a vehicle having a low degree of deterioration of parts is regarded as data before a predetermined period from the time when the function recovery treatment is performed, and the data is used as post-treatment data. As a result, compared to the case where the initial data before the vehicle travels is used as the post-treatment data among the related regulation data, the deterioration of the parts has not progressed, but it has been updated to some extent by the reinforcement learning accompanying the actual driving. It is possible to use the related regulation data as post-treatment data.

8. The update process is the vehicle control system according to any one of 5 to 7 above, which is executed by the first execution device.
9. A vehicle control device including the first execution device according to any one of 5 to 8 above.

10. A vehicle learning device including the second execution device according to any one of 5 to 8 above.

The figure which shows the control device and its drive system which concerns on 1st Embodiment. The flow chart which shows the procedure of the process executed by the control device which concerns on the same embodiment. The flow chart which shows the detailed procedure of a part of the processing executed by the control device which concerns on this embodiment. The flow chart which shows the procedure of the process executed by the control device which concerns on the same embodiment. The figure which shows the structure of the control system for a vehicle which concerns on 2nd Embodiment. (A) and (b) are flow charts showing the procedure of processing executed by the vehicle control system.

<First Embodiment>
Hereinafter, the first embodiment of the vehicle control device will be described with reference to the drawings.
FIG. 1 shows the configuration of the drive system and the control device of the vehicle VC1 according to the present embodiment.

As shown in FIG. 1, the intake passage 12 of the internal combustion engine 10 is provided with a throttle valve 14 and a fuel injection valve 16 in this order from the upstream side, and is injected from the air or fuel injection valve 16 sucked into the intake passage 12. The fuel is discharged into the combustion chamber 24 partitioned by the cylinder 20 and the piston 22 as the intake valve 18 is opened. In the combustion chamber 24, the air-fuel mixture is subjected to combustion with the spark discharge of the ignition device 26, and the energy generated by the combustion is converted into the rotational energy of the crankshaft 28 via the piston 22. To. The air-fuel mixture used for combustion is discharged to the exhaust passage 32 as exhaust gas when the exhaust valve 30 is opened. The exhaust passage 32 is provided with a catalyst 34 as an aftertreatment device for purifying the exhaust gas.

The input shaft 52 of the transmission 50 can be mechanically connected to the crankshaft 28 via a torque converter 40 provided with a lockup clutch 42. The transmission 50 is a device that makes the gear ratio variable, which is the ratio of the rotation speed of the input shaft 52 to the rotation speed of the output shaft 54. A drive wheel 60 is mechanically connected to the output shaft 54.

The control device 70 targets the internal combustion engine 10, and in order to control the torque and the exhaust component ratio, which are the controlled amounts thereof, the control device 70 controls the operation unit of the internal combustion engine 10 such as the throttle valve 14, the fuel injection valve 16, and the ignition device 26. Manipulate. Further, the control device 70 controls the torque converter 40 and operates the lockup clutch 42 in order to control the engaged state of the lockup clutch 42. Further, the control device 70 targets the transmission 50 as a control target, and operates the transmission 50 to control the gear ratio as the control amount thereof. Note that FIG. 1 shows the operation signals MS1 to MS5 of the throttle valve 14, the fuel injection valve 16, the ignition device 26, the lockup clutch 42, and the transmission 50, respectively.

The control device 70 controls the intake air amount Ga detected by the air flow meter 80, the opening degree of the throttle valve 14 (throttle opening degree TA) detected by the throttle sensor 82, and the crank angle sensor 84 for controlling the control amount. Refer to the output signal Scr of. Further, the control device 70 refers to the depression amount of the accelerator pedal 86 (accelerator operation amount PA) detected by the accelerator sensor 88 and the acceleration Gx in the front-rear direction of the vehicle VC1 detected by the acceleration sensor 90. Further, the control device 70 refers to the position data Pgps by the Global Positioning System (GPS92).

The control device 70 includes a CPU 72, a ROM 74, an electrically rewritable non-volatile memory (storage device 76), and a peripheral circuit 78, which can communicate with each other via the local network 79. Here, the peripheral circuit 78 includes a circuit that generates a clock signal that defines the internal operation, a power supply circuit, a reset circuit, and the like.

The control program 74a and the learning program 74b are stored in the ROM 74. On the other hand, the storage device 76 contains the related regulation data DR that defines the relationship between the accelerator operation amount PA, the command value of the throttle opening degree TA (throttle opening degree command value TA *), and the retard angle amount aop of the ignition device 26. , The initial data DR0 is stored. Here, the retard angle amount aop is a retard angle amount with respect to a predetermined reference ignition timing, and the reference ignition timing is a timing on the retard side of the MBT ignition timing and the knock limit point. The MBT ignition timing is the ignition timing at which the maximum torque can be obtained (maximum torque ignition timing). The knock limit is an ignition timing advance limit that can keep knocking within an acceptable level under the best possible conditions when using a high octane fuel with a high knock limit. Further, the torque output mapping data DT is stored in the storage device 76. The torque output map defined by the torque output map data DT is a map that outputs the torque Trq by inputting the rotation speed NE, the filling efficiency η, and the ignition timing of the crankshaft 28.

FIG. 2 shows a procedure of processing executed by the control device 70 according to the present embodiment. The process shown in FIG. 2 is realized by the CPU 72 repeatedly executing the control program 74a and the learning program 74b stored in the ROM 74, for example, at a predetermined cycle. In the following, the step number of each process is indicated by a number prefixed with "S".

In the series of processes shown in FIG. 2, the CPU 72 first, as the state s, time-series data composed of six sampling values "PA (1), PA (2), ... PA (6)" of the accelerator operation amount PA. (S10). Here, each sampling value constituting the time series data is sampled at different timings from each other. In the present embodiment, time-series data is configured by six sampling values that are adjacent to each other in time series when sampling is performed at a constant sampling cycle.

Next, the CPU 72 sets the action a including the throttle opening degree command value TA * and the retard angle amount aop according to the state s acquired by the process of S10, in accordance with the policy π defined by the relational regulation data DR (S12).

In the present embodiment, the relational regulation data DR is data that defines the action value function Q and the policy π. In the present embodiment, the action value function Q is a table-type function showing the value of the expected return according to the eight-dimensional independent variables of the state s and the action a. Further, the policy π is predetermined while preferentially selecting the action a (greedy action) that is the largest among the action value functions Q in which the independent variable is given the state s when the state s is given. With the probability ε of, the rule for selecting other actions a is determined.

Specifically, the number of values that can be taken by the independent variable of the action value function Q according to the present embodiment is reduced by a part of all combinations of possible values of the state s and the action a by human knowledge or the like. It is a thing. That is, for example, in the time series data of the accelerator operation amount PA, one of the two adjacent sampling values becomes the minimum value of the accelerator operation amount PA and the other becomes the maximum value from the operation of the accelerator pedal 86 by a person. The action value function Q is not defined as it cannot occur. In the present embodiment, the possible values of the state s that defines the action value function Q are limited to 10 4 or less, more preferably 10 3 or less, by dimensionality reduction based on human knowledge or the like.

Next, the CPU 72 outputs the operation signal MS1 to the throttle valve 14 to operate the throttle opening degree TA based on the set throttle opening degree command value TA * and the retard angle amount aop, and operates the throttle opening degree TA, and at the same time, operates the operation signal MS3 to the ignition device 26. Is output to control the ignition timing (S14). Here, in the present embodiment, since the feedback control of the throttle opening degree TA to the throttle opening degree command value TA * is illustrated, even if the throttle opening degree command value TA * is the same value, the operation signal MS1 Can be different signals from each other. Further, for example, when a well-known knocking control (KCS) or the like is performed, the ignition timing is a value obtained by retarding the reference ignition timing with a retard angle amount aop and feedback-corrected by KCS. Here, the reference ignition timing is variably set by the CPU 72 according to the rotation speed NE of the crankshaft 28 and the filling efficiency η. The rotation speed NE is calculated by the CPU 72 based on the output signal Scr of the crank angle sensor 84. Further, the filling efficiency η is calculated by the CPU 72 based on the rotation speed NE and the intake air amount Ga.

Next, the CPU 72 acquires the torque Trq of the internal combustion engine 10, the torque command value Trq * for the internal combustion engine 10, and the acceleration Gx (S16). Here, the CPU 112 calculates the torque Trq by inputting the rotation speed NE, the filling efficiency η, and the ignition timing into the torque output map. Further, the CPU 72 sets the torque command value Trq * according to the accelerator operation amount PA.

Next, the CPU 72 determines whether or not the transient flag F is “1” (S18). When the transient flag F is "1", it indicates that it is in transient operation, and when it is "0", it indicates that it is not in transient operation. When the CPU 72 determines that the transient flag F is "0" (S18: NO), the CPU 72 determines whether or not the absolute value of the change amount ΔPA of the accelerator operation amount PA per unit time is equal to or greater than the predetermined amount ΔPAth. (S20). Here, the change amount ΔPA may be, for example, the difference between the latest accelerator operation amount PA at the execution timing of the process of S20 and the accelerator operation amount PA before the same timing by a unit time.

When the CPU 72 determines that the predetermined amount is ΔPAth or more (S20: YES), the CPU 72 substitutes “1” for the transient flag F (S22).
On the other hand, when the CPU 72 determines that the transient flag F is "1" (S18: YES), the CPU 72 determines whether or not a predetermined period has elapsed from the execution timing of the process of S22 (S24). Here, the predetermined period is a period until the state in which the absolute value of the change amount ΔPA of the accelerator operation amount PA per unit time is smaller than the predetermined amount ΔPAth and is equal to or less than the specified amount continues for a predetermined time. When the CPU 72 determines that the predetermined period has elapsed (S24: YES), the CPU 72 substitutes “0” for the transient flag F (S26).

When the processing of S22 and S26 is completed, the CPU 72 updates the action value function Q by reinforcement learning, assuming that one episode has ended (S28).
FIG. 3 shows the details of the processing of S28.

In the series of processes shown in FIG. 3, the CPU 72 performs time-series data including a set of three sampling values of the torque command value Trq *, the torque Trq, and the acceleration Gx in the most recently completed episode, and the state s and the action a. (S30). Here, the latest episode is a period in which the transient flag F is continuously set to "0" when the processing of S30 is performed after the processing of S22, and the processing of S26 is followed by the processing of S30. When the processing is performed, it is the period during which the transient flag F is continuously set to "1".

FIG. 3 shows that the values in parentheses that differ are the values of the variables at different sampling timings. For example, the torque command value Trq * (1) and the torque command value Trq * (2) have different sampling timings. Further, the time-series data of the action a belonging to the latest episode is defined as the action set Aj, and the time-series data of the state s belonging to the same episode is defined as the state set Sj.

Next, the CPU 72 has a condition (a) that the absolute value of the difference between the arbitrary torque Trq belonging to the latest episode and the torque command value Trq * is the specified amount ΔTrq or less, and the acceleration Gx is the lower limit value GxL or more. It is determined whether or not the logical product with the condition (a) indicating that the upper limit value is GxH or less is true (S32).

Here, the CPU 72 variably sets the specified amount ΔTrq according to the amount of change ΔPA per unit time of the accelerator operation amount PA at the start of the episode. That is, the CPU 72 considers that when the absolute value of the change amount ΔPA is large, it is an episode related to the transient time, and sets the specified amount ΔTrq to a larger value than in the case of the steady state.

Further, the CPU 72 variably sets the lower limit value GxL according to the change amount ΔPA of the accelerator operation amount PA at the start of the episode. That is, when the CPU 72 is an episode related to the transient time and the amount of change ΔPA is positive, the CPU 72 sets the lower limit value GxL to a larger value than the case of the episode related to the steady state. Further, when the CPU 72 is an episode related to the transient time and the amount of change ΔPA is negative, the CPU 72 sets the lower limit value GxL to a smaller value than the case of the episode related to the steady state.

Further, the CPU 72 variably sets the upper limit value GxH according to the change amount ΔPA per unit time of the accelerator operation amount PA at the start of the episode. That is, when the CPU 72 is an episode related to the transient time and the amount of change ΔPA is positive, the CPU 72 sets the upper limit value GxH to a larger value than the case of the episode related to the steady state. Further, when the CPU 72 is an episode related to the transient time and the amount of change ΔPA is negative, the CPU 72 sets the upper limit value GxH to a smaller value than the case of the episode related to the steady state.

When the CPU 72 determines that the logical product is true (S32: YES), it substitutes "10" for the reward r (S34), while when it determines that the logical product is false (S32: NO), the reward r is "". -10 ”is substituted (S36). When the processing of S34 and S36 is completed, the CPU 72 updates the relational regulation data DR stored in the storage device 76 shown in FIG. In this embodiment, the ε soft policy on-type Monte Carlo method is used.

That is, the CPU 72 adds the reward r to the profit R (Sj, Aj) determined by the set of each state read by the process of S30 and the corresponding action (S38). Here, "R (Sj, Aj)" is a general description of the profit R in which one of the elements of the state set Sj is a state and one of the elements of the action set Aj is an action. Next, each of the revenues R (Sj, Aj) determined by the set of each state read by the process of S30 and the corresponding action is averaged and substituted into the corresponding action value function Q (Sj, Aj) ( S40). Here, the averaging may be a process of dividing the profit R calculated by the process of S38 by the number obtained by adding a predetermined number to the number of times the process of S38 is performed. The initial value of the profit R may be the initial value of the corresponding action value function Q.

Next, the CPU 72 describes the throttle opening command value TA * and the retard angle amount aop at the maximum value of the corresponding action value functions Q (Sj, A) for the state read by the process of S30. The set of actions is substituted into the action Aj * (S42). Here, "A" indicates an arbitrary action that can be taken. The action Aj * has a different value depending on the type of the state read by the process of S30, but here, the notation is simplified and described by the same symbol.

Next, the CPU 72 updates the corresponding policy π (Aj | Sj) for each of the states read by the process of S30 (S44). That is, assuming that the total number of actions is "| A |", the selection probability of the action Aj * selected by S42 is "1-ε + ε / | A |". Further, the selection probabilities of "| A | -1" actions other than the action Aj * are set to "ε / | A |", respectively. Since the process of S44 is a process based on the action value function Q updated by the process of S40, the relationship rule data DR that defines the relationship between the state s and the action a increases the profit R. Will be updated to.

When the process of S44 is completed, the CPU 72 temporarily ends the series of processes shown in FIG.
Returning to FIG. 2, the CPU 72 temporarily ends the series of processes shown in FIG. 2 when the process of S28 is completed or when a negative determination is made in the processes of S20 and S24. The processing of S10 to S26 is realized by the CPU 72 executing the control program 74a, and the processing of S28 is realized by the CPU 72 executing the learning program 74b. Further, the relational regulation data DR at the time of shipment of the vehicle VC1 is the same as the initial data DR0. The initial data DR0 is data that has been learned in advance by executing the same processing as that shown in FIG. 2 while simulating the running of the vehicle on the test bench.

FIG. 4 shows a procedure of processing executed by the control device 70 according to the present embodiment. The process shown in FIG. 4 is realized by the CPU 72 repeatedly executing the learning program 74b stored in the ROM 74, for example, at a predetermined cycle.

In the series of processes shown in FIG. 4, the CPU 72 first determines whether or not the mileage RL of the vehicle VC1 has reached a predetermined distance (S50). Here, the predetermined distance is a plurality of distances represented by multiples of a predetermined amount, such as 10,000 km, 20,000 km, 30,000 km, and so on. When the CPU 72 determines that the distance is a predetermined distance (S50: YES), the CPU 72 stores the relational regulation data DR at that time in the storage device 76 as the updated data DR1 (S52). When the predetermined amount is "10,000 km" and the mileage RL is 20,000 km, two different data are stored in the storage device 76 as the updated data DR1. That is, every time it is determined that the mileage RL is a predetermined distance, the relational regulation data DR at that time is newly stored in the storage device 76 as the updated data DR1, and the updated data DR1 increases.

When the process of S52 is completed or when a negative determination is made in the process of S50, the CPU 72 determines whether or not the function recovery measure has been taken (S54). In the present embodiment, when the function recovery measures of the parts mounted on the vehicle VC1 are performed due to the maintenance of the vehicle VC1, a signal indicating that the function recovery measures have been performed is input from the scan tool to the control device 70. I'm assuming. Therefore, the CPU 72 determines that the function recovery treatment has been performed when the signal indicating that the function recovery treatment has been performed is input.

When it is determined that the function recovery measure has been taken (S54: YES), the CPU 72 determines whether or not the updated data DR1 exists at the time when the mileage is shorter than the current mileage RL by a predetermined amount ΔL or more. (S56). Then, when the CPU 72 determines that the updated data DR1 exists (S56: YES), the CPU 72 substitutes the updated data DR1 for the post-treatment data DRp (S58). When a plurality of data to be positively determined by the processing of S56 are stored in the storage device 76 as the updated data DR1, the CPU 72 uses the data when the mileage RL is the longest among them after the treatment. Substitute in data DRp. On the other hand, when the CPU 72 determines that the updated data DR1 does not exist (S56: NO), the CPU 72 substitutes the initial data DR0 into the post-treatment data DRp (S60).

When the processing of S58 and S60 is completed, the CPU 72 rewrites the relational regulation data DR used for the processing of S12 to the post-treatment data DRp (S62).
The CPU 72 temporarily ends the series of processes shown in FIG. 4 when the process of S62 is completed or when a negative determination is made in the process of S54.

Here, the operation and effect of this embodiment will be described.
The CPU 72 acquires time-series data of the accelerator operation amount PA in accordance with the operation of the accelerator pedal 86 by the user, and sets the action a including the throttle opening degree command value TA * and the retard angle amount aop according to the policy π. Here, the CPU 72 basically selects the action a that maximizes the expected return based on the action value function Q defined in the relational rule data DR. However, the CPU 72 searches for the action a that maximizes the expected return by selecting an action other than the action a that maximizes the expected return with a predetermined probability ε. As a result, the related regulation data DR can be updated by reinforcement learning as the user drives the vehicle VC1. Therefore, the throttle opening command value TA * and the retard angle amount aop according to the accelerator operation amount PA can be set to appropriate values in the traveling of the vehicle VC1 without excessively increasing the man-hours by the expert.

In this way, the relational regulation data DR, which is the same as the initial data DR0 at the time of shipment of the vehicle VC1, is updated as the vehicle VC1 travels. Here, for example, even if the throttle opening degree TA is the same, when deposits are deposited on the throttle valve 14 and the intake passage 12, the passage cross-sectional area of the intake passage 12 becomes small, so that the intake air amount Ga becomes smaller. Therefore, the throttle opening degree command value TA * that maximizes the expected profit according to the time-series data of the accelerator operation amount PA specified by the related regulation data DR is caused by the accumulation of deposits on the throttle valve 14. It may be updated to compensate for changes in the flow path cross-sectional area of the intake passage 12. In this way, when the relevant regulation data DR is learned so as to compensate for the aged deterioration of the parts of the vehicle VC1, after that, the parts are replaced or cleaned along with the maintenance to recover the functions of the parts of the vehicle VC1. If this is done, there is a concern that the related regulation data DR will not be appropriate data for determining actions to increase expected profits.

Therefore, each time the mileage RL increases by a predetermined amount, the CPU 72 stores the relational regulation data DR at that time as the updated data DR1. Then, when the CPU 72 determines that the function recovery treatment associated with the maintenance has been performed, if there is updated data DR1 when the mileage is shorter than the current mileage RL by a predetermined amount ΔL or more, this is used after the treatment. As the data DRp, the related regulation data DR used for setting the throttle opening command value TA * and the retard angle amount aop is rewritten. Here, the updated data DR1 is data updated with the actual traveling of the vehicle VC1 with respect to the initial data DR0, which is the relational regulation data DR at the time of shipment of the vehicle VC1. Moreover, since it is the relational regulation data DR in the mileage RL shorter than the predetermined distance ΔL or more than when the function recovery treatment is performed, it is considered that the data is less affected by deterioration of parts at the time when the function recovery treatment is performed. Be done. Therefore, by updating to the updated data DR1, it is possible to obtain the relevant regulation data DR suitable for the vehicle VC1 after the function recovery treatment.

According to the present embodiment described above, the effects described below can be further obtained.
(1) When there is no updated data DR1 in the mileage RL shorter than the predetermined distance ΔL or more than when the function recovery treatment is performed, the initial data DR0 is used as the post-treatment data DRp, and the related regulation data DR is rewritten accordingly. .. The initial data DR0 is data that has not been updated so as to compensate for the deterioration of the parts. Therefore, by rewriting to the initial data DR0, the throttle valve 14 and the ignition device 26 after the function recovery treatment are compared with the case where the relational regulation data DR before the function recovery treatment is used as it is even after the function recovery treatment is performed. You can manipulate them with data that is more appropriate for your operation.

(2) The time series data of the accelerator operation amount PA was included in the independent variable of the action value function Q. As a result, the value of the action a can be finely adjusted for various changes in the accelerator operation amount PA, as compared with the case where only a single sampling value is used as the independent variable for the accelerator operation amount PA.

(3) The throttle opening command value TA * itself is included in the independent variable of the action value function Q. This makes it easier to increase the degree of freedom of search by reinforcement learning, for example, as compared with the case where the parameters of the model formula that models the behavior of the throttle opening command value TA * are used as independent variables related to the throttle opening. Is.

<Second embodiment>
Hereinafter, the second embodiment will be described with reference to the drawings, focusing on the differences from the first embodiment.

FIG. 5 shows the configuration of the control system that executes reinforcement learning in the present embodiment. In FIG. 5, the members corresponding to the members shown in FIG. 1 are designated by the same reference numerals for convenience.

In addition to the control program 74a, the learning main program 74c is stored in the ROM 74 in the vehicle VC1 shown in FIG. Further, although the storage device 76 in the vehicle VC1 stores the torque output mapping data DT and the related regulation data DR, it does not store the initial data DR0. Further, the control device 70 includes a communication device 77. The communication device 77 is a device for communicating with the data analysis center 110 via the network 100 outside the vehicle VC1.

The data analysis center 110 analyzes the data transmitted from the plurality of vehicles VC1, VC2, .... The data analysis center 110 includes a CPU 112, a ROM 114, an electrically rewritable non-volatile memory (storage device 116), peripheral circuits 118, and a communication device 117, which are made communicable by the local network 119. Is. The learning subprogram 114a is stored in the ROM 114. Initial data DR0 is stored in the storage device 116.

FIG. 6 shows a processing procedure for dealing with the functional recovery treatment according to the present embodiment. The process shown in FIG. 6A is realized by the CPU 72 executing the learning main program 74c stored in the ROM 74 shown in FIG. Further, the process shown in FIG. 6B is realized by the CPU 112 executing the learning subprogram 114a stored in the ROM 114. Note that the processes corresponding to the processes shown in FIG. 4 in FIG. 6 are assigned the same step numbers for convenience. Hereinafter, the processing shown in FIG. 6 will be described in chronological order.

In the series of processes shown in FIG. 6A, the CPU 72 first operates the communication device 77 to transmit the identification information ID of the vehicle VC1, the mileage RL, and the position data Pgps (S70).

On the other hand, as shown in FIG. 6B, the CPU 112 receives the identification information ID, the mileage RL, and the position data Pgps (S80). Then, the CPU 112 updates the mileage RL and the position data Pgps associated with the identification information ID stored in the storage device 116 to the values received by the processing of S80 (S82).

On the other hand, as shown in FIG. 6A, when the CPU 72 executes the process of S54 and makes an affirmative determination, the CPU 72 operates the communication device 77 after appropriate measures as the relational regulation data DR used for the process of S12. A signal requesting the data DRp for use is transmitted (S72).

On the other hand, as shown in FIG. 6B, the CPU 112 determines whether or not there is a request for post-treatment data DRp (S84). Then, when the CPU 112 determines that there is a request for the post-treatment data DRp (S84: YES), the CPU 112 searches for a vehicle that is positionally close to the vehicle VC1 that transmitted the requested signal and has a short mileage (S86). ). Here, the condition that the vehicles are close to each other is that the distance from the vehicle VC1 that has transmitted the requested signal is less than or equal to the predetermined distance based on the position data Pgps of each vehicle stored by the processing of S82. To do. Further, as a vehicle having a shorter mileage than the vehicle VC1 that transmitted the requested signal, the mileage is shorter than the mileage RL of the vehicle VC1 by a predetermined amount ΔL or more, and the difference from the mileage RL of the vehicle VC1 is large. It is assumed that the vehicle has a specified amount of ΔH or less.

Here, the reason for searching for a vehicle whose distance from the vehicle VC1 is less than or equal to a predetermined distance is that the relational regulation data DR in the case of a vehicle located at a place where the distance from the vehicle VC1 is too large is due to a difference in the environment or the like. This is because there was concern that the data would not be appropriate for increasing the expected return for the vehicle VC1. Further, the vehicle whose mileage RL is shorter than the mileage RL of the vehicle VC1 by a predetermined amount ΔL or more but not shorter than the specified amount ΔH is specified as a vehicle that approximates the state before the deterioration of the parts of the vehicle VC1. It is a setting to do.

When the CPU 112 determines that the corresponding vehicle exists (S88: YES), the CPU 112 operates the communication device 117 to prompt the corresponding vehicle to transmit the related regulation data DR, and receives this as the other vehicle regulation data DRa. (S90). Next, the CPU 72 substitutes the other vehicle regulation data DRa into the post-treatment data DRp (S92). On the other hand, when the CPU 72 determines that the corresponding vehicle does not exist (S88: NO), the CPU 72 substitutes the initial data DR0 into the post-treatment data DRp (S94). When the processing of S92 and S94 is completed, the CPU 112 operates the communication device 117 to transmit the post-treatment data DRp to the vehicle VC1 that issued the request (S96). When the processing of S96 is completed or when a negative determination is made in the processing of S84, the CPU 112 temporarily ends the series of processing shown in FIG. 6B.

On the other hand, as shown in FIG. 6A, the CPU 72 receives the transmitted post-treatment data DRp (S74) and executes the process of S62.
When the processing of S62 is completed or when a negative determination is made in the processing of S54, the CPU 72 temporarily ends the series of processing shown in FIG. 6A.

<Correspondence>
The correspondence between the matters in the above-described embodiment and the matters described in the above-mentioned "means for solving the problem" column is as follows. In the following, the correspondence is shown for each number of the solution means described in the column of "Means for solving the problem".

[1] The execution device corresponds to the CPU 72 and the ROM 74, and the storage device corresponds to the storage device 76. The acquisition process corresponds to the processes of S10 and S16, the operation process corresponds to the process of S14, the reward calculation process corresponds to the processes of S32 to S36, and the update process corresponds to the processes of S38 to S44. .. The detection process corresponds to the process of S54, and the switching process corresponds to the process of S62. The updated map corresponds to the map specified by the command for executing the processes of S38 to S44 in the learning program 74b. [2] Corresponds to the process of S62 when an affirmative decision is made in the process of S56. [3] The past data maintenance process corresponds to the process of S52. [4, 6] The post-treatment data request processing corresponds to the processing of S72, and the post-treatment data reception processing corresponds to the processing of S74. [5, 8-10] The first executor corresponds to the CPU 72 and ROM 74, and the second executor corresponds to the CPU 112 and ROM 114. The post-treatment data transmission process corresponds to the process of S96, and the post-treatment data reception process corresponds to the process of S74. [7] Corresponds to the processing of S96 when an affirmative judgment is made in the processing of S88.

<Other Embodiments>
In addition, this embodiment can be implemented by changing as follows. The present embodiment and the following modified examples can be implemented in combination with each other within a technically consistent range.

・ "About detection processing"
In the above embodiment, with the scan tool connected to the control device 70, the control device 70 detects that fact by inputting a signal from the scan tool to the control device 70 to the effect that the function recovery action has been taken. However, the detection process is not limited to this. For example, when a function recovery measure is taken at a repair shop or the like, the data analysis center 110 may be notified to that effect via the network 100. Even in that case, the post-treatment data DRp can be transmitted to the control device 70 by executing the processing according to the processing of S80, 82, S86 to S96 of FIG. 6B at the data analysis center 110. ..

However, the detection process is not limited to that executed by either the control device 70 or the data analysis center 110. For example, as described in the column of "About the vehicle control system", when the vehicle control system is configured by including the mobile terminal, the mobile terminal may execute the detection process. Here, when the vehicle control system is configured by the control device 70, the mobile terminal, and the data analysis center 110, after the mobile terminal executes the detection process, a signal requesting the post-treatment data DRp is transmitted to the data analysis center 110. do it.

Further, the detection process is not limited to the process of directly detecting a signal by a repair shop or the like. For example, when a signal indicating that the function recovery treatment has been performed is transmitted to the mobile terminal and a signal to that effect is further transmitted from the mobile terminal to the control device 70, the control device 70 receives the signal from the mobile terminal. , It may be a detection process.

・ "Past data maintenance process"
In the above embodiment, each time the mileage RL increases by a predetermined distance, the relational regulation data DR at that time is stored as the updated data DR1, but the present invention is not limited to this. For example, the amount of deposits around the throttle valve 14 is quantified by the average value of the intake air amount Ga per "1%" when the throttle opening degree TA is fully opened at 100%, and the average value is only the specified value. At the time of change, the relational regulation data DR at that time may be set as the updated data DR1. Here, the specified value may be set to an upper limit value in which the influence on the intake air amount Ga can be ignored.

・ "About post-treatment data transmission processing"
The data required for the control device 70 to execute the switching process is not limited to the post-treatment data DRp. For example, as described in the column of "Detection processing", when a signal indicating that the function recovery treatment has been performed is transmitted from the repair shop to the data analysis center 110 via the network 100, the data analysis center 110 controls the control device. Data indicating that fact and post-treatment data DRp may be transmitted to 70.

The processing of S86 to S92 may be deleted, and the post-treatment data DRp transmitted from the data analysis center 110 to the control device 70 may always be the initial data DR0.
Further, for example, the processing of S28 in FIG. 2 and the processing according to the processing of S50, S52, S56 to S62 of FIG. 4 are executed by the data analysis center 110, and the post-treatment data DRp generated by the processing of S62 is executed. May be transmitted to the control device 70.

・ "About behavior variables"
In the above embodiment, the throttle opening command value TA * is exemplified as a variable related to the opening degree of the throttle valve as an action variable, but the present invention is not limited to this. For example, the responsiveness of the throttle opening command value TA * to the accelerator operation amount PA is expressed by the wasted time and the second-order lag filter, and the wasted time and the two variables that define the second-order lag filter are a total of three. The variable may be a variable related to the opening degree of the throttle valve. However, in that case, it is desirable that the state variable is a change amount per unit time of the accelerator operation amount PA instead of the time series data of the accelerator operation amount PA.

In the above embodiment, the retard angle amount aop is exemplified as a variable related to the ignition timing as an action variable, but the present invention is not limited to this. For example, it may be the ignition timing itself to be corrected by KCS.

In the above embodiment, variables related to the opening degree of the throttle valve and variables related to the ignition timing are exemplified as behavioral variables, but the present invention is not limited to this. For example, the fuel injection amount may be used in addition to the variables related to the opening degree of the throttle valve and the variables related to the ignition timing. Further, regarding these three, only the variable related to the opening degree of the throttle valve and the fuel injection amount may be adopted as the action variables, or only the variable related to the ignition timing and the fuel injection amount may be adopted. Furthermore, for those three, only one of them may be adopted as the behavioral variable.

Further, as described in the column of "About the internal combustion engine", in the case of a compression ignition type internal combustion engine, a variable related to the injection amount is used instead of the variable related to the opening degree of the throttle valve, and a variable related to the ignition timing is used instead of the variable related to the ignition timing. You can use the variables related to. In addition to the variables related to the injection timing, the variables related to the number of injections in one combustion cycle and the end timing of one of the two fuel injections adjacent in time series for one cylinder in one combustion cycle and the other It is desirable to add a variable for the time interval between the start timing.

Further, for example, when the transmission 50 is a stepped transmission, the current value of the solenoid valve for adjusting the engaged state of the clutch by flood control may be used as an action variable.
Further, for example, when a hybrid vehicle, an electric vehicle, or a fuel cell vehicle is adopted as the vehicle as described in the column of "About the vehicle" below, the torque and output of the rotary electric machine may be used as an action variable. Further, for example, when an in-vehicle air conditioner including a compressor that is rotated by the rotational power of the crankshaft of the internal combustion engine is provided, the load torque of the compressor may be included in the action variable. Further, when an electric in-vehicle air conditioner is provided, the power consumption of the air conditioner may be included in the action variable.

・ "About the condition"
In the above embodiment, the time-series data of the accelerator operation amount PA is set to the data consisting of six values sampled at equal intervals, but the present invention is not limited to this. The data may be data consisting of two or more sampling values at different sampling timings, and at this time, it is more desirable that the data consists of three or more sampling values or the sampling intervals are evenly spaced.

The state variable related to the accelerator operation amount is not limited to the time series data of the accelerator operation amount PA, but is, for example, the amount of change of the accelerator operation amount PA per unit time as described in the column of "behavior variable". May be good.

Further, for example, as described in the column of "About the action variable", when the current value of the solenoid valve is used as the action variable, the state includes the rotation speed of the input shaft 52 of the transmission, the rotation speed of the output shaft 54, and the solenoid valve. The oil pressure adjusted by may be included. Further, for example, as described in the column of "behavior variable", when the torque or output of the rotary electric machine is used as the action variable, the charge rate or temperature of the battery may be included in the state. Further, for example, as described in the column of "behavior variables", when the load torque of the compressor and the power consumption of the air conditioner are included in the behavior, the temperature inside the vehicle interior may be included in the state.

・ "About dimensionality reduction of table format data"
The method for reducing the dimension of the table format data is not limited to the one illustrated in the above embodiment. For example, since it is rare that the accelerator operation amount PA becomes the maximum value, the action value function Q is not defined for the state where the accelerator operation amount PA becomes the specified amount or more, and the accelerator operation amount PA becomes the specified amount or more. In this case, the throttle opening command value TA * and the like may be adapted separately. Further, for example, the dimension may be reduced by excluding the value at which the throttle opening degree command value TA * is equal to or more than the specified value from the values that can be taken.

・ "Regarding related regulation data"
In the above embodiment, the action value function Q is a table-type function, but the present invention is not limited to this. For example, a function approximator may be used.

For example, instead of using the action value function Q, the policy π is expressed by a function approximation device in which the state s and the action a are independent variables and the probability of taking the action a is the dependent variable, and the parameter that determines the function approximation device is expressed. , May be updated according to the reward r.

・ "About operation processing"
For example, as described in the column of "Relationship regulation data", when the action value function is used as a function approximation device, all the discrete value sets for the action that are the independent variables of the function of the table type in the above embodiment. The action a that maximizes the action value function Q may be selected by inputting the action value function Q together with the state s.

Further, for example, as described in the column of "Relationship regulation data", when the policy π is a function approximation device in which the state s and the action a are independent variables and the probability of taking the action a is the dependent variable, the policy π is used. Action a may be selected based on the probability indicated by.

・ "About updated mapping"
In the processing of S38 to S44, the method by the ε soft policy on-type Monte Carlo method was illustrated, but the present invention is not limited to this. For example, the policy-off type Monte Carlo method may be used. However, it is not limited to the Monte Carlo method, for example, the policy-off type TD method is used, the policy-on type TD method such as the SARSA method is used, and the eligibility tracing method is used as the policy-on type learning, for example. You may use it.

Further, for example, as described in the column of "Relationship regulation data", when the policy π is expressed by using a function approximation device and this is directly updated based on the reward r, the policy gradient method or the like is used to update the policy π. A map may be constructed.

Further, only one of the action value function Q and the policy π is not limited to the one directly updated by the reward r. For example, the action value function Q and the policy π may be updated, respectively, as in the actor-critic method. Further, in the actor-critic method, the value function V may be updated instead of the action value function Q, for example.

The “ε” that defines the policy π is not limited to a fixed value, and may be changed according to a predetermined rule according to the degree of learning progress.
・ "About reward calculation processing"
In the process of FIG. 3, the reward is given according to whether or not the logical product of the condition (a) and the condition (b) is true, but the reward is not limited to this. For example, a process of giving a reward depending on whether or not the condition (a) is satisfied and a process of giving a reward depending on whether or not the condition (b) is satisfied may be executed. Further, for example, with respect to two processes, that is, a process of giving a reward depending on whether or not the condition (a) is satisfied and a process of giving a reward depending on whether or not the condition (b) is satisfied, among them. Only one of the processes may be executed.

Further, for example, instead of giving the same reward uniformly when the condition (a) is satisfied, a process may be performed in which a larger reward is given when the absolute value of the difference between the torque Trq and the torque command value Trq * is smaller than when it is large. .. Further, for example, instead of giving the same reward uniformly when the condition (a) is not satisfied, it is also possible to give a smaller reward when the absolute value of the difference between the torque Trq and the torque command value Trq * is large than when it is small. Good.

Further, for example, instead of giving the same reward uniformly when the condition (a) is satisfied, the process may be such that the magnitude of the reward is variable according to the magnitude of the acceleration Gx. Further, for example, instead of giving the same reward uniformly when the condition (a) is not satisfied, the process may be such that the magnitude of the reward is variable according to the magnitude of the acceleration Gx.

In the above embodiment, the reward r is given according to whether or not the drivability standard is satisfied, but the drivability standard is not limited to the above-mentioned one, and for example, whether noise or vibration intensity satisfies the standard. It may be set according to the case. However, not limited to this, there are four conditions: whether the acceleration meets the standard, whether the followability of the torque Trq meets the standard, whether the noise meets the standard, and whether the vibration intensity meets the standard. It may be any one or more of them.

The reward calculation process is not limited to the one in which the reward r is given depending on whether or not the criteria for drivability are satisfied. For example, if the fuel consumption rate meets the criteria, it may be a process that rewards a larger amount than if it does not meet the criteria. Further, for example, it may be a process of giving a larger reward when the exhaust characteristic satisfies the standard than when it does not meet the standard. In addition, when the drivability standard is met, a larger reward is given than when it is not met, when the fuel consumption rate meets the standard, a larger reward is given than when it is not met, and when the exhaust characteristics meet the standard. It may include two or three of the three processes with the process that rewards more than the case where is not satisfied.

Further, for example, when the current value of the solenoid valve of the transmission 50 is used as the action variable as described in the column of "about the action variable", for example, the following three processes (a) to (c) are performed in the reward calculation process. At least one of these processes may be included.

(A) This is a process in which when the time required for switching the gear ratio by the transmission is within a predetermined time, a larger reward is given than when the predetermined time is exceeded.
(B) This is a process in which when the absolute value of the change speed of the rotation speed of the input shaft 52 of the transmission is equal to or less than the input side predetermined value, a larger reward is given than when the input side predetermined value is exceeded.

(C) This is a process in which when the absolute value of the change speed of the rotation speed of the output shaft 54 of the transmission is equal to or less than the predetermined value on the output side, a larger reward is given than when the predetermined value on the output side is exceeded.
In addition, for example, as described in the column of "About behavior variables", when the torque or output of a rotating electric machine is used as a behavior variable, a process of giving a larger reward than when the battery charge rate is within a predetermined range. Alternatively, it may include a process that rewards a greater amount than if the battery temperature is within a predetermined range. In addition, for example, as described in the column of "About behavior variables", when the load torque of the compressor and the power consumption of the air conditioner are included in the behavior variables, it is larger than when the temperature inside the vehicle is not within the predetermined range. A rewarding process may be added.

・ "About vehicle control system"
The vehicle control system is not limited to the one composed of the control device 70 and the data analysis center 110. For example, instead of the data analysis center 110, a mobile terminal owned by the user may be used, and the vehicle control system may be configured by the control device 70 and the mobile terminal. Further, for example, it may be composed of a control device 70, a mobile terminal, and a data analysis center 110.

・ "About the execution device"
The execution device is not limited to the one provided with the CPU 72 (112) and the ROM 74 (114) to execute software processing. For example, a dedicated hardware circuit such as an ASIC that performs hardware processing on at least a part of what has been software-processed in the above embodiment may be provided. That is, the executing device may have any of the following configurations (a) to (c). (A) A processing device that executes all of the above processing according to a program and a program storage device such as a ROM that stores the program are provided. (B) A processing device and a program storage device that execute a part of the above processing according to a program, and a dedicated hardware circuit that executes the remaining processing are provided. (C) A dedicated hardware circuit for executing all of the above processes is provided. Here, there may be a plurality of software execution devices including a processing device and a program storage device, and a plurality of dedicated hardware circuits.

・ "About storage device"
In the above embodiment, the storage device for storing the related regulation data DR and the storage device (ROM74) for storing the learning program 74b and the control program 74a are different storage devices, but the present invention is not limited to this.

・ "About internal combustion engine"
The internal combustion engine is not limited to one having a port injection valve for injecting fuel into the intake passage 12 as a fuel injection valve, and may be provided with an in-cylinder injection valve for directly injecting fuel into the combustion chamber 24. For example, it may include both a port injection valve and an in-cylinder injection valve.

The internal combustion engine is not limited to the spark ignition type internal combustion engine, and may be, for example, a compression ignition type internal combustion engine that uses light oil or the like as fuel.
・ "About the vehicle"
The vehicle is not limited to a vehicle in which the thrust generator is only an internal combustion engine, and may be, for example, a so-called hybrid vehicle including an internal combustion engine and a rotary electric machine. Further, for example, as the thrust generator, there may be a so-called electric vehicle or a fuel cell vehicle equipped with a rotating electric machine without providing an internal combustion engine.

10 ... Internal combustion engine, 12 ... Intake passage, 14 ... Throttle valve, 16 ... Fuel injection valve, 18 ... Intake valve, 20 ... Cylinder, 22 ... Piston, 24 ... Combustion chamber, 26 ... Ignition device, 28 ... Crankshaft, 30 ... exhaust valve, 32 ... exhaust passage, 34 ... catalyst, 40 ... torque converter, 42 ... lockup clutch, 50 ... transmission, 52 ... input shaft, 54 ... output shaft, 60 ... drive wheel, 70 ... control device, 72 ... CPU, 74 ... ROM, 74a ... Control program, 74b ... Learning program, 74c ... Main program for learning, 76 ... Storage device, 77 ... Communication device, 78 ... Peripheral circuit, 79 ... Local network, 80 ... Airflow meter, 82 ... Throttle sensor, 84 ... Crank angle sensor, 86 ... Accelerator pedal, 88 ... Accelerator sensor, 90 ... Acceleration sensor, 100 ... Network, 110 ... Data analysis center, 112 ... CPU, 114 ... ROM, 114a ... Learning subprogram, 116 ... storage device, 117 ... communication device, 118 ... peripheral circuit, 119 ... local network.

Claims (10)

Equipped with an execution device and a storage device
The storage device stores relationship regulation data that defines the relationship between the state of the vehicle and the behavior variable, which is a variable related to the operation of the electronic device in the vehicle.
The executing device is
The acquisition process for acquiring the detection value of the sensor that detects the state of the vehicle, and
An operation process for operating the electronic device based on the value of the action variable determined by the detection value acquired by the acquisition process and the relational regulation data, and
Based on the detected value acquired by the acquisition process, a reward calculation process that gives a larger reward than when the characteristics of the vehicle meet the criteria and does not meet the criteria.
Input of the vehicle state based on the detected value acquired by the acquisition process, the value of the action variable used for the operation of the electronic device, and the reward corresponding to the operation into a predetermined update map. And the update process to update the related regulation data,
A detection process for detecting that a functional recovery measure has been taken for a part of the parts in the vehicle that affects the state of the vehicle caused by the operation by the operation process.
When it is detected that the function recovery treatment has been performed by the detection process, the relevant regulation data used for the operation process is the post-treatment data which is data before a predetermined period from the time when the function recovery treatment is performed. Switching process to switch to, and
And
The updated map is a vehicle control device that outputs the relevant regulation data updated so as to increase the expected profit for the reward when the electronic device is operated according to the relevant regulation data. The vehicle control device according to claim 1, wherein the switching process includes a process of using the relevant regulation data updated by the update process as the vehicle travels as post-treatment data. The execution device is updated by the update process until the predetermined condition is satisfied, and after the update process is satisfied, the update process is performed separately from the related regulation data which is updated by the update process. The past data maintenance process is executed so that the relevant regulation data in which the update is avoided is stored in the storage device.
The vehicle control device according to claim 1 or 2, wherein the switching process includes a process using the relevant specified data whose state stored in the storage device is maintained by the past data maintenance process as the post-treatment data. When the execution device detects that the function recovery treatment has been performed by the detection processing, the execution device transmits a signal requesting the post-treatment data, and the result of the post-treatment data request processing and the post-treatment data request processing. , The post-treatment data reception process for receiving the transmitted post-treatment data, and the post-treatment data reception process are executed.
The vehicle control device according to claim 1 or 2, wherein the switching process includes a process of switching the related regulation data used for the operation process to the received post-treatment data. The executing device and the storage device according to claim 1 or 2 are provided.
The execution device includes a first execution device mounted on the vehicle and a second execution device different from the in-vehicle device.
The second executing device at least executes the post-treatment data transmission process for transmitting the post-treatment data when it is detected by the detection process that the function recovery treatment has been performed.
The first execution device is a vehicle control system that at least executes post-treatment data reception processing for receiving data transmitted by the acquisition processing, the operation processing, and the post-treatment data transmission processing. The first executing device executes the post-treatment data request processing for transmitting the detection process and the signal requesting the post-treatment data when it is detected that the function recovery treatment has been performed by the detection processing. Item 5. The vehicle control system according to item 5. The post-treatment data transmission process includes a process of transmitting the relevant regulation data as post-treatment data in a vehicle in which the degree of deterioration of the parts is lower than that of the vehicle to which the function recovery treatment has been performed. The vehicle control system described. The vehicle control system according to any one of claims 5 to 7, wherein the update process is executed by the first execution device. A vehicle control device including the first execution device according to any one of claims 5 to 8. A vehicle learning device including the second execution device according to any one of claims 5 to 8.

Images