JP2021143882A - Learning system and learning method for operation inference learning model that controls automatically manipulated robot - Google Patents

Abstract

To provide a learning system and a learning method, with which it is possible to suppress a decrease in the learning accuracy of an operation inference learning model that is attributable to a difference in pedal play amount between a vehicle model and an actual vehicle.SOLUTION: Provided is a learning system 10 comprising an operation inference learning model 70 for inferring a vehicle operation on the basis of the traveling state of a vehicle 2 and an automatically manipulated robot 4, operation including a pedal operation amount of one or both of an accelerator pedal and a brake pedal, the system including a first pedal play amount adjustment unit 54 and a second pedal play amount adjustment unit 55 for inputting the pedal operation amount inferred by the operation inference learning model to a vehicle model on the basis of a difference in pedal play amount between the vehicle and a vehicle model 52 that outputs a simulated traveling state. The simulated traveling state having been adjusted is applied to the operation inference learning model, whereby a learning system for the operation inference learning model that controls the automatically manipulated robot is provided.SELECTED DRAWING: Figure 2

Description

The present invention relates to a learning system and a learning method of an operation inference learning model that controls an autopilot robot.

Generally, when manufacturing and selling vehicles such as ordinary automobiles, the fuel consumption and exhaust gas when the vehicle is driven according to a specific driving pattern (mode) specified by the country or region are measured and displayed. There is a need to.
The mode can be represented by a graph as, for example, the relationship between the time elapsed from the start of traveling and the vehicle speed to be reached at that time. This vehicle speed to be reached is sometimes referred to as a command vehicle speed in terms of a command regarding the speed to be achieved given to the vehicle.
The above tests on fuel consumption and exhaust gas are carried out by placing the vehicle on the chassis dynamometer and driving the vehicle according to the mode by the autopilot robot mounted on the vehicle, the so-called drive robot (registered trademark). Will be done.

The margin of error is specified for the command vehicle speed. If the vehicle speed deviates from the permissible error range, the test becomes invalid, so that the control of the drive robot is required to have high followability to the commanded vehicle speed. For this reason, especially in recent years, the drive robot may be controlled using a machine-learned learning model to infer an operation that causes the vehicle to travel according to a commanded vehicle speed when the current state of the vehicle is input. ..
For example, Patent Document 1 discloses a vehicle driving simulation device, a driver model construction method, and a driver model construction program capable of constructing a driver model that performs a human-like pedal operation by reinforcement learning.
More specifically, the vehicle driving simulation device travels the vehicle model multiple times while changing the gain value of the driver model, and evaluates the changed gain value based on the reward value. The gain of the driver model is set automatically. The above gain value is evaluated not only by the vehicle speed reward function that evaluates the followability of the vehicle speed, but also by the accelerator reward function that evaluates the smoothness of the accelerator pedal operation and the brake reward function that evaluates the smoothness of the brake pedal operation. Is done.
As a vehicle model used in Patent Document 1 and the like, a physical model that imitates an operation is usually created for each component of the vehicle, and a physical model that combines these is created.

Japanese Unexamined Patent Publication No. 2014-115168

In the device as disclosed in Patent Document 1, an operation inference learning model for inferring the operation of the vehicle is learned based on the vehicle model. Therefore, if the reproduction accuracy of the vehicle model is low, the operation inferred by the operation inference learning model may not be suitable for the actual vehicle, no matter how precisely the operation inference learning model is trained.
One of the characteristics of the vehicle that has a great influence on the improvement of the reproduction accuracy of the vehicle model is the pedal play of the accelerator pedal or the brake pedal. As shown in FIG. 11, consider a case where the actuator of the drive robot provided in the seat of the vehicle depresses the pedal from the pedal depressing start position P1 to the depressing limit position P2 in the direction PD. That is, the depression start position P1 corresponds to the case where the pedal opening degree is 0%, and the depression limit position P2 corresponds to the case where the pedal opening degree is 100%. At this time, in reality, the drive by the accelerator pedal or the deceleration by the brake pedal is not started immediately after the pedal is depressed from the depression start position P1, but the pedal is depressed to reach the default drive start position P3. Until then, the pedal operation is not reflected in the behavior of the vehicle. Pedal play refers to such play. Further, the magnitude H of the pedal opening degree from the depression start position P1 to the drive start position P3 is hereinafter referred to as a pedal play amount.

The amount of pedal play greatly affects the absolute amount of pedal operation value when the drive robot is controlled to drive the vehicle.
For example, an operation inference learning model learned using a vehicle model in which the pedal play amount is set to a value smaller than that of the actual vehicle is such that the brake pedal is depressed slightly beyond the drive start position P3 on the vehicle model. Consider the case of inferring various operations. In such a case, the inferred operation may be an operation that does not reach the drive start position P3 in an actual vehicle in which the pedal play amount is larger than that of the vehicle model. That is, even if the operation reasoning learning model intends to infer the operation of lightly depressing the brake pedal, depending on the operation, the brake pedal cannot be depressed deeper than the driving start position P3 of the vehicle, and the brake is actually applied. There can be situations where it doesn't work.

After discovering the difference in pedal play amount between the actual vehicle and the vehicle model as described above, it is conceivable to relearn the vehicle model so that the pedal play amount is appropriate, but this requires a lot of calculation time. However, it cannot be easily implemented.

The problem to be solved by the present invention is the learning accuracy of the operation inference learning model due to the difference in the pedal play amount between the vehicle model and the actual vehicle when the operation inference learning model is machine-learned with the vehicle model as the operation execution target. It is an object of the present invention to provide a learning system and a learning method of an operation inference learning model for controlling an automatic control robot (drive robot), which can easily suppress a decrease in the number of.

The present invention employs the following means in order to solve the above problems. That is, the present invention is mounted on the vehicle and an operation inference learning model that infers the operation of the vehicle so that the vehicle travels according to a specified command vehicle speed based on the traveling state of the vehicle including the vehicle speed. A learning system for an operation inference learning model that controls an automatic control robot, comprising an automatic control robot that runs the vehicle based on the operation, and machine-learning the operation inference learning model. The operation is an accelerator pedal. The vehicle is set to perform a simulated operation including the pedal operation amount of one or both of the pedals and the brake pedal, and a simulated running state imitating the vehicle including the pedal detection amount of the pedal is output. , The pedal operation amount included in the input operation based on the operation inferred by the operation inference learning model based on the difference value between the vehicle model, the pedal play amount of the vehicle, and the pedal play amount of the vehicle model. Is adjusted according to the vehicle model, and the adjusted input operation is input to the vehicle model. The adjusted simulated running state is provided with a second pedal play amount adjusting unit that adjusts the pedal detection amount included in the simulated running state according to the vehicle and generates the adjusted simulated running state. Is applied to the operation inference learning model to provide a learning system for an operation inference learning model that controls an automatic control robot by machine learning the operation inference learning model.

Further, the present invention is mounted on the vehicle and an operation inference learning model for inferring the operation of the vehicle so as to drive the vehicle in accordance with a specified command vehicle speed based on the traveling state of the vehicle including the vehicle speed. A learning method of an operation inference learning model for controlling an automatic control robot, which machine-learns the operation inference learning model with respect to an automatic control robot that runs the vehicle based on the operation, wherein the operation is an accelerator pedal. A vehicle that includes the pedal operation amount of one or both of the brake pedals, is set to simulate the vehicle, and outputs a simulated running state that imitates the vehicle, including the pedal detection amount of the pedal. According to the model, the pedal operation amount included in the input operation based on the operation inferred by the operation inference model is calculated based on the difference value between the pedal play amount of the vehicle and the pedal play amount of the vehicle model. The adjusted and adjusted input operation is input to the vehicle model, and the pedal detection amount included in the simulated running state is adjusted to the vehicle based on the inverted difference value obtained by reversing the positive and negative of the difference value. By adjusting and generating the adjusted simulated running state and applying the adjusted simulated running state to the operation inference learning model, an automatic control robot that machine-learns the operation inference learning model is controlled. A learning method of an operation inference learning model is provided.

According to the present invention, when the operation inference learning model is machine-learned with the vehicle model as the target of operation execution, the learning accuracy of the operation inference learning model is reduced due to the difference in the amount of pedal play between the vehicle model and the actual vehicle. It is possible to provide a learning system and a learning method of an operation inference learning model that controls an automatic control robot (drive robot) that can be easily suppressed.

It is explanatory drawing of the test environment using the autopilot robot (drive robot) in embodiment of this invention. It is a block diagram which describes the process flow at the time of learning of a vehicle learning model of the learning system of the operation reasoning learning model which controls an autopilot robot in the said embodiment. It is a block diagram of the said vehicle learning model. It is a block diagram which describes the process flow at the time of the pre-learning of the operation inference learning model of the learning system of the operation inference learning model which controls the autopilot robot. It is explanatory drawing of the 1st and 2nd pedal play amount adjustment part of the said learning system. It is a block diagram which describes the process flow at the time of reinforcement learning after the pre-learning of the operation inference learning model is completed of the learning system of the operation inference learning model which controls the automatic control robot. It is a flowchart of the learning method of the operation reasoning learning model which controls an autopilot robot in the said embodiment. It is explanatory drawing of the 2nd pedal play amount adjustment part with respect to the 1st modification of the said embodiment. It is explanatory drawing when it took time to infer the vehicle model in the adjustment of the pedal play amount in the 1st modification. It is explanatory drawing of the 2nd pedal play amount adjustment part with respect to the 2nd modification of the said embodiment. It is explanatory drawing of pedal play.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
In the present embodiment, since the drive robot (registered trademark) is used as the autopilot robot, the autopilot robot will be referred to as a drive robot below.

FIG. 1 is an explanatory diagram of a test environment using a drive robot in the embodiment. The test device 1 includes a vehicle 2, a chassis dynamometer 3, and a drive robot 4.
The vehicle 2 is provided on the floor surface. The chassis dynamometer 3 is provided below the floor surface. The vehicle 2 is positioned so that the drive wheels 2a of the vehicle 2 are placed on the chassis dynamometer 3. When the vehicle 2 travels and the drive wheels 2a rotate, the chassis dynamometer 3 rotates in the opposite direction.
The drive robot 4 is mounted on the driver's seat 2b of the vehicle 2 to drive the vehicle 2. The drive robot 4 includes a first actuator 4c and a second actuator 4d, which are provided so as to come into contact with the accelerator pedal 2c and the brake pedal 2d of the vehicle 2, respectively.

The drive robot 4 is controlled by the learning control device 11 described in detail later. The learning control device 11 changes and adjusts the opening degrees of the accelerator pedal 2c and the brake pedal 2d of the vehicle 2 by controlling the first actuator 4c and the second actuator 4d of the drive robot 4.
The learning control device 11 controls the drive robot 4 so that the vehicle 2 travels according to a specified command vehicle speed. That is, the learning control device 11 controls the traveling of the vehicle 2 so as to follow the defined traveling pattern (mode) by changing the opening degrees of the accelerator pedal 2c and the brake pedal 2d of the vehicle 2. More specifically, the learning control device 11 controls the traveling of the vehicle 2 so as to follow the commanded vehicle speed, which is the vehicle speed to reach each time, as time elapses from the start of traveling.

The learning system 10 includes the test device 1 and the learning control device 11 as described above.
The learning control device 11 includes a drive robot control unit 20 and a learning unit 30.
The drive robot control unit 20 controls the drive robot 4 by generating a control signal for controlling the drive robot 4 and transmitting the control signal to the drive robot 4. The learning unit 30 performs machine learning as described later to generate a vehicle learning model, an operation inference learning model, and a value inference learning model. The control signal for controlling the drive robot 4 as described above is generated by the operation inference learning model.
The drive robot control unit 20 is, for example, an information processing device such as a controller provided outside the housing of the drive robot 4. The learning unit 30 is an information processing device such as a personal computer.

FIG. 2 is a block diagram of the learning system 10. In FIG. 2, the lines connecting the components are shown only for those that have data transmission / reception during machine learning of the vehicle learning model, and therefore do not indicate the transmission / reception of all data between the components. No.
The test device 1 includes a vehicle condition measuring unit 5 in addition to the vehicle 2, the chassis dynamometer 3, and the drive robot 4 as described above. The vehicle state measuring unit 5 is various measuring devices for measuring the state of the vehicle 2. The vehicle state measuring unit 5 may be, for example, a camera or an infrared sensor for measuring the amount of operation of the accelerator pedal 2c or the brake pedal 2d.

The drive robot control unit 20 includes a pedal operation pattern generation unit 21, a vehicle operation control unit 22, and a drive state acquisition unit 23. The learning unit 30 includes a command vehicle speed generation unit 31, an inference data molding unit 32, a learning data molding unit 33, a learning data generation unit 34, a learning data storage unit 35, a reinforcement learning unit 40, and a test device model 50. The reinforcement learning unit 40 includes an operation content inference unit 41, a state behavior value inference unit 42, and a reward calculation unit 43. The test device model 50 includes a drive robot model 51, a vehicle model 52, a chassis dynamometer model 53, a first pedal play amount adjusting unit 54, and a second pedal play amount adjusting unit 55.
Each component of the learning control device 11 other than the learning data storage unit 35 may be, for example, software or a program executed by the CPU in each of the above-mentioned information processing devices. Further, the learning data storage unit 35 may be realized by a storage device such as a semiconductor memory or a magnetic disk provided inside or outside each of the information processing devices.

As will be described later, the operation content inference unit 41 infers the operation of the vehicle 2 after the time so as to follow the commanded vehicle speed based on the traveling state at a certain time. In order to effectively infer the operation of the vehicle 2, the operation content inference unit 41 is provided with a machine learning device as will be described later, and after the operation of the drive robot 4 based on the inferred operation. A learning model (operation inference learning model) 70 is generated by machine learning the machine learning device based on the reward calculated based on the running state at the time of. When actually controlling the running of the vehicle 2 for performance measurement, the operation content inference unit 41 infers the operation of the vehicle 2 by using the operation inference learning model 70 for which this learning has been completed.
That is, the learning system 10 is roughly divided into two types of operations: learning the operation during reinforcement learning and inferring the operation when controlling the running of the vehicle for performance measurement. In order to simplify the explanation, first, each component of the learning system 10 at the time of learning the operation is explained, and then the behavior of each component at the time of inferring the operation at the time of measuring the performance of the vehicle is explained. do.

First, the behavior of the components of the learning control device 11 at the time of learning the operation will be described.
The learning control device 11 collects running record data (running record) used at the time of learning as a running record prior to learning the operation. Specifically, the drive robot control unit 20 generates operation patterns for measuring vehicle characteristics of the accelerator pedal 2c and the brake pedal 2d, thereby controlling the vehicle 2 to travel and collecting travel record data.
The pedal operation pattern generation unit 21 generates operation patterns for measuring vehicle characteristics of the pedals 2c and 2d. As the pedal operation pattern, for example, in another vehicle similar to the vehicle 2, the actual value of the pedal operation when traveling in the WLTC (World Harmonized Light Vehicles Test Cycle) mode or the like can be used.
The pedal operation pattern generation unit 21 transmits the generated pedal operation pattern to the vehicle operation control unit 22.

The vehicle operation control unit 22 receives the pedal operation pattern from the pedal operation pattern generation unit 21, converts it into commands to the first and second actuators 4c and 4d of the drive robot 4, and converts the pedal operation pattern into commands to the drive robot 4. Send.
When the drive robot 4 receives a command to the actuators 4c and 4d, the drive robot 4 causes the vehicle 2 to travel on the chassis dynamometer 3 based on the command.
The drive state acquisition unit 23 acquires the actual drive state of the drive robot 4, such as the positions of the actuators 4c and 4d. As the vehicle 2 travels, the traveling state of the vehicle 2 changes sequentially. The running state of the vehicle 2 is measured by various measuring instruments provided in the drive state acquisition unit 23, the vehicle state measurement unit 5, and the chassis dynamometer 3. For example, the drive state acquisition unit 23 measures the detected amount of the accelerator pedal 2c and the detected amount of the brake pedal 2d as the running state as described above. Further, the measuring instrument provided in the chassis dynamometer 3 measures the vehicle speed as a traveling state.
The measured running state of the vehicle 2 is transmitted to the learning data forming unit 33 of the learning unit 30.
The learning data forming unit 33 receives the traveling state of the vehicle 2, converts the received data into formats used in various subsequent learnings, and stores the received data in the learning data storage unit 35 as traveling record data.

When the traveling state of the vehicle 2, that is, the collection of the traveling record data is completed, the learning data generation unit 34 acquires the traveling record data from the learning data storage unit 35, forms it into an appropriate format, and transmits it to the test apparatus model 50.
The vehicle model 52 of the test device model 50 of the learning unit 30 acquires the running record data formed from the learning data generation unit 34, and uses this to machine-learn the machine learning device 60 to obtain the vehicle learning model 60. Generate. The vehicle learning model 60 is set to perform a simulated operation of the vehicle 2 based on the travel record data which is the actual travel record of the vehicle 2. In the present embodiment, the machine learning is performed, and when an operation for the vehicle 2 is received, this is performed. Based on this, a simulated running state imitating the vehicle 2 is output. That is, the machine learning device 60 of the vehicle model 52 generates a learned model 60 in which appropriate learning parameters are learned, which is used as a program module that is a part of artificial intelligence software.
In this embodiment, the vehicle learning model 60 is realized by a neural network.
Hereinafter, for the sake of simplicity, both the machine learning device included in the vehicle model 52 and the learning model generated by learning the machine learning device will be referred to as a vehicle learning model 60.

FIG. 3 is a block diagram of the vehicle learning model 60. In the present embodiment, the vehicle learning model 60 is realized by a fully connected neural network of all five layers with three intermediate layers. The vehicle learning model 60 includes an input layer 61, an intermediate layer 62, and an output layer 63. In FIG. 3, each layer is drawn as a rectangle, and the nodes included in each layer are omitted.

In the present embodiment, the input of the vehicle learning model 60 includes a series of vehicle speeds from the past to the reference time for a predetermined first time in the travel record data with an arbitrary reference time as a base point. Further, in the present embodiment, the input of the vehicle learning model 60 is the sequence of the operation amount of the accelerator pedal 2c and the operation amount of the brake pedal 2d from the reference time to the future time by a predetermined second time. Includes series. The series of the operation amount of the accelerator pedal 2c and the series of the operation amount of the brake pedal 2d are actually the detection amounts of the accelerator pedal 2c after the reference time in the running record data stored in the learning data storage unit 35. And the detected amount of the brake pedal 2d, which are input to the vehicle learning model 60 as an operation applied to the vehicle 2 at the reference time.
The input layer 61 is a vehicle speed series i1 which is a series of vehicle speeds as described above, a model input which is a series of operation amounts of the accelerator pedal 2c, a model input which is a series of operation amounts of the accelerator pedal operation amount series im2, and a model input which is a series of operation amounts of the brake pedal 2d. It is provided with input nodes corresponding to each of the brake pedal operation amount series im3.
As described above, each of the inputs i1, im2, and im3 is a series, and each is realized by a plurality of values. For example, in FIG. 3, the input corresponding to the vehicle speed series i1 shown as one rectangle is actually provided with an input node so as to correspond to each of a plurality of values of the vehicle speed series i1. There is.
The vehicle model 52 stores the value of the corresponding travel record data in each input node.

The intermediate layer 62 includes a first intermediate layer 62a, a second intermediate layer 62b, and a third intermediate layer 62c.
In each node of the intermediate layer 62, from each node of the previous layer (for example, the input layer 61 in the case of the first intermediate layer 62a and the first intermediate layer 62a in the case of the second intermediate layer 62b), the previous layer. An operation is performed based on the value stored in each node of the above and the weight from each node of the previous layer to the node of the intermediate layer 62, and the operation result is stored in the node of the intermediate layer 62.
In the output layer 63, the same calculation as in each of the intermediate layers 62 is performed, and the calculation result is stored in each output node provided in the output layer 63.
In the present embodiment, the output of the vehicle learning model 60 is an estimated vehicle speed series o1 which is a series of estimated vehicle speeds from the reference time to a future time (later time) by a predetermined third time. It is a simulated running state om including a model output accelerator pedal detection amount series om2 which is a series of detection amounts of the accelerator pedal 2c and a model output brake pedal detection amount series om3 which is a series of detection amounts of the brake pedal 2d. In FIG. 3, each of the simulated running states om shown as one rectangle is actually provided with an output node so as to correspond to each of the above-mentioned plurality of values.

In the vehicle learning model 60, the running record at the reference time is input as described above, and learning is performed so that the simulated running state om that imitates the running of the vehicle 2 at a later time can be output.
More specifically, the vehicle model 52 uses the running record from the reference time to the future time by the third hour, which is separately transmitted from the learning data storage unit 35 via the learning data generation unit 34, as teacher data. Receive. The vehicle model 52 reverses the error of each parameter constituting the neural network, such as a weight and a bias value, so that the training data and the mean square error of the simulated running state om output by the vehicle learning model 60 become small. Adjust by propagation method and stochastic gradient descent method.
The vehicle model 52 calculates the least squares error of the teacher data and the simulated running state om each time while repeating the learning of the vehicle learning model 60, and if this is smaller than a predetermined value, the learning of the vehicle learning model 60 ends.

When the learning of the vehicle learning model 60 is completed, the reinforcement learning unit 40 of the learning system 10 pre-learns the operation inference learning model 70 for inferring the operation of the vehicle 2 provided in the operation content inference unit 41. FIG. 4 is a block diagram of the learning system 10 showing the data transmission / reception relationship at the time of pre-learning. In the present embodiment, the operation reasoning learning model 70 is machine-learned by reinforcement learning. That is, the operation inference learning model 70 becomes a learned model in which appropriate learning parameters are learned, which is used as a program module that is a part of artificial intelligence software by learning the machine learning device.
The learning system 10 reinforces the operation inference learning model 70 in advance by applying the simulated running state output by the vehicle learning model 60 for which learning has already been completed to the operation inference learning model 70. As will be described later, after the reinforcement learning of the operation inference learning model 70 progresses and the preliminary reinforcement learning is completed, the vehicle 2 is actually driven and acquired based on the operation output by the operation inference learning model 70. By applying the traveling state to the operation inference learning model 70, the operation inference learning model 70 is further strengthened and learned. In this way, the learning system 10 changes the execution target of the inferred operation and the acquisition target of the running state from the vehicle learning model 60 to the actual vehicle 2 according to the learning stage of the operation inference learning model 70.

As will be described later, the operation content inference unit 41 outputs the operation of the vehicle 2 from the present time to the future time by the third time by the operation inference learning model 70 in the middle of learning, and outputs the operation of the vehicle 2 to the drive robot. Send to model 51. In the present embodiment, the operation content inference unit 41 outputs, in particular, a sequence of operations of the accelerator pedal 2c and the brake pedal 2d, that is, a pedal operation amount.
By learning the vehicle learning model 60, the test device model 50 is configured to simulate each of the test devices 1 as a whole. The test apparatus model 50 receives a sequence of operations.

The drive robot model 51 is configured to simulate the drive robot 4. Based on the operation inferred by the operation inference learning model 70 received from the operation content inference unit 41, the drive robot model 51 converts the expression of the operation system into the value of the actual pedal operation amount for the vehicle 2 and inputs it. Generate an operation. More specifically, the drive robot model 51 generates an accelerator pedal operation amount series i2 and a brake pedal operation amount series i3, which are a series of pedal operation amounts, as input operations and transmits them to the first pedal play amount adjustment unit 54.
The chassis dynamometer model 53 is configured to simulate the chassis dynamometer 3. The chassis dynamometer 3 detects the vehicle speed of the vehicle learning model 60 during simulated running and records it internally at any time. The chassis dynamometer model 53 generates a vehicle speed series i1 from the record of the past vehicle speed and transmits it to the first pedal play amount adjusting unit 54.

The first pedal play amount adjusting unit 54 adjusts the input operation received from the drive robot model 51, that is, the pedal operation amount included in the accelerator pedal operation amount series i2 and the brake pedal operation amount series i3, as will be described in detail later. Then, the model input accelerator pedal operation amount series (adjusted input operation) im2 and the model input brake pedal operation amount series (adjusted input operation) im3 input to the vehicle model 52 are generated to generate the chassis dynamometer model 53. It is transmitted to the vehicle model 52 together with the vehicle speed series i1 received from.
The vehicle model 52 receives the vehicle speed series i1, the model input accelerator pedal operation amount series im2, and the model input brake pedal operation amount series im3, and inputs these to the vehicle learning model 60. When the vehicle learning model 60 outputs the simulated running state om, the vehicle model 52 transmits the simulated running state om to the chassis dynamometer model 53 and the second pedal play amount adjusting unit 55.
The second pedal play amount adjusting unit 55 adjusts the pedal detection amount included in the model output accelerator pedal detection amount series om2 and the model output brake pedal detection amount series om3 in the simulated running state om, as will be described in detail later. Then, the accelerator pedal detection amount series o2 and the brake pedal detection amount series o3 are generated.
The chassis dynamometer model 53 detects the vehicle speed included in the simulated running state om and updates the internal state.
The accelerator pedal detection amount series o2, the brake pedal detection amount series o3, and the vehicle speed series held in the chassis dynamometer model 53 adjusted by the second pedal play amount adjusting unit 55 are set as the adjusted simulated running state o. It is transmitted to the inference data forming unit 32 and the reinforcement learning unit 40.

The command vehicle speed generation unit 31 holds the command vehicle speed generated based on the information regarding the mode. The command vehicle speed generation unit 31 generates a sequence of command vehicle speeds to be followed by the vehicle learning model 60 from the present time to a future time by a predetermined fourth time, and transmits the sequence to the inference data forming unit 32.
The inference data forming unit 32 receives the adjusted simulated running state o and the command vehicle speed sequence, appropriately forms them, and then transmits them to the reinforcement learning unit 40.
The reinforcement learning unit 40 transmits these adjusted simulated running states o and the command vehicle speed series to the operation content inference unit 41 as running states.

When the operation content inference unit 41 receives the traveling state at a certain time, the operation inference learning model 70 during learning infers a series of operations after the time based on the driving state.
In the present embodiment, the operation inference learning model 70 is a neural network including an input layer having input nodes corresponding to each of the running states, a plurality of intermediate layers, and an output layer having a plurality of output nodes. be.
When the corresponding running state value is input to each of the input nodes, an operation based on the weight is performed, and the operation result is performed for each of the intermediate nodes of the intermediate layer provided as the next stage of the input node. Is stored. Such an operation and storage of the operation result in the intermediate node of the next stage are sequentially executed for each intermediate layer. Finally, the same operation is performed based on the operation result stored in the intermediate node in the intermediate layer of the final stage, and the result is stored in the output node.
Each of the output nodes of the operation inference learning model 70 is provided so as to correspond to each of the operations. In the present embodiment, the objects of operation are the accelerator pedal 2c and the brake pedal 2d, and in response to this, the operation inference learning model 70 sets, for example, a sequence of accelerator pedal operations and a sequence of brake pedal operations as operations. Infer.

The operation content reasoning unit 41 transmits the accelerator pedal operation and the brake pedal operation generated in this manner to the drive robot model 51. Based on this, the drive robot model 51 generates an accelerator pedal operation amount series i2 and a brake pedal operation amount series i3, which are input operations, and transmits them to the first pedal play amount adjustment unit 54. The first pedal play amount adjusting unit 54 adjusts the input operation to generate the model input accelerator pedal operation amount series im2 and the model input brake pedal operation amount series im3, and transmits the model input brake pedal operation amount series im3 to the vehicle learning model 60. The vehicle learning model 60 receives these and infers the next simulated running state om. The second pedal play amount adjusting unit 55 generates the simulated running state o adjusted from the simulated running state om. In this way, the next running state is generated.
The learning of the operation inference learning model 70, that is, the adjustment of the values of each parameter constituting the neural network by the error back propagation method and the stochastic gradient descent method is not performed at this stage, and the operation inference learning model 70 infers the operation. Only. The learning of the operation inference learning model 70 is later performed in association with the learning of the value inference learning model 80.

The reward calculation unit 43 uses an appropriately designed formula based on the running state, the operation inferred by the operation inference learning model 70 corresponding to the running state, and the running state newly generated based on the operation. Calculate the reward. The reward is designed so that the operation and the newly generated running conditions associated therewith have an undesirably small value and a desirablely large value. The state action value inference unit 42, which will be described later, calculates the action value so that the larger the reward, the higher the action value, and the operation inference learning model 70 outputs an operation such that the action value becomes higher. Will be done.
The reward calculation unit 43 transmits the running state, the operation inferred corresponding to the running state, the running state newly generated based on the operation, and the calculated reward to the learning data forming unit 33. The learning data forming unit 33 appropriately forms these and stores them in the learning data storage unit 35. These data are used for learning the value inference learning model 80, which will be described later.
In this way, the inference of the operation by the operation content inference unit 41, the inference of the simulated running state om by the vehicle model 52 corresponding to this operation, and the calculation of the reward are sufficient data for learning the value inference learning model 80. Is repeated until is accumulated.

When a sufficient amount of running data for learning the value inference learning model 80 is accumulated in the learning data storage unit 35, the state behavior value inference unit 42 learns the value inference learning model 80. The value inference learning model 80 becomes a learned model in which appropriate learning parameters are learned, which is used as a program module that is a part of artificial intelligence software by learning a machine learning device.
As a whole, the reinforcement learning unit 40 calculates an action value indicating how appropriate the operation inferred by the operation inference learning model 70 is, and the operation inference learning model 70 outputs an operation such that the action value becomes high. As you can see, reinforcement learning is performed. The action value is expressed as a function designed so that the larger the reward, the higher the action value, with the running state and the operation for it as arguments. In the present embodiment, the calculation of this function is performed by the learning model 80 as a function approximator designed to output the action value by inputting the running state and the operation.

The operation learning data generation unit 34 forms the learning data in the learning data storage unit 35 and transmits it to the state / behavior value inference unit 42.
The state-behavioral value inference unit 42 receives the formed learning data and causes the value inference learning model 80 to be machine-learned.
In the present embodiment, the value inference learning model 80 is a neural network including an input layer having input nodes corresponding to each of a running state and an operation, a plurality of intermediate layers, and an output node corresponding to an action value. Is. Since the value inference learning model 80 is realized by a neural network having the same structure as the operation inference learning model 70, detailed structural explanation is omitted.

The state action value inference unit 42 reduces the TD (Temporal Difference) error, that is, the error between the action value before executing the operation and the action value after executing the operation, and outputs an appropriate value as the action value. The values of each parameter constituting the neural network, such as the weight and bias values, are adjusted by the error back propagation method and the stochastic gradient descent method. In this way, the value inference learning model 80 is trained so that the operations inferred by the current operation inference learning model 70 can be appropriately evaluated.
As the learning of the value inference learning model 80 progresses, the value inference learning model 80 comes to output a more appropriate action value value. That is, since the value of the action value output by the value inference learning model 80 is different from that before learning, the operation inference learning model 70 designed to output an operation that increases the action value is updated accordingly. There is a need. Therefore, the operation content inference unit 41 learns the operation inference learning model 70.
Specifically, the operation content inference unit 41 uses a negative value of the action value as a loss function, and outputs an operation that makes it as small as possible, that is, increases the action value. The operation inference learning model 70 is trained by adjusting the values of each parameter constituting the neural network, such as the values, by the error back propagation method and the stochastic gradient descent method.
When the operation inference learning model 70 is learned and updated, the output operation changes, so the driving data is accumulated again, and the value inference learning model 80 is learned based on this.
In this way, the learning unit 30 reinforces the learning models 70 and 80 by repeating the learning of the operation inference learning model 70 and the value inference learning model 80.

The learning unit 30 executes reinforcement learning using the vehicle learning model 60 as an operation execution target as the pre-learning until the pre-learning end criterion is satisfied.

Next, the behaviors of the first pedal play amount adjusting unit 54 and the second pedal play amount adjusting unit 55 in the pre-learning of the operation inference learning model 70 using the vehicle learning model 60 as described above will be described.
In order for the operation inference learning model 70 to be learned accurately, it is necessary to improve the reproduction accuracy of the vehicle 2 in the vehicle learning model 60 in the vehicle model 52. At this time, what is particularly important is to accurately reproduce the pedal play amount of the accelerator pedal 2c and the brake pedal 2d of the vehicle 2 also in the vehicle model 52.
However, for example, when collecting travel record data to be used as learning data when learning the vehicle learning model 60, there may be a problem in installing an instrument that detects the pedal operation amount of the accelerator pedal 2c and the brake pedal 2d. For example, when the running record data is collected under such a situation that the pedal play amount is different from the actual one, the pedal play amount of the vehicle model 52 becomes a value different from the original pedal play amount of the vehicle 2. ..
Alternatively, when the drive robot 4 is re-installed in the vehicle 2, if the drive robot 4 is installed at a position different from the previous one, or if the pedal is calibrated, the pedal play is also performed in the same vehicle 2. The amount can fluctuate.
In such a case, the first pedal play amount adjusting unit 54 and the second pedal play amount adjusting unit 55 adjust the pedal play amount between the vehicle 2 and the vehicle model 52.

FIG. 5 is an explanatory diagram of the first and second pedal play amount adjusting units 54 and 55.
As will be described later, the operation inferred by the operation inference learning model 70 is transmitted to the drive robot 4 via the vehicle operation control unit 22 and used to operate the vehicle 2. Therefore, the operation output by the operation inference learning model 70 reflects the amount of pedal play of the vehicle 2.
Therefore, the first pedal play amount adjusting unit 54 adjusts the pedal operation amount included in the input operation based on the operation inferred by the operation inference learning model 70 according to the vehicle model 52, and adjusts the input operation. Is generated and the adjusted input operation is input to the vehicle model 52.

ここで、関数ｆは、次の式２として示されるような、ペダル操作量を下限である０％から上限である１００％の範囲内に収めるための関数である。

More specifically, the first pedal play amount adjusting unit 54 is an input operation generated by the drive robot model 51 based on the operation inferred by the operation inference learning model 70, that is, the accelerator pedal operation amount series i2 and the brake pedal operation amount. Receives the series i3.
Pedal play amount _{d r} of the vehicle 2, the pedal play amount of the vehicle model 52 is taken as _{d v,} the pedal play amount difference (difference value) is defined as _d diff _{= d} r -d _v. The first pedal play amount adjusting unit 54 adjusts the accelerator pedal operation amount series i2 and the brake pedal operation amount series i3, that is, the pedal operation amount a _ri included in the input operation to the vehicle model 52 by the following equation 1. It is adjusted and the adjusted pedal operation amount _avi is generated.

Here, the function f is a function for keeping the pedal operation amount within the range of 0%, which is the lower limit, and 100%, which is the upper limit, as shown by the following equation 2.

Thus, the first pedal play amount adjusting unit 54, based on a pedal play amount d _r of the vehicle 2, the pedal play amount difference d _diff is a difference value between the pedal play amount d _v of the vehicle model 52, an input operation _{The pedal operation amount a r-i} included in the accelerator pedal operation amount series i2 and the brake pedal operation amount series i3 is adjusted according to the vehicle model 52 by subtracting the difference value from the pedal operation amount a _r-i. do. As a result, the first pedal play amount adjusting unit 54 generates the adjusted input operation, that is, the model input accelerator pedal operation amount series im2 and the model input brake pedal operation amount series im3.
_{The value a r-i-} d _diff passed to the function f in the above equation 1 can be transformed into (a _r-i- d _r ) + d _v. The value of (a _{r-i -d r),} upon operating the pedal in the vehicle 2, is the amount the pedal is depressed beyond the drive start position of the pedal. The value of a _v-i of equation 1, this is a pedal play amount d _v of the vehicle model 52 is added the value, the pedal play amount d _v of the vehicle model 52 to the amount depressed beyond the drive starting position It is a reflected value.
Thus, for example, as shown in FIG. 5, the range R1 which is located above the pedal play amount d _r of the vehicle 2 in the accelerator pedal operation amount sequence i2, the model input accelerator pedal operation amount which is converted by Equation 1 also in series im2, it is located above the pedal play amount d _v of the vehicle model 52.

Incidentally, the pedal play amount d _r, d _v, for example the accelerator pedal operation amount sequence i2 and the brake pedal operation amount sequence i3 in FIG. 5, the model input accelerator pedal operation amount sequence im2 and model input different values for the brake pedal operation amount sequence im3 As shown as, the accelerator pedal 2c and the brake pedal 2d can take different values.
Therefore, the pedal play amount difference d _diff can also have different values between the accelerator pedal 2c and the brake pedal 2d.
Therefore, the above equation 1 is actually different for the accelerator pedal 2c and the brake pedal 2d. The first pedal play amount adjusting unit 54 applies Equation 1 in which the corresponding values are reflected to the respective operating amounts of the accelerator pedal 2c and the brake pedal 2d.
This also applies to the formula 3 described below with respect to the second pedal play amount adjusting unit 55, and the formulas 4 and 5 described later as a modification of the present embodiment.

The second pedal play amount adjusting unit 55 inputs the model input accelerator pedal operation amount series im2 and the model input brake pedal operation amount series im3 to the vehicle model 52.
The vehicle model 52 infers the simulated running state om, that is, the model output accelerator pedal detection amount series om2 and the model output brake pedal detection amount series om3 by the vehicle learning model 60.

Second pedal play amount adjustment unit 55 receives a simulated running state om, pedal detected amount a _v-o contained therein, and adjusted for vehicle 2 by the following equation 3, the pedal detection amount which is adjusted Generate a _r-o.

Thus, the second pedal play amount adjustment unit 55, a pedal detected amount a _v-o contained in the simulated running state om adjusted in accordance with the vehicle 2, adjusted simulated running state o, i.e. the accelerator pedal detected The quantity series o2 and the brake pedal detection quantity series o3 are generated.
Of formula 3 above, the value _{a v-o + d diff} passed to the function f _can be modified with _{(a v-o -d v)} + d r. The value of _{(a v-o} -d _v), in the vehicle model 52, is the amount the pedal is depressed beyond the drive start position of the pedal. The value of a _r-o of the formula 3, this is a value that pedal play amount d _r of the vehicle 2 is added, the pedal play amount d _r of the vehicle 2 is reflected in the amounts depressed beyond the drive starting position It is a value.
Thus, for example, as shown in FIG. 5, the range located above the pedal play amount d _v of the vehicle model 52 in the model output accelerator pedal detected amount sequence om2 R2 is an accelerator pedal detection which is converted by Equation 3 also in the amount sequence o2, it is located above the pedal play amount d _r of the vehicle 2.

The above equation 3 is a symmetric operation with the equation 1.
That is, in Equation 1, the pedal play amount difference (difference value) d _diff is subtracted _{from the pedal operation amount a r-i included in the input operation.}
On the other hand, in Equation 3, the value of _{−d diff} , which is a value obtained by reversing the positive / negative of the pedal play amount difference d _{diff from the pedal detection amount a r— included in the simulated running state om (reversal difference value).} It is an operation to subtract.
The above-described operation, the second pedal play amount adjustment unit 55, a pedal detected amount a _v-o contained in the simulated running state om, inversion based on a difference value, and more particularly pedal detected amount a _v- By adding the pedal play amount difference (difference value) d _{diff to o} , the pedal detection amount is adjusted according to the vehicle 2, and the adjusted simulated running state o is generated.
The adjusted simulated running state o generated by the second pedal play amount adjusting unit 55 is applied to the operation inference learning model 70, so that the operation inference learning model 70 is machine-learned.

When the pre-learning using the vehicle learning model 60 of the operation inference learning model 70 and the value inference learning model 80 as the execution target of the operation is completed, the learning unit 30 operates the actual vehicle 2 instead of the vehicle learning model 60. The operation inference learning model 70 and the value inference learning model 80 are further strengthened and learned as execution targets. FIG. 6 is a block diagram of the learning system 10 showing the data transmission / reception relationship at the time of reinforcement learning after the completion of the pre-learning.

The operation content inference unit 41 outputs the operation of the vehicle 2 from the present time to the future time for the third time, and transmits this to the vehicle operation control unit 22.
The vehicle operation control unit 22 converts the received operation into commands to the first and second actuators 4c and 4d of the drive robot 4 and transmits them to the drive robot 4.
When the drive robot 4 receives a command to the actuators 4c and 4d, the drive robot 4 causes the vehicle 2 to travel on the chassis dynamometer 3 based on the command.
The chassis dynamometer 3 and the vehicle condition measuring unit 5 detect the vehicle speed of the vehicle 2 and the operating amounts of the accelerator pedal 2c and the brake pedal 2d, generate their respective series, and transmit them to the inference data forming unit 32.
The command vehicle speed generation unit 31 generates a command vehicle speed series and transmits it to the inference data forming unit 32.
The inference data forming unit 32 receives each series, appropriately forms them, and then transmits them to the reinforcement learning unit 40 as a running state.

The reinforcement learning unit 40 uses each of the above series instead of the adjusted simulated running state o generated by the test device model 50, as described above, as in the case of the pre-learning described with reference to FIG. The learning data is stored in the learning data storage unit 35 using the actual vehicle 2 as the execution target of the operation. When a sufficient amount of driving data is accumulated, the reinforcement learning unit 40 learns the value inference learning model 80, and then learns the operation inference learning model 70.
The learning unit 30 reinforces the learning models 70 and 80 by repeating the accumulation of learning data and the learning of the operation inference learning model 70 and the value inference learning model 80.

The learning unit 30 executes reinforcement learning using the vehicle 2 as an operation execution target until the learning end criterion is satisfied.

Next, the behavior of each component of the learning system 10 in the case of inferring the operation when measuring the performance of the vehicle 2, that is, after the reinforcement learning of the operation inference learning model 70 is completed will be described.

The vehicle speed of the vehicle 2, the detection amount of the accelerator pedal 2c, the detection amount of the brake pedal 2d, etc. are measured by various measuring instruments provided in the drive state acquisition unit 23, the vehicle state measurement unit 5, and the chassis dynamometer 3. NS. These values are transmitted to the inference data forming unit 32.
The command vehicle speed generation unit 31 generates a command vehicle speed series and transmits it to the inference data forming unit 32.
The inference data molding unit 32 receives the vehicle speed, the detected amount of the accelerator pedal 2c, the detected amount of the brake pedal 2d, and the commanded vehicle speed series, and after appropriately molding, transmits them to the reinforcement learning unit 40 as a running state.
When the operation content inference unit 41 receives the traveling state, the operation content inference unit 41 infers the operation of the vehicle 2 by the learned operation inference learning model 70 based on the received driving state.
Operation content The inference unit 41 transmits the inferred operation to the vehicle operation control unit 22.
The vehicle operation control unit 22 receives an operation from the operation content inference unit 41, and operates the drive robot 4 based on this operation.

Next, a learning method of the operation inference learning model 70 for controlling the drive robot 4 using the above learning system 10 will be described with reference to FIGS. 1 to 6 and 7. FIG. 7 is a flowchart of the learning method.
The learning control device 11 collects running record data (running record) used at the time of learning as a running record prior to learning the operation. Specifically, the drive robot control unit 20 generates an operation pattern for measuring vehicle characteristics of the accelerator pedal 2c and the brake pedal 2d, thereby controlling the vehicle 2 to travel and collecting travel performance data (step S1). ).
The vehicle model 52 acquires the travel record data formed from the learning data generation unit 34, and uses this to perform machine learning on the machine learning device 60 to generate the vehicle learning model 60 (step S3).

When the learning of the vehicle learning model 60 is completed, the reinforcement learning unit 40 of the learning system 10 pre-learns the operation inference learning model 70 that infers the operation of the vehicle 2 (step S5). More specifically, the learning system 10 reinforces the operation inference learning model 70 in advance by applying the simulated running state output by the vehicle learning model 60 for which learning has already been completed to the operation inference learning model 70. At this time, the first pedal play amount adjusting unit 54 and the second pedal play amount adjusting unit 55 adjust the pedal play amount between the vehicle 2 and the vehicle model 52 by using the formulas 1 to 3.
The learning unit 30 executes reinforcement learning using the vehicle learning model 60 as an operation execution target as the pre-learning until the pre-learning end criterion is satisfied. If the pre-learning end criterion is not satisfied (No in step S7), the pre-learning is continued. When the pre-learning end criterion is satisfied (Yes in step S7), the pre-learning ends.

When the pre-learning using the vehicle learning model 60 of the operation inference learning model 70 and the value inference learning model 80 as the execution target of the operation is completed, the learning unit 30 operates the actual vehicle 2 instead of the vehicle learning model 60. The operation inference learning model 70 and the value inference learning model 80 are further strengthened and learned as execution targets (step S9).

Next, the effects of the learning system and learning method of the operation inference learning model that controls the drive robot will be described.

The learning system 10 of the present embodiment includes an operation inference learning model 70 that infers the operation of the vehicle 2 so that the vehicle 2 travels according to a specified command vehicle speed based on the traveling state of the vehicle 2 including the vehicle speed, and the vehicle. A learning system for an operation inference learning model 70 that controls a drive robot 4 and is equipped with a drive robot (automatic control robot) 4 that is mounted on the vehicle 2 and runs the vehicle 2 based on operations, and machine-learns the operation inference learning model 70. a 10, the operation includes a pedal operation amount both of the accelerator pedal 2c and the brake pedal 2d, is configured to simulate operation of the vehicle 2, includes pedals 2c, a pedal detected amount a _v-o of 2d, and outputs a simulated running state om simulating a vehicle 2, based on the vehicle model 52, and the pedal play amount d _r of the vehicle 2, the difference value d _diff pedal play amount d _v of the vehicle model 52, the operation inference learning model _{The pedal operation amount a r-i} included in the input operations i2 and i3 based on the operation inferred by 70 is adjusted according to the vehicle model 52, and the adjusted input operations im2 and im3 are input to the vehicle model 52. Based on the first pedal play amount adjusting unit 54 and _{the inverted difference value −d diff in} which the positive and negative of the _{difference value d diff are reversed, the pedal detection amount a v o} included in the simulated running state om is adjusted to the vehicle 2. An operation inference learning model is provided by providing a second pedal play amount adjusting unit 55 that adjusts and generates an adjusted simulated running state o, and by applying the adjusted simulated running state o to the operation inference learning model 70. Machine learning 70.
Further, the learning control method of the present embodiment includes an operation inference learning model 70 that infers the operation of the vehicle 2 so as to cause the vehicle 2 to travel according to a specified command vehicle speed based on the traveling state of the vehicle 2 including the vehicle speed. , Learning the operation inference learning model 70 that controls the drive robot 4 by machine learning the operation inference learning model 70 with respect to the drive robot (automatic control robot) 4 that is mounted on the vehicle 2 and runs the vehicle 2 based on the operation. a method, operating comprises pedal operation amount both of the accelerator pedal 2c and the brake pedal 2d, is configured to simulate operation of the vehicle 2, includes pedals 2c, a pedal detected amount a _v-o of 2d, in accordance with the vehicle model 52 for outputting a simulated running state om simulating a vehicle 2, based on a pedal play amount d _r of the vehicle 2, the difference value d _diff pedal play amount d _v of the vehicle model 52, the operation inference model 70 is based on the operations inferred input operation i2, i3 adjust the pedal operation amount _{a r-i} included in the adjusted input operation im2, im3 input to the vehicle model 52, the positive and negative difference values _{d diff The pedal detection amount av-o} included in the simulated running state om is adjusted according to the vehicle 2 based on the inverted difference value −d _diff, which is the reverse of the above, and the adjusted simulated running state o is generated and adjusted. By applying the simulated running state o to the operation inference learning model 70, the operation inference learning model 70 is machine-learned.
According to the above structure, the first pedal play amount adjuster 54, and the pedal play amount d _r of the vehicle 2, based on the difference value d _diff pedal play amount d _v of the vehicle model 52, the operation reasoning _{The pedal operation amount a r-i} included in the input operations i2 and i3 based on the inference result of the learning model 70 is adjusted according to the vehicle model 52. Therefore, the pedal play amount d _r of the vehicle 2 and the vehicle model _52, a difference of d _v is appropriately reflected, the pedal operation amount is appropriately adjusted.
The second pedal play amount adjusting unit 55, based on the inverted difference value _{-d diff} obtained by inverting the sign of the difference value _{d diff,} pedal detected amount _{a v} contained in the simulated running state om the vehicle model 52 is output _{Adjust −o according} to the vehicle 2. Therefore, the adjustment in the first pedal play amount adjuster 54 is performed in the backward adjustment, the pedal play amount d _r of the vehicle 2 and the vehicle model _52, a difference of d _v is appropriately reflected, pedal detected amount Is adjusted appropriately.
In this way, the difference in pedal play amount between the vehicle 2 and the vehicle model 52 is appropriately absorbed. As a result, even if the operation inference learning model 70 that infers the operation of the vehicle 2 is machine-learned with the vehicle model 52 as the operation execution target, the learning accuracy of the operation inference learning model 70 due to the difference in the amount of pedal play is achieved. The decrease is suppressed.
Further, since the difference in the pedal play amount is absorbed by converting the value of the pedal play amount in the input / output portion of the vehicle model 52, it is necessary to relearn the vehicle model 52 even if the difference in the pedal play amount occurs. The calculation cost required for the learning system 10 is reduced. Therefore, it becomes easy to suppress a decrease in learning accuracy of the operation inference learning model 70.

Further, the first pedal play amount adjusting unit 54 adjusts the pedal operation amount a _r-i included in the input operations i2 and i3 by subtracting the difference value d _diff from the pedal operation amount a _r-i. , The adjusted pedal operation amount _av-i is generated, and the second pedal play amount adjustment unit 55 sets the pedal detection amount _av-o included in the simulated running state om to the pedal detection amount _av-o . The adjusted pedal detection amount a _r-o is generated by adjusting by adding the difference value d _diff.
According to the above configuration, as already described using the equations 1 and 3, the difference in the amount of pedal play between the vehicle 2 and the vehicle model 52 can be effectively absorbed.

Further, the vehicle model 52 is a vehicle that outputs a simulated running state om based on the input operations im2 and im3 that are machine-learned and adjusted based on the actual running performance of the vehicle 2 so as to perform a simulated operation of the vehicle 2. It is equipped with a learning model 60.
In particular, in the present embodiment, the vehicle learning model 60 is realized by a neural network.
According to the above configuration, the learning system 10 can be appropriately realized.

Further, the operation reasoning learning model 70 is reinforcement-learned.
In the initial stage of reinforcement learning, the operation inference learning model 70 learned by reinforcement learning is impossible for humans to operate pedals 2c and 2d with extremely high frequency, and puts a burden on the actual vehicle. , May output undesired operations.
According to the above configuration, in the initial stage of such reinforcement learning, the vehicle learning model 60 is a traveling state s that imitates the vehicle 2 based on the operation inferred by the operation inference learning model 70. By outputting the simulated running state om and applying it to the operation inference learning model 70, the operation inference learning model 70 is reinforced and learned in advance. That is, in the initial stage of reinforcement learning, the operation inference learning model 70 can be reinforcement-learned without using the actual vehicle 2. Therefore, the burden on the actual vehicle 2 can be reduced.
Further, when the pre-learning is completed, the operation inference learning model 70 is further strengthened and learned using the actual vehicle 2, so that the operation inference learning model 70 is strengthened and learned by using only the vehicle learning model 60. The learning accuracy of the operation output by the operation inference learning model 70 can be improved.
In particular, in the above configuration, since the pre-learning is performed with the vehicle learning model 60 as the operation execution target, the learning time is longer than when the vehicle 2 is the operation execution target in the entire process of the pre-learning. It can be reduced.

[First Modified Example of Embodiment]
Next, with reference to FIG. 8, a first modification of the learning system and the learning method of the operation inference learning model for controlling the drive robot shown in the above embodiment will be described. FIG. 8 is an explanatory diagram of the operation of the second pedal play amount adjusting unit in the first modification. The learning system in the first modification is different from the learning system 10 of the above embodiment in the processing content of the second pedal play amount adjusting unit 55A.

In the example of FIG. 5, than the pedal play amount d _r of the vehicle 2, the pedal play amount d _v of the vehicle model 52 is small. In such a case, for example, when the model output brake pedal detection amount series om3 output by the vehicle model 52 is converted by the second pedal play amount adjusting unit 55 of the above embodiment, the output brake pedal detection amount series o3 is converted. There may be a portion where the adjusted pedal detection amount _{avo value becomes a constant value, as shown as R3.}
This is due to the following reasons. That is, when the pedal operation amount a _r-i included in the input operation is smaller than the pedal play amount difference d _diff , the pedal operation adjusted by the calculation using Equation 1 in the first pedal play amount adjusting unit 54. The value of the quantity a _vi is 0. Therefore, after applying it to the vehicle model 52, the pedal detected amount a _v-o contained in the simulated running state om also becomes a value close to 0. In the second pedal play amount adjusting unit 55, the pedal play amount difference _ddiff is added to this value close to 0 by the equation 3. As a result, in the accelerator pedal detection amount series o2 and the brake pedal detection amount series o3, the portion where the pedal operation amount a _r-i is smaller than the pedal play amount difference d _{diff in the accelerator pedal operation amount series i2 and the brake pedal operation amount series i3.} However, it is uniformly converted into the value of the pedal play amount difference d _diff.

上式においては、アクセルペダル操作量系列ｉ２、ブレーキペダル操作量系列ｉ３の、ペダル操作量ａ_ｒ−ｉがペダル遊び量差ｄ_ｄｉｆｆよりも小さい部分においては、アクセルペダル操作量系列ｉ２、ブレーキペダル操作量系列ｉ３に含まれるペダル操作量ａ_ｒ−ｉを、調整されたペダル検出量ａ_ｒ−ｏの値としている。これにより、図８にＲ４として示されるように、上記実施形態においては一定の値が出力されていた部分Ｒ３が、アクセルペダル操作量系列ｉ２、ブレーキペダル操作量系列ｉ３に近い、なだらかな線となっている。 In contrast, the second pedal play amount adjusting unit 55A in this variation, using the following equation 4, to adjust the pedal detected amount a _v-o contained in the simulated running state om.

In the above equation, in the part of the accelerator pedal operation amount series i2 and the brake pedal operation amount series i3 where the pedal operation amount a _r-i is smaller than the pedal play amount difference d _diff , the accelerator pedal operation amount series i2 and the brake pedal _{The pedal operation amount a r-i} included in the operation amount series i3 is used as the value of the adjusted pedal detection amount a _r-o . As a result, as shown as R4 in FIG. 8, the portion R3 in which a constant value is output in the above embodiment becomes a gentle line close to the accelerator pedal operation amount series i2 and the brake pedal operation amount series i3. It has become.

In the learning system of this modification, the second pedal play amount adjusting unit 55A is when the value obtained by subtracting the difference value d _{diff from the pedal operation amount a r-i included in the input operations i2 and i3 is 0 or more. Adjusts the pedal detection amount a v-o} included in the simulated running state om by adding the difference value d _diff to the pedal detection amount a _v-o , and adjusts the pedal detection amount a _{r-o. In other cases, the value of the pedal operation amount a r-i} included in the input operations i2 and i3 is set as the adjusted pedal detection amount a _r-o .
According to the above configuration, the adjusted pedal detection amount a _r-o can be expressed as a value close _{to the pedal operation amount a r-i} included in the input operations i2 and i3. Therefore, the adjusted simulated running state o that is the input of the operation inference learning model 70 can be set to a state close to the actual pedal operation. Therefore, the reduction of the learning accuracy of the operation inference learning model 70 is effective. Can be suppressed.
Needless to say, this modification has other effects similar to those of the embodiments already described.

[Second variant of the embodiment]
Next, with reference to FIGS. 9 and 10, a second modification of the learning system and learning method of the operation inference learning model for controlling the drive robot shown in the above embodiment will be described. FIG. 9 is an explanatory diagram when it takes time to infer the vehicle model in adjusting the pedal play amount in the first modification. FIG. 10 is an explanatory diagram of the operation of the second pedal play amount adjusting unit in the second modification. The learning system in the second modification is a further modification of the learning system in the first modification, and the processing content of the second pedal play amount adjusting unit 55B is different.

When the accelerator pedal operation amount series i2 as shown in the upper left of FIG. 9 is input to the learning system described as the first modification, the first pedal play amount adjusting unit 54 as shown in the upper right of FIG. Model input The accelerator pedal operation amount series is converted to im2, which is input to the vehicle model.
Here, if it takes time to infer the model output accelerator pedal detection amount series om2 by the vehicle model, as shown in the lower right of FIG. 9, the model input accelerator pedal operation amount series im2 and the model output accelerator pedal detection amount series om2 There is a delay between the two, as shown as the delay time D.
In such a situation, when the model output accelerator pedal detection amount series om2 is input to the second pedal play amount adjustment unit 55A of the first modification, the accelerator pedal detection amount series o2 as shown in the lower left of FIG. 9 is output. NS. More specifically, in the accelerator pedal operation amount series i2, the accelerator pedal operation is performed by the equation 4 between the time T1 and the delay time D _{in which the pedal operation amount a r-i} is equal to or less than the pedal play amount difference d _diff. The value of the quantity series i2 is adopted as the accelerator pedal detection quantity series o2. As a result, as shown as the range R5, in the falling portion of the accelerator pedal detection amount series o2, the portion where it would be desirable to take a value larger than the _{pedal play amount difference d diff is the pedal play.} The value is smaller than the amount difference d _diff.

すなわち、上式においては、模擬走行状態ｏｍに含まれるペダル検出量ａ_ｖ−ｏが０より大きな値を有する場合には、入力操作ｉ２、ｉ３に含まれるペダル操作量ａ_ｒ−ｉではなく、模擬走行状態ｏｍに含まれるペダル検出量ａ_ｖ−ｏの値を調整されたペダル検出量ａ_ｒ−ｏに反映している。これにより、図１０に範囲Ｒ６として示されるように、少なくともペダル遊び量差ｄ_ｄｉｆｆよりも大きな部分においては、モデル出力アクセルペダル検出量系列ｏｍ２の値はアクセルペダル操作量系列ｉ２の値に変換されずに、維持されている。 In the second pedal play amount adjusting section 55B of the present modification, using Equation 5 follows, adjusting the pedal detected amount a _v-o contained in the simulated running state om.

That is, in the above equation, when the pedal detected amount a _v-o contained in the simulated running state om has a value greater than 0, the input operation i2, instead pedal operation amount a _r-i included in the i3, It reflects the pedal detected amount a pedal detected amount value is adjusted _v-o a _r-o contained in the simulated running state om. As a result, as shown as the range R6 in FIG. 10, the value of the model output accelerator pedal detection amount series om2 is converted to the value of the accelerator pedal operation amount series i2 _{at least in a portion larger than the pedal play amount difference d diff.} It is maintained without.

In the learning system of this modified example, in the second pedal play amount adjusting unit 55B, whether the value obtained by subtracting the difference value _{ddiff from the pedal operation amount a r-i included in the input operations i2 and i3 is 0 or more.} When the pedal detection amount a _v-o included in the simulated running state om is larger than 0, the pedal detection amount a _v-o included in the simulated running state om is added to the pedal detection amount a _v-o by a difference value d. _The adjusted pedal detection amount a _r-o is generated by adding the diff, and in other cases, the value of the _{pedal operation amount a r-i} included in the input operations i2 and i3 is set. Let the adjusted pedal detection amount a _r-o .
According to the above configuration, even when it takes time to infer the vehicle model, it is possible to effectively suppress a decrease in learning accuracy of the operation inference learning model 70.
Needless to say, this modification has other effects similar to those of the embodiments already described.

The learning system and learning method of the operation inference learning model for controlling the drive robot of the present invention are not limited to the above-described embodiment and each modification described with reference to the drawings, but are within the technical scope thereof. Various other variants are possible.

For example, in the above embodiment and each modification, the vehicle model 52 outputs a series of pedal detection amounts, that is, a plurality of pedal detection amount values, but the present invention is not limited to this, and only one pedal detection amount value is output. It may be output.
Further, in the above-described embodiment and each modification, the operation includes both the accelerator pedal 2c and the brake pedal 2d, but the operation is not limited to this, and may be only the accelerator pedal 2c or only the brake pedal 2d. It doesn't matter.
Further, in the above embodiment and each modification, the operation inferred by the operation inference learning model 70 is converted into the value of the actual pedal operation amount by the drive robot model 51, and an input operation is generated. Not limited to. For example, the operation inference learning model 70 may infer the input operation itself, and the input operation may be input to the vehicle model 52 without going through the drive robot model 51.

Further, in the above-described embodiment and each modification, the vehicle model 52 includes a vehicle learning model 60 realized as a neural network, and the vehicle 2 is simulated by the vehicle learning model 60, but the present invention is not limited to this. .. That is, if the vehicle model is set to perform a simulated operation of the vehicle 2, and when an operation is input, a simulated running state imitating the vehicle 2 including the pedal detection amount of the pedal is output, machine learning is performed. It does not have to be a learning model.
For example, when it is not necessary to match the dynamic characteristics of the vehicle 2, a physical model that does not depend on the vehicle 2 to be learned may be used as the vehicle model. By using the first and second pedal play amount adjusting units 54 and 55 described in the present embodiment and each modification, the vehicle to be learned by the operation inference learning model 70 and the vehicle model 52 used for pre-learning. The amount of pedal play between is adjusted and absorbed. Therefore, for example, as the vehicle model 52, a physical model of a vehicle different from the learning target of the operation inference learning model 70 can be used.
One of the purposes of pre-learning the operation inference learning model 70 is, as already explained, in the early stage of learning where humans cannot infer undesired operations that are impossible and burden the actual vehicle. , Do not use the actual vehicle. Therefore, when pre-learning with this as the main purpose, for example, if it is premised that the dynamic characteristics of the vehicle 2 are learned using the actual vehicle 2 after the pre-learning, it does not depend on the vehicle 2 to be learned. Even a physical model can be sufficiently used by pre-learning.
In such a case, if the environment is such that some vehicle model can be prepared, the operation inference learning model 70 can be pre-learned without machine learning the vehicle learning model 60. Therefore, learning of the operation inference learning model 70 is easy.

In addition to this, as long as the gist of the present invention is not deviated, the configurations given in the above-described embodiment and each modification can be selected or changed to other configurations as appropriate.

1 Test equipment 2 Vehicle 2c Accelerator pedal 2d Brake pedal 3 Chassis dynamometer 4 Drive robot (autopilot robot)
10 Learning system 11 Learning control device 20 Drive robot control unit 30 Learning unit 40 Reinforcement learning unit 41 Operation content Reasoning unit 50 Test device model 51 Drive robot model 52 Vehicle model 53 Chassis dynamometer model 54 First pedal play amount adjustment unit 55, 55A, 55B 2nd pedal play amount adjustment unit 60 Vehicle learning model 70 Operation reasoning learning model i1 Vehicle speed series i2 Accelerator pedal operation amount series (input operation)
i3 Brake pedal operation amount series (input operation)
im2 model input accelerator pedal operation amount series (adjusted input operation)
im3 model input brake pedal operation amount series (adjusted input operation)
om Simulated running state o Adjusted simulated running state a _r-i Pedal operation amount included in input operation a _v-i Adjusted pedal operation amount a _{v o} Pedal detection amount included in simulated running state a _r-o Adjusted pedal detection amount _dr Vehicle pedal play amount d _v Vehicle model pedal play amount d _diff Pedal play amount difference (difference value)

Claims (8)

An operation inference learning model that infers the operation of the vehicle such that the vehicle is driven according to a specified command vehicle speed based on the traveling state of the vehicle including the vehicle speed, and an operation inference learning model mounted on the vehicle and based on the operation. It is a learning system of an operation inference learning model that controls an autopilot robot and includes an autopilot robot that runs the vehicle and machine-learns the operation inference learning model.
The operation includes the amount of pedal operation of one or both of the accelerator pedal and the brake pedal.
A vehicle model that is set to perform a simulated operation of the vehicle and outputs a simulated running state that imitates the vehicle, including a pedal detection amount of the pedal.
Based on the difference value between the pedal play amount of the vehicle and the pedal play amount of the vehicle model, the pedal operation amount included in the input operation based on the operation inferred by the operation inference learning model is applied to the vehicle model. A first pedal play amount adjusting unit that adjusts and inputs the adjusted input operation to the vehicle model, and
A second pedal that adjusts the pedal detection amount included in the simulated running state according to the vehicle and generates the adjusted simulated running state based on the inverted difference value obtained by reversing the positive and negative of the difference value. Play amount adjustment part and
With
A learning system for an operation inference learning model that controls an automatic control robot, which machine-learns the operation inference learning model by applying the adjusted simulated running state to the operation inference learning model. The first pedal play amount adjusting unit adjusts the pedal operation amount included in the input operation by subtracting the difference value from the pedal operation amount to generate the adjusted pedal operation amount.
The second pedal play amount adjusting unit adjusts the pedal detection amount included in the simulated running state by adding the difference value to the pedal detection amount to generate the adjusted pedal detection amount. , A learning system for an operation inference learning model that controls the automatic control robot according to claim 1. When the value obtained by subtracting the difference value from the pedal operation amount included in the input operation is 0 or more, the second pedal play amount adjusting unit determines the pedal detection amount included in the simulated running state. The adjusted pedal detection amount is generated by adding the difference value to the pedal detection amount.
In other cases, learning of the operation inference learning model for controlling the automatic control robot according to claim 2, wherein the value of the pedal operation amount included in the input operation is the adjusted pedal detection amount. system. In the second pedal play amount adjusting unit, the value obtained by subtracting the difference value from the pedal operation amount included in the input operation is 0 or more, or the pedal detection amount included in the simulated running state is larger than 0. In this case, the pedal detection amount included in the simulated running state is adjusted by adding the difference value to the pedal detection amount to generate the adjusted pedal detection amount.
In other cases, learning of the operation inference learning model for controlling the automatic control robot according to claim 2, wherein the value of the pedal operation amount included in the input operation is the adjusted pedal detection amount. system. The vehicle model is a vehicle learning model that is machine-learned to simulate the vehicle based on the actual running performance of the vehicle and outputs the simulated running state based on the adjusted input operation. The learning system of the operation inference learning model for controlling the automatic control robot according to claim 1. The vehicle learning model is a learning system of an operation inference learning model that controls an autopilot robot according to claim 5, which is realized by a neural network. The operation reasoning learning model is a learning system of an operation reasoning learning model that controls an automatic control robot according to any one of claims 1 to 3, which is reinforcement-learned. An operation inference learning model that infers the operation of the vehicle such that the vehicle is driven according to a specified command vehicle speed based on the traveling state of the vehicle including the vehicle speed, and an operation inference learning model mounted on the vehicle and based on the operation. It is a learning method of an operation inference learning model that controls an automatic control robot, which machine-learns the operation inference learning model with respect to the automatic control robot that runs the vehicle.
The operation includes the amount of pedal operation of one or both of the accelerator pedal and the brake pedal.
The pedal play amount of the vehicle and the pedal of the vehicle model are set according to the vehicle model that is set to perform the simulated operation of the vehicle and outputs a simulated running state that imitates the vehicle, including the pedal detection amount of the pedal. Based on the difference value of the play amount, the pedal operation amount included in the input operation based on the operation inferred by the operation inference model is adjusted, and the adjusted input operation is input to the vehicle model.
Based on the inverted difference value obtained by reversing the positive and negative of the difference value, the pedal detection amount included in the simulated traveling state is adjusted according to the vehicle, and the adjusted simulated traveling state is generated.
A learning method of an operation inference learning model that controls an automatic control robot, which machine-learns the operation inference learning model by applying the adjusted simulated running state to the operation inference learning model.

Images