CN108791302A - Driving behavior modeling - Google Patents

Abstract

The invention discloses a driver behavior modeling system, which specifically includes a feature extractor for extracting and constructing reward function features; a reward function generator for obtaining the reward function required for constructing a driving strategy; a driving strategy acquirer for completing the construction of the driving strategy ; The determiner judges whether the optimal driving strategy constructed by the acquirer meets the evaluation criteria; if not, rebuilds the reward function, repeats the construction of the optimal driving strategy, and iterates repeatedly until the evaluation criteria are met; finally obtains a description of the real driving Demonstrate driving strategies. This application can be applied to new state scenarios to obtain corresponding actions, which greatly improves the generalization ability of the established driver behavior model, and has wider applicable scenarios and stronger robustness.

Description

Driver Behavior Modeling System

technical field

The invention relates to a modeling method, in particular to a driver behavior modeling system.

Background technique

Autonomous driving is an important part of the field of intelligent transportation. Due to current technology and other reasons, autonomous vehicles still require an intelligent driving system (intelligent assisted driving system) and human drivers to cooperate with each other to complete driving tasks. In this process, driver modeling is an essential and important step, whether it is to better quantify driver information for intelligent system decision-making, or to provide people with personalized services by distinguishing different drivers.

In the current method of driver modeling, the reinforcement learning method has a good solution to the complex sequential decision-making problem with large-scale continuous space and multiple optimization objectives such as the driver's driving in the vehicle, so it is also a kind of Efficient method for modeling driver behavior. Reinforcement learning, as a problem-solving method based on MDP, needs to interact with the environment, take actions to obtain feedback signals from the environment, that is, rewards, and maximize long-term rewards.

Through the search of existing literature, it is found that the existing methods for setting the reward function in driver behavior modeling mainly include: the traditional method of setting manually by researchers for different scene states, and the use of reverse reinforcement learning method to set the method. Traditional methods rely heavily on the subjectivity of researchers, and the quality of the reward function depends on the ability and experience of researchers. At the same time, because in the process of vehicle driving, in order to correctly set the reward function, it is necessary to balance a large number of decision variables. These variables have great incompatibility or even contradiction, and researchers often cannot design a reward function that can balance various requirements. .

With the help of driving demonstration data, reverse reinforcement learning assigns appropriate weights to various driving characteristics, and can automatically learn to obtain the required reward function, thereby solving the shortcomings of the original human decision-making. However, the traditional reverse reinforcement learning method can only learn the existing scene states in the driving demonstration data. In actual driving, due to different factors such as weather and scenery, the real driving scene often exceeds the driving demonstration range. Therefore, the method of reverse reinforcement learning solves the problem that the relationship between scenes and decision-making actions in driving demonstration data shows insufficient generalization ability.

The existing driver behavior modeling methods based on reinforcement learning theory mainly have two ideas: the first idea is to adopt the traditional reinforcement learning method, and the setting of the reward function depends on the analysis, sorting, screening and induction of the scene by the researchers, and then obtains A series of characteristics related to driving decisions, such as: the distance in front of the car, whether it is far away from the curb, whether it is far away from pedestrians, reasonable speed, lane change frequency, etc.; and then according to the needs of driving scenarios, a series of experiments are designed to obtain these characteristics in the corresponding The weight ratio in the reward function in the scene environment, and finally complete the overall design of the reward function, and use it as a model to describe the driver's driving behavior. Idea 2: Based on the probabilistic model modeling method, maximum entropy reverse reinforcement learning is used to solve the driving behavior characteristic function. First, it is assumed that there is a potential and specific probability distribution, which generates a driving demonstration trajectory; then, it is necessary to find a probability distribution that can fit the driving demonstration, and the problem of obtaining this probability distribution can be transformed into a nonlinear programming problem. which is:

max-plogp

ΣP=1

P refers to the probability distribution of the demonstration trajectory. After the probability distribution is obtained by solving the above formula, it is determined by

After obtaining relevant parameters, the reward function r=θ ^T f(s _t ) can be obtained.

The traditional driver's driving behavior model uses known driving data to analyze, describe and reason about driving behavior. However, the collected driving data cannot completely cover the endless driving behavior characteristics, and it is even more impossible to obtain the corresponding actions of all states. In the actual driving scene, due to the different weather, scenes, and objects, there are many possible driving states, and it is impossible to traverse all states. Therefore, the generalization ability of the traditional driver's driving behavior model is weak, the model assumes many conditions, and the robustness is poor.

Secondly, in the actual driving problem, only relying on the method of setting the reward function by the researchers needs to balance too many requirements for various features, all completely relying on the experience settings of the researchers, repeated manual adjustments, time-consuming and labor-intensive, and more deadly is too subjective. In different scenarios and environments, researchers need to face too many scene states; at the same time, even for a certain scene state, different requirements will lead to changes in driving behavior characteristics. In order to accurately describe the driving task, a series of weights must be assigned to accurately describe these factors. In the existing method, the reverse reinforcement learning based on the probability model mainly starts from the existing demonstration data, takes the demonstration data as the existing data, and then seeks the distribution corresponding to the current data, and based on this, the action selection in the corresponding state can be obtained. However, the distribution of known data cannot represent the distribution of all data. To obtain the distribution correctly, it is necessary to obtain the corresponding actions of all states.

Contents of the invention

In order to solve the problem of weak generalization of driver modeling, that is, the technical problem in the prior art that it is impossible to establish a corresponding reward function for driver behavior modeling when the driving scene does not include demonstration data, this application Provides a driver behavior modeling system, which can be applied to new state scenarios to obtain its corresponding actions, which greatly improves the generalization ability of the established driver behavior model, with wider applicable scenarios and stronger robustness.

In order to achieve the above object, the technical gist of the solution of the present invention is: a driver behavior modeling system, specifically comprising:

Feature extractor, extracting and constructing reward function features;

Reward function generator to obtain driving strategy;

The driving strategy acquirer completes the construction of the driving strategy;

Determiner, which judges whether the optimal driving strategy constructed by the acquirer meets the evaluation criteria; if not, then rebuilds the reward function, repeats the construction of the optimal driving strategy, and iterates repeatedly until the evaluation criteria are met; finally obtains a description of the real driving demonstration driving strategy.

Further, the specific implementation process of the feature extractor to extract and construct the features of the reward function is:

S11. During the driving process of the vehicle, use the camera placed behind the windshield of the vehicle to sample the driving video, and obtain N groups of pictures of road conditions in different vehicle driving environments; at the same time, corresponding to the driving operation data, that is, the steering angle under the road environment situation, jointly build up the training data;

S12. Translating, cropping, and changing brightness operations on the collected pictures to simulate scenes with different lighting and weather;

S13. Construct a convolutional neural network, use the processed picture as input, and use the operation data of the corresponding picture as the label value for training, and use the optimization method based on the Nadam optimizer to find the optimal solution for the mean square error loss to optimize the neural network. The weight parameter;

S14. Save the network structure and weights of the trained convolutional neural network to build a new convolutional neural network and complete the state feature extractor.

Further, the convolutional neural network established in step S13 includes 1 input layer, 3 convolutional layers, 3 pooling layers, and 4 fully connected layers; the input layer is sequentially connected to the first convolutional layer, the first Pooling layer, then connect the second convolutional layer, the second pooling layer, then connect the third convolutional layer, the third pooling layer, and finally connect the first fully connected layer, the second fully connected layer The connection layer, the third fully connected layer, and the fourth fully connected layer.

Further, the convolutional neural network after the training in step S14 does not include an output layer.

Further, the specific implementation process of the reward function generator to obtain the driving strategy is:

S21. Obtaining expert driving demonstration data: the driving demonstration data comes from the sampling and extraction of demonstration driving video data, and a continuous driving video is sampled according to a certain frequency to obtain a set of trajectory demonstration; one expert demonstration data includes multiple trajectories, and the overall Remember to do:

Among them, D _E represents the overall driving demonstration data, (s _j , a _j ) represents the data pair consisting of the corresponding state j and the corresponding decision instruction of the state, M represents the total number of driving demonstration data, _NT represents the number of driving demonstration trajectories , L _i represents the number of state-decision instruction pairs (s _j , a _j ) contained in the i-th driving demonstration trajectory;

S22. Obtaining the characteristic expectation value of the driving demonstration;

Firstly, each state s _t describing the driving environment in the driving demonstration data DE is input into the state feature extractor to obtain the characteristic situation _f (s _t , a _t ) in the corresponding state s _t , f(s _t , a _t ) refers to a set of characteristic values of the driving environment scene corresponding to s _t that affect the driving decision-making result, and then calculate the characteristic expected value of the driving demonstration based on the following formula:

Among them, γ is the discount factor, which should be set according to different problems;

S23. Obtain the state-action set under the greedy strategy;

S24. Calculate the weight of the reward function.

Furthermore, the specific steps to obtain the state-action set under the greedy strategy are: since the reward function generator and the driving strategy obtainer are two parts of the cycle; first, obtain the neural network in the driving strategy obtainer: the driving demonstration The state feature _f (s _t , a _t ) obtained by extracting the data DE and describing the environment is input into the neural network, and the output g _w (st _t ) is obtained; g _w (st _t ) is a set of Q values describing the state s _t Set, namely [Q(s _t ,a ₁ ),...,Q(s _t ,a _n )] ^T , and Q(s _t ,a _i ) represents the state-action value, used to describe the current driving scene In the state s _t , select the quality of the decision-making driving action a _i , and obtain it based on the formula Q(s,a)=θ·μ(s,a), where θ in the formula refers to the weight in the current reward function , μ(s,a) refers to the expected value of the feature.

Then based on the ε-greedy strategy, select and describe the driving decision-making action corresponding to the driving scene state s _t Select the decision-making action that maximizes the Q value in the Q value set under the current driving scene s _t Otherwise, randomly select selected Afterwards, record the

In this way, for the state feature _f (s _t , a _t ) of each state in the driving demonstration DE, input the neural network, and obtain a total of M state-action pairs (s _t , a _t ), which describe the time t The driving decision-making action a _t is selected under the driving scene state s _t ; at the same time, based on the action selection situation, the Q values of M corresponding state-action pairs are obtained, denoted as Q.

Furthermore, the specific steps to obtain the weight of the reward function are:

First, construct the objective function based on the following formula:

Represents the loss function, that is, according to whether the current state-action pair exists in the driving demonstration, if it exists, it is 0, otherwise it is 1; is the corresponding state-action value recorded above; Be the product of the weight θ of the driving demonstration feature expectation and reward function obtained in S22; is a regular item;

The objective function is minimized by means of the gradient descent method, that is, t=min _θ J(θ), and the variable θ that minimizes the objective function is obtained, and the θ is the weight of the required reward function.

Furthermore, the specific implementation process of obtaining the driving strategy by the reward function generator also includes: S25. Based on the obtained corresponding reward function weight θ, construct the reward function generation according to the formula r(s, a)=θ ^T f(s, a) device.

As a further step, the specific implementation process for the driving strategy acquirer to complete the construction of the driving strategy is as follows:

S31 Construct the training data of the driving strategy acquirer

Obtain training data, each data includes two parts: one is the driving decision feature f(s _t ) obtained by inputting the driving scene state at time t into the driving state extractor, and the other is obtained based on the following formula

Among them, r _θ (s _t , a _t ) is the reward function generated based on the driving demonstration data by means of the reward function generator; Q ^π (s _t , a _t ) and Q ^π (s _t+1 , a _t+1 ) come from For the Q value recorded in S23, select the Q value that describes the driving scene s _t at time t and select the Q value that describes the driving scene s _t+ 1 at time t+1;

S32. Establish neural network

The neural network consists of three layers, the first layer is used as the input layer, and the number of neurons in it is the same as the output feature type of the feature extractor, which is k, which is used to input the feature f(s _t , a _t ) of the driving scene, and the second layer The number of hidden layers in the first layer is 10, and the number of neurons in the third layer is the same as the number n of driving actions for decision-making in the action space; the activation functions of the input layer and the hidden layer are both sigmoid functions, that is i.e. have:

z=w ⁽¹⁾ x=w ⁽¹⁾ [1,f _t ] ^T

h=sigmoid(z)

g _w (s _t )=sigmoid(w ⁽²⁾ [1,h] ^T )

where w ⁽¹⁾ is the weight of the hidden layer; f _t is the feature of the state s _t of the driving scene at time t, that is, the input of the neural network; z is the output of the network layer without the sigmoid activation function of the hidden layer; h is Hidden layer output after sigmoid activation function; w ⁽²⁾ is the weight of the output layer;

The g _w (s _t ) output by the network is the Q set of the driving scene state s _t at time t, that is, [Q( _st ,a ₁ ),...,Q(s _t ,a _n )] ^T , in S31 Q ^π (s _t , a _t ) is obtained by inputting the state s _t into the neural network and selecting the item a _t in the output;

S33. Optimizing the neural network

For the optimization of the neural network, the established loss function is the cross-entropy cost function, and the formula is as follows:

Among them, N refers to the number of training data; Q ^π ( _st , a _t ) is the value obtained by inputting the driving scene state s _t describing the driving scene at time t into the neural network, and selecting the corresponding driving decision-making action a _t item in the output; is the value obtained in S31; is a regular term, where W={w ⁽¹⁾ ,w ⁽²⁾ } refers to the weight in the above neural network;

Input the training data obtained in S31 into the neural network optimization cost function; use the gradient descent method to minimize the cross-entropy cost function, obtain the optimized neural network, and then obtain the driving strategy acquirer.

As a further step, the specific implementation process of the determiner includes:

Consider the current reward function generator and driving strategy acquirer as a whole, and check the current t value in S22 to see if t<ε, ε is the threshold for judging whether the objective function satisfies the requirement, that is, to judge whether it is currently used to obtain the driving strategy Whether the return function of the strategy meets the requirements; its value can be set differently according to specific needs;

When the value of t does not satisfy the formula; the return function generator needs to be rebuilt. At this time, the neural network required in S23 needs to be replaced with the optimized new neural network in S33, which will be used to generate the description in Under the driving scene state s _t , the selected network of Q(st _t , a _i ) value for decision-making of the driving action a _i is replaced with the new network structure optimized by the gradient descent method in S33; and then rebuild the reward The function generator obtains the driving strategy acquirer, and judges whether the value of t meets the requirement again;

When the formula is satisfied, the current θ is the weight of the required reward function; the reward function generator meets the requirements, and the driving strategy acquirer also meets the requirements; then the driving data of a driver who needs to build a driver model is collected, That is, the environment scene image and the corresponding operation data in the driving process are input into the driving environment feature extractor to obtain the decision-making features for the current scene; then the extracted features are input into the reward function generator to obtain the reward function corresponding to the state of the scene; then Input the collected decision features and the calculated reward function into the driving strategy acquirer to obtain the corresponding driving strategy of the driver.

Compared with the prior art, the present invention has the beneficial effects that: the present invention is used to describe the driver's decision-making and the method for establishing the driver's behavior model. Because the neural network is used to describe the strategy, when the neural network parameters are determined, the state and action One-to-one correspondence, then the possible cases for state-action pairs are no longer limited to the exemplary trajectories. Therefore, in the actual driving situation, the large state space corresponding to various driving scenarios caused by weather, scenery, etc., with the help of the excellent ability of the neural network to approximate any function, this kind of policy expression can be approximated as Black box: By inputting the characteristic value of the state, the corresponding state-action value is output, and at the same time, the action is further selected according to the output value to obtain the corresponding action. As a result, the applicability of inverse reinforcement learning for driver behavior modeling is greatly enhanced. The traditional method attempts to fit the demonstration trajectory with the help of a certain probability distribution, so the optimal strategy obtained is still limited by the existing parameters in the demonstration trajectory. However, the present invention can be applied to new state scenarios to obtain corresponding actions, which greatly improves the generalization ability of the established driver behavior model, and has wider applicable scenarios and stronger robustness.

Description of drawings

Figure 1 is a new deep convolutional neural network;

Fig. 2 is a driving video sampling diagram;

Figure 3 is a block diagram of the workflow of the system;

FIG. 4 is a structural diagram of the neural network established in step S32.

Detailed ways

The present invention will be further described below in conjunction with the accompanying drawings. The following examples are only used to illustrate the technical solution of the present invention more clearly, but not to limit the protection scope of the present invention.

This embodiment provides a driver behavior modeling system, including:

1. The feature extractor extracts and constructs the features of the return function, the specific method is:

S11. Sampling the driving video obtained by using the camera placed behind the windshield of the vehicle during the driving process of the vehicle, as shown in FIG. 2 .

Obtain pictures of N groups of different vehicle driving road environment road conditions and corresponding steering angle conditions. Including N1 straight roads and N2 curves, the values of N1 and N2 can be N1>=300, N2>=3000, corresponding to the driving operation data, and jointly construct training data.

S12. Perform relevant operations such as translation, cropping, and brightness change on the collected images to simulate scenes with different lighting and weather.

S13. Construct a convolutional neural network, use the processed picture as input, and use the operation data of the corresponding picture as the label value for training; use the optimization method based on the Nadam optimizer to find the optimal solution for the mean square error loss to optimize the neural network The weight parameters of the network.

The convolutional neural network includes 1 input layer, 3 convolutional layers, 3 pooling layers, and 4 fully connected layers. The input layer is connected to the first convolutional layer and the first pooling layer in turn, then connected to the second convolutional layer, the second pooling layer, and then connected to the third convolutional layer and the third pooling layer, Then connect the first fully connected layer, the second fully connected layer, the third fully connected layer, and the fourth fully connected layer in sequence.

S14. Save the network structure and weights of the trained convolutional neural network except for the final output layer to establish a new convolutional neural network and complete the state feature extractor.

2. The reward function generator obtains the driving strategy, the specific method is:

The reward function is used as the standard for action selection in the reinforcement learning method. In the process of obtaining the driving strategy, the quality of the reward function plays a decisive role. It directly determines the quality of the obtained driving strategy and whether the obtained strategy is consistent with the real one. The strategy corresponding to the driving demonstration data is the same. The formula of the reward function is reward=θ ^T f(s _t , a _t ), f(s _t , a _t ) refers to a group of driving decisions that affect the state s _t at time t in the corresponding driving environment scene "vehicle surrounding environment" The eigenvalues of the result are used to describe the scene situation around the vehicle. And θ refers to a set of weights corresponding to the features that affect driving decisions. The value of the weights shows the proportion of the corresponding environmental features in the reward function, reflecting the importance. On the basis of the state feature extractor, this weight θ needs to be solved to construct the reward function that affects the driving strategy.

S21. Obtain driving demonstration data from experts

The driving demonstration data comes from the sampling and extraction of demonstration driving video data (different from the data used by the previous driving environment feature extractor). A continuous driving video can be sampled at a frequency of 10hz to obtain a set of trajectory demonstrations. An expert demonstration should have multiple trajectories. Overall remember to do: Where D _E represents the overall driving demonstration data, (s _j , a _j ) represents the corresponding state j (the video picture of the driving environment at the sampling time j) and the state corresponding to the decision instruction (such as the steering angle in the steering instruction) Data pair, M represents the total number of driving demonstration data, _NT represents the number of driving demonstration trajectories, L _i represents the number of state-decision instruction pairs (s _j , a _j ) contained in the i-th driving demonstration trajectory

S22. Obtaining the characteristic expectation of driving demonstration

Firstly, each state s _t describing the driving environment in the driving demonstration data DE is input into the state feature extractor to obtain the characteristic situation _f (s _t , a _t ) in the corresponding state s _t , f(s _t , a _t ) Refers to a set of eigenvalues of driving environment scenes that affect driving decision-making results corresponding to s _t , and then calculate the characteristic expectation of driving demonstration based on the following formula:

Among them, γ is the discount factor, which should be set according to different problems, and the reference value can be set to 0.65.

S23. Obtain the state-action set under the greedy strategy

First, acquire the neural network in the driving strategy acquirer in S32. (Because the reward function generator and the driving strategy acquirer are two parts in a cycle, the neural network is the neural network just initialized in S32 at the beginning. As the cycle progresses, each step in the cycle is: Construction of a reward function that affects driving decision-making, and then obtain the corresponding optimal driving strategy in the driving strategy obtainer based on the current reward function, and judge whether the criteria for ending the cycle are met. If not, the optimized driving strategy in the current S34 put the neural network into the reconstructed reward function)

The state feature _f (s _t , a _t ) describing the environment obtained by extracting the driving demonstration data DE is input into the neural network to obtain the output g _w (s _t ); g _w ( _st ) is about describing the state s _t A set of Q values, that is, [Q( _st ,a ₁ ),...,Q( _st ,a _n )] ^T , and Q(st _t ,a _i ) represents the state-action value and is used to describe In the current driving scene state s _t , the quality of the decision-making driving action a _i can be selected based on the formula Q(s,a)=θ·μ(s,a), where θ refers to the current The weight in the reward function, μ(s,a) refers to the feature expectation.

Then based on the ε-greedy strategy, if ε is set to 0.5, select the driving decision-making action corresponding to the description driving scene state s _t That is to say, there is a 50% possibility to select the decision-making action that maximizes the Q value in the set of Q values under the current driving scene s _t Otherwise, randomly select selected Afterwards, record the

In this way, for the state feature _f (s _t , a _t ) of each state in the driving demonstration DE, input the neural network, and obtain a total of M state-action pairs (s _t , a _t ), which describe the Select the driving decision-making action a _{t in the driving scene state s t .} At the same time, based on the situation of action selection, the Q values of M corresponding state-action pairs are obtained, denoted as Q.

S24. Find the weight of the return function

First, construct the objective function based on the following formula:

Represents the loss function, that is, according to whether the current state-action pair exists in the driving demonstration, if it exists, it is 0, otherwise it is 1. is the corresponding state-action value recorded above. is the product of the driving demonstration feature expectation obtained in S22 and the weight θ of the reward function. As a regular term, in case of over-fitting problems, the γ can be 0.9.

The objective function is minimized by means of the gradient descent method, that is, t=min _θ J(θ), and the variable θ that minimizes the objective function is obtained, and the θ is the weight of the required reward function.

S25. Based on the obtained corresponding reward function weight θ, construct a reward function generator according to the formula r(s, a)=θ ^T f(s, a).

3. The driving strategy acquirer completes the construction of the driving strategy. The specific method is:

Construction of training data for S31 driving strategy acquirer

Get training data. The data comes from the sampling of the previous demonstration data, but it needs to be processed to obtain a new set of data of N types. Each data in the data includes two parts: one is the driving decision feature f(s _t ) obtained by inputting the driving scene state at time t into the driving state extractor, and the other is obtained based on the following formula

The formula includes a reward function generated by the parameter _{r θ} (s _t ,at ) based on the driving demonstration data by means of a reward function generator. Q ^π (s _t , a _t ) and Q ^π (s _{t +1} , a _t+1 ) come from the set of Q values Q recorded in S23, select the Q value and select It describes the Q value of driving scene s _t+ 1 at time t+1.

S32. Establish neural network

The neural network consists of three layers, the first layer is used as the input layer, and the number of neurons in it is the same as the output feature type of the feature extractor, which is k, which is used to input the feature f(s _t , a _t ) of the driving scene, and the second layer The number of hidden layers in the first layer is 10, and the number of neurons in the third layer is the same as the number n of driving actions for decision-making in the action space; the activation functions of the input layer and the hidden layer are both sigmoid functions, that is i.e. have:

z=w ⁽¹⁾ x=w ⁽¹⁾ [1,f _t ] ^T

h=sigmoid(z)

g _w (s _t )=sigmoid(w ⁽²⁾ [1,h] ^T )

Among them, w ⁽¹⁾ refers to the weight of the hidden layer; f _t refers to the characteristics of the state s _t of the driving scene at time t, which is the input of the neural network; z refers to the network layer without the sigmoid activation function of the hidden layer Output; h refers to the hidden layer output after the sigmoid activation function; w ⁽²⁾ refers to the weight of the output layer; the network structure is shown in Figure 3:

The g _w (s _t ) output by the network is the Q set of the driving scene state s _t at time t, that is, [Q( _st ,a ₁ ),...,Q(s _t ,a _n )] ^T , in S31 Q ^π (s _t , at ) _{is obtained by inputting the state s t into the} neural network and selecting the at item in the output.

S33. Optimizing the neural network

For the optimization of the neural network, the established loss function is the cross-entropy cost function, and the formula is as follows:

Where N refers to the number of training data. Q ^π (s _t , a _t ) is the value obtained by inputting the description of the driving scene state s _{t at time t} into the neural network, and selecting the corresponding driving decision-making action a _t item in the output. is the value obtained in S31. It is also a regular item, which is set to prevent overfitting. This γ may also be 0.9. Among them, W={w ⁽¹⁾ ,w ⁽²⁾ } refers to the weight value in the above neural network.

Input the training data obtained in S31 into the neural network optimization cost function. The gradient descent method is used to minimize the cross-entropy cost function, and the optimized neural network is obtained to obtain the driving strategy acquirer.

4. The determiner judges whether the optimal driving strategy constructed by the acquirer meets the evaluation criteria; if not, rebuilds the reward function, repeats the construction of the optimal driving strategy, and iterates repeatedly until the evaluation criteria are met; finally obtains the true description Driving strategies for driving demonstrations.

Consider the current reward function generator and driving strategy acquirer as a whole, and check the current t value in S22 to see if t<ε, ε is the threshold for judging whether the objective function satisfies the requirement, that is, to judge whether it is currently used to obtain the driving strategy Whether the reward function of the strategy meets the requirements. Its value can be set differently according to specific needs.

When the value of t does not satisfy the formula. It is necessary to rebuild the reward function generator. At this time, the neural network required in the current S23 needs to be replaced with the optimized new neural network in S33, which will be used to generate and describe the selected driving decision in the driving scene state s _t The network of Q(st _t , a _i ) value of action a _i is replaced by the new network structure optimized by the gradient descent method in S33. Then rebuild the reward function generator, obtain the driving strategy acquirer, and judge whether the value of t meets the requirement again.

When the formula is satisfied, the current θ is the weight of the required reward function. The reward function generator meets the requirements, and the driving strategy acquirer also meets the requirements. Therefore, it is possible to: collect driving data of a driver who needs to establish a driver model, that is, an image of an environment scene during driving and corresponding operation data, such as a driving steering angle. Input the driving environment feature extractor to obtain the decision features for the current scene. Then input the extracted features into the reward function generator to obtain the reward function corresponding to the scene state. Then input the collected decision features and the calculated reward function into the driving strategy acquirer to obtain the corresponding driving strategy of the driver.

In a Markov decision process, a policy needs to connect states to their corresponding actions. But for a state space with a large range, it is difficult to describe a definite strategy representation for the untraversed area, and the description of this part is also ignored in the traditional method, which is only based on the demonstration trajectory to illustrate the whole The probability model of the trajectory distribution does not give a specific policy representation for the new state, that is, it does not give a specific method for the possibility of taking certain actions in the new state. In the present invention, the policy is described by means of a neural network. The neural network has excellent generalization ability because it can approximate the characteristics of any function with any accuracy. With the help of the representation of state features, on the one hand, those states which are not included in the exemplary trajectory can be represented, and on the other hand, by means of inputting the corresponding state features into the neural network. The corresponding action value can be obtained, so as to obtain the desired action according to the strategy. Therefore, the problem that the traditional method cannot generalize the driving demonstration data to the untraversed driving scene state is solved.

The above is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto, any person familiar with the technical field within the technical scope of the disclosure of the present invention, according to the present invention Any equivalent replacement or change of the created technical solution and its inventive concept shall be covered within the scope of protection of the present invention.

Claims (10)

1. The driver behavior modeling system is characterized in that it specifically includes: Feature extractor, extracting and constructing reward function features; Reward function generator to obtain driving strategy; The driving strategy acquirer completes the construction of the driving strategy; The determiner judges whether the optimal driving strategy constructed by the obtainer meets the evaluation criteria; if not, rebuilds the reward function, repeats the construction of the optimal driving strategy, and iterates repeatedly until the evaluation criteria are met. 2. according to the described driver's behavior modeling system of claim 1, it is characterized in that, the concrete realization process that feature extractor extracts and constructs reward function feature is: S11. During the driving process of the vehicle, use the camera placed behind the windshield of the vehicle to sample the driving video, and obtain N groups of pictures of different vehicle driving environment road conditions and the corresponding steering angle conditions; at the same time, corresponding to the driving operation data, jointly construct Get up the training data; S12. Translating, cropping, and changing brightness operations on the collected pictures to simulate scenes with different lighting and weather; S13. Construct a convolutional neural network, use the processed picture as input, and use the operation data of the corresponding picture as the label value for training, and use the optimization method based on the Nadam optimizer to find the optimal solution for the mean square error loss to optimize the neural network. The weight parameter; S14. Save the network structure and weights of the trained convolutional neural network to build a new convolutional neural network and complete the state feature extractor. 3. The driver behavior modeling system according to claim 2, wherein the convolutional neural network set up in step S13 comprises 1 input layer, 3 convolutional layers, 3 pooling layers, 4 fully connected layer; the input layer is connected to the first convolutional layer and the first pooling layer in turn, then connected to the second convolutional layer, the second pooling layer, and then connected to the third convolutional layer and the third pooling layer Layers, and finally connect the first fully connected layer, the second fully connected layer, the third fully connected layer, and the fourth fully connected layer. 4. The driver behavior modeling system according to claim 2, wherein the convolutional neural network after the training in step S14 does not include an output layer. 5. according to the described driver's behavior modeling system of claim 1, it is characterized in that, reward function generator obtains driving strategy concrete implementation process is: S21. Obtaining expert driving demonstration data: the driving demonstration data comes from the sampling and extraction of demonstration driving video data, and a continuous driving video is sampled according to a certain frequency to obtain a set of trajectory demonstration; one expert demonstration data includes multiple trajectories, and the overall Remember to do: D _E ＝{(s ₁ ,a ₁ ),(s ₂ ,a ₂ ),...,(s _M ,a _M )} Among them, D _E represents the overall driving demonstration data, (s _j , a _j ) represents the data pair consisting of the corresponding state j and the corresponding decision instruction of the state, M represents the total number of driving demonstration data, _NT represents the number of driving demonstration trajectories , L _i represents the number of state-decision instruction pairs (s _j , a _j ) contained in the i-th driving demonstration trajectory; S22. Obtaining the characteristic expectation value of the driving demonstration; Firstly, each state s _t describing the driving environment in the driving demonstration data DE is input into the state feature extractor to obtain the characteristic situation _f (s _t , a _t ) in the corresponding state s _t , f(s _t , a _t ) refers to a set of characteristic values of the driving environment scene corresponding to s _t that affect the driving decision-making result, and then calculate the characteristic expected value of the driving demonstration based on the following formula: Among them, γ is the discount factor, which should be set according to different problems; S23. Obtain the state-action set under the greedy strategy; S24. Calculate the weight of the reward function. 6. according to the described driver behavior modeling system of claim 5, it is characterized in that, the specific step of obtaining the state-action set under the greedy strategy is: because the reward function generator and the driving strategy acquirer are two parts of the cycle; First, obtain the neural network in the driving strategy acquirer: extract the state feature _f (s _t , a _t ) describing the environmental situation obtained by extracting the driving demonstration data DE, input it into the neural network, and obtain the output g _w (s _t ); g _w (s _t ) is a set of Q values describing the state s _t , that is, [Q(s _t ,a ₁ ),...,Q(s _t ,a _n )] ^T , and Q(s _t , a _i ) represents the state-action value, which is used to describe the pros and cons of selecting a decision-making driving action a _i under the current driving scene state s _t , and calculate it based on the formula Q(s,a)=θ·μ(s,a) So, θ in this formula refers to the weight in the current reward function, and μ(s, a) refers to the expected value of the feature. Then based on the ε-greedy strategy, select and describe the driving decision-making action corresponding to the driving scene state s _t Select the decision-making action that maximizes the Q value in the Q value set under the current driving scene s _t Otherwise, randomly select selected Afterwards, record the In this way, for the state feature _f (s _t , a _t ) of each state in the driving demonstration DE, input the neural network, and obtain a total of M state-action pairs (s _t , a _t ), which describe the time t The driving decision-making action a _t is selected under the driving scene state s _t ; at the same time, based on the action selection situation, the Q values of M corresponding state-action pairs are obtained, denoted as Q. 7. according to the described driver's behavior modeling system of claim 5, it is characterized in that, the concrete step of seeking the weight value of return function is: first based on following formula, construct objective function: Represents the loss function, that is, according to whether the current state-action pair exists in the driving demonstration, if it exists, it is 0, otherwise it is 1; is the corresponding state-action value recorded above; Be the product of the weight θ of the driving demonstration feature expectation and reward function obtained in S22; is a regular item; The objective function is minimized by means of the gradient descent method, that is, t=min _θ J(θ), and the variable θ that minimizes the objective function is obtained, and the θ is the weight of the required reward function. 8. According to the described driver behavior modeling system of claim 5, it is characterized in that, the specific implementation process of the reward function generator obtaining the driving strategy also includes: S25. Based on the corresponding reward function weight θ obtained, according to the formula r(s, a)=θ ^T f(s,a) builds a reward function generator. 9. according to the described driver behavior modeling system of claim 1, it is characterized in that, the specific realization process that driving strategy acquirer completes driving strategy construction is: S31 Construct the training data of the driving strategy acquirer Obtain training data, each data includes two parts: one is the driving decision feature f(s _t ) obtained by inputting the driving scene state at time t into the driving state extractor, and the other is obtained based on the following formula Among them, r _θ (s _t , a _t ) is the reward function generated based on the driving demonstration data by means of the reward function generator; Q ^π (s _t , a _t ) and Q ^π (s _t+1 , a _t+1 ) come from For the Q value recorded in S23, select the Q value that describes the driving scene s _t at time t and select the Q value that describes the driving scene s _t+ 1 at time t+1; S32. Establish neural network The neural network consists of three layers, the first layer is used as the input layer, and the number of neurons in it is the same as the output feature type of the feature extractor, which is k, which is used to input the feature f(s _t , a _t ) of the driving scene, and the second layer The number of hidden layers in the first layer is 10, and the number of neurons in the third layer is the same as the number n of driving actions for decision-making in the action space; the activation functions of the input layer and the hidden layer are both sigmoid functions, that is i.e. have: z=w ⁽¹⁾ x=w ⁽¹⁾ [1,f _t ] ^T h=sigmoid(z) g _w (s _t )=sigmoid(w ⁽²⁾ [1,h] ^T ) where w ⁽¹⁾ is the weight of the hidden layer; f _t is the feature of the state s _t of the driving scene at time t, that is, the input of the neural network; z is the output of the network layer without the sigmoid activation function of the hidden layer; h is Hidden layer output after sigmoid activation function; w ⁽²⁾ is the weight of the output layer; The g _w (s _t ) output by the network is the Q set of the driving scene state s _t at time t, that is, [Q( _st ,a ₁ ),...,Q(s _t ,a _n )] ^T , in S31 Q ^π (s _t , a _t ) is obtained by inputting the state s _t into the neural network and selecting the item a _t in the output; S33. Optimizing the neural network For the optimization of the neural network, the established loss function is the cross-entropy cost function, and the formula is as follows: Among them, N refers to the number of training data; Q ^π ( _st , a _t ) is the value obtained by inputting the driving scene state s _t describing the driving scene at time t into the neural network, and selecting the corresponding driving decision-making action a _t item in the output; is the value obtained in S31; is a regular term, where W={w ⁽¹⁾ ,w ⁽²⁾ } refers to the weight in the above neural network; Input the training data obtained in S31 into the neural network optimization cost function; use the gradient descent method to minimize the cross-entropy cost function, obtain the optimized neural network, and then obtain the driving strategy acquirer. 10. The driver behavior modeling system according to claim 1, wherein the specific implementation process of the determiner comprises: Consider the current reward function generator and driving strategy acquirer as a whole, and check the current t value in S22 to see if t<ε, ε is the threshold for judging whether the objective function satisfies the requirement, that is, to judge whether it is currently used to obtain the driving strategy Whether the return function of the strategy meets the requirements; its value can be set differently according to specific needs; When the value of t does not satisfy the formula; the return function generator needs to be rebuilt. At this time, the neural network required in S23 needs to be replaced with the optimized new neural network in S33, which will be used to generate the description in Under the driving scene state s _t , the selected network of Q(st _t , a _i ) value for decision-making of the driving action a _i is replaced with the new network structure optimized by the gradient descent method in S33; and then rebuild the reward The function generator obtains the driving strategy acquirer, and judges whether the value of t meets the requirement again; When the formula is satisfied, the current θ is the weight of the required reward function; the reward function generator meets the requirements, and the driving strategy acquirer also meets the requirements; then the driving data of a driver who needs to build a driver model is collected, That is, the environment scene image and the corresponding operation data in the driving process are input into the driving environment feature extractor to obtain the decision-making features for the current scene; then the extracted features are input into the reward function generator to obtain the reward function corresponding to the state of the scene; then Input the collected decision features and the calculated reward function into the driving strategy acquirer to obtain the corresponding driving strategy of the driver.