CN111079851B - Vehicle type identification method based on reinforcement learning and bilinear convolution network - Google Patents

Abstract

The invention discloses a vehicle identification method based on reinforcement learning and bilinear convolutional network, constructs a deep network model, sets hyperparameters of a fine-grained classification network and initializes the network, and establishes a Markov decision model for optimizing salient features; Perform scale transformation on the dataset; optimize the attention area: when the parameters of the fine-grained classification network are fixed, input the dataset into the fine-grained classification network, and use the reinforcement learning algorithm to optimize the saliency area and select the optimal attention area; A loss function is established to update the parameters of the fine-grained classification network; after the features are fused, the network is repeatedly trained until the attention region no longer changes; the model images to be tested are input into the trained model to obtain the corresponding detection results. The invention utilizes the reinforcement learning network to extract the salient features of the bottom layer, and uses the bilinear interpolation method to fuse the high-level semantic features and the salient features of the lower layers to improve the recognition accuracy.

Description

Vehicle Recognition Method Based on Reinforcement Learning and Bilinear Convolutional Network

technical field

The invention relates to a vehicle identification method, in particular to a vehicle identification method based on reinforcement learning and bilinear convolutional network.

Background technique

The vehicle model recognition problem can be viewed as an applied branch of the fine-grained classification problem, that is, classifying different sub-classes of the same class with very similar appearance. Since the daily collected model pictures are easily affected by factors such as posture, viewing angle and occlusion, there are small differences between models of different brands, but there are large differences between models of the same brand. How to effectively identify vehicle models is an urgent application problem in fine-grained classification.

Bilinear convolutional network is a model that can achieve fine-grained classification with high accuracy in recent years. It has the advantages of simple structure and efficient training, but it only uses the features of the last layer as the input features of classification. When training with class features, more detailed information will be lost, and most of the high-level features will be retained. Since the objects of fine-grained classification are often objects with similar appearance but different details in performance, the characterization of detailed features has a great impact on the recognition rate of fine-grained classification. If the low-level features of the bilinear network and the high-level features are directly fused, because the scale of the low-level features is relatively large, some methods need to be used for dimensionality reduction when merging with the high-level features. When the main information of the feature loss obtained after dimensionality reduction is the detailed information, it will not only fail to improve the accuracy of classification, but will prolong the training time of the network and the final classification efficiency.

Reinforcement learning is a method for solving sequence decision problems. By modeling the problem to be solved as an MDP model, the classical methods in reinforcement learning such as time difference algorithm, least square time difference algorithm and actor-critic algorithm are used. to find the optimal strategy. Therefore, reinforcement learning is a very suitable method for extracting saliency in low-level features.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a vehicle identification method based on reinforcement learning and bilinear convolutional network, which can improve the accuracy of vehicle identification in the case of fewer vehicle pictures.

The technical scheme of the present invention is as follows: a vehicle identification method based on reinforcement learning and bilinear convolutional network, comprising the following steps:

(1) Build a deep network model: build a fine-grained classification network based on reinforcement learning and bilinear convolutional network for vehicle recognition;

(2) Setting the hyperparameters of the fine-grained classification network: the hyperparameters include the learning rate, the number of iterations and the batch size of the network;

(3) Initialize the network: initialize the weights and thresholds of the fine-grained classification network;

(4) Establish a Markov decision model to optimize saliency features;

(5) Preprocessing data set: scale transformation of the data set;

(6) Optimize the attention area: when the parameters of the fine-grained classification network are fixed, input the data set into the fine-grained classification network, and use the reinforcement learning algorithm to optimize the saliency area and select the optimal attention area;

(7) Construct loss function: establish a loss function for updating fine-grained classification network parameters, the loss function is defined as the sum of squares of errors between the true label of the data and the predicted label of the data;

(8) Fusion features: For each data in the data set, the final fusion result can be obtained by using the attention region optimized in step (6) and the features of the fifth convolutional layer, and used for classification;

(9) Training the network: In the case of fixing the optimal attention area, the fine-grained classification network is retrained by using the dataset and gradient descent method until the training error is less than the preset threshold;

(10) Alternate training: Repeat (6)-(9) until the attention area no longer changes;

(11) Input the image of the vehicle model to be tested into the trained deep network model to obtain the corresponding detection result.

Further, the parallel feature extraction layer of the bilinear convolutional network described in the step (1) adopts the first convolutional layer to the fifth convolutional layer of VGG16, and the first convolutional layer to the fifth convolutional layer. The features output by the convolution layer transition from detailed features to high-level semantic feature attention. After the fifth convolutional layer, a bilinear vector is obtained through the outer product operation, and finally the fully connected layer is connected, and soft max is performed on the output. To realize the identification and classification of vehicle models.

Further, the step (4) of establishing the Markov decision model for optimizing the salient features includes:

401) The state space X is set as a set composed of all sub-features of the fifth convolutional layer in the feature mesoscale generated by the third convolutional layer, X _{= {} x1, _x2 ,...,xn};

402) The action space U is set as the set formed by the movement of the state in the up, down, left, and right of the state space;

403) The state transition function is f:X×U→X, for any state x∈X in the state space, from any action u∈U in the action space, the next state is the state after the action u occurs, and the state is The scale in the features generated by the third convolutional layer is a sub-feature of the fifth convolutional layer;

404) The reward function is: r:X×U→R, for any x∈X in the state space, the immediate reward obtained from any action u∈U in the action space.

Preferably, the action space U={0, 1, 2, 3}, where 0 represents an upward movement of the state, 1 represents a leftward movement of the state, 2 represents a downward movement of the state, and 3 represents a rightward movement of the state.

Further, the (6) optimization of the attention region includes the steps:

601) Set the values of parameters: discount rate γ, decay factor λ, number of iterations e, maximum time step T corresponding to each iteration, learning rate α, exploration rate ε;

初始化Q₀1(x,u)＝0，Q₀2(x,u)＝0；602) For

Initialize Q ₀ 1(x, u)=0, Q ₀ 2(x, u)=0;

603) It is judged that the number of episodes has reached the maximum value E: if it is reached, go to step 612); otherwise, go to step 604);

604) Determine whether the maximum time step is reached: if it is reached, go to step 603); otherwise, go to step 605)

605) Initialize the current state x=x ₀ ;

606) Randomly generate a probability p between (0, 1), and judge whether p<ε is true: if it is true, the action selected in the current state is: u=argmax _u (Q ₁ (x,u)+Q ₂ ( x, u)); otherwise, randomly select any action in the action set;

607) Execute the currently selected action u to obtain its corresponding next state x';

608) Determine whether the classification result obtained by the output layer is the same as the real label: if they are the same, immediately reward r=1; otherwise, immediately reward r=0;

609) Randomly generate a probability p between (0, 1), and judge whether p<0.5 holds: if it holds, update the Q value: Q ₁ (x,u)=r+γmaxuQ ₁ (x′,u); otherwise Update the Q value: Q ₂ (x,u)=r+γmaxuQ ₂ (x′,u);

610) Update the current time step: t=t+1, and go to step 604) to judge;

611) Update the current plot: e=e+1;

612) Output the current optimal policy and value functions Q ₁ (x, u), Q ₂ (x, u).

Further, the loss function in the step (7) is:

Among them, y represents the model classification result obtained by the network, and y′ represents the real label of the model image.

The beneficial effect of the technical solution provided by the present invention is that the bilinear convolutional network is used as the basic deep network structure, the reinforcement learning network is used to extract the salient features of the bottom layer, and the bilinear interpolation method is used to analyze the high-level semantic features. It is fused with the salient features of the lower layers, and finally, the specific vehicle recognition is carried out through the fully connected layer of the bilinear convolutional network and the soft maximization operation, which improves the recognition accuracy of the vehicle. Combined with the reinforcement learning network, the salient features of the model pictures can be well extracted when there are few pictures of the model, which is suitable for online vehicle recognition and can be applied to online real-time recognition in the field of video surveillance.

Description of drawings

Fig. 1 is the flow chart of the inventive method;

Fig. 2 is the network model diagram of the method of the present invention;

FIG. 3 is a detailed diagram of a single network model of the bilinear model in the method of the present invention.

Detailed ways

Please refer to Figure 1. The vehicle recognition method based on reinforcement learning and bilinear convolutional network involved in this example includes the following steps:

(1) Build a deep network model: Build a fine-grained classification network based on reinforcement learning and bilinear convolutional network for vehicle recognition. The model diagrams are shown in Figure 2 and Figure 3. The parallel feature extraction layer of the bilinear convolutional network adopts the first convolutional layer to the fifth convolutional layer of VGG16, and the features output from the first convolutional layer to the fifth convolutional layer are from detailed features to high-level semantic feature attention. In the transition, a bilinear vector is obtained through the outer product operation after the fifth convolutional layer, and finally the fully connected layer is connected, and a soft maximization operation is performed on the output to realize the identification and classification of vehicle models.

(2) Set the hyperparameters of the network: the learning rate of the network is 0.02, the number of iterations is 10,000, the batch size is 10 images, and the training threshold is 0.01;

(3) Initialize the network: set the ownership value and threshold of the network to 0.00001;

(4) Build MDP model: build a Markov decision model that optimizes saliency features. The established MDP model is as follows:

401) State space modeling: The state space is based on the output feature map of Conv3, and all the features that can be obtained from the output features of the third convolution layer using the scale of the fifth convolution layer constitute the state space. The space contains 4 feature maps containing the four corners of the edge;

402) Action space modeling: the action space modeling moves upward, downward, and rightward, and uses numbers 0, 1, 2, and 3 to describe the action, respectively;

403) Migration function modeling: Assuming that the position corresponding to the feature corresponding to the current state is (x, y), then:

If the upward action is taken, the position of the next state is (x, y-1);

If the action to the left is taken, the position of the next state is (x-1,y);

If the downward action is taken, the position of the next state is (x,y+1)

If the action to the right is taken, the position of the next state is (x+1,y)

404) Reward function modeling: The modeling of the reward function depends on the current output of the deep network, that is, when a certain vehicle model is input into the deep network, the current optimal attention region is used to obtain the vehicle type. When the model category is the same as the real category, the immediate reward is 1; otherwise, the reward is 0.

(5) Preprocessing data set: Download the data set, and perform scale transformation on the data set, that is, operations such as translation and rotation, and expand the original data set. The purpose of the expansion is to increase the robustness of the network, that is, in some With noisy graphs, the network has good recognition ability while preventing overfitting during training. The download address of the dataset Car-196 is: Car-196: https://ai.stanford.edu/~jkrause/cars/car_dataset.html.

In order to make the network have better generalization ability, the bird dataset CUB-200 and the aircraft class FGVC-Aircraft are also used for training in the training phase. The download addresses are:

CUB-200: http://www.vision.caltech.edu/visipedia/CUB-200.html and FGVC-Aircraft: http://www.robots.ox.ac.uk/~vgg/data/fgvc-aircraft /.

(6) Optimize the attention area: Use the reinforcement learning algorithm to optimize the saliency area, train the network, and select the optimal attention area. The specific implementation process of optimization can be described as:

601) Set the values of parameters: discount rate γ=0.9, decay factor λ=0.95, number of iterations E=200, maximum time step T=1000 corresponding to each iteration, learning rate α=0.5, exploration rate ε=0.1 ;

初始化Q₀1(x,u)＝0，Q₀2(x,u)＝0，判断情节数已达到最大值E：602) For

Initialize Q ₀ 1(x, u)=0, Q ₀ 2(x, u)=0, and judge that the number of episodes has reached the maximum value E:

If it reaches:

Go to step

otherwise:

Go to step 604);

603) Determine whether the maximum time step is reached:

If it reaches:

Go to step 603)

otherwise:

Go to step 605)

604) Randomly initialize the current state x=x ₀ ;

605) Randomly generate a probability p between (0, 1), and judge whether p<ε holds:

If true:

The action selected in the current state is: u=argmax _u (Q ₁ (x,u)+Q ₂ (x,u))

otherwise:

Randomly select any one of the four actions in the action set;

606) Execute the currently selected action u to obtain its corresponding next state x'

607) Determine whether the classification result obtained by the output layer is the same as the real label:

If the same:

Immediate reward r=1

otherwise:

Immediate reward r=0

608) Randomly generate a probability p between (0, 1), and judge whether p<0.5 holds:

If true:

Update Q value: Q ₁ (x,u)=r+γmax _u Q ₁ (x′,u)

otherwise:

Update Q value: Q ₂ (x,u)=r+γmax _u Q ₂ (x′,u)

609) Update the current time step: t=t+1, and go to step 4) for judgment

610) Update current episode: e=e+1

611) Output the current optimal policy and value functions Q ₁ (x, u), Q ₂ (x, u)

(7) Constructing the loss function: The loss function for constructing the network training is:

Among them, y represents the model classification result obtained by the network, and y′ represents the real label of the model image.

(8) Fusion features: After obtaining the feature region of the optimal value, fix the region and use it to fuse the high-level features (the output of the fifth convolution module) by summing to obtain the fused high-level features, The output of each layer feature and the output of the fusion feature are shown in Figure 2;

(9) Train the network: In the case of fixing the optimal attention area, use the data set and use the gradient descent method to retrain the network until the training error is less than the preset threshold;

(10) Alternate training: Repeat (6)-(9) until the attention area no longer changes;

(11) Input the vehicle image to be tested into the deep network model to obtain the corresponding detection result.

The recognition accuracy rate of the vehicle type recognition method in each data set using the method of the present invention is as follows:

Claims (4)

1. a vehicle type identification method based on reinforcement learning and bilinear convolutional network is characterized by comprising the following steps:

(1) constructing a depth network model: constructing a fine-grained classification network for vehicle identification based on reinforcement learning and a bilinear convolutional network;

(2) setting the hyper-parameters of the fine-grained classification network: the hyper-parameters comprise the learning rate, the iteration times and the batch size of the network;

(3) initializing the network: initializing a weight value and a threshold value of the fine-grained classification network;

(4) establishing a Markov decision model for optimizing the significance characteristics:

401) the state space X is set as a set formed by all sub-features of the fifth convolutional layer with the feature middle scale size generated by the third convolutional layer, and X is { X ═ X₁,x₂,…,x_n}；

402) The motion space U is a set of movements of the state in the state space up, down, left, and right;

403) the state transition function is f, multiplied by U → X, for any state X in the state space belonging to X, any action U belonging to U in the action space, the next state is the state after the action U occurs, and the state is a certain sub-feature of the fifth convolutional layer in the feature generated by the third convolutional layer;

404) the reward function is: r is X U → R, and for any X in the state space belonging to X, any action U in the action space belonging to U is rewarded immediately;

(5) preprocessing a data set: carrying out scale transformation on the data set;

(6) optimizing the attention area: under the condition that the parameters of the fine-grained classification network are fixed, the data set is input into the fine-grained classification network, a reinforcement learning algorithm is adopted to optimize the saliency region, and the optimal attention region is selected, wherein the method comprises the following steps:

601) setting the values of the parameters: discount rate gamma, attenuation factor lambda, iteration round number e, maximum time step T corresponding to each iteration, learning rate alpha and exploration rate;

602) for the

Initialization Q₀1(x,u)＝0，Q₀2(x,u)＝0；

603) Judging that the number of the episodes reaches the maximum value E: if so, go to step 612); otherwise go to step 604);

604) judging whether the maximum time step is reached: if yes, go to step 603); otherwise, go to step 605)

605) Initializing current state x ═ x₀；

606) Randomly generating a probability p between (0,1), judging p<Whether or not: if so, the actions selected in the current state are: u-argmax_u(Q₁(x,u)+Q₂(x, u)); otherwise, randomly selecting any one action in the action set;

607) executing the currently selected action u to obtain a next state x' corresponding to the action u;

608) judging whether the classification result obtained by the output layer is the same as the real label or not: if the two are the same, immediately rewarding r to be 1; otherwise, the prize r is 0 immediately;

609) randomly generating a probability p between (0,1), judging p<Whether or not 0.5 holds: if true, update Q: q₁(x,u)＝r+γmaxuQ₁(x', u); otherwise, updating the Q value: q₂(x,u)＝r+γmaxuQ₂(x′,u)；

610) Updating the current time step: t +1, and go to step 604) to make a determination;

611) and updating the current plot: e + 1;

612) outputting the current optimal strategy and value function Q₁(x,u)、Q₂(x,u)；

(7) Constructing a loss function: establishing a loss function for updating the fine-grained classification network parameters, wherein the loss function is defined as the sum of squares of errors of a real label of the data and a predicted label of the data;

(8) fusion characteristics: for each data in the data set, obtaining a final fusion result by using the attention area optimized in the step (6) and the characteristics of the fifth convolution layer, and using the final fusion result for classification;

(9) training a network: under the condition of fixing the optimal attention area, the data set is utilized and a gradient descent method is adopted to train the fine-grained classification network again until the training error is smaller than a preset threshold value;

(10) alternate training: repeating the steps (6) to (9) until the attention area is not changed any more;

(11) and inputting the vehicle type image to be tested into the trained deep network model to obtain a corresponding detection result.

2. The vehicle type identification method based on reinforcement learning and bilinear convolutional network of claim 1, wherein the parallel feature extraction layer of the bilinear convolutional network in step (1) adopts the first convolutional layer to the fifth convolutional layer of VGG16, the features output by the first convolutional layer to the fifth convolutional layer are transited from the detail features to the high-level semantic feature attention, a bilinear vector is obtained by an outer product operation after the fifth convolutional layer, and finally, a full connection layer is connected, and a soft maximization operation is performed on the output, so that the vehicle type identification and classification are realized.

3. The vehicle type identification method based on reinforcement learning and bilinear convolutional network of claim 1, wherein the motion space U ═ {0,1,2,3}, 0 represents upward movement of state, 1 represents leftward movement of state, 2 represents downward movement of state, and 3 represents rightward movement of state.

4. The vehicle type identification method based on reinforcement learning and bilinear convolutional network of claim 1, wherein the loss function in step (7) is:

wherein y represents the vehicle type classification result obtained by the network, and y' represents the real label of the vehicle type picture.

Images