WO2018205245A1 - 一种用于车辆自动驾驶的策略网络模型的生成方法及装置 - Google Patents

一种用于车辆自动驾驶的策略网络模型的生成方法及装置 Download PDF

Info

Publication number
WO2018205245A1
WO2018205245A1 PCT/CN2017/084081 CN2017084081W WO2018205245A1 WO 2018205245 A1 WO2018205245 A1 WO 2018205245A1 CN 2017084081 W CN2017084081 W CN 2017084081W WO 2018205245 A1 WO2018205245 A1 WO 2018205245A1
Authority
WO
WIPO (PCT)
Prior art keywords
state information
information set
vehicle
feature
network model
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2017/084081
Other languages
English (en)
French (fr)
Inventor
李慧云
王峥
刘玢玢
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shenzhen Institute of Advanced Technology of CAS
Original Assignee
Shenzhen Institute of Advanced Technology of CAS
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shenzhen Institute of Advanced Technology of CAS filed Critical Shenzhen Institute of Advanced Technology of CAS
Priority to PCT/CN2017/084081 priority Critical patent/WO2018205245A1/zh
Publication of WO2018205245A1 publication Critical patent/WO2018205245A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N5/00—Computing arrangements using knowledge-based models
    • G06N5/02—Knowledge representation; Symbolic representation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N99/00—Subject matter not provided for in other groups of this subclass

Definitions

  • the invention belongs to the technical field of computers, and in particular relates to a method and a device for generating a strategy network model for automatic driving of a vehicle.
  • a driverless car that is, driving on the road through its own assisted driving system without a driver, is environmentally aware.
  • the assisted driving system makes control decisions through the environmental information acquired by the sensors, and various sensors and products for assisting the driving system have emerged, such as: laser radar, machine vision, adaptive cruise, vehicle approach notification device, night vision assistance, Adaptive front lighting system, etc.
  • the control method of the assisted driving system is mainly based on the rule-based control decision, that is, the expert rule system for controlling the decision output of the vehicle environment information is constructed based on the known driving experience.
  • the shallow learning algorithm such as the expert rule system can be regarded as the slave. The process of finding regularity between the marked data, when the rules are difficult to abstract into formulas or simple logic, shallow learning can not be effective, however, the types of autonomous driving scenes are diverse, the road conditions are complex, and it is difficult to define them with limited rules.
  • the present invention provides a method for generating a policy network model for automatic driving of a vehicle, the method comprising the steps of:
  • the vehicle state information collected at each test moment constitutes a state information set, and the state information set is processed to obtain a feature state information set;
  • a policy network model for the automatic driving of the vehicle is generated according to the training result of the network model of the extreme learning machine.
  • the present invention provides a device for generating a policy network model for automatic driving of a vehicle, the device comprising:
  • a state processing module configured to form vehicle state information collected at each test moment to form a state information set, and process the state information set to obtain a feature state information set;
  • An enhanced learning module configured to search for a vehicle action that obtains a maximum reward value under each state information of the feature state information set according to a preset reward value function and a pre-built vehicle action set;
  • An extreme learning machine training module configured to train a network model of the extreme learning machine according to the feature state information set, the maximum reward value corresponding to each state information in the feature state information set, and the vehicle action that obtains the maximum reward value; as well as
  • a policy network generating module configured to generate a policy network model for the automatic driving of the vehicle according to the training result of the network model of the extreme learning machine.
  • the invention constructs a state information set by collecting vehicle state information at each test moment, and processes the state information set to preserve the feature of the state information set while reducing the amount of data in the state information set, and obtain the feature state information set according to the pre-
  • the set value function and the pre-built vehicle action set are used to find the vehicle action that obtains the maximum return value under each state information of the feature state information set, according to the state information of the feature state information set, and the maximum return corresponding to each state information.
  • FIG. 1 is a flowchart showing an implementation of a method for generating a policy network model for automatic driving of a vehicle according to Embodiment 1 of the present invention
  • FIG. 2 is a diagram showing an example of a structure of a policy network model for automatic driving of a vehicle according to Embodiment 1 of the present invention
  • FIG. 3 is a schematic structural diagram of a device for generating a policy network model for automatic driving of a vehicle according to Embodiment 2 of the present invention
  • FIG. 4 is a schematic diagram of a preferred structure of a device for generating a policy network model for automatic driving of a vehicle according to Embodiment 2 of the present invention.
  • Embodiment 1 is a diagrammatic representation of Embodiment 1:
  • FIG. 1 is a flowchart showing an implementation process of a method for generating a policy network model for automatic driving of a vehicle according to Embodiment 1 of the present invention. For convenience of description, only parts related to the embodiment of the present invention are shown, which are described in detail as follows:
  • step S101 the vehicle state information collected at each test time constitutes a state information set, and the state information set is processed to obtain a feature state information set.
  • the invention is applicable to an interactive platform established based on a racing simulation platform or a racing simulator (for example, the open racing car simulator TORCS), on which the driving interaction test of the driverless vehicle is performed.
  • vehicle state information is collected by a plurality of sensors preset on the vehicle, and the vehicle state information may include a distance of the vehicle from the centerline of the road, an angle between the forward direction of the vehicle and the tangential direction of the road, and laser ranging in front of the vehicle. The distance value of the instrument and the speed component of the vehicle in the tangential direction of the road.
  • the vehicle state information of each experiment time is collected, and all the vehicle state information are together constitute a state information set.
  • the state information set is Processing is performed to generate a feature state information set, thereby effectively reducing the amount of calculation of the subsequent process by processing the state information set.
  • the feature state information set may include features of representative state information or state information.
  • the state information set may be divided into several clusters by a preset clustering algorithm (for example, K-means clustering), and cluster center points capable of representing each cluster may be clustered, and state information of the cluster center points may be clustered.
  • the feature state information set is formed.
  • features in the state information set may be extracted by a preset feature extraction algorithm (for example, a self-encoder feature extraction algorithm), and the feature state information set is formed by the features.
  • step S102 a vehicle action for obtaining a maximum return value under each state information of the feature state information set is searched according to a preset reward value function and a pre-built vehicle action set.
  • the reward value function is used to calculate an immediate return value obtained after the vehicle performs a certain action in a certain state
  • the vehicle action concentration may include a vehicle action such as a straight line or a brake.
  • the sequence of reward values corresponding to each state information of the feature state information set is initialized, and may be initialized to zero, and the sequence of reward values of each state information includes a return value corresponding to each vehicle action under the state information, wherein the state
  • the return value of the vehicle action s in the sequence of reward values of the information s can be expressed as
  • the reward value sequence of the state information of the feature state information is updated multiple times until the values in all the sequence of reward values no longer change, and finally the sequence of all the reward values
  • the maximum return value of each state information in the feature state information set and the vehicle action corresponding to the maximum return value are selected, so that the vehicle action with the best state information of each state in the feature state information set is determined through reinforcement learning.
  • the return value function can be expressed as:
  • the reward value when the action a traversed to is performed under the state information s of the feature state information set, It is the maximum return value of the next state information s' of the state information s, r is the immediate return value when the action a is traversed under the state information s, ⁇ is a preset parameter and 0 ⁇ ⁇ ⁇ 1.
  • the formula for the reward value function can be:
  • r ⁇ dis*cos( ⁇ *angle)*sgn(trackPos-threshold), where r is the immediate return value calculated by the return value function, ⁇ dis is the effective distance that the vehicle ran at the adjacent test time, and ⁇ is the preset
  • the weight scaling factor, angle is the angle between the current direction of the vehicle and the tangential direction of the road
  • trackPos is the distance of the vehicle from the centerline of the road
  • threshold is the preset threshold.
  • the return value function can also consider driving speed, unit fuel consumption, smoothness, and the like.
  • step S103 the network model of the preset extreme learning machine is trained according to the feature state information set, the maximum reward value corresponding to each state information in the feature state information set, and the vehicle action that obtains the maximum reward value.
  • the extreme learning machine is a single hidden layer feedforward neural network learning algorithm.
  • the network model of the extreme learning machine can be expressed as:
  • g( ) is the activation function of the hidden layer node
  • Q j is the jth column vector in the output value
  • M is the number of hidden layer nodes .
  • the maximum reward value corresponding to each state information in the feature state information set may be set as an output value in a training process of the extreme learning machine network model, and the state information and each state information in the feature state information set are set.
  • the vehicle that obtains the maximum return value is the input value of the extreme learning machine network model in one training process, thereby training the output weight vector of the hidden layer node and the network output layer node in the extreme learning machine network model.
  • the Extreme Learning Machine network model can be trained multiple times by performing multiple vehicle automatic driving tests.
  • the activation function of the hidden layer node is infinitely different, the input weight and the hidden layer node of the network model of the extreme learning machine
  • the threshold can be randomly assigned.
  • the matrix W can be regarded as a constant.
  • step S104 a policy network model for vehicle automatic driving is generated based on the training result of the network model of the extreme learning machine.
  • FIG. 2 is an example of a policy network model. It can be seen that the output weight vector of the hidden layer node and the network output layer node in the extreme learning machine network model, that is, the weight value of the hidden layer in the policy network model. Finally, the strategy network model consisting of state data processing, reinforcement learning and extreme learning machine network model is obtained.
  • the state information set formed by all the collected state information is processed to reduce the amount of data in the state information set and retain the features in the state information set to generate feature state information.
  • Set, through reinforcement learning obtain the maximum return value of each state information in the feature state information set and the vehicle action corresponding to the maximum return value, and collect each state information, the maximum return value of each state information, and the maximum return through the feature state information.
  • the utility model effectively reduces the consumption of computing resources, and effectively improves the generalization ability and the generation efficiency of the vehicle automatic driving strategy network model.
  • Embodiment 2 is a diagrammatic representation of Embodiment 1:
  • FIG. 3 is a diagram showing the structure of a device for generating a policy network model for automatic driving of a vehicle according to Embodiment 2 of the present invention. For the convenience of description, only parts related to the embodiment of the present invention are shown, including:
  • the state processing module 31 is configured to form the state information set by collecting the vehicle state information collected at each test time, and process the state information set to obtain the feature state information set.
  • the vehicle state information of each experiment time is collected, and all the vehicle state information are together constitute a state information set.
  • the state information set is Processing is performed to generate a feature state information set, thereby effectively reducing the amount of calculation of the subsequent process by processing the state information set.
  • the feature state information set may include features of representative state information or state information.
  • the reinforcement learning module 32 is configured to search for vehicle actions that obtain a maximum reward value under each state information of the feature state information set according to a preset reward value function and a pre-built vehicle action set.
  • the sequence of reward values of each state information includes a reward value corresponding to each vehicle action under the state information, and a sequence of reward values for the state information of the feature state information according to the reward value function and the vehicle action set. Perform multiple updates until the values in all the sequence of reward values no longer change, and finally select the maximum return value of each state information in the feature state information set from all the sequence of reward values. And the vehicle action corresponding to the maximum return value, thereby determining the vehicle behavior that is optimal for each state information in the feature state information set by reinforcement learning.
  • the return value function can be expressed as:
  • the reward value when the action a traversed to is performed under the state information s of the feature state information set, It is the maximum return value of the next state information s' of the state information s, r is the immediate return value when the action a is traversed under the state information s, ⁇ is a preset parameter and 0 ⁇ ⁇ ⁇ 1.
  • the extreme learning machine training module 33 is configured to train the network model of the extreme learning machine according to the feature state information set, the maximum return value corresponding to each state information in the feature state information set, and the vehicle action that obtains the maximum reward value.
  • the network model of the extreme learning machine can be expressed as:
  • g( ) is the activation function of the hidden layer node
  • Q j is the jth column vector in the output value
  • M is the number of hidden layer nodes .
  • the maximum reward value corresponding to each state information in the feature state information set may be set as an output value in a training process of the extreme learning machine network model, and the state information and each state information in the feature state information set are set.
  • the vehicle that obtains the maximum return value is the input value of the extreme learning machine network model in one training process, thereby training the output weight vector of the hidden layer node and the network output layer node in the extreme learning machine network model. Can be tested by multiple auto-driving tests, Secondary training extreme learning machine network model.
  • the activation function of the hidden layer node is infinitely different, the input weight and the hidden layer node of the network model of the extreme learning machine
  • the threshold can be randomly assigned.
  • the matrix W can be regarded as a constant.
  • the policy network generating module 34 is configured to generate a policy network model for automatic driving of the vehicle according to the training result of the network model of the extreme learning machine.
  • the output weight vector of the hidden layer node and the network output layer node in the network model of the extreme learning machine that is, the weight value of the hidden layer in the strategy network model, finally obtains state data processing, reinforcement learning, and extreme learning.
  • a strategic network model consisting of a machine network model.
  • the state processing module 31 includes:
  • the state processing sub-module 411 is configured to cluster the state information set by using a preset clustering algorithm, or perform feature extraction on the state information set by using a preset feature extraction algorithm to generate a feature state information set.
  • the reinforcement learning module 32 includes a reward value initialization module 421, a reward value update module 422, and a maximum reward value acquisition module 423, wherein:
  • the reward value initialization module 421 is configured to initialize a sequence of reward values corresponding to each state information of the feature state information set, and the sequence of reward values includes performing a reward value corresponding to all vehicle actions in the vehicle action set;
  • a return value update module 422 for updating the sequence of all reward values multiple times according to the reward value function and the vehicle action set until all the sequence of reward values no longer changes;
  • the maximum reward value obtaining module 423 is configured to obtain a vehicle action corresponding to the maximum reward value and the maximum reward value from the sequence of reward values corresponding to each state information of the feature state information.
  • the reward value update module 422 includes an action traversal module and a reward value update sub-module, wherein:
  • An action traversal module configured to traverse the vehicle action set, and calculate an immediate return value when performing the traversed vehicle action under the state information of the feature state information set according to the reward value function;
  • the reward value update submodule is configured to update, according to the immediate reward value and the preset update formula, a reward value corresponding to the vehicle action traversed by the state information of the feature state information set.
  • the extreme learning machine training module 33 includes a learning machine setting module 431 and a network training module 432, wherein:
  • the learning machine setting module 431 is configured to set an input and an output of the extreme learning machine network model according to the feature state information set, the maximum return value corresponding to each state information in the feature state information set, and the vehicle action that obtains the maximum return value;
  • the network training module 432 is configured to train the extreme learning machine network model to obtain an output weight of the extreme learning machine network model.
  • the state information set formed by all the collected state information is processed to reduce the amount of data in the state information set and retain the features in the state information set, generate a feature state information set, and obtain the feature state through reinforcement learning.
  • the generalization ability and generation efficiency of the vehicle automatic driving strategy network model are effectively improved.
  • each module of the device for the strategic network model of the vehicle automatic driving can be implemented by a corresponding hardware or software module, and each module can be an independent software and hardware module, or can be integrated into a software and hardware module. This is not intended to limit the invention.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Physics & Mathematics (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Computational Linguistics (AREA)
  • Artificial Intelligence (AREA)
  • Feedback Control In General (AREA)

Abstract

提供了一种用于车辆自动驾驶的策略网络模型的生成方法及装置，所述方法包括：将每个试验时刻采集到的车辆状态信息构成状态信息集，对状态信息集进行处理，获得特征状态信息集（S101）；根据预设的回报值函数和预先构建的车辆动作集，查找在特征状态信息集的每个状态信息下获得最大回报值的车辆动作（S102）；根据特征状态信息集、特征状态信息集中每个状态信息对应的最大回报值和获得最大回报值的车辆动作，训练极限学习机的网络模型（S103）；根据极限学习机的网络模型的训练结果，生成用于车辆自动驾驶的策略网络模型（S104），从而有效地降低了计算资源的消耗，有效地提高了车辆自动驾驶策略网络模型的生成效率。

Description

一种用于车辆自动驾驶的策略网络模型的生成方法及装置 技术领域
本发明属于计算机技术领域，尤其涉及一种用于车辆自动驾驶的策略网络模型的生成方法及装置。
背景技术
随着经济的发展和城镇化的推进，全球汽车保有量和道路里程逐步增加，导致诸如交通拥堵、事故、污染、土地资源紧缺等一系列传统汽车无法妥善解决的问题日益凸显。无人驾驶汽车技术被视为这些问题的有效解决方案，其发展被受瞩目，美国电子和电子工程师协会(IEEE)预测，至2040年自动驾驶车辆数目所占比例将达到75％。
无人驾驶汽车，即在没有司机的情况下通过自身的辅助驾驶系统在道路上行驶，具备环境感知能力。辅助驾驶系统通过传感器获取的环境信息做出控制决策，已出现了多种用于辅助驾驶系统的传感器和产品，例如：激光雷达、机器视觉、自适应巡航、车辆接近通报装置、夜视辅助、自适应前照明系统等。
目前，辅助驾驶系统的控制方法主要为基于规则的控制决策，即根据已知的驾驶经验构建对车辆环境信息输出控制决策的专家规则系统，专家规则系统这类浅层学习算法可看作从被标记的数据之间寻找规律的过程，当规则很难被抽象成公式或简单逻辑时，浅层学习就无法奏效，然而，自动驾驶场景类别多样、路况复杂，很难用有限的规则定义清楚。
随着深度强化学习技术的快速发展，一些研究机构提出“端对端”式的自动驾驶算法，通过深度网络构建辅助驾驶系统中的控制决策模型。这类方法不需要对车辆的状态进行基于规则的识别。然而，对车辆自动驾驶进行深度学习要求强大的计算资源，例如需要数百瓦的图形处理器(GPU)等长时间(数天到 数周)的训练，才能得到一个决策网络，如果外界条件改变，则需要重新训练。
发明内容
本发明的目的在于提供一种用于车辆自动驾驶的策略网络模型的生成方法及装置，旨在解决现有技术中策略网络模型的生成效率较低的问题。
一方面，本发明提供了一种用于车辆自动驾驶的策略网络模型的生成方法，所述方法包括下述步骤：
将每个试验时刻采集到的车辆状态信息构成状态信息集，对所述状态信息集进行处理，获得特征状态信息集；
根据预设的回报值函数和预先构建的车辆动作集，查找在所述特征状态信息集的每个状态信息下获得最大回报值的车辆动作；
根据所述特征状态信息集、所述特征状态信息集中每个状态信息对应的最大回报值和所述获得最大回报值的车辆动作，训练极限学习机的网络模型；
根据所述极限学习机的网络模型的训练结果，生成用于所述车辆自动驾驶的策略网络模型。
另一方面，本发明提供了一种用于车辆自动驾驶的策略网络模型的生成装置，所述装置包括：
状态处理模块，用于将每个试验时刻采集到的车辆状态信息构成状态信息集，对所述状态信息集进行处理，获得特征状态信息集；
强化学习模块，用于根据预设的回报值函数和预先构建的车辆动作集，查找在所述特征状态信息集的每个状态信息下获得最大回报值的车辆动作；
极限学习机训练模块，用于根据所述特征状态信息集、所述特征状态信息集中每个状态信息对应的最大回报值和所述获得最大回报值的车辆动作，训练极限学习机的网络模型；以及
策略网络生成模块，用于根据所述极限学习机的网络模型的训练结果，生成用于所述车辆自动驾驶的策略网络模型。
本发明将每个试验时刻采集到的车辆状态信息构成状态信息集，对状态信息集进行处理，以在降低状态信息集中数据量的同时保留状态信息集中的特征，获得特征状态信息集，根据预设的回报值函数和预先构建的车辆动作集，查找在特征状态信息集的每个状态信息下获得最大回报值的车辆动作，根据特征状态信息集中的状态信息、每个状态信息对应的最大回报值以及每个最大回报值对应的车辆动作，训练预设的极限学习机的网络模型，并根据训练结果，生成车辆自动驾驶的策略模型，从而通过对状态信息的处理、通过采用强化学习得到每个状态信息的最大回报值和最大回报值对应的车辆动作、通过训练极限学习及的网络模型，有效地降低了计算资源的消耗，有效地提高了车辆自动驾驶策略网络模型的泛化能力和生成效率。
附图说明
图1是本发明实施例一提供的用于车辆自动驾驶的策略网络模型的生成方法的实现流程图；
图2是本发明实施例一提供的用于车辆自动驾驶的策略网络模型的结构示例图；
图3是本发明实施例二提供的用于车辆自动驾驶的策略网络模型的生成装置的结构示意图；以及
图4是本发明实施例二提供的用于车辆自动驾驶的策略网络模型的生成装置的优选结构示意图。
具体实施方式
为了使本发明的目的、技术方案及优点更加清楚明白，以下结合附图及实施例，对本发明进行进一步详细说明。应当理解，此处所描述的具体实施例仅仅用以解释本发明，并不用于限定本发明。
以下结合具体实施例对本发明的具体实现进行详细描述：
实施例一：
图1示出了本发明实施例一提供的用于车辆自动驾驶的策略网络模型的生成方法的实现流程，为了便于说明，仅示出了与本发明实施例相关的部分，详述如下：
在步骤S101中，将每个试验时刻采集到的车辆状态信息构成状态信息集，对状态信息集进行处理，获得特征状态信息集。
本发明适用于基于赛车仿真模拟平台或赛车模拟器(例如开放式赛车模拟器TORCS，The open racing car simulation)建立的交互平台，在该交互平台上进行无人驾驶汽车的行驶交互试验。在当前的交互试验过程中，通过车辆上预设的多个传感器采集车辆状态信息，车辆状态信息可包括车辆离道路中线的距离、车辆前进方向与道路切向的夹角、车辆前方激光测距仪的距离值和车辆在道路切向上的速度分量。
在本发明实施例中，采集每个实验时刻的车辆状态信息，并将所有的车辆状态信息一起构成状态信息集，为了减少状态信息集中的数据量并保留状态信息集的特征，对状态信息集进行处理，生成特征状态信息集，从而通过对状态信息集的处理有效地降低后续过程的计算量。其中，特征状态信息集可包含具有代表性的状态信息或状态信息的特征。
具体地，可通过预设的聚类算法(例如K-means聚类)将状态信息集划分为若干个簇，并聚类出能够代表每个簇的簇中心点，这些簇中心点的状态信息构成特征状态信息集，此外，还可通过预设的特征提取算法(例如自编码机特征提取算法)提取状态信息集中的特征，由这些特征构成特征状态信息集。
在步骤S102中，根据预设的回报值函数和预先构建的车辆动作集，查找在特征状态信息集的每个状态信息下获得最大回报值的车辆动作。
在本发明实施例中，回报值函数用来计算在某个状态下车辆做出某个动作后获得的立即回报值，车辆动作集中可包括直行、刹车等车辆动作。先对特征状态信息集的每个状态信息对应的回报值序列进行初始化，可初始化为零，每 个状态信息的回报值序列包括在该状态信息下每个车辆动作对应的回报值，其中，状态信息s的回报值序列中车辆动作s的回报值可表示为
Figure PCTCN2017084081-appb-000001
在本发明实施例中，根据回报值函数和车辆动作集，对特征状态信息集中状态信息的回报值序列进行多次更新，直至所有回报值序列中的值不再变化，最后从所有回报值序列中选取特征状态信息集中每个状态信息的最大回报值以及该最大回报值对应的车辆动作，从而通过强化学习确定了特征状态信息集中每个状态信息最佳的车辆动作。
具体地，在更新回报值序列时，在车辆动作集中遍历可以在特征状态信息集的当前状态信息下执行的车辆动作，执行遍历得到的动作，并通过回报值函数计算执行后的立即回报值，根据该立即回报值和预设的更新公式，即可对在该当前状态信息下遍历到的车辆动作所对应的回报值。其中，回报值函数可表示为：
Figure PCTCN2017084081-appb-000002
其中，
Figure PCTCN2017084081-appb-000003
为在特征状态信息集的状态信息s下执行遍历到的动作a时的回报值，
Figure PCTCN2017084081-appb-000004
为状态信息s的下一状态信息s'的最大回报值，r为在状态信息s下执行遍历到的动作a时的立即回报值，γ为预设参数且0≤γ<1。
作为示例地，回报值函数的公式可为：
r＝Δdis*cos(α*angle)*sgn(trackPos-threshold)，其中，r为回报值函数计算得到的立即回报值，Δdis为车辆在相邻试验时刻跑过的有效距离，α为预设的权重缩放因子，angle为车辆当前行驶方向与道路切向的夹角，trackPos为车辆离道路中线的距离，threshold为预设阈值，当trackPos大于threshold时，r为无穷小，可表示对车辆太靠近道路边界时的惩罚。另外，回报值函数还可考虑行驶速度、单位耗油量、平稳度等。
在步骤S103中，根据特征状态信息集、特征状态信息集中每个状态信息对应的最大回报值和获得最大回报值的车辆动作，训练预设极限学习机的网络模型。
在本发明实施例中，极限学习机是一种单隐层前馈神经网络学习算法。其中，极限学习机的网络模型可表示为：
Figure PCTCN2017084081-appb-000005
其中，g( )为隐层节点的激活函数，βi＝[βi1,βi2,...,βin]T为连接第i个隐层节点与网络输出层节点的输出权值向量，ωi＝[ω1i,ω2i,...,ωmi]为连接第i个隐层节点的随机输入权值向量，极限学习机网络模型的输出值为Qi＝[Qi1,Qi2,...,Qin]T，Qj为输出值中的第j列向量，j＝1,2,...,N，N为输出节点的个数，M为隐层节点的个数。
在本发明实施例中，可将特征状态信息集中每个状态信息对应的最大回报值设置为极限学习机网络模型一次训练过程中的输出值，将特征状态信息集中的状态信息和每个状态信息下获得最大回报值的车辆动作最为极限学习机网络模型一次训练过程中的输入值，从而训练得到极限学习机网络模型中隐层节点与网络输出层节点的输出权值向量。可通过进行多次的车辆自动驾驶试验，多次地训练极限学习机网络模型。
在本发明实施例中，极限学习机网络模型的表示方式还可写为矩阵形式Wβ＝Q，当隐层节点的激活函数无限可微时，极限学习机网络模型的输入权值和隐层节点阈值可随机赋值，此时可将矩阵W视为常数，极限学习机的学习过程可等价为求取线性系统Wβ＝Q最小范数的最小二乘解。
在步骤S104中，根据极限学习机的网络模型的训练结果，生成用于车辆自动驾驶的策略网络模型。
在本发明实施例中，图2为策略网络模型的示例，由此可见，极限学习机网络模型中隐层节点与网络输出层节点的输出权值向量，即策略网络模型中隐藏层的权重值，最后得到了状态数据处理、强化学习和极限学习机网络模型构成的策略网络模型。
在本发明实施例中，对采集到的所有状态信息构成的状态信息集进行处理，以降低状态信息集中的数据量并保留状态信息集中的特征，生成特征状态信息 集，通过强化学习得到特征状态信息集中每个状态信息的最大回报值和该最大回报值对应的车辆动作，通过特征状态信息集中每个状态信息、每个状态信息的最大回报值、和最大回报值对应的车辆动作，训练极限学习机的网络模型，最后根据训练的到的参数向量，得到车辆自动驾驶的策略网络模型，从而通过对状态信息的处理、通过强化学习和极限学习机的结合，有效地降低了计算资源的消耗，有效地提高了车辆自动驾驶策略网络模型的泛化能力和生成效率。
本领域普通技术人员可以理解实现上述实施例方法中的全部或部分步骤是可以通过程序来指令相关的硬件来完成，所述的程序可以存储于一计算机可读取存储介质中，所述的存储介质，如ROM/RAM、磁盘、光盘等。
实施例二：
图3示出了本发明实施例二提供的用于车辆自动驾驶的策略网络模型的生成装置的结构，为了便于说明，仅示出了与本发明实施例相关的部分，其中包括：
状态处理模块31，用于将每个试验时刻采集到的车辆状态信息构成状态信息集，对状态信息集进行处理，获得特征状态信息集。
在本发明实施例中，采集每个实验时刻的车辆状态信息，并将所有的车辆状态信息一起构成状态信息集，为了减少状态信息集中的数据量并保留状态信息集的特征，对状态信息集进行处理，生成特征状态信息集，从而通过对状态信息集的处理有效地降低后续过程的计算量。其中，特征状态信息集可包含具有代表性的状态信息或状态信息的特征。
强化学习模块32，用于根据预设的回报值函数和预先构建的车辆动作集，查找在特征状态信息集的每个状态信息下获得最大回报值的车辆动作。
在本发明实施例中，每个状态信息的回报值序列包括在该状态信息下每个车辆动作对应的回报值，根据回报值函数和车辆动作集，对特征状态信息集中状态信息的回报值序列进行多次更新，直至所有回报值序列中的值不再变化，最后从所有回报值序列中选取特征状态信息集中每个状态信息的最大回报值以 及该最大回报值对应的车辆动作，从而通过强化学习确定了特征状态信息集中每个状态信息最佳的车辆动作。
具体地，在更新回报值序列时，在车辆动作集中遍历可以在特征状态信息集的当前状态信息下执行的车辆动作，执行遍历得到的动作，并通过回报值函数计算执行后的立即回报值，根据该立即回报值和预设的更新公式，即可对在该当前状态信息下遍历到的车辆动作所对应的回报值。其中，回报值函数可表示为：
Figure PCTCN2017084081-appb-000006
其中，
Figure PCTCN2017084081-appb-000007
为在特征状态信息集的状态信息s下执行遍历到的动作a时的回报值，
Figure PCTCN2017084081-appb-000008
为状态信息s的下一状态信息s'的最大回报值，r为在状态信息s下执行遍历到的动作a时的立即回报值，γ为预设参数且0≤γ<1。
极限学习机训练模块33，用于根据特征状态信息集、特征状态信息集中每个状态信息对应的最大回报值和获得最大回报值的车辆动作，训练极限学习机的网络模型。
在本发明实施例中，极限学习机的网络模型可表示为：
Figure PCTCN2017084081-appb-000009
其中，g( )为隐层节点的激活函数，βi＝[βi1,βi2,...,βin]T为连接第i个隐层节点与网络输出层节点的输出权值向量，ωi＝[ω1i,ω2i,...,ωmi]为连接第i个隐层节点的随机输入权值向量，极限学习机网络模型的输出值为Qi＝[Qi1,Qi2,...,Qin]T，Qj为输出值中的第j列向量，j＝1,2,...,N，N为输出节点的个数，M为隐层节点的个数。
在本发明实施例中，可将特征状态信息集中每个状态信息对应的最大回报值设置为极限学习机网络模型一次训练过程中的输出值，将特征状态信息集中的状态信息和每个状态信息下获得最大回报值的车辆动作最为极限学习机网络模型一次训练过程中的输入值，从而训练得到极限学习机网络模型中隐层节点与网络输出层节点的输出权值向量。可通过进行多次的车辆自动驾驶试验，多 次地训练极限学习机网络模型。
在本发明实施例中，极限学习机网络模型的表示方式还可写为矩阵形式Wβ＝Q，当隐层节点的激活函数无限可微时，极限学习机网络模型的输入权值和隐层节点阈值可随机赋值，此时可将矩阵W视为常数，极限学习机的学习过程可等价为求取线性系统Wβ＝Q最小范数的最小二乘解。
策略网络生成模块34，用于根据极限学习机的网络模型的训练结果，生成用于车辆自动驾驶的策略网络模型。
在本发明实施例中，极限学习机网络模型中隐层节点与网络输出层节点的输出权值向量，即策略网络模型中隐藏层的权重值，最后得到了状态数据处理、强化学习和极限学习机网络模型构成的策略网络模型。
优选地，如图4所示，状态处理模块31包括：
状态处理子模块411，用于通过预设的聚类算法对状态信息集进行聚类、或通过预设的特征提取算法对状态信息集进行特征提取，以生成特征状态信息集。
优选地，强化学习模块32包括回报值初始化模块421、回报值更新模块422和最大回报值获取模块423，其中：
回报值初始化模块421，用于对特征状态信息集的每个状态信息对应的回报值序列进行初始化，回报值序列包括执行车辆动作集中所有车辆动作对应的回报值；
回报值更新模块422，用于根据回报值函数和车辆动作集，对所有回报值序列进行多次更新，直至所有回报值序列不再变化；以及
最大回报值获取模块423，用于从特征状态信息的每个状态信息对应的回报值序列中获取最大回报值和最大回报值对应的车辆动作。
回报值更新模块422包括动作遍历模块和回报值更新子模块，其中：
动作遍历模块，用于遍历车辆动作集，并根据回报值函数计算在特征状态信息集的状态信息下执行遍历到的车辆动作时的立即回报值；以及
回报值更新子模块，用于根据立即回报值和预设的更新公式，更新在特征状态信息集的状态信息下执行遍历到的车辆动作对应的回报值。
优选地，极限学习机训练模块33包括学习机设置模块431和网络训练模块432，其中：
学习机设置模块431，用于根据特征状态信息集、特征状态信息集中每个状态信息对应的最大回报值和获得最大回报值的车辆动作，设置极限学习机网络模型的输入和输出；以及
网络训练模块432，用于对极限学习机网络模型进行训练，获得极限学习机网络模型的输出权重。
在本发明实施例中，对采集到的所有状态信息构成的状态信息集进行处理，以降低状态信息集中的数据量并保留状态信息集中的特征，生成特征状态信息集，通过强化学习得到特征状态信息集中每个状态信息的最大回报值和该最大回报值对应的车辆动作，通过特征状态信息集中每个状态信息、每个状态信息的最大回报值、和最大回报值对应的车辆动作，训练极限学习机的网络模型，最后根据训练的到的参数向量，得到车辆自动驾驶的策略网络模型，从而通过对状态信息的处理、通过强化学习和极限学习机的结合有效地降低了计算资源的消耗，有效地提高了车辆自动驾驶策略网络模型的泛化能力和生成效率。
在本发明实施例中，用于车辆自动驾驶的策略网络模型的装置的各模块可由相应的硬件或软件模块实现，各模块可以为独立的软、硬件模块，也可以集成为一个软、硬件模块，在此不用以限制本发明。
以上所述仅为本发明的较佳实施例而已，并不用以限制本发明，凡在本发明的精神和原则之内所作的任何修改、等同替换和改进等，均应包含在本发明的保护范围之内。

Claims (10)

  1. 一种用于车辆自动驾驶的策略网络模型的生成方法，其特征在于，所述方法包括下述步骤：
    将每个试验时刻采集到的车辆状态信息构成状态信息集，对所述状态信息集进行处理，获得特征状态信息集；
    根据预设的回报值函数和预先构建的车辆动作集，查找在所述特征状态信息集的每个状态信息下获得最大回报值的车辆动作；
    根据所述特征状态信息集、所述特征状态信息集中每个状态信息对应的最大回报值和所述获得最大回报值的车辆动作，训练预设极限学习机的网络模型；
    根据所述极限学习机的网络模型的训练结果，生成用于所述车辆自动驾驶的策略网络模型。
  2. 如权利要求1所述的方法，其特征在于，将每个试验时刻采集到的车辆状态信息构成状态信息集，对所述状态信息集进行处理，获得特征状态信息集的步骤，包括：
    通过预设的聚类算法对所述状态信息集进行聚类、或通过预设的特征提取算法对所述状态信息集进行特征提取，以生成所述特征状态信息集。
  3. 如权利要求1所述的方法，其特征在于，根据预设的回报值函数和预先构建的车辆动作集，查找在所述特征状态信息集的每个状态信息下获得最大回报值的车辆动作的步骤，包括：
    对所述特征状态信息集的每个状态信息对应的回报值序列进行初始化，所述回报值序列包括执行所述车辆动作集中所有车辆动作对应的回报值；
    根据所述回报值函数和所述车辆动作集，对所述所有回报值序列进行多次更新，直至所述所有回报值序列不再变化；
    从所述特征状态信息的每个状态信息对应的回报值序列中获取最大回报值和所述最大回报值对应的车辆动作。
  4. 如权利要求3所述的方法，其特征在于，根据所述回报值函数和所述车 辆动作集，对所述所有回报值序列进行多次更新，直至所述所有回报值序列不再变化的步骤，包括：
    遍历所述车辆动作集，并根据所述回报值函数计算在所述特征状态信息集的状态信息下执行遍历到的车辆动作时的立即回报值；
    根据所述立即回报值和预设的更新公式，更新在所述特征状态信息集的状态信息下执行所述遍历到的车辆动作对应的回报值，所述更新公式为：
    Figure PCTCN2017084081-appb-100001
    其中，
    Figure PCTCN2017084081-appb-100002
    为在所述特征状态信息集的状态信息s下执行所述遍历到的动作a时的回报值，
    Figure PCTCN2017084081-appb-100003
    为所述状态信息s的下一状态信息s'的最大回报值，r为在所述状态信息s下执行所述遍历到的动作a时的立即回报值，γ为预设参数且0≤γ<1。
  5. 如权利要求1所述的方法，其特征在于，根据所述特征状态信息集、所述特征状态信息集中每个状态信息对应的最大回报值和所述获得最大回报值的车辆动作，训练预设极限学习机的网络模型的步骤，包括：
    根据所述特征状态信息集、所述特征状态信息集中每个状态信息对应的最大回报值和所述获得最大回报值的车辆动作，设置所述极限学习机网络模型的输入和输出；
    对所述极限学习机网络模型进行训练，获得所述极限学习机网络模型的输出权重。
  6. 一种用于车辆自动驾驶的策略网络模型的生成装置，其特征在于，所述装置包括：
    状态处理模块，用于将每个试验时刻采集到的车辆状态信息构成状态信息集，对所述状态信息集进行处理，获得特征状态信息集；
    强化学习模块，用于根据预设的回报值函数和预先构建的车辆动作集，查找在所述特征状态信息集的每个状态信息下获得最大回报值的车辆动作；
    极限学习机训练模块，用于根据所述特征状态信息集、所述特征状态信息集中每个状态信息对应的最大回报值和所述获得最大回报值的车辆动作，训练 极限学习机的网络模型；以及
    策略网络生成模块，用于根据所述极限学习机的网络模型的训练结果，生成用于所述车辆自动驾驶的策略网络模型。
  7. 如权利要求6所述的装置，其特征在于，所述状态处理模块包括：
    状态处理子模块，用于通过预设的聚类算法对所述状态信息集进行聚类、或通过预设的特征提取算法对所述状态信息集进行特征提取，以生成所述特征状态信息集。
  8. 如权利要求6所述的装置，其特征在于，所述强化学习模块包括：
    回报值初始化模块，用于对所述特征状态信息集的每个状态信息对应的回报值序列进行初始化，所述回报值序列包括执行所述车辆动作集中所有车辆动作对应的回报值；
    回报值更新模块，用于根据所述回报值函数和所述车辆动作集，对所述所有回报值序列进行多次更新，直至所述所有回报值序列不再变化；以及
    最大回报值获取模块，用于从所述特征状态信息的每个状态信息对应的回报值序列中获取最大回报值和所述最大回报值对应的车辆动作。
  9. 如权利要求8所述的装置，其特征在于，所述回报值更新模块包括：
    动作遍历模块，用于遍历所述车辆动作集，并根据所述回报值函数计算在所述特征状态信息集的状态信息下执行遍历到的车辆动作时的立即回报值；以及
    回报值更新子模块，用于根据所述立即回报值和预设的更新公式，更新在所述特征状态信息集的状态信息下执行所述遍历到的车辆动作对应的回报值，所述更新公式为：
    Figure PCTCN2017084081-appb-100004
    其中，
    Figure PCTCN2017084081-appb-100005
    为在所述特征状态信息集的状态信息s下执行所述遍历到的动作a时的回报值，
    Figure PCTCN2017084081-appb-100006
    为所述状态信息s的下一状态信息s'的最大回报值，r为在所述状态信息s下执行所述遍历到的动作a时的立即回报值，γ为预设参数且0≤γ<1。
  10. 如权利要求6所述的装置，其特征在于，所述极限学习机训练模块包括：
    学习机设置模块，用于根据所述特征状态信息集、所述特征状态信息集中每个状态信息对应的最大回报值和所述获得最大回报值的车辆动作，设置所述极限学习机网络模型的输入和输出；以及
    网络训练模块，用于对所述极限学习机网络模型进行训练，获得所述极限学习机网络模型的输出权重。
PCT/CN2017/084081 2017-05-12 2017-05-12 一种用于车辆自动驾驶的策略网络模型的生成方法及装置 Ceased WO2018205245A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
PCT/CN2017/084081 WO2018205245A1 (zh) 2017-05-12 2017-05-12 一种用于车辆自动驾驶的策略网络模型的生成方法及装置

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2017/084081 WO2018205245A1 (zh) 2017-05-12 2017-05-12 一种用于车辆自动驾驶的策略网络模型的生成方法及装置

Publications (1)

Publication Number Publication Date
WO2018205245A1 true WO2018205245A1 (zh) 2018-11-15

Family

ID=64104132

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2017/084081 Ceased WO2018205245A1 (zh) 2017-05-12 2017-05-12 一种用于车辆自动驾驶的策略网络模型的生成方法及装置

Country Status (1)

Country Link
WO (1) WO2018205245A1 (zh)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111738098A (zh) * 2020-05-29 2020-10-02 浪潮(北京)电子信息产业有限公司 一种车辆识别方法、装置、设备及存储介质
CN112258097A (zh) * 2020-12-23 2021-01-22 睿至科技集团有限公司 一种基于大数据的辅助驾驶方法和系统
CN121209404A (zh) * 2025-11-28 2025-12-26 天津市博瑞特旅游观光火车有限公司 一种双向行驶轮式观光火车的智能控制方法及系统

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6493614B1 (en) * 2001-12-24 2002-12-10 Samsung Electronics Co., Ltd. Automatic guided system and control method thereof
CN103605285A (zh) * 2013-11-21 2014-02-26 南京理工大学 汽车驾驶机器人系统的模糊神经网络控制方法
CN106080590A (zh) * 2016-06-12 2016-11-09 百度在线网络技术（北京）有限公司 车辆控制方法和装置以及决策模型的获取方法和装置
CN106428000A (zh) * 2016-09-07 2017-02-22 清华大学 一种车辆速度控制装置和方法
CN107229973A (zh) * 2017-05-12 2017-10-03 中国科学院深圳先进技术研究院 一种用于车辆自动驾驶的策略网络模型的生成方法及装置

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6493614B1 (en) * 2001-12-24 2002-12-10 Samsung Electronics Co., Ltd. Automatic guided system and control method thereof
CN103605285A (zh) * 2013-11-21 2014-02-26 南京理工大学 汽车驾驶机器人系统的模糊神经网络控制方法
CN106080590A (zh) * 2016-06-12 2016-11-09 百度在线网络技术（北京）有限公司 车辆控制方法和装置以及决策模型的获取方法和装置
CN106428000A (zh) * 2016-09-07 2017-02-22 清华大学 一种车辆速度控制装置和方法
CN107229973A (zh) * 2017-05-12 2017-10-03 中国科学院深圳先进技术研究院 一种用于车辆自动驾驶的策略网络模型的生成方法及装置

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111738098A (zh) * 2020-05-29 2020-10-02 浪潮(北京)电子信息产业有限公司 一种车辆识别方法、装置、设备及存储介质
CN111738098B (zh) * 2020-05-29 2022-06-17 浪潮(北京)电子信息产业有限公司 一种车辆识别方法、装置、设备及存储介质
CN112258097A (zh) * 2020-12-23 2021-01-22 睿至科技集团有限公司 一种基于大数据的辅助驾驶方法和系统
CN112258097B (zh) * 2020-12-23 2021-03-26 睿至科技集团有限公司 一种基于大数据的辅助驾驶方法和系统
CN121209404A (zh) * 2025-11-28 2025-12-26 天津市博瑞特旅游观光火车有限公司 一种双向行驶轮式观光火车的智能控制方法及系统

Similar Documents

Publication Publication Date Title
CN107229973B (zh) 一种用于车辆自动驾驶的策略网络模型的生成方法及装置
CN114358128B (zh) 一种训练端到端的自动驾驶策略的方法
Ashraf et al. Optimizing hyperparameters of deep reinforcement learning for autonomous driving based on whale optimization algorithm
US11899411B2 (en) Hybrid reinforcement learning for autonomous driving
CN112099496B (zh) 一种自动驾驶训练方法、装置、设备及介质
CN107169567B (zh) 一种用于车辆自动驾驶的决策网络模型的生成方法及装置
CN111483468B (zh) 基于对抗模仿学习的无人驾驶车辆换道决策方法和系统
Li et al. Humanlike driving: Empirical decision-making system for autonomous vehicles
CN112508164B (zh) 一种基于异步监督学习的端到端自动驾驶模型预训练方法
CN113239986B (zh) 车辆轨迹评价网络模型的训练方法、装置和存储介质
Li et al. Simulation of vehicle interaction behavior in merging scenarios: A deep maximum entropy-inverse reinforcement learning method combined with game theory
CN113609784B (zh) 一种交通极限场景生成方法、系统、设备和存储介质
CN111572562A (zh) 自动驾驶方法、装置、设备、系统、车辆和计算机可读存储介质
CN114701517B (zh) 基于强化学习的多目标复杂交通场景下自动驾驶解决方法
CN114926823A (zh) 基于wgcn的车辆驾驶行为预测方法
CN110196587A (zh) 车辆自动驾驶控制策略模型生成方法、装置、设备及介质
CN115700626A (zh) 用于车辆的奖励函数
CN115031753B (zh) 基于安全势场和dqn算法的行车工况局部路径规划方法
Tang et al. Personalized decision-making and control for automated vehicles based on generative adversarial imitation learning
WO2018205245A1 (zh) 一种用于车辆自动驾驶的策略网络模型的生成方法及装置
Lu et al. Enhancing interpretability of autonomous driving via human-like cognitive maps: A case study on lane change
Wang et al. An end-to-end deep reinforcement learning model based on proximal policy optimization algorithm for autonomous driving of off-road vehicle
CN113188556A (zh) 基于行车安全场的智能网联汽车轨迹规划方法及装置
CN113753049B (zh) 基于社会偏好的自动驾驶超车决策确定方法及系统
CN110390398A (zh) 在线学习方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 17909026

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 17909026

Country of ref document: EP

Kind code of ref document: A1

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 25.08.2020)

122 Ep: pct application non-entry in european phase

Ref document number: 17909026

Country of ref document: EP

Kind code of ref document: A1