JP2019096012A - Method and device for controlling mobile body - Google Patents

Abstract

To provide a method and a device for controlling a mobile body that can consider a signal state and situations of obstacles at one time and can reduce the time to a destination, for example.SOLUTION: The method for controlling a mobile body according to the present invention includes: grasping the position of a mobile body recognized by a camera, for example, the distance to a vehicle in front of or behind the mobile body, the distance to a vehicle in the next lane in front of or behind the mobile body, and the state of cycles of the lamp of a signal obtained from an optical beacon; calculating a feature amount vector used for reinforcement learning from the state; and calculating a control guideline, using a reward value obtained from the feature amount vector and the control guideline at the present moment, by the reinforcement learning.SELECTED DRAWING: Figure 1

Description

  The present disclosure relates to technology for controlling the operation of a mobile.

  As a technology for controlling the movement of a mobile, research has been made to steer the mobile so as to avoid an obstacle.

U. S. of Japan, “Amis: Advanced mobile information systems”, http: // www. utms. or. jp / english / system / amis. html. VICS, "Beacon and fm broadcasting", https: // www. vics. or. jp / en / vics / beacon. html. HONDA, “Honda sensing technology”, http: // www. honda. co. jp / hondasensing /. W. Liu, J.J. Liu, J.J. Peng, and Z. Zhu, “Cooperative multi-agent traffic signal control system using fast gradient-descent function approximation for v2i networks”, in Proc. IEEE International Conference on Communications (ICC), 2014, pp. 2562-2567. W. Lu, Y. Zhang, and Y. Xie, "A multi-agent adaptive traffic signal control system using swarm intelligence and neuro-fuzzy reinforcement learning", in Proc. IEEE Forum on Integrated and Sustainable Transportation System (FISTS), 2011, pp. 233-238. TOYOTA, “Toyota to boost investment in artificial intelligence by strength relationship with preferred networks inc”. http: // newsroom. toyota. co. jp / en / detail / 10679722 /. R. S. Sutton and A. G. Barto, Introduction to reinforcement learning. MIT Press Cambridge, 1998, vol. 135. V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski et al. , "Human-level control through deep reinforcement learning", Nature, vol. 518, no. 7540, pp. 529-533, 2015. H. Van Hasselt, A. Guez, and D. Silver, “Deep reinforcement learning with double q-learning”. in AAAI, 2016, pp. 2094-2100. Masakazu Mukai, Hiroshi Aoki, Taketoshi Kawashima, "Model prediction type fuel saving driving control by mixed integer programming using traffic light information", Proc. Of the Society of Measurement and Control Engineers, vol. 51, no. 12, pp. 866-872, 2015. V. Nair and G. E. Hinton, “Rectified linear units improved restricted boltzmann machines”, in Proc. International Conference on Machine Learning (ICML), 2010, pp. 807-814. S. Adam, L. Bosoniu, and R. Babuska, "Experience replay for realtime reinforcement learning control", IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), vol. 42, no. 2, pp. 201-212, 2012.

  In recent years, regarding acquisition of signal information, using the advanced light beacon installed on the roadside by recent technological development, route signal information from the traffic control center (distance information to a signal in the traveling direction, traffic signals installed at intersections Since it became possible to provide a car with a remaining time information of a red light, etc., the feasibility of automation of driving assistance rapidly increased (see, for example, non-patent documents 1 and 2). When considering driving assistance on a general road based on these technologies, after acquiring the signal status, the driving assistance technology that can arrive at the destination quickly by adding the signal status and the behavior of other vehicles around is available. It is considered necessary.

  With regard to driving support technology that takes into account signal conditions, techniques aimed at avoiding red light and reducing waiting time have been proposed, and acceleration / deceleration adjustment is performed to avoid stopping at red light from the signal condition one ahead. (See, for example, Non-Patent Document 3) or a method in which the signal itself adjusts the distance between lamps (for example, see Non-Patent Documents 4 and 5). There are also many technologies that drive safely from the behavior of other vehicles, and in recent years, technologies have been proposed that use the deep learning method to detect the situation of obstacles and other vehicles and automatically perform avoidance actions. (For example, refer to nonpatent literature 6).

  The driving support technology that takes into account the signal state takes into account only the latest signal, not the signal state ahead, so the stop time by the red signal to the destination and the time to the destination Etc. can not be shortened. The existing driving support technology that takes into consideration environmental changes can avoid the obstacle (movement of another vehicle), but there is no driving support technology that simultaneously considers the signal status and the like.

  Therefore, an object of the present invention is to provide a moving object control method and a moving object control device capable of shortening the time to the destination etc. by simultaneously considering the signal state and the situation of the obstacle.

  In order to achieve the above object, according to the mobile object control method of the present invention, the position of the mobile object recognized by a camera or the like, the distance between the front and rear of the mobile, and the distance between the front and rear of the adjacent lane , And the state such as the lamp cycle of the signal obtained from the light beacon, calculate the feature quantity vector used in reinforcement learning from the state, calculate the reward value obtained from the feature quantity vector and control guideline at the current time by reinforcement learning We used it to calculate control guidelines.

Specifically, the mobile object control method according to the present invention is
A state grasping procedure for acquiring a position of a moving object, an alerting time until a plurality of stop commands for the moving object are issued, a distance from the moving object to each of the stop commands, and a distance to another moving object; ,
The spatio-temporal distance of the number of stop instructions is calculated from the position of the moving body from the current speed of the moving body and the distance from the notification time to the stop command, the current speed, the space-time A feature amount extraction procedure for obtaining a feature amount vector configured by a distance and a distance to the other moving object;
Reward value representing the avoidance of the stop command and the avoidance of the contact with the other moving object, obtained as a result of performing a control guideline for causing the moving body to perform at least one of acceleration and deceleration and direction change with respect to the feature quantity vector. A learning control procedure for performing reinforcement learning using a controller, calculating a new control guideline, and controlling the mobile object;
I do.

Further, according to the mobile object control device of the present invention,
A state grasping unit that acquires a position of a moving object, an alerting time until a plurality of stop commands for the moving object are issued, a distance from the moving object to each of the stop commands, and a distance to another moving object ,
The spatio-temporal distance of the number of stop instructions is calculated from the position of the moving body from the current speed of the moving body and the distance from the notification time to the stop command, the current speed, the space-time A feature amount extraction unit that obtains a feature amount vector configured by a distance and a distance to the other moving object;
Reward value representing the avoidance of the stop command and the avoidance of the contact with the other moving object, obtained as a result of performing a control guideline for causing the moving body to perform at least one of acceleration and deceleration and direction change with respect to the feature quantity vector. A learning control unit that performs reinforcement learning using the control unit, calculates a new control guideline, and controls the mobile unit;
Equipped with

  Since the distance to the surrounding car and the situation of the signal to the destination (the timing of the red light) are included in the feature quantity vector to derive control guidelines for the acceleration and deceleration of the vehicle, the speed of the vehicle is adjusted and the stop time is adjusted. Can be shortened. Therefore, the present invention can provide a mobile control method and a mobile control apparatus capable of shortening the time to the destination etc. by simultaneously considering the signal status and the situation of the obstacle.

  The present invention can provide a mobile control method and a mobile control apparatus capable of shortening the time to the destination etc. by simultaneously considering the signal status and the situation of the obstacle.

It is a flow chart explaining the mobile control method concerning the present invention. It is a figure explaining the feature-value vector of the mobile used by the mobile control method which concerns on this invention. It is a figure explaining the result of acceleration of the vehicle with respect to a signal state among the reward functions judged by the moving body control method which concerns on this invention. It is a figure explaining the mobile control device concerning the present invention.

  Embodiments of the invention will be described with reference to the accompanying drawings. The embodiments described below are examples of the present invention, and the present invention is not limited to the following embodiments. In the present specification and drawings, components having the same reference numerals denote the same components.

  Reinforcement learning is a method of setting values of states, actions, and rewards according to the environment, and calculating an action that maximizes the cumulative sum of rewards in all the set states. It is applied. The driving support technology in which the signal information and the behavior of the other vehicle, which is the object of the present invention, is taken into consideration is achieved using three modules, which include the state grasping unit 11, the feature extraction unit 12, and the learning control unit 13. (See FIG. 1).

The mobile control device 301 of FIG.
A state grasping unit 11 that acquires the position of a moving object, an alerting time until a plurality of stop commands for the moving object is issued, the distance from the moving object to each of the stop commands, and the distance to another moving object. When,
The spatio-temporal distance of the number of stop instructions is calculated from the position of the moving body from the current speed of the moving body and the distance from the notification time to the stop command, the current speed, the space-time A feature amount extraction unit 12 for obtaining a feature amount vector configured by a distance and a distance to the other moving object;
Reward value representing the avoidance of the stop command and the avoidance of the contact with the other moving object, obtained as a result of performing a control guideline for causing the moving body to perform at least one of acceleration and deceleration and direction change with respect to the feature quantity vector. A learning control unit 13 that performs reinforcement learning using the control unit, calculates a new control pointer, and controls the mobile unit;
Equipped with

FIG. 1 is a flow chart for explaining a mobile control method according to the present embodiment. This mobile control method is
The state grasping unit 11 determines the position of the moving body, the notification time until a plurality of stop commands are issued to the moving body, the distance between the moving body and each of the stop commands, and the distance to another moving body. With S11 to acquire,
The feature amount extraction unit 12 calculates the spatio-temporal distance of the number of the stop commands from the position of the moving body to the current velocity of the moving body, and the distance between the notification time and the stop command, A feature quantity extraction procedure S12 for acquiring a feature quantity vector composed of the velocity, the space-time distance, and the distance to the other moving object;
Avoidance of the stop command and contact with the other moving object obtained as a result of the learning control unit 13 performing a control guideline for causing the moving body to perform at least one of acceleration and deceleration and direction change with respect to the feature quantity vector A learning control procedure S13 for performing reinforcement learning using a reward value representing the avoidance of the following and calculating a new control guideline to control the mobile object;
I do.

[Status grasping unit]
The state grasping part 11 can acquire the current position of the moving object, the lamp cycle of the signal obtained from the light beacon, the distance of the vehicle to the front and rear of the moving object, and the distance to the vehicle of the front and rear in both adjacent lanes I assume. In addition, about an acquisition method, a vehicle-mounted sensor * camera etc. can be used.

ここで、ｖは移動体の現在の速度（履歴）、（ｄｔ_１、ｄｔ_２、・・・、ｄｔ_ｎ）は得られた複数の信号情報から各信号の赤信号（停止指令）になるまでの時間と赤信号までの距離を加味したｎ個の時空間距離、（ｄｆ_１、ｄｆ_２、ｄｆ_３）は現在の車線と両隣の車線の前方の車までの距離、（ｄｂ_１、ｄｂ_２、ｄｂ_３）は現在の車線と両隣の車線の後方の車までの距離である。なお、特徴量ベクトルの各距離は、任意の定数との除算によって［０，１］に正規化し、除算結果が１を超える場合は１とみなす。 [Feature extraction unit]
The feature quantity extraction unit 12 creates a feature quantity vector used in reinforcement learning from the information obtained from the state grasping unit 11 and passes it to the learning control unit 13. The equation 1 is an example of the feature quantity vector s _t .

Here, v is the current velocity (history) of the mobile unit, (dt ₁ , dt ₂ ,..., Dt _n ) until the red signal (stop command) of each signal is obtained from the plurality of obtained signal information N space-time distances (df ₁ , df ₂ , df ₃ ) taking into account the time and the distance to the red light, the distance to the car ahead of the current lane and both adjacent lanes, (db ₁ , db ₂ , Db ₃ ) is the distance to the car behind the current lane and both adjacent lanes. Note that each distance of the feature amount vector is normalized to [0, 1] by division with an arbitrary constant, and when the division result exceeds 1, it is regarded as 1.

  FIG. 2 is a diagram for explaining the outline of the space-time distance. The horizontal axis represents time, and the vertical axis represents the direction of travel to the destination. The position of each signal and the timing of the red signal are described here, and a vector including the distance from the vehicle to the red signal and the time is the space-time distance. In FIG. 2, a broken line indicates an ideal route (a route of a controlled mobile body) traveling while avoiding a red light.

  Experiments have shown that it is possible to avoid such red light as well as the most recent red light by using such space-time distances. In addition, regarding the distance to other vehicles, it is also possible to use the transition history of the distance of other vehicles. In addition, when the number of lanes is 2 or less, the distance to the car of the front back in the non-existent lane is set to 0.

[Learning control unit]
With respect to the obtained feature quantity vector, the learning control unit 13 avoids a red light section shown in FIG. Determine the change etc.) and execute. With this control guideline, driving support can be achieved in consideration of the signal state and the behavior of other vehicles. The learning control unit 13 uses reinforcement learning. In reinforcement learning, the current (time t), for the feature quantity vector s _t that observed number 2 obtained when executing the control pointer a _t using a compensation value, the control pointer a in s _t Update the value Q (s _t , a _t ) as in _equation 3.

α (0 ≦ α ≦ 1) indicates a learning rate, and γ (0 ≦ γ ≦ 1) indicates a discount rate. When α is large, the latest reward is emphasized, and when α is 1, the past reward is not considered at all. In addition, γ represents the influence of the control evaluation value for the transition destination state on the current control evaluation value, and when γ is 0, the control evaluation value for the transition destination state s _{t + 1 is} the control evaluation value of the current state s _t Does not depend on

  This update formula is called Q learning (see, for example, non-patent document 7), and the evaluation value Q of control which can obtain the largest reward value by recursively performing the above update. It is possible to maximize s, a) theoretically.

Next, the reward function for avoiding contact with red light and other vehicles, the result B (a _t) of the acceleration of the vehicle with respect to the signal _condition, the sum T of the space-time distances in the current state s _t, et acceleration collision determination that vehicle C (a _t), and using the sum D of the distance to the front and rear side of the car lane of the current lane and both sides, expressed as follows.

結果Ｂ（ａ_ｔ）は、下記の３つの値域をとる値であり、図３にその概要を示す。

衝突判定Ｃ（ａ_ｔ）は下記の二つの値を取る。

総和Ｄは、数８の通りである。

The parameters are as follows.
The sum T is as shown in Equation 5.

Result B (a _t) is a value taking three range below shows the outline in Figure 3.

Collision determination C (a _t) takes two values below.

The total sum D is as shown in Equation 8.

It has been confirmed by experiments that reinforcement learning using the feature quantity vector and the reward function defined above can operate at high speed while avoiding red light and other vehicles. Note that Q (s _t , a _t ) may be enormous depending on the number of feature amounts and the value range. In this case, calculation time can be shortened by using deep reinforcement learning (see, for example, non-patent documents 8 and 9) (see, for example, non-patent documents 10-12).

11: state grasping unit 12: feature amount extraction unit 13: learning control unit

Claims (2)

A state grasping procedure for acquiring a position of a moving object, an alerting time until a plurality of stop commands for the moving object are issued, a distance from the moving object to each of the stop commands, and a distance to another moving object; ,
The spatio-temporal distance of the number of stop instructions is calculated from the position of the moving body from the current speed of the moving body and the distance from the notification time to the stop command, the current speed, the space-time A feature amount extraction procedure for obtaining a feature amount vector configured by a distance and a distance to the other moving object;
Reward value representing the avoidance of the stop command and the avoidance of the contact with the other moving object, obtained as a result of performing a control guideline for causing the moving body to perform at least one of acceleration and deceleration and direction change with respect to the feature quantity vector. A learning control procedure for performing reinforcement learning using a controller, calculating a new control guideline, and controlling the mobile object;
Do mobile control method. A state grasping unit that acquires a position of a moving object, an alerting time until a plurality of stop commands for the moving object are issued, a distance from the moving object to each of the stop commands, and a distance to another moving object ,
The spatio-temporal distance of the number of stop instructions is calculated from the position of the moving body from the current speed of the moving body and the distance from the notification time to the stop command, the current speed, the space-time A feature amount extraction unit that obtains a feature amount vector configured by a distance and a distance to the other moving object;
Reward value representing the avoidance of the stop command and the avoidance of the contact with the other moving object, obtained as a result of performing a control guideline for causing the moving body to perform at least one of acceleration and deceleration and direction change with respect to the feature quantity vector. A learning control unit that performs reinforcement learning using the control unit, calculates a new control guideline, and controls the mobile unit;
Mobile control device comprising:

Images