JP7381804B1 - Route control device and route control method - Google Patents

Abstract

An object of the present invention is to remotely control the course of a vehicle using a simpler configuration. [Solution] Course control that controls the course of the vehicle 2 from the position of the communication area A1 at the initial point to the position of the communication area An at the destination point in the movement space of the vehicle 2 defined by a plurality of communication areas A1 to An. A first acquisition unit of the device 1 that acquires the position of the communication area A associated with the location registration signal transmitted from the vehicle 2 when crossing the communication area A as the current position of the vehicle 2; 10, and the current position acquired by the first acquisition unit 10 as the current position of the vehicle 2 based on the route the vehicle 2 should take sequentially from the position of each communication area A, which is learned using the learning model. A determining unit 15 determines the next route for the vehicle 2 from the position of the communication area A, and a determining unit 15 determines the next route for the vehicle 2 determined by the determining unit 15 via the core network 3 of a predetermined communication standard. and a route control unit 16 that instructs the vehicle 2 to follow the directions. [Selection diagram] Figure 1

Description

The present invention relates to a course control device and a course control method.

Conventionally, vehicle position estimation technology for autonomous vehicle driving has been based on signals from positioning satellites such as GPS, and six-axis inertial sensors (inertial measurement units) that detect vehicle behavior to complement signals from positioning satellites. Distance Measuring Instruments (DMI) that measure the distance traveled by a vehicle by measuring the number of rotations of tires are known.

In situations where communication conditions are poor, such as when a vehicle is driving in a tunnel, GPS signals from positioning satellites may not be received. However, even with the IMU and DMI used to supplement GPS, the accuracy of estimating the vehicle's position may not be sufficient depending on the situation. For example, self-vehicle position estimation using an IMU has the disadvantage that errors tend to accumulate. In self-vehicle position estimation using DMI, measurement accuracy may decrease when the vehicle speed or direction of the vehicle changes.

In this way, in conventional self-driving vehicle position estimation, when the reception of radio waves from GPS positioning satellites is poor, the accuracy of self-position estimation decreases, and the self-driving ECU of the self-driving vehicle is unable to determine the course or route. It was sometimes difficult to make accurate selections.

Therefore, when controlling automatic driving using a combination of GPS and map information, when a communication delay occurs, Patent Document 1 collects information on the area where the vehicle is located, creates current situation map information, and sets the The present invention discloses a technology that performs predictive calculation of the travel area of a vehicle to a destination point based on the destination point and current situation map information.

However, the technology disclosed in Patent Document 1 collects various information such as vehicle speed and surrounding information, generates a map, and performs predictive calculations of the vehicle's driving area, which makes the predictive calculations complicated and increases the calculation load. It will be huge.

JP 2021-111329 Publication

As described above, with the conventional technology, it has not been possible to remotely control the course of a vehicle using a simpler configuration.

The present invention has been made to solve the above-mentioned problems, and an object of the present invention is to remotely control the course of a vehicle with a simpler configuration.

In order to solve the above-mentioned problems, a route control device according to the present invention provides a route control device for moving a vehicle from a position of a communication area at an initial point to a position of a communication area at a destination point in a movement space of a vehicle defined by a plurality of communication areas. A route control device that controls the route of a vehicle, the location of which is associated with a location registration signal from the vehicle transmitted when crossing a communication area, as the current location of the vehicle. The first acquisition unit is configured to acquire information about the vehicle based on the strategy of the route that the vehicle should take sequentially from the position of each communication area, which has been learned using a learning model. a determining unit configured to determine the next course that the vehicle should take from the current communication area position acquired as the current position; and a route control unit configured to instruct the vehicle on a desired route via a core network of a predetermined communication standard.

Further, in the route control device according to the present invention, the vehicle may move from the position of each of the communication areas until the vehicle reaches the position of the communication area of the destination point from the position of the communication area of the initial point. A reward function is applied to the estimation result of calculating the route to be taken sequentially, and the reward function is updated so that the reward for the vehicle to reach the position of the communication area of the destination point is maximized. a learning section configured to use the learning model to learn strategies for the course to proceed sequentially from the position of , and a memory configured to store the strategies for the route learned by the learning section. The determination unit may read out the route strategy from the storage unit and determine the route the vehicle should take next.

In order to solve the above-mentioned problems, the route control device according to the present invention further includes a second acquisition unit configured to acquire the position of a communication area corresponding to an area where traffic congestion occurs in the movement space. And, the reward function uses as variables the degree of arrival of the vehicle to the position of the communication area related to the destination point and the degree of arrival of the vehicle to the position of the communication area corresponding to the area where the traffic jam occurs. May contain.

Further, in the route control device according to the present invention, the learning model is a neural network model including an input layer, a hidden layer, and an output layer, and the learning unit is configured to calculate the current communication area position from the neural network model. is given as an input, the neural network model is calculated, and the next course that the vehicle should take from the current communication area position is obtained when each action including turning right, turning left, and going straight is taken. The learning unit outputs a first estimated value of an action value function representing an expected value of the cumulative value of the reward in the future, and the learning unit further uses the position of the communication area that the vehicle reaches next as an input to the neural network model. and calculates the neural network model to output a second estimated value of the action value function, and the learning unit is configured to make the first estimated value a target value calculated from the second estimated value. The weight parameters of the neural network model may be learned, and the storage unit may store the learned weight parameters.

Further, in the route control device according to the present invention, the first acquisition unit acquires the current communication area position of the vehicle from an integrated data repository that manages subscriber information included in the core network; The route control unit may transmit an instruction regarding the next route to the vehicle via a user plane function included in the core network.

In order to solve the above-mentioned problems, the route control method according to the present invention provides the route control method from the position of the communication area of the initial point to the position of the communication area of the destination point in the movement space of the vehicle defined by a plurality of communication areas. A route control method for controlling the route of a vehicle, wherein the current location of the vehicle is determined by determining the location of the communication area associated with a location registration signal from the vehicle that is transmitted when the vehicle crosses the communication area. The current position of the vehicle is acquired in the first acquisition step based on the strategy of the course that the vehicle should take sequentially from the position of each communication area, which is learned using a learning model. a determining step of determining the next course that the vehicle should take based on the current communication area position acquired as the position; and a route control step of instructing the vehicle via a core network.

Further, in the route control method according to the present invention, the vehicle may sequentially move from the communication area position of each of the communication areas until the vehicle reaches the communication area position of the destination point from the communication area position of the initial point. A reward function is applied to the estimated result of calculating the course to be taken, and the reward for the vehicle to reach the communication area of the destination point is updated so as to maximize the reward for the vehicle to reach the position of the communication area of the destination point. a learning step configured to learn, using the learning model, a strategy for the route to be taken sequentially from a position; and a storage step for storing in a storage unit the strategy for the route learned in the learning step. In the determining step, the route strategy may be read from the storage unit to determine the route the vehicle should take next.

Further, in the route control method according to the present invention, the second obtaining step is configured to obtain a position of a communication area corresponding to an area where traffic congestion occurs in the moving space, and the reward function includes: The variables may include a degree of arrival of the vehicle to a position in a communication area related to the destination point and a degree of arrival of the vehicle to a position of a communication area corresponding to the area where the traffic jam occurs.

Further, in the route control method according to the present invention, the learning model is a neural network model including an input layer, a hidden layer, and an output layer, and the learning step includes determining the current communication area position from the neural network model. is given as an input, the neural network model is calculated, and the future path obtained when the vehicle takes each action, including turning right, turning left, and going straight, is calculated as the next course the vehicle should take from the current communication area. outputting a first estimated value of an action value function representing an expected value of the cumulative value of the reward; the learning step further includes providing a position of a communication area that the vehicle reaches next as an input to the neural network model; Calculating the neural network model and outputting a second estimated value of the action value function, the learning step is such that the first estimated value becomes a target value calculated from the second estimated value. The step of learning weight parameters of the neural network model and storing the learned weight parameters may be stored in the storage unit.

Further, in the route control method according to the present invention, the first acquisition step acquires the current communication area position of the vehicle from an integrated data repository that manages subscriber information included in the core network; The route control step may include transmitting an instruction regarding the next route to the vehicle via a user plane function included in the core network.

According to the present invention, the next course that the vehicle should take from the current communication area position is determined based on the course that the vehicle should take sequentially from the position of each communication area, which is learned using a learning model. . Therefore, the route of the vehicle can be controlled remotely with a simpler configuration.

FIG. 1 is a block diagram showing the configuration of a route control system including a route control device according to an embodiment of the present invention. FIG. 2 is a diagram for explaining an overview of the route control system according to the present embodiment. FIG. 3 is a diagram for explaining learning processing by the learning section according to the present embodiment. FIG. 4 is a block diagram showing the configuration of the learning section according to this embodiment. FIG. 5 is a block diagram showing the hardware configuration of the route control device according to this embodiment. FIG. 6 is a flowchart showing the learning process of the route control device according to the present embodiment. FIG. 7 is a flowchart showing the learning process of the route control device according to the present embodiment. FIG. 8 is a flowchart showing the route control process of the route control device according to the present embodiment.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to FIGS. 1 to 8.

FIG. 1 is a block diagram showing the configuration of a route control system including a route control device 1 according to an embodiment of the present invention. The route control system according to the present embodiment uses a position registration signal sent when the vehicle 2 crosses the communication areas A1 to An in a movement space of the vehicle 2 defined by a plurality of communication areas A1 to An. The course of the vehicle 2 from the initial point in the communication area A1 to the destination point in the communication area An is controlled.

[Configuration of route control system]
First, an outline of a course control system including a course control device 1 according to an embodiment of the present invention will be explained. As shown in FIG. 1, the route control system includes a route control device 1, a vehicle 2, base stations BS1 to BSn, and a core network 3, which are compatible with, for example, an SA type 5G wireless communication system.

The base stations BS1 to BSn are configured of wireless base stations compatible with the 5G system, and relay communications between the core network 3 and the vehicles 2 located in the communication areas A1 to An. In the following, when base stations BS1 to BSn and communication areas A1 to An are not distinguished from each other, they may be collectively referred to as base station BS and communication area A, respectively.

As shown in FIG. 1, the communication areas A1 to An of each base station BS1 to BSn define a movement space in which the vehicle 2 moves. Further, each of the communication areas A1 to An is arranged to cover a section of a road such as a main road along which the vehicle 2 travels from the initial point to the destination point. In this embodiment, it is assumed that communication areas A1 to An have cells of the same size.

FIG. 2 is a diagram schematically showing a movement space to be controlled by the course control system. Each dotted circle in FIG. 2 indicates a communication area A1 to A16 covered by each base station BS1 to BS16 arranged in the moving space. The vehicle 2 moves from the initial point S to the destination point G in the communication area A16 along a course indicated by an arrow, using the communication areas A1 to A16 of the base stations BS1 to BS16 as waypoints.

In this embodiment, the vehicle 2 reaches from the initial point S to the destination point G by turning right, turning left, or going straight in each of the communication areas A1 to A16 according to the route instructions given by the route control system. Further, it is assumed that the route instruction is to turn right, left, or go straight based on the direction in which the vehicle 2 is traveling. For example, if there is a direction instruction to go straight at the position of the communication area A1, the vehicle 2 will go straight from the position of the communication area A1 and reach the next communication area A2. Further, if a left turn course instruction is given from the position of the communication area A2, the vehicle 2 turns left and reaches the next communication area A6. In this way, the vehicle 2 moves in the movement space according to the route instructions for each of the communication areas A1 to A16.

The vehicle 2 is equipped with a communication terminal 20 . Vehicles 2 include automobiles, motorized vehicles, motorcycles, and the like. The communication terminal 20 includes a processor, a main storage device, an auxiliary storage device, a communication interface, etc., and is a terminal device installed in the vehicle 2, a mobile communication terminal such as a smartphone of a user using the vehicle 2, a tablet computer, etc. It is realized as.

Specifically, the communication terminal 20 includes a SIM 21. The vehicle 2 is uniquely identified by the IMSI (International Mobile Subscriber Identity) of the SIM 21 included in the communication terminal 20 .

When the communication terminal 20 straddles the communication areas A1 to An as the vehicle 2 moves, the processor of the communication terminal 20 sends a location registration signal (TAU) to base stations BS1 to BSn in the new communication areas A1 to An. Send.

The vehicle 2 also includes an ECU (Electronic Control Unit) 22, which processes route instructions from the route control device 1 as steering control, drive control, brake control, and automatic driving control of the vehicle 2. The vehicle 2 can be equipped with a GPS module having a GPS function (not shown), a car navigation system, and various sensors such as a camera and LiDAR.

The route control device 1 and the core network 3 are connected via a network NW such as a LAN or WAN. Further, the base stations BS1 to BSn constituting the radio access network and the core network 3 are connected via a network L such as a backhaul link.

The core network 3 includes nodes within the C-plane, such as an AMF (Access and Mobility Management Function) 30, a UDM (Unified Data Management) 31, and a UDR (Unified Data Repository). sitory) 32. The core network 3 also includes a UPF (User Plane Function) 33 as a node within the U-plane. The route control device 1 acquires the position information of the communication area A associated with the position registration signal from the vehicle 2 via the communication interface 32a of the UDR 32. Further, the route control device 1 sends instructions for the determined route to the vehicle 2 via the communication interface 33a of the UPF 33.

For example, as shown in FIG. 1, when the vehicle 2 crosses from the communication area A1 of the base station BS1 to the communication area A2 of the base station BS2, the communication terminal 20 registers the location with the UDM 31 via the base station BS2 and the AMF 30. Sends a location registration signal to make a request.

The AMF 30 transmits the received signal to the UDM 31, and the UDM 31 performs location registration using the terminal identification information of the communication terminal 20 included in the vehicle 2. Furthermore, the location registration signal and terminal identification information transmitted by the communication terminal 20 provided in the vehicle 2 are stored in the UDR 32 as identification information regarding the base station BS and communication area A located in the area, and a transmission time stamp (date and time) of the location registration signal. information shown).

The route control device 1 according to the present embodiment transmits, via the network NW, the identification information or communication area of each base station BS that has received the location registration signal, which is associated with the location registration signal stored in the UDR 32. Obtain A's identification information. Further, the route control device 1 stores position information such as GPS coordinates of longitude and latitude of the base station BS and the communication area A arranged in the moving space in the setting information storage unit 14, which will be described later, as setting information. There is. In this embodiment, the position of the vehicle 2 is treated as the position of the base station BS in the communication area A where the vehicle 2 is located. In addition, in the following description, the position of the communication area A shall refer to the position of the corresponding base station BS.

In this way, the route control device 1 can acquire the position of the vehicle 2 at the time indicated by the time stamp of the position registration signal by acquiring the position registration signal from the vehicle 2 from the UDR 32.

[Functional block of route control device]
The course control device 1 includes a first acquisition section 10 , a second acquisition section 11 , a learning section 12 , a learning model storage section 13 , a setting information storage section 14 , a determination section 15 , and a course control section 16 . The route control device 1 determines the route the vehicle 2 should take next based on the position registration signal from the vehicle 2, and instructs the vehicle 2 on the route.

The first acquisition unit 10 acquires, as the current position of the vehicle 2, the position in the communication area A associated with the location registration signal from the vehicle 2, which is transmitted when the vehicle 2 crosses the communication area A. do. More specifically, the first acquisition unit 10 acquires a location registration signal and information associated with the location registration signal transmitted by the communication terminal 20 of the vehicle 2 from the communication interface 32a of the UDR 32 via the network NW. get. The information associated with the location registration signal includes the SIM 21 of the communication terminal 20, the identification information of the base station BS that received the location registration signal or the identification information of the communication area A, and the timestamp of the time when the location registration signal was transmitted. is included.

The first acquisition unit 10 acquires GPS coordinates of communication areas A1 to An associated with identification information of base stations BS1 to BSn or identification information of communication areas A1 to An stored in a setting information storage unit 14, which will be described later. By referring to the position information, it is possible to obtain the position of the vehicle 2 at the time when the position registration signal was transmitted.

The second acquisition unit 11 acquires the position of the communication area A corresponding to the area where traffic congestion occurs in the moving space of the vehicle 2. The second acquisition unit 11 can acquire traffic congestion information and position coordinates where traffic regulations are occurring in the moving space from an external traffic information server (not shown) via the network NW. The second acquisition unit 11 refers to the position information of the communication areas A1 to An arranged in the moving space, which is stored in the setting information storage unit 14, and the communication area corresponding to the coordinates of the area where the traffic jam is occurring. Obtain the positions of A1 to An.

For example, as shown in FIG. 2, the second acquisition unit 11 acquires the positions of communication areas A7 and A11 corresponding to area J where traffic congestion occurs. The communication areas A7 and A11 related to congestion and traffic regulations are set to be excluded from route selection when performing route control for the vehicle 2.

The second acquisition unit 11 can acquire the position of the communication area A related to congestion or traffic regulation at the time when the initial point and destination point of the vehicle 2 are set and route control is started. Alternatively, the second acquisition unit 11 can acquire the position of the communication area A related to congestion or traffic regulation at regular intervals.

The learning unit 12 uses the estimated result of calculating the course that the vehicle 2 should take sequentially from the position of each communication area A1 to An until it reaches the position of the communication area An of the destination point from the position of the communication area A1 of the initial point. The reward function is applied to update so that the reward for the vehicle 2 to reach the destination point in the communication area An is maximized, and the route that the vehicle 2 should take sequentially from each communication area A1 to An is determined. , learn using a learning model.

In the present embodiment, a case will be exemplified in which the vehicle 2 adopts three actions: right turn, left turn, and straight ahead, as the route the vehicle 2 should take sequentially from the position of each communication area A1 to An. However, depending on the area covered by the communication area A, the interval between the communication areas A arranged according to the shape of the road, etc., more detailed actions can be learned as a course strategy.

In this embodiment, the learning unit 12 uses a neural network model including an input layer, a hidden layer, and an output layer as shown in FIG. 3 as a learning model. In addition, as a neural network model, a neural network that receives the state s _t that is the position of the vehicle 2 and outputs all action values Q (s _t , go straight), Q (s _t , left turn), and Q (s _t , right turn) The network Deep Q-Network (DQN) is adopted.

More specifically, the learning unit 12 provides the current position of the communication area A indicating the current position of the vehicle 2 as an input to the neural network model, calculates the neural network model, and determines whether the vehicle 2 is in the current communication area. The first estimated value Q1 of the action value function that represents the expected value of the cumulative value of future rewards obtained when each action including turning right, turning left, and going straight is taken as the next course from position A. Output.

The reward is a state s indicating the current position of the vehicle 2, an action a in which the vehicle 2 turns right, turns left, or goes straight, and a reward function r=r(s , a, s'). In this embodiment, the reward function is the degree to which the vehicle 2 reaches the position in the communication area A corresponding to the destination point, and the degree to which the vehicle 2 reaches the position in the communication area A corresponding to the area where traffic congestion occurs. Contains as a variable. For example, when the vehicle 2 moves closer to the destination point or reaches the destination point by the shortest distance due to an action related to turning right or left or going straight, the reward, which is a scalar amount, is set to a larger value.

On the other hand, if the vehicle 2 moves away from the destination point or reaches a communication area A7 or A11 related to traffic congestion or traffic regulation as shown in FIG. 2, a negative reward value (for example, r = -1) is given. The design can be made as follows. In this way, by setting the reward for communication area A related to traffic congestion and traffic regulation as a negative value, the vehicle 2 can avoid these points and reach the destination point.

Furthermore, the learning unit 12 provides the position of the communication area A that the vehicle 2 has next arrived at as an input to the neural network model, performs calculations on the neural network model, and outputs the second estimated value Q2 of the action value function. The learning unit 12 learns the weight parameters of the neural network model so that the first estimated value Q1 becomes the target value calculated from the second estimated value Q2.

When the weight parameter of the neural network model is θ and the action value function is expressed as Q(s, a; θ), the learning minimization loss function is given by the following equation (1).
L (θ) = 1/2 {r + γmax _a Q (s', a'; θ) - Q (s, a; θ)} ²
...(1)

In the above formula (1), r is a reward (immediate reward), and γ indicates a discount rate. Q(s, a; θ) corresponds to the first estimated value Q1, and Q(s', a'; θ) corresponds to the action value in the state s', which is one step advanced, that is, the second estimated value Q2. do. The target value is expressed as r+γmax _a' Q(s', a'; θ).

The learning unit 12 can update the weight parameters of the neural network model by backpropagating the gradient of the loss function given by the above equation (1).

More specifically, the learning unit 12 can employ a Fixed Target Q-Network that uses two neural networks, a main QN 121 and a target QN 123, as shown in FIG. The main QN 121 selects the optimal action and updates the action value function Q. On the other hand, the target QN 123 estimates and evaluates the value of the action a' to be taken in the next state s' as a result of the action. The main QN 121 and the target QN 123 have neural networks with the same layer structure, but the parameter of the main QN 121 is "θ", and the parameter of the target QN 123 is given by "θ ^- ".

The main QN 121 receives the current position of the vehicle 2 from the environment 120 as the state s. The environment 120 is a moving space system in which the vehicle 2 is placed. Under this environment 120, the vehicle 2 moves to another communication area A by taking action a of turning right or left and going straight, and enters the next state s. 'At the same time, the reward r is obtained from the environment 120.

The learning unit 12 inputs the state s related to the current position of the vehicle 2 to the main QN 121 and calculates the action value function Q(s, a; θ). The learning unit 12 calculates the behavior a using, for example, the ε-greedy method, or determines the current optimal behavior argmax _a Q(s, a; θ) of turning left, right, and going straight. In the environment 120, the vehicle 2 performs the current optimal behavior of turning left, right, and going straight argmax _a Q(s, a; θ). The environment 120 observes the position of the communication area A to which the vehicle 2 has moved as a next state s' as a result of the action argmax _a Q(s, a; θ), and outputs the reward r. Experience data 124 stores experience (s, a, r, s') output from environment 120.

The learning unit 12 calculates a loss function L in a DQN loss calculation 122, and updates the weight of the main QN 121 with the gradient of the loss function L.

The learning unit 12 periodically copies the weight of the main QN 121 to the target QN 123 and performs synchronization. The synchronization of the target QN 123 is performed at a lower frequency than the weight update frequency of the main QN 121. The learning unit 12 extracts the experience from the experience data 124, inputs the past state to the target QN 123, and outputs the estimated value max _a' Q(s', a'; θ ⁻ ). The learning unit 12 calculates the DQN loss using the target value r+γmax _{a '} Q(s', a'; θ ^- ) based on the estimated value max a'Q(s',a'; θ ^- ) output by the target QN 123. In calculation 122, learning of the weight of the main QN 121 is performed.

Returning to FIG. 1, the learning model storage unit 13 stores the weights of the learned neural network model.

The setting information storage unit 14 stores identification information of the vehicle 2 and the communication terminal 20, position information of a movement space where the course of the vehicle 2 is controlled, and position information of each communication area A arranged in the movement space. . Furthermore, the setting information storage unit 14 stores positional information of the initial point and destination point of the vehicle 2 that has been acquired in advance. The setting information storage unit 14 can store the positions of the communication area A corresponding to the positions of the initial point and the destination point. In addition, the setting information storage unit 14 can also store map information of the moving space.

The determining unit 15 determines the current position of the communication area A acquired by the first acquisition unit 10 based on the route the vehicle 2 should take sequentially from the position of each communication area A, which is learned using the learning model. From this, the next route for the vehicle 2 is determined. The determining unit 15 reads out the learned weights stored in the learning model storage unit 13, provides the current position of the communication area A as an input to the learned neural network model, and calculates the learned neural network model. The robot determines the best course to take next, either by turning left, right, or going straight.

The route control unit 16 instructs the vehicle 2 via the core network 3 about the route that the vehicle 2 should take next, which has been determined by the determination unit 15 . Specifically, the route control unit 16 transmits a route instruction to the vehicle 2 via the UPF 33. The route control unit 16 instructs the vehicle 2 on the route until the vehicle 2 reaches the communication area of the destination point.

[Hardware configuration of route control device]
Next, an example of a hardware configuration that implements the route control device 1 having the above-described functions will be described using FIG. 5.

As shown in FIG. 5, the route control device 1 includes, for example, a computer connected via a bus 101, including a processor 102, a main storage device 103, a communication interface 104, an auxiliary storage device 105, and an input/output I/O 106; This can be realized by a program that controls these hardware resources.

The main storage device 103 stores in advance programs for the processor 102 to perform various controls and calculations. The processor 102 and the main storage device 103 realize each function of the route control device 1, such as the first acquisition unit 10, second acquisition unit 11, learning unit 12, determination unit 15, and route control unit 16 shown in FIG. Ru.

The communication interface 104 is an interface circuit for establishing a network connection between the route control device 1 and various external electronic devices.

The auxiliary storage device 105 includes a readable and writable storage medium and a drive device for reading and writing various information such as programs and data to and from the storage medium. For the auxiliary storage device 105, a semiconductor memory such as a hard disk or a flash memory can be used as a storage medium.

The auxiliary storage device 105 has a program storage area that stores a route control program executed by the route control device 1. Further, the auxiliary storage device 105 has an area for storing a learning program for learning a neural network model. The auxiliary storage device 105 realizes the learning model storage unit 13 and the setting information storage unit 14 described in FIG. Furthermore, for example, it may have a backup area for backing up the above-mentioned data, programs, and the like.

The input/output I/O 106 is an input/output device that inputs signals from external devices and outputs signals to external devices.

[Operation of course control device]
Next, the operation of the route control device 1 having the above-described configuration will be explained with reference to the flowcharts of FIGS. 6 to 8.

First, the learning process by the route control device 1 will be explained with reference to FIG. First, the route control device 1 acquires the positions of the initial point and destination point of the vehicle 2 (step S1). For example, the route control device 1 can acquire the destination point and the current position of the vehicle 2 input into the car navigation system of the vehicle 2.

Next, the second acquisition unit 11 acquires the position of the communication area corresponding to the area where traffic congestion occurs in the moving space (step S2). For example, the second acquisition unit 11 can acquire traffic congestion information or location information of an area where traffic restrictions are occurring from an external traffic information server, and can specify the location of the corresponding communication area. Alternatively, the second acquisition unit 11 may be configured to acquire the position of a communication area related to congestion or traffic regulation at regular intervals.

Next, the first acquisition unit 10 acquires the current position of the communication area A in which the vehicle 2 is located, as the current position of the vehicle 2 (step S3). Specifically, the first acquisition unit 10 obtains, from the UDR 32 of the core network 3, identification information of the communication area A or base station BS associated with the location registration signal transmitted when the vehicle 2 crosses the communication area; and get the timestamp. The first acquisition unit 10 acquires the communication area A stored in the setting information storage unit 14 or the position information of the communication area A associated with the identification information of the base station BS as the current position of the vehicle 2.

Next, the learning unit 12 inputs the current position of the communication area A in which the vehicle 2 is located, which is the current state of the vehicle 2 acquired in step S3, to the neural network model, and Calculate the expected value of the cumulative value of future rewards obtained when vehicle 2 takes each action including turning right, turning left, and going straight as the next course to proceed from the current communication area A. The first estimated value Q1 of the represented action value function is output (step S4).

Furthermore, the learning unit 12 provides the position of the communication area An that the vehicle 2 has reached next as an input to the neural network model, performs calculations on the neural network model, and outputs the second estimated value Q2 of the action value function (step S6). The learning unit 12 calculates a target value from the second estimated value Q2 (step S7). Subsequently, the learning unit 12 learns the weight parameters of the neural network model so that the first estimated value Q1 becomes the target value calculated from the second estimated value Q2 (step S8). Specifically, the learning unit 12 updates the weight parameters of the neural network model so as to minimize the loss function of equation (1) above.

The learning model storage unit 13 stores the learned weights obtained in step S8 (step S9).

Next, with reference to FIG. 7, learning processing by the learning unit 12 will be described when a Fixed Target Q-Network using two neural networks, the main QN 121 and the target QN 123, is adopted.

The processes from step S1 to step S3 are similar to the steps of the learning process explained with reference to FIG. After that, the learning unit 12 inputs the position of the communication area A in which the vehicle 2 is located, which was acquired in step S3, to the main QN 121, performs neural network calculations, and outputs the action value function Q. , calculates the next course a to follow (step S120).

Next, the learning unit 12 returns the behavior of the vehicle 2 along the route a obtained in step S120 to the environment 120, and determines the position of the communication area A to which the vehicle 2 has proceeded, which is the next state s' of the vehicle 2. Obtain a reward r (step S121). Note that the reward r given by the reward function can be configured to reflect the degree of arrival at the position of the communication area A where the traffic jam is occurring, which is obtained at regular intervals in step S2, at any time.

The learning unit 12 stores the experience (s, a, r, a') obtained in step S121 in the experience data 124 (step S122). Next, the learning unit 12 calculates the loss function L in the DQN loss calculation 122, and updates the weight of the main QN 121 with the slope of the loss function L (step S123). The learning unit 12 repeats the processing from step S120 to step S123 a set number of times.

After that, the learning unit 12 periodically copies the weight of the main QN 121 to the target QN 123 and performs synchronization (step S124). The synchronization of the target QN 123 is performed at a lower frequency than the weight update frequency of the main QN 121. Next, the learning unit 12 extracts the experience from the experience data 124, inputs the past state to the target QN 123, and outputs the estimated value max _a' Q(s', a'; θ ^- ) (step S126). .

Next, the learning unit 12 calculates the target value r+γmax _{a '} Q(s', a'; θ ^- ) based on the estimated value max a'Q(s',a'; θ ^- ) output by the target QN 123. (Step S127). Next, the learning unit 12 calculates the loss function L in the DQN loss calculation 122 using the target value calculated in step S127 (step S128). Next, the learning unit 12 learns the weights of the main QN 121 so as to minimize the loss given by the loss function L (step S129). Thereafter, the learned weights are stored in the learning model storage unit 13 (step S9).

Next, with reference to FIG. 8, route control processing by the route control device 1 will be described. First, the determining unit 15 loads the trained neural network model from the learning model storage unit 13 (step S40). In this embodiment, the determining unit 15 loads the learned DQN. Next, the first acquisition unit 10 acquires the current position of the vehicle 2, which is the position of the communication area A in which the vehicle 2 is located (step S41).

Next, the determining unit 15 determines whether the vehicle 2 should move from the current communication area position based on the learned neural network model loaded in step S40, that is, the route the vehicle 2 should take sequentially from the position of each communication area A. The next course to follow is determined (step S42). Specifically, the determining unit 15 inputs the current position of the communication area A obtained in step S41 to the trained neural network model, performs calculations on the trained neural network model, and determines whether the vehicle 2 Decide which route to take: go right, left, or go straight. The determining unit 15 selects the action with the highest probability value from among the action value functions Q for each action of turning right, turning left, and going straight, which are output from the learned neural network model, and determines the action as the course.

Thereafter, the route control unit 16 instructs the vehicle 2, via the UPF 33 of the core network 3, about the route of the vehicle 2 determined in step S42 (step S43). When the vehicle 2 receives the route instruction, the ECU 22 included in the vehicle 2 outputs a control command to each control actuator according to the route instruction, so that the vehicle 2 moves to the next communication area A.

Next, when the vehicle 2 reaches the destination point, the process ends (step S44: YES). On the other hand, if the vehicle 2 has not reached the destination point (step S44: NO), the processes from step S41 to step S43 are repeated. For example, a communication area in which the position of the associated communication area A is set as the destination point by a location registration signal transmitted when the vehicle 2 crosses the communication area A to which it has moved in accordance with route instructions. Based on whether or not the location matches the location of A, it is possible to determine whether or not the destination point has been reached.

As explained above, according to the route control device 1 according to the present embodiment, the location of the base station BS associated with the location registration signal sent when the vehicle 2 crosses the communication area A, that is, the location of the communication area A. The position is acquired as the current position of the vehicle 2, and the next course to be taken is determined based on the course strategy obtained by the learned neural network. Therefore, the course of the vehicle 2 can be controlled remotely with a simpler configuration.

Further, according to the route control device 1 according to the present embodiment, the position registration signal is used to determine the next route to be taken based on the route strategy acquired by the trained neural network. Even in situations where radio waves from satellites cannot be received, the vehicle's position can be estimated and the optimal course selected.

Further, according to the route control device 1 according to the present embodiment, since DQN is employed as a learning model, it is possible to select a route that avoids traffic jams and points related to traffic regulations.

In addition, in the above-mentioned embodiment, although the case where it is a route control system based on 5G was illustrated, the route control system based on LTE or 6G may be sufficient.

Although the embodiments of the route control device and route control method of the present invention have been described above, the present invention is not limited to the described embodiments, and those skilled in the art can imagine it within the scope of the invention described in the claims. Various possible modifications can be made.

DESCRIPTION OF SYMBOLS 1... Course control device, 10... First acquisition part, 11... Second acquisition part, 12... Learning part, 13... Learning model storage part, 14... Setting information storage part, 15... Determination part, 16... Course control part, 2... Vehicle, 20... Communication terminal, 21... SIM, 22... ECU, 3... Core network, 30... AMF, 31... UDM, 32... UDR, 33... UPF, 101... Bus, 102... Processor, 103... Main memory Device, 32a, 33a, 104... Communication interface, 105... Auxiliary storage device, 106... Input/output I/O, 120... Environment, 121... Main QN, 122... DQN loss calculation, 123... Target QN, 124... Experience data, BS1 to BSn...Base station, A1 to An...Communication area, L, NW...Network.

Claims (10)

A route control device that controls the route of the vehicle from a communication area position at an initial point to a communication area position at a destination point in a vehicle movement space defined by a plurality of communication areas,
a first acquisition unit configured to acquire, as the current position of the vehicle, a position in the communication area associated with a location registration signal transmitted from the vehicle when crossing the communication area;
of the current communication area, which is acquired as the current position of the vehicle by the first acquisition unit, based on a course that the vehicle should take sequentially from the position of each communication area, which is learned using a learning model. a determining unit configured to determine, from the position, the next course the vehicle should take;
A route control device comprising: a route control unit configured to instruct the vehicle on the next route determined by the determination unit, via a core network of a predetermined communication standard. The course control device according to claim 1,
Further, a reward function is applied to an estimation result of calculating a course that the vehicle should sequentially take from the position of each communication area until the vehicle reaches the position of the communication area of the destination point from the position of the communication area of the initial point. is applied so that the reward for the vehicle to reach the communication area position of the destination point is maximized, and the route policy for the vehicle to proceed sequentially from the communication area position is updated, a learning unit configured to learn using the learning model;
a storage unit configured to store the course strategy learned by the learning unit;
The route control device according to claim 1, wherein the determining unit reads out the route strategy from the storage unit and determines the route the vehicle should take next. The course control device according to claim 2,
Further, a second acquisition unit configured to acquire the position of a communication area corresponding to an area where traffic congestion occurs in the moving space;
The reward function includes, as variables, the degree of arrival of the vehicle to a position in a communication area related to the destination point, and the degree of arrival of the vehicle to a position in a communication area corresponding to the area where the traffic jam occurs. A course control device featuring: The course control device according to claim 3,
The learning model is a neural network model including an input layer, a hidden layer, and an output layer,
The learning unit provides the current communication area position as an input to the neural network model, performs calculations on the neural network model, and determines a right turn as the next course the vehicle should take from the current communication area position. outputting a first estimated value of an action value function representing an expected value of the future cumulative value of the reward obtained when each action including turning left and going straight is taken;
The learning unit further provides a position of a communication area that the vehicle next arrives at as an input to the neural network model, performs calculations on the neural network model, and outputs a second estimated value of the action value function,
The learning unit learns weight parameters of the neural network model so that the first estimated value becomes a target value calculated from the second estimated value,
The course control device, wherein the storage unit stores learned weight parameters. The course control device according to any one of claims 1 to 4,
The first acquisition unit acquires the current communication area position of the vehicle from an integrated data repository included in the core network,
The route control device, wherein the route control unit transmits an instruction regarding the next route to be taken to the vehicle via a user plane function included in the core network. A route control method for controlling the route of a vehicle from a communication area position at an initial point to a communication area position at a destination point in a vehicle movement space defined by a plurality of communication areas, the method comprising:
a first acquisition step of acquiring, as the current position of the vehicle, a position in the communication area associated with a location registration signal transmitted from the vehicle when crossing the communication area;
The current communication area acquired as the current position of the vehicle in the first acquisition step is based on the route the vehicle should take sequentially from the position of each communication area, which is learned using a learning model. determining the next course the vehicle should take from the location;
A route control method comprising: a route control step of instructing the vehicle, via a core network of a predetermined communication standard, on the route the vehicle should take next, determined in the determining step. In the course control method according to claim 6,
Further, a reward function is applied to an estimation result of calculating a course that the vehicle should sequentially take from the position of each communication area until the vehicle reaches the position of the communication area of the destination point from the position of the communication area of the initial point. is applied so that the reward for the vehicle to reach the communication area position of the destination point is maximized, and the route policy for the vehicle to proceed sequentially from the communication area position is updated, a learning step configured to learn using the learning model;
a storage step of storing the course strategy learned in the learning step in a storage unit;
The route control method, wherein the determining step reads the route plan from the storage unit and determines the route the vehicle should take next. In the course control method according to claim 7,
Further, a second acquisition step configured to acquire a position of a communication area corresponding to an area where traffic congestion occurs in the moving space;
The reward function includes, as variables, the degree of arrival of the vehicle to a position in a communication area related to the destination point, and the degree of arrival of the vehicle to a position in a communication area corresponding to the area where the traffic jam occurs. A path control method characterized by: In the course control method according to claim 8,
The learning model is a neural network model including an input layer, a hidden layer, and an output layer,
In the learning step, the position of the current communication area is given as an input to the neural network model, the neural network model is operated, and the next course the vehicle should take from the current communication area is to turn right or turn left. , and outputting a first estimated value of an action value function representing the expected value of the cumulative value of the future reward obtained when each action including walking straight is taken;
The learning step further includes providing the position of the next communication area that the vehicle has arrived at as an input to the neural network model, performing calculations on the neural network model, and outputting a second estimated value of the action value function;
The learning step learns weight parameters of the neural network model so that the first estimated value becomes a target value calculated from the second estimated value,
The course control method, wherein the storing step stores learned weight parameters in the storage unit. In the route control method according to any one of claims 6 to 9,
The first obtaining step obtains the current communication area position of the vehicle from an integrated data repository included in the core network;
The route control method is characterized in that, in the route control step, an instruction regarding the next route to be taken is transmitted to the vehicle via a user plane function included in the core network.