JP2023171455A - Route prediction device, in-vehicle device therewith, route prediction system, route prediction method, and computer program - Google Patents

Abstract

To provide a route prediction device that can predict a pedestrian's path of movement earlier before the pedestrian actually takes action, not only at a specific location such as a pedestrian crossing, etc., an in-vehicle device therewith, a route prediction system, a route prediction method, and a computer program.SOLUTION: A route prediction device includes: a collection unit that collects sensor data acquired by at least one of in-vehicle sensors and roadside sensors via wireless or wired communication; an analysis unit that detects a moving object from the sensor data collected by the collection unit and identifies the positions and signs of the moving object; a prediction unit that predicts a route including a future location of the moving object using the positions and signs of the moving object identified by the analysis unit; a storage unit that stores the positions and indications identified by the analyzing section, corresponding to each moving object; and a learning unit that trains the prediction unit using the positions and signs stored in the storage unit for a predetermined period of time.SELECTED DRAWING: Figure 6

Description

The present disclosure relates to a route prediction device, a vehicle-mounted device equipped with the same, a route prediction system, a route prediction method, and a computer program.

BACKGROUND ART A system has been proposed in which sensor information from fixedly installed sensors such as street surveillance cameras (hereinafter also referred to as infrastructure sensors) is uploaded to a server computer (hereinafter simply referred to as a server), and is analyzed and monitored. Furthermore, it has been proposed that various sensors be mounted on automobiles, motorcycles, etc. (hereinafter referred to as vehicles), and that information from these sensors be uploaded to a server, analyzed, and used for driving support. It has also been proposed that the results of analysis of data from in-vehicle sensors and the results of analysis of data from infrastructure sensors be combined and used for driving support.

Patent Document 1 listed below discloses that when there is a risk of a collision between a pedestrian or a bicycle attempting to cross a road and a vehicle, driving support information is appropriately distributed to the vehicle to prevent a collision. system is disclosed. This system uses sensors to detect pedestrians or bicycles, predict their behavior, and determine the possibility of a collision with a vehicle. It also uses sensors installed on the roadside to predict the route of pedestrians or vehicles in conjunction with traffic light information and road surface information. Pedestrian route prediction is performed using the positional relationship with crosswalks and a motion model of uniform linear motion.

Japanese Patent Application Publication No. 2002-360192

In Patent Document 1, when predicting a pedestrian's route, a plurality of possible routes are presented as candidates, but in driving support, it is preferable to have fewer predicted routes. In addition, since predictions are made only from a motion model based on the pedestrian's position, geographical conditions, speed, etc., it is possible to predict which route the actual pedestrian will take when starting to cross the crosswalk or just before starting to cross. There are some problems that cannot be understood until then. It would be desirable to be able to limit pedestrian routes sooner.

In Patent Document 1, it is assumed that pedestrians cross only on crosswalks, and there is a problem that other dangerous crossings that are difficult for drivers to handle cannot be predicted. Furthermore, since a moving object is detected only by a road condition grasping sensor provided on the roadside, there is a problem that prediction cannot be made outside of the detection by the roadside sensor.

Therefore, the present disclosure provides a route prediction device that can quickly predict a pedestrian's movement route, not only at a specific location such as a crosswalk, but also before the pedestrian actually moves, an in-vehicle device equipped with the same, and a route prediction device. The present invention aims to provide a system, a route prediction method, and a computer program.

A route prediction device according to an aspect of the present disclosure includes a collection unit that collects sensor data acquired by at least one of an in-vehicle sensor and a roadside sensor via wireless communication or wired communication; An analysis unit that detects a moving object from sensor data and identifies the position and signs of the moving object, and uses the position and signs of the moving object identified by the analysis unit to predict the path including the future position of the moving object. A prediction unit that stores the positions and signs identified by the analysis unit in association with each moving object, and a learning unit that uses the positions and signs stored for a predetermined period in the storage unit to learn the prediction unit. including the section.

A route prediction system according to another aspect of the present disclosure includes an on-vehicle device mounted on a vehicle including a first sensor, a roadside device including a second sensor installed on the road, and a route prediction device, the on-board device transmits the sensor data acquired by the first sensor to the prediction device via wireless communication, and the roadside device transmits the sensor data acquired by the second sensor to the prediction device via wireless communication or wired communication. The route prediction device includes a receiving unit that receives sensor data transmitted from the in-vehicle device and sensor data transmitted from the roadside device, and detects a moving object from the sensor data received by the receiving unit, and detects a moving object from the sensor data received by the receiving unit. an analysis unit that identifies the position and signs of the moving object; a prediction unit that predicts a path including the future position of the moving object using the position and signs of the moving object identified by the analysis unit; and a prediction unit that predicts the path including the future position of the moving object; and a storage unit that stores signs and signs in correspondence with each moving object, and a learning unit that causes the route prediction unit to learn using the positions and signs stored for a predetermined period in the storage unit.

A route prediction system according to yet another aspect of the present disclosure includes an in-vehicle device installed in a vehicle that includes a sensor, and a distribution device that delivers predetermined information to the in-vehicle device via wireless communication, the in-vehicle device including: a receiving unit that receives predetermined information; an analysis unit that detects a moving object from sensor data detected by the sensor and identifies the position and signs of the moving object; and the position and signs of the moving object identified by the analysis unit. and a prediction unit that predicts a route including the future position of the moving object using the predetermined information received by the reception unit, the predetermined information being acquired by at least one of an on-vehicle sensor and a roadside sensor. A predictor of the same model as the prediction unit is trained using a set of positions and signs obtained by repeating the process of detecting a moving object from sensor data and identifying the position and signs of the moving object for a predetermined period. These are the learning parameters obtained by

An in-vehicle device according to yet another aspect of the present disclosure includes a sensor, an analysis unit that detects a moving object from sensor data detected by the sensor, and identifies the position and sign of the moving object; a prediction unit that predicts a route including the future position of the moving object using the position and sign of the moving object and predetermined information received by the reception unit, and a learning parameter that the prediction unit uses to predict the route. , and a storage unit that stores data for each predetermined area including roads that vehicles can pass, and the learning parameters detect a moving object from sensor data acquired by at least one of an on-vehicle sensor and a roadside sensor, and These are learning parameters obtained by training a predictor of the same model as the prediction unit using a set of positions and signs obtained by repeating the process of specifying the position and signs of an object for a predetermined period of time.

A route prediction method according to yet another aspect of the present disclosure includes a collection step of collecting sensor data acquired by at least one of an in-vehicle sensor and a roadside sensor via wireless communication or wired communication; an analysis step of detecting a moving object from the sensor data and identifying the position and signs of the moving object; and a path including the future position of the moving object using the position and signs of the moving object identified in the analysis step. a prediction step that predicts the movement of the object; a storage step that stores the positions and signs identified in the analysis step in correspondence with each moving object; and a learning step that uses the positions and signs stored for a predetermined period in the storage step. and a learning step.

A computer program according to yet another aspect of the present disclosure provides a computer with a collection function that collects sensor data acquired by at least one of an in-vehicle sensor and a roadside sensor via wireless communication or wired communication; An analysis function that detects a moving object from sensor data collected by the sensor data and identifies the position and signs of the moving object, and uses the position and signs of the moving object identified by the analysis function to predict the future position of the moving object. A prediction function that predicts the included route, a memory function that stores positions and signs identified by the analysis function in correspondence with each moving object, and a prediction function that uses the positions and signs stored for a predetermined period by the storage function. A learning function for learning is realized.

Note that a part or all of each of the route prediction device, the in-vehicle device, and the route prediction system can be realized as a semiconductor integrated circuit. In addition, computer-readable recording media (USB memory, memory card (SD card, etc.), magnetic disk (HDD, etc.), magneto-optical disk, optical disk (DVD disk, Blue-ray disk, etc.), etc.) on which the above computer program is recorded, etc. ) can also be realized.

According to the present disclosure, it is possible to predict the movement route of a pedestrian more quickly, not only at a specific location such as a crosswalk, but before the pedestrian actually moves.

FIG. 1 is a schematic diagram showing the configuration of a route prediction system according to a first embodiment of the present disclosure. FIG. 2 is a plan view schematically showing a state in which the route prediction system of FIG. 1 is placed in an area including a crosswalk. FIG. 3 is a block diagram showing the configuration of the route prediction device of FIG. 1. FIG. 4 is a block diagram showing the configuration of an on-vehicle device mounted on the vehicle of FIG. 1. FIG. 5 is a block diagram showing the configuration of the roadside sensor of FIG. 1. FIG. 6 is a block diagram showing the functions of the route prediction device. FIG. 7 is a diagram showing feature amounts (signs) of a pedestrian who is a detection target. FIG. 8 is a flowchart showing processing (route prediction processing) by the route prediction device. FIG. 9 is a plan view showing an example of application of the route prediction system. FIG. 10 is a flowchart showing processing (learning processing) by the route prediction device. FIG. 11 is a schematic diagram showing learning data used for machine learning of the route prediction device. FIG. 12 is a schematic diagram showing the configuration of a route prediction system according to a second embodiment of the present disclosure. FIG. 13 is a block diagram showing the functions of the in-vehicle device according to the second embodiment. FIG. 14 is a schematic diagram showing an example of a prediction model and learning parameters. FIG. 15 is a flowchart showing the processing (route prediction processing) performed by the in-vehicle device of FIG. 13. FIG. 16 is a flowchart showing processing by the server (parameter distribution device) according to the second embodiment.

[Description of embodiments of the present disclosure]
First, the contents of the embodiments of the present disclosure will be listed and explained. At least some of the embodiments described below may be combined arbitrarily.

(1) The information providing device according to the first aspect of the present disclosure includes a collection unit that collects sensor data acquired by at least one of an in-vehicle sensor and a roadside sensor via wireless communication or wired communication; an analysis unit that detects a moving object from sensor data collected by the sensor data and identifies the position and signs of the moving object; and an analysis unit that uses the position and signs of the moving object identified by the analysis unit to determine the future position of the moving object A prediction unit that predicts a route including a path, a storage unit that stores positions and signs identified by the analysis unit in correspondence with each moving object, and a prediction unit that uses the positions and signs stored for a predetermined period in the storage unit. and a learning section for learning the sections. As a result, the moving route of a moving object such as a pedestrian can be predicted more quickly before the moving object actually moves.

(2) Preferably, the moving object is a person, and the sign includes information representing the direction of the person's face. Thereby, the accuracy of route prediction for people (pedestrians, etc.) can be improved.

(3) More preferably, the sign further includes information representing the orientation of the person's body. Thereby, the accuracy of route prediction for people (pedestrians, etc.) can be further improved.

(4) More preferably, the learning unit causes the prediction unit to learn by machine learning, and among the time-series data of the position and sign identified for each moving object over a predetermined time and stored in the storage unit, First time-series data of positions and signs that belong to the past are used as input data for machine learning, and time-series data of positions and signs that are identified for each moving object over a predetermined period of time and stored in a storage unit. Among them, time series data that does not include the first time series data is used as training data for machine learning. Thereby, data for machine learning can be automatically generated, and the prediction unit can be efficiently trained.

(5) Preferably, the route prediction device further includes a distribution unit that distributes the route predicted by the prediction unit to the in-vehicle device. This allows the in-vehicle device to use the travel route as driving support information.

(6) More preferably, the distribution unit distributes the route predicted by the prediction unit as well as image data of the moving object corresponding to the route. Thereby, the in-vehicle device can visually present to the driver an image of a moving object, such as a pedestrian, which is the target of the predicted route, so that the reliability of the received route can be confirmed.

(7) More preferably, the route prediction device uses the route predicted by the prediction unit to evaluate the possibility of a collision between the moving object to be predicted on the route and a moving object different from the moving object. The evaluation unit further includes an evaluation unit that performs evaluation. This makes it possible to present the possibility of a collision and prevent it.

(8) Preferably, the collection unit collects the sensor data acquired by the on-vehicle sensor and the sensor data acquired by the roadside sensor, and collects the first moving object detected from the sensor data acquired by the onboard sensor and the roadside The prediction unit further includes a determination unit that determines whether the second moving object detected from the sensor data acquired by the sensor is the same moving object, and the prediction unit determines whether the first moving object and The locations and signatures identified for each of the second moving objects are used together to predict a path. As a result, the moving route of a moving object such as a pedestrian can be predicted even earlier before the moving object actually moves.

(9) An in-vehicle device according to a second aspect of the present disclosure includes the route prediction device described above. Thereby, the in-vehicle device can predict the movement route of a moving object, such as a pedestrian, more quickly before the moving object actually moves.

(10) A route prediction system according to a third aspect of the present disclosure includes an on-vehicle device installed in a vehicle including a first sensor, a roadside device including a second sensor installed on the road, and a route prediction device. The in-vehicle device transmits the sensor data acquired by the first sensor to the route prediction device via wireless communication, and the roadside device transmits the sensor data acquired by the second sensor via wireless communication or wired communication. The route prediction device includes a receiving unit that receives sensor data transmitted from the in-vehicle device and sensor data transmitted from the roadside device, and detects a moving object from the sensor data received by the receiving unit. an analysis unit that detects and specifies the position and signs of the moving object; a prediction unit that predicts a path including the future position of the moving object using the position and signs of the moving object identified by the analysis unit; The prediction unit includes a storage unit that stores positions and signs identified by the moving object in association with each moving object, and a learning unit that causes the prediction unit to learn using the positions and signs stored for a predetermined period in the storage unit. As a result, the vehicle-mounted device alone can predict the movement route of a moving object, such as a pedestrian, more quickly before the moving object actually moves.

(11) A route prediction system according to a fourth aspect of the present disclosure includes an in-vehicle device installed in a vehicle that includes a sensor, and a distribution device that delivers predetermined information to the in-vehicle device via wireless communication. The device includes a reception unit that receives predetermined information, an analysis unit that detects a moving object from sensor data detected by the sensor, and identifies the position and signs of the moving object, and an analysis unit that detects the moving object identified by the analysis unit. a prediction unit that predicts a route including the future position of the moving object using the position and the sign and predetermined information received by the reception unit, the predetermined information being at least one of an on-vehicle sensor and a roadside sensor. Detect a moving object from the sensor data acquired by These are learning parameters obtained by learning. As a result, the movement route of a moving object such as a pedestrian can be predicted more accurately using appropriate predetermined information (parameters) before the moving object actually moves.

(12) Preferably, the route prediction system includes a plurality of distribution devices, and each of the plurality of distribution devices is arranged for each predetermined area including a road through which a vehicle can pass, and transmits predetermined information corresponding to the predetermined area wirelessly. The information is distributed via communication, and the receiving unit of the vehicle-mounted device receives predetermined information corresponding to the predetermined area from among the plurality of delivery devices that is disposed in the predetermined area including the current location of the vehicle. This can prevent the in-vehicle device from storing unnecessary data.

(13) More preferably, the in-vehicle device further includes a transmitter that transmits a transmission request requesting transmission of predetermined information corresponding to the predetermined region to a distribution device disposed in a predetermined region including the current location of the vehicle. The distribution device that includes the above information and receives the transmission request transmits predetermined information corresponding to the predetermined area in which the distribution device is placed to the in-vehicle device that transmitted the request. Thereby, transmission of unnecessary predetermined information to the in-vehicle device from the distribution device can be suppressed.

(14) More preferably, the receiving unit receives a version of the predetermined information corresponding to the predetermined area distributed from a distribution device disposed in the predetermined area including the current position of the vehicle, and the in-vehicle device receives the A storage unit stores learning parameters used by the unit to predict a route for each predetermined region, and a version acquired by the reception unit stores a predetermined learning parameter including the current position of the vehicle among the learning parameters stored in the storage unit. and a determination unit that determines whether the version of the learning parameters corresponding to the area is newer than the version of the learning parameters that correspond to the region, and the determination unit determines whether the version acquired by the receiving unit is newer than the version of the learning parameters stored in the storage unit, In response to determining that the learning parameter version is newer than the version of the learning parameter corresponding to the predetermined area including the current location, the transmitter transmits a transmission request. Thereby, transmission of unnecessary predetermined information to the in-vehicle device from the distribution device can be further suppressed.

(15) The in-vehicle device according to the fifth aspect of the present disclosure includes a sensor, an analysis unit that detects a moving object from sensor data detected by the sensor, and identifies the position and symptoms of the moving object; A prediction unit that predicts a route including the future position of the moving object from the location and signs of the moving object identified by and a storage unit that stores information for each area, and the learning parameters detect a moving object from sensor data acquired by at least one of an on-vehicle sensor and a roadside sensor, and perform processing to identify the position and signs of the moving object. These are learning parameters obtained by training a predictor of the same model as the prediction unit using a set of positions and signs repeatedly obtained for a predetermined period of time. As a result, the on-vehicle device alone can predict the travel path of a moving object, such as a pedestrian, more quickly and accurately using predetermined information (parameters) appropriate for the region, before the object actually moves. .

(16) The route prediction method according to the sixth aspect of the present disclosure includes a collection step of collecting sensor data acquired by at least one of an in-vehicle sensor and a roadside sensor via wireless communication or wired communication; An analysis step that detects a moving object from the sensor data collected in the step and identifies the position and signs of the moving object; and a future position of the moving object using the position and signs of the moving object identified in the analysis step. a prediction step for predicting a route including a path, a storage step for storing the positions and signs identified in the analysis step in correspondence with each moving object, and a prediction step using the positions and signs stored for a predetermined period in the storage step. and a learning step for learning the steps. As a result, the moving route of a moving object such as a pedestrian can be predicted more quickly before the moving object actually moves.

(17) A computer program according to a seventh aspect of the present disclosure provides a computer with a collection function for collecting sensor data acquired by at least one of an in-vehicle sensor and a roadside sensor via wireless communication or wired communication. , an analysis function that detects a moving object from the sensor data collected by the collection function and identifies the position and signs of the moving object, and a future prediction of the moving object using the position and signs of the moving object identified by the analysis function. A prediction function that predicts a route including the location of , and a learning function that learns a prediction function. As a result, the moving route of a moving object such as a pedestrian can be predicted more quickly before the moving object actually moves.

[Details of embodiments of the present disclosure]
In the following embodiments, identical parts are given the same reference numbers. Their names and functions are also the same. Therefore, detailed description thereof will not be repeated.

(First embodiment)
[overall structure]
Referring to FIG. 1, a route prediction system 100 according to a first embodiment of the present disclosure includes a route prediction device 102, a roadside sensor 104 fixedly installed on a road and its surroundings (hereinafter also referred to as a road), It includes a vehicle 106, a mobile communication base station 108, a network 110, and a road traffic signal (pedestrian signal) 112. The pedestrian 200 is a detection target of the roadside sensor 104 and the sensor mounted on the vehicle 106. It is assumed that communication between the elements constituting the route prediction system 100 is performed via the base station 108 or by direct wireless communication or wired communication not via the base station 108. The base station 108 provides mobile communication services using LTE (Long Term Evolution) lines, 5G (5th generation mobile communication system) lines, and the like.

The on-vehicle device mounted on the vehicle 106 has a communication function using an LTE line, a 5G line, and the like. The roadside sensor 104 is a device equipped with an imaging function installed on the road and its surroundings, and has a communication function with the base station 108. The roadside sensor 104 is, for example, a digital surveillance camera.

Referring to FIG. 2, route prediction system 100 is placed in an area including an intersection, for example. The route prediction device 102 receives and analyzes sensor information (video image data, etc., hereinafter also referred to as sensor data) transmitted from the roadside sensor 104 and the in-vehicle device of the vehicle 106, and detects pedestrians 200 and 202. The movement routes 210 and 212 of each detected pedestrian are predicted. The predicted travel route (hereinafter referred to as a predicted route) is distributed (for example, broadcast) to the on-vehicle device. Vehicles 114, 116, and 118 traveling on surrounding roads can use the received predicted route as driving support information.

Although FIGS. 1 and 2 typically show one base station 108 and one infrastructure sensor (roadside sensor 104), a plurality of base stations and a plurality of infrastructure sensors are normally provided. The route prediction device 102 communicates with more infrastructure sensors and on-vehicle devices of vehicles, collects and analyzes sensor information, and uses it to predict a travel route.

[Hardware configuration of route prediction device]
Referring to FIG. 3, the route prediction device 102 includes a control unit 120 for controlling each unit, a memory 122 for storing data, a communication unit 124 for communicating, and a bus 126 for exchanging data between each unit. including. The control unit 120 includes a CPU (Central Processing Unit), and realizes functions described below by controlling each unit. The memory 122 includes a large capacity storage device such as a rewritable semiconductor nonvolatile memory and a hard disk drive (hereinafter referred to as HDD). The communication unit 124 receives sensor information (image data) uploaded from the roadside sensor 104 placed on the road and sensor information uploaded from an in-vehicle device such as the vehicle 106 . The data received by the communication unit 124 is transmitted to and stored in the memory 122. Thereby, the route prediction device 102 realizes a route prediction function as described later.

[Hardware configuration and functions of in-vehicle device]
With reference to FIG. 4, an example of the hardware configuration of the on-vehicle device 140 mounted on the vehicle 106 is shown. The in-vehicle device 140 includes an interface section (hereinafter referred to as I/F section) 144 connected to a sensor section 142 mounted on the vehicle 106, a communication section 146 that performs wireless communication, a memory 148 that stores data, and controls them. It includes a control section 150 for presenting information, a presentation section 152 for presenting information, and a bus 154 for exchanging data between the various sections.

The sensor unit 142 is a known video image capturing device (for example, a digital camera (CCD camera, CMOS camera)) mounted on the vehicle 106. The sensor section 142 outputs a predetermined video signal (analog signal or digital data). A signal from the sensor section 142 is input to the I/F section 144. The I/F section 144 includes an A/D conversion section, and when an analog signal is input, it samples it at a predetermined frequency, generates and outputs digital data (sensor information). The generated digital data is transmitted to and stored in memory 148. If the output signal from the sensor section 142 is digital data, the I/F section 144 stores the input digital data in the memory 148. The memory 148 is, for example, a rewritable nonvolatile semiconductor memory or an HDD.

The communication unit 146 has a mobile communication function such as an LTE line and a 5G line, and communicates with the route prediction device 102. Communication between the in-vehicle device 140 and the route prediction device 102 is performed via the base station 108 or by direct wireless communication not via the base station 108. The communication unit 146 includes an IC for modulating and multiplexing that is used in LTE lines, 5G lines, etc., an antenna for emitting and receiving radio waves of a predetermined frequency, an RF circuit, etc. . The communication unit 146 is equipped with multiple mobile communication functions such as an LTE line and a 5G line, so that when the vehicle 106 is traveling in an area where 5G line communication service is not provided, the LTE line communication becomes possible.

The presentation unit 152 includes, for example, an image display device such as a liquid crystal display, and an audio device such as a speaker, and displays driving support information as an image and outputs it as sound. The control unit 150 includes a CPU, and realizes the functions of the in-vehicle device 140 by controlling each unit. For example, the control unit 150 receives a predicted route of a moving object (such as a pedestrian) from the route prediction device 102, and performs processing such as controlling the travel of the vehicle 106 and providing information to support the driver. . Furthermore, the control unit 150 obtains the current position of the vehicle 106 using GPS.

[Hardware configuration and functions of infrastructure sensor]
The roadside sensor 104, which is an infrastructure sensor, is basically configured similarly to the in-vehicle device 140. With reference to FIG. 5, an example of the hardware configuration of the roadside sensor 104 is shown. The roadside sensor 104 includes an I/F unit 162 connected to the sensor unit 160, a communication unit 164 that performs wireless or wired communication, a memory 166 that stores data, a control unit 168 that controls them, and It includes a bus 170 for exchanging data.

The sensor unit 160 is a known video image capturing device (for example, a digital camera). The sensor unit 160 outputs a predetermined video signal. A signal from the sensor section 160 is input to the I/F section 162. The I/F section 162 includes an A/D conversion section, and when an analog signal is input, it generates and outputs digital data (sensor information). The generated digital data is transmitted to and stored in memory 166. If the output signal from the sensor unit 160 is digital data, the I/F unit 162 stores the input digital data in the memory 166. The memory 166 is, for example, a rewritable nonvolatile semiconductor memory or an HDD.

The communication unit 164 has a mobile communication function and communicates with the route prediction device 102 via the base station 108 or by direct wireless communication or wired communication not via the base station 108. Since the roadside sensor 104 is fixedly installed, it does not need to be compatible with multiple mobile communication systems, but is compatible with the mobile communication system (for example, LTE line or 5G line) provided by a nearby base station 108. All you have to do is do it. The communication unit 164 includes an IC for performing modulation and multiplexing, an antenna for emitting and receiving radio waves of a predetermined frequency, an RF circuit, and the like. Note that the communication function of the fixedly installed roadside sensor 104 is not limited to the communication function via the base station 108, but is arbitrary. The communication function may be a wired LAN or a wireless LAN such as WiFi. In the case of WiFi communication, a device (such as a wireless router) that provides a WiFi service is provided separately from the mobile communication base station 108, and the roadside sensor 104 communicates with the route prediction device 102 via the network 110.

The control unit 168 includes a CPU, and realizes the functions of the in-vehicle device 140 by controlling each unit. That is, the control unit 168 reads out the moving image data captured by the sensor unit 160 and stored in the memory 166 at predetermined time intervals, generates packet data, and transmits it from the communication unit 164 to the route prediction device 102.

[Functional configuration of route prediction device]
The functions of the route prediction device 102 will be described with reference to FIG. 6. The route prediction device 102 includes a packet reception unit 180 that receives packet data, a packet transmission unit 182 that transmits packet data, an analysis processing unit 184 that performs predetermined analysis processing on the received sensor information, and a route prediction and a prediction unit 188 that executes a process of predicting a future movement path (predicted path) of a moving object. The packet receiving unit 180 receives packet data including sensor data from the roadside sensor 104 and the in-vehicle device 140. The received sensor data is input to the analysis processing section 184.

The analysis processing unit 184 performs analysis processing on input sensor data (moving image data, etc.) to detect a moving object. Here, the detection target is a moving pedestrian, and "pedestrian" is not limited to a walking person, but means a person moving at any speed, Also includes. If multiple pedestrians are included in the uploaded video data, each pedestrian is detected. The analysis process can use known object detection processing, including processing for determining differences between frames, and feature extraction processing. For example, feature quantities (detection time, position, , moving speed, moving direction, size, shape, color, etc.). Thereby, it is possible to distinguish between pedestrians and other moving objects, and it is also possible to determine whether the pedestrians are the same pedestrian or different pedestrians.

Furthermore, the analysis processing unit 184 detects the position and signs as feature amounts for each detected pedestrian. "Sign" means information for predicting the future position of the detected pedestrian. Here, the body direction and face direction of the detected pedestrian are detected as signs. Any method can be used to detect the orientation of the body and the orientation of the face. For example, a detection method using HOG (Histograms of Oriented Gradients) features or Haar-Like features can be used. Furthermore, a detection method that combines one of the HOG feature amount and the Haar-Like feature amount with AdaBoost (Adaptive Boosting) may be used. Alternatively, a detection method using a convolutional neural network may be used. As an example, FIG. 7 shows part of one frame of image data. In FIG. 7, target frames 220 and 230 representing detected pedestrians are displayed. For example, regarding the target frame 220, a face area (indicated by a face area frame 222) is specified, and the face direction 224 is calculated by analyzing the face area. Further, regarding the target frame 220, a body region (indicated by a body region frame 226) is specified, and the body direction 228 is calculated by analyzing the body region. Each of the face direction 224 and the body direction 228 is calculated as a two-dimensional vector in a horizontal plane, for example. The analysis processing section 184 outputs the analysis results (position, body direction, and face direction of each pedestrian) to the learning section 186 and the prediction section 188.

When the learning unit 186 receives the analysis results (position, body orientation, and face orientation of each pedestrian) from the analysis processing unit 184, the learning unit 186 stores the analysis results for a predetermined period of time. The learning unit 186 stores time series data of position, time series data of body orientation, and time series data of face orientation as one set for each detection target. The learning unit 186 then performs learning for route prediction using multiple sets (multiple detection targets) of time-series data. The learning target is the same model as the prediction unit 188 described later. The learning algorithm is arbitrary. For example, depending on the model used, neural networks, Markov decision processes, Gaussian processes, etc. can be used. The learning unit 186 outputs learning results (for example, learning parameters (hereinafter also simply referred to as parameters)) to the prediction unit 188.

The prediction unit 188 applies the learning results input from the learning unit 186 to a predetermined prediction model, inputs the analysis results input from the analysis processing unit 184 to the prediction model (prediction model after learning), and predicts future predictions. Derive a travel route (predicted route). The prediction unit 188 outputs the derived predicted route to the packet transmission unit 182.

The packet transmitting unit 182 generates packet data including the predicted route input from the predicting unit 188 and distributes (eg, broadcasts) the packet data to the in-vehicle device. The in-vehicle device that receives the predicted route can use the predicted route as driving support information.

The functions of the packet receiving section 180 are realized by, for example, the communication section 124 and the memory 122 in FIG. 3. The functions of the analysis processing unit 184, learning unit 186, and prediction unit 188 are realized by, for example, the control unit 120 and memory 122 in FIG. The function of the packet transmitter 182 is realized by, for example, the communication unit 124 and memory 122 in FIG. 3. The route prediction device 102 may implement the functions of the analysis processing section 184, the learning section 186, and the prediction section 188 using dedicated hardware (circuit board, ASIC, etc.).

[Operation of route prediction device]
(route prediction processing)
With reference to FIG. 8, the route prediction process by the route prediction device 102 will be described in more detail. The processing shown in FIG. 8 is realized by the control unit 120 reading a predetermined program from the memory 122 and executing it.

In step 300, control unit 120 determines whether sensor data has been received. If it is determined that it has been received, control moves to step 302. Otherwise, step 300 is repeated.

In step 302, the control unit 120 analyzes the sensor data received in step 300, detects a pedestrian, and detects its position and signs (body orientation and face orientation). Control then moves to step 304. The process in step 302 corresponds to the function performed by the analysis processing unit 184 in FIG.

In step 304, the control unit 120 determines whether to perform route prediction. If it is determined to execute, control moves to step 306. Otherwise, control transfers to step 310. For example, as a result of the analysis in step 302, if a pedestrian is detected and its position and signs (body orientation and face orientation) can be detected, the determination result will be YES. As a result of the analysis in step 302, if no pedestrian is detected, the determination result is NO. As a result of the analysis in step 302, even if a pedestrian is detected, if its position and signs cannot be detected, the determination result is NO.

In step 306, the control unit 120 uses the position and signs detected in step 302 to predict the movement route of each detected pedestrian. Control then transfers to step 308. The process in step 306 corresponds to the function of the prediction unit 188 in FIG.

In step 308, the control unit 120 broadcasts the predicted route generated in step 306. Control then transfers to step 310.

In step 310, control unit 120 determines whether an instruction to end has been received. If it is determined that a termination instruction has been received, this program terminates. Otherwise, control returns to step 300. The instruction to terminate is given by, for example, operating the route prediction device 102 by an administrator or the like.

Thereby, the predicted route transmitted from the route prediction device 102 is received by the in-vehicle device. The in-vehicle device that receives the predicted route can use the predicted route as driving support information. For example, the in-vehicle device can determine whether or not the traveling direction of the own vehicle intersects with the predicted route, and if they intersect, the on-vehicle device can call the driver's attention by presenting a warning or the like.

By using the detected position and signs of a moving object such as a pedestrian, the moving path of the moving object can be predicted more quickly before the moving object actually moves.

Since the direction of the pedestrian's body does not change until the time when the pedestrian changes the direction of travel, the accuracy of a route predicted based only on the direction of the pedestrian's body is not very high. On the other hand, when outdoor pedestrians change their direction of travel, they usually turn their heads toward their surroundings, including the new direction of travel, to avoid collisions with other moving objects (vehicles, pedestrians, etc.). Check. Therefore, by using the direction of the face as a sign of changing course, the predicted route can be determined with high accuracy. In particular, by using the body orientation and face orientation, the predicted route can be determined with higher accuracy.

For example, with reference to FIG. 2, the traffic light 112 (pedestrian traffic light) and vehicle traffic light 206 for movement in the left and right direction is a green light, and the pedestrian traffic light 204 and vehicle traffic light 208 for movement in the up and down direction are red lights. shall be. In FIG. 2, the current direction of travel of a person or vehicle is shown by a solid line arrow, and the predicted future travel route, ie, the predicted route, is shown by a broken line. The detection areas of the roadside sensor 104 and the on-vehicle sensor of the vehicle 106 are shown in fan shapes with one-dot chain lines. The sensor data transmitted from the roadside sensor 104 to the route prediction device 102 includes an image of the pedestrian 200, and the route prediction device 102 detects the pedestrian 200 from the sensor data received from the roadside sensor 104. A predicted route 210 can be determined by detecting its location and signs (body orientation and facial orientation) and using them to perform route prediction. The route prediction device 102 distributes information specifying the determined predicted route 210. As a result, the in-vehicle device of the vehicle 116 that is about to turn right and has received the information specifying the predicted route 210 displays a warning to the driver that the predicted route 210 overlaps the crosswalk that the vehicle 116 is about to enter. It can be performed.

Further, the sensor data transmitted from the on-board device of the vehicle 106 includes an image of the pedestrian 202, and the route prediction device 102 detects the pedestrian 200 from the sensor data received from the on-board device of the vehicle 106. The predicted route 212 can be determined by detecting the position and signs (body orientation and facial orientation) of the route 212, and using them to perform route prediction. The route prediction device 102 distributes information specifying the determined predicted route 212. Thereby, the in-vehicle devices of vehicles 106 and 118 that have received the information specifying the predicted route 212 can detect that the pedestrian 202 is about to enter the roadway, and can make a presentation that calls for the driver's attention. Thereby, the on-vehicle device of the vehicle 118 can detect the presence of the pedestrian 202 who is about to cross in front of the vehicle 118 before the driver of the vehicle 118 visually sees it, and can alert the pedestrian 202 to the pedestrian 202 who is about to cross the street. Furthermore, the on-vehicle device of the vehicle 106 can also alert the driver in advance, so that even if the pedestrian 202 is within the field of view of the driver of the vehicle 106, the driver can warn the driver that the pedestrian 202 is crossing the road. This is effective when you cannot recognize what you are trying to do.

It is possible to predict the route of a pedestrian near an intersection using only the sensor data of the roadside sensor 104, but if the roadside sensor and the pedestrian are far apart, depending on the performance of the roadside sensor 104, the pedestrian's path may be predicted. Difficult to detect signs. For example, if the image resolution is not sufficient, the direction of a pedestrian's face cannot be determined. Therefore, as described above, it is effective to use not only the sensor data of the roadside sensor 104 but also the sensor data of the in-vehicle device. It is effective to use the sensor data of the in-vehicle device not only in the case described above with reference to FIG. 2 but also in the case shown in FIG. 9, for example. Referring to FIG. 9, sensor data transmitted from the on-board device of vehicle 106 to route prediction device 102 includes an image of pedestrian 202, and route prediction device 102 receives sensor data from the on-board device of vehicle 106. The predicted route 212 can be determined by detecting the pedestrian 202 from the sensor data, detecting its position and signs (body orientation and face orientation), and performing route prediction using these. The route prediction device 102 distributes information specifying the determined predicted route 212. As a result, when the vehicle 114 is about to turn left, the on-vehicle device of the vehicle 114 that has received the information specifying the predicted route 212 detects that the traveling direction of the vehicle 114 after the left turn intersects with the predicted route of the pedestrian 202. It is possible to present a warning to the driver. Even after the vehicle 114 makes a left turn, the pedestrian 202 located in the blind spot due to an obstacle 240 such as a tree is not visible to the driver of the vehicle 114, so it is effective to be able to alert the driver of the vehicle 114 in advance.

Note that not only sensor data transmitted from an on-vehicle device of a running vehicle but also sensor data transmitted from an on-vehicle sensor of a stopped vehicle may be used for route prediction. For example, a pedestrian attempts to cross a road by entering between multiple vehicles parked in parallel. In such a case, it is preferable to detect pedestrians and predict their travel routes using sensor data transmitted from an on-vehicle device of a parked vehicle.

(Learning process)
With reference to FIG. 10, learning processing related to route prediction by the route prediction device 102 will be described in more detail. The processing shown in FIG. 10 is realized by the control unit 120 reading a predetermined program from the memory 122 and executing it.

In step 400, control unit 120 determines whether sensor data has been received. If it is determined that it has been received, control moves to step 402. Otherwise, step 400 is repeated.

In step 402, the control unit 120 analyzes the sensor data received in step 400, detects a pedestrian, and detects its position and signs (body orientation and face orientation). Control then transfers to step 404. The process in step 402 corresponds to the function performed by the analysis processing unit 184 in FIG.

In step 404, the control unit 120 stores the position and sign detected in step 402 for each detected pedestrian. Control then transfers to step 406.

In step 406, the control unit 120 determines whether learning is to be performed. If it is determined to execute, control moves to step 408. Otherwise, control transfers to step 412. For example, when the time series data of positions and signs stored in step 404 exceeds a predetermined amount, it is determined that learning should be performed. Alternatively, it may be determined that learning is to be performed every time a predetermined time elapses or at a predetermined time.

In step 408, the control unit 120 uses the stored time series data of positions and signs to perform learning of a prediction model used for route prediction. Control then transfers to step 410. For example, learning input data and teacher data are automatically generated from stored time-series data of locations and signs, and are used to learn a predictive model. Specifically, referring to FIG. 11, a route (two-dimensional route) from a starting point A to an ending point C via an intermediate point B is stored as time-series data of the detected position of a certain pedestrian. If so, the data from the starting point A to the intermediate point B (time-series position data) is used as input data. Furthermore, data from intermediate point B to end point C is assumed to be teacher data. Similarly, time-series data of signs (body orientation, face orientation) is divided into input data and teacher data. In this way, for example, one piece of input data and the corresponding teacher data are generated from time-series data of the position and signs regarding a certain pedestrian. Similarly, input data and teacher data are generated from time-series data of positions and signs regarding other pedestrians. The parameters of the prediction model are determined by performing learning using the plurality of input data and teacher data thus generated. The process in step 408 corresponds to the function of the learning unit 186 in FIG.

In step 410, the control unit 120 stores the learning results (parameters) in step 408. Control then transfers to step 412.

In step 412, control unit 120 determines whether an instruction to end has been received. If it is determined that a termination instruction has been received, this program terminates. Otherwise, control returns to step 400. The instruction to terminate is given by, for example, operating the route prediction device 102 by an administrator or the like.

Thereby, the learning results (parameters) stored in step 410 are used in step 306 of the route prediction process described above. As described above, learning can be performed efficiently by automatically generating learning input data and teacher data.

In the above, a case has been described in which the route prediction device 102 and the roadside sensor 104 are arranged to observe an intersection with a crosswalk, but the present invention is not limited to this. The route prediction device 102 and the roadside sensor 104 can be placed anywhere around a road on which a vehicle travels to detect pedestrians and predict their routes. For example, by installing the route prediction device 102 and the roadside sensor 104 at locations where pedestrians tend to cross, even if they are not on crosswalks, a large amount of route data can be acquired, improving the accuracy of route prediction. can.

Furthermore, when the roadside sensor 104 is not arranged and only the route prediction device 102 is arranged, it is possible to detect pedestrians and predict their routes by using sensor data from sensors mounted on the vehicle. can.

Although a case has been described above in which route prediction is performed using only sensor data, the present invention is not limited to this. Traffic light information may be used for route prediction. For example, if a pedestrian traffic light is red, it is generally considered that the possibility of a travel route that involves crossing the pedestrian crossing is low. Therefore, by using information on traffic lights (including pedestrian traffic lights and vehicle traffic lights) for route prediction, the accuracy of route prediction can be improved.

Sensor data is not limited to data captured by a camera. The sensor may be any detection device that detects pedestrians through analysis and outputs data that allows calculation of their characteristic quantities (position and signs), a device that repeatedly captures still images at short intervals, or RF (millimeter wave, etc.) Alternatively, a sensor using a laser may be used.

In the above, pedestrians were the detection target, but the detection target is not limited to this. The detection target can be a moving object that could cause damage if the vehicle collides with it, or even a person riding a bicycle.

Since the vehicle travels, the range in which the on-vehicle sensor can acquire sensor data is different from the range in which the roadside sensor can acquire sensor data. The range in which in-vehicle sensors of different vehicles can acquire sensor data also differs. On the other hand, the same pedestrian may be included in sensor data acquired by a roadside sensor and a vehicle-mounted sensor. Therefore, in order to detect pedestrians over a wider range and determine their predicted routes, sensor data from roadside sensors and in-vehicle sensors, as well as sensor data from multiple in-vehicle sensors, must be combined and used. It is preferable. As a result, more information can be obtained regarding the same pedestrian detected over a wider range, and the position and signs of the pedestrian can be detected with higher accuracy. Therefore, the predicted route can be determined with higher accuracy.

In the above, a case has been described in which the route prediction device 102 distributes only the predicted route, but the present invention is not limited to this. In addition to the predicted route, auxiliary data may be distributed. For example, image data of a detected moving object corresponding to the predicted route may be transmitted. Thereby, the in-vehicle device can visually present to the driver an image of a moving object, such as a pedestrian, which is the target of the predicted route, so that the reliability of the received route can be confirmed.

In the above, a case has been described in which the detected position, body direction, and face direction of the pedestrian are used for route prediction, but the present invention is not limited thereto. In addition to these, for example, the detected moving speed of the pedestrian may be used. Furthermore, in place of the body orientation and face orientation, or in addition to at least one of the body orientation and face orientation, another physical state of the pedestrian that can be observed before the pedestrian changes the direction of travel, May be used as a sign.

In the above, a case has been described in which the fixedly installed route prediction device 102 executes route prediction using sensor data received from roadside sensors and in-vehicle devices, but the present invention is not limited to this. The route prediction device may be mounted on the vehicle. For example, the in-vehicle device may perform learning and route prediction for route prediction.

(Second embodiment)
The same prediction device (prediction model) after learning for route prediction may be used for roads across the country, but the prediction accuracy depends on past data used for learning. In other words, the sensor data used for learning includes the geographical conditions (road shape, road width, number of lanes, presence or absence of sidewalks, width of sidewalks, presence or absence of traffic lights, This is thought to reflect the location of traffic lights, traffic volume, etc. Preferably, a route prediction device that predicts a route in a specific area is trained using sensor data acquired in that area. Therefore, in the second embodiment, the in-vehicle device has a route prediction function, and uses the learning results determined by area-by-area learning to predict the route in the area where the vehicle is traveling.

Referring to FIG. 12, the route prediction system according to the second embodiment includes a server 500 located in an area 510, a server 502 located in an area 512 different from the area 510, and each of a plurality of vehicles 504 and 508. In-vehicle equipment. Servers 500 and 502 are computers and are configured similarly to FIG. 3. The memory of the server 500 stores learning parameters determined by learning a prediction model as described above using sensor data from roadside sensors (not shown) placed in the area 510. Similarly, the memory of the server 502 stores learning parameters determined using sensor data from roadside sensors (not shown) located in the area 512. Each of the servers 500 and 502 appropriately distributes learning parameters stored in its own memory and functions as a parameter distribution device. Although FIG. 12 shows two areas 510 and 512, a predetermined area including roads (for example, the whole country, each prefecture, etc.) is divided into multiple areas, and each area has at least one server ( It is assumed that a system (that stores and distributes learning parameters) is installed.

(Functional configuration of in-vehicle device)
The on-vehicle devices of vehicles 504 and 506 are configured in the same manner as in FIG. 4 . FIG. 13 shows the functional configuration of the on-vehicle devices of vehicles 504 and 506. Referring to FIG. 13, in-vehicle device 520 includes a packet receiving section 522, a parameter storage section 524, a sensor section 526, an analysis processing section 528, a prediction section 530, and an information presentation section 532.

The packet receiving unit 522 receives packet data (learning parameters transmitted from the servers 500 and 502), and outputs the received learning parameters to the parameter storage unit 524. The parameter storage unit 524 stores learning parameters corresponding to each area in advance. The parameter storage unit 524 appropriately stores learning parameters input from the packet receiving unit 522.

The sensor unit 526 is configured in the same manner as the sensor unit 142 and I/F unit 144 in FIG. 4, for example, and acquires sensor data. The acquired sensor data is output to the analysis processing section 528. The analysis processing unit 528 analyzes the input sensor data in the same manner as the analysis processing unit 184 in FIG. 6, detects a pedestrian, and determines its position and signs (body orientation and face orientation of the pedestrian). To detect. The analysis processing unit 528 outputs the detected position and symptom to the prediction unit 530.

When the position and signs are input from the analysis processing unit 528, the prediction unit 530 calculates the area in which the vehicle is currently traveling (the area where the vehicle is currently located) from among the learning parameters stored in the parameter storage unit 524. The learning parameters are read and route prediction is performed in the same manner as the prediction unit 188 in FIG. 6 . The prediction unit 530 outputs the obtained route prediction to the information presentation unit 532.

An example of learning parameters will be described with reference to FIG. 14. FIG. 14 shows a predictor (prediction model) using a neural network. Measured data is input to each node of the input layer. When data is input to a node in a certain layer, that node outputs a value obtained by multiplying the input value by a predetermined weight to each node in the next layer. By repeating this process between adjacent layers, data is output from the output layer. If you change the weights set for the nodes of each layer, the output data of the output layer will change. In learning, each weight is adjusted using a large amount of input data so that the output data for each input data approaches the teacher data. In the case of such neural network models, the weights are the learning parameters.

In advance, such a neural network model is trained using input data and teacher data generated from sensor data acquired in each area, and the resulting weights are stored as learning parameters in a server located in that area. I'll keep it. For example, as shown in FIG. 11, the input data and teacher data can be automatically generated from time-series data of the pedestrian's position and signs (body orientation and face orientation) detected from sensor data.

Although FIG. 14 shows a case where there are two intermediate layers, there may be one, three or more intermediate layers. Furthermore, predictive models are not limited to neural networks. Any prediction model may be used as long as the learning parameters are determined by learning using the pedestrian's position and signs (body orientation and face orientation) detected from sensor data. The prediction unit 530 of the in-vehicle device 520 may employ a prediction model having the same configuration as the prediction model that determined the learning parameters.

Returning to FIG. 13, the information presentation unit 532 corresponds to the bus 154 in FIG. , a series of two-dimensional coordinates) is displayed as an image. It also determines whether the predicted route intersects with the direction of travel of the vehicle, and if it is determined that the predicted route intersects with the direction of travel of the vehicle, a warning sound is emitted to call for attention.

(Operation of in-vehicle device)
With reference to FIG. 15, the route prediction process by the in-vehicle device 520 will be described in more detail. As described above, since the in-vehicle device 520 is configured in the same manner as in FIG. 4, reference numerals in FIG. 4 will be referred to below. The processing shown in FIG. 15 is realized by the control unit 150 reading a predetermined program from the memory 148 and executing it. Here, it is assumed that the memory 148 stores in advance information specifying an area (map information, etc.) and learning parameters corresponding thereto.

In step 600, control unit 150 identifies the area in which the vehicle is currently traveling. Specifically, the control unit 150 acquires the current position of the vehicle using GPS or the like, and specifies the area that includes the position from the area information stored in the memory 148. The control unit 150 stores information representing the identified area (hereinafter referred to as traveling area information) in a predetermined area of the memory 148.

In step 602, control unit 150 determines whether the area in which the vehicle is traveling has been changed. Specifically, similarly to step 600, control unit 150 identifies the area in which the vehicle is currently traveling, and determines that the identified area is the area identified by the traveling area information stored in memory 148. Determine whether they are different. If it is determined that the area is different (the area has been changed), the area control proceeds to step 604 after overwriting the information representing the specified area over the running area information in the memory 148. Otherwise, control transfers to step 616.

In step 604, control unit 150 determines whether learning parameters corresponding to the area identified in step 602 are stored in memory 148. If it is determined that the information has been stored, control proceeds to step 606. Otherwise, control transfers to step 612.

In step 606, the control unit 150 queries the server in the area where the vehicle is traveling about the version of the learning parameters. Specifically, the control unit 150 broadcasts a predetermined code (hereinafter referred to as a version inquiry code). As will be described later, the server that has received the version inquiry code transmits information (version information) representing the stored version of the learning parameters to the in-vehicle device that has transmitted the version inquiry code. For example, when the in-vehicle device of vehicle 504 traveling in area 510 broadcasts a version inquiry code, server 500 transmits version information to vehicle 504 . On-vehicle devices running around vehicle 504 ignore the version inquiry code even if they receive it.

In step 608, control unit 150 determines whether version information has been received. If it is determined that it has been received, control moves to step 610. Otherwise, the process of step 608 is repeated.

In step 610, the control unit 150 determines whether the version information acquired in step 608 is newer than the version of the learning parameter corresponding to the area identified in step 602 among the learning parameters stored in the memory 148. Determine. If it is determined that the learning parameters are new (ie, the learning parameters stored in memory 148 are old), control moves to step 612. Otherwise (ie, if the learning parameters stored in memory 148 are new or equivalent), control transfers to step 616.

In step 612, the control unit 150 requests the server in the area where the vehicle is traveling to transmit learning parameters. Specifically, the control unit 150 broadcasts a predetermined code (hereinafter referred to as a transmission request code). As will be described later, the server that has received the transmission request code transmits the stored learning parameters to the in-vehicle device that has transmitted the transmission request code. For example, when the in-vehicle device of vehicle 504 traveling in area 510 broadcasts a transmission request code, server 500 transmits learning parameters to vehicle 504 . Even if the on-vehicle device of a vehicle traveling around the vehicle 504 receives the transmission request code, it ignores it.

In step 614, control unit 150 determines whether learning parameters have been received. If it is determined that the learning parameters have been received, the received learning parameters are stored in the memory 148 in association with the area specified in step 620. Control then passes to step 616. Otherwise, step 614 is repeated.

In step 616, the control unit 150 determines whether the sensor data has been updated, that is, whether new sensor data has been acquired from the sensor unit 142. If it is determined that it has been received, control moves to step 618. Otherwise, control transfers to step 624.

In step 618, the control unit 150 analyzes the sensor data acquired in step 616 (corresponding to the function of the analysis processing unit 528 in FIG. 13), and determines that the detection target (pedestrian) has been detected and its position and signs have been determined. Determine whether or not it has been detected. If it is determined that it has been detected, control moves to step 620. Otherwise (eg, if the detection target is not detected, or if the detection target is detected but one of its location and signature is not detected), control transfers to step 624.

In step 620, the control unit 150 reads the learning parameters corresponding to the area identified in step 602, and uses the read learning parameters to determine the position and signs (body orientation and face orientation) detected in step 616. From this, the movement route of the pedestrian detected in step 616 is predicted (corresponding to the function of the prediction unit 530 in FIG. 13), and the predicted route that is the result is stored in the memory 148.

In step 622, the control unit 150 reads the predicted route, which is the prediction result of step 620, from the memory 148 and presents it to the presentation unit 152 (function of the information presentation unit 532 in FIG. 13).

In step 624, control unit 150 determines whether an instruction to end has been received. If it is determined that a termination instruction has been received, this program terminates. Otherwise, control returns to step 602. The instruction to terminate is given, for example, by turning off the engine and electrical system of the vehicle in which the on-vehicle device is mounted.

(server operation)
With reference to FIG. 16, the processing by servers 500 and 502 will be described in more detail. As mentioned above, the servers 500 and 502 are configured in the same manner as in FIG. 3, so the reference numerals in FIG. 3 will be referred to below. The processing shown in FIG. 16 is realized by the control unit 120 reading a predetermined program from the memory 122 and executing it. The memory 122 stores the learning parameters of the corresponding area and their version information. The learning parameters stored by server 500 are different from the learning parameters stored by server 502.

In step 700, the control unit 120 determines whether there is an inquiry about the learning parameter version. Specifically, control unit 120 determines whether or not a version inquiry code has been received. If it is determined that it has been received, control moves to step 702. Otherwise, control transfers to step 708. The version inquiry code is transmitted from the in-vehicle device as described above (see step 606 in FIG. 15).

In step 702, the control unit 120 transmits the version information of the learning parameters stored in the memory 122 to the in-vehicle device that transmitted the version inquiry code. Control then transfers to step 704. Information (address) specifying the in-vehicle device that transmitted the version inquiry code can be found from the source address of the packet containing the version inquiry code received in step 700. The transmitted version information is received by the in-vehicle device that transmitted the version inquiry code (see step 608 in FIG. 15).

In step 704, the control unit 120 determines whether there is a request for learning parameters. Specifically, control unit 120 determines whether a transmission request code has been received.
If it is determined that it has been received, control moves to step 706. Otherwise, control transfers to step 708. The transmission request code is transmitted from the in-vehicle device as described above (see step 612 in FIG. 15).

In step 706, the control unit 120 reads the learning parameters from the memory 122 and transmits them to the in-vehicle device that sent the transmission request code. Control then transfers to step 708. Information (address) specifying the in-vehicle device that transmitted the transmission request code can be found from the source address of the packet containing the transmission request code received in step 704. The transmitted learning parameters are received by the in-vehicle device that transmitted the transmission request code (see step 614 in FIG. 15).

In step 708, the control unit 120 determines whether an instruction to end has been received. If it is determined that a termination instruction has been received, this program terminates. Otherwise, control returns to step 700. The instruction to terminate is issued by, for example, operating the server including the control unit 120 by an administrator or the like.

As described above, if the in-vehicle device stores learning parameters corresponding to the area in which the vehicle is traveling, it can use the learning parameters to determine the predicted route of the detected pedestrian. On the other hand, if the in-vehicle device does not store the learning parameters corresponding to the area where the vehicle is traveling, or if the stored learning parameters are older than the learning parameters that can be obtained from the server, the in-vehicle device learns from the server. The parameters can be obtained and used to determine the predicted path of the detected pedestrian.

For example, with reference to FIG. 12, the in-vehicle device of vehicle 504 traveling in area 510 obtains learning parameters suitable for area 510 from server 500 as needed, and uses them to predict a route. . The on-vehicle device of the vehicle 504 analyzes the sensor data of the sensor mounted on the vehicle 504, and when a pedestrian 590 is detected, for example, based on the position and signs of the pedestrian, the on-vehicle device of the vehicle 504 analyzes the sensor data of the sensor mounted on the vehicle 504, and uses the learning parameters received from the server 500 to determine whether the pedestrian is walking or not. A predicted route 592 for a person 590 can be determined. Further, when the vehicle 504 starts traveling in the area 512, the on-vehicle device of the vehicle 504 can acquire learning parameters suitable for the area 512 from the server 502, if necessary. The in-vehicle device of the vehicle 504 analyzes the sensor data of the sensor mounted on the vehicle 504, and when a pedestrian 594 is detected, for example, based on the position and signs of the pedestrian, the on-vehicle device of the vehicle 504 analyzes the sensor data of the sensor mounted on the vehicle 504, and uses the learning parameters received from the server 502 to determine whether a pedestrian 594 is walking. A predicted route 596 for a person 594 can be determined.

In this way, the on-vehicle device can more accurately determine the predicted route of the detected pedestrian by using learning parameters suitable for the area in which the vehicle in which it is mounted is traveling. Therefore, the in-vehicle device can provide more appropriate driving support to the driver, such as urging caution, based on a highly accurate predicted route.

By transmitting learning parameters to the requested vehicle-mounted device only when the server receives a request from the vehicle-mounted device, it is possible to suppress transmission of unnecessary learning parameters from the server to the vehicle-mounted device.

In the above, a case has been described in which the in-vehicle device stores learning parameters for each of a plurality of areas in a predetermined region in advance. As a result, the on-vehicle device alone can predict the travel path of a moving object, such as a pedestrian, more quickly and accurately using predetermined information (parameters) appropriate for the region, before the object actually moves. .

On the other hand, by enabling the in-vehicle device to acquire learning parameters from servers placed in each area as needed, the in-vehicle device does not have to store learning parameters in advance and can store unnecessary data. can be restrained from doing so.

Although the present invention has been described above by describing the embodiments, the above-described embodiments are merely examples, and the present invention is not limited to the above-described embodiments. The scope of the present invention is indicated by each claim, with reference to the description of the detailed description of the invention, and all changes within the scope and meaning equivalent to the words described therein are defined. include.

100 Route prediction system 102 Route prediction device 104 Roadside sensors 106, 114, 116, 118, 504, 506 Vehicle 108 Base station 110 Network 112 Traffic light (pedestrian traffic light)
120, 150, 168 Control section 122, 148, 166 Memory 124, 146, 164 Communication section 126, 154, 170 Bus 140, 520 On-vehicle device 142, 160, 526 Sensor section 144, 162 I/F section 152 Presentation section 180, 522 Packet receiving section 182 Packet transmitting section 184, 528 Analysis processing section 186 Learning section 188, 530 Prediction section 200, 202, 590, 594 Pedestrian 204 Pedestrian traffic light 206, 208 Vehicle traffic signal 210, 212, 592, 596 Prediction Routes 220, 230 Target frame 222 Face area frame 224 Face orientation 226 Body area frame 228 Body orientation 240 Obstacles 500, 502 Servers 510, 512 Area 524 Parameter storage unit 532 Information presentation unit

Claims (17)

a collection unit that collects sensor data acquired by at least one of the in-vehicle sensor and the roadside sensor via wireless communication or wired communication;
an analysis unit that detects a moving object from the sensor data collected by the collection unit and identifies the position and symptoms of the moving object;
a prediction unit that predicts a path including a future position of the moving object using the position of the moving object and the sign specified by the analysis unit;
a storage unit that stores the position and the symptom identified by the analysis unit in association with each of the moving objects;
A route prediction device comprising: a learning unit that causes the prediction unit to learn using the position and the symptom stored in the storage unit for a predetermined period. The moving object is a person,
The route prediction device according to claim 1, wherein the sign includes information representing the direction of the person's face. The route prediction device according to claim 2, wherein the sign further includes information representing the direction of the person's body. The learning department is
Learning the prediction unit by machine learning,
Among the time-series data of the position and the sign that are specified for each moving object over a predetermined period of time and stored in the storage unit, first time-series data of the position and the sign that belong to the past; is the input data for the machine learning,
Among the time series data of the position and the symptom that are specified for each moving object over a predetermined time and stored in the storage unit, time series data that does not include the first time series data, The route prediction device according to any one of claims 1 to 3, which is used as training data for the machine learning. The route prediction device according to any one of claims 1 to 4, further comprising a distribution unit configured to distribute the route predicted by the prediction unit to an in-vehicle device. The route prediction device according to claim 5, wherein the distribution unit distributes image data of the moving object corresponding to the route along with the route predicted by the prediction unit. Claim further comprising: an evaluation unit that uses the route predicted by the prediction unit to evaluate the possibility of a collision between the moving object that is a prediction target of the route and a moving object different from the moving object. The route prediction device according to any one of claims 1 to 6. The collection unit collects sensor data acquired by the in-vehicle sensor and sensor data acquired by the roadside sensor,
Determining whether a first moving object detected from the sensor data obtained by the in-vehicle sensor and a second moving object detected from the sensor data obtained by the roadside sensor are the same moving object. further including a determination section;
The prediction unit predicts the route by using both the position and the sign specified for each of the first moving object and the second moving object that are determined to be the same moving object. 8. The route prediction device according to claim 7. An on-vehicle device comprising the route prediction device according to any one of claims 1 to 8. an on-vehicle device mounted on a vehicle including a first sensor;
a roadside device including a second sensor installed on the road;
a route prediction device;
The in-vehicle device transmits sensor data acquired by the first sensor to the route prediction device via wireless communication,
The roadside device transmits sensor data acquired by the second sensor to the route prediction device via wireless communication or wired communication,
The route prediction device includes:
a receiving unit that receives the sensor data transmitted from the in-vehicle device and the sensor data transmitted from the roadside device;
an analysis unit that detects a moving object from the sensor data received by the reception unit and identifies the position and symptoms of the moving object;
a prediction unit that predicts a path including a future position of the moving object using the position of the moving object and the sign specified by the analysis unit;
a storage unit that stores the position and the symptom identified by the analysis unit in association with each of the moving objects;
A route prediction system comprising: a learning section that causes the prediction section to learn using the position and the symptom stored in the storage section for a predetermined period. An on-vehicle device installed in a vehicle including a sensor,
a distribution device that distributes predetermined information to the in-vehicle device via wireless communication,
The in-vehicle device includes:
a receiving unit that receives the predetermined information;
an analysis unit that detects a moving object and identifies the position and signs of the moving object from the sensor data detected by the sensor;
a prediction unit that predicts a route including a future position of the moving object using the position and the symptom of the moving object specified by the analysis unit and the predetermined information received by the receiving unit; including,
The predetermined information includes the position and the sign obtained by repeating a process for a predetermined period of time to detect a moving object from sensor data acquired by at least one of an in-vehicle sensor and a roadside sensor, and to specify the position and symptoms of the moving object. A route prediction system in which learning parameters are obtained by training a predictor of the same model as the prediction unit using the set of symptoms. including a plurality of the distribution devices,
Each of the plurality of distribution devices includes:
Placed in each predetermined area including roads where vehicles can pass,
distributing the predetermined information corresponding to the predetermined area via the wireless communication;
The receiving unit of the in-vehicle device receives the predetermined information corresponding to the predetermined area from one of the plurality of delivery devices that is disposed in the predetermined area including the current position of the vehicle. The route prediction system according to item 11. The in-vehicle device further includes a transmitter that transmits a transmission request to the distribution device disposed in the predetermined area including the current position of the vehicle, requesting transmission of the predetermined information corresponding to the predetermined area,
The route prediction according to claim 12, wherein the distribution device that has received the transmission request transmits the predetermined information corresponding to the predetermined area in which the distribution device is placed to the in-vehicle device that has transmitted the request. system. The receiving unit receives a version of the predetermined information corresponding to the predetermined area, which is distributed from the distribution device disposed in the predetermined area including the current position of the vehicle,
The in-vehicle device includes:
a storage unit that stores the learning parameters used by the prediction unit to predict the route for each of the predetermined areas;
Determining whether the version acquired by the receiving unit is newer than the version of the learning parameter corresponding to the predetermined area including the current position of the vehicle, among the learning parameters stored in the storage unit. further comprising a determination unit that determines
The determination unit determines that the version acquired by the reception unit is newer than the version of the learning parameter corresponding to the predetermined area including the current position of the vehicle, among the learning parameters stored in the storage unit. The route prediction system according to claim 13, wherein the transmitter transmits the transmission request in response to the determination. sensor and
an analysis unit that detects a moving object and identifies the position and signs of the moving object from the sensor data detected by the sensor;
a prediction unit that predicts a route including a future position of the moving object from the position and the symptom of the moving object specified by the analysis unit;
a storage unit that stores learning parameters used by the prediction unit to predict the route for each predetermined area including roads through which vehicles can pass;
The learning parameters are the positions and signs obtained by repeating for a predetermined period a process of detecting a moving object from sensor data acquired by at least one of an in-vehicle sensor and a roadside sensor, and identifying the position and signs of the moving object. The vehicle-mounted device is a learning parameter obtained by training a predictor of the same model as the prediction unit using the set of symptoms. a collection step of collecting sensor data acquired by at least one of the in-vehicle sensor and the roadside sensor via wireless communication or wired communication;
an analysis step of detecting a moving object from the sensor data collected in the collecting step and identifying the position and signs of the moving object;
a prediction step of predicting a path including a future position of the moving object using the position of the moving object identified in the analysis step and the sign;
a storage step of storing the position and the symptom identified in the analysis step in correspondence with each of the moving objects;
A route prediction method, comprising: a learning step of learning the prediction step using the position and the symptom stored for a predetermined period in the storage step. to the computer,
a collection function that collects sensor data acquired by at least one of the in-vehicle sensor and the roadside sensor via wireless communication or wired communication;
an analysis function that detects a moving object from the sensor data collected by the collection function and identifies the position and symptoms of the moving object;
a prediction function that predicts a path including a future position of the moving object using the position of the moving object and the signs identified by the analysis function;
a memory function that stores the position and the symptom identified by the analysis function in correspondence with each of the moving objects;
and a learning function for learning the prediction function using the position and the symptom stored for a predetermined period by the storage function.

Images