JP2022135676A - Moving body control system and moving body control method - Google Patents

Abstract

To provide a moving body control system which can control a motion of a moving body according to a situation of a person in the surroundings of the moving body.SOLUTION: A moving body control system comprises: an acquisition unit which acquires an image of a person existing in the surroundings of a moving body; an estimation unit which estimates consciousness of the person to the moving body on the basis of the image; and a notification unit which notifies the moving body of an estimation result of the estimation unit.SELECTED DRAWING: Figure 5

Description

TECHNICAL FIELD The present invention relates to a technique for assisting traveling of a moving body.

2. Description of the Related Art In recent years, as one type of driving support for a moving object such as a vehicle, extracting risk factors that exist around the moving object that should be noted when traveling and providing guidance about the extracted risk factors has been performed.

In Patent Document 1, a judgment target area is set for judging risk factors existing in the surrounding environment of a mobile object, and machine learning is performed using captured images and map information images around the mobile object as input images. Extraction of regional risk factors based on training data is described.

JP 2019-164611 A

However, with the configuration described in Patent Document 1, although it is possible to determine the risk factors that exist around the moving object, it is not possible to determine the consciousness of people around the moving object with respect to the moving object. There is a problem. Therefore, even if the driver expects the moving body to respond to the risk factor, the moving body cannot respond.

SUMMARY OF THE INVENTION The present invention has been made in view of the problems described above, and an object thereof is to provide a mobile body control system capable of controlling the operation of a mobile body according to the situation of people around the mobile body. .

A moving body control system according to the present invention includes acquisition means for acquiring an image of a person around the moving body, estimation means for estimating the consciousness of the person to the moving body based on the image, and estimation by the estimation means. and notification means for notifying the moving body of the result.

ADVANTAGE OF THE INVENTION According to this invention, it becomes possible to provide the mobile body control system which can control operation|movement of a mobile body according to the situation of the person around a mobile body.

1 is a diagram showing the configuration of a mobile body control system according to a first embodiment of the present invention; FIG. 4 is a diagram showing the hardware configuration of each device in the first embodiment; FIG. FIG. 2 is a diagram showing a hardware configuration and a software configuration realized by using a program according to the first embodiment; FIG. A conceptual diagram showing an input/output structure using a learning model. The figure explaining operation|movement of the mobile control system in 1st Embodiment. 4 is a flowchart showing the flow of learning in the learning phase of the first embodiment; 4 is a flow chart showing the flow of estimation in the estimation phase of the first embodiment; The figure which shows the hardware constitutions of each apparatus in 2nd Embodiment. The figure which shows the software configuration implement|achieved by using the hardware configuration and program in 2nd Embodiment. The figure explaining operation|movement of the mobile body control system in 2nd Embodiment. The figure explaining operation|movement of the mobile control system in 3rd Embodiment. 10 is a flow chart showing the flow of learning in the learning phase of the third embodiment; 9 is a flow chart showing the flow of estimation in the estimation phase of the third embodiment;

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. In addition, the following embodiments do not limit the invention according to the scope of claims. Although multiple features are described in the embodiments, not all of these multiple features are essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant description is omitted.

(First embodiment)
FIG. 1 is a diagram showing the configuration of a mobile body control system according to the first embodiment of the present invention. In the present embodiment, a vehicle will be described as an example of a moving body, but a bicycle or the like may be used as long as it moves.

In FIG. 1 , a vehicle-mounted camera 102 is connected to a vehicle 103 via a local network, and an estimation server 104 is connected to the vehicle 103 via the Internet 101 . Note that the vehicle 103 may be equipped with the estimation server 104, and the vehicle 103 and the estimation server 104 may all be connected via a local network.

FIG. 2 is a diagram showing the hardware configuration of each device that constitutes the mobile body control system of FIG.

The vehicle 103 includes a vehicle system 219 that controls the vehicle, a vehicle sensor 211 that monitors the state of the vehicle, a navigation system 220 that guides the destination of the vehicle, an input unit 212 that receives operations and the like, and a display unit that displays information about the guidance route. 213. Furthermore, the vehicle 103 includes a speaker 221 that emits a warning sound, etc., a vehicle-mounted camera 102 that captures the surroundings of the vehicle, a CPU 214 that performs arithmetic processing, a ROM 215 that stores a program for selecting people, and a RAM 216 that expands the program. Further, the vehicle 103 includes an HDD 217 storing image data and the like, a GPU 222 for image processing, and a NIC 218 for network communication.

The estimation server 104 includes a CPU 201 that performs arithmetic processing, a ROM 202 that stores an estimation program, a RAM 203 that stores the program, an HDD 204 that stores training data and the like, and a GPU 205 that performs image processing. Further, the estimation server 104 includes an input unit 207 that receives operations and the like, a display unit 208 that displays data, and a NIC 206 for network communication.

FIG. 3 is a diagram showing a software configuration realized by using the hardware configuration and programs shown in the hardware configuration diagram of FIG.

A data collection/selection unit 303 performs a selection process on image data acquired by an image acquisition unit (imaging unit) 301 of the vehicle-mounted camera 102 to determine whether or not a person is captured. The data collecting/sorting unit 303 is implemented by the CPU 201 developing a sorting program stored in the ROM 215 in the RAM 216 and executing it. In this case, the GPU 222 is also used for efficient image processing. The CPU 214 and the GPU 222 cooperate to perform the calculation of the data collection/selection unit 303 . Note that the processing of the data collection/selection unit 303 may be performed by either the CPU 214 or the GPU 222 only.

The image of the person selected by the data collection/selection unit 303 is transmitted from the data transmission unit 304 to the estimation server 104 . In the estimation server 104, the estimation unit 310 performs estimation processing for estimating the consciousness of the person in the image toward the vehicle 103 (awareness corresponding to the image) using the received image data as input data. The estimation unit 310 is implemented by the CPU 201 developing an estimation program stored in the ROM 202 in the RAM 203 and executing the program. In this case, the GPU 205 is also used for efficient image processing. The CPU 201 and the GPU 205 cooperate to perform the calculation of the estimation unit 310 . In addition, the processing of the estimation unit 310 may be performed by only one of the CPU 201 and the GPU 205 . Estimating section 310 transmits the estimation result to data receiving section 305 of vehicle 103 .

FIG. 4 is a conceptual diagram showing an input/output structure using the learning model 403 in the first embodiment. An image of a person photographed by the vehicle 103 is input to the input X401, and the consciousness of the photographed person toward the vehicle 103 is output to the output Y402.

FIG. 5 is a diagram for explaining the operation of the mobile body control system using the structure of the learning model 403 shown in FIG.

The CPU 214 of the vehicle 103 takes an image of a person around the vehicle 103 using the vehicle-mounted camera 102, and transfers the acquired image to the data transmission unit 304 of the vehicle 103 (S1 in the figure). The data transmission unit 304 transmits the received images of people around the vehicle 103 to the estimation server 104, and requests the estimation server 104 to estimate the intentions of the people in the image toward the vehicle 103 (S2 in the figure). The estimation server 104 uses the transmitted image as an input to estimate the consciousness of the person in the image toward the vehicle 103 (S3 in the figure). The estimation server 104 transmits the estimation result to the vehicle 103 (S4 in the figure). Upon receiving the estimation result, the CPU 214 of the vehicle 103 outputs the estimation result through the speaker 221 and the display unit 213 to notify the driver (S5 in the figure).

By performing the operation shown in FIG. 5 , the driver of the vehicle 103 is notified of the awareness of the people around the vehicle 103 , and the effect can be obtained that this can be reflected in the operation of the vehicle 103 .

FIG. 6 is a flow chart showing the detailed flow of learning in the learning phase.

First, the image acquisition unit 301 of the vehicle 103 captures an image around the vehicle 103 using the vehicle-mounted camera 102 in order to collect data (S601). Then, the data collection/selection unit 303 selects an image in which a person appears from the captured images (S602).

The learning unit 309 of the estimation server 104 acquires the selected image and information about the person's awareness of the vehicle 103 (S611). Then, the learning unit 309 learns the correspondence relationship using the selected image as input data and the person's awareness of the vehicle 103 as teacher data (S612), and creates a learned model (S613).

Specific algorithms of machine learning include nearest neighbor method, naive Bayes method, decision tree, support vector machine, and the like. Another example is deep learning in which a neural network is used to generate feature values and connection weighting coefficients for learning. As appropriate, any of the above algorithms that can be used can be used and applied to the present embodiment.

A description will be added regarding the learning process in S612. Here, the learning unit 309 in FIG. 3 may include an error detection unit and an update unit. In this case, the error detection unit detects an error between the output data output from the output layer of the neural network according to the input data input to the input layer and the teacher data. Specifically, the error detector uses a loss function to calculate the error between the output data from the neural network and the teacher data.

Based on the error obtained by the error detection unit, the update unit updates the weighting coefficients for coupling between nodes of the neural network so that the error is reduced. This updating unit updates the connection weighting coefficients and the like using, for example, the error backpropagation method. The error backpropagation method is a method of adjusting the connection weighting coefficients and the like between nodes of each neural network so as to reduce the above error.

FIG. 7 is a flow chart showing the detailed flow of estimation in the estimation phase.

The estimation unit 310 of the estimation server determines whether or not the image of the person acquired by the vehicle-mounted camera 102 and the request for estimating the consciousness of the person to the vehicle 103 have been received from the vehicle 103 (S701). If received in S701, the estimation unit 310 inputs the received image to the trained model (S702). If it is not received, it waits as it is.

The estimation unit 310 estimates the person's awareness of the vehicle 103 from the received image (S703). Furthermore, the result of the estimated person's awareness of the vehicle 103 is transmitted to the vehicle 103 (S704). When the data receiving unit 305 of the vehicle 103 receives the estimation result of the person's consciousness toward the vehicle 103 from the estimation server 104, it outputs the received result to the speaker 221 and the display unit 213 of the vehicle 103 to inform the driver (S705). . Alternatively, only necessary information may be conveyed to the driver based on the received results.

It should be noted that, since the determination results for one image may result in variations in the estimation results, images may be acquired and determined repeatedly to output results with improved accuracy.

(Second embodiment)
Next, a second embodiment will be described. The configuration of the mobile body control system of the second embodiment is the same as that of the first embodiment. FIG. 8 is a diagram showing the hardware configuration of each device that constitutes the system. It is the same as the first embodiment except that the vehicle 103 is equipped with an automatic driving module 823 for automatic driving.

FIG. 9 is a diagram showing a software configuration realized by using the hardware and programs shown in the hardware configuration diagram of FIG. It is the same as the first embodiment except that an automatic driving module 911 is arranged in the vehicle 103 .

FIG. 10 is a diagram for explaining the operation of the mobile body control system. The steps up to the estimation server 104 transmitting the estimation result to the vehicle 103 are the same as in the first embodiment. The CPU 214 of the vehicle 103 that has received the estimation result inputs the estimation result to the automatic driving module 911 (S4 in the figure) and reflects it in the driving behavior of the vehicle 103 (S5 in the figure).

For example, when the estimation result of the consciousness of people around the vehicle indicates that they want the person to get on the vehicle, the vehicle decelerates, stops in front of the person, opens the door, and performs an operation to get the person on the vehicle. If the person is not paying attention to the vehicle, the vehicle continues to run.

The learning phase and estimation phase are the same as in the first embodiment.

(Third Embodiment)
In this third embodiment, a diagram showing a mobile object control system, a hardware configuration diagram of each device constituting the system, and a software configuration diagram realized by using the hardware and programs shown in the hardware configuration diagram are the same as in the first and second embodiments.

The conceptual diagram showing the input/output structure using the learning model is the same as in the first and second embodiments. , the intention of the photographed person with respect to the vehicle 103 and the estimation result of the movement direction are output.

FIG. 11 is a diagram for explaining the operation of the mobile body control system using the structure of the learning model.

The CPU 214 of the vehicle 103 takes an image of a person around the vehicle 103 using the vehicle-mounted camera 102, and transfers the acquired image to the data transmission unit 304 of the vehicle 103 (S1 in the figure). The data transmission unit 304 transmits the received image of the person around the vehicle 103 to the estimation server 104, and the estimation server 104 receives the result of estimation of the consciousness of the person in the image toward the vehicle 103 and the estimation of the moving direction of the person in the image. Request the result (S2 in the figure). The estimation server 104 receives the transmitted image and estimates the intention of the person in the image toward the vehicle 103 and the moving direction of the person in the image (S3 in the figure). The estimation server 104 transmits the estimation result to the vehicle 103 (S4 in the figure). The CPU 214 of the vehicle 103 that has received the estimation result outputs the estimation result through the speaker 221 or the display unit 213 to notify the driver or reflect it in automatic driving (S5 in the figure).

By performing the operation shown in FIG. 11 , it is possible to obtain an effect that the awareness of the vehicle 103 and the moving direction of the people around the vehicle 103 can be reflected in the operation of the vehicle 103 .

For example, in the case of the estimation result that the person wants the vehicle to stop and the estimation result that the person is moving in the traveling direction of the vehicle, the speaker 221 informs the driver that there is a person who wants the vehicle to stop, It prompts the driver to slow down, and automatically slows down in the case of autonomous driving.

FIG. 12 is a flow chart showing the detailed flow of learning in the learning phase.

First, the image acquisition unit 301 of the vehicle 103 captures an image around the vehicle 103 using the vehicle-mounted camera 102 in order to collect data (S1201). Then, the data collection/selection unit 303 selects an image in which a person appears from the captured images (S1202).

The learning unit 309 of the estimating server 104 acquires the selected image and the information about the person's awareness of the vehicle 103 and the moving direction (S1211). Then, the learning unit 309 learns the selected image as input data and the person's awareness of the vehicle 103 and the movement direction as teacher data (S1212), and creates a learned model (S1213).

FIG. 13 is a flow chart showing the detailed flow of estimation in the estimation phase.

The estimation unit 310 of the estimation server determines whether or not the image of the person acquired by the vehicle-mounted camera 102 and the request for estimating the person's consciousness and movement direction with respect to the vehicle 103 have been received from the vehicle 103 (S1301). If received in S1301, the estimation unit 310 inputs the received image to the trained model (S1302). If it is not received, it waits as it is.

The estimating unit 310 estimates the person's awareness of the vehicle 103 and the moving direction from the received image (S1303). Further, the result of the estimated person's awareness of the vehicle 103 and the movement direction is transmitted to the vehicle 103 (S1304). When the data receiving unit 305 of the vehicle 103 receives the result of estimation of the person's consciousness toward the vehicle 103 and the movement direction from the estimation server 104, the data receiving unit 305 outputs the received result to the speaker 221 and the display unit 213 of the vehicle 103 to inform the driver. Alternatively, it is input to the automatic driving module 911 and reflected in the driving behavior of the vehicle 103 (S1305). Alternatively, only necessary information may be conveyed to the driver based on the received results.

It should be noted that, since the determination results for one image may result in variations in the estimation results, images may be acquired and determined repeatedly to output results with improved accuracy.

(Other embodiments)
In addition, the present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus reads the program. It can also be realized by executing processing. It can also be implemented by a circuit (eg, ASIC) that implements one or more functions.

The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the claims are appended to make public the scope of the invention.

101: Internet, 102: Vehicle-mounted camera, 103: Vehicle, 104: Estimation server

Claims (11)

Acquisition means for acquiring an image of a person around the moving object;
an estimating means for estimating a person's awareness of moving objects based on the image;
a notification means for notifying the moving object of the estimation result of the estimation means;
A mobile body control system comprising: The estimating means estimates a person's awareness of moving objects by inputting the images into a learning model that has learned a relationship between a person's image and the person's awareness of moving objects corresponding to the image. The mobile body control system according to claim 1. 3. The locomotion according to claim 2, wherein the learning model is a learned learning model obtained by learning using images of people as input data and information about people's awareness of moving bodies corresponding to the images as teacher data. body control system. 4. The moving body control system according to any one of claims 1 to 3, further comprising imaging means for capturing images of people around the moving body. 5. The mobile body control system according to any one of claims 1 to 4, wherein said notification means notifies a driver of said mobile body of said estimation result. 6. The moving body control system according to claim 5, wherein said notification means uses at least one of a speaker and display means to notify a driver of said moving body of said estimation result. 7. The mobile body control system according to any one of claims 1 to 6, further comprising automatic driving means for controlling movement of said mobile body according to a notification from said notification means. 3. The moving body control system according to claim 2, wherein said estimation means further estimates a movement direction of a person based on said image. The learning model is a learning model that has been learned by using an image of a person as input data, and information about the person's awareness of moving objects corresponding to the image and information about the direction of movement of the person as teacher data. The mobile body control system according to claim 8 . 10. The mobile body control system according to any one of claims 1 to 9, wherein said mobile body and said estimation means are connected via the Internet. an acquisition step of acquiring an image of a person in the vicinity of the moving object;
an estimation step of estimating a person's awareness of moving objects based on the image;
a notification step of notifying the moving object of the estimation result of the estimation step;
A moving body control method, comprising:

Images