JP6947091B2 - Driving support device, driving support method, driving support program, motion control device, motion control method, and motion control program - Google Patents

Description

The present invention relates to a driving support device, a driving support method, a driving support program, an operation control device, an operation control method, and an operation control program.

In recent years, the development of technologies for assisting the driving of a vehicle, such as assisting the steering of a steering wheel, automatically performing a brake operation, and performing automatic driving that does not depend on the driver's driving operation, has been progressing. For example, Patent Document 1 proposes an autonomous driving vehicle having a manual driving mode and an automatic driving mode, which switches from the manual driving mode to the automatic driving mode according to a predetermined in-vehicle condition.

JP-A-2015-133050

The present inventors have found that the conventional technology for supporting the driving of a vehicle has the following problems. That is, in the conventional technology, for example, support for a specific driving operation is associated with a specific operation, such as determining an auxiliary force for steering according to the steering angle of the steering wheel or performing automatic driving at a constant speed. Was being done. Therefore, it is not possible to provide support for a specific driving operation by assuming in advance individual situations caused by individual users, vehicle characteristics, etc. that occur during actual driving, and there are variations in the mode of supporting driving. I couldn't.

On the other hand, in order to give variation to the mode of supporting driving, it is conceivable to construct a system that accepts a desired operation when supporting driving from individual users. However, in such a system, it takes time and effort to input an operation suitable for an individual situation and to register the input operation content, and such a registration work is performed according to an individual situation. It was inefficient as much as it was done.

It should be noted that this problem is not peculiar to the scene of supporting the driving operation. For example, a similar problem may occur in every situation where the operation of a target device used by a user, such as a robot on a production line, is controlled. That is, if the desired operation is accepted from each user in order to give variation in the mode for controlling the operation of the target device, it is inefficient because the registration work is performed according to the individual situation.

One aspect of the present invention has been made in view of such circumstances, and an object of the present invention is to provide a technique for giving variation to a mode for controlling the operation of a target device in a simple and efficient manner. Is.

The present invention employs the following configuration in order to solve the above-mentioned problems.

That is, the driving support device according to one aspect of the present invention is a support command for instructing support for driving a vehicle, and is a support command for driving the vehicle based on a support command including a command value related to a predetermined attribute of the driving operation. A driving control unit that controls assistance, an image acquisition unit that acquires a captured image from a photographing device arranged to photograph a occupant in the vehicle, and a driving operation for correcting the driving operation according to the situation reflected in the image. By inputting the captured image into the learned learning device that has undergone machine learning and executing the arithmetic processing of the learned learning device, the corrected value for the command value of the support command is corrected to the learned learning device. The operation correction unit includes an operation correction unit acquired from the above, and the operation control unit corrects the command value by the acquired correction value, and supports the operation of the vehicle based on the support command after the command value is corrected. To control.

In this configuration, by using the learned learner, the captured image obtained from the photographing device arranged to photograph the occupant with respect to the command value of the support command instructing the driving assistance of the vehicle can be obtained. It is possible to obtain different correction values according to the individual situations in which the image is captured. By applying this correction value to the support command, it is possible to realize a variation of the operation that supports the driving, so that it is possible to save the trouble of inputting the operation suitable for each situation. In addition, in this configuration, since the variation of the operation that supports the driving is realized by using the correction value obtained from the learned learner, the operation that supports the driving suitable for each situation can be individually performed from the beginning. It is not necessary to generate it, and it is possible to efficiently give variation to the operation that supports the operation. Therefore, according to the configuration, it is possible to have variations in the mode of supporting driving in a simple and efficient manner, whereby driving suitable for individual situations due to individual users, vehicle characteristics, and the like can be provided. Support can be realized.

In addition, "driving support" may be intended for all kinds of support for driving operation, and may include, for example, assistance of the driver's driving operation during manual driving, implementation of automatic driving, and the like. The "occupant" may be intended for any person present in the vehicle and may include an occupant in the driver's seat (ie, the driver) and an occupant in a seat other than the driver's seat. The "learner" is composed of a learning model such as a neural network that can acquire the ability to make a predetermined inference by machine learning. The type of the learning device does not have to be particularly limited, and may be appropriately determined according to the embodiment. The "learned learner" may be referred to as a "discriminator" or a "classifier".

In the driving support device according to the one aspect, the learned learning device may be constructed so as to correct the driving motion according to the state of the person appearing in the image by the machine learning. According to this configuration, it becomes possible to correct the operation of the driving support according to the situation of the occupant.

In the driving support device according to the one aspect, the learned learning device may be constructed so as to correct the driving motion according to the facial expression of the person in the image by the machine learning. According to this configuration, it becomes possible to correct the driving support operation according to the facial expression of the occupant.

In the driving support device according to the one aspect, the learned learner performs correction to reduce or cancel the driving support according to the fear of the facial expression of the person in the image by the machine learning. It may be built in. According to this configuration, the operation of supporting the driving can be reduced or canceled depending on the fear of the occupant's facial expression.

In the driving support device according to the one aspect, the learned learning device may be constructed so as to correct the driving motion according to the gesture of the person appearing in the image by the machine learning. According to this configuration, it becomes possible to correct the operation of the driving support according to the gesture of the occupant.

In the driving support device according to the one aspect, the motion correction unit further inputs operation information indicating an operation command of the vehicle by the driver to the learned learner, and performs arithmetic processing of the learned learner. By executing this, the correction value may be acquired from the learned learner. According to this configuration, it becomes possible to correct the operation of the driving support according to the operation of the driver and the situation reflected in the captured image.

In the driving support device according to the one aspect, the predetermined attribute of the driving operation may be at least one of the steering angle of the steering wheel, the braking amount, and the accelerator amount. According to this configuration, at least one of the steering angle of the steering wheel, the amount of braking, and the amount of accelerator in the driving support operation can be corrected according to the situation shown in the captured image.

The driving support device according to each of the above modes may be changed so as to be applicable to any situation in which the operation of the target device used by the user is controlled.

For example, the motion control device according to one aspect of the present invention is an instruction command for instructing a target device to perform a predetermined operation, and is based on an instruction command including a command value related to a predetermined attribute of the operation of the target device. An operation control unit that controls the operation, an image acquisition unit that acquires a photographed image from a photographing device arranged so as to photograph a user who uses the target device, and a motion control unit that acquires the photographed image, and corrects the operation according to the situation reflected in the image. By inputting the captured image into the learned learning device that has been machine-learned for the purpose and executing the arithmetic processing of the learned learning device, the correction value for the command value of the instruction command is corrected to the learned learning device. The operation correction unit includes an operation correction unit acquired from the device, the operation control unit corrects the command value by the acquired correction value, and operates the target device based on an instruction command after correcting the command value. To control.

The "target device" may include any device whose operation can be controlled, and may include, for example, an industrial robot, a belt conveyor, or the like used in a production line. The "command value" is supplied to the target device and defines the attribute value of the operation of the target device. The “predetermined attribute” may include any attribute of the operation of the target device, and may be, for example, a driving amount of a motor of an industrial robot or a belt conveyor.

Further, as another aspect of the driving support device and the operation control device according to each of the above modes, an information processing method or a program that realizes each of the above configurations may be used, or such a program may be used. It may be a stored storage medium that can be read by a computer or other device, machine, or the like. Here, the storage medium that can be read by a computer or the like is a medium that stores information such as a program by electrical, magnetic, optical, mechanical, or chemical action.

For example, the driving support method according to one aspect of the present invention is a support command in which a computer instructs a vehicle to support driving, and the vehicle is based on a support command including a command value related to a predetermined attribute of a driving operation. A step of controlling the driving support of the vehicle, a step of acquiring a photographed image from a photographing device arranged to photograph the occupant in the vehicle, and a machine for correcting the driving motion according to the situation reflected in the image. By inputting the captured image into the learned learning device that has been trained and executing the arithmetic processing of the learned learning device, the correction value for the command value of the support command is corrected from the learned learning device. The step of acquiring, the step of correcting the command value by the acquired correction value, and the step of controlling the driving support of the vehicle based on the support command after correcting the command value are executed. It is an information processing method.

Further, for example, the driving support program according to one aspect of the present invention is a support command for instructing a computer to support driving of a vehicle, and is based on a support command including a command value related to a predetermined attribute of driving operation. A step of controlling the driving support of the vehicle, a step of acquiring a photographed image from a photographing device arranged to photograph an occupant in the vehicle, and a step of correcting the driving operation according to the situation reflected in the image. By inputting the captured image into the learned learning device that has undergone machine learning and executing the arithmetic processing of the learned learning device, the correction value for the command value of the support command is corrected to the learned learning device. The step of acquiring the command value from the device, the step of correcting the command value by the acquired correction value, and the step of controlling the driving support of the vehicle based on the support command after correcting the command value are executed. It is a program to make it.

Further, for example, the operation control method according to one aspect of the present invention is an instruction command in which a computer instructs a target device to perform a predetermined operation, and is based on an instruction command including a command value relating to a predetermined attribute of the operation. , A step of controlling the operation of the target device, a step of acquiring a captured image from a photographing device arranged to photograph a user using the target device, and correcting the operation according to the situation reflected in the image. By inputting the captured image into the trained learner that has been machine-learned to perform the operation and executing the arithmetic processing of the trained learner, the correction value for the command value of the instruction command has been learned. A step of acquiring the command value from the learner, a step of correcting the command value by the acquired correction value, and a step of controlling the operation of the target device based on an instruction command after correcting the command value are executed. It is an information processing method.

Further, for example, the operation control program according to one aspect of the present invention is an instruction command for instructing a target device to perform a predetermined operation based on an instruction command including a command value related to a predetermined attribute of the operation. , A step of controlling the operation of the target device, a step of acquiring a captured image from a photographing device arranged to photograph a user using the target device, and correcting the operation according to the situation reflected in the image. By inputting the captured image into the trained learner that has been machine-learned to perform the operation and executing the arithmetic processing of the trained learner, the correction value for the command value of the instruction command has been learned. A step of acquiring the command value from the learner, a step of correcting the command value by the acquired correction value, and a step of controlling the operation of the target device based on an instruction command after correcting the command value are executed. It is a program to make it.

According to the present invention, it is possible to provide a technique for giving variation to a mode of controlling the operation of a target device by a simple and efficient method.

FIG. 1 schematically illustrates an example of a situation in which the present invention is applied. FIG. 2 schematically illustrates an example of the hardware configuration of the driving support device according to the embodiment. FIG. 3 schematically illustrates an example of the hardware configuration of the learning device according to the embodiment. FIG. 4 schematically illustrates an example of the software configuration of the driving support device according to the embodiment. FIG. 5 schematically illustrates an example of the software configuration of the learning device according to the embodiment. FIG. 6 illustrates an example of the processing procedure of the driving support device according to the embodiment. FIG. 7 illustrates an example of the processing procedure of the learning device according to the embodiment. FIG. 8 schematically illustrates an example of another situation to which the present invention is applied. FIG. 9 schematically illustrates an example of the hardware configuration of the motion control device according to another form. FIG. 10 schematically illustrates an example of the software configuration of the motion control device according to another form.

Hereinafter, embodiments according to one aspect of the present invention (hereinafter, also referred to as “the present embodiment”) will be described with reference to the drawings. However, the embodiments described below are merely examples of the present invention in all respects. Needless to say, various improvements and modifications can be made without departing from the scope of the present invention. That is, in carrying out the present invention, a specific configuration according to the embodiment may be appropriately adopted. The data appearing in the present embodiment are described in natural language, but more specifically, the data is specified in a pseudo language, a command, a parameter, a machine language, etc. that can be recognized by a computer.

§1 Application example First, an example of the basic configuration of the present invention will be described. In order to give variation in the mode of controlling the operation of the target device, if a system that accepts a desired operation from an individual user is constructed, the following problems may occur. That is, it takes time and effort to input an operation suitable for an individual situation and to register the input operation content, and it is inefficient because such registration work is performed according to an individual situation. Is.

Therefore, in one example of the present invention, the operation of the target device is controlled based on an instruction command that instructs the target device to perform a predetermined operation and includes a command value related to a predetermined attribute of the operation. The target device may include any device whose operation can be controlled. The command value is supplied to the target device and defines the attribute value of the operation of the target device. A given attribute may include any attribute of the operation of the target device.

Next, a photographed image is acquired from a photographing device arranged so as to photograph a user who uses the target device. Subsequently, the acquired captured image is input to the learned learning device that has undergone machine learning to correct the operation of the target device according to the situation reflected in the image, and the arithmetic processing of the learned learning device is executed. As a result, the correction value for the command value of the instruction command is acquired from the learned learner. Then, the command value is corrected by the acquired correction value, and the operation of the target device is controlled based on the instruction command after the command value is corrected.

Thereby, in one example of the present invention, by using the learned learner, a photographing device arranged to photograph the user with respect to the command value of the instruction command instructing the target device to perform a predetermined operation. It is possible to obtain different correction values according to the individual situations reflected in the captured image obtained from the above. By applying this correction value, it is possible to realize variations in the operation of the target device, so that it is possible to save the trouble of inputting the operation suitable for each situation. Further, in the example of the present invention, since the variation of the operation is realized by using the correction value obtained from the learned learner, it is not necessary to individually generate the operation suitable for each situation from scratch. , It is possible to efficiently give variation to the operation of the target device. Therefore, according to an example of the present invention, it is possible to have variations in the mode of controlling the operation of the target device by a simple and efficient method.

Next, an example of a situation in which the present invention is applied will be described with reference to FIG. FIG. 1 schematically illustrates an example of a scene applied to the driving support of the vehicle of the present invention. However, the scope of application of the present invention is not limited to the example of vehicle driving support described below. The present invention can be applied to any situation that supports the operation of the target device.

The driving support device 1 illustrated in FIG. 1 is a computer configured to control the driving support of the vehicle 100. The driving support device 1 according to the present embodiment is a support command for instructing the driving support of the vehicle 100, and supports the driving of the vehicle 100 based on the support command including the command value related to a predetermined attribute of the driving operation. Control. The vehicle 100 is an example of the "target device" of the present invention. Driving assistance is an example of the "predetermined operation" of the present invention. The support command is an example of the "instruction command" of the present invention.

Here, the driving assistance may include any assistance for the driving operation, and may include, for example, assisting the driving operation of the driver D in the manual driving mode. Examples of driving operations to be assisted include steering wheel operation, brake operation, accelerator operation, and the like. Further, when the vehicle 100 is configured to be able to execute an automatic driving mode that does not depend on the driving operation of the driver D, the driving support may include the implementation of the automatic driving mode. The predetermined attribute of the driving operation defined by the command value of the support command instructing the driving assistance may be, for example, at least one of the steering angle of the steering wheel of the vehicle 100, the brake amount, and the accelerator amount.

Further, the driving support device 1 acquires a photographed image from a camera 32 arranged so as to photograph an occupant in the vehicle 100. The camera 32 is an example of the "photographing device" of the present invention. In the example of FIG. 1, the camera 32 is arranged so that the driver D can be photographed. However, the arrangement of the camera 32 does not have to be limited to such an example. The occupant to be photographed may be any person existing in the vehicle 100, and may include a driver D who is in the driver's seat and an occupant who is in a seat other than the driver's seat.

Subsequently, the driving support device 1 inputs a captured image into a learned learning device (neural network 6 described later) that has been machine-learned to correct the driving motion according to the situation reflected in the image, and has learned the image. Executes the arithmetic processing of the learner. As a result, the driving support device 1 acquires the correction value for the command value of the support command from the learned learner. The learned learner may be referred to as a "discriminator" or a "classifier".

Then, the driving support device 1 corrects the command value based on the acquired correction value, and controls the driving support of the vehicle 100 based on the support command after the command value is corrected. For example, when the attribute of the driving operation defined by the command value to be corrected is the accelerator amount of the vehicle 100, the driving support device 1 adjusts the accelerator amount of the vehicle 100 by the correction value acquired from the learned learner. Correct (change).

On the other hand, the learning device 2 according to the present embodiment constructs the learning device used in the driving support device 1, that is, the learning device outputs the correction value of the driving operation according to the situation reflected in the input captured image. It is a computer that carries out machine learning. Specifically, the learning device 2 acquires a combination of a captured image (in this embodiment, further including operation information) and a correction value as a learning data set. Of these, the captured image is used as input data (training data), and the correction value is used as teacher data (correct answer data). That is, the learning device 2 trains the learning device (neural network 7 described later) so as to output a correction value for the command value of the support command when the captured image is input.

As a result, the learning device 2 can create a learned learning device to be used in the driving support device 1. The driving support device 1 can acquire the learned learner created by the learning device 2 via the network, for example. The network type may be appropriately selected from, for example, the Internet, a wireless communication network, a mobile communication network, a telephone network, a dedicated network, and the like.

As described above, according to the present embodiment, by using the learned learner, a camera arranged to photograph the occupant with respect to the command value of the support command instructing the driving support of the vehicle 100. It is possible to obtain different correction values according to the individual situations reflected in the captured image obtained from 32. By applying this correction value to the command value of the support command, it is possible to realize a variation of the operation that supports the driving, so that it is possible to save the trouble of inputting the operation suitable for each situation. In addition, in the present embodiment, if a template for an operation that supports driving is prepared, a variation of the operation that supports driving can be realized by applying a correction value to the prepared template. Therefore, it is not necessary to individually generate an operation that supports driving suitable for each situation from scratch, and it is possible to efficiently give variations to the operation that supports the operation. Therefore, according to the present embodiment, it is possible to have variations in the mode of supporting driving in a simple and efficient manner, which is suitable for individual situations caused by individual users, vehicle characteristics, and the like. Driving support can be realized.

§2 Configuration example [Hardware configuration]
<Driving support device>
Next, an example of the hardware configuration of the driving support device 1 according to the present embodiment will be described with reference to FIG. FIG. 2 schematically illustrates an example of the hardware configuration of the driving support device 1 according to the present embodiment.

As shown in FIG. 2, the driving support device 1 according to the present embodiment is a computer to which the control unit 11, the storage unit 12, and the external interface 13 are electrically connected. In FIG. 2, the external interface is described as "external I / F".

The control unit 11 includes a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), and the like, which are hardware processors, and is configured to execute information processing based on a program and various data. NS. The storage unit 12 is an example of a memory, and is composed of, for example, a RAM, a ROM, or the like. In the present embodiment, the storage unit 12 stores various information such as the driving support program 121 and the learning result data 122.

The driving support program 121 is a program for causing the driving support device 1 to execute information processing (FIG. 6) described later that controls driving support of the vehicle 100, and includes a series of instructions for the information processing. The learning result data 122 is data for setting the learned learning device used in the driving support device 1. Details will be described later.

The external interface 13 is an interface for connecting to an external device, and is appropriately configured according to the external device to be connected. In the present embodiment, the external interface 13 is connected to the navigation device 30, the operation unit 31, and the camera 32 via, for example, CAN (Controller Area Network).

The navigation device 30 is a computer that provides route guidance when the vehicle 100 is traveling. A known car navigation device may be used as the navigation device 30. The navigation device 30 is configured to measure the position of the own vehicle based on a GPS (Global Positioning System) signal and provide route guidance by using map information and peripheral information related to surrounding buildings and the like. In the following, information indicating the position of the own vehicle measured based on the GPS signal will be referred to as "GPS information".

The operation unit 31 is, for example, a steering wheel, an accelerator, a brake, an operation panel, or the like, and is used for driving the vehicle 100. The type of the operation unit 31 does not have to be particularly limited, and may be appropriately selected according to the embodiment. By connecting the driving support device 1 to the operation unit 31 via the external interface 13, it is possible to acquire operation information indicating the content of an operation command corresponding to the operation operation of the operation unit 31 by the driver D.

The camera 32 is arranged so as to photograph an occupant in the vehicle 100. In the example of FIG. 1, the camera 32 is arranged above the front of the driver's seat so that at least the upper body of the driver D is the shooting range. However, the location of the camera 32 does not have to be limited to such an example, and if it is possible to photograph an occupant in the vehicle 100 such as a driver D and an occupant other than the driver D, the camera 32 may be arranged. It may be appropriately determined according to the form. A general digital camera, a video camera, or the like may be used as the camera 32. By connecting the driving support device 1 to the camera 32 via the external interface 13, it is possible to acquire a captured image of the situation inside the vehicle including the state of the occupant from the camera 32.

An external device other than the above may be connected to the external interface 13. For example, a communication module for performing data communication via a network may be connected to the external interface 13. The external device connected to the external interface 13 does not have to be limited to each of the above devices, and may be appropriately selected according to the embodiment. Further, in the example of FIG. 2, the driving support device 1 includes one external interface 13. However, the external interface 13 may be provided for each external device to be connected. The number of external interfaces 13 can be appropriately selected according to the embodiment.

Regarding the specific hardware configuration of the driving support device 1, the components can be omitted, replaced, or added as appropriate according to the embodiment. For example, the control unit 11 may include a plurality of hardware processors. The hardware processor may be composed of a microprocessor, an FPGA (field-programmable gate array), an ECU (Electronic Control Unit), or the like. The storage unit 12 may be composed of a RAM and a ROM included in the control unit 11. The storage unit 12 may be composed of an auxiliary storage device such as a hard disk drive or a solid state drive. Further, as the driving support device 1, a general-purpose computer may be used in addition to the information processing device designed exclusively for the provided service.

<Learning device>
Next, an example of the hardware configuration of the learning device 2 according to the present embodiment will be described with reference to FIG. FIG. 3 schematically illustrates an example of the hardware configuration of the learning device 2 according to the present embodiment.

As shown in FIG. 3, the learning device 2 according to the present embodiment is a computer to which the control unit 21, the storage unit 22, the communication interface 23, the input device 24, the output device 25, and the drive 26 are electrically connected. .. In FIG. 3, the communication interface is described as "communication I / F".

Like the control unit 11, the control unit 21 includes a CPU, RAM, ROM, etc., which are hardware processors, and is configured to execute information processing based on a program and various data. The storage unit 22 is composed of, for example, a hard disk drive, a solid state drive, or the like. The storage unit 22 stores various information such as the learning program 221 executed by the control unit 21, the learning data set 222 used for machine learning of the learning device, and the learning result data 122 created by executing the learning program 221. ..

The learning program 221 is a program for causing the learning device 2 to execute a machine learning process (FIG. 7) described later, and to generate learning result data 122 as a result of the machine learning. The learning data set 222 is data used for machine learning for making the learner acquire the ability to determine the correction value of the driving motion according to the situation shown in the image. Details will be described later.

The communication interface 23 is, for example, a wired LAN (Local Area Network) module, a wireless LAN module, or the like, and is an interface for performing wired or wireless communication via a network. The learning device 2 may distribute the created learning result data 122 to an external device via the communication interface 23.

The input device 24 is, for example, a device for inputting a mouse, a keyboard, or the like. The output device 25 is, for example, a device for outputting a display, a speaker, or the like. The operator can operate the learning device 2 via the input device 24 and the output device 25.

The drive 26 is, for example, a CD drive, a DVD drive, or the like, and is a drive device for reading a program stored in the storage medium 92. The type of the drive 26 may be appropriately selected according to the type of the storage medium 92. The learning program 221 and the learning data set 222 may be stored in the storage medium 92.

The storage medium 92 stores the information of the program or the like by electrical, magnetic, optical, mechanical or chemical action so that the information of the program or the like recorded by the computer or other device, the machine or the like can be read. It is a medium to do. The learning device 2 may acquire the learning program 221 and the learning data set 222 from the storage medium 92.

Here, FIG. 3 illustrates a disc-type storage medium such as a CD or DVD as an example of the storage medium 92. However, the type of the storage medium 92 is not limited to the disc type, and may be other than the disc type. Examples of storage media other than the disk type include semiconductor memories such as flash memories.

Regarding the specific hardware configuration of the learning device 2, the components can be omitted, replaced, or added as appropriate according to the embodiment. For example, the control unit 21 may include a plurality of hardware processors. The hardware processor may be composed of a microprocessor, FPGA, ECU and the like. The learning device 2 may be composed of a plurality of information processing devices. Further, as the learning device 2, in addition to an information processing device designed exclusively for the provided service, a general-purpose server device, a PC, or the like may be used.

[Software configuration]
<Driving support device>
Next, an example of the software configuration of the driving support device 1 according to the present embodiment will be described with reference to FIG. FIG. 4 schematically illustrates an example of the software configuration of the driving support device 1 according to the present embodiment.

The control unit 11 of the driving support device 1 expands the driving support program 121 stored in the storage unit 12 into the RAM. Then, the control unit 11 interprets and executes the driving support program 121 expanded in the RAM by the CPU to control each component. As a result, as shown in FIG. 4, the operation support device 1 according to the present embodiment operates as a computer including an operation control unit 111, an image acquisition unit 112, and an operation correction unit 113 as software modules.

The driving control unit 111 controls the driving support of the vehicle 100 based on the support command 50 that instructs the driving support of the vehicle 100 and includes the command value related to a predetermined attribute of the driving operation. The content of the support to be implemented may be appropriately determined according to the embodiment. For example, the driving control unit 111 may acquire the operation information 51 indicating the content of the operation command corresponding to the driving operation of the driver D to the operation unit 31 of the vehicle 100. Then, the operation control unit 111 may determine the type of the support command 50 to be executed based on the content of the operation command indicated by the acquired operation information 51.

The image acquisition unit 112 acquires the captured image 52 from the camera 32 arranged so as to photograph the occupant in the vehicle 100. The motion correction unit 113 includes a trained neural network 6 that has been machine-learned to correct the driving motion according to the situation reflected in the image as a trained learner. The neural network 6 is an example of the "learner" of the present invention. The motion correction unit 113 inputs the operation information 51 and the captured image 52 to the trained neural network 6, and executes the arithmetic processing of the trained neural network 6. As a result, the motion correction unit 113 acquires the correction value 53 with respect to the command value of the support command 50 from the trained neural network 6. The operation control unit 111 corrects the command value of the support command 50 with the acquired correction value, and controls the driving support of the vehicle 100 based on the support command 50 after the command value is corrected.

(Learning device)
Next, the neural network 6 which is an example of the learner will be described. As shown in FIG. 4, the neural network 6 according to the present embodiment is a neural network having a multi-layer structure used for so-called deep learning, and is an input layer 61, an intermediate layer (hidden layer) 62, and an output layer in order from the input. It has 63.

In the example of FIG. 4, the neural network 6 includes a one-layer intermediate layer 62, the output of the input layer 61 is input to the intermediate layer 62, and the output of the intermediate layer 62 is input to the output layer 63. There is. However, the number of intermediate layers 62 is not limited to one, and the neural network 6 may include two or more intermediate layers 62.

Each layer 61-63 comprises one or more neurons. For example, the number of neurons in the input layer 61 can be set according to the input operation information 51 and the captured image 52. The number of neurons in the intermediate layer 62 can be appropriately set according to the embodiment. Further, the number of neurons in the output layer 63 can be set according to the correction value 53.

Neurons in adjacent layers are appropriately connected to each other, and a weight (connection load) is set for each connection based on the result of machine learning. In the example of FIG. 4, each neuron is connected to all neurons in the adjacent layer, but the connection of neurons does not have to be limited to such an example and is appropriately set according to the embodiment. You can. In addition, a threshold is set for each neuron based on the result of machine learning. Basically, the output of each neuron is determined by whether or not the sum of the products of each input and each weight exceeds the threshold value.

The motion correction unit 113 inputs the operation information 51 and the captured image 52 to the input layer 61, and performs firing determination of each neuron included in each layer in order from the input side as arithmetic processing of the neural network 6. As a result, the motion correction unit 113 acquires the correction value 53 with respect to the command value of the support command 50 from the output layer 63.

The configuration of such a neural network 6 (for example, the number of layers in each network, the number of neurons in each layer, the connection relationship between neurons, the transmission function of each neuron), the weight of the connection between each neuron, and each neuron Information indicating the threshold value is included in the learning result data 122. The motion correction unit 113 refers to the learning result data 122, and uses the trained neural network 6 to determine the content of the correction for the driving motion support based on the support command 50 according to the situation reflected in the captured image 52. Set.

<Learning device>
Next, an example of the software configuration of the learning device 2 according to the present embodiment will be described with reference to FIG. FIG. 5 schematically illustrates an example of the software configuration of the learning device 2 according to the present embodiment.

The control unit 21 of the learning device 2 expands the learning program 221 stored in the storage unit 22 into the RAM. Then, the control unit 21 interprets and executes the learning program 221 expanded in the RAM by the CPU to control each component. As a result, as shown in FIG. 5, the learning device 2 according to the present embodiment is configured as a computer including the learning data acquisition unit 211 and the learning processing unit 212 as software modules.

The learning data acquisition unit 211 acquires the combination of the operation information 223, the captured image 224, and the correction value 225 as the learning data set 222. The operation information 223 and the captured image 224 are used as input data (training data), and the correction value 225 is used as teacher data (correct answer data).

The operation information 223 indicates an operation command of the vehicle by the driver. This vehicle may be of the same type as the vehicle 100 or may be of a different type. Further, the captured image 224 is obtained by a photographing device arranged so as to photograph an occupant in the vehicle. This imaging device may be of the same type as the camera 32 or may be of a different type. The correction value 225 is appropriately determined so that the desired driving operation can be corrected according to the situation reflected in the operation information 223 and the captured image 224.

The learning processing unit 212 uses the acquired learning data set 222 to perform machine learning (supervised learning) of the neural network 7. That is, when the operation information 223 and the captured image 224 are input, the learning processing unit 212 performs the learning process of the neural network 7 so as to output the correction value 225 as an output value.

The neural network 7 is an example of a learning device to be learned. The neural network 7 is configured in the same manner as the neural network 6. That is, the neural network 7 includes an input layer 71, an intermediate layer (hidden layer) 72, and an output layer 73. The layers 71 to 73 are configured in the same manner as the layers 61 to 63 of the neural network 6.

When the operation information 223 and the captured image 224 are input to the input layer 71 by the learning process of the neural network, the learning processing unit 212 trains the neural network 7 so that the correction value 225 is output from the output layer 73. As a result, the trained neural network 6 is constructed. Then, the learning processing unit 212 stores information indicating the configuration of the neural network 7 (that is, the learned neural network 6) after learning, the weight of the connection between each neuron, and the threshold value of each neuron as the learning result data 122. It is stored in the unit 22.

That is, in order to make the neural network 7 acquire the ability to determine the correction content of the driving operation according to the driving operation and the situation of the occupant in the vehicle, the learning data set 222 according to the present embodiment includes the operation information 223 and the captured image. It is composed of a combination of 224 and a desired correction value 225 according to the situation reflected in the operation information 223 and the captured image 224. However, the combination of learning data sets for causing the neural network 7 to acquire the ability to determine the correction content of the driving motion according to the situation of the occupant in the vehicle does not have to be limited to such an example. For example, the learning data set may be composed of a combination of the operation information 223 and the captured image 224 and an index for obtaining a desired correction value 225. Further, the index for obtaining the desired correction value 225 may correspond to the situation (for example, the state of a person) appearing in the image.

The combination of the operation information 223 and the captured image 224 and the desired correction value 225 may be appropriately determined according to the embodiment. For example, the learning processing unit 212 may construct a trained neural network 6 so as to correct the driving motion according to the state of the person appearing in the image. In this case, the correction value 225 determined according to the state of the person captured in the captured image 224 is combined with the operation information 223 and the captured image 224. By using such a learning data set 222 for machine learning, the learning processing unit 212 constructs a trained neural network 6 that determines the correction content of the driving motion according to the state of the person in the image. Can be done.

The content (correction value) of the correction associated with the state of the person may be appropriately determined according to the embodiment. For example, the learning processing unit 212 may construct a trained neural network 6 so as to correct the driving motion according to the facial expression of a person appearing in the image. In this case, the correction value 225 determined according to the facial expression of the person captured in the captured image 224 is combined with the operation information 223 and the captured image 224. By using such a learning data set 222 for machine learning, the learning processing unit 212 constructs a trained neural network 6 that determines the correction content of the driving motion according to the facial expression of the person in the image. Can be done.

The content of the correction (correction value) associated with the facial expression of the person may be appropriately determined according to the embodiment. As a specific example, the learning processing unit 212 may construct a trained neural network 6 so as to reduce or cancel the driving support according to the fear of the facial expression of the person in the image. In this case, the correction value 225, which is determined to reduce or cancel the driving assistance, in other words, the correction value 225, which is determined to reduce or cancel the command value of the corresponding support command 50, has a scared expression. It is combined with a photographed image 224 showing a person. By using such a learning data set 222 for machine learning, the learning processing unit 212 makes a correction to reduce or cancel the driving support according to the fear of the facial expression of the person in the image. A trained neural network 6 to be determined can be constructed.

Here, the correction for reducing the support is to weaken the operation of the driving support by the support command. For example, when the accelerator amount of the vehicle is specified by the command value of the support command, the driving support is reduced. By performing the correction, this accelerator amount can be reduced. Further, the correction for canceling the support is to stop the operation of the driving support by the support command. For example, when the automatic driving mode is implemented by the support command, the correction for canceling the driving support is performed. A vehicle running in the automatic driving mode can be stopped.

Further, for example, the learning processing unit 212 may construct a trained neural network 6 so as to correct the driving motion according to the gesture of the person appearing in the image. In this case, the correction value 225 determined according to the gesture of the person appearing in the captured image 224 is combined with the operation information 223 and the captured image 224. By using such a learning data set 222 for machine learning, the learning processing unit 212 constructs a trained neural network 6 that determines the correction content of the driving motion according to the gesture of the person appearing in the image. Can be done.

The correspondence between the gesture of the person and the content of the correction (correction value) may be appropriately determined according to the embodiment. For example, a captured image 224 showing a person signing a stop may be combined with a correction value 225 determined to reduce or cancel driving assistance. In this case, the learning processing unit 212 constructs a trained neural network 6 that determines to reduce or cancel the driving support according to the person appearing in the image signing the stop. Can be done.

As described above, the combination of the operation information 223 and the captured image 224 and the desired correction value 225 may be appropriately determined according to the embodiment. When the operation information 51 and the captured image 52 used for control and the operation information 223 and the captured image 224 used for learning are called separately, the operation information 51 and the captured image 52 are the "first operation information" and the "first", respectively. It may be referred to as "captured image", and the operation information 223 and the captured image 224 may be referred to as "second operation information" and "second captured image", respectively.

<Others>
Each software module of the driving support device 1 and the learning device 2 will be described in detail in an operation example described later. In this embodiment, an example in which each software module of the driving support device 1 and the learning device 2 is realized by a general-purpose CPU is described. However, some or all of the above software modules may be implemented by one or more dedicated processors. Further, with respect to the software configurations of the driving support device 1 and the learning device 2, software modules may be omitted, replaced, or added as appropriate according to the embodiment.

§3 Operation example [Driving support device]
Next, an operation example of the driving support device 1 will be described with reference to FIG. FIG. 6 is a flowchart illustrating an example of the processing procedure of the driving support device 1. The processing procedure described below is an example of the "driving support method" of the present invention. However, the processing procedure described below is only an example, and each processing may be changed as much as possible. Further, with respect to the processing procedure described below, steps can be omitted, replaced, and added as appropriate according to the embodiment.

(Step S101)
In step S101, the control unit 11 operates as the operation control unit 111, monitors the operation of the driver D with respect to the operation unit 31 connected via the external interface 13, and issues an operation command corresponding to the operation. The operation information 51 indicating the content is acquired.

The content of the operation information 51 may be appropriately determined according to the embodiment. For example, the control unit 11 acquires operation information 51 indicating each of the steering wheel steering angle, the brake amount, and the accelerator amount in response to the operation of the steering wheel, the brake pedal, and the accelerator pedal. Further, for example, the control unit 11 acquires the operation information 51 indicating the execution of the automatic operation mode in response to the operation of switching the operation mode from the manual operation mode to the automatic operation mode on the operation panel. When the operation information 51 is acquired, the control unit 11 proceeds to the next step S102.

(Step S102)
In step S102, the control unit 11 operates as the driving control unit 111, and is a support command 50 for instructing the driving support of the vehicle 100, based on the support command 50 including a command value related to a predetermined attribute of the driving operation. , Control the driving support of the vehicle 100. When the control of the driving support is started, the control unit 11 proceeds to the next step S103.

The type of the support command 50 to be executed, that is, the content of the support for the operation to be executed may be appropriately determined according to the embodiment. For example, the driving support device 1 may store support reference information (not shown) such as a table format in which the operation command by the operation unit 31 and the support command 50 are associated with each other in the storage unit 12. In this case, the control unit 11 can determine the support command 50 to be executed in response to the operation command indicated by the operation information 51 acquired in step S101 by referring to the support reference information.

As a specific example, the assist command 50 may instruct to assist the operation of the handlebar, the brake pedal, or the accelerator pedal in response to the operation of the handlebar, the brake pedal, or the accelerator pedal. In this case, the driving operation attribute defined by the command value included in the support command 50 may be, for example, the steering angle of the steering wheel, the amount of braking, or the amount of accelerator. At this time, the command value included in the support command 50, in other words, the steering angle of the steering wheel, the brake amount, or the auxiliary amount of the accelerator amount may be given by the template.

Further, the support command 50 may instruct to execute the automatic operation mode in response to the operation of switching the operation mode from the manual operation mode to the automatic operation mode to the operation panel. In this case, the attributes of the driving operation defined by the command value included in the support command 50 may be, for example, the steering angle of the steering wheel of the vehicle 100, the brake amount, the accelerator amount, and the like during automatic driving. At this time, the command value included in the support command 50, in other words, the steering angle of the steering wheel of the vehicle 100, the brake amount, the accelerator amount, and the like during automatic driving may be given by the template. The steering angle of the steering wheel, the amount of braking, and the amount of accelerator are secondary indicators such as the traveling speed of the vehicle 100, the distance between another vehicle and the vehicle 100, and the traveling lane, which are obtained as a result of these combinations. It may be determined based on. In this case, the command value may be given to a secondary index such as a traveling speed, an inter-vehicle distance, or a traveling lane.

(Step S103)
In step S103, the control unit 11 operates as the image acquisition unit 112 and acquires the captured image 52 from the camera 32 arranged so as to photograph the occupant in the vehicle 100. The captured image 52 may be a moving image or a still image.

In the present embodiment, the driving support device 1 is connected to the camera 32 via the external interface 13. Therefore, the control unit 11 acquires the captured image 52 from the camera 32 via the external interface 13. When the captured image 52 is acquired, the control unit 11 proceeds to the next step S104.

However, the route for acquiring the captured image 52 does not have to be limited to such an example, and may be appropriately selected according to the embodiment. For example, another information processing device different from the driving support device 1 may be connected to the camera 32. In this case, the driving support device 1 may acquire the captured image 52 by accepting the transmission of the captured image 52 from another information processing device.

(Steps S104 and S105)
In step S104, the control unit 11 operates as the motion correction unit 113, inputs the captured image 52 to the trained neural network 6, and executes the arithmetic processing of the trained neural network 6. As a result, in step S105, the control unit 11 acquires the correction value 53 with respect to the command value of the support command 50 from the trained neural network 6.

Specifically, the control unit 11 inputs the captured image 52 acquired in step S103 to the input layer 61 of the trained neural network 6. Then, the control unit 11 determines the firing of each neuron included in each layer 61 to 63 in order from the input side. As a result, the control unit 11 acquires the output value corresponding to the correction value 53 from the output layer 63.

The format of the output value obtained from the output layer 63 may be appropriately selected according to the embodiment. For example, the output value obtained from the output layer 63 may be used as it is as the correction value 53. In this case, the control unit 11 can directly acquire the correction value 53 from the output layer 63 of the trained neural network 6. Further, for example, the output value obtained from the output layer 63 may indicate an index for determining the correction value 53. In this case, the driving support device 1 may store correction reference information (not shown) such as a table format in which the output value obtained from the output layer 63 and the correction value 53 are associated with each other in the storage unit 12. In step S105, the control unit 11 can determine the correction value 53 corresponding to the output value obtained from the output layer 63 by referring to the correction reference information. The index may correspond to individual situations such as, for example, the degree of drowsiness of the driver. When the correction value 53 is acquired, the control unit 11 proceeds to the next step S106.

(Steps S106 and S107)
In step S106, the control unit 11 operates as the operation control unit 111, and corrects the command value included in the running support command 50 by the correction value 53 acquired in step S105. Then, in step S107, the control unit 11 controls the driving support of the vehicle 100 based on the support command 50 after correcting the command value.

That is, the control unit 11 changes the steering angle of the steering wheel, the brake amount, the accelerator amount, and the like defined by the command value of the support command 50 by using the correction value 53 acquired in step S105. As a result, the control unit 11 corrects the content of the driving operation support by the support command 50 according to the situation reflected in the captured image 52.

For example, as described above, the trained neural network 6 may be constructed so as to correct the driving motion according to the state of the person appearing in the image. In this case, in step S105, the control unit 11 can acquire the correction value 53 determined according to the state of the person appearing in the captured image 52 acquired in step S103 from the trained neural network 6. As a result, in steps S106 and S107, the content of the driving operation support by the support command 50 can be modified according to the state of the person captured in the captured image 52.

Further, for example, the trained neural network 6 may be constructed so as to correct the driving motion according to the facial expression of a person appearing in the image. In this case, in step S105, the control unit 11 can acquire the correction value 53 determined according to the facial expression of the person captured in the captured image 52 acquired in step S103 from the trained neural network 6. As a result, in steps S106 and S107, the content of the driving operation support by the support command 50 can be modified according to the facial expression of the person reflected in the captured image 52.

As a specific example, the trained neural network 6 may be constructed to reduce or cancel the driving support according to the fear of the facial expression of the person in the image. In this case, in step S105, the control unit 11 reduces or cancels the driving support according to the person in the captured image 52 acquired in step S103 having a scared facial expression, in other words, the support command. It is possible to obtain a correction value 53 that reduces or cancels the command value of 50. As a result, in steps S106 and S107, the support for the driving operation by the support command 50 can be reduced or canceled depending on the fear of the facial expression of the person in the captured image 52. For example, when the accelerator amount of the vehicle 100 is defined by the command value of the support command 50, the control unit 11 reduces the accelerator amount of the vehicle 100 or stops the vehicle 100 in steps S106 and S107. can do.

Further, for example, the trained neural network 6 may be constructed so as to correct the driving motion according to the gesture of the person appearing in the image. In this case, in step S105, the control unit 11 can acquire the correction value 53 determined according to the gesture of the person appearing in the captured image 52 acquired in step S103 from the trained neural network 6. As a result, in steps S106 and S107, the content of the driving operation support by the support command 50 can be modified according to the gesture of the person shown in the captured image 52.

As a result, when the content of the support for the driving operation is modified, the control unit 11 ends the process related to this operation example. After that, the control unit 11 may continuously control the driving support of the vehicle 100 by repeating a series of processes from step S101. Further, the control unit 11 may repeatedly correct the command value of the support command 50 by repeating a series of processes from step S103.

[Learning device]
Next, an operation example of the learning device 2 will be described with reference to FIG. 7. FIG. 7 is a flowchart illustrating an example of the processing procedure of the learning device 2. The processing procedure related to machine learning of the learning device described below is an example of a learning method using a computer. However, the processing procedure described below is only an example, and each processing may be changed as much as possible. Further, with respect to the processing procedure described below, steps can be omitted, replaced, and added as appropriate according to the embodiment.

(Step S201)
In step S201, the control unit 21 operates as the learning data acquisition unit 211, and acquires the combination of the operation information 223, the captured image 224, and the correction value 225 as the learning data set 222.

The learning data set 222 is used for machine learning to make the neural network 7 acquire the ability to determine the content of the correction of the driving motion according to the situation shown in the image. Such a learning data set 222 may be appropriately generated according to the embodiment. For example, a vehicle equipped with a camera is prepared, the driver is made to operate the driving, and the driving support is provided by the support command, and the occupant in the vehicle is photographed under various conditions. Then, the obtained operation information 223 and the captured image 224 are combined with a desired correction value 225 according to the situation reflected in the captured image 224. As a result, the training data set 222 can be generated.

The generation of the learning data set 222 may be performed by the learning device 2. In this case, the control unit 21 may generate the learning data set 222 in response to the operation of the input device 24 by the operator. Further, the control unit 21 may appropriately determine the correction value 225 for the obtained operation information 223 and the captured image 224 by the processing of the learning program 221 and automatically generate the learning data set 222. By executing this generation process, in this step S201, the control unit 21 can acquire the learning data set 222.

Further, the learning data set 222 may be generated by an information processing device other than the learning device 2. In other information processing devices, the learning data set 222 may be manually generated by the operator or automatically generated by the processing of the program. In this case, in this step S201, the control unit 21 may acquire the learning data set 222 generated by another information processing device via the network, the storage medium 92, or the like.

The number of learning data sets 222 acquired in this step S201 may be appropriately determined according to the embodiment, and may be appropriately determined, for example, to the extent that machine learning of the neural network 7 can be performed. As a result, when the learning data set 222 is acquired, the control unit 21 proceeds to the next step S202.

As described above, the combination of the operation information 223 and the captured image 224 in the learning data set 222 and the desired correction value 225 may be appropriately determined according to the embodiment.

For example, the correction value 225 determined according to the state of the person appearing in the captured image 224 may be combined with the operation information 223 and the captured image 224. Further, for example, the correction value 225 determined according to the facial expression of the person captured in the captured image 224 may be combined with the operation information 223 and the captured image 224. As a specific example, the correction value 225 determined to reduce or cancel the driving assistance according to the fear of the facial expression of the person reflected in the captured image 224 is combined with the operation information 223 and the captured image 224. You may. Further, for example, the correction value 225 determined according to the gesture of the person appearing in the captured image 224 may be combined with the operation information 223 and the captured image 224.

(Step S202)
In step S202, the control unit 21 operates as the learning processing unit 212, and when the operation information 223 and the captured image 224 are input to the input layer 71 using the learning data set 222 acquired in step S201, it is corrected as an output value. Machine learning of the neural network 7 is performed so that the value 225 is output from the output layer 73.

Specifically, first, the control unit 21 prepares a neural network 7 to be subjected to learning processing. The configuration of the neural network 7 to be prepared, the initial value of the weight of the connection between each neuron, and the initial value of the threshold value of each neuron may be given by the template or by the input of the operator. Further, when performing re-learning, the control unit 21 may prepare the neural network 7 based on the learning result data 122 to be re-learned.

Next, the control unit 21 uses the operation information 223 and the captured image 224 included in the learning data set 222 acquired in step S201 as input data, and uses the correction value 225 as the teacher data to perform the learning process of the neural network 7. To execute. A stochastic gradient descent method or the like may be used for the learning process of the neural network 7.

For example, the control unit 21 inputs the operation information 223 and the captured image 224 to the input layer 71, and determines the firing of each neuron included in each layer 71 to 73 in order from the input side. As a result, the control unit 21 obtains an output value from the output layer 73. Next, the control unit 21 calculates an error between the output value obtained from the output layer 73 and the correction value 225. Subsequently, the control unit 21 calculates the error of the connection weight between each neuron and the error of each threshold value of each neuron by using the error of the output value calculated by the error back propagation method. Then, the control unit 21 updates the weight of the connection between each neuron and the value of each threshold value of each neuron based on each calculated error.

The control unit 21 repeats this series of processes for each learning data set 222 until the output value output from the neural network 7 matches the correction value 225. As a result, the control unit 21 can construct a neural network 7 (that is, a trained neural network 6) that outputs a corresponding correction value 225 when the operation information 223 and the captured image 224 are input.

For example, it is assumed that a learning data set 222 in which the correction value 225 determined according to the state of the person captured in the captured image 224, the operation information 223, and the captured image 224 are combined is used. In this case, according to this step S202, the control unit 21 can construct a neural network 7 that determines the correction content of the driving operation according to the state of the person appearing in the image.

Further, for example, it is assumed that a learning data set 222 in which the correction value 225 determined according to the facial expression of the person captured in the captured image 224, the operation information 223, and the captured image 224 are combined is used. In this case, according to this step S202, the control unit 21 can construct a neural network 7 that determines the correction content of the driving operation according to the facial expression of the person in the image.

As a specific example, it is assumed that a learning data set 222 is used in which a photographed image 224 showing a person with a scared expression and a correction value 225 determined to reduce or cancel driving assistance are combined. do. In this case, in this step S202, the control unit 21 constructs a neural network 7 that determines to reduce or cancel the driving support according to the fear of the facial expression of the person in the image. be able to.

Further, for example, it is assumed that a learning data set 222 is used in which the correction value 225 determined according to the gesture of the person captured in the captured image 224, the operation information 223, and the captured image 224 are combined. In this case, according to this step S202, the control unit 21 can construct a neural network 7 that determines the correction content of the driving motion according to the gesture of the person appearing in the image. When the learning process of the neural network 7 is completed, the control unit 21 proceeds to the next step S203.

(Step S203)
In step S203, the control unit 21 operates as the learning processing unit 212, and uses information indicating the configuration of the neural network 7 after machine learning, the weight of the connection between each neuron, and the threshold value of each neuron as learning result data 122. It is stored in the storage unit 22. As a result, the control unit 21 ends the learning process of the neural network 7 according to this operation example.

The control unit 21 may transfer the created learning result data 122 to the driving support device 1 after the process of step S203 is completed. Further, the control unit 21 may periodically update the learning result data 122 by periodically executing the learning processes of steps S201 to S203. Then, the control unit 21 may periodically update the learning result data 122 held by the driving support device 1 by transferring the created learning result data 122 to the driving support device 1 each time the learning process is executed. .. Further, for example, the control unit 21 may store the created learning result data 122 in a data server such as NAS (Network Attached Storage). In this case, the driving support device 1 may acquire the learning result data 122 from this data server. Further, the learning result data 122 created by the learning device 2 may be incorporated in the driving support device 1 in advance.

[feature]
As described above, the driving support device 1 according to the present embodiment acquires the operation information 51 and the captured image 52 by the processing of steps S101 and S103. Next, in steps S104 and S105, the driving support device 1 inputs the operation information 51 and the captured image 52 into the trained neural network 6, and the correction value is determined according to the situation reflected in the captured image 52. Get 53. The trained neural network 6 is generated by the learning device 2 by machine learning (supervised learning) using a learning data set 222 composed of a combination of operation information 223, captured image 224, and correction value 225. Will be done. Then, the driving support device 1 corrects the content of the driving operation by using the correction value 53 acquired in the steps S106 and S107.

Therefore, according to the present embodiment, by using the learned neural network, the camera 32 arranged to photograph the occupant with respect to the command value of the support command instructing the driving support of the vehicle 100 It is possible to obtain different correction values 53 according to the individual situations captured in the obtained captured image 52. As a result, in steps S106 and S107, by applying the correction value 53 to the command value of the support command 50, it is possible to realize a variation of the operation that supports the operation, so that the operation suitable for each situation is input. It is possible to save the trouble of performing the above. In addition, in the present embodiment, if the command value of the support command 50 is given by the template, by applying the correction value 53 to the support command 50, it is possible to realize a variation of the operation of supporting the driving. Therefore, it is not necessary to individually prepare the support command 50 that supports the driving suitable for each situation, and the operation that supports the driving can be efficiently varied. Therefore, according to the present embodiment, it is possible to provide variations in the method of supporting driving in a simple and efficient manner, which is suitable for individual situations caused by individual users, vehicle characteristics, and the like. Driving support can be realized.

§4 Modifications Although the embodiments of the present invention have been described in detail above, the above description is merely an example of the present invention in all respects. Needless to say, various improvements and modifications can be made without departing from the scope of the present invention. For example, the following changes can be made. In the following, the same reference numerals will be used for the same components as those in the above embodiment, and the same points as in the above embodiment will be omitted as appropriate. The following modifications can be combined as appropriate.

<4.1>
In the above embodiment, each neural network (6, 7) is a fully connected neural network. However, the configuration and type of each neural network (6, 7) need not be limited to such an example, and may be appropriately selected according to the embodiment. For example, each neural network (6, 7) may be composed of a convolutional neural network, a recurrent neural network, or the like.

<4.2>
In the above embodiment, a neural network is used as an example of the learner. However, the type of the learning device is not limited to the neural network as long as the captured image can be used as an input, and may be appropriately selected according to the embodiment. Examples of the learning device that can be used include a support vector machine, a self-organizing map, and a learning device that performs machine learning by reinforcement learning.

<4.3>
In the above embodiment, the training result data 122 includes information indicating the configuration of the trained neural network 6. However, the configuration of the learning result data 122 does not have to be limited to such an example, and may be appropriately determined according to the embodiment as long as it can be used for setting the learned learning device. For example, when the configuration of the neural network to be used is common to each device, the learning result data 122 does not have to include information indicating the configuration of the trained neural network 6.

<4.4>
In the above embodiment, the learning data set 222 is composed of a combination of the operation information 223, the captured image 224, and the correction value 225, and the operation information 51 is input to the trained neural network 6 together with the captured image 52. ing. However, the input of the neural network does not have to be limited to such an example, and may be appropriately determined according to the embodiment.

For example, the operation information 51 may be omitted. In this case, the learning data set 222 may be composed of a combination of the captured image 224 and the correction value 225, and only the captured image 52 may be input to the trained neural network 6. Further, for example, the support command 50 may be input instead of the operation information 51. In this case, the learning data set 222 may be composed of a support command and a combination of the captured image 224 and the correction value 225, and the support command 50 may be input to the trained neural network 6 together with the captured image 52.

<4.5>
In the above embodiment, an example in which the present invention is applied to a situation where the driving of the vehicle 100 is supported is shown. However, the scope of application of the present invention does not have to be limited to the situation where the driving of the vehicle 100 is supported. The present invention can be widely applied to all situations where the operation of the target device is controlled.

FIG. 8 schematically illustrates an example of another situation to which the present invention is applied. The motion control device 1A according to this modification is configured to control the motion of the target device RA used by the user UA. In the example of FIG. 8, the user UA is a worker on the production line, and the target device RA is an industrial robot that cooperates with the user UA on the production line. However, the target device RA does not have to be limited to such an example, and may be appropriately selected according to the embodiment. When the user UA is a worker on the production line, the target device RA may be, for example, a device provided on the production line such as a belt conveyor.

9 and 10 schematically show an example of the hardware configuration and software configuration of the operation control device 1A according to this modification. The operation control device 1A may be configured in the same manner as the driving support device 1 except that the processing target is replaced by the target device RA from the vehicle 100. That is, as shown in FIG. 9, the hardware configuration of the operation control device 1A may be the same as the hardware configuration of the operation support device 1. Further, as shown in FIG. 10, the software configuration of the operation control device 1A may be the same as the software configuration of the operation support device 1 except that the operation control unit 111 is replaced with the operation control unit 111A.

Specifically, as shown in FIG. 9, the storage unit 12 of the motion control device 1A stores various information such as the motion control program 121A and the learning result data 122A. The operation control program 121A is a program for causing the operation control device 1A to execute information processing for controlling the operation of the target device RA by the same processing procedure as the operation support device 1, and issues a series of instructions for the information processing. include. The learning result data 122A is data for setting a learned learner (learned neural network 6A) that has been machine-learned to correct the operation of the target device RA according to the situation shown in the image. .. Further, the motion control device 1A is connected to the target device RA and the camera 32 to be controlled via the external interface 13. The camera 32 is appropriately arranged at a place where the user UA of the target device RA can be photographed.

Further, as shown in FIG. 10, the motion control device 1A is a computer including the motion control unit 111A, the image acquisition unit 112, and the motion correction unit 113 as software modules by executing the motion control program 121A by the control unit 11. Works as. The operation control unit 113A controls the operation of the target device RA based on the instruction command 50A that instructs the target device RA to perform a predetermined operation and includes a command value related to a predetermined attribute of the operation.

Further, the motion correction unit 113 according to this modification includes the trained neural network 6A. The motion correction unit 113 inputs the captured image 52 into the trained neural network 6A that has been machine-learned to correct the motion of the target device RA according to the situation reflected in the image, and calculates the trained neural network 6A. By executing the process, the correction value for the command value of the instruction command 50A is acquired from the trained neural network 6A. The operation control unit 111A corrects the command value based on the acquired correction value, and controls the operation of the target device RA based on the instruction command 50A after the command value is corrected.

The trained neural network 6A is appropriately constructed so as to output a correction value (or an index for acquiring the correction value) according to the situation reflected in the input captured image when the captured image is input. .. The learning device 2 can construct the trained neural network 6A in the same processing procedure as in the above embodiment (processing in steps S201 to S203). Further, the learning device 2 stores information indicating the configuration of the constructed learned neural network 6A, the weight of the connection between each neuron, and the threshold value of each neuron as the learning result data 122A. The motion correction unit 113 sets the trained neural network 6A with reference to the learning result data 122A.

The motion control device 1A according to this modification controls the motion of the target device RA by a processing procedure substantially the same as that of the operation support device 1. Specifically, in step S102, the control unit 11 operates as the operation control unit 111A, is an instruction command 50A that instructs the target device RA to perform a predetermined operation, and includes a command value related to a predetermined attribute of the operation. The operation of the target device RA is controlled based on the instruction command 50A. The operation attribute defined by the command value may include any attribute of the operation of the target device RA. As illustrated in FIG. 8, when the target device RA is an industrial robot, the attribute of the operation defined by the command value may be the driving amount of the motor of the industrial robot or the like.

In step S103, the control unit 11 operates as the image acquisition unit 112 and acquires the captured image 52 from the camera 32 arranged to capture the user UA using the target device RA. In step S104, the control unit 11 operates as the motion correction unit 113, inputs the captured image 52 into the trained neural network 6A that has been machine-learned to correct the motion according to the situation reflected in the image, and learns. The arithmetic processing of the completed neural network 6A is executed. As a result, in step S105, the control unit 11 acquires the correction value for the command value of the instruction command 50A from the trained neural network 6A.

Similar to the above embodiment, the trained neural network 6A may be constructed so as to correct the operation of the target device RA according to the state of the person appearing in the image. The trained neural network 6A may be constructed so as to correct the operation of the target device RA according to the facial expression of the person appearing in the image. The trained neural network 6A may be constructed to reduce or cancel the operation of the target device RA depending on the fear of the facial expression of the person in the image. The trained neural network 6A may be constructed so as to correct the operation of the target device RA according to the gesture of the person appearing in the image. The learning device 2 can construct the trained neural network 6A of each form by machine learning as in the above embodiment.

In step S106, the control unit 11 operates as the operation control unit 111A, and corrects the command value of the instruction command 50A by the correction value acquired in step S105. Then, in step S107, the control unit 11 operates as the operation control unit 111A, and controls the operation of the target device RA based on the instruction command 50A after correcting the command value. As a result, it is possible to realize variations in the operation of the target device RA according to the situation captured in the captured image 52. For example, when the user UA (worker) makes a scared expression, the motion control device 1A can make a correction to reduce or cancel the motion of the target device RA (industrial robot).

1 ... Driving support device,
11 ... Control unit, 12 ... Storage unit, 13 ... External interface,
111 ... Operation control unit, 112 ... Image acquisition unit,
113 ... Motion correction unit,
121 ... Driving support program, 122 ... Learning result data,
2 ... Learning device,
21 ... Control unit, 22 ... Storage unit, 23 ... Communication interface,
24 ... Input device, 25 ... Output device, 26 ... Drive,
211 ... Learning data acquisition unit, 212 ... Learning processing unit,
221 ... Learning program, 222 ... Learning data,
92 ... Storage medium,
30 ... Navigation device,
31 ... Operation unit, 32 ... Camera,
50 ... Support command,
51 ... Operation information, 52 ... Captured image,
53 ... Correction value,
6 ... Neural network (learned learner),
61 ... Input layer, 62 ... Intermediate layer (hidden layer), 63 ... Output layer,
7 ... Neural network,
71 ... Input layer, 72 ... Intermediate layer (hidden layer), 73 ... Output layer

Claims (12)

A support command for instructing support for driving a vehicle, which is a driving control unit that controls support for driving the vehicle based on a support command including a command value related to a predetermined attribute of the driving operation, and the command value. Is given by the template, with the operation control unit ,
An image acquisition unit that acquires a photographed image from an imaging device arranged so as to photograph an occupant in the vehicle, and an image acquisition unit.
The support command is obtained by inputting the captured image into a learned learner that has undergone machine learning to correct the driving motion according to the situation shown in the image and executing arithmetic processing of the learned learner. The motion correction unit that acquires the correction value for the command value of the above from the learned learner, and
With
The driving control unit corrects the command value with the acquired correction value, and controls the driving support of the vehicle based on the support command after the command value is corrected.
Driving support device. The learned learner is constructed so as to correct the driving motion according to the state of the person in the image by the machine learning.
The driving support device according to claim 1. The learned learner is constructed so as to correct the driving motion according to the facial expression of the person in the image by the machine learning.
The driving support device according to claim 2. The learned learner is constructed so as to reduce or cancel the driving support according to the fear of the facial expression of the person in the image by the machine learning.
The driving support device according to claim 3. The learned learner is constructed so as to correct the driving motion according to the gesture of the person shown in the image by the machine learning.
The driving support device according to claim 2. The motion correction unit further inputs operation information indicating an operation command of the vehicle by the driver to the learned learner, and executes arithmetic processing of the learned learner to perform the learned learning. Obtaining the correction value from the device,
The driving support device according to any one of claims 1 to 5. The predetermined attribute of the driving operation is at least one of the steering angle of the steering wheel, the braking amount, and the accelerator amount.
The driving support device according to any one of claims 1 to 6. The computer
It is a support command for instructing the driving support of the vehicle, and is a step of controlling the driving support of the vehicle based on the support command including the command value related to a predetermined attribute of the driving operation, and the command value is a template. Given by the steps and
A step of acquiring a photographed image from a photographing device arranged so as to photograph an occupant in the vehicle, and a step of acquiring a photographed image.
The support command is obtained by inputting the captured image into a learned learner that has undergone machine learning to correct the driving motion according to the situation shown in the image and executing arithmetic processing of the learned learner. The step of acquiring the correction value for the command value of the above from the learned learner, and
A step of correcting the command value with the acquired correction value, and
A step of controlling the driving support of the vehicle based on the support command after correcting the command value, and
To execute,
Driving support method. On the computer
It is a support command for instructing the driving support of the vehicle, and is a step of controlling the driving support of the vehicle based on the support command including the command value related to a predetermined attribute of the driving operation, and the command value is a template. Given by the steps and
A step of acquiring a photographed image from a photographing device arranged so as to photograph an occupant in the vehicle, and a step of acquiring a photographed image.
The support command is obtained by inputting the captured image into a learned learner that has undergone machine learning to correct the driving motion according to the situation shown in the image and executing arithmetic processing of the learned learner. The step of acquiring the correction value for the command value of the above from the learned learner, and
A step of correcting the command value with the acquired correction value, and
A step of controlling the driving support of the vehicle based on the support command after correcting the command value, and
To execute,
Driving support program. A instruction command for instructing a predetermined operation to the target device, based on an instruction command containing the command value for a given attribute of the operation, a motion control section that controls operation of the target device, the command value Is given by the template, with the motion control unit ,
An image acquisition unit that acquires a captured image from a photographing device arranged so as to photograph a user who uses the target device, and an image acquisition unit.
The instruction command is obtained by inputting the captured image into a learned learner that has undergone machine learning to correct the movement according to the situation reflected in the image and executing arithmetic processing of the learned learner. The motion correction unit that acquires the correction value for the command value of
With
The operation control unit corrects the command value with the acquired correction value, and controls the operation of the target device based on the instruction command after the command value is corrected.
Operation control device. The computer
An instruction command for instructing a target device to perform a predetermined operation, which is a step of controlling the operation of the target device based on an instruction command including a command value related to a predetermined attribute of the operation, and the command value is a template. Given by the steps and
A step of acquiring a captured image from a photographing device arranged so as to photograph a user who uses the target device, and
The instruction command is obtained by inputting the captured image into a learned learner that has undergone machine learning to correct the movement according to the situation reflected in the image and executing arithmetic processing of the learned learner. And the step of acquiring the correction value for the command value of
A step of correcting the command value with the acquired correction value, and
A step of controlling the operation of the target device based on the instruction command after correcting the command value, and
To execute,
Operation control method. On the computer
An instruction command for instructing a target device to perform a predetermined operation, which is a step of controlling the operation of the target device based on an instruction command including a command value related to a predetermined attribute of the operation, and the command value is a template. Given by the steps and
A step of acquiring a captured image from a photographing device arranged so as to photograph a user who uses the target device, and
The instruction command is obtained by inputting the captured image into a learned learner that has undergone machine learning to correct the movement according to the situation reflected in the image and executing arithmetic processing of the learned learner. And the step of acquiring the correction value for the command value of
A step of correcting the command value with the acquired correction value, and
A step of controlling the operation of the target device based on the instruction command after correcting the command value, and
To execute,
Operation control program.

Images