WO2018163474A1

WO2018163474A1 - Drive mode switching control device, method and program

Info

Publication number: WO2018163474A1
Application number: PCT/JP2017/033149
Authority: WO
Inventors: 初美青位; 相澤　知禎; 岡地　一喜; 啓菅原; 充恵鵜野; 光司滝沢
Original assignee: オムロン株式会社
Priority date: 2017-03-09
Filing date: 2017-09-13
Publication date: 2018-09-13
Also published as: JP2018149822A; DE112017007208T5

Abstract

The purpose of this invention is to ensure that a driver has adequate time to prepare when switching from self-driving mode to manual driving mode, even if a change in driver occurs, and this invention adopts the following measures to achieve this purpose. That is to say, during an operation control period in self-driving mode, a preparatory period control unit (613) sets and counts a first preparatory period if the driver is deemed to be in a state in which he or she is capable of driving operation in manual driving mode, and a preparatory period information generating unit (614) generates first preparatory period information for notifying the driver of the time remaining in a first preparatory period. A driver state assessment unit (612) assesses whether a change in driver has occurred in the first preparatory period, and if a change is deemed to have occurred, the preparatory period control unit sets and counts a second preparatory period, and the preparatory period information generating unit generates second preparatory period information for notifying the time remaining in the second preparatory period.

Description

Operation mode switching control device, method and program

The present invention relates to an operation mode switching control device, method, and program for switching a vehicle operation mode between a manual operation mode and an automatic operation mode.

In recent years, as a driving mode of a vehicle, in addition to a manual driving mode in which the vehicle is driven based on a driving operation of the driver, an automatic driving mode in which the vehicle is driven along a predetermined route without depending on the driving operation of the driver. Development is underway. The automatic driving mode includes, for example, navigation system information using GPS (Global Positioning System), traffic information acquired by road-to-vehicle communication, and information on a surrounding monitoring system that monitors the position and movement of surrounding people and vehicles. Originally, the vehicle can be automatically driven by controlling a power unit, a steering device, a brake, and the like (see, for example, Patent Document 1).

By the way, although the automatic driving mode can be expected to reduce the burden on the driving operation of the driver and alleviate traffic congestion, it is practically difficult to automate all the driving operations from the start to the end of driving. For this reason, it is required to appropriately switch from the automatic operation mode to the manual operation mode. When switching the operation mode from the automatic operation mode to the manual operation mode, confirm that the driver is ready for manual operation prior to switching and ask the driver to prepare for manual operation. On the other hand, it is necessary to inform that the operation mode will be switched after a certain period of time (a period set for the driver to prepare when switching from the automatic operation mode to the manual operation mode, hereinafter simply referred to as “preparation period”). is there.

In the automatic operation mode, operation operations such as steering wheel operation and pedal operation are not necessary in principle, whereas in the manual operation mode, such operation operation is necessary. For this reason, during the preparatory period for switching from the automatic operation mode to the manual operation mode, it is expected that a driver change occurs such that the driver changes from a person not good at driving operation to a person good at driving operation.

However, the preparation period when switching from the automatic operation mode to the manual operation mode is set as a preparation period for one driver to perform manual operation from now on. Therefore, when the above change is made, the two drivers share the preparation period, and the preparation period may not be sufficient for the driver in charge of manual driving.

The present invention has been made paying attention to the above circumstances, and ensures a sufficient driver preparation period even when a driver change occurs when switching from the automatic operation mode to the manual operation mode. It is an object of the present invention to provide an operation mode switching control device, method and program capable of performing the above.

In order to solve the above-described problem, a first aspect of the present invention is an operation mode switching control device that performs switching control of a vehicle operation mode between a manual operation mode and an automatic operation mode. A first preparation period is set when switching to the manual operation mode, a preparation period control unit that counts the first preparation period, and first preparation period information for informing the remaining time of the first preparation period A preparation period information generation unit that generates the output, an output unit that outputs the first preparation period information, and a determination unit that determines whether or not a driver change has occurred in the first preparation period. It has. When the determination unit determines that a driver change has occurred, the preparation period control unit sets a second preparation period and counts the second preparation period, and the preparation period The information generation unit generates second preparation period information for informing the remaining time of the second preparation period, and the output unit outputs the second preparation period information.

According to the first aspect of the present invention, when the automatic operation mode is switched to the manual operation mode, the preparation period is set and counted by the preparation period control unit, and the first time for informing the remaining time of the preparation period The preparation period information is generated by the preparation period information generation unit and output from a predetermined output unit. When the determination unit determines that a driver change has occurred, the preparation period control unit sets and counts a new second preparation period and informs the remaining time of the second preparation period. Second preparation period information is generated by the preparation period information generation unit and output from a predetermined output unit. Therefore, it is possible to prevent the situation where two drivers share the limited preparation period when switching from the automatic operation mode to the manual operation mode, and a certain preparation period is always provided for the driver after the change. Can be secured. Thereby, the new driver after the change can make sufficient preparations for the manual driving operation and can perform the manual driving with a psychological margin. As a result, a safe and comfortable driving environment can be provided.

According to a second aspect of the present invention, video data about a driver acquired by the monitoring camera before the counting unit starts counting the first preparation period and the monitoring camera in the first preparation period are used. The similarity is calculated using the acquired video data about the driver, and the determination as to whether or not a driver change has occurred is executed based on the calculated similarity. is there.

According to the second aspect of the present invention, the determination unit acquires the video data about the driver acquired by the monitoring camera before starting the counting of the first preparation period and the monitoring camera in the first preparation period. The similarity is calculated using the image data of the driver, and a determination is made as to whether or not a driver change has occurred based on the calculated similarity. Therefore, based on the unique information such as the driver's face, it can be reliably determined whether or not a driver change has occurred before and after the start of the latest preparation period.

According to a third aspect of the present invention, the determination unit determines whether or not a driver change has occurred in the second preparation period, and the determination unit determines whether or not a driver change has occurred in the second preparation period. When it is determined that the occurrence has occurred, the preparation period control unit sets a third preparation period, counts the third preparation period, and the preparation period information generation unit determines the remaining of the third preparation period. Preparation period information for informing time is generated, and the output unit outputs the third preparation period information.

According to the third aspect of the present invention, when a driver change occurs in the second preparation period, a new third preparation period is set and the third preparation period is counted. Preparation period information for informing the remaining time of the new third preparation period is generated, and the third preparation period information is output from a predetermined output unit. As a result, it is possible to determine whether or not a driver change has occurred even during the newly set preparation period, and every time the driver changes until the automatic operation mode is switched to the manual operation mode. It is possible to reliably set a new preparation period for a new driver.

According to a fourth aspect of the present invention, video data about a driver acquired by the monitoring camera in the first preparation period and a driver acquired by the monitoring camera in the second preparation period by the determination unit. Similarity is calculated using video data for and the determination as to whether or not a driver change has occurred in the second preparation period is performed based on the calculated similarity. Is.

According to the fourth aspect of the present invention, the video data about the driver acquired by the monitoring camera in the first preparation period and the driver acquired by the monitoring camera in the second preparation period by the determination unit. The degree of similarity is calculated using the video data and a determination is made as to whether or not a driver change has occurred based on the calculated degree of similarity. Therefore, based on the unique information such as the driver's face, it can be reliably determined whether or not a driver change has occurred before and after the start of the latest preparation period.

According to a fifth aspect of the present invention, there is further provided an operation mode switching signal output unit that outputs a signal for switching the automatic operation mode to the manual operation mode when the latest preparation period has elapsed. is there.

According to the fifth aspect of the present invention, the operation mode switching signal output unit outputs a signal for switching the automatic operation mode to the manual operation mode when the latest preparation period has elapsed. For this reason, it is possible to switch the automatic operation mode to the manual operation mode after reliably providing a certain preparation period to the new driver after the change.

In the sixth aspect of the present invention, the length of the preparation period newly set according to a predetermined standard is adjusted by the preparation period information generation unit.

According to the sixth aspect of the present invention, the newly set preparation period can be adjusted to a predetermined length according to the current driver state, and the appropriate preparation period can be set flexibly. Can improve safety.

According to a seventh aspect of the present invention, the preparation period information generation unit further generates information notifying that the driver change has been confirmed, and the output unit outputs information notifying that the driver change has been confirmed. It is what you do.

According to the seventh aspect of the present invention, information notifying that the change of the driver has been confirmed is generated and output from the predetermined output unit. Accordingly, the driver quickly and easily grasps that the preparation period is newly set and the remaining time of the preparation period, and that the preparation period is newly set is due to the change of the driver. be able to.

That is, according to each aspect of the present invention, when switching from the automatic operation mode to the manual operation mode, even when a driver change occurs, the operation mode can sufficiently ensure the driver's preparation period. A switching control device, method, and program can be provided.

The figure which shows the whole structure of the automatic driving | operation control system provided with the driving mode switching control apparatus which concerns on one Embodiment of this invention. The block diagram which shows the function structure of the operation mode switching control apparatus which concerns on one Embodiment of this invention. The flowchart which shows the procedure and control content of the preparation period control by the operation mode switching control apparatus shown in FIG. The conceptual diagram for demonstrating the content of the preparation period control by the operation mode switching control apparatus shown in FIG. The figure which showed an example of the information output in the preparation period control by the operation mode switching control apparatus shown in FIG. The figure which showed the other example of the information output in the preparation period control by the operation mode switching control apparatus shown in FIG.

Embodiment

Embodiments according to the present invention will be described below with reference to the drawings.
[One Embodiment]
(Constitution)
FIG. 1 is a diagram showing an overall configuration of an automatic driving control system including an operation mode switching control device according to an embodiment of the present invention, and this automatic driving control system is mounted on a vehicle 1 such as a passenger car.

The vehicle 1 includes, as basic equipment, a power unit 2 including a power source and a transmission, a steering device 3 equipped with a steering wheel 4, and further includes a manual operation mode and an automatic operation mode as operation modes. . An engine and / or a motor is used as the power source.

The manual driving mode is a mode in which the vehicle 1 is driven mainly by a driving operation manually performed by a driver (hereinafter also referred to as a driver). The manual operation mode includes, for example, an operation mode for driving the vehicle based only on the driver's driving operation, and an operation mode for performing driving operation support control for supporting the driving operation of the driver while mainly driving the driver's driving operation. Is included.

The driving operation support control assists the steering torque so that the driver's steering becomes an appropriate steering amount based on the curvature of the curve when the vehicle 1 is traveling on the curve, for example. The driving operation support control includes control for assisting a driver's accelerator operation (for example, operation of an accelerator pedal) or brake operation (for example, operation of a brake pedal), manual steering (manual operation of steering), and manual speed adjustment (speed). Adjustment manual operation) is also included. In manual steering, the vehicle 1 is steered mainly by the driver's operation of the steering wheel 4. In the manual speed adjustment, the speed of the vehicle is adjusted mainly by the driver's accelerator operation or brake operation.

Note that the driving operation support control does not include control for forcibly intervening in the driving operation of the driver and automatically driving the vehicle. In other words, in the manual driving mode, the driving operation of the driver is reflected in the driving of the vehicle within a preset allowable range, but forcibly intervenes in the driving of the vehicle under certain conditions (for example, deviation from the lane of the vehicle). Control to do is not included.

On the other hand, the automatic operation mode is a mode that realizes an operation state in which the vehicle automatically travels along the road on which the vehicle travels, for example. The automatic driving mode includes, for example, a driving state in which the vehicle automatically travels toward a preset destination without driving by the driver. In the automatic driving mode, it is not always necessary to automatically control all of the vehicle, and the driving state in which the driving operation of the driver is reflected in the driving of the vehicle within the preset allowable range is also included in the automatic driving mode. That is, the automatic driving mode includes control for forcibly intervening in driving of the vehicle under certain conditions, while reflecting the driving operation of the driver in driving of the vehicle within a preset allowable range.

In FIG. 1, reference numeral 5 denotes an automatic operation control device for executing operation control in the automatic operation mode. The automatic driving control device 5 acquires sensing data from the steering sensor 11, the accelerator pedal sensor 12, the brake pedal sensor 13, the GPS receiver 14, the gyro sensor 15, and the vehicle speed sensor 16, respectively. These sensing data, route information generated by a navigation system (not shown), traffic information acquired by road-to-vehicle communication, information obtained by a peripheral monitoring system that monitors the position and movement of surrounding people and vehicles Based on this, the traveling of the vehicle 1 is automatically controlled.

Automatic control includes, for example, automatic steering (automatic steering operation) and automatic speed adjustment (automatic driving of speed). Automatic steering is an operating state in which the steering device 3 is automatically controlled. Automatic steering includes LKA (Lane Keeping Assist). For example, the LKA automatically controls the steering device 3 so that the vehicle 1 does not deviate from the traveling lane even when the driver does not perform the steering operation. Even when LKA is being executed, the driver's steering operation may be reflected in the steering of the vehicle in a range where the vehicle 1 does not deviate from the travel lane (allowable range). Note that automatic steering is not limited to LKA.

Automatic speed adjustment is an operating state in which the speed of the vehicle 1 is automatically controlled. Automatic speed adjustment includes ACC (Adaptive Cruise Control). For example, when there is no preceding vehicle ahead of the vehicle 1, ACC performs constant speed control that causes the vehicle 1 to travel at a constant speed at a preset speed, and when the preceding vehicle exists ahead of the vehicle 1. Is a follow-up control that adjusts the vehicle speed of the vehicle 1 in accordance with the inter-vehicle distance from the preceding vehicle. The automatic operation control device 5 decelerates the vehicle 1 according to the driver's brake operation (for example, operation of the brake pedal) even when ACC is being executed. In addition, the automatic operation control device 5 can perform the driver's accelerator operation (for example, accelerator) up to a preset maximum allowable speed (for example, the maximum speed legally determined on the traveling road) even during execution of ACC. The vehicle can be accelerated according to the pedal operation. The automatic speed adjustment is not limited to ACC but also includes CC (Cruise Control).

By the way, the automatic operation control system of this embodiment is an operation mode switching control device 6 and a driver camera 7 as a first monitoring sensor as devices for switching between the manual operation mode and the automatic operation mode. And a torque sensor 8 as a second monitoring sensor and an audio output device 9.

The driver camera 7 is installed at a position in front of the driver such as on the dashboard, for example, and images the driver and outputs the video signal to the driving mode switching control device 6. The torque sensor 8 detects torque generated when the driver operates the steering wheel 4, and outputs a detection signal to the operation mode switching control device 6. The audio output device 9 has a speaker and a display, and outputs the audio signal of the message output from the operation mode switching control device 6 from the speaker and displays the display signal of the message on the display.

The operation mode switching control device 6 controls the switching of the operation modes as a whole, and is configured as follows.
FIG. 2 is a block diagram showing the functional configuration.

That is, the operation mode switching control device 6 includes a control unit 61, an input / output interface unit 62, and a storage unit 63.

The input / output interface unit 62 receives the video signal and the torque detection signal output from the driver camera 7 and the torque sensor 8, respectively, and converts them into digital data. At the same time, the message output from the control unit 61 is converted into an audio signal and a display signal and output to the audio output unit 9. Further, the operation mode switching control signal output from the control unit 61 is output to the automatic operation control device 5.

The storage unit 63 includes, as a storage medium, a nonvolatile memory such as SSD (Solid State Drive) or HDD (Hard Disk Drive) that can be written and read at any time, and a volatile memory such as RAM (Random Access Memory). As used storage areas for carrying out the present embodiment, a driver monitoring video storage unit 631, a driver state storage unit 632, and a preparation period table storage unit 633 are provided.

The driver monitoring video storage unit 631 stores the driver monitoring video acquired over time by the driver monitoring video acquisition control unit 611 in association with time information.

The driver state storage unit 632 stores information indicating the determination result generated by the driver state determination unit 612 in association with a time stamp indicating the determination timing.

The preparation period table storage unit 633 stores a preparation period table created in advance. This preparation period table defines, for example, preparation periods classified for each driver state, such as “normal state”, “tension state”, and “attentiveness distraction state”.

The control unit 61 has a CPU (Central Processing Unit) and a program memory that constitute a computer, and includes a driver monitoring video acquisition control unit 611 and a driver state determination unit 612 as control functions necessary for carrying out the present embodiment. A preparation period control unit 613, a preparation period information generation unit 614, and an operation mode switching signal output unit 615. Each of these control functions is realized by causing the CPU to execute a program stored in the program memory.

The driver monitoring video acquisition control unit 611 fetches the digital data (driver monitoring video data) of the driver video signal output from the driver camera 7 from the input / output interface unit 62, and stores the fetched driver monitoring video data in the storage unit. 63 stored in the driver monitoring video storage unit 631. Further, the driver monitoring video acquisition control unit 611 creates time series local facial video data and time series global motion video data from the time series driver monitoring video data sequentially output from the driver camera 7. The driver monitoring video storage unit 631 of the storage unit 63 can also be stored.

The driver state determination unit 612 reads driver monitoring video data from the driver monitoring video storage unit 631 at a preset time interval. Each time the driver monitoring video data is read, a process is performed to determine whether or not the driver can manually perform a driving operation based on the driver monitoring video data. For example, it is determined whether or not the driver is in a sleep state by checking whether or not the driver has closed his / her eyes using local facial image data. Then, the information indicating the determination result is stored in the driver state storage unit 632 in association with the time stamp indicating the determination timing.

In addition, the driver state determination unit 612 uses a plurality of feature points that are reflected in local facial image data acquired over time by using a method such as ASM (Active Shape Model) or AAM (Active Appearance Model). (For example, each organ of the driver's face, nose, mouth, etc.) is detected, and their position is calculated. In addition, the driver state determination unit 612 calculates a face shape model for local face image data at each time by a three-dimensional fitting process using the calculated positions of the plurality of feature points. Further, the driver state determination unit 612 calculates the similarity between the face shape model corresponding to the reference time and the face shape model corresponding to another time, and compares the obtained similarity with a preset threshold value. Thus, it is determined whether or not a driver change has occurred.

The preparation period control unit 613 performs control related to the preparation period that is set for the driver to prepare when switching from the automatic operation mode to the manual operation mode. For example, the preparation period control unit 613 performs a predetermined preparation period when the driver state determination unit 612 determines that the driver can perform a driving operation in the manual operation mode during the operation control period in the automatic operation mode. Set. The preparation period control unit 613 sets a new preparation period when the driver state determination unit 612 determines that a driver change has occurred during the set preparation period.

The preparation period information generation unit 614 provides information (preparation period information) for informing the driver of at least how many seconds the remaining preparation period is regarding the preparation period set or reset by the preparation period control unit 613. And output to the audio output unit 9 and the display unit 10 via the input / output interface unit 62.

The operation mode switching signal output unit 615 outputs a control signal for switching from the automatic operation mode to the manual operation mode to the automatic operation control device 5 when the preparation period has elapsed.

(Operation)
FIG. 3 is a flowchart showing the procedure and control contents of the preparation period control by the operation mode switching control device 6. FIG. 4 is a conceptual diagram for explaining the contents of the preparation period control by the operation mode switching control device 6. The operation of the operation mode switching control device 6 will be described according to FIG. 3 with reference to FIG.

(1) Acquisition of Driver Monitoring Video When driving according to the automatic driving mode is started, the driver camera 7 is activated, and continuously captures a predetermined range including the driver's face and outputs the video signal. In this state, the operation mode switching control device 6 first receives the digital data (driver monitoring video data) of the video signal output from the driver camera 7 from the input / output interface unit 62 under the control of the driver monitoring video acquisition control unit 611. The captured driver monitoring video data is stored in the driver monitoring video storage unit 631 of the storage unit 63 (step S12).

Note that the imaging of the driver may be intermittently performed at a cycle shorter than a time interval for determining the state of the driver described later. In addition, the driver camera 7 or the input / output interface unit 62 may encode the video signal according to a predetermined encoding method. In this way, it is possible to save the storage capacity of the driver monitoring video storage unit 631 by reducing the information amount of the monitoring video data.

(2) Determination of Driver State When the acquisition of the driver monitoring video data is started, the operation mode switching control device 6 next determines that an elapse of a certain time is determined under the control of the driver state determination unit 612 ( In step S13), driver monitoring video data is read from the driver monitoring video storage unit 631. Then, it is determined from each of the read driver monitoring video data whether or not the driver can manually perform the driving operation (step S14).

For example, based on driver monitoring video data, the driver's eye open state, blinking frequency, or eye movement is detected to recognize the driver's arousal level. Then, by comparing this arousal level with a threshold value, it is determined whether or not the driver can perform a driving operation manually.

When the determination result of whether or not the manual driving operation is possible is obtained, the driver state determination unit 612 associates the information indicating the determination result with information indicating the determination timing, for example, time stamp information, in the driver state storage unit 632. Store (step S15).

(3) Determination of switching to manual operation mode The driver state determination unit 612 determines whether or not to switch the current automatic operation mode to the manual operation mode at a predetermined timing (step S16). The timing of this determination is typically tens of seconds before reaching the exit of the expressway when the vehicle is driving on the expressway in the automatic driving mode, and the external sensor of the vehicle is normal due to bad weather etc. The timing at which it is determined that the operation of the automatic operation mode is stopped and the continuation of the automatic operation mode is stopped may be mentioned.

If the driver state determination unit 612 determines that the current automatic operation mode is not switched to the manual operation mode, the process from step S11 to S16 is repeatedly executed without proceeding to step S17. On the other hand, if the driver state determination unit 612 determines that the current automatic operation mode is to be switched to the manual operation mode, a control signal for instructing control related to the preparation period associated with switching from the automatic operation mode to the manual operation mode is provided. Then, it is output from the driver state determination unit 612 to the preparation period control unit 613.

(3) Control related to preparation period When the preparation period control unit 613 receives a control signal from the driver state determination unit 612, the preparation period control unit 613 sets the preparation period to 60 seconds, for example, and starts countdown (step S17). The preparation period information generation unit 614 generates preparation period information for notifying the driver of at least how many seconds remain in the preparation period, and outputs the preparation period information to the display unit 10 via the input / output interface unit 62 (step S18). .

FIG. 5 shows an example of the preparation period information generated and output in step S18 and displayed on the display unit 10. The driver can visually recognize how many seconds from the automatic operation mode to the manual operation mode by the displayed preparation period information. Moreover, you may make it output the warning sound for notifying that the timing of switching to the manual operation mode from the automatic operation mode is near, the audio | voice of the remaining time of a preparation period from the audio | voice output device 9 as needed. The driver can recognize from the preparation period information that the switch to the manual operation mode is about to be performed, and thus can start preparation so as to correspond to the manual operation mode.

For example, based on the latest local face image data before the start of the preparation period and the local face image data sequentially acquired in the preparation period, the driver state determination unit 612 changes the driver in the preparation period. Whether or not has occurred is determined (step S19).

More specifically, the driver state determination unit 612 obtains a local facial image acquired at a predetermined reference time (in the example of FIG. 4, the latest time tk before the start of the preparation period set in step S17). Each feature such as eyes, nose and mouth of the driver's face as feature points in the data is detected, and the spatial position of each organ is calculated. In addition, the driver state determination unit 612 operates as a feature point reflected in each local face image data acquired at each time of the preparation period (in the example of FIG. 4, time tk + 1, tk + 2,...). Each organ such as eyes, nose and mouth of the person's face is detected and the spatial position of each organ is calculated. The driver state determination unit 612 calculates a face shape model for local face image data at a predetermined reference time and each time of the preparation period by a three-dimensional fitting process using the calculated positions of a plurality of feature points. To do. The driver state determination unit 612 calculates the similarity between the face shape model corresponding to the reference time and the face shape model corresponding to each time in the preparation period. The similarity calculation is not particularly limited, and for example, a general geometric pattern matching method can be used.

Furthermore, the driver state determination unit 612 compares the similarity at each time in the preparation period with a predetermined threshold value. As a result, when the similarity at a predetermined time (in the example of FIG. 4, the time tl−1) is equal to or smaller than a predetermined threshold, the driver state determination unit 612 indicates that “the driver change at the predetermined time is It is determined that it has occurred (Y in step S19), and a control signal for notifying the occurrence of a driver change is output to the preparation period control unit 613.

When the preparation period control unit 613 receives the control signal from the driver state determination unit 612, the preparation period control unit 613 sets a new preparation period as, for example, 60 seconds, and starts a countdown (step S20). The preparation period information generation unit 614 generates information (new preparation period information) for notifying the driver of at least how many seconds the newly set preparation period is remaining, and via the input / output interface unit 62 It outputs to the display part 10 (step S18 which passed through step S20).

FIG. 6 shows an example of new preparation period information generated and output in step S18 and displayed on the display unit 10. Compared with FIG. 5, in addition to the remaining time of the (new) preparation period, a message is added regarding the occurrence of a driver change and therefore the establishment of a new preparation period. Based on the displayed preparation period information, the driver can quickly and easily visually confirm that the preparation period has been newly secured in accordance with the driver change, in addition to how many seconds the automatic operation mode is switched to the manual operation mode. can do.

For example, as shown in FIG. 4, the driver state determination unit 612 applies local facial image data acquired at a new reference time (in the example of FIG. 4, the latest time tl before the start of a new preparation period). Each feature such as eyes, nose, mouth, etc. of the driver's face as a feature point is detected, and the spatial position of each organ is calculated. In addition, the driver state determination unit 612 is a feature point that is reflected in each local face image data acquired at each time of the new preparation period (in the example of FIG. 4, time tl + 1, tl + 2,...). The driver's face, eyes, nose, mouth and other organs are detected and their respective spatial positions are calculated. The driver state determination unit 612 performs a face shape model on local face image data at each time of a new reference time and a new preparation period by a three-dimensional fitting process using the calculated positions of a plurality of feature points. Calculate The driver state determination unit 612 calculates the similarity between the face shape model corresponding to the new reference time and the face shape model corresponding to each time of the new preparation period.

Furthermore, the driver state determination unit 612 compares the similarity at each time in the new preparation period with a predetermined threshold. As a result, when the similarity at a predetermined time (in the example of FIG. 4, time tm−1) is equal to or less than a predetermined threshold, the driver state determination unit 612 indicates that “the driver change at the predetermined time is It is determined that it has occurred "(Y in step S19 via step S20), and a control signal for notifying the occurrence of a driver change is output to the preparation period control unit 613.

When the preparation period control unit 613 receives the control signal from the driver state determination unit 612, the preparation period control unit 613 newly sets a new preparation period, for example, as 60 seconds, and starts countdown (step S20). The preparation period information generation unit 614 generates information (further new preparation period information) for informing the driver of at least how many seconds remain in the newly set preparation period. To the display unit 10 (step S18 via step S20).

On the other hand, in step S19, if the similarity at each time within the preparation period does not fall below the predetermined threshold, the driver state determination unit 612 determines that “no driver change has occurred” (in step S19). N). The preparation period control unit 613 continues to count down the preparation period set in step S17 (step S21).

The preparation period control unit 613 determines whether or not the countdown of the latest preparation period has been completed, and if not completed, repeats the processing from step S19 to S22 (N in step S22). Also. If the driver change occurs before the countdown of the latest preparation period is completed, the processing from steps S18 to S20 is repeatedly executed at each change.

(4) Operation mode switching signal output control When the countdown of the latest preparation period becomes zero (that is, when the latest preparation period elapses without any driver change: Y in step S22) The preparation period control unit 613 outputs a control signal for notifying that the preparation period has elapsed to the operation mode switching signal output unit 615 (step S23).

When receiving the control signal for notifying that the preparation period has elapsed from the preparation period control unit 613, the operation mode switching control apparatus 6 sends a control signal for switching from the automatic operation mode to the manual operation mode to the automatic operation control apparatus 5. Output. At this time, the preparation period information generation unit 614 generates information (switching completion information) for notifying that the switching from the automatic operation mode to the manual operation mode is completed as a result of the preparation period having elapsed, and the input / output interface unit 62 Is output to the display unit 10 (step S24). The driver grasps that the switching from the automatic operation mode to the manual operation mode is completed based on the switching completion information, and thereafter, the driving control according to the manual operation of the driver is performed.

(effect)
As described above in detail, in one embodiment of the present invention, when the automatic operation mode is switched to the manual operation mode, the preparation period control unit 613 sets and counts the preparation period and informs the remaining time of the preparation period. The preparation period information is generated by the preparation period information generation unit 614 and output from the sound output unit 9 and the display unit 10. When the driver state determination unit 612 determines that a driver change has occurred, the preparation period control unit 613 sets and counts a new preparation period and informs the remaining time of the new preparation period. For this purpose, new preparation period information is generated by the preparation period information generation unit 614 and output from the audio output device 9 and the display unit 10. Therefore, it is possible to prevent the situation where two drivers share the limited preparation period when switching from the automatic operation mode to the manual operation mode, and a certain preparation period is always provided for the driver after the change. Can be secured. Thereby, the new driver after the change can make sufficient preparations for the manual driving operation and can perform the manual driving with a psychological margin. As a result, a safe and comfortable driving environment can be provided.

Further, the driver state determination unit 612 uses the video data about the driver acquired by the monitoring camera before starting the counting of the preparation period set in step S17, and the driving acquired by the monitoring camera in the new preparation period. The similarity is calculated using the video data of the driver, and a determination is made as to whether or not a driver change has occurred based on the calculated similarity. Therefore, based on the unique information such as the driver's face, it can be reliably determined whether or not a driver change has occurred before and after the start of the latest preparation period.

Furthermore, when a driver change occurs in a new preparation period, a new preparation period is set, the new preparation period is counted, and preparation period information for informing the remaining time is generated. And output from the audio output device 9 and the display unit 10. As a result, it is possible to determine whether or not a driver change has occurred even during the newly set preparation period, and every time the driver changes until the automatic operation mode is switched to the manual operation mode. It is possible to reliably set a new preparation period for a new driver.

Further, the driver state determination unit 612 further calculates video data about the driver acquired by the monitoring camera before starting the counting of a new preparation period, and about the driver acquired by the monitoring camera in a new preparation period. The similarity is calculated using the video data, and a determination is made as to whether or not a driver change has occurred based on the calculated similarity. Therefore, based on the unique information such as the driver's face, it can be reliably determined whether or not a driver change has occurred before and after the start of the latest preparation period.

Furthermore, the operation mode switching signal output unit 615 outputs a signal for switching the automatic operation mode to the manual operation mode when the latest preparation period has elapsed, to the automatic operation control device 5. For this reason, it is possible to switch the automatic operation mode to the manual operation mode after reliably providing a certain preparation period to the new driver after the change.

Furthermore, the preparation period control unit 613 can adjust the newly set preparation period to a predetermined length according to the current driver state, and can set an appropriate preparation period flexibly. , Can increase safety.

Furthermore, information notifying that the driver change has been confirmed is generated by the preparation period information generation unit 614 and output from a predetermined output unit. Accordingly, the driver quickly and easily grasps that the preparation period is newly set and the remaining time of the preparation period, and that the preparation period is newly set is due to the change of the driver. be able to.

(Modification 1)
In the above-described embodiment, the latest monitoring video data acquired before the start of the latest preparation period is employed as the monitoring video data serving as a reference for determining the occurrence of the driver change. However, this is merely an example, and other monitoring video data can be adopted as the reference monitoring video data. Examples of criteria for selecting the monitoring video data as a reference include monitoring video data that has been successfully detected at the position of each feature point in the face area, and monitoring video data that is as close in time as possible to the start time of the latest preparation period. It is mentioned. The monitoring video data used as the reference need not be limited to the monitoring video data at a specific time, and monitoring video data at a plurality of times is selected and averaged using the face shape pattern for each monitoring video data. It is also possible to use the calculated one for similarity calculation.

With such a configuration, it is possible to realize a robust preparation time control without depending on a monitoring video at a specific time.

(Modification 2)
In the said embodiment, the preparation period newly set with generation | occurrence | production of a driver | operator's change was demonstrated as a uniform time (same 60 seconds as the preparation period before being newly set). However, this is merely an example, and it is possible to adjust the length of the newly set preparation period in accordance with the current state of the driver. In this case, the driver state determination unit 612 is, for example, a global operation image (related to the driver) acquired over time by the driver monitoring image acquisition control unit 611, a biological signal (for example, a pulse wave sensor or Based on the driver's pulse wave signal or heartbeat signal detected by the heartbeat sensor, the signal indicating the vertical movement of the diaphragm detected by the pressure sensor), etc., the driver's state is determined. As a result, when it is better to extend the preparation period than usual, such as “the new driver is in tension”, the preparation period control unit 613 prepares the preparation period table storage unit 633. The preparation period corresponding to the current driver's situation is set with reference to the table stored in (1).

With such a configuration, the preparation period considering the new driver condition can be set flexibly, and a safer and more comfortable driving environment can be provided.

(Modification 3)
In the above embodiment, as an example of the case where the monitoring video data including the driver's face obtained by the driver camera 7 is used as means for determining the occurrence of the driver change. However, this is merely an example, and any means may be used as long as it can accurately determine the occurrence of a driver change. For example, different information such as a driver's fingerprint, earprint, and retina pattern may be acquired at the time of change, and the occurrence of the driver change may be accurately determined based on this information.

(Modification 4)
In the above embodiment, when the driver state determination unit 612 determines “switch to the manual operation mode” in step S <b> 16, the preparation period control unit 613 starts setting the preparation period and counting it down. However, the trigger for setting the preparation period and starting the countdown by the preparation period control unit 613 is not limited to this example. For example, an instruction input that the driver himself “switches to the manual operation mode” by a predetermined operation may be used as a trigger, or an override operation may be performed as a trigger.

In addition, the vehicle type, the function of the automatic operation control device, the control function and control procedure of the operation mode switching control device, and the control contents can be variously modified and implemented without departing from the gist of the present invention.

That is, the present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the scope of the invention in the implementation stage. In addition, the embodiments may be appropriately combined as much as possible, and in that case, the combined effect can be obtained. Further, the above embodiments include inventions at various stages, and various inventions can be extracted by appropriately combining a plurality of disclosed constituent elements.

Some or all of the above embodiments may be described as in the following supplementary notes, but are not limited to the following.

(Appendix 1)
An operation mode switching control device for switching and controlling a vehicle operation mode between a manual operation mode and an automatic operation mode,
Having at least one hardware processor;
The at least one hardware processor comprises:
When it is determined that the driver is in a state where the driver can perform the driving operation in the manual operation mode during the operation control period in the automatic operation mode, the first preparation period is set, and the first preparation period is counted. ,
Generating first preparation period information for informing the remaining time of the first preparation period;
Outputting the first preparation period information to an output unit;
In the first preparation period, it is determined whether a driver change has occurred,
If it is determined that a driver change has occurred,
Set a second preparation period and count the second preparation period,
Generating second preparation period information for informing the remaining time of the second preparation period;
An operation mode switching control device configured to cause the output unit to output the second preparation period information.
(Appendix 2)
An operation mode switching control method executed by an apparatus for switching and controlling a vehicle operation mode between a manual operation mode and an automatic operation mode,
Using at least one hardware processor,
A control process of setting a first preparation period when switching from the automatic operation mode to the manual operation mode, and counting the first preparation period;
Generating a first preparation period information for informing the remaining time of the first preparation period;
Outputting the first preparation period information;
In the first preparation period, a determination process for determining whether or not a driver change has occurred;
Run
When it is determined that a driver change has occurred in the determination process,
A control step of setting a second preparation period and counting the second preparation period;
Generating a second preparation period information for informing the remaining time of the second preparation period;
An output process of outputting the second preparation period information;
The operation mode switching control method for executing

Claims

An operation mode switching control device for switching and controlling a vehicle operation mode between a manual operation mode and an automatic operation mode,
A preparation period controller configured to set a first preparation period when switching from the automatic operation mode to the manual operation mode, and to count the first preparation period;
A preparation period information generating unit that generates first preparation period information for informing the remaining time of the first preparation period;
An output unit for outputting the first preparation period information;
A determination unit that determines whether or not a driver change occurs in the first preparation period;
Comprising
If it is determined by the determination unit that a driver change has occurred,
The preparation period control unit sets a second preparation period and counts the second preparation period,
The preparation period information generating unit generates second preparation period information for informing a remaining time of the second preparation period;
The output unit outputs the second preparation period information;
Operation mode switching control device.
The determination unit includes video data about the driver acquired by the monitoring camera before starting the counting of the first preparation period and video data about the driver acquired by the monitoring camera during the first preparation period. The operation mode switching control device according to claim 1, wherein the determination is made as to whether or not a driver change has occurred based on the calculated similarity.
The determination unit determines whether or not a driver change has occurred in the second preparation period,
In the second preparation period, the preparation period control unit sets a third preparation period and counts the third preparation period when the determination unit determines that a driver change has occurred in the second preparation period. And
The preparation period information generating unit generates preparation period information for informing the remaining time of the third preparation period;
The operation mode switching control device according to claim 1, wherein the output unit outputs the third preparation period information.
The determination unit is similar using the video data about the driver acquired by the monitoring camera in the first preparation period and the video data about the driver acquired by the monitoring camera in the second preparation period. The operation mode switching control device according to claim 3, wherein a degree is calculated, and the determination as to whether or not a driver change has occurred in the second preparation period is performed based on the calculated similarity.
The operation according to any one of claims 1 to 4, further comprising an operation mode switching signal output unit that outputs a signal for switching the automatic operation mode to the manual operation mode when a latest preparation period has elapsed. Mode switching control device.
The operation mode switching control device according to any one of claims 1 to 5, wherein the preparation period information generation unit adjusts a length of a newly set preparation period in accordance with a predetermined standard.
The preparation period information generation unit further generates information notifying that the change of the driver has been confirmed,
The operation mode switching control device according to any one of claims 1 to 6, wherein the output unit outputs information notifying that the change of the driver has been confirmed.
An operation mode switching control method executed by an apparatus for switching and controlling a vehicle operation mode between a manual operation mode and an automatic operation mode,
A control process of setting a first preparation period when switching from the automatic operation mode to the manual operation mode, and counting the first preparation period;
Generating a first preparation period information for informing the remaining time of the first preparation period;
Outputting the first preparation period information;
In the first preparation period, a determination process for determining whether or not a driver change has occurred;
Comprising
When it is determined that a driver change has occurred in the determination process,
A control step of setting a second preparation period and counting the second preparation period;
Generating a second preparation period information for informing the remaining time of the second preparation period;
An output process of outputting the second preparation period information;
An operation mode switching control method comprising:
A program that causes a computer to function as each of the units included in the operation mode switching control device according to any one of claims 1 to 7.