JP2019074498A - Drive supporting device - Google Patents

Abstract

To provide a drive supporting device that allows a more smooth talking with a passenger.SOLUTION: The drive supporting device according to an embodiment includes: a sound output unit for outputting sound to a passenger; an acquisition unit for acquiring an image of the passenger in an image taken by an imaging device for imaging the inside of a vehicle; and a determination unit for determining whether the output of sound is to be interrupted, on the basis of the operation or the state of the passenger in the result of acquisition by the acquisition unit. The sound output unit interrupts output of sound when the determination unit has determined that the output of sound needs to be interrupted.SELECTED DRAWING: Figure 4

Description

  An embodiment of the present invention relates to a driving support device.

  2. Description of the Related Art Conventionally, there is known a driving assistance apparatus that performs dialogue with an occupant by outputting voice guidance and performing voice recognition of a response of the occupant (user). In addition, in a system that interacts with a user, there is known a technology capable of performing barge-in (interruption) by the user's voice while voice guidance is output from the system.

JP 2004-163541 A

  However, in the prior art, it has been desired that the interaction between the passenger and the driving support device be performed more smoothly.

  A driving assistance apparatus according to an embodiment of the present invention includes, as an example, a voice output unit that outputs a voice to an occupant, and an acquisition unit that acquires an occupant image included in a captured image of an imaging device that captures an interior of a vehicle. And a determination unit configured to determine whether or not to interrupt the audio output based on the operation or state of the occupant included in the acquisition result of the acquisition unit. The voice output unit interrupts the output of voice when the determination unit determines that the output of voice is interrupted. Therefore, according to the driving support device of the embodiment, since the output of the voice is interrupted based on the operation and the state of the occupant, for example, it is possible to interact with the occupant more smoothly.

  In the above-described driving support device, as an example, the predetermined action may be an action of speech, an action indicating intention of approval, an action indicating intention of denial, an action indicating intention of holding, or that the output sound can not be heard. It is one of the operations to represent. For this reason, according to the driving support device according to the embodiment, by interrupting the voice based on the intention of the occupant, for example, it is possible to suppress the output of the voice unnecessary for the occupant. Thereby, according to the driving assistance device concerning an embodiment, it can interact with a crew member more smoothly.

  The above-described driving support device further includes, as an example, a dialog control unit that executes processing corresponding to the approval of the output voice when it is determined that the occupant is performing an operation indicating the intention of the approval. Therefore, according to the driving support device according to the embodiment, for example, after confirming the intention of the occupant, it is possible to quickly start the subsequent processing in accordance with the intention of the occupant. For this reason, according to the driving support device according to the embodiment, for example, the time required for the dialogue can be shortened, and the dialogue with the occupant can be performed more smoothly.

  The above-described driving support device further includes, as an example, a dialog control unit that executes a process corresponding to negation of the output voice when it is determined that the occupant is performing an operation indicating a negative intention. Therefore, according to the driving support device according to the embodiment, for example, after confirming the intention of the occupant, it is possible to quickly start the subsequent processing in accordance with the intention of the occupant. For this reason, according to the driving support device according to the embodiment, for example, the time required for the dialogue can be shortened, and the dialogue with the occupant can be performed more smoothly.

  The above-described driving support device further includes, as an example, a voice recognition unit that recognizes a voice uttered by a passenger with respect to the voice output by the voice output unit. Also, when it is determined that the occupant is performing an operation indicating the intention to suspend, the voice recognition unit extends the recognition time of the voice spoken by the occupant. Therefore, according to the driving support device according to the embodiment, it is possible to wait until the occupant makes a decision, and to perform dialogue in accordance with the pace at which the occupant answers.

  In the above-described driving support device, as one example, when it is determined that the occupant is performing an action indicating the intention to suspend, the voice output unit is determined to be performing an operation indicating the intention to suspend. Output a voice different from the voice being output. Therefore, according to the driving support device according to the embodiment, it is possible to support the decision making of the occupant by presenting an option different from the content of the voice that is initially output.

  In the above-described driving support device, as an example, when it is determined that the occupant is performing an action indicating that the voice can not be heard, the voice output unit determines that the occupant is performing an action indicating that the voice can not be heard. The voice that was being output when it was output is output again. Therefore, according to the driving support device according to the embodiment, by re-outputting the voice that the occupant could not hear, it is possible to smoothly interact with the occupant.

FIG. 1 is an exemplary perspective view in which a part of a cabin of a vehicle according to the embodiment is seen through. FIG. 2 is a view showing an example of the arrangement of imaging devices according to the embodiment. FIG. 3 is a diagram illustrating an example of a hardware configuration of the driving support system according to the embodiment. FIG. 4 is a block diagram showing an example of functions of the ECU according to the embodiment. FIG. 5 is a flowchart illustrating an example of the procedure of the dialogue control process according to the embodiment. FIG. 6 is a diagram showing an example of the voice guidance by the ECU according to the embodiment and the dialog with the occupant. FIG. 7 is a view showing an example of an operation representing the occupant's approval intention according to the embodiment. FIG. 8 is a diagram showing an example of an operation representing the occupant's negative intention according to the embodiment. FIG. 9 is a diagram illustrating an example of the operation of the utterance of the occupant according to the embodiment. FIG. 10 is a diagram illustrating an example of an operation indicating that the output sound according to the embodiment can not be heard. FIG. 11 is a diagram illustrating an example of processing corresponding to suspension according to the fourth modification.

  In the present embodiment, the driver assistance apparatus can interrupt the voice output according to the operation of the occupant, whereby the occupant and the drive assistance apparatus can interact more smoothly. Hereinafter, the example which mounted the driving assistance device of this embodiment in the vehicle is mentioned and demonstrated.

  FIG. 1 is an exemplary perspective view in which a part of a compartment 2a of a vehicle 1 according to the present embodiment is seen through. The vehicle 1 may be, for example, an internal combustion engine automobile, an electric automobile, a fuel cell automobile, a hybrid automobile or the like, or may be an automobile provided with another drive source. In addition, the vehicle 1 can be mounted with various transmissions, and can be mounted with various devices necessary for driving an internal combustion engine or a motor, such as a system or components.

  Further, as illustrated in FIG. 1, the vehicle 1 is, for example, a four-wheeled vehicle, and has two left and right front wheels 3F and two left and right two rear wheels 3R. All of these four wheels 3 can be configured to be steerable. The system, the number, the layout, and the like of devices related to the driving of the wheels 3 in the vehicle 1 can be set variously.

  As illustrated in FIG. 1, the vehicle body 2 constitutes a passenger compartment 2 a in which a passenger (not shown) rides. A steering unit 4, an acceleration operation unit 5, a braking operation unit 6, a shift operation unit 7 and the like are provided in the passenger compartment 2a in a state of facing the driver's seat 40a (driver's seat) as a passenger.

  The steering unit 4 is, for example, a steering wheel (handle) that protrudes from the dashboard 12. In addition, the acceleration operation unit 5 is, for example, an accelerator pedal positioned under the driver's foot. The braking operation unit 6 is, for example, a brake pedal positioned under the driver's foot. Further, the shift operation unit 7 is, for example, a shift lever that protrudes from the center console. The steering unit 4, the acceleration operation unit 5, the braking operation unit 6, and the transmission operation unit 7 are not limited to these.

  Further, a monitor device 11 is provided in the passenger compartment 2a. The monitor device 11 is provided with a display device (shown in FIG. 3) and an audio output device (shown in FIG. 3). The audio output device is, for example, a speaker. The display device is, for example, a liquid crystal display (LCD), an organic electroluminescent display (OELD), or the like. The display device is covered with a transparent operation input unit (shown in FIG. 3) such as a touch panel, for example. Further, an audio output device (not shown) may be provided at another position in the compartment 2 a different from the monitor device 11. The monitor device 11 can also be used as, for example, a navigation system or an audio system.

  Further, on the ceiling of the vehicle body 2, a voice input unit 24 is provided. The voice input unit 24 is, for example, a microphone, and can input a voice of a passenger in the passenger compartment 2a. The installation position of the voice input unit 24 shown in FIG. 1 is an example, and the present invention is not limited to this.

  Further, an imaging device 15 for imaging the inside of the vehicle 1 is installed on a steering wheel column (shown in FIG. 2) connecting the steering unit 4 and the dashboard 12. The imaging device 15 is, for example, a CCD (Charge Coupled Device) camera or the like.

  FIG. 2 is a view showing an example of the arrangement of the imaging device 15 according to the present embodiment. In the present embodiment, the imaging device 15 is installed on the handle column 41. The viewing angle and the attitude of the imaging device 15 are adjusted such that the face of the occupant 42 sitting on the seat 40 a is positioned at the center of the field of view. The imaging device 15 captures the inside of the vehicle 1 and sequentially outputs captured images, which are image data obtained by imaging, to an ECU described later.

  The installation position of the imaging device 15 shown in FIG. 2 is an example, and the present invention is not limited to this. For example, the imaging device 15 may be a wide-angle camera provided on the dashboard 12 or on the monitor device 11 or the like.

  FIG. 3 is a diagram showing an example of a hardware configuration of the driving support system 10 according to the present embodiment. As shown in FIG. 3, in the driving support system 10 installed in the vehicle 1, the brake system 18, the steering angle sensor 19 (angle sensor), the accelerator sensor 20, the shift sensor 21, in addition to the ECU 14 and the monitor device 11. The wheel speed sensor 22, the steering system 13, the voice input unit 24, and the like are electrically connected via an in-vehicle network 23 as a telecommunication line. The in-vehicle network 23 is configured as, for example, a CAN (controller area network). The voice input unit 24 may be directly connected to the ECU 14 without passing through the in-vehicle network 23.

  The ECU 14 can execute various types of arithmetic processing and control of each component of the driving support system 10. More specifically, the ECU 14 controls the steering system 13, the brake system 18 and the like by transmitting control signals through the in-vehicle network 23. Further, the ECU 14 controls the display device 8 a and the voice output device 8 b included in the monitor device 11. In addition, the ECU 14 detects the torque sensor 13b, the brake sensor 18b, the steering angle sensor 19, the accelerator sensor 20, the shift sensor 21, the wheel speed sensor 22, the voice input unit 24, etc. An instruction signal (a control signal, an operation signal, an input signal, data) of the input unit 8c or the like is received. Further, the ECU 14 acquires a captured image from the imaging device 15. The ECU 14 is an example of the driving support device in the present embodiment.

  The ECU 14 has, for example, a CPU 14 a (central processing unit), a ROM 14 b (read only memory), a RAM 14 c (random access memory), a display control unit 14 d, an SSD 14 f (solid state drive, flash memory) and the like.

  The CPU 14a reads a program installed and stored in a non-volatile storage device such as the ROM 14b, and executes arithmetic processing according to the program. In addition, the display control unit 14d mainly performs image processing using image data obtained by the imaging device 15, composition of image data displayed by the display device 8a, and the like among the arithmetic processing in the ECU 14.

  The steering system 13 steers at least two wheels 3. The steering system 13 has an actuator 13a and a torque sensor 13b. The brake system 18 also has an actuator 18a and a brake sensor 18b. The brake system 18 applies a braking force to the wheel 3 and thus to the vehicle 1 via the actuator 18a.

  The configuration, arrangement, electrical connection form and the like of the various sensors and actuators described above are merely examples, and can be set (changed) in various ways.

  FIG. 4 is a block diagram showing an example of the function of the ECU 14 according to the present embodiment. As shown in FIG. 4, the ECU 14 includes a storage unit 140, an acquisition unit 141, a determination unit 142, a voice recognition unit 143, a voice output unit 144, and a dialogue control unit 145.

  Each configuration of the acquisition unit 141, the determination unit 142, the voice recognition unit 143, the voice output unit 144, and the dialogue control unit 145 is realized by the CPU 14a executing a program stored in the ROM 14b. . Note that these configurations may be configured to be realized by a hardware circuit.

  The storage unit 140 stores texts and the like of voice guidance output from a voice output unit 144 described later. In addition, the storage unit 140 is configured of, for example, a storage device such as an SSD 14 f.

  The acquisition unit 141 acquires an image of the occupant 42 included in a captured image of the imaging device 15. More specifically, the acquisition unit 141 acquires a captured image from the imaging device 15, and extracts (acquires) an image of the occupant 42 included in the captured image by image processing. In the present embodiment, an image of the occupant 42 is referred to as an acquisition result of the acquisition unit 141. A known technique can be adopted as a method of acquiring the image of the occupant 42 from the captured image.

  The determination unit 142 determines whether to interrupt the audio output by the audio output unit 144, which will be described later, based on the operation and state of the occupant 42 included in the acquisition result of the acquisition unit 141.

  More specifically, the determination unit 142 determines the operation or state of the occupant 42 from the image of the occupant 42 included in the captured image acquired by the acquisition unit 141. For example, the determination unit 142 may perform an operation of uttering, an operation indicating an intention of approval, an operation indicating an intention of denial, an operation indicating an intention of holding, or an operation indicating that the output voice can not be heard. Determine if you are doing any of them. Hereinafter, the operation of the speech is "uttered operation", the operation representing the intention of approval is "approval operation", the operation representing the intention of denial "negative operation", the operation representing the intention of holding "hold operation", and output An operation indicating that the voice can not be heard is referred to as "a motion indicating that it can not be heard." In addition, the action of uttering, the action of representing the intention of approval, the action of representing the intention of denial, or the action of representing the intention of holding, the action of representing that the output voice can not be heard are collectively referred to as a predetermined action. .

  The speech operation is, for example, an operation of opening a mouth. Generally, noise such as radio sound is present in the car, but the judgment unit 142 judges that noise or the like is erroneously judged as the speech of the occupant 42 because the judgment portion 142 judges the presence or absence of the speech operation of the occupant 42 from the captured image. it can. For this reason, the determination unit 142 of the present embodiment can determine the presence or absence of the utterance of the occupant 42 with high accuracy. In addition, since the determination unit 142 determines that the uttering operation is performed when the occupant 42 moves the mouth to emit a voice, it is possible to recognize the utterance of the occupant 42 at an earlier stage.

  The approval operation is, for example, an operation in which the neck is swung vertically and turned, which indicates that the passenger 42 has approved the content of the sound output from the sound output unit 144 described later.

  Further, the negative operation is, for example, an operation of shaking the neck side, which is an operation indicating that the occupant 42 has made negative with respect to the content of the sound output from the sound output unit 144 described later.

  The holding operation is, for example, an operation of tilting the neck. In addition, the movement of the face, such as frowning or chewing, may be used as the hold operation. The hold operation is an operation that indicates that the occupant 42 has not yet made a decision as to whether the voice output by the voice output unit 144 described later has been approved or not.

  An operation indicating that the user can not hear is, for example, an operation of opening his eyes, and is an operation that indicates that the passenger 42 can not hear the voice output from the voice output unit 144 described later.

  From the image of the occupant 42, the determination unit 142 is any of the motion of the occupant 42, the motion indicating the intention of approval, the motion indicating the intention of denial, the motion indicating the intention of holding, and the motion indicating no hearing from the image of the passenger 42. If it is determined that the user is making a noise, it is determined that the audio output is interrupted. The contents of the specific operation of the speech operation, the approval operation, the negative operation, the hold operation, and the operation indicating that the user can not hear it are not limited to the above-described example. Further, the determination unit 142 may determine whether or not to interrupt the voice output by determining the intention of the occupant 42 based not only on the operation but also on the posture of the occupant 42 or the like.

  In addition, the determination unit 142 may determine whether or not to interrupt the sound output from the input image of the occupant 42 using a deep learning method such as Recurrent Neural Network (RNN). The method of determining whether or not the determination unit 142 interrupts the output of voice is not limited to the RNN, and other methods of deep learning and deep learning may be adopted.

  In the present embodiment, the determination unit 142 determines the operation or state of the occupant 42 based on the image of the occupant 42 acquired by the acquisition unit 141. However, the acquisition unit 141 detects the operation or state of the occupant 42 You may employ the structure which In this case, the operation or state of the occupant 42 detected from the image of the occupant 42 is referred to as an acquisition result of the acquisition unit 141.

  The voice recognition unit 143 performs voice recognition processing on the voice input to the voice input unit 24 and specifies the content of the command. In the present embodiment, the voice recognition unit 143 recognizes the voice uttered by the occupant 42 from the voice output from the voice output unit 144 described later. More specifically, the voice recognition unit 143 identifies, from the voice of the occupant 42, a word indicating an approval or denial intention. Further, the voice recognition unit 143 specifies, from the voice of the occupant 42, a proper noun indicating the destination of the vehicle 1, a verb indicating the process to be executed by the vehicle 1, and the like.

  In the present embodiment, the voice recognition unit 143 recognizes the voice of the occupant 42 in the voice recognition time provided after the voice output unit 144 outputs the voice such as the question to the occupant 42. The timing at which the speech recognition unit 143 performs speech recognition is not limited to this, and speech recognition processing may always be performed. Further, the voice recognition unit 143 may perform voice recognition only when the occupant 42 operates the start of voice recognition by the operation input unit 8c or the like. Further, as a method for the speech recognition unit 143 to recognize the speech of the occupant 42, a known method can be adopted. Further, when the determination unit 142 determines that the occupant 42 performs the on-hold operation, the voice recognition unit 143 extends the voice recognition time. During the speech recognition time, the interruption of the speech output by the speech output unit 144 is continued. The extension of the speech recognition time is an example of processing corresponding to suspension in the present embodiment, and the processing performed when the determination unit 142 determines that the occupant 42 is in suspension operation is limited to this. It is not a thing.

  The voice output unit 144 outputs a voice to the occupant 42. As an example, the voice output unit 144 controls the voice output device 8b to output voice guidance of the operation of the navigation system. Further, when the determination unit 142 determines that the output of the audio is interrupted, the audio output unit 144 interrupts the output of the audio. Further, when the sound output unit 144 determines that the determination unit 142 is performing an operation indicating that the occupant 42 can not hear, it is determined that the voice output unit 144 is performing an operation indicating that the occupant 42 can not hear. Re-output the voice that was being output.

  The dialogue control unit 145 controls the dialogue between the occupant 42 and the ECU 14. Further, the dialogue control unit 145 executes the subsequent processing in accordance with the reaction of the occupant 42 with the voice output from the voice output unit 144. For example, when the determination unit 142 determines that the occupant 42 performs the approval operation, the dialogue control unit 145 executes a process corresponding to the approval for the output voice. Further, when the determination unit 142 determines that the occupant 42 performs a negative operation, the dialogue control unit 145 executes a process corresponding to the negative of the output voice. The process corresponding to the approval and the process corresponding to the negative differ depending on the content of the audio output by the audio output unit 144.

  Further, when the speech recognition unit 143 specifies the intention of approval from the voice of the occupant 42, the dialogue control unit 145 executes processing corresponding to the approval. Further, when the speech recognition unit 143 specifies the negative intention from the voice of the occupant 42, the dialogue control unit 145 executes a process corresponding to the negative.

  The process corresponding to the approval is a process that is predetermined to be executed when the occupant 42 approves the output voice. Further, the process corresponding to the negative is a process which is predetermined to be executed when the occupant 42 makes a negative with respect to the output voice. Specific examples of the process corresponding to the approval and the process corresponding to the denial will be described later.

  Next, dialogue control processing in the ECU 14 of the present embodiment configured as described above will be described using FIGS. FIG. 5 is a flowchart showing an example of the procedure of the dialogue control process according to the present embodiment.

  The voice output unit 144 controls the voice output device 8b to start voice output to the occupant 42 (S1). Moreover, FIG. 6 is a figure which shows an example of the audio | voice guidance by ECU14 concerning this embodiment, and a dialog with the passenger | crew 42. As shown in FIG. In the process of S1 shown in FIG. 5, it is assumed that the output of the voice t2 in the voice guidance shown in FIG. 6 is started.

  Further, returning to the flowchart of FIG. 5, the acquisition unit 141 acquires a captured image from the imaging device 15 (S2). The acquisition unit 141 also acquires an image of the occupant 42 from the captured image. Further, the determination unit 142 determines the operation of the occupant 42 from the image of the occupant 42 acquired by the acquisition unit 141.

  When it is determined that the occupant 42 has performed the approval operation while the voice t2 is being output (S3 “Yes”), the determination unit 142 determines to interrupt the voice output. In this case, the audio output unit 144 interrupts the output of the audio t2 (S4). Then, the dialogue control unit 145 executes a process corresponding to the approval of the voice t2 (S5).

  FIG. 7 is a view showing an example of the approval operation of the occupant 42 according to the present embodiment. As illustrated in FIG. 7, when the determination unit 142 determines that the occupant 42 performs an approval operation such as peeping in the middle of the sound t2, the sound output unit 144 interrupts the output of the sound t2. Further, the dialogue control unit 145 executes a process corresponding to the approval of the voice t2.

  In the example shown in FIG. 7, the voice t2 is a voice for requesting approval or denial with respect to the destination, so the processing corresponding to the approval of the voice t2 is performed when the occupant 42 approves the destination included in the voice t2. It is a process to be performed. As a process corresponding to the approval of the voice t2, the dialogue control unit 145 causes the voice output unit 144 to output the voice t3 in response to the approval of the occupant 42. Further, the dialogue control unit 145 sets the destination indicated by the voice t2 as the destination of the navigation system. The process corresponding to the approval of the voice t2 is not limited to this, and varies depending on the content of the output voice t2.

  Generally, although the occupant 42 approves the content of the voice t2, when the voice t2 is output to the end, the occupant 42 has to wait for the end of the voice t2, and the dialogue can be smoothly performed. It can be difficult. The voice output unit 144 according to the present embodiment interrupts the output of the voice t2 when the occupant 42 performs the approval operation in the middle of the voice t2, and the dialog control unit 145 shifts to the subsequent processing, whereby the passenger 42 smoothly. Can interact with

  Also, in general, in human-to-human dialogue, the dialogue is progressed smoothly by communicating intentions such as approval not only by speech but also by action. Since the determination unit 142 of the present embodiment can determine the operation representing the intention of the occupant 42 or the like, the voice output unit 144 can interrupt the output of the voice t2 at a more natural timing. Further, the voice output unit 144 can shorten the time required for the dialogue by interrupting the output of the voice t2 when the occupant 42 performs the approval operation.

  Returning to the flowchart of FIG. 5, when the determination unit 142 determines that the occupant 42 has performed a negative operation while the voice t2 is being output (S3 "No", S6 "Yes"), the voice output is interrupted. I will judge. In this case, the audio output unit 144 interrupts the output of the audio t2 (S7). Then, the dialogue control unit 145 executes a process corresponding to the negation of the voice t2 (S8).

  FIG. 8 is a diagram showing an example of the negative operation of the occupant 42 according to the present embodiment. As shown in FIG. 8, when the determination unit 142 determines that the occupant 42 performs a negative operation such as swinging his / her neck or the like in the middle of the voice t2, the voice output unit 144 interrupts the output of the voice t2. Further, the dialogue control unit 145 executes a process corresponding to the negation of the voice t2.

  As an example of the process corresponding to the denial of the voice t2, the dialogue control unit 145 causes the voice output unit 144 to output the voice t4 responding to the denial of the occupant 42. For example, as illustrated in FIG. 8, when the occupant 42 performs a negative operation while the voice output unit 144 outputs voice that the destination is A ranch, the dialogue control unit 145 performs voice output The part 144 is made to output an audio questioning the destination. The process corresponding to the negation of the voice t2 is not limited to this, and varies depending on the content of the output voice t2.

  When it is determined that the occupant 42 has performed a negative operation, the voice output unit 144 interrupts the output of the voice t2, and the dialog control unit 145 shifts to the subsequent processing, whereby the ECU 14 does not wait for the occupant 42. Can interact smoothly. Further, the voice output unit 144 can shorten the time required for the dialogue by interrupting the output of the voice t2 when the occupant 42 makes a negative operation.

  Returning to the flowchart of FIG. 5, when the determination unit 142 determines that the occupant 42 has made a speech operation while the voice t2 is being output (S6 “No”, S9 “Yes”), the voice output is interrupted I will judge. In this case, the audio output unit 144 interrupts the output of the audio t2 (S10).

  FIG. 9 is a view showing an example of the utterance operation of the occupant 42 according to the present embodiment. As illustrated in FIG. 9, when the determination unit 142 determines that the occupant 42 performs an utterance operation such as opening the mouth in the middle of the sound t2, the sound output unit 144 interrupts the output of the sound t2.

  The speech output unit 144 interrupts the output of the voice t2 when the occupant 42 makes an utterance operation, whereby the talker in the dialog changes from the ECU 14 to the occupant 42 at a natural timing. Therefore, the occupant 42 can speak smoothly without being disturbed by the voice t2. Further, since the voice uttered by the occupant 42 and the voice t2 output by the voice output unit 144 do not overlap, the voice recognition unit 143 can more accurately recognize the voice uttered by the occupant 42.

  In the present embodiment, when it is determined that the occupant 42 has made an utterance operation, the output of the voice t2 is interrupted and the process is ended, but after the output of the voice t2 is interrupted, the voice recognition by the voice recognition unit 143 is performed It may be in a waiting state. Further, while the output of the voice t2 is interrupted, the voice output unit is obtained when the obtaining unit 141 repeatedly obtains the captured image and repeats the determination process, and it is determined that the occupant 42 does not perform the speech operation. 144 may resume the output of the voice t2.

  Returning to the flowchart of FIG. 5, when the determination unit 142 determines that the occupant 42 performs the on-hold operation while the voice t2 is being output (S9 “No”, S11 “Yes”), the voice output is interrupted I will judge. In this case, the audio output unit 144 interrupts the output of the audio t2 (S12). Then, the process corresponding to the suspension is started (S13). In this embodiment. The process corresponding to the hold is an extension of the speech recognition time. During the speech recognition time, the speech recognition unit 143 waits for recognition that the speech of the occupant 42 is input to the speech input unit 24. When the voice of the occupant 42 is input, the voice recognition unit 143 recognizes the voice and specifies the content.

  In addition, when the determination unit 142 determines that the occupant 42 has performed an operation indicating that it can not hear the voice while the voice t2 is being output (S11 “No”, S14 “Yes”), the voice output is interrupted. to decide. In this case, the audio output unit 144 interrupts the output of the audio t2 (S15). Then, the sound output unit 144 returns to S1, and outputs again the sound t2 that has been output when it is determined that the occupant 42 has performed an operation indicating that it can not hear.

  FIG. 10 is a diagram showing an example of an operation representing inaudibility according to the present embodiment. As shown in FIG. 10, when the determination unit 142 determines that the occupant 42 performs an operation to indicate that it can not hear, such as opening its eyes, in the middle of the sound t2, the sound output unit 144 starts the sound t2 again from the beginning. Output.

  Referring back to the flowchart of FIG. 5, the determination unit 142 determines that the occupant 42 does not perform any of the approval operation, the negative operation, the utterance operation, the hold operation, and the operation indicating that it can not be heard (S14 “No” When the audio output unit 144 does not end the output of the audio t2 (S16 “No”), the processing of S2 to S16 is repeated.

  In addition, in the case where the determination unit 142 determines that the occupant 42 does not perform any of the approval operation, the negative operation, the speech operation, the hold operation, and the operation indicating that it can not be heard (S14 “No”), the voice t2 When the output of (S16 "Yes") is completed, the processing of the flowchart ends.

  When the determination unit 142 determines that the occupant 42 does not perform any of the approval operation, the negative operation, the speech operation, the hold operation, and the operation indicating that the user can not hear, and the output of the voice t2 ends, As shown in FIG. 6, the dialogue control unit 145 performs the subsequent processing according to the response by the voice uttered by the occupant 42 recognized by the voice recognition unit 143 or the response by the operation. For example, when the crew member 42 responds to the voice t2 shown in FIG. 6 as an approval response, the dialog control unit 145 executes a process corresponding to the approval for the voice t2. Further, when the occupant 42 gives a negative response to the voice t2, the dialog control unit 145 executes processing corresponding to the negative of the voice t2.

  Further, when the voice guidance continues and the voice output unit 144 outputs the next voice, the processing of the flowchart of FIG. 5 is repeatedly executed.

  The contents of the voices t1 to t4 output from the voice output unit 144 and the voices r1 to r3 uttered by the occupant 42 illustrated in FIGS. 6 to 10 are merely examples, and the present invention is not limited thereto.

  Further, the process corresponding to approval and the process corresponding to negation are not limited to the contents shown in FIGS. When it is determined that the occupant 42 has performed the approval operation or the negative operation while the voice that is not the content for requesting the approval or denial is output to the occupant 42 as illustrated in the voice t1 illustrated in FIG. The voice output unit 144 may simply interrupt the voice output.

  As described above, the ECU 14 according to the present embodiment interacts more smoothly with the occupant 42 in order to interrupt the voice output based on the operation and the state of the occupant 42 acquired from the image of the occupant 42 included in the captured image. be able to.

  Further, the ECU 14 of the present embodiment interrupts the voice output when it is determined that the occupant 42 has performed any of the speech operation, the approval operation, the negative operation, the hold operation, and the operation indicating that the user can not hear it. . Therefore, according to the ECU 14 of the present embodiment, it is possible to suppress the output of sound unnecessary for the occupant 42.

  For example, when the ECU 14 determines that the occupant 42 has made a speech operation, interrupting the output of the voice can suppress interference with the utterance of the occupant 42. Thus, according to the ECU 14 of the present embodiment, the occupant 42 can utter smoothly. Further, according to the ECU 14 of the present embodiment, when it is determined that the occupant 42 has performed the approval operation or the negative operation, the output of the voice is interrupted to suppress the output of the voice unnecessary for the occupant 42 and make the dialog The time required can be reduced and the dialogue can be conducted smoothly. Further, according to the ECU 14 of the present embodiment, the voice output is interrupted when it is determined that the occupant 42 has performed the hold operation, so that the voice is still determined while the occupant 42 is not making an approval or denial decision. It is possible to suppress the output of the Further, according to the ECU 14 of the present embodiment, when it is determined that the occupant 42 has performed an operation indicating that it can not hear, the output of the voice is interrupted, so that the voice output can be performed in a state where the occupant 42 can not hear the voice. It is possible to suppress continuation.

  Furthermore, according to the ECU 14 of the present embodiment, when it is determined that the occupant 42 has performed the approval operation, the output of the voice is interrupted, and the processing corresponding to the approval of the output voice is performed. Subsequent processing in line with the intention can be quickly initiated without waiting for the occupant 42. For this reason, according to the ECU 14 according to the present embodiment, it is possible to interact with the occupant 42 more smoothly.

  Furthermore, according to the ECU 14 of the present embodiment, when it is determined that the occupant 42 has performed a negative operation, the output of the voice is interrupted, and processing corresponding to the denial of the output voice is performed. Subsequent processing in line with the intention can be quickly initiated without waiting for the occupant 42. For this reason, according to the ECU 14 according to the present embodiment, it is possible to interact with the occupant 42 more smoothly.

  Furthermore, the ECU 14 according to the present embodiment extends the recognition time of the voice of the occupant 42 when it is determined that the occupant 42 performs the on-hold operation. For this reason, according to the ECU 14 of the present embodiment, it is possible to wait until the occupant 42 makes a decision, and it is possible to perform dialogue in accordance with the pace at which the occupant 42 responds.

  Furthermore, the ECU 14 according to the present embodiment re-outputs the voice that has been output, when it is determined that the occupant 42 has performed an operation indicating that it can not hear. According to the ECU 14 of the present embodiment, it is possible to smoothly interact with the occupant 42 by outputting again the voice that the occupant 42 could not hear. Further, according to the ECU 14 of the present embodiment, it is possible to suppress the occurrence of the trouble that the occupant 42 operates the re-output of the voice, and the interaction with the ECU 14 is re-started from the beginning.

  In the present embodiment, the voice output unit 144 and the dialogue control unit 145 are different functional units, but the functions of the voice output unit 144 and the dialogue control unit 145 may be configured to be provided by one functional unit. Further, the functions of the determination unit 142 and the dialogue control unit 145 may be configured to be included in one function unit.

(Modification 1)
In the above-mentioned embodiment, when the crew member 42 makes a wriggling motion, the motion is judged as the approval motion, and the output of the sound is interrupted. However, in general, human beings may operate continuously in a row as a compliment with the intention of prompting the speaker to speak. So, in this modification, when crew member 42 makes a fight, output of sound is not interrupted.

  From the image of the occupant 42 included in the captured image acquired by the acquisition unit 141, the determination unit 142 of the present modification example determines whether or not the occupant 42 has made a fight. More specifically, the determination unit 142 performs image processing on captured images across a plurality of frames to determine the continuous operation of the occupant 42 at a fixed time. When the occupant 42 shakes the neck vertically once, the determination unit 142 determines that the operation is an approval operation. In addition, when the occupant 42 shakes his / her neck a plurality of times in succession, the determination unit 142 determines that the motion is a compliment. In addition, the method of distinguishing an approval operation and a sumo wrestling is not limited to this. Further, the image processing for detecting the continuous motion of the occupant 42 may be performed by the acquisition unit 141. In addition, the determination unit 142 may determine that the occupant 42 is in conflict by processing the input image of the occupant 42 using a deep learning method such as RNN.

  When it is determined that the occupant 42 has made a fight, the determination unit 142 of the present modification example determines that the audio output is not interrupted. Then, when the determination unit 142 determines that the output of the audio is not interrupted, the audio output unit 144 continues the output of the audio.

  Further, the dialogue control unit 145 of the present modification associates the voice output by the voice output unit 144 with the timing when it is determined that the occupant 42 has made a fight during the output of the voice, and stores them in the storage unit 140. Do. For example, the voice output unit 144 stores the identification information for specifying the output voice, the output start time, and the time when it is determined that the occupant 42 has made a fight in the storage unit 140. The content stored in the storage unit 140 is an example, and the present invention is not limited to this.

  When it is determined that the crew member 42 has made a fight, it is highly likely that the crew member 42 agrees with the content of the outputted voice. Therefore, by storing the output voice and the timing of the reciprocation in association with each other, the occupant 42 agrees with the content of the sound output until the time when the occupant 42 is determined to have made a reciprocation. You can guarantee the credibility of what you have earned.

(Modification 2)
Although the above-mentioned embodiment explained as an example ECU14 which has a function which carries out voice recognition of the voice of crew member 42, ECU14 does not need to have a function of voice recognition. For example, when the determination unit 142 determines that the occupant 42 performs an operation such as approval or denial with respect to the sound output from the voice output unit 144, the ECU 14 does not have the voice recognition function. You can interact.

(Modification 3)
In the above-described embodiment, the processing after interrupting the output of the voice (for example, the processing corresponding to the approval and the processing corresponding to the negation) is determined in advance according to the content of the voice that has been output. However, the processing after interrupting the audio output is not limited to this.

  For example, the dialogue control unit 145 may determine processing after interrupting the output of voice using a deep learning method such as RNN. More specifically, the dialogue control unit 145 inputs, as deep learning input data, time-series data indicating the operation or state of the occupant 42 included in the captured image, and time-series data of audio output content. , And subsequent processing after interruption of speech may be determined. In addition, a captured image may be used as deep learning input data. Further, the deep learning process may be performed by the determination unit 142.

(Modification 4)
In the above embodiment, the process corresponding to the hold is an extension of the speech recognition time, but is not limited to this. For example, when it is determined that the passenger 42 is in the hold operation, the voice output unit 144 of this modification is different from the voice that is output when it is determined that the passenger 42 is in the hold operation. Output More specifically, the voice different from the voice that is output when it is determined that the occupant 42 performs the on-hold operation is a voice for making another proposal to the occupant 42.

  FIG. 11 is a diagram illustrating an example of processing corresponding to suspension according to the present modification. As shown in FIG. 11, in the process of inputting the destination by voice input, the voice output unit 144 sets “the first candidate specified by voice recognition of the voice recognized by the passenger 42 by voice recognition of the voice recognition unit 143. A voice t2 indicating "A ranch" is output. Here, when it is determined that the occupant 42 performs the holding operation, the voice output unit 144 outputs a voice t5 proposing “B ranch” as the second candidate.

  As a method of determining the second candidate, for example, the words of the first candidate and the second candidate may be specified in descending order of the possibility that the occupant 42 uttered from the speech recognition result by the speech recognition unit 143. Alternatively, the dialogue control unit 145 or the determination unit 142 is near the first destination (A ranch), and the place name, facility name, etc. of the same word category (such as ranch) are not shown. You may search from the information.

  As described above, since the ECU 14 of the present modification outputs a voice different from the voice output when the occupant 42 is determined to be on hold, the content of the voice originally output by the occupant 42 is If not approved or denied, presenting the other options can help the occupant 42 to make a decision.

  As mentioned above, although the embodiment of the present invention was illustrated, the above-mentioned embodiment and modification are an example to the last, and limiting the scope of the invention is not intended. The above embodiment and modifications can be implemented in other various forms, and various omissions, replacements, combinations, and changes can be made without departing from the scope of the invention. In addition, the configurations and shapes of the embodiments and the modifications may be partially replaced and implemented.

  Reference Signs List 1 vehicle 8 b voice output device 10 driving support system 14 ECU 15 imaging device 24 voice input unit 42 occupant 140 storage unit 141 acquisition unit 142 determination unit 142 143: speech recognition unit, 144: speech output unit, 145: dialogue control unit.

Claims (7)

An audio output unit that outputs audio to the occupant;
An acquisition unit that acquires an image of an occupant included in a captured image of an imaging device that captures an interior of a vehicle;
A determination unit that determines whether or not to interrupt the output of the sound based on the operation or state of the occupant included in the acquisition result of the acquisition unit;
The voice output unit interrupts the output of the voice when the determination unit determines that the output of the voice is interrupted.
Driving support device. The motion is any of a motion of speech, a motion of representing approval, a motion of representing negative intention, a motion of representing holding intention, or a motion of representing inability to hear the outputted voice.
The driving support device according to claim 1. A dialog control unit for executing processing corresponding to the approval of the output voice when it is determined that the occupant is performing an operation indicating the intention of the approval;
The driving support device according to claim 2. A dialog control unit for executing a process corresponding to the negation of the output voice when it is determined that the occupant is performing an action indicating the negative intention;
The driving assistance device according to claim 2 or 3. The voice recognition unit further includes a voice recognition unit that recognizes a voice uttered by the occupant with respect to the voice output by the voice output unit.
If it is determined that the occupant is operating to indicate the intention to suspend, the voice recognition unit extends the recognition time of the voice spoken by the occupant.
The driving assistance device according to any one of claims 2 to 4. When it is determined that the occupant is operating to indicate the intention to suspend, the voice output unit outputs when it is determined that the occupant is operating to express the intention to suspend. Output a voice different from the voice,
The driving assistance device according to any one of claims 2 to 4. When it is determined that the occupant is performing an action indicating that the voice can not be heard, the voice output unit determines that the occupant is performing an action indicating that the voice can not be heard. Re-output the voice that was being output
The driving assistance device according to any one of claims 2 to 6.

Images