JP6387287B2 - Unknown matter resolution processing system - Google Patents

Description

  The present invention relates to an unknown matter elimination processing system, and further relates to an unknown matter elimination processing device, an unknown matter elimination processing method, and an unknown matter elimination processing program.

  In recent years, smartphones, which are portable communication devices having both functions of a mobile phone terminal and a portable information terminal, have exploded in popularity. The spread of smartphones has a great influence on in-vehicle communication devices (sometimes referred to as in-vehicle devices or in-vehicle navigation devices) including audio functions, visual functions, navigation functions, and near field communication functions. Are connected to each other and work together, that is, smartphone cooperation has become the mainstream.

  For example, when a smartphone user (user) connects a smartphone to a vehicle-mounted device in a short-distance wireless connection in a vehicle such as an automobile, the smartphone-linked application (relay application software) operates, and the location (location information) found on a smartphone that is normally used You can easily set the destination by sending to the in-vehicle device. Thereby, the user of a smart phone can start immediately aiming at the destination displayed on the display of the vehicle equipment.

  In addition, mobile communication devices such as smartphones that have application software that has autonomous dialogue ability and virtual personality and that functions as an agent (anthropomorphic agent or communication agent) that mediates communication have become widespread.

  In agent-equipped smartphones and in-vehicle devices that operate in cooperation with such smartphones, when an unknown question (sometimes referred to as an unknown point) that a user wonders is inquired (questioned), this is triggered. The agent provides information on unknown matters. The user can resolve the unknown matter based on the provided information.

JP 2003-330487 A JP 2010-009413 A JP 2012-144066 A JP 2009-244204 A

  However, in the above-described unknown matter resolution technique, the user cannot clarify the unknown matter unless the agent actively asks the agent about the unknown matter.

  For example, when a user is driving a moving body such as a vehicle, manual operations such as taking notes on unknown matters and accessing agents are not performed smoothly, and it is possible to avoid problems. Absent.

  A first problem is to provide a technique that can solve an unknown matter while suppressing an action such as a user's manual operation.

The second problem is to provide a technology that enables an unknown matter to be resolved without the user actively asking questions about the unknown matter.

  In order to solve the above-described problem, an information processing apparatus as an unknown matter elimination processing apparatus includes a recognition unit that recognizes an image of a subject, an input unit that inputs sound, and a mouth of the subject by the recognition unit. First determination means for determining an utterance state based on the state of the subject and a facial expression based on the state of the subject's face by the recognition means; and based on a determination result of the first determination means, Second determination means for determining that an unknown matter has occurred in the subject when it is determined that there is no utterance state and the facial expression of the subject is not clear; the unknown matter has been generated by the second determination means; And specifying means for specifying the unknown matter based on the sound input by the input means; and output means for outputting information explaining the unknown matter specified by the specifying means.

  The above-described unknown matter elimination processing device can also be implemented as an unknown matter elimination processing method and an unknown matter elimination processing program.

  According to the disclosed unknown matter elimination processing device, unknown matters can be eliminated while suppressing actions such as user's manual operation. Further, even if the user does not actively ask questions about unknown matters, the unknown matters can be resolved.

  Other objects, features and advantages will become apparent upon reading the detailed description set forth below when taken in conjunction with the drawings and the appended claims.

The block diagram which shows the outline | summary structure of the unknown matter solution processing system of one embodiment. The figure for demonstrating the characteristic of the unknown matter solution processing system of one embodiment. The block diagram which shows the structure of the vehicle equipment in the unknown matter solution processing system of one embodiment. The block diagram which shows the structure of the server in the unknown matter solution processing system of one embodiment. The flowchart which shows the procedure of the unknown matter cancellation | release process of one Embodiment. The figure which shows the structure of the user management database and term database in the unknown matter solution processing system of one embodiment.

  Hereinafter, further detailed description will be given with reference to the accompanying drawings. The drawings show preferred embodiments. However, it can be implemented in many different forms and is not limited to the embodiments described herein.

[System Overview]
Referring to FIG. 1 for explaining an outline of a system in an embodiment, an unknown matter elimination processing system 1 includes an in-vehicle device (in-vehicle communication device) 2 installed in a room of a vehicle (moving body) such as an automobile. A smartphone (mobile communication device) 3 having both functions of a mobile phone terminal and a mobile information terminal, and a server (unknown matter resolution processing device) 4 are provided.

  In the unknown matter elimination processing system 1, the smartphone 3 and the vehicle-mounted device 2 are connected in a short-range wireless connection and operate in cooperation, and the smartphone 3 and the server 4 are connected via a wireless communication network (not shown).

As will be described in detail later, the in-vehicle device 2 includes at least one camera (in-vehicle camera) and at least one microphone (in-vehicle microphone). A smartphone 3 having one aspect as a Web (World Wide Web) client is equipped with a Web browser (for example, an HTML (Hyper Text Markup Language) browser, a VRML (Virtual Reality Modeling Language) browser, etc.), an agent function, a communication function, An information input function, an information output (visible display / audible display) function, an information designation function, and an information holding (memory) function are included.

  In addition, the server 4 cooperates with the in-vehicle device 2 and the smartphone 3 to perform an unknown matter resolution process. That is, referring to FIG. 1 and FIG. 2 together, in the unknown matter elimination processing system 1, the user (user) of the smartphone 3 moves the smartphone 3 and the vehicle-mounted device 2 in a coordinated manner so Distance wireless connection is made (21 in FIG. 2).

  Then, the user of the smartphone 3 activates an agent application (application software) on the smartphone 3 that has an autonomous dialogue ability and a virtual personality and functions as an agent (personification agent or communication agent) that mediates communication ( 22 in FIG.

  In the in-vehicle device 2 that operates in cooperation with the agent-equipped smartphone 3, the driver (driver) image signal and audio signal acquired through the camera and microphone are transmitted to the smartphone 3.

  When the server 4 determines that the driver does not understand the conversation in the vehicle interior based on the driver's image signal and voice signal transferred from the smartphone 3, the server 4 extracts unknown matters (unknown points). And in the server 4, the information explaining the extracted unknown point is transmitted to the smart phone 3 (23 in FIG. 2).

  In the smartphone 3, the unknown point explanation information transmitted from the server 4 is transferred to the in-vehicle device 2 via the agent.

  As a result, the agent of the vehicle-mounted device 2 explains the unclear point to the driver, so that the unclear point can be resolved without the driver, that is, the user actively questioning the unclear point to the agent (see FIG. 2). 24).

[Configuration of in-vehicle device]
FIG. 3 shows an example of the hardware configuration of the in-vehicle device (in-vehicle communication device) 2 in the unknown matter elimination processing system 1 shown in FIG.

The in-vehicle device 2 has a car navigation function for guiding a route to the current location or destination of the vehicle, a playback function for various audio / visual (AV) contents, a function for receiving broadcast waves, and a short-range communication function. doing. The in-vehicle device 2 is used in a state where it is installed near the center of a dashboard that can be easily reached by a driver or passenger in the vehicle. The in-vehicle device 2 can be realized by, for example, an in-vehicle navigation device of an AVN (Audio Visual Navigation) (registered trademark) integrated device manufactured by Fujitsu Ten Limited.

Specifically, the in-vehicle device 2 includes a control unit 100, a memory 101, a removable recording medium playback unit 102, a radio and television broadcast reception unit 103, a disk playback unit 104, a navigation unit 106, a VICS (Vehicle Information and Communication). System) information reception unit 107, GPS (Global Positioning System) information reception unit 108, distribution unit 109, audio adjustment unit 110, speaker 111, image adjustment unit 112, image output unit 113, display unit 114, remote control transmission / reception unit 115, remote control (Remote operation unit) 116 and operation unit 117 are included.

  Further, the in-vehicle device 2 includes at least one camera (in-vehicle camera) 118, at least one microphone (in-vehicle microphone) 119, a video / audio input unit 120, and a short-range communication unit 121, and wirelessly as necessary. A communication unit 122 is included.

  In the in-vehicle device 2, the control unit 100 comprehensively controls each unit of the in-vehicle device 2 or an external device based on a signal from each part of the in-vehicle device 2 or a connected external device. The control unit 100 includes, for example, a microcomputer (microcomputer), and each function is realized by the microcomputer executing various programs stored in the memory 101.

  The memory 101 is a storage unit that is mounted on the in-vehicle device 2 and stores various data and programs. The memory 101 includes, for example, a RAM (Random Access Memory) and a flash memory.

  The detachable recording medium playback unit 102 reads out and outputs data recorded on a connected detachable recording medium (not shown). The detachable recording medium is, for example, an SD card, a miniSD card, a microSD card, or another flash memory memory card. The detachable recording medium playback unit 102 includes a read signal decoding circuit and the like, and various operations such as on / off and reading position are controlled by a control signal from the control unit 100.

  The broadcast receiving unit (radio receiving unit) 103 selects and receives a broadcast wave of a specific frequency from AM or FM broadcast waves transmitted from each radio broadcast station received by the antenna, thereby receiving broadcast audio. Output a signal. The broadcast receiving unit (television receiving unit) 103 receives a broadcast wave having a specific frequency from the broadcast wave transmitted from each television broadcast station received by the antenna, and receives the received audio signal, video signal, and Output data signals. The broadcast receiving unit 103 includes a tuning circuit, a demodulation and decoding circuit, and various operations such as on / off and reception frequency are controlled by a control signal from the control unit 100.

The disc playback unit 104 reads data recorded on a disc such as a CD (Compact Disc), an MD (Mini Disc), a DVD (Digital Versatile Disc), and a Blu-ray disc by a pickup, and an audio signal based on the read data And video signals. The disk reproducing unit 104 includes an optical pickup, a pickup disk drive mechanism, a control circuit for the pickup disk drive mechanism, a read signal decoding circuit, and the like. The disc playback unit 104 is controlled in various operations such as on / off and reading position by a control signal from the control unit 100.

The navigation unit 106 displays the position of the host vehicle and the route to the destination on the map, provides direction guidance such as right and left turns by voice at an intersection, etc., and receives traffic information from the VICS information reception unit 107. Or receiving and displaying the vehicle position information from the GPS information receiving unit 108 to provide route guidance to the destination. The navigation unit 106 includes a hard disk that records map information used for navigation, a CPU (Central Processing Unit) that performs various arithmetic processes, and a memory such as a RAM that stores data for various processes. The navigation unit 106 controls various operations such as on / off in response to a control signal from the control unit 100.

  The VICS information receiving unit 107 receives traffic information related to a traffic information communication system (VICS: registered trademark), and outputs the received traffic information. The VICS information receiving unit 107 includes a receiver (FM receiver, radio beacon receiver, optical beacon receiver) that receives data from the traffic information communication system, a decoding circuit that decodes the received data, and the like.

  The GPS information receiving unit 108 detects the position of the vehicle based on the GPS signal from the GPS satellite, and outputs the detected current position information. The GPS information receiving unit 108 includes a receiving circuit that receives a GPS signal, a calculation unit that calculates the vehicle position based on the received GPS signal, and the like.

  The distribution unit 109 is a variety of sources designated to be output by the control signal of the control unit 100, that is, the removable recording medium playback unit 102, the broadcast reception unit 103, the disc playback unit 104, the navigation unit 106, and the video / audio. Audio signals and video signals (also referred to as image signals) from the input unit 120 and the like are output to the audio adjustment unit 110, the image adjustment unit 112, and the control unit 100. The distribution unit 109 includes a switch group including relays or switching transistors.

  Here, the distribution unit 109 outputs the audio signal and the image signal from the video / audio input unit 120 to the control unit 100. When the wireless communication unit 122 is provided, the distribution unit 109 outputs the audio signal and the image signal from the video / audio input unit 120 to the wireless communication unit 122 and the data signal from the wireless communication unit 122. Is output to the control unit 100.

  The sound adjustment unit 110 adjusts the volume and sound of the input sound signal according to the control signal of the control unit 100, and outputs the adjusted sound signal. The audio adjustment unit 110 includes a memory that stores audio data, an arithmetic circuit such as a digital signal processor that performs arithmetic processing on the audio data, an amplification or attenuation circuit, and a resonance circuit. The speaker 111 outputs the audio signal from the audio adjustment unit 110 as audio.

  The image adjustment unit 112 adjusts the brightness, color tone, contrast, and the like of the input image signal according to the control signal of the control unit 100, and outputs each adjusted image signal. The image adjustment unit 112 includes a calculation circuit such as a memory that stores image data and a digital signaling processor that performs calculation processing on the image data.

  The image output unit 113 inputs the image signal input from the image adjustment unit 112 and the display image signal to be displayed on the display unit 114 from the control unit 100 and performs processing such as image composition, and the image subjected to the processing The display unit 114 is driven based on the signal. The image output unit 113 includes, for example, an image ASIC that is an arithmetic processing circuit specialized for an image to be processed by arithmetic processing, a video memory that stores image data for image processing and output, and an image output video memory. An image driving circuit for driving the display unit 114 based on the stored image data is provided.

  The display unit 114 displays an image based on the image signal input from the image output unit 113. The display unit 114 is, for example, a liquid crystal display, an organic EL display, a plasma display, a cold cathode flat panel display, or the like. The display unit 114 includes a backlight, a liquid crystal panel, and a touch panel.

  Since the remote control transmission / reception unit 115, the remote control (remote operation unit) 116, and the operation unit 117 can be easily understood, detailed description thereof will be omitted.

  The camera (in-vehicle camera) 118 outputs an image signal of a driver photographed (captured) in real time to the video / audio input unit 120.

  The microphone (in-vehicle microphone) 119 outputs to the video / audio input unit 120 a driver's audio signal collected (sound collection) in real time.

The short-range communication unit 121 is, for example, a DSRC (Dedicated Short Range Comm
communication interface for short-range wireless communication such as unication, infrared communication, and Bluetooth (registered trademark). Here, the short-range communication unit 121 is short-range wirelessly connected to the smartphone 3.

  The wireless communication unit 122 includes a transmission / reception antenna, a radio frequency (RF) signal processing unit, a baseband (BB) signal processing unit, and an encoding / decoding unit. When the wireless communication unit 122 is provided, the in-vehicle device 2 and the server 4 can communicate without passing through the smartphone 3. Conversely, when this wireless communication unit 122 is omitted, when the in-vehicle device 2 and the server 4 communicate, it is necessary to pass through the smartphone 3.

[Server configuration]
FIG. 4 shows an example of the hardware configuration of the server (unknown matter elimination processing device) 4 in the unknown matter elimination processing system 1 shown in FIG.

  The server 4 is a server computer, and includes a CPU 401, a memory 402, various interfaces (I / F) 403, 405, 407, 409, 411, and 413, and peripheral devices 404 and 406 connected to the CPU 1 through these interfaces. , 408, 410, 412, and 414.

  Here, as examples of peripheral devices, a hard disk drive device 404, an input device 406, a display device 408, a communication network interface (I / F) 410, a removable storage medium drive device 412, and a memory card reader / writer 414 are shown. Has been.

  The CPU 401 executes application programs developed in the memory 402 and provides various functions of the server 4. The memory 402 holds a program in a format that can be executed by the CPU 401. The memory 402 holds data processed by the CPU 401. The memory 402 is a DRAM (Dynamic Random Access Memory), a ROM (Read Only Memory), or the like. However, a flash memory may be used as the memory 402.

  The hard disk drive 404 accesses the hard disk and stores data processed by the CPU 401, programs executed by the CPU 401, and the like. The input device 406 is a character input device such as a keyboard or a pointing device such as a mouse. The display device 408 is a liquid crystal display, an electroluminescence panel, or the like.

  The removable storage medium driving device 412 is a driving device such as a CD-ROM, DVD, flash memory card (SD memory card, USB memory card).

  The program executed by the CPU 401 is normally stored in the hard disk and expanded in the memory 402 through the communication network interface 410 or the removable storage medium driving device 412. The memory card reader / writer 414 inputs / outputs data between the loaded memory card and the CPU 401 in accordance with an instruction from the CPU 401.

  The server 4 has various databases, that is, a user management database 41 and a term database 42 in at least one of the hard disk and the memory 402.

  In order to logically realize the unknown matter resolution processing function described in detail later, the server 4 includes an unknown matter resolution processing program linked with various OS standard tools.

[Unknown matter elimination processing]
FIG. 5 shows an example of the procedure of unknown matter elimination processing in the server 4. Here, the server 4 cooperates with the in-vehicle device 2 and the smartphone 3 to describe in detail the case where the driver and passengers perform the unknown matter resolution processing at the time of conversation, but the modification processing described later is adopted. You can also

  Referring to FIG. 5 and the related diagram together, in the unknown matter elimination processing system 1, the user (that is, the driver) of the smartphone 3 who is the subject of the unknown matter elimination target operates the smartphone 3 and the in-vehicle device 2 in cooperation with each other. In the vehicle, both are short-range wirelessly connected (21 in FIG. 2).

  And the user of the smart phone 3 has the autonomous dialogue capability and virtual personality, and starts the agent application which functions as an agent which mediates communication in the smart phone 3 (22 in FIG. 2). In addition, the user of the smartphone 3 activates the agent application functioning as an agent that mediates communication with the agent of the smartphone 3 also in the in-vehicle device 2.

  In the in-vehicle device 2 that operates in cooperation with the agent-equipped smartphone 3, the control unit 100 captures the driver image acquired in real time by the video / audio input unit 120 through the camera (in-vehicle camera) 118 and the microphone (in-vehicle microphone) 119. Signals, audio signals, and the like are transmitted to the smartphone 3 via the short-range communication unit 121. Note that the image signal and audio signal of the driver transmitted to the smartphone 3 include the image signal and audio signal of a passenger who is having a conversation in the passenger compartment. Here, the camera 118 and the microphone 119 are arranged in the vicinity of the front part of the driver's seat, if necessary, in the vicinity of the rear seat.

  In the server 4 in which the unknown matter elimination processing program is resident, the CPU 401 (strictly speaking, the CPU 401 and other components cooperating) transmits the driver ID transferred from the smartphone 3 together with the user ID (user identification information). If it is determined that the driver does not understand the conversation with the passenger in the passenger compartment based on the image signal and the audio signal, an unknown point is extracted (specified). And in the server 4, CPU401 transmits the information (unknown point commentary information) explaining the extracted unknown point to the smart phone 3 (23 in FIG. 2). The driver's image signal and audio signal transferred from the smartphone 3 are associated with the user ID as image data and audio data in the user management database (DB) 41 (see FIG. 6A) in the memory 402. Stored.

  More specifically, in the server 4, the CPU 401 frequently opens and closes the driver's mouth to detect the driver's speech state (opening / closing degree) based on the driver's image signal received from the smartphone 3. (S51 in FIG. 5). If the determination is affirmative (YES), it is determined that there is no unknown point of the driver, and the process ends (S52). If the determination is negative (NO), the process proceeds to step S53. In addition, together with the processing S51, continuous recording of the driver's audio signal received from the smartphone 3, that is, storage in the user management database 41 is started.

  In the process S51, in order to detect the utterance state of the driver, an existing face recognition technique that detects the opening / closing degree of the mouth in real time from the face image can be applied.

  Next, based on the received driver image signal, the CPU 401 determines whether the facial expression of the driver is negative (in a moody state) in order to detect the driver's moodiness (S53). If the determination is negative, that is, if the facial expression is positive (in a mood state), it is determined that there is no unknown point of the driver, and the process ends (S54). When the determination is affirmative, it is assumed that there is an unclear point in the content of the conversation because the driver has not entered the conversation with the passenger, that is, the conversation is not established (S55).

  In step S53, in order to determine whether the facial expression of the driver is negative (momentary state), features such as eyebrows between eyebrows, wobbling of the viewpoint, pupil size, and movement of the nostrils are detected from the face image. In addition, existing facial expression estimation technology that estimates several facial expressions including joy, disgust, and expressionless expression in real time can be applied.

  Note that the processing order of the processing S51, S52 and the processing S53, S54 may be reversed.

  Next, the CPU 401 performs voice recognition when a negative facial expression is made based on the image data and voice data stored in the user management database 41 in real time, and the part where the content of the conversation is not known (unclear point) Is determined (S56). That is, the CPU 401 determines a point (unknown point) where the content of the conversation was not known when the occurrence of an unclear point was affirmed based on the image data and the sound data stored in the user management database 41 in real time. judge. For this determination, voice color may be considered together. Strictly speaking, the CPU 401 determines that the conversation content immediately before the negative expression (for example, 15 seconds before) could not be understood.

  Next, the CPU 401 refers to the term database 42 (FIG. 6B) to determine whether there is a word whose meaning can be understood when the voice data corresponding to the unknown point is read from the user management database 41 and converted into text. (S57). That is, the CPU 401 determines whether or not words that have been read out from the user management database 41 and converted into text are registered in the term database 42 in advance. If a negative determination is made in step S57, it is determined that an unknown point of the driver cannot be detected, and the process ends (S58). If the determination is affirmative in process S57, the process proceeds to process S59.

  Further, the CPU 401 determines with reference to the user management database 41 whether the detected conversation content is likely to be unknown to the driver (S59). For example, in the user management database 41, at least age, sex, and hobbies are registered as attribute information declared by the user when joining the unknown matter resolution processing system 1. In addition, for example, for initial setting, by performing a hobby genre, a questionnaire regarding the handling of information equipment, a simple ability test, etc., it is possible to divide the level by user, hobby genre, information equipment field Good. For example, drivers may be classified into levels according to the amount of reading, and a list of terms that may become unknown may be held in the database for each level. Further, for example, levels may be classified according to the driver's knowledge of information devices, and a list of terms that may be unknown may be held in the database for each level. If the determination in step S59 is negative, it is determined that the driver's unknown point cannot be detected, and the process ends (S60). If the determination in step S59 is affirmative, an unknown point is extracted (specified) from the detected conversation content (S61).

  Further, unless requested by the driver in advance, the CPU 401 transmits the unknown point comment information to the smartphone 3 so as to explain the unknown point to the driver in a form that only the passengers and other drivers know, and terminates the process. (S62). More specifically, for example, when an unknown point “car navigation” is identified in the process S61, the CPU 401 refers to the term database 42 and reads commentary information about “car navigation”. Then, the CPU 401 creates, for example, text-format unclear point explanation information “Unclear point / car navigation that you wondered is ...” and sends it to the smartphone 3.

  In the smartphone 3, the CPU (not shown) receives at least one text-format unknown point explanation information transmitted from the server 4 and sequentially transfers the information to the in-vehicle device 2 via the agent of the smartphone 3.

  In the vehicle-mounted device 2, the agent of the vehicle-mounted device 2 that has received the text unclear explanation information from the agent of the smartphone 3 under the control of the control unit 100 is unknown to the driver through a head-up display (HUD) (not shown). Since the points are explained (visually displayed), the driver, that is, the user, can resolve the unknown point without actively asking the agent about the unknown point (24 in FIG. 2).

  In addition, when an unclear point arises in a conversation with a passenger, the communication with the passenger is facilitated by explaining the unknown point to the driver.

[Modification 1]
For example, when a program is received by the broadcast receiving unit 103 of the in-vehicle device 2, an audio signal in the vehicle interior transmitted from the in-vehicle device 2 to the smartphone 3 includes the driver and the passenger who are talking in the vehicle interior. In addition to audio signals, audio signals (audio signals and sound signals) of programs are included. In the server 4, the CPU 401 discriminates and removes the audio signal by the unknown matter elimination processing program, and judges the content of the conversation between the driver and the passenger as in the above-described embodiment. Note that the in-vehicle device 2 may remove the audio signal from the driver and passenger's audio signal and then transmit the audio signal to the server 4.

[Modification 2]
For example, when the driver gets on the vehicle alone and the program is received by the broadcast receiving unit 103 of the in-vehicle device 2 without a conversation in the passenger compartment, the signal transmitted from the in-vehicle device 2 to the smartphone 3 includes In addition to the image signal, an audio signal (sound signal and sound signal) of the program is included. In the server 4, the CPU 401 can identify the corresponding part in the audio signal (sound signal) of the program received together with the image signal of the driver as an unknown matter, and can perform the unknown matter resolution processing.

  In the server 4, the CPU 401 can perform an unknown matter resolution process based on an image signal of the driver and a sound signal (for example, a siren of an emergency vehicle) that enters the room from the outside of the vehicle. In this case, for example, when the driver or passenger's conversation includes an expression indicated by a fire truck siren, the server 4 identifies where the fire occurred by associating the fire with the news. Explain. For example, the server searches news websites on the Internet, collects information on fires within a specified range from the current location of the vehicle specified by the GPS obtained from the agent, and converts the collected information into speech And then hand it over to the agent.

  In the server 4, when the driver operates a predetermined hard switch of the in-vehicle device 2, the CPU 401 may additionally specify the corresponding portion in the audio signal at that time as an unknown point.

[Modification 3]
The following method can be additionally applied to the unknown point extraction (specification) processing of the above-described processing S59 and S61 in the server 4.

  (1) An unknown point is specified based on data such as a heart rate and a sweating amount acquired from a wearable device worn by the driver. For example, the server 4 may determine that a question has occurred when the heart rate changes from the normal value or when the amount of sweating exceeds the normal value.

(2) The wearable device worn by the driver is used, and the uncertain point is identified by reading the shake like a lie detector.

  (3) An unknown point is specified with reference to the user management database 41 storing data (such as past utterance history, hobbies and preferences, age, and sex) for estimating personal vocabulary such as a driver. For example, a man in his 50s who likes baseball is predicted to be familiar with baseball-related conversations, and conversely, is not familiar with entertainment relations. In addition, in the user management database 41, individual weak fields are registered, and the server 4 holds a term database 42 for each field, and when it matches a term in the term database 42 in that field, It is also possible to make it an unknown point.

  (4) A user management database 41 is created in cooperation with a search word in a personal smartphone 3 such as a driver, and an unknown point is specified with reference to the user management database 41. In this case, the server 4 acquires the search word in the smartphone 3 and reflects it in the user management database 41.

  (5) Since it can be presumed that the recently started words (buzzwords) have a high unclear point rate, the server 4 creates a database and identifies the unclear points with reference to this database (term database 42).

  (6) An unknown point is specified by a gesture of the driver (for example, gesture of stroking the head three times).

[Modification 4]
The following method can be additionally applied to the explanation process of the unknown point of the process S62 in the server 4 described above.

  (1) The server 4 basically transmits unknown point commentary information to the driver so as to explain the unknown point so that only the passengers and other drivers can understand unless requested by the driver in advance. ing.

  Therefore, the server 4 detects an occupant based on the image data, and explains the unknown point to the driver when the driver gets on alone. In addition, when the driver gets off, the smartphone 3 is notified, or the driver's personal computer (PC) is notified by e-mail. The driver needs to specify one of these unknown point explanation methods in advance.

  (2) Even if there are passengers, if the driver permits, explain the unclear points in real time. For example, when the agent detects a passenger from the image data, the agent may inquire the driver of necessity of explanation.

  (3) Explain unclear points in the car in an awkward manner. This makes it unclear for the driver, but it gets ridiculous when others hear it.

  (4) The explanation of the unclear point can be triggered by an occasional request from the user. For example, the server 4 can use the time when the driver operates a predetermined hard switch of the in-vehicle device 2 as an opportunity to explain the unknown point.

  (5) Even when there is a passenger, the service can be provided depending on the relationship with the passenger (for example, a child).

(6) Even when there is a passenger, an explanation of unknown points is displayed in real time on a display that can be seen only by a driver such as a head-up display (HUD).

  (7) If the driver's unknown point cannot be extracted, the agent asks the driver what he did not understand in the conversation within a predetermined time by voice output, the driver answers by voice, and the voice is converted. The unclear point is explained with respect to the written text by the same procedure (S62 in FIG. 5) as in the above-described embodiment.

(8) If an unclear point comes out, explain in two steps (switchable in advance).
1st stage: A simple version of the bulletin level that emphasizes speed (using HUD, etc.).
Second stage: Detailed explanation (supplementary explanation) when the driver is riding alone.

  (9) The content is weighted at the time of explanation according to the genre of unknown points. For example, if the server 4 refers to the user management database 41 and is a man in his fifties who likes baseball, he can predict that he is familiar with baseball-related conversations and conversely he is not familiar with entertainment relations. In accordance with this result, the server 4 reads a simple explanation or detailed explanation from the term database 42 and weights the explanation contents.

[Modification 5]
The following content supplement method can be applied to the explanation process of the unknown point of the process S62 in the server 4 described above.

  A term database 42 for unknown points is created to understand the fields that individuals are not good at, and the fields they are not good at are explained in detail when content is provided. In this case, the server 4 determines that there is an unclear point but that it is interesting to come out in the conversation.

[Modification 6]
The following method can be additionally applied to each process described above in the server 4.

  (1) When an unclear point is explained in the in-vehicle device 2, content is additionally provided depending on the reaction of the user. The server 4 provides this additional service when it is determined from the facial expression of the user (positive (motivated state)) that the unknown point explained this time is likely to be more interesting.

  (2) When an unclear point is explained in the in-vehicle device 2, the user (driver) uses the conversation remind function to get unclear points when riding with a specific person (for example, a child). You can deepen your understanding by listening to the commentary again. In the server 4, when the driver operates a predetermined hard switch of the in-vehicle device 2, the CPU 401 specifies a corresponding portion in the audio signal at that time as a subject to be reminded and explains the unknown point again.

  (3) The server 4 enables the user to remember the content of the conversation (for example, the content spoken with the child) in the conversation remind function of (2) described above, and provides an opportunity for further conversation development. .

  (4) The server 4 can also provide an explanation of an unclear point appropriate to the user's age and hobbies / references with reference to the user management database 41 in the conversation remind function of (2) described above.

(5) Based on the user's hobbies and preferences, the content of the conversation that is not known is estimated, and the fact that it is not known is registered in the user management database 41. For example, if the server 4 refers to the user management database 41 and is a man in his fifties who likes baseball, he can predict that he is familiar with baseball-related conversations and conversely he is not familiar with entertainment relations. In accordance with this prediction, the server 4 estimates the content of the conversation that was not known, and registers the fact that it was not known in the user management database 41, so that it is utilized for the next unknown matter resolution processing.

  (6) For example, if it can be predicted that the radio broadcast song is unknown, information is provided. For example, when the music flows, the server 4 determines the driver's facial expression from the image data, and if the server 4 determines that a question has occurred, the server 4 may provide commentary information for each music data. The server 4 may hold melody data for specifying the melody for each piece of music data and a comment information database. And the server 4 should just match with the audio | voice of music, and melody data, and may identify as a music. And the server 4 should just acquire the description information of the specified music from a description information database, and should provide it to a driver.

  (7) After explaining the unknown, the agent confirms whether or not the user really did not know the unknown, and registers it in the user management database 41. In other words, the server 4 determines that the user is wondering about the unknown matter, but if it is not actually an unknown matter, the result is registered in the user management database 41 to be used for the next unknown matter resolution process. To do.

[Modification 7]
The following method can be additionally applied to each process described above in the server 4.

  Since the driver is seated in the driver's seat and is a user of the in-vehicle device 2 and the smartphone 3, the driver can register the user ID in the server 4 and become an unknown matter cancellation target person. However, the passenger ID cannot be directly registered in the server 4 like the driver.

  For this reason, in order for the fellow passenger to become a target for further solving the unknown matter receiving the explanation of the unknown point, it is only necessary to receive registration in the user management database 41 of the server 4 as the fellow passenger of the driver who registered the user ID. For example, when the server 4 detects that the driver has performed a predetermined designation operation in the in-vehicle device 2, the server 4 includes the passengers in the unknown matter elimination target person and performs the same processing to explain the unknown points.

[Modification 8]
In the above-described one embodiment and the unknown matter eliminating processing system 1 of the first to seventh modified examples, the server 4 performs an unknown matter eliminating process in cooperation with the in-vehicle device 2 and the smartphone 3. did.

  However, when the in-vehicle device 2 includes the wireless communication unit 122 for directly communicating with the server 4, the intervention of the smartphone 3 can be omitted. In this case, the user (that is, the driver) of the in-vehicle device 2 becomes the unknown matter elimination target person.

  Then, the user of the in-vehicle device 2 activates an agent application that functions as an agent that mediates communication with the server 4 in the in-vehicle device 2.

  In the vehicle-mounted device 2 equipped with the agent, the control unit 100 receives the image signal and audio signal of the driver acquired in real time by the video / audio input unit 120 through the camera (vehicle interior camera) 118 and the microphone (vehicle interior microphone) 119. Control is performed to transmit to the server 4 via the wireless communication unit 122 instead of the short-range communication unit 121. In addition, the control unit 100 receives the unknown point comment information transmitted from the server 4 via the wireless communication unit 122.

  Since the agent of the vehicle-mounted device 2 explains the unclear point to the driver, the unclear point can be resolved without the driver, that is, the user actively asking the agent about the unclear point.

  In the server 4 in which the unknown matter elimination processing program is resident, the CPU 401 determines whether the driver is in the vehicle interior based on the driver's image signal and audio signal transferred from the in-vehicle device 2 together with the user ID (user identification information). If it is determined that the conversation is not understood, unknown points are extracted (specified). Then, the server 4 transmits the extracted unknown point explanation information to the in-vehicle device 2.

  The unknown matter elimination processing including the above-described processing S51 to S62 in the server 4 is the same except for the intervention of the smartphone 3.

[Modification 9]
In the unknown matter elimination processing system 1 of the above-described embodiment and Modifications 1 to 8, the server 4 cooperates with the vehicle-mounted device 2 installed in a vehicle (moving body) such as an automobile, The case of performing the unknown matter elimination processing is illustrated.

  However, the target room for the unknown matter elimination process is not limited to the moving body, and the unknown matter elimination target person is not limited to the driver and the passenger. Further, an agent that mediates communication may be omitted.

[Other variations]
The processing in the above-described embodiment and each modification is provided as a computer-executable program, and can be provided via a non-transitory computer-readable recording medium such as a CD-ROM or a flexible disk, and further via a communication line.

  In addition, each process in the above-described embodiment and each modification can be performed by selecting and combining any or all of the processes.

[Others]
The following additional remarks are disclosed with respect to the above-described embodiment and modifications.

(Appendix 1)
Means for receiving an image signal of an unknown matter elimination target person existing in a specific room and determining an utterance state based on an opening / closing degree of the mouth of the unknown matter elimination target person included in the received image signal;
Means for determining whether or not an unknown matter has occurred based on the facial expression of the person to be solved for the unknown matter contained in the received image signal;
Means for determining the occurrence of an unknown matter when the unknown matter elimination target person is not in the utterance state and determines a moody state;
Means for identifying a corresponding portion in any sound of the audio signal and the sound signal received together with the image signal as an unknown matter;
Means for transmitting information explaining the identified unknown matter;
An unknown matter elimination processing device comprising:

(Appendix 2)
A means for determining whether or not a word obtained by reading out sound data corresponding to an unknown matter from the first storage means into a text is registered in the second storage means;
APPENDIX 1 Unknown matter elimination processing device.

(Appendix 3)
Further comprising means for referring to the first storage means to determine whether the detected conversation content is likely to be an unknown matter for the unknown matter elimination target person,
The unknown matter elimination processing device according to attachment 2.

(Appendix 4)
The specific room is a room of a moving body including a vehicle, the unknown matter elimination target person is at least one of a driver and a passenger, and the image signal and the audio signal are transmitted to at least one of the driver and the passenger. Corresponding,
APPENDIX 1 Unknown matter elimination processing device.

(Appendix 5)
Computer
Receiving an image signal of an unknown matter elimination target person existing in a specific room, and determining an utterance state based on an open / closed degree of the mouth of the unknown matter elimination target person included in the received image signal;
Determining whether or not an unknown matter has occurred based on a facial expression of the person to be solved for the unknown matter included in the received image signal;
Determining the occurrence of an unknown matter when the unknown matter elimination target person is not in the utterance state and determines the moody state;
Identifying a corresponding location in any sound of the audio signal and the sound signal received together with the image signal as an unknown matter;
Sending information explaining the identified unknown matter;
Unknown matter resolution processing method to execute.

(Appendix 6)
On the computer,
Receiving an image signal of an unknown matter elimination target person existing in a specific room, and determining an utterance state based on an open / closed degree of the mouth of the unknown matter elimination target person included in the received image signal;
Determining whether or not an unknown matter has occurred based on a facial expression of the person to be solved for the unknown matter included in the received image signal;
Determining the occurrence of an unknown matter when the unknown matter elimination target person is not in the utterance state and determines the moody state;
Identifying a corresponding location in any sound of the audio signal and the sound signal received together with the image signal as an unknown matter;
Sending information explaining the identified unknown matter;
An unknown matter elimination processing program that executes

1 Unknown matter resolution processing system 2 In-vehicle device (in-vehicle communication device)
3 Smartphone (mobile communication device)
4 server (unknown matter solution processing device)

Claims (4)

Recognition means for recognizing an image of the subject;
Input means for inputting sound;
First determination means for determining an utterance state based on the state of the subject's mouth by the recognition means and an expression based on the state of the face of the subject by the recognition means;
A second determination that an unknown matter has occurred in the subject when it is determined that there is no utterance state of the subject and the facial expression of the subject is not clear based on the judgment result of the first judging means; Determination means;
Third determination means for determining whether a word obtained by converting the sound data input when it is determined that the unknown matter has occurred is registered in the storage means;
In the case where the word is registered in the storage unit, a first determination is made as to whether or not the registered word is likely to be the unknown matter according to attribute information set for each target person. 4 determination means;
Specifying means for specifying, as the unknown matter , the word that is determined to be highly likely to be the unknown matter by the fourth determining means;
Output means for outputting information explaining the unknown matter specified by the specifying means;
An information processing apparatus comprising: The target person is at least one of a driver and a passenger existing in a room of a moving body including a vehicle, and the image and the sound correspond to at least one of the driver and the passenger.
The information processing apparatus according to claim 1. Computer
A recognition step for recognizing an image of the subject;
An input step for inputting sound;
A first determination step of determining an utterance state based on the mouth state of the subject by the recognition step and a facial expression based on the face state of the subject by the recognition step;
A second determination that an unknown matter has occurred in the subject when it is determined that the subject does not speak and the facial expression of the subject is not clear based on the determination result of the first determination step. A determination step;
A third determination step of determining whether a word obtained by converting the sound data input when it is determined that the unknown matter has occurred is registered in a storage unit;
In the case where the word is registered in the storage unit, a first determination is made as to whether or not the registered word is likely to be the unknown matter according to attribute information set for each target person. 4 determination steps;
A specifying step of specifying, as the unknown item , the word that has been determined to be likely to be the unknown item in the fourth determining step;
An output step of outputting information explaining the unknown matter specified by the specifying step;
The processing method characterized by performing. On the computer,
A recognition step for recognizing an image of the subject;
An input step for inputting sound;
A first determination step of determining an utterance state based on the mouth state of the subject by the recognition step and a facial expression based on the face state of the subject by the recognition step;
A second determination that an unknown matter has occurred in the subject when it is determined that the subject does not speak and the facial expression of the subject is not clear based on the determination result of the first determination step. A determination step;
A third determination step of determining whether a word obtained by converting the sound data input when it is determined that the unknown matter has occurred is registered in a storage unit;
In the case where the word is registered in the storage unit, a first determination is made as to whether or not the registered word is likely to be the unknown matter according to attribute information set for each target person. 4 determination steps;
A specifying step of specifying, as the unknown item , the word that has been determined to be likely to be the unknown item in the fourth determining step;
An output step of outputting information explaining the unknown matter specified by the specifying step;
A processing program characterized by causing