JP6787957B2 - Utterance control device, utterance control method, and utterance control program - Google Patents

Description

The present invention relates to an utterance control device, an utterance control method, and an utterance control program.

Conventionally, an utterance control device for controlling an utterance from an apparatus is known. For example, in Patent Document 1, an operation log showing an operation status of a device is analyzed to determine the busyness of a user who uses the device, and an utterance that controls the utterance timing from the device according to the determined busyness. The control device is disclosed.

JP-A-2017-151718

However, the utterance control device described in Patent Document 1 controls the utterance timing under preset conditions based on the operating status of the device, and may not be an appropriate utterance timing depending on the user of the device. There was room for improvement.

The present application has been made in view of the above, and an object of the present application is to provide an utterance control device, an utterance control method, and an utterance control program capable of more appropriately determining the utterance timing.

The utterance control device according to the present application considers the reaction of the past user to the utterance from the voice output device based on the context acquisition unit that acquires the context information about the user and the context information acquired by the context acquisition unit. It is provided with a timing determination unit that determines the timing of the utterance.

According to one aspect of the embodiment, it is possible to provide an utterance control device, an utterance control method, and an utterance control program capable of more appropriately determining the utterance timing.

FIG. 1 is a diagram showing a configuration example of an information processing system according to an embodiment. FIG. 2 is an explanatory diagram of the utterance control process according to the embodiment. FIG. 3 is a diagram showing a configuration example of the smart speaker according to the embodiment. FIG. 4 is a diagram showing an example of an utterance table according to the embodiment. FIG. 5 is a diagram showing a configuration example of the information providing device according to the embodiment. FIG. 6 is a diagram showing an example of an utterance table stored in the utterance table storage unit according to the embodiment. FIG. 7 is a diagram showing an example of a content table stored in the content storage unit according to the embodiment. FIG. 8 is a diagram showing an example of a voice advertisement table stored in the voice advertisement storage unit according to the embodiment. FIG. 9 is a diagram showing an example of a user information table stored in the user information storage unit according to the embodiment. FIG. 10 is a flowchart (No. 1) showing an example of the utterance control process by the information processing system according to the embodiment. FIG. 11 is a flowchart (No. 2) showing an example of the utterance control process by the information processing system according to the embodiment. FIG. 12 is a flowchart showing an example of output control processing by the information processing system according to the embodiment. FIG. 13 is a flowchart showing an example of voice information effect determination processing by the information processing system according to the embodiment. FIG. 14 is a diagram showing an example of the hardware configuration of the computer that executes the program.

Hereinafter, the utterance control device, the utterance control method, and the embodiment for implementing the utterance control program (hereinafter referred to as “the embodiment”) according to the present application will be described in detail with reference to the drawings. It should be noted that this embodiment does not limit the utterance control device, the utterance control method, and the utterance control program according to the present application.

[1. Information provision system]
FIG. 1 is a diagram showing a configuration example of an information processing system according to an embodiment. As shown in FIG. 1, the information processing system 100 according to the embodiment includes a smart speaker 1, an information providing device 2, a terminal device 3, and a plurality of devices 4 ₁ to 4 _n (n is an integer of 2 or more). , A plurality of sensor devices 5 _{1 to} 5 _m (m is an integer of 2 or more). Hereinafter, when each of the devices 4 ₁ to 4 _n is shown without distinction, it is described as the device 4, and when each of the sensor devices 5 _{1 to} 5 _m is shown without distinction, it is described as the sensor device 5.

The smart speaker 1, the information providing device 2, the terminal device 3, the device 4, and the sensor device 5 are connected to each other so as to be able to communicate with each other wirelessly or by wire via the network 6. The network 6 is, for example, a LAN (Local Area Network) or a WAN (Wide Area Network) such as the Internet, and is composed of one or more networks.

Smart speaker 1, the terminal device 3, device 4, and the sensor device 5 is arranged in the area AR ₁ surrounding the user _{U 1.} Region AR ₁ is, for example, the room or home of user U ₁ . Furthermore, also each of the regions _AR 2 to Ar _k of the illustrated non Although the user _U 2 _~U k (k is an integer of 2 or more), as in the area AR _1, the smart speaker 1, the terminal device 3, device 4 , And the sensor device 5 is arranged. Hereinafter, when each of users U _{1 to} U _k is shown without distinction, it is described as user U.

The smart speaker 1 is a speaker that can use the AI (Artificial Intelligence) assistant function corresponding to the interactive voice operation, and the user U acquires various information by interacting with the smart speaker 1. Can be done. For example, the smart speaker 1 transmits input information indicating an instruction from the user U to the information providing device 2, and the content (for example, music, news, etc.) provided from the information providing device 2 according to the input information via the network 6. It is possible to acquire various information such as traffic information and weather, and output the content acquired from the built-in audio output device.

Further, the smart speaker 1 can control the device 4 according to an instruction from the user U. For example, when the device 4 is a lighting device, the smart speaker 1 can control the on / off of the device 4 which is a lighting device according to an instruction from the user U.

The information providing device 2 can provide the content according to the instruction from the user U to the smart speaker 1 via the network 6 based on the information output from the smart speaker 1. For example, the information providing device 2 can determine the instruction of the user U based on the utterance information of the user U transmitted from the smart speaker 1, and provide the content according to the determined instruction to the smart speaker 1.

The terminal device 3 is realized by, for example, a smartphone, a desktop PC (Personal Computer), a notebook PC, a tablet terminal, a mobile phone, a PDA (Personal Digital Assistant), or the like. Such a terminal device 3 has, for example, a plurality of applications including an information notification application, and acquires and displays the content notified from the information providing device 2. The terminal device 3 can notify the smart speaker 1 and the information providing device 2 of information indicating the operating state of the terminal device 3 and information indicating the operation history of the terminal device 3 by the user U.

The device 4 is a device existing around the user U. For example, the device 4 includes various devices such as a refrigerator, a lighting device, a washing machine, an air conditioner, a television receiver, a dishwasher, a dish dryer, an electromagnetic cooker, and a microwave oven. The device 4 can notify the smart speaker 1 and the information providing device 2 of information indicating the operating state of the device 4 and information indicating the operation history of the device 4 by the user U.

The sensor device 5 is a sensor existing around the user U. The sensor device 5 includes, for example, sensors such as a temperature sensor, a humidity sensor, an illuminance sensor, a pressure sensor, and an open / close sensor that detects opening / closing of a door. Further, the sensor device 5 includes an imaging unit that images the user U. The sensor device 5 notifies the smart speaker 1 and the information providing device 2 of sensor information such as temperature measurement value, humidity measurement value, illuminance measurement value, atmospheric pressure measurement value, door opening / closing information, and captured image information. Can be done. The sensor device 5 may be built in the smart speaker 1 or the device 4.

The information processing system 100 according to the embodiment can execute an utterance control process for controlling spontaneous utterances from the voice output device of the smart speaker 1. The utterance control process will be described below. FIG. 2 is an explanatory diagram of the utterance control process according to the embodiment. In the example shown in FIG. 2, the utterance control process is executed by the smart speaker 1 which is an example of the utterance control device.

The smart speaker 1 executes a context acquisition process for acquiring context information regarding the user U (step S1). The context information regarding the user U is information indicating the context regarding the user U. For example, a context relating to user U (hereinafter, may be simply referred to as a context) is a situation relating to user U. For example, the context includes a situation relating to the user U, such as the situation of the user U, the usage of the device (eg, terminal device 3 or device 4) by the user U, and the situation around the user U.

The situation of the user U includes, for example, the conversation state of the user U, the utterance content of the user U, the motion state of the user U, the current position of the user U, the attribute state of the user U, the emotional state of the user U, and the like. The usage status of the device by the user U includes, for example, the operation history and usage time of the device by the user U.

The situation around the user U includes, for example, the existence and state of another person around the user U, the physical environment in which the user U is placed, the social environment in which the user U is placed, and the like. The physical environment in which the user U is placed includes the brightness, temperature, humidity, weather, and the like around the user U. The social environment in which the user U is placed includes the operating state of transportation around the user U, the holding state of events around the user U, the day of the week (including the distinction between holidays and weekdays, for example), and the like.

For example, the smart speaker 1 can acquire information indicating the usage status of the terminal device 3 and the device 4 by the user U as context information from the terminal device 3 and the device 4. Further, the smart speaker 1 can acquire information indicating the physical environment in which the user U is placed as context information from the sensor device 5 arranged around the user U. Further, the smart speaker 1 can acquire the utterance of the user U from a voice input device (not shown), and can obtain information indicating the conversation state of the user U from the acquired utterance of the user U. Further, the smart speaker 1 can acquire sounds other than the user U's utterance from a voice input device (not shown), and can obtain information indicating the state of the surrounding sound from the acquired sounds other than the user U's utterance.

Further, the smart speaker 1 is provided with information indicating the social environment in which the user U is placed from the information providing device 2 via the network 6 (for example, the operating state of transportation) and the physical environment in which the user U is placed (for example, the weather). ) Can be obtained. Further, when the terminal device 3 or the device 4 transmits information indicating the usage status of the terminal device 3 or the device 4 by the user U to the information providing device 2, the smart speaker 1 is the user from the information providing device 2 via the network 6. It is also possible to acquire information indicating the usage status of the terminal device 3 and the device 4 by U.

The smart speaker 1 repeatedly executes the above-mentioned context acquisition process. The smart speaker 1 executes a timing determination process for determining the utterance timing in consideration of the past reaction of the user U to the utterance from the voice output device 11 based on the context information acquired by the context acquisition process that is repeatedly executed. (Step S2). The utterance timing is the timing at which the smart speaker 1 spontaneously utters without being instructed by the user U.

The smart speaker 1 can determine whether or not the current context is a context suitable for the utterance based on the reaction information and the context information indicating the reaction of the user U to the past utterance from the voice output device 11. it can.

For example, the smart speaker 1 has a model (hereinafter, may be referred to as a timing determination model) generated by machine learning using reaction information as teacher data and context information as a feature amount. The smart speaker 1 can determine whether or not the current context is a context suitable for utterance by inputting the context information acquired in the context acquisition process into the timing determination model.

When the current context regarding the user U is a context suitable for utterance, the smart speaker 1 determines that it is time to utter, and performs an output control process for outputting a spontaneous utterance from the voice output device 11 (step). S3). As a result, the voice output device 11 outputs a spontaneous utterance (step S4).

In this way, since the smart speaker 1 determines the timing of spontaneously speaking from the smart speaker 1 in consideration of the past reaction of the user U to the speech from the voice output device 11, the smart speaker 1 speaks under preset conditions. The utterance from the smart speaker 1 can be performed at a more appropriate timing as compared with the case of controlling the timing.

In the timing determination process, the smart speaker 1 can determine the timing of utterance for each content that can be output from the voice output device 11. As a result, the smart speaker 1 can speak at a timing suitable for each content.

[2. Configuration of smart speaker 1]
Next, the configuration of the smart speaker 1 according to the embodiment will be specifically described. FIG. 3 is a diagram showing a configuration example of the smart speaker 1 according to the embodiment. As shown in FIG. 3, the smart speaker 1 includes a communication unit 10, a voice output device 11, a voice input device 12, an imaging unit 13, a storage unit 14, and a control unit 15.

The communication unit 10 is a communication interface capable of communicating with devices such as an information providing device 2, a terminal device 3, a device 4, and a sensor device 5 via a network 6. The control unit 15 can send and receive information to and from the information providing device 2, the terminal device 3, the device 4, and the sensor device 5 via the communication unit 10. The smart speaker 1 may be configured to communicate with the terminal device 3, the device 4, and the sensor device 5 by a communication unit other than the communication unit 10.

For example, when the network 6 is composed of a LAN and a WAN and the communication unit 10 is connected to the LAN, the control unit 15 transmits and receives information to and from the terminal device 3, the device 4, and the sensor device 5 via the LAN. , Sends and receives information to and from the information providing device 2 via LAN and WAN.

The voice output device 11 vibrates according to the electric signal output from the control unit 15 to output a sound wave corresponding to the electric signal to the outside of the smart speaker 1. The audio output device 11 includes, for example, a diaphragm and a drive mechanism that vibrates the diaphragm in response to an electric signal. In the example shown in FIG. 3, only one audio output device 11 is shown, but the smart speaker 1 may be provided with a plurality of audio output devices 11.

The voice input device 12 is a microphone, converts sound waves input from the outside into an electric signal, and outputs the converted electric signal to the control unit 15. In the example shown in FIG. 3, only one voice input device 12 is shown, but the smart speaker 1 may be provided with a plurality of voice input devices 12.

The imaging unit 13 has, for example, a COMS (Complementary Metal Oxide Semiconductor) image sensor, and images the surroundings of the smart speaker 1. The imaging unit 13 outputs the imaging information, which is the imaging result, to the control unit 15. The captured information includes information on a captured image around the smart speaker 1.

The storage unit 14 stores the operation history 20 of the user U to the smart speaker 1 and the utterance table 21 for spontaneously speaking.

The operation history 20 is information in which the operation content and the operation time for the smart speaker 1 are associated with each operation of the user U. The operation contents include, for example, a voice operation that starts output of various contents (for example, schedule, mail, news, music, traffic information, etc.) from the voice output device 11, and output of various contents from the voice output device 11. Various operations such as voice operation to stop are included.

FIG. 4 is a diagram showing an example of the utterance table 21 according to the embodiment. The utterance table 21 shown in FIG. 4 includes information in which the “content ID”, the “utterance content”, and the “model” are associated with each other. The "content ID" is identification information unique to the content.

The "utterance content" is, for example, utterance information for making an inquiry as to whether or not the content can be used. For example, when the content is news, the "utterance content" is "How about today's news?", "There are" X "news", and so on. In addition, the "utterance content" is, for example, "the mail has been sent by" X "" when the content is an email, and when the content is a schedule, "there are plans for" X "today." And so on. The information of "X" can be acquired by the control unit 15 from the information providing device 2 and added to the utterance content.

Further, the "utterance content" may be the content itself. In this case, the utterance content is not set in the utterance table 21, and the control unit 15 acquires the content from the information providing device 2 based on the content ID.

The "model" is a timing determination model generated based on the reaction of the past user U to the utterance and the context information about the past user U at the time of the utterance, and is different for each content. For example, the timing determination model is a regression model in which the past reaction of the user U to the utterance is used as the objective variable and the context related to the user U is used as the explanatory variable. The objective variable is also called teacher data, and the explanatory variable is also called feature or feature quantity. In the timing determination model, the larger the number of explanatory variables, the better the accuracy, but the number of explanatory variables may be one.

The control unit 15 shown in FIG. 3 includes an input processing unit 31, an information output unit 32, an information acquisition unit 33, an output processing unit 34, a context acquisition unit 35, and a timing determination unit 36. The input processing unit 31 recognizes the voice of the user U from the electric signal output from the voice input device 12.

Further, the input processing unit 31 determines the gesture of the user U from the imaging information output from the imaging unit 13. The input processing unit 31 can acquire imaging information from the sensor device 5 including the imaging unit, and can determine the gesture of the user U based on the acquired imaging information.

Further, the input processing unit 31 can detect the movement of the mouth of the user U from the imaging information output from the imaging unit 13, and determine the content of the lip-synching (silent vocalization) of the user U from the detected movement of the mouth. it can. That is, the input processing unit 31 can read the lips from the image pickup information output from the image pickup unit 13. The input processing unit 31 uses the result of lip reading as the utterance information of the user U. If the voice of the user U can be identified, the input processing unit 31 does not read the lip.

When the input processing unit 31 recognizes that the user U has spoken the wake-up word, the information output unit 32 outputs the utterance information of the user U following the wake-up word to the information providing device 2. The utterance information may be the voice information of the user U itself or text information. Further, the information output unit 32 outputs the gesture information, which is the gesture information determined by the input processing unit 31, to the information providing device 2. The wake-up word is a word preset in the smart speaker 1, but the information output unit 32 may output the utterance information and the gesture information of the user U to the information providing device 2 regardless of the utterance of the wake-up word. it can.

Further, the information output unit 32 can continuously and repeatedly output the context information acquired by the context acquisition unit 35 to the information providing device 2 when, for example, the context information is not acquired from the information providing device 2.

Further, the information output unit 32 inputs to the voice input device 12 during a preset period (hereinafter, may be referred to as an advertisement output period) from the time when the sound advertisement output from the voice output device 11 is started. The sound information including the sound to be produced and the imaging information including the imaging information output from the imaging unit 13 are output to the information providing device 2. The preset period is, for example, a period that starts when the output of the voice advertisement from the voice output device 11 is started and ends when the output of the voice advertisement is stopped or ended, or a voice output. It is a period that starts when the output of the voice advertisement from the device 11 is started and ends after a certain period of time after the output of the voice advertisement is stopped or ended. The information output unit 32 acquires sound information including the sound input to the voice input device or the sensor device 5 of the terminal device 3 from the terminal device 3 or the sensor device 5 during the advertisement output period, and sends the information providing device 2 to the information providing device 2. It can also be output.

The information acquisition unit 33 can acquire model information including the timing determination model for each content from the information providing device 2, and set the acquired timing determination model information in the utterance table 21. In addition, the information acquisition unit 33 acquires the content from the information providing device 2. The output processing unit 34 converts the content acquired by the information acquisition unit 33 into an electric signal and outputs the content to the audio output device 11. As a result, the content is output as sound from the smart speaker 1.

When the content is acquired as character information from the information providing device 2, the output processing unit 34 converts the character information into a voice signal (electric signal) by voice synthesis processing and outputs the content to the voice output device 11. Further, when the content is acquired as audio information from the information providing device 2, the output processing unit 34 converts the audio information into an audio signal (electric signal) by digital-to-analog conversion and outputs the content to the audio output device 11.

The context acquisition unit 35 executes a content acquisition process for acquiring context information regarding the user U. The context acquisition unit 35 can acquire context information directly from the terminal device 3, the device 4, and the sensor device 5 or via the information providing device 2.

Specifically, the context acquisition unit 35 can acquire surrounding information indicating the surrounding situation of the user U as at least a part of the context information. For example, the context acquisition unit 35 can acquire surrounding information from sensor information output from one or more sensor devices 5 existing around the user U.

The sensor information includes, for example, at least one of illuminance information indicating the brightness around the user U, temperature information indicating the air temperature around the user U, and humidity information indicating the humidity around the user U. The context acquisition unit 35 can acquire illuminance information, temperature information, and humidity information as ambient information.

Further, the context acquisition unit 35 can acquire surrounding information from device information indicating the state of one or more devices existing around the user U. Here, the one or more devices are one or more devices among the smart speaker 1, the terminal device 3, and the device 4. The device information is, for example, information indicating the operating state of the device such as turning on / off the device, or information indicating the operating state.

For example, in the case of the terminal device 3, the information indicating the operating state includes the type of the application displayed on the terminal device 3 and the content of the displayed content. When the device is an air conditioner, the information indicating the operating state includes information such as the set air volume and the set temperature of the air conditioner.

Further, the context acquisition unit 35 can acquire surrounding information from the operation history information indicating the operation history of one or more devices existing around the user U. The one or more devices are one or more devices among the smart speaker 1, the terminal device 3, and the device 4. The operation history information includes, for example, information associated with the operation content of the device and the operation time for each operation of the user U. The context acquisition unit 35 can acquire the operation history information of the smart speaker 1 from the storage unit 14.

Further, the context acquisition unit 35 can acquire, for example, imaging information indicating the imaging information of the user U from the imaging unit 13, the terminal device 3, the device 4, or the sensor device 5. The context acquisition unit 35 can acquire context information such as information indicating the status of the user U from the acquired imaging information.

Further, the context acquisition unit 35 obtains, for example, the conversation state of the user U, the utterance state of the user U, and the sound around the user U (the sound of the device 4) from the sound information including the sound input to the voice input device 12. Context information such as) can be acquired.

The timing determination unit 36 determines the utterance timing, which is the timing of the utterance in consideration of the past reaction of the user U to the utterance from the voice output device 11, based on the context information acquired by the context acquisition unit 35.

For example, the timing determination unit 36 inputs the context information acquired by the context acquisition unit 35 as input information into the timing determination model for each content included in the utterance table stored in the storage unit 14, and performs an operation using the model. Do. The timing determination unit 36 determines whether or not the score, which is the calculation result of the timing determination model, is a preset threshold value.

When the timing determination unit 36 determines that the score, which is the calculation result of the timing determination model, is equal to or greater than the threshold value, the timing determination unit 36 outputs the utterance content associated with the timing determination model whose score is equal to or greater than the threshold value in the utterance table 21. Judge that it has become. Further, when there are two or more timing determination models having a score equal to or higher than the threshold value, the timing determination unit 36 outputs the utterance content associated with the timing determination model having the highest score among the timing determination models having the score equal to or higher than the threshold value. Judge that it is time to do.

Even if the timing determination unit 36 determines that the score, which is the calculation result of the timing determination model, is equal to or greater than the threshold value, it cannot determine that it is time to output the utterance content. For example, when the timing determination unit 36 does not elapse a preset period (hereinafter referred to as an output prohibition period) since the last output of the utterance content associated with the timing determination model whose score is equal to or higher than the threshold value. It cannot be determined that the utterance timing has come.

Further, the timing determination unit 36 presets the utterance content associated with the timing determination model whose score is equal to or higher than the threshold value from the current time to a period before a preset period (hereinafter, referred to as a set period). When the number of times (hereinafter referred to as the output upper limit number of times) is exceeded, it can not be determined that the utterance timing has come.

Further, even when the timing determination unit 36 determines that the score, which is the calculation result of the timing determination model, is equal to or higher than the threshold value, the content or utterance is output from the user U while the smart speaker 1 is being operated or from the voice output device 11. If it is medium, it is not determined that the speech timing has come. In this case, the timing determination unit 36 ends the operation of the smart speaker 1 by the user U if there is a timing determination model in which the score is continuously equal to or higher than the threshold value when the operation of the smart speaker 1 by the user U is completed. After a certain period of time, it can be determined that it is time to output the utterance content.

Further, the timing determination unit 36 can set the above-mentioned output prohibition period and output upper limit number of times in the utterance table 21 for each content based on the voice operation by the user U. The information that can be set for each content is not limited to the output prohibition period and the maximum number of output times. Further, information such as the output prohibition period and the maximum number of times of output may be set in advance in the utterance table 21 regardless of the setting of the user U.

In this way, the timing determination unit 36 can determine the utterance timing and the utterance content determined to be output at the utterance timing (hereinafter, may be described as the output target utterance content) based on the context information. it can. The timing determination unit 36 determines the utterance timing and the output target utterance content in consideration of the past reaction of the user U to the utterance from the voice output device 11, and thus determines the utterance timing and the output target utterance content. Can be determined more appropriately.

For example, suppose that the user U is located in a dark place and there is a collection vehicle (for example, a garbage truck) around the user U, and no matter how many times the user U speaks, there is no reaction of the user U to the utterance. In this case, the timing determination model is generated so that the score output when the user U is located in a dark place and indicates that there is a collection vehicle around the user U is smaller than the threshold value. Therefore, when the context information indicates that the user U is located in a dark place and there is a collecting vehicle around the user U, the timing determination unit 36 determines that the utterance timing is not reached.

Further, it is assumed that the user U does not respond to the utterance when both the dishwasher and the microwave oven are used. In this case, the timing determination model is generated so that the score output when both the dishwasher and the microwave oven are used is smaller than the threshold value. Therefore, when the context information indicates that the dishwasher and the microwave oven are used together, the timing determination unit 36 determines that it is not the utterance timing.

Further, when the user U is operating the terminal device 3 (for example, the terminal device 3 is browsing a web page or the terminal device 3 is playing music), it is assumed that the user U does not respond to the utterance. In this case, the timing determination model is generated so that the score output when the user U is operating the terminal device 3 is smaller than the threshold value. Therefore, when the context information indicates that the user U is operating the terminal device 3, the timing determination unit 36 determines that it is not the utterance timing.

Further, for example, when the conversation of the user U continues after the utterance is started or when the user U utters "stop", the timing determination model uses the context information at the time of the utterance as the feature amount as incorrect answer data. Will be generated. In this case, the timing determination unit 36 can determine that it is not the utterance timing in the context in which the user U continues the conversation of the user U or in the context in which the user U speaks "stop".

In this way, the timing determination unit 36 can determine the utterance timing when the current context regarding the user U is a context suitable for utterance. Further, since the timing determination model is generated for each content, an appropriate utterance timing for each content is determined. For example, in the morning hours, the user U often reacts to content related to the operating status of transportation or news content. Therefore, a timing determination model is generated in which the morning time zone is likely to be the utterance timing for the content related to the operation state of the transportation system and the content of the news.

The output processing unit 34 outputs the output target utterance content determined by the timing determination unit 36 from the voice output device 11 at the utterance timing determined by the timing determination unit 36. For example, when the utterance table 21 is shown in FIG. 4 and the utterance content to be output is "utterance content XA", the output processing unit 34 outputs a telegraph signal based on the utterance content XA to the voice output device 11. , The utterance content XA is output by voice from the voice output device 11.

[3. Configuration of information providing device 2]
Next, the configuration of the information providing device 2 according to the embodiment will be specifically described. FIG. 5 is a diagram showing a configuration example of the information providing device 2 according to the embodiment. As shown in FIG. 5, the information providing device 2 includes a communication unit 41, a storage unit 42, and a control unit 43.

The communication unit 41 is a communication interface capable of communicating with devices such as the smart speaker 1, the terminal device 3, the device 4, and the sensor device 5 via the network 6. The control unit 43 can send and receive information to and from the information providing device 2, the smart speaker 1, the terminal device 3, the device 4, and the sensor device 5 via the communication unit 41.

The storage unit 42 includes an utterance table storage unit 51, a content storage unit 52, a voice advertisement storage unit 53, a user information storage unit 54, a context storage unit 55, and an output mode determination information storage unit 56.

The utterance table storage unit 51 stores the information of the utterance table 21 for each smart speaker 1. FIG. 6 is a diagram showing an example of an utterance table stored in the utterance table storage unit 51 according to the embodiment. The utterance table 71 shown in FIG. 6 includes information in which the “content ID”, the “utterance content”, and the “model” are associated with each other for each “device ID”.

The "content ID", "utterance content", and "model" in the utterance table 71 are the same information as the "content ID", "utterance content", and "model" in the utterance table 21. The "device ID" is identification information unique to each smart speaker 1.

The content storage unit 52 shown in FIG. 5 stores various contents to be provided to the smart speaker 1. FIG. 7 is a diagram showing an example of a content table stored in the content storage unit 52 according to the embodiment. The content table 72 shown in FIG. 7 is information in which the “content ID” and the “content” are associated with each other.

The "content" includes a content for auditory output and a content for visual output. The auditory output content is content that is output by voice, and the visual output content is content that is output in a mode other than voice, such as characters and images.

The voice advertisement storage unit 53 shown in FIG. 5 stores voice advertisement information and the like. FIG. 8 is a diagram showing an example of a voice advertisement table stored in the voice advertisement storage unit 53 according to the embodiment. The voice advertisement table 73 shown in FIG. 8 is information in which the “advertisement ID”, the “voice advertisement”, the “output count”, the “acceptance count”, and the “acceptance rate” are associated with each other. The "advertisement ID" is identification information unique to each voice advertisement.

The “voice advertisement” is the content of the voice advertisement, and is output from, for example, the voice output device 11 of the smart speaker 1 or the voice output device of the terminal device 3. The "voice advertisement" in the voice advertisement table 73 is the content of the voice advertisement itself, but may be information indicating a storage location of the content of the voice advertisement.

The “number of outputs” is information indicating the number of times the voice advertisement is provided to the user U, and is incremented by the control unit 43 each time the voice advertisement is output to the smart speaker 1 or the terminal device 3, for example. The “number of acceptances” is the number of times the voice advertisement is received by the user U, and is incremented by the control unit 43 each time the voice advertisement is received by the user U, for example. The "acceptance rate" is the ratio of the number of acceptances to the number of outputs, and is calculated by, for example, the control unit 43.

For example, in the voice advertisement table 73 shown in FIG. 8, the voice advertisement of the advertisement ID “A101” shows that the number of outputs is 29817, the number of acceptances is 827, and the acceptance rate is 0.0278. Further, the voice advertisement of the advertisement ID "A102" shows that the number of outputs is 8372, the number of acceptances is 352, and the acceptance rate is 0.0420.

The user information storage unit 54 shown in FIG. 5 stores the information of the user U. FIG. 9 is a diagram showing an example of a user information table stored in the user information storage unit 54 according to the embodiment. The user information table 74 shown in FIG. 9 is information in which a "user ID", a "user attribute", a "device ID", and a "device address" are associated with each other.

The "user ID" is identification information unique to each user U. The "user attribute" is information indicating the attribute of the user U. The attributes of the user U are, for example, demographic attributes such as address and occupation in addition to gender and age, but may include psychographic attributes indicating the preferences of the user U and the like. The "device ID" includes identification information unique to the smart speaker 1 owned by the user U and identification information unique to the terminal device 3 owned by the user U. The “device address” is an address on the network 6 of the smart speaker 1 or the terminal device 3 owned by the user U.

The output mode determination information storage unit 56 includes output mode determination information for determining the output mode of the content subject to the output request when the smart speaker 1 requests the output of the content. As described above, the output mode includes at least one of the content output type and the content output destination.

The output mode determination information is, for example, information that defines the relationship between the context regarding the user U and each output mode, and includes a model or a table. The table included in the output mode determination information is a table that defines the relationship between the context regarding the user U and each output mode. Further, the model included in the output mode determination information is a model generated by learning by the learning unit 64.

The control unit 43 shown in FIG. 5 includes an information acquisition unit 61, an information output unit 62, a context acquisition unit 63, a learning unit 64, an output mode determination unit 65, a detection unit 66, a determination unit 67, and an advertisement. It is provided with an effect updating unit 68.

The information acquisition unit 61 acquires the information transmitted from the smart speaker 1. For example, the information acquisition unit 61 can acquire input information (for example, utterance information, information indicating the operation content by gesture) indicating the instruction of the user U from the smart speaker 1. Further, the information acquisition unit 61 can acquire, for example, imaging information indicating the captured image of the user U from the smart speaker 1, the terminal device 3, the device 4, or the sensor device 5.

Further, the information acquisition unit 61 provides operation history information indicating the operation history to one or more devices (for example, smart speaker 1, terminal device 3, device 4, etc.) existing around the user U to the smart speaker 1, terminal device. It can be obtained from 3 or the device 4.

The information output unit 62 acquires the content (auditory output content) corresponding to the input information of the user U from the content storage unit 52 based on the output mode determined by the output mode determination unit 65. For example, when the output type determined by the output mode determination unit 65 is auditory output, the information output unit 62 acquires audio content from the content storage unit 52, which is content corresponding to the input information of the user U. ..

Further, when the output type determined by the output mode determination unit 65 is visual output, the information output unit 62 is content corresponding to the input information of the user U and is character or image content (visual output content). ) Is acquired from the content table 72 of the content storage unit 52. Further, when the output types determined by the output mode determination unit 65 are the auditory output and the visual output, the information output unit 62 is the content corresponding to the input information of the user U, and is voice and characters (or images). The content including the above is acquired from the content table 72 of the content storage unit 52.

The information output unit 62 outputs the content acquired from the storage unit 42 to at least one of the smart speaker 1 and the terminal device 3 based on the output mode determined by the output mode determination unit 65. The information output unit 62 acquires the device address of the output destination determined by the output mode determination unit 65 from the user information table 74 of the storage unit 42, and transmits the content to the acquired device address.

For example, when the output destination determined by the output mode determination unit 65 is the smart speaker 1, the information output unit 62 transmits the content acquired from the storage unit 42 to the device address of the smart speaker 1. Further, when the output destination determined by the output mode determination unit 65 is the terminal device 3, the information output unit 62 transmits the content acquired from the storage unit 42 to the device address of the terminal device 3.

The context acquisition unit 63 executes a content acquisition process for acquiring context information regarding the user U. The context acquisition unit 63 can acquire context information from the information acquired by the information acquisition unit 61. The context information acquired by the context acquisition unit 63 is the same as the context information acquired by the context acquisition unit 35, but even if the information is partially or completely different from the context information acquired by the context acquisition unit 35. Good.

The context information acquired by the context acquisition unit 63 includes input type information indicating the type of instruction of the user U. The input type information is, for example, information indicating whether the instruction of the user U is a voice, a gesture, or a movement of the mouth. When the captured image of the user U is transmitted from the smart speaker 1, the terminal device 3, the device 4, or the sensor device 5 to the information providing device 2, the context acquisition unit 63 makes a gesture of the user U from the captured image of the user U. It is also possible to acquire input type information by determining the movement of the or mouth.

The learning unit 64 can generate a timing determination model for the utterance table 71 for each user U and each content based on the context information for each user U acquired by the context acquisition unit 63. As described above, such a timing determination model is a model generated based on the reaction of the past user U to the utterance and the context information regarding the past user U at the time of the utterance.

In the learning unit 64, the operation of the user U or the utterance of the user U included in the context information becomes a spontaneous utterance from the smart speaker 1 within a preset period after the spontaneous utterance from the smart speaker 1 is started. Whether or not the reaction is positive is used as the teacher data.

For example, when the user U's utterance is positive with respect to the spontaneous utterance of the smart speaker 1, the learning unit 64 determines that the reaction is positive with respect to the spontaneous utterance of the smart speaker 1. For example, the learning unit 64 responds positively to a spontaneous utterance such as "How about today's news?" When the utterance of the user U is, for example, "Thank you" or "Yeah". Can be determined.

Further, when the user U does not speak to the spontaneous utterance of the smart speaker 1 or the user U's utterance to the spontaneous utterance of the smart speaker 1 is negative, the learning unit 64 causes the smart speaker 1 to speak spontaneously. It is judged that the reaction is not positive. For example, the learning unit 64 responds to a spontaneous utterance such as "How about today's news?" When there is no utterance of the user U, or when the utterance of the user U is, for example, "don't need" or "stop". In some cases, it can be determined that the reaction is not positive.

Whether or not User U's reaction to spontaneous utterance is positive is not limited to the above-mentioned example. For example, the learning unit 64 can determine that the user U has a positive reaction to the spontaneous utterance when the user U nods. Further, the learning unit 64 can determine that the user U does not have a positive reaction to the spontaneous utterance when the user U moves away from the smart speaker 1.

As described above, the learning unit 64 uses the teacher data as to whether or not it is a positive reaction to the spontaneous utterance, and one or more included in the context information in a preset period after the start of the spontaneous utterance. Machine learning is performed using the information in the above as a feature quantity to generate and update a timing judgment model.

The timing determination model is not limited to the above-mentioned example, and may be, for example, a model generated by using SVM (Support Vector Machine) or another machine learning method. Further, the timing determination model may be generated by using a technique of deep learning. For example, the timing determination model may be generated by appropriately using various deep learning techniques such as DNN (Deep Neural Network), RNN (Recurrent Neural Network), and CNN (Convolutional Neural Network).

The learning unit 64 sets the generated timing determination model in the utterance table 71. Further, the learning unit 64 can update the timing determination model set in the utterance table 71 based on the reaction of the user U to the newly acquired utterance and the context information. For example, the learning unit 64 can update the timing determination model in a time zone (for example, midnight) when the processing load of the information providing device 2 is small.

Further, the learning unit 64 can generate a timing determination model common to a plurality of users U (hereinafter, may be referred to as a common determination model) for each content or for a specific content. In this case, the learning unit 64 can generate a common determination model based on the reactions of the plurality of past users U to the utterance and the context information regarding the past plurality of users U at the time of the utterance.

Further, the learning unit 64 generates a common judgment model for each content, and then gives each user U a reaction and context information to a newly acquired utterance based on the common judgment model. It is also possible to generate a unique timing determination model.

Further, the learning unit 64 can also generate a timing determination model that is common to a plurality of contents and common to a plurality of users U. In this case, the learning unit 64 has a timing determination model common to the plurality of users U based on the reactions of the past plurality of users U to an arbitrary utterance and the context information regarding the past plurality of users U at the time of an arbitrary utterance. Can be generated.

Further, the learning unit 64 can generate an output mode determination model for each output mode based on the reaction of the past user U to the content and the context information regarding the past user U. For example, machine learning can be performed using the teacher data as to whether or not the reaction of the user U is a negative reaction or a positive reaction, and the context information at the time of the reaction of the user U as a feature amount. ..

The negative reaction is, for example, a negative utterance of the user U (for example, "don't need" or "stop") when the content is output as voice from the smart speaker 1. Further, the negative reaction is, for example, a non-operation of the user U with respect to the terminal device 3 when the content is output as a character or an image from the terminal device 3.

Further, the positive reaction is, for example, a state in which there is no negative utterance of the user U when the content is output as voice from the smart speaker 1. The positive reaction is, for example, an operation on the terminal device 3 when the content is output as a character or an image from the terminal device 3.

When there is a content output request, the output mode determination unit 65 determines the output mode of the content provided to the user U based on the context information acquired by the context acquisition unit 63. The output mode includes an output type and an output destination, but the output mode determining unit 65 can also determine only one of the output type and the output destination. The output mode determination unit 65 determines the output mode using the above-mentioned output mode determination information stored in the output mode determination information storage unit 56.

The output mode determination unit 65 can determine the output mode based on the status information indicating the status of the user U included in the context information. For example, when the user U is moving, the output mode determination unit 65 can determine a mode in which the output type is auditory output and the output destination is the smart speaker 1 as the content output mode. As a result, the user U can grasp the content without looking at the screen of the terminal device 3 while moving.

Further, the output mode determination unit 65 can set the device of the smart speaker 1 and the terminal device 3 that is close to the current position of the user U as the output destination, which makes it easy for the user U to grasp the content. it can.

Further, when the user U is in a conversation, the output mode determining unit 65 can determine a mode in which the output type is visual output and the output destination is the terminal device 3 as the content output mode. As a result, the user U can grasp the content without interrupting the conversation.

Further, the output mode determining unit 65 can determine the output mode based on the surrounding information indicating the surrounding situation of the user U included in the context information. For example, the output mode determination unit 65 can determine a mode in which the output type is visual output and the output destination is the terminal device 3 as the content output mode when there is another person around the user U. .. As a result, for example, when the content is the schedule of the user U or the mail to the user U, the smart speaker 1 does not output the voice as the content, so that it is possible to prevent the schedule or mail of the user U from being known to others. it can.

The output mode determining unit 65 may determine as a content output mode a mode in which the output type is visual output and the output destination is the terminal device 3 when another person existing around the user U is sleeping. it can. As a result, for example, it is possible to avoid a situation in which another person sleeping around the user U is awakened by the output of the content.

Further, the output mode determination unit 65 can determine a mode in which the output type is visual output and the output destination is the terminal device 3 as the content output mode when the sound around the user U is loud. As a result, it is possible to avoid a situation in which the content cannot be grasped due to the noise around the user U or the sound emitted by the device 4.

Further, the output mode determining unit 65 sets the output type to auditory output and the output destination to the smart speaker 1 when there is no other person around the user U or when the sound around the user U is small. It can be determined as the output mode of the content. As a result, the user U can grasp the content without looking at the screen of the terminal device 3.

Further, when the input type to the smart speaker 1 is gesture or mouth movement, the output mode determining unit 65 sets the output type to visual output and the output destination to the terminal device 3, as the content output mode. Can be decided. Further, in the case of utterance to the smart speaker 1 (sound utterance), the output mode determining unit 65 determines as the content output mode the mode in which the output type is auditory output and the output destination is the smart speaker 1. be able to.

As a result, the user U can input to the smart speaker 1 with an input type according to the desired output mode, and the user U can confirm the content from the smart speaker 1 or the terminal device 3 in the desired output mode. it can. When the output mode determination unit 65 determines the output mode according to the input type based on the output mode determination model, the output mode determination model may, for example, increase the weight with the input type as the feature amount or input the feature amount. The output mode determination unit 65 can determine the output mode according to the input type by setting only the type. The output mode determination unit 65 can also determine the output mode according to the input type based on the output mode determination table in which the input type and the output mode are associated with each input type.

Further, when the output mode determination unit 65 includes the output mode determination model for each output mode as the output mode determination information, the output mode determination unit 65 inputs the context information to the output mode determination model for each output mode. The output mode determination unit 65 determines the output mode of the content based on the output of the output mode determination model for each output mode.

For example, when the output mode determination model is generated using whether or not the reaction of the user U is a negative reaction as teacher data, the output mode determination unit 65 corresponds to the output mode determination model having the lowest output score. The output mode can be determined as the output mode of the content. Further, when the output mode determination model is generated using whether or not the reaction of the user U is a positive reaction as teacher data, the output mode determination unit 65 corresponds to the output mode determination model having the highest output score. The output mode can be determined as the output mode of the content.

In this way, the output mode determination unit 65 can determine the output mode of the content according to the situation of the user U and the situation around the user U, so that the content can be appropriately provided to the user U. it can.

The detection unit 66 detects the behavior of the user U when a voice advertisement is output from the voice output device 11 of the smart speaker 1 or the voice output device (not shown) of the terminal device 3. The detection unit 66 can detect the behavior of the user U when the voice advertisement is output by performing image analysis of the image pickup information acquired by the information acquisition unit 61.

For example, the detection unit 66 moves the line of sight of the user U, the movement of the head of the user U, the movement of the mouth of the user U, the movement of the hand of the user U, based on the imaging information acquired by the information acquisition unit 61. And at least one physical behavior of the user U's foot movement can be detected as the user U's behavior.

Further, the detection unit 66 can detect the state of the work performed by the user U based on the imaging information acquired by the information acquisition unit 61. For example, the detection unit 66 can detect whether or not the user U has interrupted operations such as washing dishes, sewing with a sewing machine, and cooking.

Further, the detection unit 66 can detect the behavior of the user U when the voice advertisement is output by acoustically analyzing the sound information acquired by the information acquisition unit 61. For example, the detection unit 66 asks the user U's behavior in conversation, the user U's utterance to the smart speaker 1, the behavior in the work performed by the user U, and the user U's device 4 based on the sound information. Behavior etc. can be detected.

Specifically, the detection unit 66 can detect behaviors such as the user U in conversation stopping the utterance and the user U in the conversation lowering the volume of the utterance based on the sound information. In addition, the detection unit 66 can detect behaviors such as an utterance for the user U to search for information and an utterance for the user U to confirm the information.

Further, the detection unit 66 can detect whether or not the user U has interrupted the work such as washing dishes and cooking based on the sound information. For example, the detection unit 66 can determine that the dishwashing is interrupted when the sound of water discharged from the faucet disappears or the sound of washing the dishes disappears.

Further, when the user U turns off the device 4, the device sound, which is the sound output from the device 4, is stopped. Therefore, the detection unit 66 behaves as the user U turns off the device 4 based on the sound information. It can be detected. The device 4 in which the detection unit 66 detects off based on sound information is, for example, a device that emits a relatively loud sound, such as a microwave oven, a washing machine, a dishwasher, a sewing machine, a television receiver, and a radio receiver. ..

Further, the detection unit 66 can detect the behavior of the user U to the smart speaker 1, the terminal device 3, or the device 4 based on the operation history information acquired by the information acquisition unit 61. For example, the detection unit 66 can detect the behavior of the user U turning the device off and on, the behavior of the user U searching for information, and the like based on the operation history information.

The determination unit 67 determines whether or not the voice advertisement is accepted by the user U based on the behavior of the user U detected by the detection unit 66. For example, the determination unit 67 determines that the voice advertisement has been accepted by the user U when the physical behavior of the user U detected by the detection unit 66 is a specific behavior.

Specific behaviors include, for example, the user U directs the line of sight to the smart speaker 1 for a certain period of time or longer during the advertisement output period, the moving user U stops moving for a certain period of time or longer during the advertisement output period at the start of the output of the voice advertisement, and the like. This is a behavior such as the user U nodding during the advertisement output period. Further, the specific behavior is such that the user U who has been continuously moving before the start of the output of the voice advertisement stops his / her hand for a certain period of time or more during the advertisement output period, and the user U makes a specific gesture. ..

Further, the determination unit 67 determines that the voice advertisement has been accepted by the user U when the behavior of the user U detected by the detection unit 66 based on the sound information is a specific behavior. Specific behaviors include, for example, user U in conversation stops speaking, user U in conversation reduces the volume of utterance, user U speaks about a voice advertisement, user U interrupts work, and user U. Is the behavior such as turning off the device. The utterances related to the voice advertisement by the user U are, for example, "So?" And "Continued?".

Further, the determination unit 67 determines that the voice advertisement has been accepted by the user U when the behavior of the user U detected by the detection unit 66 based on the operation history information is a specific behavior. The specific behavior is, for example, a behavior in which the user U searches for a product or service to be advertised in a voice advertisement, and the user U turns off the device.

The determination unit 67 can determine that the voice advertisement is not accepted by the user U when the behavior of the user U is a negative behavior. Negative behavior is, for example, behavior such that the user U makes a negative utterance and the user U makes a negative physical behavior.

For example, the determination unit 67 can determine that the user U has behaved negatively when the user U makes negative utterances such as "stop", "do not want to hear", and "dislike". Further, for example, the determination unit 67 can determine that the user U has behaved negatively when the user U closes his / her ear with his / her hand.

If the determination unit 67 determines that the behavior of the user U is not a specific behavior, it can also determine that the voice advertisement is not accepted by the user U. As a result, the determination unit 67 does not have to determine whether or not the user U has behaved negatively, and the processing load is reduced.

Further, when the determination unit 67 determines that the behavior of the user U is neither a specific behavior nor a negative behavior, the determination unit 67 determines that the non-acceptance of the voice advertisement by the user U is low, and the behavior of the user U is negative. In the case of behavior, it can be determined that the non-acceptance of the voice advertisement by the user U is high.

Further, the determination unit 67 may not perform the process of determining that the voice advertisement has been accepted by the user U when the behavior of the user U is a specific behavior. For example, the determination unit 67 determines that the voice advertisement is not accepted by the user U when the behavior of the user U is a negative behavior, and when the behavior of the user U is not a negative behavior, the voice It can be determined that the advertisement is accepted by the user U.

The specific behaviors and negative behaviors described above are set according to the attributes of the user U. User U's attributes include, for example, at least one of gender, age, address, and occupation. For example, the determination unit 67 determines that the voice advertisement has been accepted by the user U when the user U is a child, the user U jumps, and the user U begins to dance.

The determination unit 67 can also change a specific behavior as a criterion for determining acceptance by the user U according to the time length and type of the voice advertisement. Further, the determination unit 67 determines whether or not the voice advertisement is accepted by the user U every predetermined period (for example, 5 seconds) in the period (for example, 30 seconds) during which one voice advertisement is output. You can also do it.

When the advertisement effect updating unit 68 determines that the voice advertisement has been accepted by the user U, the advertisement effect updating unit 68 updates the number of times the voice advertisement is received in the voice advertisement table 73. As a result, it is possible to obtain an advertising effect equivalent to a click of a text advertisement or a banner advertisement for a voice advertisement. Then, the advertisement effect updating unit 68 can calculate the acceptance rate, which is the ratio of the number of times the voice advertisement is received to the number of times the voice advertisement is output, and set the calculated acceptance rate in the voice advertisement table 73 as the voice advertisement effect. As a result, it is possible to obtain an advertising effectiveness index corresponding to CTR in voice advertising.

In addition, the advertisement effect updating unit 68 can acquire the number of conversions of the voice advertisement from the external device. The number of conversions of an audio advertisement is the number of times that the purpose of an audio advertisement such as purchase of goods or services, application for samples, and application for pamphlets is achieved. The advertisement effect updating unit 68 can calculate the conversion rate, which is the ratio of the number of conversions of the voice advertisement to the number of times the voice advertisement is received, and can use the calculated conversion rate as the voice advertisement effect. As a result, it is possible to obtain an advertising effectiveness index corresponding to CVR (Conversion Rate) in voice advertising.

When the non-acceptance degree is determined by the determination unit 67, the advertisement effect updating unit 68 can calculate the ratio of the non-acceptance number for each non-acceptance degree of the voice advertisement to the output number of voice advertisements. In this case, the advertisement effect updating unit 68 may, for example, subtract the number of non-acceptances with a non-acceptance degree of 2 from the number of acceptances of the voice advertisement, and use the subtraction result as the number of acceptances of the voice advertisement.

[4. Processing flow of information processing system 100]
Next, the procedure of the utterance control process by the information processing system 100 according to the embodiment will be described. 10 and 11 are flowcharts showing an example of the utterance control process by the information processing system 100 according to the embodiment.

First, the utterance control process of the smart speaker 1 will be described. As shown in FIG. 10, the control unit 15 of the smart speaker 1 determines whether or not the content output processing is in progress (step S10). In the process of step S10, the control unit 15 treats, for example, the period from when the control unit 15 receives the content output request from the user U until the output of the content corresponding to the output request is completed as the content output process.

When the control unit 15 determines that the content is not being output (step S10: No), the control unit 15 determines whether or not the utterance process is in progress (step S11). In the process of step S11, the control unit 15 treats, for example, the period from the start of the output of the utterance from the voice input device 12 to the end of the period during which the user U's request for the utterance can be received as the utterance process. ..

When the control unit 15 determines that the utterance process is not in progress (step S11: No), the control unit 15 acquires the context information regarding the user U (step S12), and applies the acquired context information to each timing determination model included in the utterance table 21. Input (step S13).

Subsequently, the control unit 15 determines whether or not there is a timing determination model that outputs a score equal to or greater than a preset threshold value (step S14). When the control unit 15 determines that there is a timing determination model that outputs a score equal to or higher than the threshold value (step S14: Yes), the control unit 15 determines whether or not there are a plurality of timing determination models that output a score equal to or higher than the threshold value (step). S15).

When the control unit 15 determines that there are a plurality of timing determination models (step S15: Yes), the control unit 15 selects the timing determination model having the highest score (step S16). Further, when the control unit 15 determines that the number of timing determination models is not plural (step S15: No), the control unit 15 selects a timing determination model that outputs a score equal to or greater than the threshold value (step S17). The control unit 15 acquires the utterance content associated with the selected timing determination model from the utterance table 21, and outputs the acquired utterance content from the voice input device 12 (step S18).

When the process of step S18 is completed, the control unit 15 determines that the content is being output (step S10: Yes), the control unit 15 determines that the utterance process is in progress (step S11: Yes), or is on the threshold value. When it is determined that there is no timing determination model for outputting the score (step S14: No), the process shown in FIG. 10 is terminated.

Next, the processing of the information providing device 2 will be described. As shown in FIG. 11, the control unit 43 of the information providing device 2 acquires reaction information indicating the reaction of the user U when the smart speaker 1 spontaneously speaks (step S20). Further, the control unit 43 acquires context information regarding the user U when the smart speaker 1 spontaneously speaks (step S21). The control unit 43 stores the reaction information acquired in step S20 and the context information acquired in step S21 in the storage unit 42 (step S22).

Subsequently, the control unit 43 generates or updates the timing determination model for each content based on the reaction information and the context information stored in the storage unit 42 (step S23), and ends the process shown in FIG. For example, when the control unit 43 generates the timing determination model of the content for which the timing determination model is not generated in the utterance table 71, the control unit 43 adds the timing determination model to the utterance table 71. Further, when the timing determination model of the utterance table 71 is updated, the control unit 43 overwrites the updated timing determination model on the utterance table 71.

Next, the output control process by the information processing system 100 will be described. FIG. 12 is a flowchart showing an example of output control processing by the information processing system 100 according to the embodiment.

As shown in FIG. 12, the control unit 43 determines whether or not there is a content output request from the smart speaker 1 (step S30). When the control unit 43 determines that there is a content output request (step S30: Yes), the control unit 43 acquires context information regarding the user U of the smart speaker 1 that has made the output request (step S31).

Then, the control unit 43 determines the output mode of the content requested to be output based on the acquired context information (step S32). The control unit 43 outputs the content in the output mode determined in step S32 (step S33). When the process of step S33 is completed or when it is determined that there is no output request (step S30: No), the control unit 43 ends the process shown in FIG.

Next, the voice information effect determination process by the information processing system 100 will be described. FIG. 13 is a flowchart showing an example of the voice information effect determination process by the information processing system 100 according to the embodiment.

As shown in FIG. 13, the control unit 43 of the information providing device 2 determines whether or not it is the output timing of the voice advertisement (step S40). When the control unit 43 determines that it is the output timing of the voice advertisement (step S40: Yes), the control unit 43 outputs the voice advertisement to the smart speaker 1 (step S41).

Subsequently, the control unit 43 increments the number of times the voice advertisement output in step S41 is output in the voice advertisement table 73 (step S42). Further, the control unit 43 acquires the imaging information, the sound information, and the device operation information during the audio output period from at least one of the smart speaker 1, the terminal device 3, the device 4, and the sensor device 5 (step S43).

The control unit 43 detects the behavior of the user U based on the information acquired in step S43 (step S44), and determines whether or not the detected behavior is a specific behavior (step S45). When the control unit 43 determines that the behavior detected in step S44 is a specific behavior (step S45: Yes), the control unit 43 determines that the voice advertisement has been accepted by the user U (step S46), and in the voice advertisement table 73, The number of times the voice advertisement received in step S41 is received is incremented (step S47).

On the other hand, when the control unit 43 determines that the behavior detected in step S44 is not a specific behavior (step S45: No), the control unit 43 determines that the voice advertisement is not accepted by the user U (step S48). When the process of step S47 or step S48 is completed, or when it is determined that it is not the output timing of the voice advertisement (step S40: No), the control unit 43 ends the process shown in FIG.

[5. Modification example]
In the above example, the information providing device 2 is provided with the learning unit 64 for generating the timing determination model, but the smart speaker 1 may be provided with the learning unit 64. Further, in the above-described example, the smart speaker 1 is provided with the context acquisition unit 35 and the timing determination unit 36, but the information providing device 2 may be provided with the context acquisition unit 35 and the timing determination unit 36.

Further, in the above-mentioned example, the utterance timing is determined by using the timing determination model, but the utterance timing is not limited to the above-mentioned example as long as the utterance timing can be determined in consideration of the reaction of the user U to the past utterance. For example, in the smart speaker 1, instead of the timing determination model, whether or not the current context is a context suitable for the utterance is based on the determination condition information generated based on the reaction information and the context information for the past utterance. Can also be determined. The determination condition information includes the condition of the context for determining the utterance timing.

Further, in the above-described example, the gesture of the user U and the movement of the mouth are detected by the smart speaker 1, but the gesture of the user U and the movement of the mouth may be detected by the information providing device 2.

Further, in the above-described example, the voice advertisement has been described as an example of the voice information output from the smart speaker 1, but the voice information output from the smart speaker 1 is not limited to the voice advertisement. For example, the control unit 43 of the information providing device 2 outputs the push-speech type voice information such as "Today is sunny" and "Today is a holiday" from the smart speaker 1, and the behavior of the user U. Based on the above, it can be determined whether or not the voice information has been accepted by the user U. Then, the control unit 43 of the information providing device can calculate the ratio of the number of times the voice information is received to the number of times the voice information is output, and use the calculated ratio as the voice information effect. As a result, an effect index corresponding to CTR can be obtained in voice information.

The voice information to be determined for the effect is not limited to the voice information output from the information providing device 2 to the smart speaker 1, and may be the voice information stored in the smart speaker 1.

Further, in the above-described example, the information processing system 100 may determine that the user U has accepted even if the reaction is not positive for the user U, but the user U has a positive reaction. Only, it can be determined that the user U has accepted. That is, the information processing system 100 accepts the state of the user U, which is determined to have had a positive reaction of the user U in the utterance control process and the output control process in the voice information effect determination process, and other than that. Can be non-acceptable to the user. Further, in the information processing system 100, in the utterance control process and the output control process, the state of the user U determined to be accepted by the user U in the voice information effect determination process is regarded as a positive reaction of the user U, and the other states are set as a positive reaction. It can be a non-positive reaction of the user.

[6. program〕
Each of the smart speaker 1 and the information providing device 2 in the above-described embodiment is realized by executing a program by a computer 200 having a configuration as shown in FIG. 14, for example. FIG. 14 is a diagram showing an example of the hardware configuration of the computer that executes the program. The computer 200 includes a CPU (Central Processing Unit) 201, a RAM (Random Access Memory) 202, a ROM (Read Only Memory) 203, an HDD (Hard Disk Drive) 204, a communication interface (I / F) 205, and an input / output interface (I). It includes a / F) 206 and a media interface (I / F) 207.

The CPU 201 operates based on the program stored in the ROM 203 or the HDD 204, and controls each part. The ROM 203 stores a boot program executed by the CPU 201 when the computer 200 is started, a program that depends on the hardware of the computer 200, and the like. HDD 204 stores data and the like used by a program executed by CPU 201. The communication interface 205 receives data from another device via the network 6 and sends it to the CPU 201, and transmits the data generated by the CPU 201 to the other device via the network 6.

The CPU 201 controls an output device such as a display or a printer, and an input device such as a keyboard or a mouse via the input / output interface 206. The CPU 201 acquires data from the input device via the input / output interface 206. Further, the CPU 201 outputs the generated data to the output device via the input / output interface 206.

The media interface 207 reads the program or data stored in the recording medium 208 and provides it to the CPU 201 via the RAM 202. The CPU 201 loads the program from the recording medium 208 onto the RAM 202 via the media interface 207, and executes the loaded program. The recording medium 208 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or PD (Phase change rewritable Disk), a magneto-optical recording medium such as MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory. And so on.

When the computer 200 functions as the smart speaker 1 according to the above-described embodiment, the CPU 201 of the computer 200 executes the program loaded on the RAM 202 to execute the input processing unit 31 and the information output unit 32 shown in FIG. Each function of the information acquisition unit 33, the output processing unit 34, the context acquisition unit 35, and the timing determination unit 36 is realized. Further, for example, the HDD 204 stores the same information as the storage unit 14 shown in FIG.

Further, when the computer 200 functions as the information providing device 2 according to the above-described embodiment, the CPU 201 of the computer 200 executes the program loaded on the RAM 202 to output the information acquisition unit 61 shown in FIG. Each function of the unit 62, the context acquisition unit 63, the learning unit 64, the output mode determination unit 65, the detection unit 66, the determination unit 67, and the advertisement effect updating unit 68 is realized. Further, for example, the HDD 204 stores the same information as the storage unit 42 shown in FIG.

The CPU 201 of the computer 200 reads and executes the programs from the recording medium 208, but as another example, these programs may be acquired from another device via the network 6.

[7. effect〕
As described above, the information processing system 100 (an example of the utterance control device) according to the embodiment is based on the context acquisition unit 35 that acquires the context information regarding the user U and the context information acquired by the context acquisition unit 35. The timing determination unit 36 for determining the timing of the utterance in consideration of the past reaction of the user U to the utterance from the voice output device 11 is provided. As a result, the utterance timing can be determined more appropriately than in the case where the utterance timing is controlled under preset conditions.

Further, the timing determination unit 36 determines the timing of the utterance for each content in consideration of the past reaction of the user U to the utterance for each content from the voice output device 11 based on the context information acquired by the context acquisition unit 35. To do. As a result, it is possible to determine an appropriate utterance timing according to the content.

Further, the timing determination unit 36 inputs the context information acquired by the context acquisition unit 35 into the timing determination model generated based on the reaction of the past user U to the utterance and the context information regarding the past user U at the time of the utterance. Then, the timing of the utterance is decided. As a result, the utterance timing can be easily determined based on the past reaction of the user U.

Further, the context acquisition unit 35 acquires surrounding information indicating the surrounding situation of the user U as at least a part of the context information. Thereby, for example, it is possible to determine a more appropriate utterance timing according to the content from the surrounding situation that affects the reaction of the user U.

Further, the context acquisition unit 35 acquires the surrounding information from the sensor information output from one or more sensor devices 5 existing around the user U. Thereby, for example, contexts such as brightness, temperature, and humidity around the user U can be used, and a more appropriate utterance timing can be determined according to the content.

Further, the context acquisition unit 35 acquires surrounding information from device information indicating the state of one or more devices (for example, smart speaker 1, terminal device 3, or device 4) existing around the user U. As a result, it is possible to use a context such as the operation state of the device by the user U, and it is possible to determine a more appropriate utterance timing according to the content.

Further, the context acquisition unit 35 acquires the surrounding information from the operation history information indicating the operation history of one or more devices (for example, the smart speaker 1, the terminal device 3, or the device 4) existing around the user U. As a result, it is possible to use a context such as a device operation log by the user U, and it is possible to determine a more appropriate utterance timing according to the content.

Further, the timing determination model is a model commonly generated by a plurality of users U. As a result, the processing load for generating the timing determination model can be reduced as compared with the case where the timing determination model is generated for each user U.

Further, the timing determination model is a model unique to the user U at least in part. As a result, it is possible to determine a more appropriate utterance timing according to the characteristics of the user U.

Further, the information processing system 100 includes a learning unit 64 that updates the timing determination model based on the past reaction of the user U to the utterance from the voice output device 11 and the context information. As a result, even when there is a characteristic change in the user U or around the user U, a more appropriate utterance timing can be determined.

Further, the reaction of the user U used in the timing determination model includes whether or not the user's utterance to the utterance from the voice output device 11 is positive. This makes it possible to determine a more appropriate utterance timing.

Further, the reaction of the user U used in the timing determination model is determined in the state of the conversation of the user U after the output of the utterance is started from the voice output device 11 when the user U is in a conversation. For example, the reaction of the user U is that the user U interrupts the conversation, the user U changes the topic of the conversation, and so on.

Further, the utterance from the voice output device 11 is an utterance relating to whether or not the content can be output. As a result, even if the utterance timing is not appropriate, it is possible to reduce the annoyance of the utterance given to the user U as compared with the case where the content is spontaneously output.

[8. Others]
Further, among the processes described in the above-described embodiments and modifications, all or part of the processes described as being automatically performed can be manually performed, or the processes described as being manually performed. It is also possible to automatically perform all or part of the processed processing by a known method.

In addition, the processing procedure, specific name, and information including various data and parameters shown in the above document and drawings can be arbitrarily changed unless otherwise specified. For example, the various information shown in each figure is not limited to the illustrated information.

Further, each component of each of the illustrated devices is a functional concept, and does not necessarily have to be physically configured as shown in the figure. That is, the specific form of distribution / integration of each device is not limited to the one shown in the figure, and all or part of the device is functionally or physically distributed / physically in arbitrary units according to various loads and usage conditions. It can be integrated and configured.

For example, the information processing system 100 may have a configuration in which at least one of an input processing unit 31, a context acquisition unit 35, and a timing determination unit 36 is provided in an information providing device 2 different from the smart speaker 1 or another device. Further, the smart speaker 1 may have a configuration capable of performing a part or all of the processing of the information providing device 2 described above. The information providing device 2 may be realized by a plurality of server computers, and depending on the function, the information providing device 2 may be realized by calling an external platform or the like by API (Application Programming Interface) or network computing, and the configuration is flexible. Can be changed to.

In addition, the above-described embodiments and modifications can be appropriately combined as long as the processing contents do not contradict each other. Further, the above-mentioned "section, module, unit" can be read as "means" or "circuit". For example, the input processing unit 31 can be read as an input processing means or an input processing circuit.

1 Smart speaker 2 Information providing device 3 Terminal device 4, 4 ₁ to 4 _n device 5, 5 _{1 to} 5 _m Sensor device 6 Network 10, 41 Communication unit 11 Audio output device 12 Audio input device 13 Imaging unit 14, 42 Storage unit 15,43 Control unit 20 Operation history 21 Speech table 31 Input processing unit 32,62 Information output unit 33,61 Information acquisition unit 34 Output processing unit 35,63 Context acquisition unit 36 Timing determination unit 51 Speech table storage unit 52 Content storage unit 53 Voice advertisement storage unit 54 User information storage unit 55 Context storage unit 56 Output mode judgment information storage unit 64 Learning unit 65 Output mode determination unit 66 Detection unit 67 Judgment unit 68 Advertising effect update unit 71 Speech table 72 Content table 73 Voice advertisement table 74 User information table 100 Information processing system

Claims (15)

A model that uses the teacher data as to whether or not the past user's reaction to the utterance from the voice output device is positive, is generated by machine learning using the past context information about the user, and is used to determine the timing of the utterance. A memory unit that memorizes
And context acquisition unit for acquiring context information for the user,
An utterance control device including a timing determination unit that inputs the context information acquired by the context acquisition unit into the model stored in the storage unit and determines the timing of the utterance. The storage unit
The model is memorized for each content,
The timing determination unit
The utterance control device according to claim 1, wherein the context information acquired by the context acquisition unit is input to the model for each content to determine the timing of utterance for each content. The context acquisition unit
The utterance control device according to claim 1 or 2 , wherein the surrounding information indicating the surrounding situation of the user is acquired as at least a part of the context information. The context acquisition unit
The utterance control device according to claim 3 , wherein the surrounding information is acquired from sensor information output from one or more sensor devices existing around the user. The context acquisition unit
The utterance control device according to claim 3 or 4 , wherein the surrounding information is acquired from device information indicating the state of one or more devices existing around the user. The context acquisition unit
The utterance control device according to any one of claims 3 to 5 , wherein the surrounding information is acquired from the operation history information indicating the operation history of one or more devices existing around the user. The model is
The utterance control device according to any one of claims 1 to 6 , wherein the model is generated in common by a plurality of users. The model is
The utterance control device according to any one of claims 1 to 6 , wherein at least a part of the model is unique to the user. The invention according to any one of claims 1 to 8 , further comprising a learning unit that updates the model based on the past user's reaction to the utterance from the voice output device and the context information. Speech control device. The utterance control device according to any one of claims 1 to 8 , wherein the reaction of the user includes whether or not the utterance of the user is positive with respect to the utterance from the voice output device. Any of claims 1 to 8 , wherein the reaction of the user is determined in the state of the conversation after the output of the utterance is started from the voice output device when the user is in a conversation. The speech control device described in one. The utterance control device according to any one of claims 1 to 11 , wherein the utterance from the voice output device is an utterance relating to whether or not the content can be output. The utterance control device according to any one of claims 1 to 12 , wherein the voice output device is a voice output device included in a smart speaker. It is a speech control method executed by a computer.
The context acquisition process to acquire context information about the user,
Whether or not the user's past reaction to the utterance from the voice output device is positive is used as the teacher data, and is generated by machine learning using the past context information about the user and used to determine the timing of the utterance. An utterance control method comprising: a timing determination step of inputting the context information acquired by the context acquisition step into the model of a storage unit for storing the model to determine the timing of the utterance. The context acquisition procedure to acquire context information about the user, and
Whether or not the user's past reaction to the utterance from the voice output device is positive is used as the teacher data, and is generated by machine learning using the past context information about the user and used to determine the timing of the utterance. A timing determination procedure for inputting the context information acquired by the context acquisition procedure into the model of the storage unit for storing the model to determine the timing of the utterance, and
An utterance control program characterized by having a computer execute.

Images