JP2019164710A - Device, method, and program for processing information - Google Patents

Abstract

To provide a device, method and program for processing information, which allow for measuring effect of voice information.SOLUTION: An information processing device according to an embodiment comprises: a detection unit configured to detect behavior of a user when voice information is output from an audio output unit; and a determination unit configured to determine whether the voice information was acknowledged by the user or not on the basis of the behavior detected by the detection unit.SELECTED DRAWING: Figure 5

Description

  The present invention relates to an information processing apparatus, an information processing method, and an information processing program.

  In recent years, with the rapid spread of networks such as the Internet, advertisement distribution via networks has been actively performed. As one form of such advertisement distribution, a voice advertisement is known (for example, see Patent Document 1). Since voice advertisements are easy to draw attention and information can be obtained while performing other tasks, it is expected that the advertising opportunities and the advertising effects will be improved more than the advertisements aimed at visual effects.

JP, 2006-61997, A

  However, unlike advertisements such as video advertisements, advertisement banners, and text advertisements that are displayed on the screen, voice advertisements do not have a screen for displaying advertisements, and therefore cannot determine the effect of advertisement by clicking. In the technique described in Patent Document 1, the advertisement effect can be improved by estimating advertisement delivery candidates and outputting a voice advertisement, but does not determine the advertisement effect of the output voice advertisement. Further, not only the advertising effect of the voice advertisement but also the voice information provided to the user, it is desirable to measure the effect of the voice information when the voice information is provided to the user.

  The present application has been made in view of the above, and an object thereof is to provide an information processing apparatus, an information processing method, and an information processing program capable of measuring the effect of audio information.

  An information processing apparatus according to the present application includes: a detection unit that detects a user's behavior when audio information is output from an audio output device; and the audio information is received by the user based on the behavior detected by the detection unit. And a determination unit for determining whether or not.

  According to one aspect of the embodiment, an information processing apparatus, an information processing method, and an information processing program that can measure the effect of audio information can be provided.

FIG. 1 is a diagram illustrating a configuration example of an information processing system according to an embodiment. FIG. 2 is an explanatory diagram of a sound information effect determination process according to the embodiment. FIG. 3 is a diagram illustrating a configuration example of the smart speaker according to the embodiment. FIG. 4 is a diagram illustrating an example of an utterance table according to the embodiment. FIG. 5 is a diagram illustrating a configuration example of the information providing apparatus according to the embodiment. FIG. 6 is a diagram illustrating an example of an utterance table stored in the utterance table storage unit according to the embodiment. FIG. 7 is a diagram illustrating an example of a content table stored in the content storage unit according to the embodiment. FIG. 8 is a diagram illustrating an example of a voice advertisement table stored in the voice advertisement storage unit according to the embodiment. FIG. 9 is a diagram illustrating an example of a user information table stored in the user information storage unit according to the embodiment. FIG. 10 is a flowchart (part 1) illustrating an example of an utterance control process by the information processing system according to the embodiment. FIG. 11 is a flowchart (part 2) illustrating an example of an utterance control process by the information processing system according to the embodiment. FIG. 12 is a flowchart illustrating an example of output control processing by the information processing system according to the embodiment. FIG. 13 is a flowchart illustrating an example of a sound information effect determination process by the information processing system according to the embodiment. FIG. 14 is a diagram illustrating an example of a hardware configuration of a computer that executes a program.

  Hereinafter, an information processing apparatus, an information processing method, and a form for implementing an information processing program (hereinafter referred to as “embodiment”) according to the present application will be described in detail with reference to the drawings. Note that the information processing apparatus, the information processing method, and the information processing program according to the present application are not limited by this embodiment.

[1. Information providing system)
FIG. 1 is a diagram illustrating a configuration example of an information processing system according to an embodiment. As illustrated in FIG. 1, the information processing system 100 according to the embodiment includes a smart speaker 1, an information providing device 2, a terminal device 3, and a plurality of devices 4 ₁ to 4 _n (n is an integer equal to or greater than 2). And a plurality of sensor devices 5 _{1 to} 5 _m (m is an integer of 2 or more). Hereinafter, when each of the devices 4 ₁ to 4 _n is illustrated without being distinguished from each other, the device 4 is referred to as a device 4, and when each of the sensor devices 5 _{1 to} 5 _m is illustrated without being distinguished from each other, it is referred to as a sensor device 5.

  The smart speaker 1, the information providing device 2, the terminal device 3, the device 4, and the sensor device 5 are connected to be communicable with each other wirelessly or via a network 6. The network 6 is, for example, a LAN (Local Area Network) or a WAN (Wide Area Network) such as the Internet, and is composed of one or more networks.

Smart speaker 1, the terminal device 3, device 4, and the sensor device 5 is arranged in the area AR ₁ surrounding the user _{U 1.} The area AR ₁ is, for example, a room or a house of the user U ₁ . Furthermore, also each of the regions _AR 2 to Ar _k of the illustrated non Although the user _U 2 _~U k (k is an integer of 2 or more), as in the area AR _1, the smart speaker 1, the terminal device 3, device 4 , And a sensor device 5 are arranged. Hereinafter, when each of the users U _{1 to} U _k is shown without being distinguished, it is referred to as a user U.

  The smart speaker 1 is a speaker that can use an AI (Artificial Intelligence) assistant function corresponding to interactive voice operation, and the user U obtains various information by interacting with the smart speaker 1. Can do. For example, the smart speaker 1 transmits input information indicating an instruction from the user U to the information providing apparatus 2, and content (for example, music, news, etc.) provided from the information providing apparatus 2 according to the input information via the network 6. (Various information such as traffic information and weather) can be acquired, and the acquired content can be output from the built-in audio output device.

  In addition, the smart speaker 1 can control the device 4 in accordance with an instruction from the user U. For example, when the device 4 is a lighting device, the smart speaker 1 can control on and off of the device 4 that is the lighting device in accordance with an instruction from the user U.

  The information providing apparatus 2 can provide content corresponding to an instruction from the user U to the smart speaker 1 via the network 6 based on information output from the smart speaker 1. For example, the information providing apparatus 2 can determine the instruction of the user U based on the utterance information of the user U transmitted from the smart speaker 1 and can provide the smart speaker 1 with content corresponding to the determined instruction.

  The terminal device 3 is realized by, for example, a smartphone, a desktop PC (Personal Computer), a notebook PC, a tablet terminal, a mobile phone, a PDA (Personal Digital Assistant), or the like. The terminal device 3 has a plurality of applications including an information notification application, for example, and acquires and displays content notified from the information providing device 2. The terminal device 3 can notify the smart speaker 1 and the information providing device 2 of information indicating the operation state of the terminal device 3 and information indicating the operation history of the terminal device 3 by the user U.

  The device 4 is a device that exists around the user U. For example, the device 4 includes various devices such as a refrigerator, a lighting device, a washing machine, an air conditioner, a television receiver, a dishwasher, a dish dryer, an electromagnetic cooker, and a microwave oven. The device 4 can notify the smart speaker 1 and the information providing device 2 of information indicating the operation state of the device 4 and information indicating the operation history of the device 4 by the user U.

  The sensor device 5 is a sensor that exists around the user U. The sensor device 5 includes, for example, sensors such as a temperature sensor, a humidity sensor, an illuminance sensor, an atmospheric pressure sensor, and an open / close sensor that detects opening / closing of a door. The sensor device 5 includes an imaging unit that images the user U. The sensor device 5 notifies the smart speaker 1 and the information providing device 2 of sensor information such as temperature measurement values, humidity measurement values, illuminance measurement values, atmospheric pressure measurement values, door opening / closing information, and captured image information. Can do. The sensor device 5 may be built in the smart speaker 1 or the device 4.

  The information processing system 100 according to the embodiment can execute a sound information effect determination process for determining whether or not the sound information output from the sound output device of the smart speaker 1 has been received by the user U. Hereinafter, the audio information effect determination process will be specifically described. FIG. 2 is an explanatory diagram of the sound information effect determination process according to the embodiment. In the example illustrated in FIG. 2, the sound information effect determination process is executed by the information providing apparatus 2 which is an example of the sound information effect determination process. In the following, a voice advertisement will be described as an example of voice information.

  The information providing apparatus 2 outputs the voice advertisement to the smart speaker 1 via the network 6 (Step S1). Thereby, a voice advertisement is output from the smart speaker 1. The voice advertisement is a voice advertisement content, and includes, for example, human voice, music, sound effects and the like.

  The information providing device 2 acquires information for detecting the behavior of the user U when the voice advertisement is output from the smart speaker 1 from the smart speaker 1 or the sensor device 5, and based on the acquired information, the user U's A behavior is detected (step S2).

  In step S <b> 2, the information providing device 2 performs the behavior of the user U based on the captured image captured by the smart speaker 1, the terminal device 3, or the sensor device 5 after the output of the voice advertisement from the smart speaker 1 is started. Can be detected. For example, the information providing apparatus 2 is a physical behavior of the user U (for example, eye movement, head movement, hand movement, foot movement, mouth movement, etc.) or a behavior of the user U to the device 4. (For example, the device 4 is turned off) can be detected.

  In addition, the information providing apparatus 2 can detect the behavior of the user U based on sound information input to the smart speaker 1 during the audio advertisement output period. For example, the information providing apparatus 2 can detect the behavior of the user U's utterance, the behavior of the work that the user U was performing, and the like based on the sound information input to the smart speaker 1.

  Based on the detected behavior of the user U, the information providing apparatus 2 determines whether or not the voice advertisement has been received by the user U (step S3). For example, the information providing apparatus 2 can determine that the voice advertisement has been received by the user U when the user U in conversation stops speaking or the user U in conversation reduces the volume of speech. On the other hand, for example, when the user U continues the conversation without reducing the volume, the information providing apparatus 2 can determine that the voice advertisement is not accepted by the user U.

  Further, the information providing apparatus 2 can determine that the voice advertisement has been received by the user U when, for example, the user U speaks about the voice advertisement or the user U performs an action of turning off the device 4. . On the other hand, the information providing apparatus 2 can determine that the voice advertisement is not accepted by the user U when the user U makes a negative utterance with respect to the advertisement target product or service.

  Further, the information providing apparatus 2 may, for example, provide a voice message when the user U turns his / her line of sight toward the smart speaker 1 for a certain time or more during the advertisement output period, or when the moving user U stops moving for a certain time during the advertisement output period. It can be determined that the advertisement is received by the user U. On the other hand, for example, when the moving user U continues moving after the voice advertisement is output, the information providing apparatus 2 can determine that the voice advertisement is not accepted by the user U.

  When the information providing apparatus 2 determines that the voice advertisement is received by the user U, the information providing apparatus 2 updates the advertising effect of the voice advertisement (step S4). For example, the information providing apparatus 2 adds an advertisement effect corresponding to a click on a text advertisement or a banner advertisement. Specifically, the information providing apparatus 2 can calculate the ratio of the number of times the voice advertisement is received with respect to the number of times the voice advertisement is output, and can set the calculated ratio as the voice advertisement effect. Thereby, an advertisement effect index corresponding to CTR (Click Through Rate) can be obtained in the voice advertisement.

  As described above, the information providing apparatus 2 can measure a new advertisement effect by the voice advertisement, and can thereby provide an appropriate advertisement effect index to the advertiser.

[2. Configuration of Smart Speaker 1]
Next, the configuration of the smart speaker 1 according to the embodiment will be specifically described. FIG. 3 is a diagram illustrating a configuration example of the smart speaker 1 according to the embodiment. As shown in FIG. 3, the smart speaker 1 includes a communication unit 10, an audio output device 11, an audio input device 12, an imaging unit 13, a storage unit 14, and a control unit 15.

  The communication unit 10 is a communication interface capable of communicating with devices such as the information providing device 2, the terminal device 3, the device 4, and the sensor device 5 via the network 6. The control unit 15 can transmit / receive information to / from the information providing device 2, the terminal device 3, the device 4, and the sensor device 5 via the communication unit 10. The smart speaker 1 may be configured to communicate with the terminal device 3, the device 4, and the sensor device 5 by a communication unit other than the communication unit 10.

  For example, when the network 6 includes a LAN and a WAN and the communication unit 10 is connected to the LAN, the control unit 15 transmits and receives information to and from the terminal device 3, the device 4, and the sensor device 5 via the LAN. , Information is transmitted to and received from the information providing apparatus 2 via the LAN and WAN.

  The audio output device 11 outputs a sound wave corresponding to the electric signal to the outside of the smart speaker 1 by performing vibration according to the electric signal output from the control unit 15. The audio output device 11 includes, for example, a diaphragm and a drive mechanism that vibrates the diaphragm according to an electric signal. In the example illustrated in FIG. 3, only one audio output device 11 is illustrated, but the smart speaker 1 may be provided with a plurality of audio output devices 11.

  The voice input device 12 is a microphone, converts a sound wave input from the outside into an electric signal, and outputs the converted electric signal to the control unit 15. In the example illustrated in FIG. 3, only one voice input device 12 is illustrated, but the smart speaker 1 may be provided with a plurality of voice input devices 12.

  The imaging unit 13 includes, for example, a complementary metal oxide semiconductor (COMS) image sensor, and images the periphery of the smart speaker 1. The imaging unit 13 outputs imaging information that is an imaging result to the control unit 15. The imaging information includes information on captured images around the smart speaker 1.

  The storage unit 14 stores an operation history 20 of the user U to the smart speaker 1 and an utterance table 21 for performing spontaneous utterance.

  The operation history 20 is information in which the operation content and operation time for the smart speaker 1 are associated with each operation of the user U. The operation details include, for example, an audio operation for starting output of various contents (for example, schedule, mail, news, music, traffic information, etc.) from the audio output device 11 and outputs from the audio output device 11 of various contents. Various operations such as a voice operation to stop are included.

  FIG. 4 is a diagram illustrating an example of the utterance table 21 according to the embodiment. The utterance table 21 illustrated in FIG. 4 includes information in which “content ID”, “utterance content”, and “model” are associated with each other. “Content ID” is identification information unique to the content.

  “Speech content” is, for example, utterance information for inquiring whether to use the content. For example, “content of utterance” is “How is the news today?” Or “There are“ X ”news” if the content is news. In addition, “content of utterance” is “If the content is an email”, “E-mail is“ X ”received”, etc. If the content is a schedule, “There are“ X ”schedules today” Etc. The information “X” can be acquired from the information providing apparatus 2 by the control unit 15 and added to the utterance content.

  Further, the “utterance content” may be the content itself. In this case, the utterance content is not set in the utterance table 21, and the control unit 15 acquires content from the information providing apparatus 2 based on the content ID.

  The “model” is a timing determination model generated based on the past user U's reaction to the utterance and the context information regarding the past user U at the time of the utterance, and is different for each content. For example, the timing determination model is a regression model in which the reaction of the past user U to an utterance is an objective variable and the context related to the user U is an explanatory variable. The objective variable is also called teacher data, and the explanatory variable is also called a feature or a feature amount. In the timing determination model, the accuracy increases as the number of explanatory variables increases, but the number of explanatory variables may be one.

  The control unit 15 illustrated in FIG. 3 includes an input processing unit 31, an information output unit 32, an information acquisition unit 33, an output processing unit 34, a context acquisition unit 35, and a timing determination unit 36. The input processing unit 31 recognizes the user U's voice from the electrical signal output from the voice input device 12.

  Further, the input processing unit 31 determines the gesture of the user U from the imaging information output from the imaging unit 13. Note that the input processing unit 31 can acquire imaging information from the sensor device 5 including the imaging unit, and can determine the gesture of the user U based on the acquired imaging information.

  Further, the input processing unit 31 detects the movement of the mouth of the user U from the imaging information output from the imaging unit 13, and determines the content of the user U's mouth (silent utterance) from the detected movement of the mouth. it can. That is, the input processing unit 31 can read a lip from the imaging information output from the imaging unit 13. The input processing unit 31 uses the result of lip reading as utterance information of the user U. Note that the input processing unit 31 does not perform lip reading when the voice of the user U can be identified.

  When the input processing unit 31 recognizes that the user U has uttered a wake-up word, the information output unit 32 outputs the utterance information of the user U following the wake-up word to the information providing apparatus 2. The utterance information may be the voice information itself of the user U or text information. Further, the information output unit 32 outputs gesture information, which is information on the gesture determined by the input processing unit 31, to the information providing apparatus 2. Note that the wake-up word is a word preset in the smart speaker 1, but the information output unit 32 may output the utterance information and gesture information of the user U to the information providing apparatus 2 regardless of the utterance of the wake-up word. it can.

  In addition, for example, when the context information is not acquired from the information providing device 2, the information output unit 32 can continuously output the context information acquired by the context acquiring unit 35 to the information providing device 2 repeatedly.

  In addition, the information output unit 32 is input to the voice input device 12 during a preset period (hereinafter sometimes referred to as an advertisement output period) from when the voice advertisement output from the voice output device 11 is started. Imaging information including the sound information including the sound to be output and the imaging information output from the imaging unit 13 is output to the information providing apparatus 2. Note that the preset period starts, for example, when the output of the audio advertisement from the audio output device 11 is started and ends when the output of the audio advertisement is stopped or ended, or the audio output This is a period that starts when the output of the audio advertisement from the device 11 is started and ends after a certain period after the output of the audio advertisement is stopped or ended. The information output unit 32 acquires sound information including sound input to the voice input device or the sensor device 5 of the terminal device 3 from the terminal device 3 or the sensor device 5 to the information providing device 2 during the advertisement output period. It can also be output.

  The information acquisition unit 33 can acquire model information including a timing determination model for each content from the information providing device 2 and set the acquired timing determination model information in the utterance table 21. The information acquisition unit 33 acquires content from the information providing device 2. The output processing unit 34 converts the content acquired by the information acquisition unit 33 into an electrical signal and outputs it to the audio output unit 11. As a result, the content is output as sound from the smart speaker 1.

  When the content is acquired as character information from the information providing device 2, the output processing unit 34 converts the character information into a voice signal (electric signal) by voice synthesis processing and outputs the voice signal to the voice output device 11. Further, when the content is acquired as audio information from the information providing device 2, the output processing unit 34 converts the audio information into an audio signal (electric signal) by digital / analog conversion and outputs the audio signal to the audio output device 11.

  The context acquisition unit 35 executes content acquisition processing for acquiring context information regarding the user U. The context acquisition unit 35 can acquire context information from the terminal device 3, the device 4, and the sensor device 5 directly or via the information providing device 2.

  Specifically, the context acquisition unit 35 can acquire the surrounding information indicating the situation around the user U as at least a part of the context information. For example, the context acquisition unit 35 can acquire ambient information from sensor information output from one or more sensor devices 5 present around the user U.

  The sensor information includes at least one of illuminance information indicating brightness around the user U, temperature information indicating temperature around the user U, and humidity information indicating humidity around the user U, for example. The context acquisition unit 35 can acquire illuminance information, temperature information, and humidity information as ambient information.

  Further, the context acquisition unit 35 can acquire ambient information from device information indicating the state of one or more devices existing around the user U. Here, the one or more devices are one or more devices among the smart speaker 1, the terminal device 3, and the device 4. The device information is, for example, information indicating an operation state of the device such as device on / off or information indicating an operation state.

  For example, in the case of the terminal device 3, the information indicating the operation state includes the type of application being displayed on the terminal device 3 and the content of the content being displayed. When the device is an air conditioner, the information indicating the operation state includes information such as a set air volume and a set temperature of the air conditioner.

  Further, the context acquisition unit 35 can acquire the surrounding information from the operation history information indicating the operation history for one or more devices existing around the user U. The one or more devices are one or more devices among the smart speaker 1, the terminal device 3, and the device 4. The operation history information includes, for example, information associated with the operation content on the device and the operation time for each operation of the user U. The context acquisition unit 35 can acquire the operation history information of the smart speaker 1 from the storage unit 14.

  The context acquisition unit 35 can acquire, for example, imaging information indicating the imaging information of the user U from the imaging unit 13, the terminal device 3, the device 4, or the sensor device 5. The context acquisition unit 35 can acquire context information such as information indicating the situation of the user U from the acquired imaging information.

  In addition, the context acquisition unit 35, for example, from the sound information including the sound input to the voice input device 12, the conversation state of the user U, the speech state of the user U, the sounds around the user U (the sound of the device 4). Context information) can be acquired.

  Based on the context information acquired by the context acquisition unit 35, the timing determination unit 36 determines an utterance timing that is an utterance timing in consideration of a past user U's reaction to an utterance from the audio output device 11.

  For example, the timing determination unit 36 inputs the context information acquired by the context acquisition unit 35 as input information to the timing determination model for each content included in the utterance table stored in the storage unit 14 and performs an operation using the model. Do. The timing determination unit 36 determines whether or not the score that is the calculation result of the timing determination model is a preset threshold value.

  When the timing determination unit 36 determines that the score, which is the calculation result of the timing determination model, is greater than or equal to the threshold, the timing determination unit 36 outputs the utterance content associated with the timing determination model whose score is greater than or equal to the threshold in the utterance table 21. It is determined that it has become. In addition, when there are two or more timing determination models having a score equal to or higher than the threshold, the timing determination unit 36 outputs the utterance content associated with the timing determination model having the highest score among the timing determination models having a score equal to or higher than the threshold. It is determined that the timing has been reached.

  Note that the timing determination unit 36 can not determine that it is time to output the utterance content even when the score, which is the calculation result of the timing determination model, is determined to be greater than or equal to the threshold value. For example, when the timing determination unit 36 has not passed a preset period (hereinafter referred to as an output prohibition period) since the utterance content associated with the timing determination model whose score is equal to or greater than the threshold value was previously output, It cannot be determined that the utterance timing has come.

  In addition, the timing determination unit 36 sets the utterance content associated with the timing determination model having a score equal to or higher than the threshold value in advance from the current time to a preset period (hereinafter referred to as a set period). When the number of times (hereinafter referred to as the output upper limit number) is exceeded, it cannot be determined that the speech timing has come.

  Even when the timing determination unit 36 determines that the score, which is the calculation result of the timing determination model, is equal to or greater than the threshold, the content or utterance is output from the audio output device 11 when the user U is operating the smart speaker 1. If it is medium, it is not determined that the utterance timing has come. In this case, the timing determination unit 36 ends the operation of the smart speaker 1 by the user U if there is a timing determination model whose score is continuously equal to or higher than the threshold when the operation of the smart speaker 1 by the user U ends. It can be determined that it is time to output the utterance content after a certain period of time.

  Moreover, the timing determination part 36 can set the output prohibition period mentioned above and the output upper limit frequency in the speech table 21 for every content based on the voice operation by the user U. The information that can be set for each content is not limited to the output prohibition period and the output upper limit number. Information such as the output prohibition period and the output upper limit count may be set in the utterance table 21 in advance regardless of the setting of the user U.

  As described above, the timing determination unit 36 can determine the utterance timing and the utterance content determined to be output at the utterance timing (hereinafter, may be described as the output target utterance content) based on the context information. it can. The timing determination unit 36 determines the utterance timing and the output target utterance content by considering the utterance timing and the output target utterance content in consideration of the past reaction of the user U to the utterance from the voice output device 11. Can be determined more appropriately.

  For example, it is assumed that the user U does not react to the utterance no matter how many times the utterance is made in a situation where the user U is located in a dark place and there is a collection vehicle (for example, a garbage collection vehicle) around the user U. In this case, the timing determination model is generated so that the score to be output when the user U is located in a dark place and the collection vehicle is around the user U is smaller than the threshold value. Therefore, when the context information indicates that the user U is located in a dark place and there is a collection vehicle around the user U, the timing determination unit 36 determines that it is not the utterance timing.

  Further, it is assumed that the user U does not react to the utterance in a state where the dishwasher and the microwave oven are used together. In this case, the timing determination model is generated so that the score output when the dishwasher and the microwave oven are both used is smaller than the threshold value. Therefore, when the context information indicates that both the dishwasher and the microwave oven are used, the timing determination unit 36 determines that it is not the utterance timing.

  Further, when the user U is operating the terminal device 3 (for example, browsing a web page on the terminal device 3 or playing music on the terminal device 3), it is assumed that the user U does not react to the utterance. In this case, the timing determination model is generated so that the score output when the user U is operating the terminal device 3 is smaller than the threshold value. For this reason, when the context information indicates that the user U is operating the terminal device 3, the timing determination unit 36 determines that it is not the utterance timing.

  Further, for example, when the user U's conversation continues after the start of the utterance or when the user U utters “Stop”, the timing determination model uses the incorrect information as the answer data and the context information at the time of the utterance as the feature amount. Generated. In this case, the timing determination unit 36 can determine that it is not the utterance timing in a context in which the user U continues the conversation of the user U or a context in which the user U utters “Stop”.

  In this manner, the timing determination unit 36 can determine that it is the utterance timing when the context regarding the current user U is a context suitable for utterance. In addition, since the timing determination model is generated for each content, an appropriate speech timing for each content is determined. For example, in the morning time zone, the user U often reacts to content related to the operation state of transportation facilities or news content. For this reason, a timing determination model in which the morning time zone is likely to be the utterance timing is generated for the content related to the operation state of the transportation facility and the content of the news.

  The output processing unit 34 outputs the output target utterance content determined by the timing determination unit 36 from the voice output device 11 at the utterance timing determined by the timing determination unit 36. For example, when the utterance table 21 is in the state shown in FIG. 4 and the output utterance content is “utterance content XA”, the output processing unit 34 outputs a telegraph signal based on the utterance content XA to the voice output device 11. The utterance content XA is output from the audio output unit 11 as audio.

[3. Configuration of information providing apparatus 2]
Next, the configuration of the information providing apparatus 2 according to the embodiment will be specifically described. FIG. 5 is a diagram illustrating a configuration example of the information providing apparatus 2 according to the embodiment. As illustrated in FIG. 5, the information providing apparatus 2 includes a communication unit 41, a storage unit 42, and a control unit 43.

  The communication unit 41 is a communication interface that can communicate with devices such as the smart speaker 1, the terminal device 3, the device 4, and the sensor device 5 via the network 6. The control unit 43 can exchange information with the information providing device 2, the smart speaker 1, the terminal device 3, the device 4, and the sensor device 5 through the communication unit 41.

  The storage unit 42 includes an utterance table storage unit 51, a content storage unit 52, a voice advertisement storage unit 53, a user information storage unit 54, a context storage unit 55, and an output mode determination information storage unit 56.

  The utterance table storage unit 51 stores information of the utterance table 21 for each smart speaker 1. FIG. 6 is a diagram illustrating an example of an utterance table stored in the utterance table storage unit 51 according to the embodiment. The utterance table 71 illustrated in FIG. 6 includes information in which “content ID”, “utterance content”, and “model” are associated with each other for each “device ID”.

  The “content ID”, “utterance content”, and “model” in the utterance table 71 are the same information as the “content ID”, “utterance content”, and “model” in the utterance table 21. “Device ID” is identification information unique to each smart speaker 1.

  The content storage unit 52 illustrated in FIG. 5 stores various types of content to be provided to the smart speaker 1. FIG. 7 is a diagram illustrating an example of a content table stored in the content storage unit 52 according to the embodiment. The content table 72 illustrated in FIG. 7 is information in which “content ID” and “content” are associated with each other.

  “Content” includes audio output content and visual output content. The audio output content is content that is output in sound, and the visual output content is content that is output in a form other than sound, such as characters and images.

  The voice advertisement storage unit 53 illustrated in FIG. 5 stores voice advertisement information and the like. FIG. 8 is a diagram illustrating an example of a voice advertisement table stored in the voice advertisement storage unit 53 according to the embodiment. The voice advertisement table 73 shown in FIG. 8 is information in which “advertisement ID”, “voice advertisement”, “output count”, “acceptance count”, and “acceptance rate” are associated with each other. “Advertisement ID” is identification information unique to each voice advertisement.

  The “voice advertisement” is a content of the voice advertisement, and is output from, for example, the voice output device 11 of the smart speaker 1 or the voice output device of the terminal device 3. The “voice advertisement” in the voice advertisement table 73 is the voice advertisement content itself, but may be information indicating the storage location of the voice advertisement content.

  The “number of times of output” is information indicating the number of times the voice advertisement is provided to the user U, and is incremented by the control unit 43 every time the voice advertisement is output to the smart speaker 1 or the terminal device 3, for example. The “acceptance count” is the number of times the voice advertisement is accepted by the user U, and is incremented by the control unit 43 every time the voice advertisement is accepted by the user U, for example. “Acceptance rate” is the ratio of the number of times of acceptance to the number of outputs, and is calculated by the control unit 43, for example.

  For example, the voice advertisement table 73 shown in FIG. 8 indicates that the voice advertisement with the advertisement ID “A101” has an output count of 29817, an acceptance count of 827, and an acceptance rate of 0.0278. The voice advertisement with the advertisement ID “A102” indicates that the output count is 8372, the acceptance count is 352, and the acceptance rate is 0.0420.

  The user information storage unit 54 illustrated in FIG. 5 stores user U information. FIG. 9 is a diagram illustrating an example of a user information table stored in the user information storage unit 54 according to the embodiment. The user information table 74 illustrated in FIG. 9 is information in which “user ID”, “user attribute”, “device ID”, and “device address” are associated with each other.

  “User ID” is identification information unique to each user U. “User attribute” is information indicating the attribute of the user U. The attribute of the user U is, for example, demographic attributes such as address and occupation in addition to gender and age, but may include psychographic attributes indicating the preference of the user U and the like. The “device ID” includes identification information unique to the smart speaker 1 owned by the user U and identification information unique to the terminal device 3 owned by the user U. The “device address” is an address on the network 6 of the smart speaker 1 or the terminal device 3 owned by the user U.

  The output mode determination information storage unit 56 includes output mode determination information for determining the output mode of the content that is the target of the output request when there is a content output request from the smart speaker 1. As described above, the output mode includes at least one of the content output type and the content output destination.

  The output mode determination information is, for example, information that defines the relationship between the context related to the user U and each output mode, and includes a model or a table. The table included in the output mode determination information is a table that defines the relationship between the context related to the user U and each output mode. The model included in the output mode determination information is a model generated by learning by the learning unit 64.

  5 includes an information acquisition unit 61, an information output unit 62, a context acquisition unit 63, a learning unit 64, an output mode determination unit 65, a detection unit 66, a determination unit 67, and an advertisement. An effect updating unit 68.

  The information acquisition unit 61 acquires information transmitted from the smart speaker 1. For example, the information acquisition unit 61 can acquire input information indicating the instruction of the user U from the smart speaker 1 (for example, utterance information, information indicating operation content by gesture). Moreover, the information acquisition part 61 can acquire the imaging information which shows the user U's captured image from the smart speaker 1, the terminal device 3, the apparatus 4, or the sensor apparatus 5, for example.

  In addition, the information acquisition unit 61 displays operation history information indicating an operation history for one or more devices (for example, the smart speaker 1, the terminal device 3, the device 4, and the like) existing around the user U. 3 or from the device 4.

  Based on the output mode determined by the output mode determination unit 65, the information output unit 62 acquires content (acoustic output content) corresponding to the input information of the user U from the content storage unit 52. For example, when the output type determined by the output mode determination unit 65 is an auditory output, the information output unit 62 acquires content corresponding to the input information of the user U and audio content from the content storage unit 52. .

  Further, when the output type determined by the output mode determination unit 65 is visual output, the information output unit 62 is content corresponding to the input information of the user U and is content of characters or images (visual output content). ) Is acquired from the content table 72 of the content storage unit 52. In addition, when the output type determined by the output mode determination unit 65 is an auditory output or a visual output, the information output unit 62 is content corresponding to the input information of the user U, and includes voice and characters (or images). Is acquired from the content table 72 of the content storage unit 52.

  The information output unit 62 outputs the content acquired from the storage unit 42 to at least one of the smart speaker 1 and the terminal device 3 based on the output mode determined by the output mode determination unit 65. The information output unit 62 acquires the device address of the output destination determined by the output mode determination unit 65 from the user information table 74 of the storage unit 42, and transmits the content to the acquired device address.

  For example, when the output destination determined by the output mode determination unit 65 is the smart speaker 1, the information output unit 62 transmits the content acquired from the storage unit 42 to the device address of the smart speaker 1. In addition, when the output destination determined by the output mode determination unit 65 is the terminal device 3, the information output unit 62 transmits the content acquired from the storage unit 42 to the device address of the terminal device 3.

  The context acquisition unit 63 executes content acquisition processing for acquiring context information regarding the user U. The context acquisition unit 63 can acquire context information from the information acquired by the information acquisition unit 61. The context information acquired by the context acquisition unit 63 is the same as the context information acquired by the context acquisition unit 35, but even if the context information acquired by the context acquisition unit 35 is partly or entirely different information. Good.

  The context information acquired by the context acquisition unit 63 includes input type information indicating the type of instruction from the user U. The input type information is information indicating, for example, whether the instruction of the user U is voice, gesture, or mouth movement. When the captured image of the user U is transmitted from the smart speaker 1, the terminal device 3, the device 4, or the sensor device 5 to the information providing device 2, the context acquisition unit 63 determines the user U's gesture from the captured image of the user U. The input type information can also be acquired by determining the movement of the mouth.

  The learning unit 64 can generate the timing determination model of the utterance table 71 for each user U and for each content based on the context information for each user U acquired by the context acquisition unit 63. As described above, the timing determination model is a model generated based on the past user U's reaction to the utterance and the context information regarding the past user U at the time of the utterance.

  The learning unit 64 starts the spontaneous utterance from the smart speaker 1, and the operation of the user U included in the context information or the utterance of the user U becomes a spontaneous utterance from the smart speaker 1 during a preset period. Whether it is a positive reaction or not is used as teacher data.

  For example, when the user U's utterance with respect to the spontaneous utterance of the smart speaker 1 is positive, the learning unit 64 determines that the reaction is a positive reaction with respect to the spontaneous utterance of the smart speaker 1. For example, the learning unit 64 responds positively to a spontaneous utterance such as “How about today's news?” If the utterance of the user U is, for example, “Thank you” or “Yes”. Can be determined.

  Further, the learning unit 64 does not utter the user U with respect to the spontaneous utterance of the smart speaker 1 or if the utterance of the user U with respect to the spontaneous utterance with the smart speaker 1 is negative, the spontaneous utterance of the smart speaker 1. It is determined that the response is not positive. For example, the learning unit 64 responds to a spontaneous utterance such as “How about today's news?” In the case where there is no utterance of the user U, or the utterance of the user U is “not required”, “stop”, etc. In some cases, it can be determined that there is no positive response.

  Note that whether or not the reaction of the user U to the spontaneous utterance is positive is not limited to the above-described example. For example, the learning unit 64 can determine that the reaction is a positive response to the spontaneous utterance when the user U speaks. In addition, the learning unit 64 can determine that the reaction is not a positive response to the spontaneous utterance when the user U moves away from the smart speaker 1.

  The learning unit 64 uses the teacher data as to whether or not the reaction is a positive response to the spontaneous utterance as described above, and is included in the context information in a predetermined period after starting the spontaneous utterance. The timing determination model is generated and updated by performing machine learning using the above information as a feature amount.

  The timing determination model is not limited to the above-described example, and may be a model generated using, for example, SVM (Support Vector Machine) or other machine learning methods. The generation of the timing determination model may be performed using a deep learning technique. For example, the generation of the timing determination model may be appropriately performed using various deep learning techniques such as DNN (Deep Neural Network), RNN (Recurrent Neural Network), and CNN (Convolutional Neural Network).

  The learning unit 64 sets the generated timing determination model in the utterance table 71. The learning unit 64 can update the timing determination model set in the utterance table 71 based on the reaction of the user U to the newly acquired utterance and the context information. For example, the learning unit 64 can update the timing determination model in a time zone (for example, midnight) when the processing load on the information providing apparatus 2 is low.

  The learning unit 64 can generate a timing determination model common to a plurality of users U (hereinafter, sometimes referred to as a common determination model) for each content or specific content. In this case, the learning unit 64 can generate a common determination model based on the past responses of the plurality of users U to the utterance and the context information regarding the plurality of past users U at the time of utterance.

  In addition, after generating a common determination model for each content, the learning unit 64 determines each user U based on the reaction and context information of each user U with respect to an utterance newly acquired based on the common determination model. A unique timing determination model can also be generated.

  The learning unit 64 can also generate a timing determination model common to a plurality of contents and common to a plurality of users U. In this case, the learning unit 64 uses a timing determination model common to the plurality of users U based on the reactions of the plurality of users U in the past with respect to an arbitrary utterance and the context information regarding the plurality of users U in the past at an arbitrary utterance. Can be generated.

  Further, the learning unit 64 can generate an output mode determination model for each output mode based on the past user U's reaction to the content and the context information regarding the past user U. For example, machine learning can be performed by using whether the user U's reaction is a negative reaction or a positive reaction as teacher data and using context information at the time of the user U's reaction as a feature amount. .

  The negative reaction is, for example, a negative utterance (for example, “do not need” or “stop”) of the user U when the content is output as sound from the smart speaker 1. Moreover, a negative reaction is non-operation with respect to the terminal device 3 of the user U when a content is output as a character or an image from the terminal device 3, for example.

  Moreover, a positive reaction is a state in which there is no negative utterance of the user U when content is output as audio from the smart speaker 1, for example. The positive reaction is, for example, an operation on the terminal device 3 when the content is output from the terminal device 3 as characters or images.

  When there is a content output request, the output mode determination unit 65 determines the output mode of content provided to the user U based on the context information acquired by the context acquisition unit 63. Although the output type includes an output type and an output destination, the output mode determination unit 65 can also determine only one of the output type and the output destination. The output mode determination unit 65 determines the output mode using the above-described output mode determination information stored in the output mode determination information storage unit 56.

  The output mode determination unit 65 can determine the output mode based on the status information indicating the status of the user U included in the context information. For example, when the user U is moving, the output mode determination unit 65 can determine a mode in which the output type is an auditory output and the output destination is the smart speaker 1 as the content output mode. As a result, the user U can grasp the content without looking at the screen of the terminal device 3 while moving.

  In addition, the output mode determination unit 65 can use a device close to the current position of the user U among the smart speaker 1 and the terminal device 3 as an output destination, thereby facilitating the grasping of the content by the user U. it can.

  In addition, when the user U is in conversation, the output mode determination unit 65 can determine a mode in which the output type is visual output and the output destination is the terminal device 3 as the content output mode. Thereby, the user U can grasp the content without interrupting the conversation.

  Moreover, the output mode determination part 65 can determine an output mode based on the surrounding information which shows the surrounding condition of the user U contained in context information. For example, when there is another person around the user U, the output mode determination unit 65 can determine the mode in which the output type is the visual output and the output destination is the terminal device 3 as the content output mode. . Thus, for example, when the content is a schedule of the user U or an email to the user U, the content is not output as voice from the smart speaker 1, so that the schedule or email of the user U can be prevented from being known to others. it can.

  The output mode determination unit 65 may determine, as a content output mode, a mode in which the output type is a visual output and the output destination is the terminal device 3 when another person around the user U is sleeping. it can. Thereby, for example, it is possible to avoid a situation in which another person sleeping around the user U is awakened by the output of the content.

  Further, the output mode determination unit 65 can determine a mode in which the output type is a visual output and the output destination is the terminal device 3 as the content output mode when the sound around the user U is loud. As a result, it is possible to avoid a situation in which the content cannot be grasped by noise around the user U or sound generated by the device 4.

  Further, the output mode determination unit 65 sets the mode in which the output type is an auditory output and the output destination is the smart speaker 1 when there is no other person around the user U or when the sound around the user U is small. It can be determined as a content output mode. Thereby, the user U can grasp the content without looking at the screen of the terminal device 3.

  In addition, when the input type to the smart speaker 1 is a gesture or movement of the mouth, the output mode determination unit 65 sets the output type as a visual output and sets the output destination as the terminal device 3 as the content output mode. Can be determined. Further, in the case of utterance (speech utterance) to the smart speaker 1, the output mode determination unit 65 determines a mode in which the output type is an auditory output and the output destination is the smart speaker 1 as the content output mode. be able to.

  Thereby, the user U can input to the smart speaker 1 with the input type according to the desired output mode, and the user U can confirm the content in the desired output mode from the smart speaker 1 or the terminal device 3. it can. When the output mode determination unit 65 determines the output mode according to the input type based on the output mode determination model, for example, the output mode determination model increases the weight using the input type as the feature amount or inputs the feature amount. By making only the type, the output mode determination unit 65 can determine the output mode according to the input type. Note that the output mode determination unit 65 can also determine the output mode according to the input type based on the output mode determination table in which the input type and the output mode are associated with each input type.

  Moreover, when the output mode determination unit 65 includes an output mode determination model for each output mode as the output mode determination information, the output mode determination unit 65 inputs context information to the output mode determination model for each output mode. The output mode determination unit 65 determines the output mode of the content based on the output of the output mode determination model for each output mode.

  For example, when an output mode determination model is generated using teacher data as to whether or not the reaction of the user U is a negative response, the output mode determination unit 65 corresponds to the output mode determination model with the lowest output score. The output mode can be determined as the content output mode. Further, when the output mode determination model is generated using whether or not the reaction of the user U is a positive response as teacher data, the output mode determination unit 65 corresponds to the output mode determination model with the highest score to be output. The output mode can be determined as the content output mode.

  Thus, since the output mode determination unit 65 can determine the output mode of the content according to the situation of the user U and the situation around the user U, the content can be appropriately provided to the user U. it can.

  The detection unit 66 detects the behavior of the user U when an audio advertisement is output from the audio output device 11 of the smart speaker 1 or the audio output device (not shown) of the terminal device 3. The detection unit 66 can detect the behavior of the user U when a voice advertisement is output by performing image analysis on the imaging information acquired by the information acquisition unit 61.

  For example, the detection unit 66, based on the imaging information acquired by the information acquisition unit 61, the movement of the eyes of the user U, the movement of the head of the user U, the movement of the mouth of the user U, the movement of the hand of the user U, In addition, at least one physical behavior among the movements of the feet of the user U can be detected as the behavior of the user U.

  Further, the detection unit 66 can detect the state of work performed by the user U based on the imaging information acquired by the information acquisition unit 61. For example, the detection unit 66 can detect whether or not the user U has interrupted operations such as washing dishes, sewing with a sewing machine, and cooking.

  In addition, the detection unit 66 can detect the behavior of the user U when a voice advertisement is output by acoustically analyzing the sound information acquired by the information acquisition unit 61. For example, based on the sound information, the detection unit 66 behaves in the conversation of the user U, asks the user U by speaking to the smart speaker 1, the behavior in the work performed by the user U, and the user U's device 4 The behavior can be detected.

  Specifically, based on the sound information, the detection unit 66 can detect such behavior that the user U during conversation stops speaking and the user U during conversation decreases the volume of speech. In addition, the detection unit 66 can detect behaviors such as an utterance for the user U to search for information and an utterance for the user U to confirm information.

  Moreover, the detection part 66 can detect whether the user U interrupted operations, such as dishwashing and cooking, based on sound information. For example, the detection unit 66 can determine that the dishwashing is interrupted when the sound discharged from the water faucet disappears or the sound of washing the dishes disappears.

  In addition, since the device sound that is the sound output from the device 4 stops when the user U turns off the device 4, the detection unit 66 behaves to turn off the device 4 based on the sound information. Can be detected. The device 4 that detects the off based on the sound information by the detection unit 66 is a device that emits a relatively loud sound such as a microwave oven, a washing machine, a dishwasher, a sewing machine, a television receiver, and a radio receiver. .

  Further, the detection unit 66 can detect the behavior of the user U with respect to the smart speaker 1, the terminal device 3, or the device 4 based on the operation history information acquired by the information acquisition unit 61. For example, the detection unit 66 can detect a behavior in which the user U turns off or on the device, a behavior in which the user U searches for information, and the like based on the operation history information.

  The determination unit 67 determines whether or not the voice advertisement has been received by the user U based on the behavior of the user U detected by the detection unit 66. For example, the determination unit 67 determines that the voice advertisement has been received by the user U when the physical behavior of the user U detected by the detection unit 66 is a specific behavior.

  The specific behavior is, for example, that the user U directs his / her line of sight to the smart speaker 1 for a certain time or more in the advertisement output period, the user U who is moving at the start of outputting the voice advertisement stops moving for a certain time or more in the advertisement output period, and This is a behavior such as the user U whispering during the advertisement output period. In addition, the specific behavior is such that the user U who has been continuously moving before starting the output of the voice advertisement stops his hand for a certain time or more during the advertisement output period, and the user U has made a specific gesture. .

  The determination unit 67 determines that the voice advertisement has been received by the user U when the behavior of the user U detected by the detection unit 66 based on the sound information is a specific behavior. Specific behaviors are, for example, that the user U in conversation stops speaking, the user U in conversation decreases the volume of speech, the user U speaks about voice advertisements, the user U interrupts work, and the user U Behaves like turning off the device. Note that the utterance regarding the voice advertisement by the user U is, for example, “So then?”, “Continue?”, And the like.

  The determination unit 67 determines that the voice advertisement has been received by the user U when the behavior of the user U detected by the detection unit 66 based on the operation history information is a specific behavior. The specific behavior is, for example, a behavior in which the user U searches for a product or service to be advertised for voice advertisement, and the user U turns off the device.

  The determination unit 67 can determine that the voice advertisement is not accepted by the user U when the behavior of the user U is a negative behavior. The negative behavior is, for example, a behavior in which the user U makes a negative utterance and the user U makes a negative physical behavior.

  For example, the determination unit 67 can determine that the user U behaves negatively when the user U makes a negative utterance such as “stop”, “do not want to hear”, and “dislike”. Further, for example, the determination unit 67 can determine that the user U behaves negatively when the user U covers his ear with his hand.

  Note that when the determination unit 67 determines that the behavior of the user U is not a specific behavior, the determination unit 67 can also determine that the voice advertisement is not accepted by the user U. Thereby, the determination part 67 does not need to determine whether the user U behaved negatively, and a processing load is reduced.

  In addition, when the determination unit 67 determines that the behavior of the user U is neither a specific behavior nor a negative behavior, the determination unit 67 determines that the non-acceptance of the voice advertisement by the user U is low, and the behavior of the user U is negative. In the case of behavior, it can be determined that the non-acceptance level of the voice advertisement by the user U is high.

  Further, the determination unit 67 may not perform a process of determining that the voice advertisement is received by the user U when the behavior of the user U is a specific behavior. For example, the determination unit 67 determines that the voice advertisement is not accepted by the user U when the behavior of the user U is negative, and the voice of the user U when the behavior of the user U is not negative. It can be determined that the advertisement is received by the user U.

  The specific behavior and the negative behavior described above are set according to the attribute of the user U. The attribute of the user U includes, for example, at least one of sex, age, address, and occupation. For example, when the user U is a child, the determination unit 67 determines that the voice advertisement has been received by the user U when the user U jumps and the user U dances.

  In addition, the determination part 67 can also change the specific behavior used as the reference | standard which determines with having received by the user U according to the time length and kind of a voice advertisement. Further, the determination unit 67 determines whether or not the voice advertisement is received by the user U every predetermined period (for example, 5 seconds) in a period (for example, 30 seconds) in which one voice advertisement is output. You can also

  When the advertisement effect update unit 68 determines that the voice advertisement has been received by the user U, the advertisement effect update unit 68 updates the number of times the voice advertisement is received in the voice advertisement table 73. Thereby, the advertisement effect equivalent to the click of a text advertisement or a banner advertisement can be acquired with respect to a voice advertisement. Then, the advertisement effect update unit 68 can calculate the acceptance rate that is the ratio of the number of times the voice advertisement is received with respect to the number of times the voice advertisement is output, and can set the calculated acceptance rate in the voice advertisement table 73 as the sound advertisement effect. Thereby, the advertisement effect index corresponding to CTR can be obtained in the voice advertisement.

  Moreover, the advertisement effect update unit 68 can acquire the number of conversions of the voice advertisement from an external device. The number of conversions of the voice advertisement is the number of times that the purpose of the voice advertisement such as purchase of products and services, sample application, and brochure application is achieved. The advertisement effect updating unit 68 can calculate a conversion rate that is a ratio of the number of conversions of the voice advertisement to the number of receptions of the voice advertisement, and can set the calculated conversion rate as the voice advertisement effect. Thereby, an advertisement effect index corresponding to CVR (Conversion Rate) can be obtained in the voice advertisement.

  When the non-acceptance level is determined by the determination unit 67, the advertisement effect update unit 68 can calculate the ratio of the non-acceptance frequency for each non-acceptance level of the audio advertisement to the output frequency of the audio advertisement. In this case, for example, the advertisement effect update unit 68 can subtract the number of non-acceptances with a non-acceptance level of 2 from the number of accepted voice advertisements, and set the subtraction result as the number of accepted voice advertisements.

[4. Processing flow of information processing system 100]
Next, a procedure of speech control processing by the information processing system 100 according to the embodiment will be described. 10 and 11 are flowcharts illustrating an example of the utterance control process by the information processing system 100 according to the embodiment.

  First, the speech control process of the smart speaker 1 will be described. As shown in FIG. 10, the control unit 15 of the smart speaker 1 determines whether or not content output processing is being performed (step S10). In the process of step S10, the control unit 15 treats, for example, a period from when the control unit 15 receives a content output request from the user U to when the output of the content corresponding to the output request is completed, during the content output process.

  When determining that the content is not being output (step S10: No), the control unit 15 determines whether or not the speech process is being performed (step S11). In the process of step S11, for example, the control unit 15 treats the period from the start of the utterance output from the voice input device 12 to the end of the period during which the request of the user U for the utterance can be received as the utterance process is in progress. .

  When it determines with the utterance process not being performed (step S11: No), the control part 15 acquires the context information regarding the user U (step S12), and uses the acquired context information for each timing determination model included in the utterance table 21. Input (step S13).

  Subsequently, the control unit 15 determines whether there is a timing determination model that outputs a score equal to or higher than a preset threshold (step S14). If it is determined that there is a timing determination model that outputs a score equal to or higher than the threshold (step S14: Yes), the control unit 15 determines whether there are a plurality of timing determination models that output a score equal to or higher than the threshold (step S14). S15).

  When it is determined that there are a plurality of timing determination models (step S15: Yes), the control unit 15 selects the timing determination model with the highest score (step S16). Moreover, the control part 15 selects the timing determination model which outputs the score more than a threshold value, when it determines with there being not two or more timing determination models (step S15: No) (step S17). The control unit 15 acquires the utterance content associated with the selected timing determination model from the utterance table 21, and outputs the acquired utterance content from the voice input device 12 (step S18).

  When the process of step S18 is completed, the control unit 15 determines that the content is being output (step S10: Yes), determines that the speech process is being performed (step S11: Yes), or exceeds the threshold value. When it determines with there being no timing determination model which outputs a score (step S14: No), the process shown in FIG. 10 is complete | finished.

  Next, processing of the information providing apparatus 2 will be described. As illustrated in FIG. 11, the control unit 43 of the information providing device 2 acquires reaction information indicating the reaction of the user U when the smart speaker 1 speaks spontaneously (step S20). Moreover, the control part 43 acquires the context information regarding the user U when the smart speaker 1 speaks spontaneously (step S21). The control unit 43 stores the reaction information acquired in step S20 and the context information acquired in step S21 in the storage unit 42 (step S22).

  Subsequently, the control unit 43 generates or updates a timing determination model for each content based on the reaction information and context information stored in the storage unit 42 (step S23), and ends the process illustrated in FIG. For example, when the timing determination model of the content for which the timing determination model is not generated in the utterance table 71 is generated, the control unit 43 adds the timing determination model to the utterance table 71. In addition, when the timing determination model of the utterance table 71 is updated, the control unit 43 overwrites the updated timing determination model on the utterance table 71.

  Next, output control processing by the information processing system 100 will be described. FIG. 12 is a flowchart illustrating an example of output control processing by the information processing system 100 according to the embodiment.

  As shown in FIG. 12, the control unit 43 determines whether or not there is a content output request from the smart speaker 1 (step S30). When it is determined that there is a content output request (step S30: Yes), the control unit 43 acquires context information regarding the user U of the smart speaker 1 that has made the output request (step S31).

  Then, the control unit 43 determines the output mode of the content requested to be output based on the acquired context information (step S32). The control unit 43 outputs the content in the output mode determined in step S32 (step S33). When the process of step S33 is completed or when it is determined that there is no output request (step S30: No), the control unit 43 ends the process illustrated in FIG.

  Next, audio information effect determination processing by the information processing system 100 will be described. FIG. 13 is a flowchart illustrating an example of the sound information effect determination process by the information processing system 100 according to the embodiment.

  As illustrated in FIG. 13, the control unit 43 of the information providing apparatus 2 determines whether it is the output timing of the voice advertisement (Step S <b> 40). When it determines with it being the output timing of an audio advertisement (step S40: Yes), the control part 43 outputs an audio advertisement to the smart speaker 1 (step S41).

  Subsequently, the control unit 43 increments the number of times the voice advertisement is output in step S41 in the voice advertisement table 73 (step S42). Further, the control unit 43 acquires imaging information, sound information, and device operation information in the audio output period from at least one of the smart speaker 1, the terminal device 3, the device 4, and the sensor device 5 (step S43).

  The control unit 43 detects the behavior of the user U based on the information acquired in step S43 (step S44), and determines whether or not the detected behavior is a specific behavior (step S45). When the control unit 43 determines that the behavior detected in step S44 is a specific behavior (step S45: Yes), the control unit 43 determines that the voice advertisement is received by the user U (step S46). The number of times of acceptance of the voice advertisement output in step S41 is incremented (step S47).

  On the other hand, when the control unit 43 determines that the behavior detected in step S44 is not a specific behavior (step S45: No), the control unit 43 determines that the voice advertisement is not accepted by the user U (step S48). When the process of step S47 or step S48 ends, or when it is determined that it is not the output timing of the voice advertisement (step S40: No), the control unit 43 ends the process shown in FIG.

[5. (Modification)
In the example described above, the learning unit 64 that generates the timing determination model is provided in the information providing apparatus 2, but the learning unit 64 may be provided in the smart speaker 1. In the example described above, the context acquisition unit 35 and the timing determination unit 36 are provided in the smart speaker 1, but the context acquisition unit 35 and the timing determination unit 36 may be provided in the information providing apparatus 2.

  In the example described above, the utterance timing is determined using the timing determination model. However, the utterance timing may be determined in consideration of the reaction of the user U with respect to the past utterance, and is not limited to the above example. For example, the smart speaker 1 determines whether the current context is a context suitable for utterance based on determination condition information generated based on reaction information and context information with respect to a past utterance instead of the timing determination model. Can also be determined. The determination condition information includes a context condition for determining that it is the utterance timing.

  In the above-described example, the user U's gesture and mouth movement are detected by the smart speaker 1, but the user U's gesture and mouth movement may be detected by the information providing apparatus 2.

  In the above-described example, the voice advertisement is described as an example of the voice information output from the smart speaker 1, but the voice information output from the smart speaker 1 is not limited to the voice advertisement. For example, the control unit 43 of the information providing device 2 behaves as the user U when the smart speaker 1 outputs push utterance type voice information such as “Today is sunny” or “Today is a work holiday”. Based on the above, it can be determined whether or not the audio information has been received by the user U. And the control part 43 of an information provision apparatus can calculate the ratio of the frequency | count of reception of the audio | voice information with respect to the frequency | count of output of audio | voice information, and can make the calculated ratio the audio | voice information effect. Thereby, the effect parameter | index equivalent to CTR can be obtained in audio | voice information.

  Note that the audio information that is the target of the effect determination is not limited to the audio information output from the information providing device 2 to the smart speaker 1, and may be audio information stored in the smart speaker 1.

  Further, in the above-described example, the information processing system 100 may determine that the user U has received even if the user U is not a positive reaction, but the user U is a positive reaction. It can also be determined that the user U has received it. That is, the information processing system 100 accepts the state of the user U determined to have received a positive reaction from the user U in the speech control process and the output control process in the voice information effect determination process, and otherwise Can be unacceptable to the user. Further, the information processing system 100 sets the state of the user U determined to be accepted by the user U in the speech information effect determination process in the speech control process and the output control process as a positive reaction of the user U, and otherwise This can be a negative user response.

[6. program〕
Each of the smart speaker 1 and the information providing apparatus 2 in the above-described embodiment is realized by, for example, a computer 200 configured as shown in FIG. 14 executing a program. FIG. 14 is a diagram illustrating an example of a hardware configuration of a computer that executes a program. A computer 200 includes a central processing unit (CPU) 201, a random access memory (RAM) 202, a read only memory (ROM) 203, a hard disk drive (HDD) 204, a communication interface (I / F) 205, an input / output interface (I). / F) 206 and media interface (I / F) 207.

  The CPU 201 operates based on a program stored in the ROM 203 or the HDD 204 and controls each unit. The ROM 203 stores a boot program executed by the CPU 201 when the computer 200 is started up, a program depending on the hardware of the computer 200, and the like. The HDD 204 stores data used by programs executed by the CPU 201. The communication interface 205 receives data from other devices via the network 6 and sends the data to the CPU 201, and transmits data generated by the CPU 201 to other devices via the network 6.

  The CPU 201 controls an output device such as a display and a printer and an input device such as a keyboard and a mouse via the input / output interface 206. The CPU 201 acquires data from the input device via the input / output interface 206. Further, the CPU 201 outputs the generated data to the output device via the input / output interface 206.

  The media interface 207 reads a program or data stored in the recording medium 208 and provides it to the CPU 201 via the RAM 202. The CPU 201 loads the program from the recording medium 208 onto the RAM 202 via the media interface 207, and executes the loaded program. The recording medium 208 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or PD (Phase change rewritable disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory. Etc.

  When the computer 200 functions as the smart speaker 1 according to the above-described embodiment, the CPU 201 of the computer 200 executes a program loaded on the RAM 202, whereby the input processing unit 31, the information output unit 32, and the like illustrated in FIG. Each function of the information acquisition unit 33, the output processing unit 34, the context acquisition unit 35, and the timing determination unit 36 is realized. For example, the HDD 204 stores the same information as the storage unit 14 illustrated in FIG.

  Further, when the computer 200 functions as the information providing apparatus 2 according to the above-described embodiment, the CPU 201 of the computer 200 executes the program loaded on the RAM 202, whereby the information acquisition unit 61 and the information output illustrated in FIG. The functions of the unit 62, the context acquisition unit 63, the learning unit 64, the output mode determination unit 65, the detection unit 66, the determination unit 67, and the advertisement effect update unit 68 are realized. For example, the HDD 204 stores the same information as the storage unit 42 illustrated in FIG.

  The CPU 201 of the computer 200 reads the program from the recording medium 208 and executes it, but as another example, the program may be acquired from another device via the network 6.

[7. effect〕
As described above, the information providing apparatus 2 (an example of an information processing apparatus) according to the embodiment detects the behavior of the user U when the voice advertisement is output from the voice output device 11, and the detection unit 66. And a determination unit 67 that determines whether or not the voice information is received by the user U based on the behavior detected by the user U. Thereby, the effect of the audio | voice information provided to the user U can be measured.

  The detection unit 66 detects the behavior of the user U based on the captured image of the user U output from the imaging unit that captures the user U. Thereby, even when there is no reaction to the voice information due to the utterance of the user U, it is possible to determine whether or not the voice information is received by the user U.

  The detection unit 66 detects the user U's utterance as the behavior of the user U. The determination unit 67 determines whether or not the audio information has been received by the user U based on the state of the utterance of the user U. For example, the determination unit 67 determines that the audio information has been received by the user U when the user U's speech stops after the output of the audio information is started. Further, the determination unit 67 determines that the voice information has been received by the user U when the user U inputs a voice related to the voice information to the voice input device 12 after the output of the voice information is started. Thereby, it can be determined more appropriately whether or not the voice information is received by the user U.

  Further, the detection unit 66 detects a change in the device sound, which is a sound output from the device 4 existing around the user U, as the behavior of the user U. The determination unit 67 determines that the audio information is received by the user U when the device sound is stopped or changed after the output of the audio information is started. Thereby, even if there is no utterance or physical behavior from the user U, it is possible to appropriately determine whether or not the voice information is received by the user U.

  Further, the detection unit 66 detects the behavior of the user U based on the operation history of the device 4 by the user U after the start of outputting the audio information. Thereby, even if there is no utterance or physical behavior from the user U, it is possible to appropriately determine whether or not the voice information is received by the user U.

  In addition, the detection unit 66 detects at least one physical behavior among the eye movement of the user U, the head movement, the mouth movement, the hand movement, and the foot movement. Thereby, for example, when the user U behaves physically to receive voice information, it is possible to determine whether or not the voice information has been received by the user U.

  The detection unit 66 detects the behavior of the user U based on the sound input to the voice input device 12 existing around the user U. Thereby, for example, even when there is no user U or an imaging unit that captures the surroundings of the user U, the behavior of the user U can be detected.

  Further, the determination unit 67 determines that the audio information is received by the user U when a search related to the audio information is included in the operation history of the device by the user U when the audio information is output from the audio output device 11. Thereby, even if there is no utterance or physical behavior from the user U, it is possible to appropriately determine whether or not the voice information is received by the user U.

  The determination unit 67 determines that the audio information has been received by the user U when the behavior of the user U detected by the detection unit 66 is a specific behavior. The specific behavior is set according to the attribute of the user U. Thereby, it can be determined appropriately whether the audio | voice information was received by the user U. FIG.

  Further, the information providing device 2 updates the effect of the voice information based on the number of times that the voice information is output to the user U (an example of the number of times of provision) and the number of times the user U receives the voice information. An example of an effect update unit). Thereby, the effect of audio | voice information can be measured accurately.

[8. Others]
In addition, among the processes described in the above-described embodiments and modifications, all or a part of the processes described as being automatically performed can be manually performed, or are described as being performed manually. All or part of the processing can be automatically performed by a known method.

  In addition, the processing procedures, specific names, and information including various data and parameters shown in the document and drawings can be arbitrarily changed unless otherwise specified. For example, the various types of information illustrated in each drawing is not limited to the illustrated information.

  Further, each component of each illustrated apparatus is functionally conceptual, and does not necessarily need to be physically configured as illustrated. In other words, the specific form of distribution / integration of each device is not limited to that shown in the figure, and all or a part thereof may be functionally or physically distributed or arbitrarily distributed in arbitrary units according to various loads or usage conditions. Can be integrated and configured.

  For example, the information processing system 100 may have a configuration in which at least one of the input processing unit 31, the context acquisition unit 35, and the timing determination unit 36 is provided in the information providing device 2 different from the smart speaker 1 or in another device. The smart speaker 1 may be configured to be able to perform part or all of the processing of the information providing apparatus 2 described above. The information providing apparatus 2 may be realized by a plurality of server computers. Depending on the function, the information providing apparatus 2 may be realized by calling an external platform or the like with an API (Application Programming Interface) or network computing. Can be changed.

  In addition, the above-described embodiments and modifications can be combined as appropriate within a range that does not contradict processing contents. In addition, the “section (module, unit)” described above can be read as “means” or “circuit”. For example, the input processing unit 31 can be read as input processing means or an input processing circuit.

1 Smart Speaker 2 information providing apparatus 3 terminal device 4, 4 ₁ to 4 _n devices 5,5 ₁ to 5 _m sensor device 6 network 10,41 communication unit 11 audio output device 12 a voice input unit 13 imaging unit 14, 42 memory unit 15, 43 Control unit 20 Operation history 21 Utterance table 31 Input processing unit 32, 62 Information output unit 33, 61 Information acquisition unit 34 Output processing unit 35, 63 Context acquisition unit 36 Timing determination unit 51 Utterance table storage unit 52 Content storage unit 53 voice advertisement storage unit 54 user information storage unit 55 context storage unit 56 output mode determination information storage unit 64 learning unit 65 output mode determination unit 66 detection unit 67 determination unit 68 advertisement effect update unit 71 utterance table 72 content table 73 voice advertisement table 74 User information table 100 Information processing Stem

Claims (16)

A detection unit that detects a user's behavior when audio information is output from the audio output device;
An information processing apparatus comprising: a determination unit that determines whether or not the voice information has been received by the user based on behavior detected by the detection unit. The detector is
Detecting user behavior when a voice advertisement is output as the voice information from the voice output device,
The determination unit
The information processing apparatus according to claim 1, wherein it is determined whether or not the voice advertisement is received by the user based on a behavior detected by the detection unit. The detector is
The information processing apparatus according to claim 1, wherein the behavior is detected based on a sound input to an audio input device existing around the user. The detector is
Detecting the user's utterance as the behavior,
The determination unit
The information processing apparatus according to claim 3, wherein it is determined whether or not the voice information has been received by the user based on a state of the user's utterance. The determination unit
The information processing apparatus according to claim 4, wherein when the user stops speaking after the start of outputting the voice information, the voice information is determined to be accepted by the user. The determination unit
6. The determination according to claim 4, wherein, when a user inputs a voice related to the voice information to the voice input device after starting to output the voice information, the voice information is determined to be received by the user. Information processing device. The detector is
Detecting a change in device sound, which is a sound output from a device around the user, as the behavior,
The determination unit
The information according to any one of claims 3 to 6, wherein when the device sound is stopped or changed after the output of the sound information is started, it is determined that the sound information is received by the user. Processing equipment. The detector is
The information processing apparatus according to claim 1, wherein the behavior is detected based on a captured image of the user output from an imaging unit that captures the user. The detector is
The information processing apparatus according to claim 8, wherein at least one physical behavior is detected among the movement of the user's eyes, the movement of the head, the movement of the mouth, the movement of the hand, and the movement of the foot. . The detector is
The information processing apparatus according to claim 1, wherein the behavior of the user is detected based on an operation history of the device by the user after the output of the audio information is started. The determination unit
When the user operation history of the device when the audio information is output from the audio output device includes a search related to the audio information, it is determined that the audio information is received by the user. The information processing apparatus according to claim 10. The determination unit
When the behavior detected by the detection unit is a specific behavior, it is determined that the voice information is received by the user,
The information processing apparatus according to claim 1, wherein the specific behavior is set according to an attribute of the user. The effect update part which updates the effect of the said audio | voice information based on the frequency | count of provision to the said user of the said audio | voice information and the frequency | count of acceptance of the said audio | voice information by the said user is provided. The information processing apparatus according to claim 1. The information processing apparatus according to claim 1, wherein the audio output device is an audio output device included in a smart speaker. An information processing method executed by a computer,
A detection step for detecting a user's behavior when audio information is output from the audio output device;
A determination step of determining whether or not the voice information has been received by the user based on the behavior detected by the detection step. A detection procedure for detecting a user's behavior when audio information is output from the audio output device;
A determination procedure for determining whether or not the audio information has been received by the user based on the behavior detected by the detection procedure;
An information processing program for causing a computer to execute.

Images