JP7269812B2 - Home appliances - Google Patents

Description

Embodiments of the present invention relate to consumer electronic devices .

A detection unit that detects the occurrence of a predetermined event that triggers the home appliance to start a dialogue, and a determination unit that determines whether a predetermined start condition is satisfied when the detection unit detects the occurrence of the event. Household appliances have been proposed.

By the way, assuming a case where a home appliance is operated using a voice interaction device arranged independently of the home appliance, a trigger word for activating the voice interaction device and a command for specifying the home appliance to be operated are assumed. There is room for improvement in the user's convenience because it is necessary to utter words.

JP 2017-84177 A

A problem to be solved by the present invention is to provide a home electric appliance capable of improving user convenience.

A home appliance according to an embodiment has a control unit. The control unit transmits a first signal for activating the voice interaction function of the external device to the external device or the server device when a predetermined situation is detected. The external device activates the voice interaction function when the user utters a trigger word and when it receives a trigger signal that is the first signal or a signal generated based on the first signal. A waiting time is set for maintaining the activated state of the voice interaction function even when the user does not speak, and a first predetermined time is set as the waiting time when the user speaks the trigger word. Then, when the trigger signal is received, the waiting time is set to a second predetermined time longer than the first predetermined time, and the control unit, when the predetermined situation is detected, sets the waiting time , the first signal for setting the second predetermined time is transmitted to the external device or the server device.

1 is a diagram showing the overall configuration of an information processing system according to a first embodiment; FIG. FIG. 2 is a block diagram showing the functional configuration of the smart speaker according to the first embodiment; FIG. 4A and 4B are perspective views showing some examples of the home appliance of the first embodiment; FIG. FIG. 2 is a block diagram showing the functional configuration of the home appliance according to the first embodiment; FIG. FIG. 4 is a diagram showing an example of the contents of a display screen according to the first embodiment; FIG. 2 is a block diagram showing the functional configuration of the voice interaction server device of the first embodiment; FIG. The figure which shows an example of the content of voice operation management DB of 1st Embodiment. 2 is a block diagram showing the functional configuration of the device control server device according to the first embodiment; FIG. 4 is a sequence diagram showing an example of the overall flow of control according to the first embodiment; FIG. 4 is a flow chart showing the flow of processing relating to determination of an utterance target according to the first embodiment; The figure which shows an example of the determination rule of 1st Embodiment. The figure which shows the whole structure of the information processing system of 2nd Embodiment.

Home appliances and information processing systems according to embodiments will be described below with reference to the drawings. In the following description, the same reference numerals are given to components having the same or similar functions. Duplicate descriptions of these configurations may be omitted.

As used herein, "based on XX" means "based on at least XX", and includes the case of being based on another element in addition to XX. Also, "based on XX" is not limited to the case of using XX directly, but also includes the case of being based on XX that has undergone calculation or processing. "XX" is an arbitrary element (for example, arbitrary information). As used herein, "to cause YY" or "to cause YY" is not limited to cases where the primary purpose is to cause YY, and Also includes the case where YY is realized secondarily later. "YY" is, for example, an arbitrary action. In this specification, "database" is written as "DB".

(First embodiment)
First, the first embodiment will be described. The first embodiment is an example in which a smart speaker 200 arranged independently of a home appliance 100 has a voice interaction function. Also, the first embodiment is an example in which a trigger signal, which will be described later, is transmitted from the home appliance 100 to the smart speaker 200 .

[1. overall structure]
FIG. 1 is a diagram showing the overall configuration of an information processing system 1 according to the first embodiment. The information processing system 1 includes, for example, one or more (for example, multiple) home appliances 100, a smart speaker 200, a home router 300, and a voice interaction server device 400 (hereinafter referred to as "voice interaction server device 400"). , a server device 500 for home appliance control (hereinafter referred to as “device control server device 500”), and a terminal device 600 . In this embodiment, the information processing unit of the information processing system 1 is controlled by the control unit 250 of the smart speaker 200, the information processing unit 430 of the voice interaction server device 400, and the information processing unit 530 of the device control server device 500, which will be described later. has been realized. The network NW described later includes, for example, one or more of the Internet, a cellular network, a Wi-Fi network, a WAN (Wide Area Network), a LAN (Local Area Network), a public line, a telephone line, a wireless base station, and the like. .

Home appliance 100 is placed in a user's home. Home appliance 100 is connected to network NW via router 300 and can communicate with voice interaction server device 400 and device control server device 500 . Also, the home appliance 100 can communicate with the smart speaker 200 via the router 300 or directly. Furthermore, home appliance 100 can communicate with terminal device 600 via router 300 or via router 300 and network NW. In this embodiment, an example in which a refrigerator 100A, a microwave oven 100B, and a washing machine 100C are arranged as the home appliances 100 is taken up. However, the home appliance 100 is not limited to the above example, and may be other home appliances such as a vacuum cleaner, a rice cooker, or a television receiver. Hereinafter, when the type of the home appliance 100 does not matter, the home appliance 100 is simply referred to as "the home appliance 100".

Smart speaker 200 is an example of a "voice interaction device." Smart speaker 200 is also an example of an “external device” viewed from home appliance 100 . Smart speaker 200 is located in the user's home. Smart speaker 200 is connected to network NW via router 300 , for example, and can communicate with voice interaction server device 400 and device control server device 500 . One smart speaker 200 is arranged for a plurality of home appliances 100, for example, one smart speaker 200 is arranged in a kitchen. In this embodiment, the smart speaker 200 is a general-purpose smart speaker, and in addition to the operation of the home appliance 100, it can respond to questions such as the weather and time, and can also respond to music streaming. However, the “voice interaction device” and the “external device” referred to in this specification may be voice interaction devices dedicated to operating home appliance 100 .

Router 300 is located in the user's home. The router 300 is, for example, a wireless router, but is not limited to this. In other words, at least one of the home appliance 100 and the smart speaker 200 may be able to communicate with the router 300 by being connected to the router 300 via a wire (for example, a power line).

Voice interaction server device 400 is, for example, a cloud server connected to network NW. Voice interaction server device 400 analyzes the content of the user's speech to smart speaker 200, and notifies device control server device 500 of an answer by voice through smart speaker 200 and a control instruction for home appliance 100 according to the content of the user's speech. and so on. Voice interaction server device 400 is, for example, a server device provided by a manufacturer or distributor of smart speaker 200 . Note that some or all of the later-described functions of voice interaction server device 400 may be provided in smart speaker 200 .

The device control server device 500 is, for example, a cloud server connected to the network NW. When device control server device 500 is notified of a control instruction for home appliance 100 from voice interaction server device 400, device control server device 500 controls home appliance 100 based on the notified control instruction. The device control server device 500 is, for example, a server device provided with services by a manufacturer or a distributor of the electric home appliance 100 . A plurality of device control server devices 500 may exist for each manufacturer or distributor of the home appliance 100 . It should be noted that voice interaction server device 400 and device control server device 500 do not need to be separate devices, and may be realized by one device.

The terminal device 600 is, for example, a mobile terminal device owned by a user, such as a smart phone or a tablet terminal device. For example, the terminal device 600 is installed with an application program for setting the home appliance 100 and performing remote control. However, the terminal device 600 may be a personal computer, a television receiver having an input function, or the like.

Home appliance 100, smart speaker 200, voice interaction server device 400, and device control server device 500 will be described in detail below. For convenience of explanation, the smart speaker 200, the home appliance 100, the voice interaction server device 400, and the device control server device 500 will be described in this order.

[2. smart speaker]
First, the smart speaker 200 will be described.
FIG. 2 is a block diagram showing the functional configuration of the smart speaker 200. As shown in FIG. The smart speaker 200 has, for example, a microphone 210 , a speaker 220 , a wireless communication module 230 , a storage section 240 and a control section (information processing section) 250 .

The microphone 210 acquires the user's voice spoken to the smart speaker 200 . Speaker 220 outputs sound to the outside of smart speaker 200 . The wireless communication module 230 includes a high-frequency circuit, an antenna, etc., and can wirelessly communicate with the router 300, for example.

Storage unit 240 stores voice DB 241 . The voice DB 241 includes, for example, a simple voice recognition DB 241a. In the simple voice recognition DB 241a, for example, a trigger word (for example, a word uttered in advance by a user such as "Hello, OO") that is a voice command for activating the smart speaker 200 is registered.

The control unit 250 controls the smart speaker 200 as a whole. The control unit 250 has, for example, an audio reception unit 251, an activation determination unit 252, a standby time setting unit 253, a data transmission/reception unit 254, and an audio output unit 255. Note that some of these functional units (for example, the activation determination unit 252 and the standby time setting unit 253) need not be installed when the smart speaker 200 is shipped. It may be realized by

The voice receiving unit 251 receives the user's speech input to the microphone 210 as voice data.

The activation determination unit 252 determines whether or not to activate the voice interaction function of the smart speaker 200 . "Activation" means, for example, from a dormant state in which power consumption is suppressed to a predetermined level or less (a state in which it does not respond to user's utterances other than trigger words), it can also respond to user's utterances other than trigger words (interactive). It means changing to a standby state (a state where voice commands can be received). The "idle state" may also be referred to as the "wake state." For example, when the voice data received by the voice receiving unit 251 includes a trigger word registered in the simple voice recognition DB 241a, the activation determination unit 252 determines that activation is necessary, and activates the voice dialogue function. Further, in this embodiment, when the data transmission/reception unit 254 receives a trigger signal, which will be described later, the activation determination unit 252 determines that activation is necessary, and activates the voice interaction function.

When the activation determination unit 252 determines that activation of the voice interaction function is necessary and the activation of the voice interaction function is activated, the standby time setting unit 253 sets the standby time (time until timeout) for maintaining the standby state.

Here, the standby time setting unit 253 may vary the standby time between when the voice interactive function is activated based on the trigger word and when the voice interactive function is activated based on the trigger signal. For example, when activating the voice interaction function based on the trigger word, the standby time setting unit 253 maintains the standby time for a first predetermined time (for example, 10 seconds) even when there is no user's speech. . On the other hand, when the voice interaction function is activated based on the trigger signal, the standby time setting unit 253 sets a second predetermined time longer than the first predetermined time (for example, 10 seconds) even when the user does not speak. to about 30 seconds, or about 10 minutes to 30 minutes). The length of the second predetermined time is, for example, more than twice the length of the first predetermined time.

For example, when the second predetermined time is set to less than 5 minutes (more specifically, about 10 seconds to 30 seconds), the user performs work on the home appliance 100 (for example, the home appliance 100 Even if the user does not speak immediately, it is possible to reduce the possibility of failing to receive the user's instructions spoken during or after the work. .

For example, when the second predetermined time is set to 5 minutes or longer (more specifically, about 10 minutes to 30 minutes), the user is associated with the home appliance 100 or the installation location of the home appliance 100 at the timing when the trigger signal is transmitted. Even if the user is doing work (for example, cooking, cleaning up after cooking, preparing laundry, cleaning the room, etc.) and the user does not speak immediately, when using the smart speaker 200, each time There is no need to call out (for example, it is possible to hear the cooking procedure without calling out each time during cooking).

For example, the standby time setting unit 253 uses the latter long-time standby function described above (while the user is performing work related to the home appliance 100 or the location where the home appliance 100 is installed, the smart speaker 200 is kept activated). A setting function that turns on a function (for example, a function that keeps the smart speaker 200 active for a long period of time, such as 10 to 30 minutes) may be provided. This function is hereinafter referred to as a "long-time standby function".

For example, the standby time setting unit 253 may switch ON/OFF of the above-described long standby function based on a predetermined voice command uttered by the user. When the long-time standby function is OFF, the standby time setting unit 253 smartly detects that there is no user's speech for the first predetermined time (for example, 10 seconds) or the above-described short second predetermined time (for example, less than 5 minutes). Terminate the active state of the speaker 200 . On the other hand, if the standby time setting unit 253 receives a voice command to turn on the long standby function after entering the normal standby state based on the trigger word (or the trigger signal from the home appliance 100), It will be in a state of waiting for time. After that, if there is no user's speech, the smart speaker 200 ends the activation state after a predetermined long standby time (for example, about 10 to 30 minutes) has elapsed. Alternatively, the smart speaker 200 may terminate the active state of the smart speaker 200 when receiving a voice command to turn off the long standby function.

Data transmission/reception unit 254 controls wireless communication module 230 to transmit/receive signals and data to/from home appliance 100, voice interaction server device 400, and the like. For example, the data transmission/reception unit 254 transmits voice data other than the trigger word acquired by the voice reception unit 251 to the voice interaction server device 400 . Data transmission/reception unit 254 also receives, from voice interaction server device 400 , answer voice data prepared by voice interaction server device 400 . The data transmission/reception unit 254 outputs the received answer voice data to the voice output unit 255 .

Furthermore, in this embodiment, the data transmission/reception unit 254 receives a predetermined signal (a trigger signal, an auxiliary signal, etc., which will be described later) from the home appliance 100 . In addition, upon receiving the predetermined signal, the data transmission/reception unit 254 transmits information (identification information) that can be used to identify the home appliance 100 that is the transmission source of the predetermined signal from the home appliance 100. receive. The “identification information” referred to in this specification is not limited to information indicating the ID number (eg, serial number) of the home appliance 100, and may be information indicating the communication address (eg, IP address) of the home appliance 100. Any information that can distinguish the home appliance 100 from another home appliance 100 may be used. A case where the data transmission/reception unit 254 acquires the communication address of the home appliance 100 as the identification information will be described below as an example. Data transmission/reception unit 254 associates information indicating that the predetermined signal was received, information indicating the time at which the predetermined signal was received, and identification information of home appliance 100 that transmitted the predetermined signal. It transmits to voice interaction server device 400 .

Voice output unit 255 outputs voice data transmitted from voice interaction server device 400 and received by data transmission/reception unit 254 to the outside as voice through speaker 220 .

[3. home appliances]
Next, the home appliance 100 will be described.
FIG. 3 is a perspective view showing some examples of home appliance 100. FIG. Home appliance 100 includes, for example, housing 101 and door 102 . The housing 101 has an opening 101a for putting things in and out of the internal space of the housing 101 . The door 102 opens and closes the opening 101a of the housing 101 .

FIG. 4 is a block diagram showing the functional configuration of the home appliance 100. As shown in FIG. The home appliance 100 has, for example, a door open/close detection sensor 110, a human sensor 120, a trigger dedicated operation section 130, a power supply operation section 140, a display/operation section 150, a wireless communication module 160, an actuator 170, and a control section 180. Note that the home appliance 100 may not have some of these configurations.

The door open/close detection sensor 110 detects opening/closing of the door 102 with respect to the opening 101 a of the housing 101 . The door open/close detection sensor 110 is provided, for example, between the housing 101 and the door 102, and is a door open/close switch that switches ON/OFF states when the door 102 is opened or closed. However, the door opening/closing detection sensor 110 may be another sensor (for example, a camera) capable of detecting opening/closing of the door 102 . Human sensor 120 detects the presence of a person around home appliance 100 (for example, in front of home appliance 100).

The trigger-dedicated operation unit 130 is a dedicated operation unit operated by the user when a trigger signal to be described later is desired to be transmitted. That is, the trigger-dedicated operation unit 130 is an operation unit that receives an operation for activating the voice interaction function of the smart speaker 200 . The trigger dedicated operation section 130 includes, for example, a dedicated button or switch for transmitting a trigger signal. The trigger dedicated operation section 130 is an example of a "trigger operation section". However, the "trigger operating section" referred to in this specification is not limited to a dedicated operating section, and may be an operating section shared with other functions.

The power operation unit 140 is an operation unit that is operated by the user when turning the power of the home appliance 100 from an OFF state to an ON state or from an ON state to an OFF state. The power operation unit 140 includes a button or switch for switching the power ON state and OFF state.

The display/operation unit 150 includes, for example, a display device and a touch input type operation unit. The display device is, for example, a liquid crystal display or an organic EL (Organic Electro-Luminescence) display, and includes a display screen on which images and videos are displayed. The operation unit includes a touch sensor that overlaps the display screen of the display device. The display/operation unit 150 receives, for example, a user's operation for starting the operation of the home appliance 100 or changing the operation mode.

In this embodiment, the trigger dedicated operation section 130, the power supply operation section 140, and the display/operation section 150 constitute an example of an operation reception section R that receives a user's operation. However, the operation reception unit R can be used instead of or in addition to the trigger dedicated operation unit 130, the power supply operation unit 140, and the display/operation unit 150. It may have general buttons, switches, and the like that accept operations.

Wireless communication module 160 includes a high-frequency circuit, an antenna, and the like, and is capable of wireless communication with router 300, for example.

Actuator 170 is a drive unit for operating home appliance 100 . For example, if the home appliance 100 is a refrigerator, the actuator 170 is a compressor that compresses refrigerant, a blower that circulates cool air in the refrigerator, or the like. When home appliance 100 is a microwave oven, actuator 170 is a microwave generator or the like. If home appliance 100 is a washing machine, actuator 170 is a motor or the like that rotates a washing tub.

The control unit 180 controls the home appliance 100 as a whole. The control unit 180 has, for example, a trigger signal generation unit 181, a trigger mode setting unit 182, an auxiliary signal generation unit 183, a device control unit 184, and a data transmission/reception unit 185.

The trigger signal generation unit 181 generates a trigger signal for activating the voice interaction function of the smart speaker 200 when a predetermined situation is detected. A trigger signal is an example of a "first signal." The “predetermined situation” is a situation preset in the home appliance 100 as a situation for transmitting the trigger signal. The “predetermined situation” is, for example, when the trigger operation unit 130 is operated, when the power operation unit 140 is operated, when the door 102 is opened with respect to the opening 101 a of the housing 101 , Any one or more of the case where the door 102 is closed with respect to the opening 101a and the case where the presence of a person near the home appliance 100 is detected. “When a predetermined situation is detected” means, for example, when the trigger operation unit 130 detects that the trigger operation unit 130 has been operated, the power supply operation unit 140 operates (for example, turns on the power of the home appliance 100). When the power supply operation unit 140 detects that the switch is turned from the OFF state to the ON state), When the door open/close detection sensor 110 detects that the door 102 is opened with respect to the opening 101a of the housing 101 , when the door opening/closing detection sensor 110 detects that the door 102 is closed with respect to the opening 101a of the housing 101, and when the human sensor 120 detects the presence of a person. That's it.

For example, the trigger signal generation unit 181 generates a signal when the trigger operation unit 130 is operated, when the power supply operation unit 140 is operated, when the door 102 is opened with respect to the opening 101a of the housing 101, and when the housing 101 is opened. Different trigger signals may be generated when the door 102 is closed with respect to the opening 101a of the body 101 and when the human sensor 120 detects the presence of a person. According to such a configuration, in the later-described voice interaction server device 400, when it is necessary to determine the home appliance 100 that is the user's utterance target from among the plurality of home appliances 100 that have transmitted the trigger signal, the home appliance 100 contextual prioritization can be done. For example, voice interaction server device 400 detects that door 102 is opened to opening 101a of housing 101 by home appliance 100 to which a trigger signal has been transmitted due to detection of the presence of a person by human sensor 120. The priority of the home appliance 100 to which the trigger signal is transmitted can be increased by (or being closed), and the priority of the home appliance 100 to which the trigger signal is transmitted can be increased by operating the trigger operation unit 130. can be even higher. In addition, depending on the type of various operation units and sensors that also function as trigger operation units, such as the power supply operation unit 140 and the door open/close detection sensor 100, the smart speaker 200 is in the activated state (voice reception state). Different trigger signals may be assigned to different types of trigger operation units so that they are different. That is, when different types of trigger operation units are operated, different trigger signals may be transmitted.

By the way, depending on the smart speaker 200, the standby time for maintaining the standby state (activated state) may be fixed. That is, the case where the smart speaker 200 does not have the standby time setting unit 253 as described above and activates the voice interaction function based on the trigger word and the case where the voice interaction function is activated based on the trigger signal are the same. A waiting time of a predetermined time (first predetermined time, for example, 5 minutes) may be set.

In the present embodiment, the trigger signal generation unit 181 corresponding to the smart speaker 200 with a fixed standby time generates a voice interaction of the smart speaker 200 even when the user does not speak when the predetermined situation is detected. An additional signal may be generated to cause the function to remain idle for a second predetermined period of time longer than the first predetermined period of time. The additional signal may be the same signal as the trigger signal, or may be another signal. In this case, the data transmission/reception unit 185, which will be described later, transmits the additional signal to the smart speaker 200 before the first predetermined time elapses after transmitting the trigger signal to the smart speaker 200, or transmits such an additional signal. By repeatedly transmitting, the waiting time of smart speaker 200 may be extended to a second predetermined time. According to such a configuration, even if the smart speaker 200 does not have a special configuration, it is possible to activate the voice interaction function based on the trigger word and activate the voice interaction function based on the trigger signal. Wait times can be different.

For example, the trigger signal generation unit 181 generates a trigger signal when the door 102 of the home appliance 100 is opened or closed, the trigger dedicated operation unit 130 is operated, or the power supply operation unit 140 is operated. Then, the data transmitting/receiving unit 185 transmits the data to the smart speaker 200 . After that, the trigger signal generator 181 generates an additional signal at a predetermined cycle as long as the presence of a person continues to be detected by the human sensor 120 (for example, as long as the presence of a person in the kitchen continues to be detected). 185 to the smart speaker 200 . According to such a configuration, the standby state of the smart speaker 200 can be maintained while the user is near the home appliance 100 after operating the home appliance 100 , while the smart speaker 200 can be smartly Activation of the speaker 200 can be suppressed.

Trigger mode setting unit 182 determines whether or not to transmit a trigger signal regarding one or more situations based on at least one of the user's operation on operation reception unit R of home appliance 100 and the user's operation on terminal device 600. change the settings.

FIG. 5 is a diagram showing an example of the content of the display screen 151 displayed on the display/operation unit 150 or the terminal device 600. As shown in FIG. For example, the display screen 151 displays a screen for accepting a change as to whether or not to transmit the trigger signal regarding one or more situations that the home appliance 100 can detect. The trigger mode setting unit 182 changes the setting of whether or not to transmit the trigger signal for one or more situations based on the user's operation on the display screen 151 .

Returning to FIG. 4, the rest of the configuration of the home appliance 100 will be described. Auxiliary signal generation unit 183 generates a signal that assists association between "user's utterance" and "home appliance to be uttered" in voice interaction server device 400, which will be described later. For example, when the home electric device 100 is in the first specific situation, the auxiliary signal generation unit 183 may be used to associate the user's utterance to the smart speaker 200 with the home electric device 100 for processing, or 100 to generate a first auxiliary signal for increasing the priority of processing. The first specific situation is a situation in which the user's speech should be preferentially associated with the home appliance 100 . The first auxiliary signal is an example of a "second signal." Specific examples of these will be described later.

In addition, when the home electric device 100 is in the second specific situation, the auxiliary signal generation unit 183 prevents association between the user's utterance to the smart speaker 200 and the home electric device 100, or between the utterance and the home electric device 100. generates a second auxiliary signal for lowering the priority of processing associated with . The second specific situation is a situation in which the user's utterance and home appliance 100 should not be preferentially associated. The second auxiliary signal is an example of a "third signal". Specific examples of these will be described later. Hereinafter, when the first auxiliary signal and the second auxiliary signal are not distinguished from each other, they are simply referred to as "auxiliary signals".

The device control unit 184 controls the home appliance 100 based on the user's operation received by the operation receiving unit R and the control command received from the device control server device 500 . For example, the device control section 184 drives the actuator 170 based on the user's operation received by the operation receiving section R and the control command received from the device control server device 500 .

Data transmission/reception unit 185 controls wireless communication module 160 to transmit/receive signals and data to/from smart speaker 200, device control server device 500, and the like. For example, the data transmission/reception unit 185 receives a control command for controlling the home appliance 100 from the device control server device 500 . Further, when the trigger signal generation unit 181 generates the trigger signal and the additional signal, the data transmission/reception unit 185 transmits the generated trigger signal and the additional signal to the smart speaker 200 . When the auxiliary signal generator 183 generates the auxiliary signal, the data transmitter/receiver 185 transmits the generated auxiliary signal to the smart speaker 200 .

Data transmitter/receiver 185 transmits identification information that can be used to identify device control server device 500 that controls home appliance 100 in addition to the trigger signal when the above-described predetermined situation is detected. From another point of view, the data transmission/reception unit 185 transmits identification information that can be used to identify the type of the home appliance 100 in addition to the trigger signal when the predetermined situation is detected. Further, when transmitting an additional signal or an auxiliary signal, the data transmitter/receiver 185 transmits the identification information in addition to the additional signal or the auxiliary signal.

As described above, the identification information is not limited to information indicating the ID number of the electric home appliance 100, and may be information indicating the communication address of the electric home appliance 100 or the like. For example, when the identification information is the communication address of home appliance 100, voice interaction server device 400 refers to voice operation management DB 422, which will be described later. The type can be specified. In this specification, "in addition to the signal, the identification information is transmitted" is not limited to the case where the identification information is transmitted at the same time as the signal, and the case where the identification information is transmitted before the signal is transmitted or after the signal is transmitted. Also includes

Further, home appliance 100 may transmit the trigger signal and identification information to voice interaction server device 400 or device control server device 500 instead of transmitting the trigger signal and identification information to smart speaker 200 . In this case, voice interaction server device 400 or device control server device 500 may transmit (for example, transfer) the trigger signal received from home appliance 100 to smart speaker 200 .

In addition, instead of transmitting the trigger signal itself that activates the voice interaction function of the smart speaker 200, the home appliance 100, when the above-described predetermined situation is detected, sends a signal conforming to the trigger signal (hereinafter, for convenience of explanation, " sub-trigger signal”) may be transmitted to voice interaction server device 400 or device control server device 500. The sub-trigger signal is a signal indicating that the above-mentioned predetermined situation has occurred. In this case, voice interaction server device 400 or device control server device 500 generates a trigger signal that activates the voice interaction function of smart speaker 200 based on the sub-trigger signal received from home appliance 100, and transmits the generated trigger signal to smart speaker 200. You may send. A sub-trigger signal is another example of a "first signal." Also, "trigger signal" in the following description may be read as "sub-trigger signal".

These are the same for the additional signals and auxiliary signals described above. Note that in this specification, "the signal is transmitted from the home appliance 100 to the voice interaction server device 400" is not limited to the case where the signal is directly transmitted from the home appliance 100 to the voice interaction server device 400. It also includes the case where the device 100 transmits to the voice interaction server device 400 via the device control server device 500 .

[4. Voice Dialogue Server Device]
Next, voice interaction server device 400 will be described.
FIG. 6 is a block diagram showing the functional configuration of voice interaction server device 400. As shown in FIG. Voice interaction server device 400 has, for example, communication module 410 , storage unit 420 , and information processing unit 430 .

Communication module 410 includes a high frequency circuit and the like. Communication module 410 can communicate with smart speaker 200 via network NW and router 300 . Also, the communication module 410 can communicate with the device control server device 500 via the network NW.

Storage unit 420 stores voice DB 421 and voice operation management DB 422 . The voice DB 421 includes, for example, a voice recognition DB 421a and an utterance DB 421b. In the voice recognition DB 421a, dictionaries used for analyzing the user's speech to the smart speaker 200, information defining relationships between things, and the like are registered. Voice data (utterance data) for generating an answer output from the smart speaker 200 is registered in the utterance DB 421b.

FIG. 7 is a diagram showing an example of contents of the voice operation management DB 422. As shown in FIG. Information for identifying the home appliance 100 based on identification information (for example, a communication address) transmitted from the home appliance 100 is registered in the voice operation management DB 422 . For example, in the voice operation management DB 422, identification information (for example, a communication address) of the home appliance 100, a home appliance ID capable of identifying the home appliance 100, a user ID capable of identifying the user of the home appliance 100, and the home appliance The ID of the device control server device 500 corresponding to 100 and the type of the home appliance 100 are associated with each other.

Returning to FIG. 6, the rest of the configuration of voice interaction server device 400 will be described. The information processing unit 430 has, for example, a voice processing unit 431, an utterance target determination unit 432, a control content identification unit 433, a control instruction generation unit 434, a response generation unit 435, and a data transmission/reception unit 436.

The voice processing unit 431 analyzes the voice data of the user's utterance acquired by the smart speaker 200 . For example, the voice processing unit 431 converts voice data into text by performing voice recognition processing. In addition, the speech processing unit 431 performs semantic interpretation (primary semantic interpretation) by performing natural language processing on the speech data converted into text. The user's utterance data on which these processes have been performed is output to the utterance target determination unit 432 and the control content identification unit 433 .

When the trigger signal is transmitted from a plurality of home appliances 100, the utterance target determination unit 432 determines the home appliance 100 that is the user's utterance target from among the plurality of home appliances 100 that have transmitted the trigger signal. For example, the utterance target determination unit 432 determines the trigger signal and identification information transmitted from the home appliance 100, the user utterance content semantically interpreted by the voice processing unit 431, the content registered in the voice operation management DB 422 (each The home appliance 100 to be spoken by the user is determined based on the type of the home appliance 100 and the presence or absence of an auxiliary signal from the home appliance 100 . Note that the processing of the utterance target determination unit 432 will be described later in detail.

Based on the content of the user's utterance whose semantic interpretation is performed by the voice processing unit 431 and the type of the home appliance 100 that is the utterance target determined (specified) by the utterance target determination unit 432, the control content identification unit 433 A deeper semantic interpretation (secondary semantic interpretation) is performed on the content of the user's utterance, and the control content for the home appliance 100 is specified. In other words, the control content identification unit 433 analyzes the user's utterance content based on the type of the home appliance 100 . “Interpreting the content of the user's utterance based on the type of the home appliance 100” is an example of “predetermined processing performed in association with the user's utterance and the home appliance identified based on the identification information”.

For example, when the home appliance 100 is a refrigerator, the contents of control for the home appliance 100 are "rapid freezing", "rapid ice making", "thawing", and the like. If the home appliance 100 is a microwave oven, the control contents for the home appliance 100 are "heating for XX minutes", "thawing", and the like. When the home appliance 100 is a washing machine, the control contents for the home appliance 100 are "normal amount of water", "dry course", and the like. Control content identifying section 433 outputs the identified control content to response generating section 435 and control instruction generating section 434 .

The control instruction generation unit 434 generates a control instruction for the home appliance 100 according to the control content identified by the control content identification unit 433 .

The response generator 435 refers to the utterance DB 421b when there is content to be answered to the user's utterance content semantically interpreted by the control content identification unit 433, or when there is other content to be notified to the user. Audio data to be output from the speaker 200 is generated. The content to be answered in response to the content of the user's utterance is, for example, "I instructed the refrigerator to make ice quickly." The content to be notified is, for example, "Ice will be made in XX minutes".

In addition, when response generation unit 435 notifies the user of home appliance 100 that has transmitted the trigger signal (home appliance 100 specified based on the identification information), preferable information is sent from device control server device 500 to voice interaction server device 400 . In this case, based on the information, audio data to be output from the smart speaker 200 is generated. Information that is preferably notified to the user is information related to the home appliance 100 that has transmitted the trigger signal. For example, if the home appliance 100 is a refrigerator, it is inventory information and recipe information, and if the home appliance 100 is a washing machine. is pollen information for the most recent week, and the like.

Data transmission/reception unit 436 controls communication module 410 to transmit/receive signals and data to/from smart speaker 200, device control server device 500, and the like. The data transmitter/receiver 436 receives voice data of user's utterance from the smart speaker 200 . The data transmission/reception unit 436 outputs the received audio data to the audio processing unit 431 . The data transmission/reception unit 436 also receives information indicating that a predetermined signal (trigger signal, auxiliary signal, etc.) has been received, information indicating the time at which the predetermined signal was received, and the home appliance 100 that transmitted the predetermined signal. identification information from the smart speaker 200 .

The data transmission/reception unit 436 also receives a control instruction for the home appliance 100 from the control instruction generation unit 434 . The data transmitting/receiving unit 436 refers to the voice operation management DB 422 and specifies the device control server device 500 corresponding to the home appliance 100 to be spoken by the user. The data transmission/reception unit 436 then transmits the control command received from the control command generation unit 434 to the specified device control server device 500 . "Specifying the device control server device 500 corresponding to the home appliance 100 targeted by the user's utterance and transmitting a control command to the specified device control server device 500 is a process of ``specifying based on the user's utterance and identification information. This is another example of "predetermined processing performed in association with the home appliance that has been received."

The data transmitter/receiver 436 also receives voice data to be output by the smart speaker 200 from the response generator 435 . The data transmitter/receiver 436 transmits the audio data received from the response generator 435 to the smart speaker 200 .

[5. Device control server device]
Next, the device control server device 500 will be described.
FIG. 8 is a block diagram showing the functional configuration of the device control server device 500. As shown in FIG. The device control server device 500 has, for example, a communication module 510, a storage section 520, and an information processing section 530.

Communication module 510 includes a high frequency circuit and the like. Communication module 510 can communicate with home appliance 100 via network NW and router 300 . Communication module 510 can communicate with voice interaction server device 400 via network NW.

Storage unit 520 stores control command DB 521 and information management DB 522 . Various control commands for operating the home appliance 100 are stored in the control command DB 521 . Information related to the home appliance 100 is stored in the information management DB 522 . For example, when the home appliance 100 is a refrigerator, the information management DB 522 stores inventory information (presence or absence of inventory, date of receipt, expiry date, etc.) of foodstuffs in the refrigerator. For example, if the home appliance 100 is a washing machine, the amount of pollen for the most recent week or so is stored.

The information processing section 530 has a control command generation section 531, an information collection section 532, an information selection section 533, and a data transmission/reception section 534, for example.

Control command generation unit 531 generates a control command corresponding to home appliance 100 based on the control instruction received from voice interaction server device 400 .

The information collection unit 532 collects information related to the home appliance 100 to be stored in the information management DB 522 . For example, the information collecting unit 532 collects information input to the user's terminal device 600 (for example, refrigerator inventory information) and information provided by various websites connected to the network NW. Collected information is stored in the information management DB 522 .

The information selection unit 533 selects information that is preferable when notified to the user from the smart speaker 200 from information registered in the information management DB 522 .

Data transmitting/receiving unit 534 transmits/receives signals and data to/from home appliance 100 and voice interaction server device 400 by controlling communication module 510 . When data transmission/reception unit 534 receives a control instruction for home appliance 100 from voice interaction server device 400 , data transmission/reception unit 534 outputs the received control instruction to control command generation unit 531 . The data transmitter/receiver 534 transmits the control command generated by the control command generator 531 to the home appliance 100 . Data transmission/reception unit 534 also transmits information selected by information selection unit 533 to voice interaction server device 400 .

[6. Overall flow of control]
Next, the overall control flow will be described.
FIG. 9 is a sequence diagram showing an example of the overall control flow of the information processing system 1. As shown in FIG. First, the trigger signal generator 181 of the home appliance 100 monitors whether or not a predetermined situation occurs. Then, when a predetermined situation occurs, the trigger signal generation unit 181 detects that the predetermined situation has occurred based on the detection result of the door opening/closing detection sensor 110, the operation of the operation reception unit R, and the like (S101).

Next, the trigger signal generator 181 of the home appliance 100 generates a trigger signal when it is detected that a predetermined situation has occurred. The data transmission/reception unit 185 of the home appliance 100 transmits the generated trigger signal and the identification information of the home appliance 100 to the smart speaker 200 (S102). When the trigger signal is received from the home appliance 100, the activation determination unit 252 of the smart speaker 200 activates the voice interaction function of the smart speaker 200 and places the smart speaker 200 in a standby state (S103). Data transmission/reception unit 254 of smart speaker 200 also transmits information indicating that the trigger signal has been received from home appliance 100 and identification information of home appliance 100 that has sent the trigger signal to voice interaction server device 400 .

Next, the voice receiver 251 of the smart speaker 200 determines whether or not the user speaks to the microphone 210 (S104). When the user does not speak into the microphone 210 (S104: NO), the voice receiving unit 251 determines whether or not a predetermined time (waiting time) for keeping the smart speaker 200 in the waiting state has elapsed (S105).

When the predetermined time has passed (S105: YES), the voice receiving unit 251 terminates the voice interaction function by timing out (S106). On the other hand, when the predetermined time has not elapsed (S105: NO), the voice receiving unit 251 returns to the process of step S103.

If there is a user's speech into the microphone 210 (S104: YES), the voice receiving unit 251 receives the user's speech input to the microphone 210 as voice data. The data transmission/reception unit 254 transmits the voice data received by the voice reception unit 251 to the voice interaction server device 400 (S107).

The voice processing unit 431 of the voice interaction server device 400 refers to the voice recognition DB 421, etc., and analyzes the voice data of the user's utterance (S108). The control content identification unit 433 of the voice interaction server device 400 identifies the control content intended by the user for the home appliance 100 based on the voice data analyzed by the voice processing unit 431 or the like (S109).

Control instruction generating section 434 of voice interaction server device 400 generates a control instruction for home appliance 100 based on the control content specified by control content specifying section 433 . The data transmission/reception unit 436 of the voice interaction server device 400 transmits the control instruction generated by the control instruction generation unit 434 to the device control server device 500 (S110).

The control command generator 531 of the device control server device 500 generates a control command for the home appliance 100 based on the control instruction received from the voice interaction server device 400 (S111). The data transmission/reception unit 534 of the device control server device 500 transmits the generated control command to the home appliance 100 (S112).

The data transmission/reception unit 185 of the home appliance 100 transmits a control response to the received control command to the device control server device 500 (S113). The control response indicates, for example, whether or not the control command has been normally notified to the home appliance 100 . The device control unit 184 of the home appliance 100 controls the home appliance 100 based on the received control command (S114).

Data transmission/reception unit 534 of device control server device 500 transmits the control response received from home appliance 100 to voice interaction server device 400 (S115). The response generation unit 435 of the voice interaction server device 400 generates voice data to be output by the smart speaker 200 based on the control response received from the device control server device 500 (S116). The data transmission/reception unit 436 of the voice interaction server device 400 transmits the generated voice data to the smart speaker 200 (S117). The voice output unit 255 of the smart speaker 200 outputs voice corresponding to the control response from the speaker 220 based on the voice data received from the voice interaction server device 400 (S118). Note that the processes of S113 and S115 to S118 may be performed at the timing when the control of the home appliance 100 is completed.

[7. Determination processing of utterance target]
[7.1 Basic processing]
Next, processing related to determination of an utterance target of voice interaction server device 400 will be described.
FIG. 10 is a flow chart showing the flow of processing relating to determination of a speech target. First, the voice receiving unit 251 of the voice interaction server device 400 acquires the voice data of the user speaking to the smart speaker 200 (S201).

When voice data is acquired by voice reception unit 251, utterance target determination unit 432 of voice interaction server device 400 determines whether voice data is acquired from a plurality of home appliances 100 during the same standby time before voice data is acquired by voice reception unit 251. It is determined whether or not a trigger signal has been received (S202).

When the trigger signal is received from only one home appliance 100 (S202: NO), the utterance target determination unit 432 determines that the home appliance 100 is the user's utterance target. In this case, the information processing unit 430 of the voice interaction server device 400 associates the single home appliance 100 with the user's utterance and performs predetermined processing. The “predetermined process” is, for example, specifying the type of the home appliance 100 based on the identification information of the home appliance 100 and the voice operation management DB 422, and interpreting the semantic content of the user's utterance based on the specified type of the home appliance 100. The device control server device 500 corresponding to the home appliance 100 is specified based on the first processing to be performed, the identification information of the home appliance 100, and the voice operation management DB 422, and the specified device control server device 500 performs processing based on the content of the user's utterance. and a second process of transmitting a control instruction. However, the "predetermined process" referred to in this specification may be at least one of the first process and the second process, or may be another process different from the first process and the second process. Note that even when the trigger signal is received from only one home appliance 100, the utterance target determination unit 432 performs affinity determination, which will be described later. It may be determined that the speech is unrelated to the control of the home appliance 100, and no special processing may be performed.

On the other hand, when the trigger signal is received from a plurality of home appliances 100 (S202: YES), the control content identification unit 433 determines which home appliance 100 among the plurality of home appliances 100 is the speech target. do. In the present embodiment, the control content specifying unit 433 gives higher priority to association with speech to the home appliance 100 that received the trigger signal later. For this reason, the control content specifying unit 433 first determines whether the affinity between the home appliance 100 that last transmitted the trigger signal among the plurality of home appliances 100 and the content of the user's utterance is equal to or higher than a reference (threshold or higher). (S204). Note that "affinity is equal to or higher than the reference (or affinity is lower than the reference)" means that the value indicating affinity is equal to or higher than the reference (or the value indicating affinity is lower than the reference). The ``value indicating affinity'' is, for example, ``a value preset according to the affinity between the type of home appliance and the content of the speech'' A value set in advance according to the affinity".

Here, for example, the contents of the utterance such as "cooling" and "cooling" match the basic operation of the refrigerator 100A, so the affinity with the refrigerator 100A is determined to be equal to or higher than the standard. On the other hand, since contents such as "cooling" and "cooling" do not match the basic operations of the microwave oven 100B and the washing machine 100C, the compatibility with the microwave oven 100B and the washing machine 100C is determined to be below the standard. Conversely, utterances such as "warm", "toast", and "500W" match the basic operation of the microwave oven 100B, and thus the compatibility with the microwave oven 100B is determined to be above the standard. On the other hand, utterances such as "warm", "toast", and "500W" do not match the basic operations of the refrigerator 100A and the washing machine 100C. be done.

If the affinity between the home appliance 100 and the content of the user's utterance is equal to or higher than the reference (S204: YES), the utterance target determination unit 432 determines that the home appliance 100 is the user's utterance target. In this case, the information processing unit 430 of the voice interaction server device 400 performs the predetermined process by associating the user's utterance with the home appliance 100 determined to be the user's utterance target (S205).

On the other hand, when the utterance target determination unit 432 determines that the affinity is less than the reference (S204: NO), next, among the plurality of home appliances 100, the home appliance that transmitted the trigger signal one before. It is determined whether or not the affinity between 100 and the content of the user's utterance is equal to or higher than a standard (S206). Then, when the affinity between the home appliance 100 and the content of the user's utterance is equal to or higher than the reference (S206: YES), the utterance target determination unit 432 determines that the home appliance 100 is the user's utterance target. In this case, the information processing unit 430 of the voice interaction server device 400 associates the home appliance 100 determined as the user's speech target with the user's speech, and performs the above-described predetermined process (S207).

When the trigger signal is received from N home appliances 100 (N is an integer equal to or greater than 1), the utterance target determination unit 432 repeats the above determination process for the N home appliances 100 (S208). Then, when the affinity between the home appliance 100 and the content of the user's utterance is equal to or higher than the reference (S208: YES), the utterance target determination unit 432 determines that the home appliance 100 is the user's utterance target. In this case, the information processing unit 430 of the voice interaction server device 400 performs the predetermined process by associating the home appliance 100 determined as the user's utterance target with the user's utterance (S209).

After that, the response generation unit 435 informs the user of the specified control content (the control content to be executed) based on the control content for the home appliance 100 specified by performing semantic interpretation based on the type of the home appliance 100. The audio data to be played is generated (S210). Then, the data transmission/reception unit 436 transmits the audio data generated by the response generation unit 435 to the smart speaker 200, and causes the smart speaker 200 to announce the specified control content. As a result, for example, the smart speaker 200 outputs a voice saying, "Start heating in the microwave oven." Such notification makes it easier to avoid control contrary to the user's intention even if the estimation in voice interaction server device 400 is erroneous.

Instead of the above, the response generation unit 435 may generate voice data for confirming with the user whether there is any problem with the specified control content. The data transmission/reception unit 436 transmits the audio data generated by the response generation unit 435 to the smart speaker 200, and causes the smart speaker 200 to output audio for confirming with the user whether there is a problem with the specified control content. As a result, the smart speaker 200 outputs, for example, a voice saying "Would you like to start heating in the microwave?" Such notification makes it easier to avoid control contrary to the user's intention even if the estimation in voice interaction server device 400 is erroneous.

On the other hand, if the home appliance 100 having affinity with the content of the user's utterance equal to or higher than the standard is not found even after repeating the determination process for N home appliances 100 (S208: NO), the utterance target determination unit 432 , it is determined that the user's utterance is irrelevant to the control of the home appliance 100, and no special processing is performed. As a result, when the user utters a content unrelated to the home appliance 100, it is possible to avoid performing unintended processing by the user.

[7.2 Another process]
Next, some other processing will be described. These processes may be executed in combination with the above-described basic process, or may be executed instead of the above-described basic process.

[7.2.1 Processing 1]
For example, in principle, the utterance target determination unit 432 assigns a higher priority to the home appliance 100 that transmitted the trigger signal later, in association with the user's utterance. Here, when the smart speaker 200 receives the first auxiliary signal (signal for raising the priority) from the home appliance 100, it performs processing to raise the priority of the home appliance 100 that has transmitted the first auxiliary signal. For example, in a state in which trigger signals are received from a plurality of home appliances 100, the utterance target determination unit 432 determines the affinity with priority given to the home appliance 100 that has transmitted the first auxiliary signal. Alternatively, the utterance target determination unit 432 may perform the predetermined processing by associating the user's utterance with the home appliance 100 that most recently transmitted the first auxiliary signal without determining affinity.

A specific example will be described. For example, the microwave oven 100B transmits a trigger signal when the door 102 is opened and closed, and the mode selection button (button for selecting the oven mode, the range mode, etc.) of the operation reception unit R is pressed. If the heating start button of the operation reception unit R is not pressed within the predetermined time (an example of the first specific situation), the first auxiliary signal is transmitted. In a certain scene, the door 102 of the microwave oven 100B is opened and closed, and the mode selection button of the microwave oven 100B is pressed. After that, the door 102 of the refrigerator 100A is opened and closed without pressing the heating start button, and the trigger is triggered from the refrigerator 100A. Suppose a signal is sent and then a user speaks. In this case, although the home appliance 100 that last transmitted the trigger signal is the refrigerator 100A, the utterance target determination unit 432 associates the microwave oven 100B that transmitted the first auxiliary signal with the user's utterance. takes precedence over association with utterances of

According to such a configuration, for example, although the operation mode of the microwave oven 100B is selected, the heating time is not known, and the food stored in the refrigerator 100A is checked on the package notation, and the heating time is specified by speaking. Even in such a case, the home appliance 100 and the user's speech can be appropriately associated. The first specific state is, for example, a state in which the user's intention to operate the home appliance 100 is presumed, but the home appliance 100 has not yet started to operate. “Pressing the mode selection button” is an example of an event that is presumed by the user's intention to operate the home appliance 100 .

[7.2.2 Processing 2]
For example, in principle, the utterance target determination unit 432 assigns a higher priority to the home appliance 100 that transmitted the trigger signal later, in association with the utterance content of the user. Here, when the smart speaker 200 receives the second auxiliary signal (signal for lowering the priority) from the home appliance 100, it performs processing to lower the priority of the home appliance 100 that has transmitted the second auxiliary signal. . For example, in a state in which trigger signals are being received from a plurality of home appliances 100, the utterance target determination unit 432 determines the above-described affinity by putting the home appliance 100 that has transmitted the second auxiliary signal later. In addition, the utterance target determination unit 432 may exclude the home appliance 100 that has transmitted the second auxiliary signal from the affinity determination target.

A specific example is introduced. For example, the microwave oven 100B transmits a trigger signal when the door 102 is opened and closed, and transmits a second auxiliary signal when the heating start button of the operation reception unit R is pressed (an example of the second specific situation). shall be In a certain scene, the door 102 of the refrigerator 100A is opened and closed, and a trigger signal is transmitted from the refrigerator 100A. After that, the door 102 of the microwave oven 100B is opened and closed, and the trigger signal is transmitted from the microwave oven 100B. Assume that the heating start button is pushed and the second auxiliary signal is transmitted. In this case, although the home appliance 100 that has transmitted the trigger signal last is the microwave oven 100B, the utterance target determination unit 432 lowers the priority of the microwave oven 100B that has transmitted the second auxiliary signal and prioritizes the refrigerator 100A first. Affinity of user's utterance content is determined.

According to such a configuration, for example, even when the operation by speaking to refrigerator 100A is postponed, home appliance 100 can be appropriately associated with the user's speech. The second specific state is, for example, a state in which the home appliance 100 has already started operating according to the user's intention. A “heating start button” is an example of an operation unit that starts the operation of home appliance 100 according to the user's intention.

[7.2.3 Processing 3]
FIG. 11 is a diagram showing an example of the determination rule of the utterance target determination unit 432. As shown in FIG. FIG. 11 shows a case where an utterance includes a word whose affinity is equal to or higher than the standard for any of the plurality of home electric appliances 100 . In such a case, the utterance target determination unit determines the utterance target according to one or more of the following determination rules (A) to (E).

(A) In principle, the utterance target determination unit 432 transmits a trigger signal immediately before (at the end) when the utterance is an utterance that instructs only the start without specific content, such as "Please do something later" or "Please take care of me later." Giving priority to the home appliance 100 that has been selected, association between the home appliance 100 and the user's utterance is performed.

(B) The utterance target determination unit 432 determines whether the home appliance 100 ( For example, the home appliance 100 (for example, the microwave oven 100B) whose basic function works only when operated is given priority over the refrigerator 100A), and the home appliance 100 and the user's utterance are associated. For example, even when the refrigerator 100A is opened and closed after the oven range 100B is opened and closed, the oven range 100B is prioritized to associate the home appliance 100 with the speech.

(C) If the utterance is an utterance that instructs only power saving or energy saving, such as "Setsumu mode", the utterance target determination unit 432 gives priority to the home appliance 100 that transmitted the trigger signal immediately before (latest). Then, the home appliance 100 is associated with the user's utterance.

(D) The utterance target determination unit 432 gives an instruction to which space (eg, refrigerator compartment, freezer compartment, ice making compartment) in home appliance 100 (eg, refrigerator 100A), such as "Keep it cool" and "Kyuurei." If it is unclear whether the space is open or closed, the user's utterance is associated with the previously operated (opened/closed) space.

(E) If the utterance is an utterance that instructs only to hurry up more than usual, such as "hurry up", the utterance target determination unit 432 gives priority to the home appliance 100 that transmitted the trigger signal immediately before (latest). , to associate the home appliance 100 with the user's speech.

[7.2.4 Processing 4]
For example, the home appliance 100 includes a sensor capable of detecting the presence or absence of an object in the space of the housing 101 (for example, the storage chamber of the refrigerator 100A, the heating chamber of the microwave oven 100B, the inside of the washing tub of the washing machine 100C). Then, when the sensor detects that there is no object in the space of the housing 101 (the space is empty), the trigger signal generation unit 181 of the home appliance 100 detects that the space in the housing 101 is empty. may be output to the voice interaction server device 400 via the smart speaker 200 . When information indicating that there is no content in the space inside housing 101 is received, utterance target determination unit 432 of voice interaction server device 400 determines the association between home appliance 100 that transmitted the information and the user's utterance. Decrease priority.

Further, the trigger signal generation unit 181 of the home electric appliance 100 detects that the space of the housing 101 is empty when the sensor detects that an object has been thrown into the space of the housing 101 . information indicating that an object has been newly introduced into the space may be output to the voice interaction server device 400 via the smart speaker 200 . When information indicating that a new object has been thrown into the space of housing 101 is received, utterance target determination unit 432 of voice interaction server device 400 determines the relationship between home appliance 100 that transmitted the information and the user's utterance. The priority of association may be increased.

[7.2.5 Processing 5]
If the timing at which the trigger signal is transmitted and the timing at which the user speaks are within a predetermined period of time (for example, the trigger signal is transmitted just before the case), the home appliance 100 that transmitted the trigger signal may be associated with the user's utterance.

[8. advantage]
As a comparative example, consider the case where utterance of a trigger word is required to activate the voice interaction function of smart speaker 200 . In this case, the user needs to remember the trigger word and say the trigger word each time to activate the voice interaction function of the smart speaker 200 .

On the other hand, in the present embodiment, the control unit 180 of the home appliance 100 transmits a trigger signal for activating the voice interaction function of the smart speaker 200 to the smart speaker 200 when a predetermined situation is detected. Such an arrangement eliminates the need to remember the trigger word and to issue the trigger word. Thereby, user convenience can be improved.

In this embodiment, the control unit of the home appliance 100 transmits a trigger signal when the trigger dedicated operation unit 130 is operated. According to such a configuration, the user can transmit the trigger signal at any timing. As a result, user convenience can be further enhanced. For example, assume that smart speaker 200 is placed in the kitchen. The user is doing household chores in order to prepare meals. When the user checks the inside of the refrigerator 100A and thinks about what to make, the user operates the trigger-dedicated operation unit 130 in the refrigerator 100A to open the door 102 of the refrigerator 100A. Therefore, when the user looks at the beef in the refrigerator 100A and says, "Tell me a recipe using beef," the smart speaker 200 inquires of the voice interaction server device 400 and suggests a recipe using beef. By using it in this way, voice operation can be performed without trouble while doing household chores.

In this embodiment, the control unit 180 of the home appliance 100 transmits a trigger signal when the door 102 is opened with respect to the opening 101a of the housing 101 . According to such a configuration, it is possible to activate the voice interaction function of the smart speaker 200 without issuing a trigger word when an object is put inside the home appliance 100 . As an example, when the voice interaction function is activated at the timing when the door 102 of the refrigerator 100A is opened, the user, while confirming the inside of the refrigerator 100A, registers the inventory status of the refrigerator 100A through voice interaction with the smart speaker 200. It can be carried out.

In this embodiment, the control unit 180 of the home appliance 100 transmits a trigger signal when the door 102 is closed with respect to the opening 101a of the housing 101 . According to such a configuration, it is possible to activate the voice interaction function of smart speaker 200 without issuing a trigger word after completing work on the inside of home appliance 100 . As an example, when the voice interaction function is activated at the timing when the door 102 of the refrigerator 100A is closed, the user can instruct quick freezing of the refrigerator 100A through voice interaction with the smart speaker 200. For example, assume that the smart speaker 200 is installed in a washroom. The power of the washing machine 100C is turned on, the laundry is put into the washing tub, and the door 102 is closed. At this time, a trigger signal is transmitted from washing machine 100C to smart speaker 200 . Here, when "wash the blanket" is uttered, smart speaker 200 inquires of voice interaction server device 400 (or device control server device 500) and transmits a blanket washing mode control command to washing machine 100C.

In this embodiment, the control unit 180 of the home appliance 100 transmits a trigger signal when the presence of a person is detected by the human sensor 120 . According to such a configuration, the user can control the home appliance 100 by voice interaction with the smart speaker 200 without trouble, just by approaching the home appliance 100 . For example, assume that smart speaker 200 is placed in the kitchen. A human sensor 120 is mounted on the refrigerator 100A. When it is detected that the user has come in front of refrigerator 100A in the kitchen, refrigerator 100A transmits a trigger signal to smart speaker 200 . When the user utters "tell me a recommended recipe", the smart speaker 200 inquires of the voice interaction server device 400 and proposes a recommended recipe.

In this embodiment, the control unit 180 of the home appliance 100 transmits a trigger signal when the power operation unit 140 is operated (for example, when the power is turned on). According to such a configuration, mode setting and the like after power-on can be easily performed by voice interaction with the smart speaker 200 . For example, there are home electric appliances 100 such as a washing machine 100C and a microwave oven 100B that are normally powered off. In such a home appliance 100, when the power operation unit 140 of the washing machine 100C is operated, or when the door 102 of the microwave oven 100B is opened and the power is turned on, the home appliance 100 turns on the smart speaker 200. Send a trigger signal. Here, the user can easily perform a voice operation by uttering a washing mode, a recipe, or the like. Also, the control unit 180 of the home appliance 100 may transmit a trigger signal when the power is turned off from the on state. According to such a configuration, even if a user forgets to set a reservation for an operation of the electric home appliance 100, it can be performed through voice interaction while the power of the electric home appliance 100 is turned off.

Here, it is assumed that some users may want to activate the voice dialogue function and others may not want to activate the voice dialogue function depending on how they operate or their habits. Therefore, in the present embodiment, the control unit 180 of the home appliance 100 performs one operation based on at least one of the user's operation on the operation receiving unit R of the home appliance 100 and the user's operation on the terminal device 600 . Change the setting of whether to send a trigger signal for one or more situations. According to such a configuration, it is possible to arbitrarily set the operation and the situation for transmitting the trigger signal according to the usability of the user. Furthermore, if the setting can be made by the user's operation on the terminal device 600 different from the home appliance 100, the setting can be made even if the home appliance 100 does not have a user interface including a display screen.

In this embodiment, when a predetermined situation is detected, the control unit 180 of the home appliance 100 transmits information used for specifying the device control server device 500 that controls the home appliance 100 in addition to the trigger signal. . According to such a configuration, the user can operate the home appliance 100 by voice without uttering a word for specifying the home appliance 100 . As a result, user convenience can be further enhanced.

In this embodiment, the information processing unit 250 of the smart speaker 200 activates the voice interaction function of the smart speaker 200 arranged independently of the home appliance 100 when the trigger signal is transmitted from the home appliance 100 . According to such a configuration, the voice interaction function of smart speaker 200 can be activated without uttering a predetermined trigger word. Thereby, user convenience can be improved.

(Second embodiment)
Next, a second embodiment will be described. In this embodiment, one home appliance 100 (hereinafter referred to as voice interaction home appliance 100V) installed in the user's house is an example having a voice interaction function. The second embodiment is an example in which a trigger signal, which will be described later, is transmitted from the home appliance 100 to the voice interaction home appliance 100V. Configurations other than those described below are the same as those of the first embodiment.

FIG. 12 is a diagram showing the overall configuration of the information processing system 1 of the second embodiment. The information processing system 1 includes, for example, one or more (e.g., multiple) home appliances 100, voice interaction home appliances 100V, home routers 300, voice interaction server devices 400, device control server devices 500, and terminal devices 600. . Voice interaction server device 400 and device control server device 500 may be configured by one server device.

The voice interaction home appliance 100V is an example of an “external device” viewed from the home appliance 100. FIG. The voice interaction home appliance 100V has the same functions as the normal home appliance 100 (for example, a refrigerator function, a microwave oven function, or a washing machine function), and also has a voice interaction device 200 . The voice interaction device 200 may have, for example, the same functions as the smart speaker 200 of the first embodiment, or may be a voice interaction device dedicated to operating the home appliance 100 .

Although the first and second embodiments have been described above, the embodiments are not limited to the above examples. For example, in the first and second embodiments, the control command for the home appliance 100 is transmitted from the device control server device 500 to the home appliance 100, but the invention is not limited to this. A control command for home appliance 100 is sent from device control server device 500 (or voice interaction server device 400) to smart speaker 200 or voice interaction home appliance 200, and sent from smart speaker 200 or voice interaction home appliance 200 to home appliance 100. may be Ordinal numbers such as “first” and “second” in this specification are for convenience of description and may be renumbered as appropriate.

The storage unit 240 of the smart speaker 200, the storage unit 420 of the voice interaction server device 400, and the storage unit 520 of the device control server device 500 are, for example, semiconductor memory chips, SSDs (Solid State Drives), or HDDs (Hard Disk Drives). ) and the like.

Control unit 180 of home appliance 100, control unit 250 of smart speaker 200, information processing unit 430 of voice interaction server device 400, and information processing unit 530 of device control server device 500 are, for example, CPUs (Central Processing Units). hardware processor executes a program (software). However, all or part of each of the control unit 180, the control unit 250, the information processing unit 430, and the information processing unit 530 is an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). ) or other hardware (including circuitry), or by cooperation between software and hardware.

According to at least one embodiment described above, the home appliance has a control unit that transmits the first signal for activating the voice interaction function of the external device to the external device or the server device, thereby improving user convenience. It is possible to improve the quality.
<Appendix>
Some home appliances and information processing systems are listed below.
(1) a control unit that, when a predetermined situation is detected, transmits a first signal for activating the voice interaction function of the external device to the external device or the server device;
home appliances with
(2) further comprising a trigger operation unit that receives an operation for activating the voice interaction function, wherein the control unit transmits the first signal when the trigger operation unit is operated;
The home appliance according to (1).
(3) a housing having an opening;
a door that closes the opening,
The control unit transmits the first signal when the door is opened with respect to the opening.
The home appliance according to (1) or (2).
(4) a housing having an opening;
a door that closes the opening,
The control unit transmits the first signal when the door is closed with respect to the opening.
The home appliance according to any one of (1) to (3).
(5) further comprising a human sensor,
The control unit transmits the first signal when the presence of a person is detected by the human sensor.
The home appliance according to any one of (1) to (4)
(6) further comprising a power operation unit;
The control unit transmits the first signal when the power operation unit is operated.
The home appliance according to any one of (1) to (5).
(7) Based on at least one of a user's operation on an operation reception unit of the home appliance and a user's operation on a terminal device other than the home appliance, the control unit performs one or more of the first conditions. 1 change the setting of whether to send a signal,
The home appliance according to any one of (1) to (6).
(8) When the predetermined situation is detected, the control unit transmits, in addition to the first signal, information that can be used to identify a server device that controls the home appliance.
The home appliance according to any one of (1) to (7).
(9) When the predetermined situation is detected, the control unit transmits information that can be used to identify the type of the home appliance in addition to the first signal.
The home appliance according to any one of (1) to (8).
(10) When the home appliance is in the first specific situation, the control unit performs processing in association with the user's utterance to the external device and the home appliance, or associates the utterance with the home appliance. sending a second signal to the external device or the server device for increasing the priority of processing by
The home appliance according to any one of (1) to (9).
(11) When the home appliance is in the second specific situation, the control unit performs processing for not associating the user's utterance to the external device with the home appliance, or by associating the utterance with the home appliance. sending a third signal to the external device or the server device for lowering the priority of
The home appliance according to any one of (1) to (10).
(12) when the external device activates the voice interaction function based on a trigger word uttered by a user, the external device terminates the voice interaction function when there is no user utterance for a first predetermined time;
When the predetermined situation is detected, the control unit maintains the activated state of the voice interaction function for a second predetermined time longer than the first predetermined time even if the user does not speak. to the external device or the server device,
The home appliance according to any one of (1) to (11).
(13) an information processing unit that activates a voice interaction function of a device arranged independently of the home appliance when the first signal is transmitted from the home appliance;
Information processing system with
(14) When the information processing unit receives the first signal and identification information that can be used to identify the home appliance, the information processing unit performs perform a predetermined process in association with the home appliance identified by
The information processing system according to (13).
(15) When the user speaks to the voice interaction function after receiving the first signal from a plurality of home appliances, the information processing unit outputs the first signal last among the plurality of home appliances. Performing the predetermined process by associating the transmitted home appliance with the utterance;
The information processing system according to (14).
(16) The predetermined process includes analyzing the content of the utterance based on the type of the home appliance.
The information processing system according to (14) or (15).
(17) When the user speaks to the voice interaction function after receiving the first signal from a plurality of home appliances, the information processing unit outputs the first signal last among the plurality of home appliances. determining whether or not affinity between the transmitted home appliance and the contents of the utterance is equal to or higher than a reference, and if the affinity is equal to or higher than the reference, finally the home appliance that transmitted the first signal and the utterance; performs a predetermined process in association with the contents of
(13) The information processing system according to any one of (13) to (16).
(18) If the affinity between the home appliance that last transmitted the first signal among the plurality of home appliances and the content of the utterance is less than the reference, the information processing unit Determines the affinity between the home appliance that transmitted the first signal one before and the content of the utterance,
The information processing system according to (17).
(19) When the information processing unit receives the first signal and identification information that can be used to identify the home appliance, the information processing unit outputs information about the home appliance identified based on the identification information. notified by the voice interaction function;
The information processing system according to any one of (13) to (18).

While several embodiments of the invention have been described, these embodiments have been presented by way of example and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, replacements, and modifications can be made without departing from the scope of the invention. These embodiments and their modifications are included in the scope and spirit of the invention, as well as the scope of the invention described in the claims and equivalents thereof.

Reference Signs List 1 information processing system 100 home appliance 100V voice interaction home appliance (external device) 101 housing 101a opening 102 door 110 door open/close detection sensor 120 human sensor 130 Trigger dedicated operation unit 140 Power operation unit 150 Display/operation unit R Input reception unit 180 Control unit 200 Smart speaker (external device) 250 Control unit (information processing unit) 400 Server device for voice interaction, 430... Information processing unit, 500... Server device for home appliance control, 530... Information processing unit.

Claims (15)

a control unit that, when a predetermined situation is detected, transmits a first signal for activating the voice interaction function of the external device to the external device or the server device;
with
The external device activates the voice interaction function when the user utters a trigger word and when it receives a trigger signal that is the first signal or a signal generated based on the first signal. A waiting time is set for maintaining the activated state of the voice interaction function even when the user does not speak, and a first predetermined time is set as the waiting time when the user speaks the trigger word. and setting a second predetermined time longer than the first predetermined time as the standby time when the trigger signal is received,
When the predetermined situation is detected, the control unit transmits the first signal for setting the second predetermined time as the waiting time to the external device or the server device.
home appliances. The second predetermined time is a time longer than or equal to twice the first predetermined time.
The home appliance according to claim 1. The second predetermined time is a time of 5 minutes or more,
The home appliance according to claim 2. a control unit that, when a predetermined situation is detected, transmits a first signal for activating the voice interaction function of the external device to the external device or the server device;
with
The external device activates the voice interaction function when the user utters a trigger word and when it receives a trigger signal that is the first signal or a signal generated based on the first signal. setting a standby time for activating the voice interaction function and maintaining the activated state of the voice interaction function even when the user does not speak;
If the user does not speak after transmitting the first signal to the external device or the server device, the control unit transmits an additional signal for extending the waiting time before the waiting time elapses, Send to the external device or the server device at least once;
home appliances. The control unit repeatedly transmits the additional signal when the predetermined situation continues to be detected.
The home appliance according to claim 4. further comprising a trigger operation unit that receives an operation for activating the voice interaction function;
The control unit transmits the first signal when the trigger operation unit is operated.
The home appliance according to any one of claims 1 to 5 . a housing having an opening;
a door that closes the opening,
The control unit transmits the first signal when the door is opened with respect to the opening.
A home appliance according to any one of claims 1 to 6 . a housing having an opening;
a door that closes the opening,
The control unit transmits the first signal when the door is closed with respect to the opening.
A home appliance according to any one of claims 1 to 7 . Equipped with a human sensor,
The control unit transmits the first signal when the presence of a person is detected by the human sensor.
A home appliance according to any one of claims 1 to 8 . It also has a power operation unit,
The control unit transmits the first signal when the power operation unit is operated.
A home appliance according to any one of claims 1 to 9 . The control unit outputs the first signal regarding one or more situations based on at least one of a user's operation on an operation reception unit of the home appliance and a user's operation on a terminal device other than the home appliance. Change the setting of whether to send or not,
A home appliance according to any one of claims 1 to 10 . When the predetermined situation is detected, the control unit transmits information that can be used to identify a server device that controls the home appliance in addition to the first signal.
A home appliance according to any one of claims 1 to 11 . When the predetermined situation is detected, the control unit transmits information that can be used to identify the type of the home appliance in addition to the first signal.
A home appliance according to any one of claims 1 to 12 . When the home appliance is in the first specific situation, the control unit causes the user's utterance to the external device and the home appliance to be processed in association with each other, or causes the utterance and the home appliance to be processed in association with each other. Sending a second signal for increasing the priority to the external device or the server device;
A home appliance according to any one of claims 1 to 13 . The control unit, when the home appliance is in a second specific situation, has a priority order for not associating the user's utterance to the external device with the home appliance, or for associating the utterance with the home appliance for processing. sending a third signal to the external device or the server device to lower
15. A home appliance according to any one of claims 1-14 .

Images