CN112242939A

CN112242939A - Household electrical appliance and information processing system

Info

Publication number: CN112242939A
Application number: CN202010473413.7A
Authority: CN
Inventors: 丸谷裕树; 中川达也; 泷川正史; 金山将也; 神田博纪
Original assignee: Toshiba Lifestyle Products and Services Corp
Current assignee: Toshiba Lifestyle Products and Services Corp
Priority date: 2019-07-18
Filing date: 2020-05-29
Publication date: 2021-01-19
Also published as: JP7269812B2; JP2021018543A

Abstract

The invention relates to a household appliance and an information processing system, which can improve the convenience of users. The home appliance of the embodiment has a control unit. When a predetermined situation is detected, the control unit transmits a first signal for activating a voice dialogue function of an external device to the external device or the server apparatus.

Description

Household electrical appliance and information processing system

Technical Field

Embodiments of the present invention relate to a home appliance and an information processing system.

Background

A home appliance is provided with a detection unit that detects the occurrence of a predetermined phenomenon that triggers the start of a session of the home appliance, and a determination unit that determines whether or not a predetermined start condition is satisfied if the detection unit detects the occurrence of the phenomenon.

However, if it is assumed that the home appliance is operated by a voice dialogue device disposed independently of the home appliance, it is necessary to sound a trigger word for activating the voice dialogue device or a word for specifying the home appliance to be operated, and there is room for improvement in user convenience.

Documents of the prior art

Patent document

Patent document 1: japanese patent laid-open publication No. 2017-84177

Disclosure of Invention

Problems to be solved by the invention

The present invention addresses the problem of providing a home appliance and an information processing system that can improve user convenience.

Means for solving the problems

The home appliance of the embodiment has a control unit. The control unit transmits a first signal for activating a voice dialogue function of an external device to the external device or the server apparatus when a predetermined situation is detected.

Effects of the invention

According to the above configuration, the convenience of the user can be improved for the voice dialogue function.

Drawings

Fig. 1 is a diagram showing an overall configuration of an information processing system according to a first embodiment.

Fig. 2 is a block diagram showing a functional configuration of the smart speaker according to the first embodiment.

Fig. 3 is a perspective view showing some examples of the home appliance of the first embodiment.

Fig. 4 is a block diagram showing a functional configuration of the home appliance according to the first embodiment.

Fig. 5 is a diagram showing an example of the contents of the display screen according to the first embodiment.

Fig. 6 is a block diagram showing a functional configuration of the voice dialogue server device according to the first embodiment.

Fig. 7 is a diagram showing an example of the contents of the voice operation management DB according to the first embodiment.

Fig. 8 is a block diagram showing a functional configuration of the device control server apparatus according to the first embodiment.

Fig. 9 is a timing chart showing an example of the overall flow of control in the first embodiment.

Fig. 10 is a flowchart showing a flow of processing related to the determination of a sound emission target according to the first embodiment.

Fig. 11 is a diagram showing an example of the determination rule according to the first embodiment.

Fig. 12 is a diagram showing the overall configuration of the information processing system according to the second embodiment.

Detailed Description

Hereinafter, a home appliance and an information processing system according to an embodiment will be described with reference to the drawings. In the following description, the same reference numerals are given to the components having the same or similar functions. In addition, a repetitive description of these configurations may be omitted.

In the present specification, "based on XX" means "based on at least XX", and includes cases where the "is based on other elements in addition to XX. "based on XX" is not limited to the case of using XX directly, and includes the case of using the result of calculation or processing performed on XX. "XX" is an arbitrary element (e.g., arbitrary information). In the present specification, "for executing YY" or "for executing YY" is not limited to the case of executing YY as a main purpose, but includes the case of executing YY secondarily together with or before or after the execution of the main purpose. "YY" is an arbitrary action, for example. In the present specification, the "database" is referred to as "DB".

(first embodiment)

First, a first embodiment will be explained. The first embodiment is an example in which the smart speaker 200, which is provided independently of the home appliance 100, has a voice conversation function. The first embodiment is an example in which a trigger signal described later is transmitted from home appliance 100 to smart speaker 200.

[1. Overall Structure ]

Fig. 1 is a diagram showing an overall configuration of an information processing system 1 according to a first embodiment. The information processing system 1 includes, for example, one or more (for example, a plurality of) home appliances 100, a smart speaker 200, an in-home router 300, a server apparatus 400 for audio conversation (hereinafter, referred to as "audio conversation server apparatus 400"), a server apparatus 500 for home appliance control (hereinafter, referred to as "apparatus control server apparatus 500"), and a terminal device 600. In the present embodiment, the control unit 250 of the smart speaker 200, the information processing unit 430 of the audio dialog server device 400, and the information processing unit 530 of the device control server device 500, which will be described later, respectively, implement the information processing unit of the information processing system 1. The network NW to be described later includes, for example, one or more of the internet, a cellular network, a Wi-Fi network, a wan (wide Area network), a lan (local Area network), a public line, a telephone line, a radio base station, and the like.

The home appliance 100 is disposed in a home of a user. The home appliance 100 is connected to the network NW via the router 300, and can communicate with the audio session server apparatus 400 and the device control server apparatus 500. In addition, the home appliance 100 can communicate with the smart speaker 200 via the router 300 or directly. The home appliance 100 can communicate with the terminal device 600 via the router 300 or via the router 300 and the network NW. In the present embodiment, a refrigerator 100A, a toaster microwave 100B, and a washing machine 100C are disposed as examples of the home appliance 100. However, the home appliance 100 is not limited to the above example, and may be other home appliances such as a vacuum cleaner, an electric cooker, or a television receiver. Hereinafter, the home appliance 100 is simply referred to as "home appliance 100" regardless of the type of the home appliance 100.

The smart speaker 200 is an example of a "voice dialog device". The smart speaker 200 is also an example of an "external device" viewed from the home appliance 100. The smart speaker 200 is disposed at the home of the user. Smart speaker 200 is connected to network NW via router 300, for example, and can communicate with voice conversation server apparatus 400 and device control server apparatus 500. One smart speaker 200 is disposed in a kitchen or the like with respect to a plurality of home appliances 100. In the present embodiment, the smart speaker 200 is a general-purpose smart speaker, and can cope with operations of the home appliance 100, answers to questions such as weather and time, and streaming media of music. Note that the "audio dialog apparatus" and the "external device" referred to in this specification may be dedicated audio dialog apparatuses for operating the home appliance 100.

The router 300 is configured at the home of the user. The router 300 is, for example, a wireless router, but is not limited thereto. In other words, at least one of the home appliance device 100 and the smart speaker 200 can communicate with the router 300 by connecting to the router 300 via a wire (e.g., a power line).

The voice conversation server apparatus 400 is, for example, a cloud server connected to the network NW. The voice dialogue server 400 analyzes the content of the sound emitted from the user to the smart speaker 200, and notifies the device control server 500 of the answer from the sound emitted from the smart speaker 200 and the control instruction for the home appliance 100 corresponding to the content of the sound emitted from the user. The voice conversation server apparatus 400 is a server apparatus that provides services to the manufacturer or the seller of the smart speaker 200, for example. In addition, a part or all of the functions of the voice dialogue server 400, which will be described later, may be provided in the smart speaker 200.

The device control server apparatus 500 is, for example, a cloud server connected to a network NW. When a control instruction for the home appliance 100 is notified from the voice session server apparatus 400, the appliance control server apparatus 500 controls the home appliance 100 based on the notified control instruction. The device control server 500 is a server that provides services to the manufacturer or the seller of the home appliance 100, for example. There may be a plurality of device control server apparatuses 500 for each manufacturer or vendor of home appliances 100. The voice conversation server apparatus 400 and the device control server apparatus 500 do not need to be separate apparatuses, and may be implemented by one apparatus.

The terminal device 600 is, for example, a mobile terminal device owned by a user, and is a smartphone, a tablet terminal device, or the like. For example, the terminal device 600 has an application program installed therein for setting and remotely operating the home appliance 100. However, the terminal device 600 may be a personal computer, a television receiver having an input function, or the like.

Hereinafter, the home appliance 100, the smart speaker 200, the voice dialogue server 400, and the appliance control server 500 will be described in detail. Here, for convenience of explanation, the smart speaker 200, the home appliance 100, the voice dialogue server apparatus 400, and the device control server apparatus 500 will be explained in this order.

[2. Intelligent speaker ]

First, the smart speaker 200 will be explained. Fig. 2 is a block diagram showing a functional configuration of the smart speaker 200. The smart speaker 200 includes, for example, a microphone 210, a speaker 220, a wireless communication module 230, a storage unit 240, and a control unit (information processing unit) 250.

The microphone 210 takes the sound emitted by the user with respect to the smart speaker 200. The speaker 220 outputs sound toward the outside of the smart speaker 200. The wireless communication module 230 includes a high-frequency circuit, an antenna, and the like, and is capable of wireless communication with the router 300, for example.

The storage unit 240 stores an audio DB 241. The sound DB241 includes, for example, a simple sound recognition DB241 a. For example, a trigger word (for example, a word uttered by a predetermined user such as "Hello, good quality" or the like) serving as a voice command for activating the smart speaker 200 is registered in the simple voice recognition DB241 a.

The control unit 250 controls the entire smart speaker 200. The control unit 250 includes, for example, an audio receiving unit 251, a start determining unit 252, a standby time setting unit 253, a data transmitting/receiving unit 254, and an audio output unit 255. However, some of these functional units (for example, the activation determination unit 252 and the standby time setting unit 253) do not need to be installed at the time of factory shipment of the smart speaker 200, and may be realized by installing an additional application program in the smart speaker 200, for example.

The sound receiving unit 251 receives an utterance input by the user with respect to the microphone 210 as sound data.

The activation determination unit 252 determines whether or not the voice dialogue function of the smart speaker 200 needs to be activated. The "activation" refers to, for example, changing from a rest state (a state in which no response is made to a user utterance other than a trigger word) in which power consumption is suppressed to a predetermined level or less to a standby state (a state in which a voice command can be received) in which a response is made (dialogue is possible) to a user utterance other than a trigger word. The "standby state" may also be referred to as the "startup state". For example, when the sound data received by the sound receiving unit 251 includes a trigger word registered in the simple sound recognition DB241a, the activation determination unit 252 determines that activation is necessary and activates the sound dialogue function. In the present embodiment, when the data transmission/reception unit 254 receives a trigger signal described later, the activation determination unit 252 determines that activation is necessary and activates the audio dialogue function.

When the start determination unit 252 determines that the audio dialog function needs to be started and the audio dialog function is started, the standby time setting unit 253 sets a standby time (time out) for maintaining the standby state.

Here, the standby time setting unit 253 may set the standby time to be different between a case where the audio dialogue function is activated based on the trigger word and a case where the audio dialogue function is activated based on the trigger signal. For example, when the voice dialogue function is activated based on a trigger word, the standby time setting unit 253 maintains the standby time for a first predetermined time (for example, 10 seconds) even when the user does not make a sound. On the other hand, when the voice dialogue function is activated based on the trigger signal, the standby time setting unit 253 maintains the standby time for a second predetermined time (for example, about 10 seconds to 30 seconds, or about 10 minutes to 30 minutes) longer than the first predetermined time even when the user does not make a sound. The length of the second predetermined time is, for example, 1 time or more of the length of the first predetermined time.

For example, when the second predetermined time is set to less than 5 minutes (more specifically, about 10 seconds to 30 seconds), even when the user is performing an operation on the home appliance 100 (for example, accessing an article to the inside of the home appliance 100) at the timing when the trigger signal is transmitted and the user does not immediately sound, it is possible to reduce the possibility that the instruction reception to the user issued during or after the operation is impaired.

For example, when the second predetermined time is set to 5 minutes or longer (in short, about 10 to 30 minutes), even when the user is performing an operation related to the home appliance 100 or the installation location of the home appliance 100 (for example, cooking, post-cooking collection, preparation for laundry, cleaning of a room, or other housework) at the timing when the trigger signal is transmitted and the user does not immediately sound, it is not necessary to call every time when the smart speaker 200 is used (for example, even when the user does not call every time during cooking, the user can call the cooking).

For example, the standby time setting unit 253 may have a setting function of turning ON (ON) the latter long-time standby function (a function of maintaining the smart speaker 200 in an activated state while the user is performing an operation related to the home appliance 100 or an installation location of the home appliance 100, for example, a function of maintaining the smart speaker 200 in an activated state for a long time of about 10 minutes to 30 minutes). Hereinafter, this function is referred to as a "long standby function".

For example, the standby time setting unit 253 may switch the long standby function ON/OFF (ON/OFF) based ON a predetermined voice command generated by a user's voice. When the long-time standby function is turned off, if the user does not make a sound for a first predetermined time (for example, 10 seconds) or a second predetermined time (for example, less than 5 minutes), the standby time setting unit 253 ends the activation state of the smart speaker 200. On the other hand, after the standby state is established as usual based on the trigger word (or the trigger signal from the home appliance 100), when the voice command for turning on the long standby function is received, the standby time setting unit 253 enters the long standby state. Then, if there is no sound from the user, the standby time setting unit 253 ends the activation state of the smart speaker 200 after a predetermined standby time (for example, about 10 minutes to 30 minutes) for a long time. Alternatively, the standby time setting unit 253 may terminate the activation state of the smart speaker 200 when receiving a voice command to turn off the long-time standby function.

The data transmitting/receiving unit 254 controls the wireless communication module 230 to transmit and receive signals and data to and from the home appliance 100, the voice session server apparatus 400, and the like. For example, the data transmission/reception unit 254 transmits the voice data other than the trigger word acquired by the voice reception unit 251 to the voice conversation server apparatus 400. The data transmitting/receiving unit 254 receives the voice data for response prepared by the voice dialogue server apparatus 400 from the voice dialogue server apparatus 400. The data transceiver 254 outputs the received voice data for response to the voice output unit 255.

In the present embodiment, the data transmission/reception unit 254 receives a predetermined signal (a trigger signal, an auxiliary signal, and the like, which will be described later) from the home appliance 100. In addition, in response to the reception of the predetermined signal, the data transmission/reception unit 254 receives information (identification information) usable for identifying the home appliance 100 from the home appliance 100 that is the transmission source of the predetermined signal. The "identification information" in the present specification is not limited to information indicating an ID number (for example, a serial number) of the home appliance 100, and may be information indicating a communication address (for example, an IP address) of the home appliance 100, and the like, as long as the home appliance 100 can be distinguished from other home appliances 100. Hereinafter, a case where the data transmission/reception unit 254 acquires the communication address of the home appliance 100 as the identification information will be described as an example. The data transmitting/receiving unit 254 transmits information indicating that the predetermined signal is received, information indicating the time at which the predetermined signal is received, and identification information of the home appliance 100 that transmitted the predetermined signal to the voice conversation server apparatus 400 in association with each other.

The audio output unit 255 outputs audio data, which is transmitted from the audio session server apparatus 400 and received by the data transmission/reception unit 254, to the outside as audio through the speaker 220.

[3. household electric appliances ]

Next, the home appliance 100 will be explained.

Fig. 3 is a perspective view showing some examples of the home appliance 100. The home appliance 100 includes, for example, a housing 101 and a door 102. The housing 101 has an opening 101a for accessing items with respect to the internal space of the housing 101. The door 102 is closed to be able to open and close the opening 101a of the housing 101.

Fig. 4 is a block diagram showing a functional configuration of the home appliance 100. The home appliance 100 includes, for example, a door opening/closing detection sensor 110, a human sensing sensor 120, a trigger-dedicated operation unit 130, a power supply operation unit 140, a display/operation unit 150, a wireless communication module 160, an actuator 170, and a control unit 180. The home appliance 100 may not have a part of these configurations.

The door opening/closing detection sensor 110 detects opening/closing of the door 102 with respect to the opening 101a of the housing 101. The door opening/closing detection sensor 110 is, for example, a door opening/closing switch that is provided between the housing 101 and the door 102 and switches between an on state and an off state when the door 102 is opened and closed. However, the door opening/closing detection sensor 110 may be another sensor (for example, a camera) capable of opening/closing the door 102. The human perception sensor 120 detects the presence of a human in the vicinity of the home device 100 (e.g., the front of the home device 100).

The trigger-dedicated operation unit 130 is a dedicated operation unit that is operated by the user when the user wants to transmit a trigger signal described later. That is, the trigger-dedicated operation unit 130 is an operation unit that receives an operation for starting the voice dialogue function of the smart speaker 200. The trigger-dedicated operation unit 130 includes, for example, a dedicated button or switch for transmitting a trigger signal. The trigger-dedicated operation unit 130 is an example of a "trigger operation unit". However, the "trigger operation unit" referred to in the present specification is not limited to a dedicated operation unit, and may be an operation unit used in combination with other functions.

The power source operation unit 140 is an operation unit operated by a user when the power source of the home appliance 100 is turned from the off state to the on state or from the on state to the off state. The power source operation unit 140 includes a button or a switch for switching the on state and the off state of the power source.

The display/operation section 150 includes, for example, a display device and a touch input type operation section. The display device is, for example, a liquid crystal display, an Organic EL (Electro-Luminescence) display, or the like, and includes a display screen on which images and videos are displayed. The operation unit includes a touch sensor arranged to overlap with a display screen of the display device. The display/operation unit 150 receives, for example, an operation of a user for starting the operation of the home appliance 100 and changing the operation mode.

In the present embodiment, the trigger-dedicated operation unit 130, the power supply operation unit 140, and the display/operation unit 150 constitute an example of an operation reception unit R that receives an operation by a user. However, the operation reception unit R may include a general button, a switch, and the like for receiving a user operation for starting the operation of the home appliance 100 or changing the operation mode, instead of or in addition to the trigger-dedicated operation unit 130, the power source operation unit 140, and the display/operation unit 150.

The wireless communication module 160 includes a high-frequency circuit, an antenna, and the like, and is capable of wireless communication with the router 300, for example.

The actuator 170 is a driving unit for operating the home appliance 100. For example, when the household appliance 100 is a refrigerator, the actuator 170 is a compressor for compressing a refrigerant, a blower for circulating cold air in the refrigerator, or the like. In the case where the electric home appliance 100 is a microwave oven, the actuator 170 is a microwave generator or the like. When the home appliance 100 is a washing machine, the actuator 170 is a motor or the like that rotates a washing tub.

The control unit 180 controls the entire home appliance 100. The control unit 180 includes, for example, a trigger signal generation unit 181, a trigger mode setting unit 182, an assist signal generation unit 183, a device control unit 184, and a data transmission/reception unit 185.

When a predetermined situation is detected, the trigger signal generating unit 181 generates a trigger signal for activating the audio dialogue function of the smart speaker 200. The trigger signal is an example of the "first signal". The "predetermined status" is a status that is set in advance for the home appliance 100 as a status for transmitting the trigger signal. The "predetermined condition" is, for example, any one or more of a case where the trigger operation unit 130 is operated, a case where the power operation unit 140 is operated, a case where the door 102 is opened with respect to the opening 101a of the casing 101, a case where the door 102 is closed with respect to the opening 101a of the casing 101, and a case where the presence of a person is detected in the vicinity of the home appliance 100. The "case where the predetermined condition is detected" is, for example, any one or more of a case where the trigger operation unit 130 is detected by the trigger operation unit 130 as being operated, a case where the power operation unit 140 is detected as being operated (for example, an operation of turning the power of the home appliance 100 from an off state to an on state) by the power operation unit 140, a case where the door 102 is detected as being opened with respect to the opening 101a of the housing 101 by the door opening/closing detection sensor 110, a case where the door 102 is detected as being closed with respect to the opening 101a of the housing 101 by the door opening/closing detection sensor 110, and a case where the presence of a person is detected by the person detection sensor 120.

For example, the trigger signal generating part 181 may generate different trigger signals in a case where the trigger operation part 130 is operated, a case where the power operation part 140 is operated, a case where the door 102 is opened with respect to the opening 101a of the casing 101, a case where the door 102 is closed with respect to the opening 101a of the casing 101, and a case where the presence of a person is detected by the person sensing sensor 120. With such a configuration, in the voice dialogue server apparatus 400 described later, when it is necessary to determine the home appliance 100 to which the user is to generate sound from among the plurality of home appliances 100 that have transmitted the trigger signal, it is possible to prioritize the home appliances 100 according to their statuses. For example, the audio session server apparatus 400 can increase the priority of the home appliance 100 that has transmitted the trigger signal when the door 102 is opened (or closed) with respect to the opening 101a of the housing 101, and can increase the priority of the home appliance 100 that has transmitted the trigger signal when the trigger operation unit 130 is operated, as compared to the home appliance 100 that has transmitted the trigger signal when the presence of a person is detected by the person detection sensor 120. In addition, different trigger signals may be assigned to different types of trigger operation units so that the smart speaker 200 may have different standby times for maintaining the activated state (sound receiving state) depending on the types of various operation units and sensors having functions as the trigger operation units, such as the power supply operation unit 140 and the door opening/closing detection sensor 100. That is, when different types of trigger operation portions are operated, different trigger signals may be transmitted.

However, according to the smart speaker 200, the standby time for maintaining the standby state (the activated state) may be fixed. That is, the smart speaker 200 does not have the standby time setting unit 253 as described above, and may set the standby time for the same predetermined time (first predetermined time, for example, 5 minutes) in a case where the voice dialogue function is activated based on the trigger word or in a case where the voice dialogue function is activated based on the trigger signal.

In the present embodiment, when the predetermined condition is detected, the trigger signal generating unit 181 corresponding to the smart speaker 200 having a fixed standby time can generate an additional signal for maintaining the standby state of the voice dialogue function of the smart speaker 200 for a second predetermined time longer than the first predetermined time even when the user does not make a voice. The additional signal may be the same signal as the trigger signal or another signal. In this case, the data transceiver 185, which will be described later, may extend the standby time of the smart speaker 200 to the second predetermined time by transmitting the additional signal to the smart speaker 200 before the first predetermined time elapses from the transmission of the trigger signal to the smart speaker 200, or by repeatedly transmitting such additional signal. With such a configuration, even if smart speaker 200 does not have a special configuration, the standby time can be made different between a case where the voice dialogue function is activated based on a trigger word and a case where the voice dialogue function is activated based on a trigger signal.

In one example, the trigger signal generating unit 181 generates a trigger signal when the door 102 of the home appliance 100 is opened or closed, the trigger-dedicated operation unit 130 is operated, or the power operation unit 140 is operated, and transmits the trigger signal to the smart speaker 200 through the data transmitting/receiving unit 185. Then, as long as the presence of a person is continuously detected by the human perception sensor 120 (for example, as long as the presence of a person in the kitchen is continuously detected), the trigger signal generation unit 181 may generate an additional signal at a predetermined cycle and transmit the additional signal to the smart speaker 200 through the data transmission/reception unit 185. With this configuration, the standby state of the smart speaker 200 can be maintained while the user is in the vicinity after operating the home appliance 100, while the smart speaker 200 can be prevented from being activated when the user passes through the vicinity regardless of the home appliance 100.

The trigger mode setting unit 182 changes the setting as to whether or not the trigger signal is transmitted in one or more situations based on at least one of the operation receiving unit R of the home appliance 100 by the user and the operation of the terminal device 600 by the user.

Fig. 5 is a diagram showing an example of the content of the display screen 151 displayed on the display/operation unit 150 or the terminal device 600. For example, a screen for accepting a change as to whether or not a trigger signal is transmitted in relation to one or more statuses detectable by the home appliance 100 is displayed on the display screen 151. The trigger mode setting unit 182 changes the setting as to whether or not a trigger signal is transmitted in one or more situations based on an operation for the display screen 151.

Referring back to fig. 4, the remaining configuration of the home appliance 100 will be described. The auxiliary signal generator 183 generates a signal for assisting the association between "the user uttering" and "the home appliance to be uttered" in the audio session server apparatus 400, which will be described later. For example, when the home appliance 100 is in the first specific state, the auxiliary signal generating unit 183 generates the first auxiliary signal for associating and processing the sound emission of the smart speaker 200 with the home appliance 100 or for increasing the priority of associating and processing the sound emission with the home appliance 100. The first specific condition is a condition in which the utterance of the user and the home appliance 100 should establish an association with priority. The first auxiliary signal is an example of the "second signal". Specific examples thereof will be described later.

When the home appliance 100 is in the second specific state, the auxiliary signal generator 183 generates the second auxiliary signal for causing the user to not associate the sound emission from the smart speaker 200 with the home appliance 100 or for lowering the priority for associating and processing the sound emission with the home appliance 100. The second specific condition is a condition in which the utterance of the user and the home appliance 100 should not preferentially establish an association. The second auxiliary signal is an example of a "third signal". Specific examples thereof will be described later. Hereinafter, the first auxiliary signal and the second auxiliary signal are simply referred to as "auxiliary signals" without distinguishing them from each other.

The appliance control unit 184 controls the home appliance 100 based on the user operation received by the operation receiving unit R and the control command received from the appliance control server device 500. For example, the equipment control unit 184 drives the actuator 170 based on the user operation received by the operation receiving unit R and the control command received from the equipment control server device 500.

The data transmitting/receiving unit 185 controls the wireless communication module 160 to transmit and receive signals and data to and from the smart speaker 200, the device control server apparatus 500, and the like. For example, the data transceiver 185 receives a control command for controlling the home appliance 100 from the appliance control server device 500. When the trigger signal and the additional signal are generated by the trigger signal generation unit 181, the data transmission/reception unit 185 transmits the generated trigger signal and the additional signal to the smart speaker 200. When the auxiliary signal is generated by the auxiliary signal generating unit 183, the data transmitting/receiving unit 185 transmits the generated auxiliary signal to the smart speaker 200.

When the predetermined condition is detected, the data transmission/reception unit 185 transmits, in addition to the trigger signal, identification information that can be used to identify the device control server apparatus 500 that controls the home appliance 100. From another point of view, when the predetermined condition is detected, the data transmission/reception unit 185 transmits identification information that can be used to identify the type of the home appliance 100, in addition to the trigger signal. In addition, when the additional signal or the auxiliary signal is transmitted, the data transmission/reception unit 185 transmits the identification information in addition to the additional signal or the auxiliary signal.

As described above, the identification information is not limited to information indicating the ID number of the home appliance 100, and may be information indicating the communication address of the home appliance 100 or the like. For example, when the identification information is the communication address of the home appliance 100, the voice session server apparatus 400 can specify the device control server apparatus 500 that controls the home appliance 100 and the type of the home appliance 100 by referring to the voice operation management DB422 described later. Note that "transmitting identification information in addition to a signal" in this specification is not limited to the case of transmitting identification information simultaneously with a signal, and includes the case of transmitting identification information before or after transmission of a signal.

In addition, instead of transmitting the trigger signal and the identification information to the smart speaker 200, the home appliance 100 may transmit the trigger signal and the identification information to the voice dialogue server apparatus 400 or the appliance control server apparatus 500. In this case, voice conversation server apparatus 400 or device control server apparatus 500 may transmit (for example, forward) the trigger signal received from home appliance 100 to smart speaker 200.

In addition, instead of transmitting the trigger signal itself for activating the voice dialogue function of the smart speaker 200, the home appliance 100 may transmit a signal (hereinafter, referred to as a "sub-trigger signal" for convenience of description) in accordance with the trigger signal to the voice dialogue server 400 or the device control server 500 when the predetermined situation is detected. The sub trigger signal is a signal indicating that the predetermined condition has occurred. In this case, voice session server apparatus 400 or device control server apparatus 500 may generate a trigger signal for activating the voice session function of smart speaker 200 based on the sub-trigger signal received from home appliance 100, and may transmit the generated trigger signal to smart speaker 200. The sub trigger signal is another example of the "first signal". In addition, the "trigger signal" in the following description may be replaced with a "sub-trigger signal".

These are also the same for the above-described additional signal and auxiliary signal. In the present specification, "a signal is transmitted from the home appliance 100 to the voice session server apparatus 400" is not limited to a case where a signal is directly transmitted from the home appliance 100 to the voice session server apparatus 400, and includes a case where a signal is transmitted from the home appliance 100 to the voice session server apparatus 400 via the device control server apparatus 500.

[4. Voice conversation Server device ]

Next, the voice conversation server apparatus 400 will be explained.

Fig. 6 is a block diagram showing a functional configuration of the voice conversation server apparatus 400. The voice conversation server apparatus 400 includes, for example, a communication module 410, a storage unit 420, and an information processing unit 430.

The communication module 410 includes a high frequency circuit and the like. The communication module 410 is capable of communicating with the smart speaker 200 via the network NW and the router 300. In addition, the communication module 410 can communicate with the device control server apparatus 500 via the network NW.

The storage unit 420 stores a voice DB421 and a voice operation management DB 422. The sound DB421 includes, for example, a sound recognition DB421a and a vocalization DB421 b. In the voice recognition DB421a, a dictionary used for analyzing utterances of the smart speaker 200 by the user, information defining the relationship of objects, and the like are registered. The utterance DB421b has registered therein sound data (utterance data) for generating a response output from the smart speaker 200.

Fig. 7 is a diagram showing an example of the contents of the voice operation management DB 422. The voice operation management DB422 has registered therein information for specifying the home appliance 100 based on identification information (e.g., a communication address) transmitted from the home appliance 100. For example, in the voice operation management DB422, identification information (for example, a communication address) of the home appliance 100, a home appliance ID capable of identifying the home appliance 100, a user ID capable of identifying a user of the home appliance 100, an ID of the device control server apparatus 500 corresponding to the home appliance 100, and a type of the home appliance 100 are associated with each other.

Returning to fig. 6, the remaining configuration of the audio session server apparatus 400 will be described. The information processing unit 430 includes, for example, a sound processing unit 431, a sound generation object determination unit 432, a control content specifying unit 433, a control instruction generation unit 434, a response generation unit 435, and a data transmission/reception unit 436.

The sound processing unit 431 analyzes the sound data of the utterance of the user acquired by the smart speaker 200. For example, the voice processing unit 431 performs voice recognition processing to convert voice data into text. The voice processing unit 431 performs meaning interpretation (primary meaning interpretation) by performing natural language processing on the text-converted voice data. The sound data of the user having performed these processes is output to the sound object determination unit 432 and the control content specification unit 433.

When the trigger signal is transmitted from the plurality of home appliances 100, the sound emission target determination unit 432 determines the home appliance 100 to which the user is to emit sound from among the plurality of home appliances 100 that have transmitted the trigger signal. For example, the sound-emission target determination unit 432 determines the home appliance 100 to which the user is to emit sound, based on the trigger signal and the identification information transmitted from the home appliance 100, the sound-emission content of the user whose meaning has been interpreted by the sound processing unit 431, the content (the type of each home appliance 100, and the like) registered in the sound operation management DB422, the presence or absence of the auxiliary signal from the home appliance 100, and the like. The processing of the sound emission target determination unit 432 will be described in detail later.

The control content specifying unit 433 performs a deeper meaning interpretation (secondary meaning interpretation) of the sound emission content of the user based on the sound emission content of the user whose meaning interpretation is performed by the sound processing unit 431 and the type of the home appliance 100 of the sound emission target determined (specified) by the sound emission target determining unit 432, and specifies the control content for the home appliance 100. In other words, the control content determination unit 433 analyzes the sound content of the user based on the type of the home appliance 100. The "explaining the utterance of the user based on the type of the home appliance 100" is an example of "a predetermined process performed by associating the utterance of the user with the home appliance specified based on the identification information".

For example, if the electric appliance 100 is a refrigerator, the control contents for the electric appliance 100 are "quick freeze", "quick ice make", "thaw", and the like. In the case where the home appliance 100 is an oven microwave oven, the control contents for the home appliance 100 are "good heating quality for minutes", "thawing", and the like. When the home appliance 100 is a washing machine, the control contents for the home appliance 100 are "normal water amount", "dry cleaning program", and the like. The control content determining unit 433 outputs the determined control content to the response generating unit 435 and the control instruction generating unit 434.

The control instruction generating unit 434 generates a control instruction for the home appliance 100 corresponding to the control content determined by the control content determining unit 433.

When there is a content to be answered for the utterance content of the user that is meant to be interpreted by the control content specifying unit 433, or when there is another content to be reported to the user, the response generating unit 435 generates the voice data output from the smart speaker 200 with reference to the utterance DB421 b. The content to which the user's utterance should be answered is, for example, "rapid ice making is instructed to the refrigerator", or the like. The content to be reported is, for example, "good quality again is good for ice in minutes" or the like.

When information that the home appliance 100 that has transmitted the trigger signal (the home appliance 100 specified based on the identification information) desires to be notified of the user is transmitted from the appliance control server apparatus 500 to the voice dialogue server apparatus 400, the response generation unit 435 generates voice data output from the smart speaker 200 based on the information. The information desired to be reported to the user is information related to the home appliance 100 that has transmitted the trigger signal, for example, inventory information and recipe information in the case where the home appliance 100 is a refrigerator, and pollen information in the latest 1 week in the case where the home appliance 100 is a washing machine.

The data transceiver 436 controls the communication module 410 to transmit and receive signals and data to and from the smart speaker 200, the device control server apparatus 500, and the like. The data transceiver 436 receives voice data uttered by the user from the smart speaker 200. The data transceiver 436 outputs the received audio data to the audio processing unit 431. The data transceiver 436 receives information indicating that a predetermined signal (a trigger signal, an auxiliary signal, or the like) is received, information indicating the time when the predetermined signal is received, and identification information of the home appliance 100 that transmitted the predetermined signal, from the smart speaker 200.

The data transceiver 436 acquires a control command for the home appliance 100 from the control command generator 434. The data transceiver 436 refers to the voice operation management DB422 to specify the device control server apparatus 500 corresponding to the home appliance 100 to which the user is to generate a voice. Next, the data transmission/reception unit 436 transmits the control command acquired from the control command generation unit 434 to the specified device control server apparatus 500. "identifying the device control server apparatus 500 corresponding to the home appliance 100 to which the user is to generate the voice, and transmitting the control command to the identified device control server apparatus 500" is another example of "performing a predetermined process by associating the voice generated by the user with the home appliance identified based on the identification information".

The data transceiver 436 acquires the audio data output from the smart speaker 200 from the response generator 435. The data transceiver 436 transmits the audio data acquired from the response generator 435 to the smart speaker 200.

[5. device control Server apparatus ]

Next, the device control server apparatus 500 will be explained.

Fig. 8 is a block diagram showing a functional configuration of the device control server apparatus 500. The device control server apparatus 500 includes, for example, a communication module 510, a storage unit 520, and an information processing unit 530.

The communication module 510 includes a high frequency circuit and the like. The communication module 510 can communicate with the home appliance 100 via the network NW and the router 300. The communication module 510 can communicate with the voice conversation server apparatus 400 via the network NW.

The storage unit 520 stores a control command DB521 and an information management DB 522. The control command DB521 stores various control commands for operating the home appliance 100. The information management DB522 stores information related to the home appliance 100. For example, when the home appliance 100 is a refrigerator, the information management DB522 stores stock information (presence or absence of stock, date of stock, expiration date, and the like) of food materials in the refrigerator. For example, when the household electrical appliance 100 is a washing machine, the pollen amount in the last 1 week or so is stored.

The information processing unit 530 includes, for example, a control command generating unit 531, an information collecting unit 532, an information selecting unit 533, and a data transmitting/receiving unit 534.

The control command generating unit 531 generates a control command corresponding to the home appliance 100 based on the control command received from the voice session server apparatus 400.

The information collection unit 532 collects information on the home appliance 100 stored in the information management DB 522. For example, the information collection unit 532 collects information (for example, stock information of a refrigerator) input by the user to the terminal device 600 and information provided from various websites connected to the network NW. The collected information is stored in the information management DB 522.

The information selecting unit 533 selects information desired to be reported from the smart speaker 200 to the user from the information registered in the information management DB 522.

The data transceiver 534 controls the communication module 510 to transmit and receive signals and data to and from the home appliance 100 and the voice conversation server apparatus 400. When receiving a control instruction for the home appliance 100 from the voice conversation server apparatus 400, the data transmission/reception unit 534 outputs the received control instruction to the control instruction generation unit 531. The data transceiver 534 transmits the control command generated by the control command generator 531 to the home appliance 100. The data transmitter/receiver 534 transmits the information selected by the information selector 533 to the voice conversation server apparatus 400.

[6. Overall flow of control ]

Next, the overall flow of control will be described.

Fig. 9 is a sequence diagram showing an example of the overall flow of control of the information processing system 1. First, the trigger signal generation unit 181 of the home appliance 100 monitors the presence or absence of occurrence of a predetermined situation. Next, when a predetermined situation occurs, the trigger signal generation unit 181 detects that the predetermined situation has occurred based on the detection result of the door opening/closing detection sensor 110, the operation on the operation reception unit R, and the like (S101).

Next, the trigger signal generating unit 181 of the home appliance 100 generates a trigger signal when detecting that a predetermined situation has occurred. The data transceiver 185 of the home appliance 100 transmits the generated trigger signal and the identification information of the home appliance 100 to the smart speaker 200 (S102). When the trigger signal is received from the home appliance 100, the activation determination unit 252 of the smart speaker 200 activates the audio dialogue function of the smart speaker 200 to put the smart speaker 200 in a standby state (S103). The data transmitting/receiving unit 254 of the smart speaker 200 transmits information indicating that the trigger signal is received from the home appliance 100, identification information of the home appliance 100 that transmitted the trigger signal, and the like to the voice conversation server apparatus 400.

Next, the sound receiving unit 251 of the smart speaker 200 determines whether or not the user has uttered the sound to the microphone 210 (S104). When the user does not generate a sound to the microphone 210 (no in S104), the sound receiving unit 251 determines whether or not a predetermined time (standby time) for which the smart speaker 200 remains in the standby state has elapsed (S105).

When the predetermined time has elapsed (yes in S105), the sound receiving unit 251 stops the sound dialogue function by pausing (S106). On the other hand, if the predetermined time has not elapsed (no in S105), the sound receiving unit 251 returns to the process of step S103.

When there is an utterance of the user with respect to the microphone 210 (S104: yes), the sound receiving section 251 receives the utterance of the user input to the microphone 210 as sound data. The data transmission/reception unit 254 transmits the audio data received by the audio reception unit 251 to the audio session server apparatus 400 (S107).

The voice processing unit 431 of the voice dialogue server 400 refers to the voice recognition DB421 or the like, and analyzes the voice data of the utterance of the user (S108). The control content specifying unit 433 of the voice conversation server apparatus 400 specifies the control content intended by the user for the home appliance 100 based on the voice data analyzed by the voice processing unit 431 and the like (S109).

The control instruction generating unit 434 of the voice conversation server apparatus 400 generates a control instruction for the home appliance 100 based on the control content determined by the control content determining unit 433. The data transmission/reception unit 436 of the voice conversation server apparatus 400 transmits the control instruction generated by the control instruction generation unit 434 to the device control server apparatus 500 (S110).

The control command generation unit 531 of the device control server apparatus 500 generates a control command for the home appliance 100 based on the control command received from the voice session server apparatus 400 (S111). The data transmission/reception unit 534 of the device control server apparatus 500 transmits the generated control command to the home appliance 100 (S112).

The data transmitting/receiving unit 185 of the home appliance 100 transmits a control response to the received control command to the appliance control server device 500 (S113). The control response is, for example, a response indicating whether or not the control command is normally notified to the home appliance 100. The device control unit 184 of the home device 100 controls the home device 100 based on the received control command (S114).

The data transmitter/receiver 534 of the device control server apparatus 500 transmits the control response received from the home appliance 100 to the voice session server apparatus 400 (S115). The response generation unit 435 of the voice conversation server apparatus 400 generates voice data to be output from the smart speaker 200 based on the control response received from the device control server apparatus 500 (S116). The data transmission/reception unit 436 of the voice conversation server apparatus 400 transmits the generated voice data to the smart speaker 200 (S117). The audio output unit 255 of the smart speaker 200 outputs audio corresponding to the control response from the speaker 220 based on the audio data received from the audio session server 400 (S118). The processes in S113 and S115 to S118 may be performed at the timing when the control of the home appliance 100 is ended.

[7. judgment processing of Sound-producing object ]

[7.1 basic treatment ]

Next, processing related to the determination of the sound emission target by the sound dialogue server device 400 will be described.

Fig. 10 is a flowchart showing a flow of processing related to the determination of the sound emission target. First, the voice receiving unit 251 of the voice dialogue server 400 acquires voice data emitted from the user to the smart speaker 200 (S201).

When the audio data is acquired by the audio receiving unit 251, the sound emission target determination unit 432 of the audio session server device 400 determines whether or not trigger signals have been received from the plurality of home electric appliances 100 before the audio data is acquired by the audio receiving unit 251 in the same standby time (S202).

When the trigger signal is received from only one home appliance 100 (S202: no), the sound emission target determination unit 432 determines that the home appliance 100 is a sound emission target of the user. In this case, the information processing unit 430 of the voice conversation server apparatus 400 associates the one home electric appliance 100 with the utterance of the user and performs predetermined processing. The "prescribed processing" includes, for example: a first process of determining the type of the home appliance 100 based on the identification information of the home appliance 100 and the voice operation management DB422, and performing the meaning interpretation of the utterance of the user based on the determined type of the home appliance 100; and a second process of specifying the device control server apparatus 500 corresponding to the home appliance 100 based on the identification information of the home appliance 100 and the voice operation management DB422, and transmitting a control instruction based on the sound emission content of the user to the specified device control server apparatus 500. Note that the "predetermined process" referred to in the present specification may be at least one of the first process and the second process, or may be another process different from the first process and the second process. Even when the trigger signal is received from only one of the home appliances 100, the sound-emission target determination unit 432 may perform the determination of the affinity described later, and when the affinity is lower than the reference, it may determine that the sound emission of the user is a sound emission unrelated to the control of the home appliance 100, and does not perform any special processing.

On the other hand, when the trigger signal is received from the plurality of home appliances 100 (yes in S202), the control content specifying unit 433 determines which home appliance 100 of the plurality of home appliances 100 is the sound emission target. In the present embodiment, the control content specifying unit 433 gives higher priority to the correlation between the home appliance 100 and the sound emission as the home appliance 100 that is closer to the time when the trigger signal is received. Therefore, the control content specifying unit 433 first determines whether or not the home appliance 100 that has transmitted the trigger signal last among the plurality of home appliances 100 has an affinity with the sound content of the user equal to or greater than a reference (equal to or greater than a threshold) (S204). The phrase "the affinity is not less than the reference (or the affinity is less than the reference)" means that the value indicating the affinity is not less than the reference (or the value indicating the affinity is less than the reference). The "value indicating affinity" is, for example, "a value set in advance according to the affinity between the type of the home appliance and the content of the utterance" or "a value set in advance according to the affinity between the type of the operation performed on the home appliance and the content of the utterance".

Here, for example, the content of the sound of "freezing" or "cooling" is the content that matches the basic operation of refrigerator 100A, and therefore it is determined that the affinity with refrigerator 100A is equal to or higher than the reference. On the other hand, since the contents of "freezing" and "cooling" do not match the basic operations of microwave oven 100B and washing machine 100C, it is determined that the affinity with microwave oven 100B and washing machine 100C is lower than the reference. On the contrary, the contents of the "heating" or "baking" and "500W" are the same as the basic operation of the microwave oven 100B, and therefore, the affinity with the microwave oven 100B is determined to be equal to or higher than the reference. On the other hand, since the contents of the sounds "heating" or "baking" and "500W" do not match the basic operations of the refrigerator 100A and the washing machine 100C, it is determined that the affinity with the refrigerator 100A and the washing machine 100C is lower than the reference.

When the affinity of the home appliance 100 with the sound content of the user is equal to or greater than the reference (yes in S204), the sound emission target determination unit 432 determines that the home appliance 100 is the sound emission target of the user. In this case, the information processing unit 430 of the voice conversation server apparatus 400 associates the home appliance 100 determined as the sound generation target of the user with the sound generation of the user and performs the predetermined processing described above (S205).

On the other hand, if it is determined that the affinity is lower than the reference (no in S204), the sound emission target determination unit 432 then determines whether or not the affinity between the home appliance 100 that transmitted the trigger signal and the sound emission content of the user is equal to or higher than the reference (S206). When the affinity of the home electric appliance 100 with the sound content of the user is equal to or greater than the reference (yes in S206), the sound emission target determination unit 432 determines that the home electric appliance 100 is the sound emission target of the user. In this case, the information processing unit 430 of the voice conversation server apparatus 400 associates the home appliance 100 determined as the sound emission target of the user with the sound emission of the user and performs the predetermined processing (S207).

When receiving the trigger signals from the N home appliances 100 (N is an integer equal to or greater than 1), the sound emission target determination unit 432 repeats the determination process for the N home appliances 100 (S208). When the affinity of the home electric appliance 100 with the sound content of the user is equal to or greater than the reference (yes in S208), the sound emission target determination unit 432 determines that the home electric appliance 100 is the sound emission target of the user. In this case, the information processing unit 430 of the voice conversation server apparatus 400 associates the home appliance 100 determined as the user 'S utterance target with the user' S utterance and performs the predetermined processing (S209).

Then, the response generation unit 435 generates sound data for reporting the determined control content (control content scheduled to be executed) to the user, based on the control content for the home appliance 100 determined by the meaningful interpretation according to the type of the home appliance 100 (S210). Next, the data transceiver 436 transmits the audio data generated by the response generator 435 to the smart speaker 200, and causes the smart speaker 200 to report the determined control content. Thereby, for example, a sound of "start heating by microwave oven" is output from the smart speaker 200. By performing such a report, even if the inference in the voice dialogue server device 400 is erroneous, it is easy to avoid control against the intention of the user.

Alternatively, instead of the above-described processing, the response generation unit 435 may generate sound data for confirming to the user whether or not there is no problem with the determined control content. The data transceiver 436 transmits the sound data generated by the response generator 435 to the smart speaker 200, and causes the smart speaker 200 to output a sound for confirming to the user whether the determined control content has no problem. Thus, for example, "start heating by microwave oven? "this sound. By performing such a report, even if the inference in the voice dialogue server device 400 is erroneous, it is easy to avoid control against the intention of the user.

On the other hand, if the home appliance 100 having the affinity with the utterance of the user equal to or greater than the reference is not found even if the above determination process is repeated for N home appliances 100 (no in S208), the utterance target determination unit 432 determines that the utterance of the user is an utterance unrelated to the control of the home appliance 100, and does not perform any special process. This makes it possible to avoid processing that is not intended by the user when the user has issued content unrelated to the home appliance 100.

[7.2 other treatments ]

Next, several other processes will be explained. These processes may be executed in combination with the basic processes described above, or may be executed instead of the basic processes described above.

[7.2.1 treatment 1]

For example, as a principle, the more the home appliance 100 that transmits the trigger signal later, the higher the priority order of the sound emission of the user with respect to the home appliance 100 is. Here, when the smart speaker 200 receives the first auxiliary signal (signal for increasing the priority order) from the home appliance 100, the process of increasing the priority order of the home appliance 100 that transmitted the first auxiliary signal is performed. For example, in a state where the trigger signals are received from the plurality of home appliances 100, the sound-emission target determination unit 432 preferentially determines the affinity from the home appliance 100 that has transmitted the first auxiliary signal. The sound emission target determination unit 432 may perform the predetermined process by associating the home appliance 100 that has transmitted the first auxiliary signal last with the sound emission of the user without performing the determination of the affinity.

A specific example will be explained. For example, the microwave oven 100B transmits a trigger signal when the door 102 is opened or closed, and transmits a first assist signal when a mode selection button (a button for selecting an oven mode, a microwave oven mode, or the like) of the operation receiving unit R is pressed but a heating start button of the operation receiving unit R is not pressed within a predetermined time thereafter (an example of a first specific situation). In a certain scenario, it is assumed that, in a state where door 102 of microwave oven 100B is opened and closed, and the mode selection button of microwave oven 100B is pressed but then the heating start button is not pressed, door 102 of refrigerator 100A is opened and closed, and a trigger signal is transmitted from refrigerator 100A, and then, a user makes a sound. In this case, the home appliance 100 that has transmitted the trigger signal last is the refrigerator 100A, but the sound-emission target determination unit 432 prioritizes association between the microwave oven 100B that has transmitted the first auxiliary signal and the sound emission of the user over association between the refrigerator 100A and the sound emission of the user.

With such a configuration, for example, even when the operation mode of microwave oven 100B is selected, the heating time is not clear, the description of the package of the food stored in refrigerator 100A is confirmed, and the heating time is designated by sound emission, it is possible to appropriately associate the sound emission of home appliance 100 with the sound emission of the user. The first specific state is, for example, a state in which the intention of the user who operates the home appliance 100 is estimated but the operation of the home appliance 100 has not yet started. The "mode selection button is pressed" is an example of a phenomenon in which the intention of the user to operate the home appliance 100 is inferred.

[7.2.2 treatment 2]

For example, as a principle, the more the home appliance 100 that transmits the trigger signal later, the higher the priority order associated with the utterance content of the user is, the uttered object determination unit 432 increases. Here, when the smart speaker 200 receives the second auxiliary signal (signal for lowering the priority order) from the home appliance 100, the process of lowering the priority order of the home appliance 100 that transmitted the second auxiliary signal is performed. For example, in a state where the trigger signals are received from the plurality of home appliances 100, the sound-emission target determination unit 432 delays the determination of the affinity of the home appliance 100 that has transmitted the second auxiliary signal. The sound emission target determination unit 432 may remove the home appliance 100 that transmitted the second auxiliary signal from the determination target of the affinity.

A specific example will be described. For example, it is assumed that microwave oven 100B transmits a trigger signal when door 102 is opened and closed, and transmits a second assist signal when a heating start button of operation receiving unit R is pressed (an example of a second specific situation). In a certain scenario, it is assumed that the door 102 of the refrigerator 100A is opened and closed to transmit the trigger signal from the refrigerator 100A, then the door 102 of the toaster microwave 100B is opened and closed to transmit the trigger signal from the toaster microwave 100B, and then the heating start button of the toaster microwave 100B is pressed to transmit the second auxiliary signal. In this case, although the home appliance 100 that has transmitted the trigger signal last is the microwave oven 100B, the sound-emission target determination unit 432 determines the affinity between the refrigerator 100A and the sound-emission content of the user first by lowering the priority of the microwave oven 100B that has transmitted the second auxiliary signal.

With such a configuration, even when an operation related to sound emission of refrigerator 100A is delayed, home appliance 100 can be appropriately associated with sound emission of the user. The second specific state is, for example, a state in which the operation of the home appliance 100 has been started according to the user's intention. The "heating start button" is an example of an operation unit that starts the operation of the home appliance 100 according to the intention of the user.

[7.2.3 treatment 3]

Fig. 11 is a diagram showing an example of the content of the determination rule of the sound emission target determination unit 432. Fig. 11 shows a case where the utterances include a language having an affinity equal to or higher than a reference for each of the plurality of home appliances 100. In such a case, the sound emission target determination unit determines the sound emission target according to one or more determination rules of (a) to (E) shown below.

(A) When only the initial sound generation is instructed by a non-specific content such as "after rakat" or "after appropriate processing", the sound generation object determination unit 432 preferentially associates the sound generation of the home appliance 100 and the user with each other by giving priority to the home appliance 100 that transmitted the trigger signal immediately before (last).

(B) Even in the case where only the initial sound emission is instructed by unspecified contents such as "after-bailout" and "after-treatment", the sound emission target judgment section 432 gives priority to the home appliance 100 (for example, the microwave oven 100B) whose basic function is operated only when operated over the home appliance 100 (for example, the refrigerator 100A) whose basic function is always operated, and associates the sound emission of the home appliance 100 and the user. For example, even when refrigerator 100A is opened or closed after toaster 100B is opened or closed, toaster 100B is prioritized to associate home appliance 100 with sound production.

(C) When only a sound generation instructing power saving or energy saving is performed, such as the "energy saving mode", sound generation target determination unit 432 preferentially associates the sound generation of home appliance 100 and the user with the sound generation of the home appliance 100, which has transmitted the trigger signal immediately before (last).

(D) If it is not clear which space (for example, a refrigerating compartment, a freezing compartment, or an ice-making compartment) is indicated in home appliance 100 (for example, refrigerator 100A) as "freezing" or "quick freezing", sound emission target determination unit 432 associates the sound emission of the user with the space that was operated (opened or closed) immediately before.

(E) When only an emergency sound emission is instructed, such as "fast-forward", the sound emission target determination unit 432 preferentially associates the sound emission of the home appliance 100 and the user with each other by giving priority to the home appliance 100 that transmitted the trigger signal immediately before (last).

[7.2.4 treatment 4]

For example, the home appliance 100 includes a sensor capable of detecting whether or not there is something in the space of the housing 101 (for example, the storage room of the refrigerator 100A, the heating room of the oven microwave 100B, and the inside of the washing tub of the washing machine 100C). When the sensor detects that there is nothing (empty) in the space of the casing 101, the trigger signal generating unit 181 of the home appliance 100 may output information indicating that there is no content in the space of the casing 101 to the voice conversation server apparatus 400 via the smart speaker 200. When receiving information indicating that the content does not exist in the space in the casing 101, the sound emission target determination unit 432 of the voice conversation server apparatus 400 lowers the priority of the association between the home appliance 100 that has transmitted the information and the sound emission of the user.

In addition, when the sensor detects that the space is filled with something from a state where nothing (empty) is not present in the space of the casing 101, the trigger signal generation unit 181 of the home appliance 100 may output information indicating that the space of the casing 101 is newly filled with something to the voice dialogue server apparatus 400 via the smart speaker 200. When receiving information indicating that a new item is inserted into the space of the casing 101, the sound emission target determination unit 432 of the voice conversation server apparatus 400 may increase the priority of the association between the home appliance 100 that has transmitted the information and the sound emission of the user.

[7.2.5 treatment 5]

When the timing at which the trigger signal is transmitted and the timing at which the user utters the voice are within a predetermined time (for example, when the trigger signal is transmitted immediately before the timing at which the user utters the voice), the utterance target determination unit 432 of the voice conversation server apparatus 400 may associate the home appliance 100 that transmitted the trigger signal with the utterance of the user.

[8. advantage ]

As a comparative example, a case is considered in which it is necessary to trigger the utterance of a word in order to activate the voice dialogue function of the smart speaker 200. In this case, the user needs to memorize the trigger word and must issue the trigger word each time the voice dialogue function of the smart speaker 200 is activated.

On the other hand, in the present embodiment, when a predetermined situation is detected, the control unit 180 of the home appliance 100 transmits a trigger signal for activating the voice dialogue function of the smart speaker 200 to the smart speaker 200. With such a configuration, it is not necessary to memorize the trigger word and issue the trigger word. This can improve user convenience.

In the present embodiment, the control unit of the home appliance 100 transmits a trigger signal when the trigger-dedicated operation unit 130 is operated. With this configuration, the user can transmit the trigger signal at an arbitrary timing. This can further improve the convenience of the user. For example, consider the case where the smart speaker 200 is configured in a kitchen. The user performs housework in order to prepare food. When the user checks what to do with refrigerator 100A, the user operates trigger-dedicated operation unit 130 of refrigerator 100A to open door 102 of refrigerator 100A. In view of this, if seeing beef placed in the refrigerator 100A and emitting "a recipe teaching me beef", the smart speaker 200 inquires of the voice dialog server apparatus 400 to have it provide the recipe of beef. By using this, it is possible to perform a sound operation without a lot of effort while performing housework.

In the present embodiment, when the door 102 is opened with respect to the opening 101a of the housing 101, the control unit 180 of the home appliance 100 transmits a trigger signal. With such a configuration, the voice dialogue function of smart speaker 200 can be activated without generating a trigger word when a user puts something into home appliance 100. As an example, if the voice dialogue function is activated at a timing when the door 102 of the refrigerator 100A is opened, the user can register the stock state of the refrigerator 100A by voice dialogue with the smart speaker 200 while confirming the inside of the refrigerator 100A.

In the present embodiment, when the door 102 is closed with respect to the opening 101a of the housing 101, the control unit 180 of the home appliance 100 transmits a trigger signal. With such a configuration, the voice dialogue function of the smart speaker 200 can be activated without generating a trigger word after the operation on the inside of the home appliance 100 is completed. As an example, if the audio dialogue function is activated at a timing when the door 102 of the refrigerator 100A is closed, the user can give an instruction to quickly freeze the refrigerator 100A through the audio dialogue with the smart speaker 200. For example, consider a case where the smart speaker 200 is set in a washroom. The power of the washing machine 100C is turned on, the laundry is put into the tub, and the door 102 is closed. At this time, a trigger signal is transmitted from the washing machine 100C to the smart speaker 200. Here, if the sound is "cleaning the carpet", the smart speaker 200 inquires of the voice dialogue server 400 (or the appliance control server 500) to transmit a control command of the carpet cleaning mode to the washing machine 100C.

In the present embodiment, when the presence of a person is detected by the person sensing sensor 120, the control unit 180 of the home appliance 100 transmits a trigger signal. With such a configuration, the user can control the home appliance 100 through a voice conversation with the smart speaker 200 without any trouble, simply by approaching the home appliance 100. For example, consider a case where the smart speaker 200 is provided in a kitchen. The refrigerator 100A is mounted with a human perception sensor 120. In case it is detected that the user comes in front of the refrigerator 100A in the kitchen, the refrigerator 100A transmits a trigger signal to the smart speaker 200. If the user utters "teach me recommended recipes", the smart speaker 200 queries the voice dialog server device 400 to provide the recommended recipes.

In the present embodiment, when the power operation unit 140 is operated (for example, when an operation to turn the power on is performed), the control unit 180 of the home appliance 100 transmits a trigger signal. With this configuration, the mode setting after power-on and the like can be performed through a voice dialogue with the smart speaker 200 without a lot of effort. For example, there are home appliances 100 such as a washing machine 100C and a microwave oven 100B that are normally powered off. When the power operation unit 140 of the washing machine 100C is operated in the home appliance 100, or when the door 102 of the toaster microwave 100B is opened and the power is turned on, the home appliance 100 transmits a trigger signal to the smart speaker 200. By the user saying here a laundry mode, a recipe, etc., a voice operation can be simply performed. The control unit 180 of the home appliance 100 may transmit a trigger signal when the power supply is turned off from the on state. With such a configuration, even when the reservation setting of the operation of the home appliance 100 is forgotten, the operation can be performed through a voice conversation in a state where the power of the home appliance 100 is turned off.

Here, it is assumed that there are a behavior in which the user wants to activate the voice dialogue function and a behavior in which the user does not want to activate the voice dialogue function depending on the manner, habit, and the like of the operation. In view of this, in the present embodiment, the control unit 180 of the home appliance 100 changes the setting as to whether or not the trigger signal is transmitted for one or more situations, based on at least one of the operation reception unit R of the home appliance 100 by the user and the operation of the terminal device 600 independent of the home appliance 100 by the user. With this configuration, the operation and situation of transmitting the trigger signal can be set arbitrarily according to the convenience of use of the user. Further, if the setting can be performed by the user operating the terminal device 600 independent of the home appliance device 100, the setting can be performed even when the home appliance device 100 does not include a user interface for displaying a screen.

In the present embodiment, when a predetermined situation is detected, the control unit 180 of the home appliance 100 transmits information used for identifying the appliance control server device 500 that controls the home appliance 100, in addition to the trigger signal. With this configuration, the user can operate the home appliance 100 by voice without uttering a word for specifying the home appliance 100. This can further improve the convenience of the user.

In the present embodiment, when a trigger signal is transmitted from the home appliance 100, the information processing unit 250 of the smart speaker 200 activates the voice dialogue function of the smart speaker 200 disposed independently of the home appliance 100. With this configuration, the voice dialogue function of the smart speaker 200 can be activated without generating a predetermined trigger word. This can improve user convenience.

(second embodiment)

Next, a second embodiment will be explained. In the present embodiment, 1 home appliance 100 (hereinafter, referred to as a voice interactive home appliance 100V) disposed in the home of the user has an audio interactive function. The second embodiment is an example in which a trigger signal described later is transmitted from the home appliance 100 to the voice-interactive home appliance 100V. The configuration other than the configuration described below is the same as that of the first embodiment.

Fig. 12 is a diagram showing the overall configuration of the information processing system 1 according to the second embodiment. The information processing system 1 includes, for example, one or more (for example, a plurality of) home appliances 100, a voice conversation home appliance 100V, a router 300 in a home, a voice conversation server apparatus 400, an apparatus control server apparatus 500, and a terminal apparatus 600. The voice conversation server apparatus 400 and the device control server apparatus 500 may be constituted by 1 server apparatus.

The voice dialog home device 100V is an example of an "external device" that is viewed from the home device 100. The audio dialogue home appliance 100V has a voice dialogue device 200 in addition to the same functions as those of the normal home appliance 100 (for example, a function of a refrigerator, a function of an oven or a function of a washing machine). The voice dialogue device 200 may have the same function as the smart speaker 200 according to the first embodiment, for example, or may be a dedicated voice dialogue device for operating the home appliance 100.

The first and second embodiments have been described above, but the embodiments are not limited to the above examples. For example, in the first and second embodiments, the control command for the home appliance 100 is transmitted from the appliance control server device 500 to the home appliance 100, but the present invention is not limited to this. The control command for home appliance 100 may be transmitted from device control server apparatus 500 (or voice session server apparatus 400) to smart speaker 200 or voice session home appliance 200, or transmitted from smart speaker 200 or voice session home appliance 200 to home appliance 100. The ordinal numbers such as "first" and "second" in the present specification are for convenience of description, and may be modified as appropriate.

The storage unit 240 of the smart speaker 200, the storage unit 420 of the audio session server device 400, and the storage unit 520 of the device control server device 500 are configured by one or more of, for example, a semiconductor memory chip, ssd (solid State drive), hdd (hard Disk drive), or the like.

The control unit 180 of the home appliance 100, the control unit 250 of the smart speaker 200, the information Processing unit 430 of the audio session server apparatus 400, and the information Processing unit 530 of the appliance control server apparatus 500 are realized by executing a program (software) by a hardware processor such as a cpu (central Processing unit), for example. However, all or a part of each of the control unit 180, the control unit 250, the information processing unit 430, and the information processing unit 530 may be realized by hardware (circuit unit including circuit) such as asic (application Specific Integrated circuit), pld (Programmable Logic device), fpga (field Programmable Gate array), or the like, or may be realized by cooperation of software and hardware.

According to at least one embodiment described above, the home appliance includes the control unit that transmits the first signal for activating the voice dialogue function of the external device to the external device or the server device, thereby improving user convenience.

Several embodiments of the present invention have been described, but these embodiments are merely illustrative and are not intended to limit the scope of the present invention. These embodiments can be implemented in other various ways, and various omissions, substitutions, and changes can be made without departing from the spirit of the invention. These embodiments and modifications are included in the invention described in the claims and the equivalent scope thereof as long as the scope and gist of the invention are included.

Description of reference numerals:

1 … information processing system, 100 … home electric appliance, 100V … voice dialogue home electric appliance (external device), 101 … casing, 101a … opening, 102 … door, 110 … door opening/closing detection sensor, 120 … human perception sensor, 130 … trigger special operation part, 140 … power supply operation part, 150 … display/operation part, R … input reception part, 180 … control part, 200 … smart speaker (external device), 250 … control part (information processing part), 400 … voice dialogue server device, 430 … information processing part, 500 … home electric appliance control server device, 530 … information processing part.

Claims

1. A household electrical appliance, characterized in that,

the server device includes a control unit that transmits a first signal for activating a voice dialogue function of an external device to the external device or the server device when a predetermined situation is detected.

2. The home device of claim 1,

further comprises a trigger operation unit for receiving an operation for starting the voice conversation function,

the control section transmits the first signal when the trigger operation section is operated.

3. The household appliance according to claim 1 or 2,

further comprises a housing having an opening and a door for closing the opening,

the control portion transmits the first signal in a case where the door is opened with respect to the opening.

4. The household appliance according to any one of claims 1 to 3,

the control portion transmits the first signal in a case where the door is closed with respect to the opening.

5. The household appliance according to any one of claims 1 to 4,

and a human perception sensor is also provided,

the control unit transmits the first signal when the presence of a person is detected by the person sensing sensor.

6. The household appliance according to any one of claims 1 to 5,

and a power supply operation part is also provided,

the control section transmits the first signal in a case where the power source operation section is operated.

7. The household appliance according to any one of claims 1 to 6,

the control unit changes a setting as to whether or not the first signal is transmitted in one or more situations, based on at least one of an operation of the operation reception unit by the user with respect to the home appliance and an operation of the terminal device independent of the home appliance by the user.

8. The household appliance according to any one of claims 1 to 7,

when the predetermined condition is detected, the control unit transmits, in addition to the first signal, information usable for specifying a server device that controls the home appliance.

9. The household appliance according to any one of claims 1 to 8,

when the predetermined condition is detected, the control unit transmits information usable for specifying the type of the home appliance, in addition to the first signal.

10. The household appliance according to any one of claims 1 to 9,

when the home appliance is in a first specific state, the control unit transmits a second signal for associating and processing a sound emission of the external device with the home appliance or for increasing a priority order for associating and processing the sound emission with the home appliance to the external device or the server apparatus.

11. The household appliance according to any one of claims 1 to 10,

when the home appliance is in a second specific state, the control unit transmits a third signal for causing the user to associate the sound generated by the external device with the home appliance or for lowering the priority for associating the sound generated by the user with the home appliance and performing processing, to the external device or the server device.

12. The household appliance according to any one of claims 1 to 11,

the external device ends the voice dialogue function when the voice dialogue function is activated based on a trigger word uttered by the user and when the user utters no voice for a first predetermined time,

when the predetermined condition is detected, the control unit transmits, to the external device or the server apparatus, an additional signal for maintaining the activated state of the voice dialogue function for a second predetermined time longer than the first predetermined time even when no user utters a voice.

13. An information processing system characterized in that,

the information processing unit activates a voice dialogue function of a device disposed independently of a home appliance when a first signal is transmitted from the home appliance.

14. The information processing system according to claim 13,

the information processing unit, when receiving the first signal and identification information usable to identify the home appliance, associates and performs a predetermined process on the utterance of the voice dialogue function and the home appliance identified by the identification information.

15. The information processing system according to claim 14,

when the user utters the voice dialogue function after receiving the first signal from the plurality of home appliances, the information processing unit associates the home appliance that has transmitted the first signal last among the plurality of home appliances with the utterance and performs the predetermined processing.

16. The information processing system according to claim 14 or 15,

the predetermined processing includes processing for analyzing the content of the utterance based on the type of the home appliance.

17. The information processing system according to any one of claims 13 to 16,

when the user utters the voice dialogue function after receiving the first signal from the plurality of home appliances, the information processing unit determines whether or not affinity between the home appliance that has transmitted the first signal last and content of the utterance is equal to or greater than a reference, and when the affinity is equal to or greater than the reference, associates the home appliance that has transmitted the first signal last with the content of the utterance and performs predetermined processing.

18. The information processing system according to claim 17,

when the affinity between the home appliance that has transmitted the first signal last among the plurality of home appliances and the content of the utterance is lower than the reference, the information processing unit determines the affinity between the home appliance that has transmitted the first signal last among the plurality of home appliances and the content of the utterance.

19. The information processing system according to any one of claims 13 to 18,

the information processing unit, when receiving the first signal and identification information usable for identifying the home appliance, reports information about the home appliance identified based on the identification information by the voice dialogue function.