WO2018041711A1

WO2018041711A1 - Interactive control device

Info

Publication number: WO2018041711A1
Application number: PCT/EP2017/071337
Authority: WO
Inventors: Arne Rost; Cristina Rico Garcia
Original assignee: BSH Hausgeräte GmbH
Priority date: 2016-08-31
Filing date: 2017-08-24
Publication date: 2018-03-08
Also published as: DE102016216409A1

Abstract

The invention relates to a device for controlling an interaction between a user and a household appliance, comprising a base section; an optical reference mark, which is movable relative to the base section and which is designed to give the user the optical impression of an eye; a camera; an acoustic input device; an acoustic output device; an interface for connection to the household appliance; and a processing device. The processing device according to the invention is designed to detect the start of a natural language dialog between a user and the device; to capture an image of the user by means of the camera; to align the reference mark to the user during the dialog; and to control the household appliance by means of the interface on the basis of the dialog.

Description

Interactive control device

The invention relates to a device for operating a household appliance. In particular, the invention relates to an interactive device adapted to maintain a dialogue with a user.

State of the art

A home appliance is adapted to perform a predetermined task within a household. For example, the domestic appliance may comprise a vacuum cleaner, a stove or an extractor hood. Some home appliances can operate autonomously and rarely require input or output. Other home appliances must be interactively controlled. For example, a vacuum cleaner may be autonomous, while a cooker usually needs to be operated by a user during cooking. The operation of a domestic appliance can be difficult if, for example, the user has dirty hands, is under time pressure or attempts to concatenate several processes involving different household appliances. One option for advanced home appliance control includes voice input. The household appliance may include a processing device that is configured to implement a natural-language instruction of the user into a control function of the domestic appliance. Similarly, for example, an operating state of the domestic appliance by means of a corresponding processing device as a spoken text, just natural language, are output. Nevertheless, the interaction between the user and the home appliance is often difficult. For example, it is not always clear when the home appliance is expecting an input, whether it has understood the input, or is performing an activity that corresponds to the activity intended by the user. US 2010/0023163 A1 proposes to equip a robot to assist a user in the observance of predetermined nutritional habits in such a way that eye contact with the user can be simulated. It is an object of the present invention to provide a technique that allows for improved interaction between a user and a home appliance. The invention solves this problem by means of the subjects of the independent claims. Subclaims give preferred embodiments again.

Disclosure of the invention

A device according to the invention for controlling an interaction between a user and a household appliance comprises a base section; an optical reference mark movable relative to the base portion and arranged to convey to the user the visual impression of an eye; a camera; an acoustic input device; an acoustic output device; an interface for connection to the household appliance and a processing device. The processing means is adapted to detect the beginning of a natural language dialogue between a user and the device; to take a picture of the user by means of the camera; align the reference mark to the user during the dialog; and control the home appliance via the interface based on the dialogue.

The device may allow to visually support and design the dialogue with the user. As a result, the dialogue can be made more intuitive and the teaching of certain dialog patterns by the user can be omitted. By aligning the reference mark with the user, it can automatically give the impression of attention, so that he knows that he is in a dialogue. The user can then improve an instruction that can be processed by the device. Conversely, turning away the reference mark from the user can signal to the user that no dialogue is currently taking place. This allows an immediate feedback to the user that a possible linguistic utterance is not processed by him and is not interpreted as an instruction. The feeling of surveillance therefore can not be adjusted by the user.

Preferably, the processing device is further configured to capture an eye of the user based on the image and to align the reference mark with the user's eye. This can increase the impression of eye contact between the device and the user. In further embodiments, the dialogue can be additionally solidified by the way the reference mark is displayed. For example, the visual impression of the eye can be changed in such a way that simple feelings such as attention, absence, concentration, etc. are displayed in a supportive manner.

In yet another embodiment, the apparatus includes a head portion that is movable in at least one dimension relative to the base portion and to which the reference mark is attached. In this case, the processing device is configured to move the head section relative to the base section and the reference mark relative to the head section in order to align the reference mark with the user. Optionally, the reference mark is aligned with an eye of the user, as explained above. Through the head section, the impression of an understanding being with the user can increase, so that, for example, reservations of the user with respect to a technical device can not tarnish the dialogue.

In a first variant, the reference mark is formed three-dimensionally and a drive device is provided for moving the reference mark. A realistic representation of an eye can be used so easily to realize the above-described fixation on the user. A graphical display is not required. If, nevertheless, a graphical display device is provided, it can be used to output other information, in particular visual or textual information.

In another variant, the device comprises a two-dimensional graphical pointing device, and the processing device is configured to display and align the reference mark on the display device. In this variant, increased flexibility in the representation of the reference mark can be achieved. In particular, simple feelings can be simulated better by adjusting the representation of the reference mark accordingly.

The acoustic input device may comprise a microphone array with the aid of which an acoustic position of the user can be determined. Preferably, the camera is then set up in a predetermined window around the particular acoustic one Position of the user to be pivoted to find a face of the user.

An apparatus for controlling an interaction between a user and a home appliance includes an optical reference mark arranged to convey to the user the visual impression of an eye. A method of controlling the device according to the invention comprises steps of detecting the beginning of a natural language dialogue between the user and the device; scanning an image of the user; aligning the reference mark to the user during the dialogue and controlling the home appliance based on the dialogue. The method according to the invention can be carried out in particular on the device described above. For this purpose, the device may have a processing device which comprises a programmable microcomputer or microcontroller. The method can be realized in particular as a computer program product with program code means. The method can be used to control one or more domestic appliances. It is preferred that the method additionally comprises converting a natural language input of the user into a machine-understandable signal and / or converting a machine-understandable signal into a user-friendly signal

natural language statement to the user. Thereby, the method can more effectively control the interaction between the user and a device, while a device function of a home appliance is machine-controlled in a known manner. In particular, the machine controller may include the exchange of status, command or request information.

The dialogue preferably comprises at least one natural language utterance of the user and a natural language utterance of the device, the two utterances being related to each other. For example, the user may issue a request, which, according to the method, returns an acknowledgment of the understood request. In another example, the user may request a device state of the home appliance, and according to the method, the device state is obtained and output in natural language.

The beginning of the dialogue can be detected after a user-spoken keyword has been captured. The keyword can be fixed and For example, a name of a device performing the method include. In another embodiment, the keyword may be predetermined by the user.

It is further preferred that the alignment of the reference mark with the user be canceled when the end of the dialogue has been determined. An impression of

Observed by the user can thereby be avoided. The improved delimitation of a dialog phase from a dialog-free phase can improve the user's overall interaction with the method or an executing device.

The dialogue is preferably determined to have ended when no natural language utterance has been exchanged with the user for longer than a predetermined period. The utterance can alternatively be made by the method or the executing device or by the user. In an alternative embodiment, the dialogue may also be determined to have ended when it emerges from the dialogue or sequence of natural language utterances.

In addition, the invention comprises a domestic appliance with a device according to the invention.

The invention has been explained with reference to a device, a domestic appliance and a method. Unless otherwise specified, features of one claim category are analogously applicable to the other claim categories.

Brief description of the figures

The invention will now be described in more detail with reference to the attached figures, in which:

Figure 1 is a schematic representation of a device for controlling an interaction between a user and a home appliance.

Fig. 2 shows the device in dialogue with a user;

3 shows an exemplary representation of an optical reference mark of the device; 4 shows exemplary positions of a pair of reference marks; and 5 illustrates a flow chart of a method for controlling the interaction of the device with a user. Detailed description of an embodiment

FIG. 1 shows a schematic representation of a device 100 for controlling an interaction between a user 105 and a domestic appliance 1 10. In the exemplary representation, two domestic appliances 110 are shown in the form of a cooker and an extractor hood. Both have a preferably wireless interface 15 for connection to the device 100. However, other domestic appliances 110 can also be connected to the device 100, in particular kitchen appliances or cleaning appliances. The device 100 comprises a processing device 120, which is connected to an acoustic input device 125, an acoustic output device 130 and a preferably wireless interface 135 for connection to one or more domestic appliances 110. The device 100 further comprises a base portion 140, which may be used, for example, to place the device 100 on a stand or to attach it to a wall, furniture, household appliance or other object. Elements of the device 100 may be received within the base portion 140. The device 100 preferably additionally comprises a head section 145 which is movably mounted in at least one dimension relative to the base section 140. For moving the head section 145 relative to the base section 140, a drive device 150 can be provided, which can be controlled by the processing device 120. For example, the head portion 145 may be pivotable about a vertical axis through the base portion 140. The head portion 145 may also be pivotable about a transverse axis to pitch relative to the base portion 140.

At the base portion 140, or at the head portion 145, if present, is mounted a movable optical reference mark 155 which is adapted to be perceived by the user 105 as an eye. The reference mark 155 may be formed plastically and be mechanically movable, for example, by means of a further drive device 150. In the illustrated embodiment, a preferably two-dimensional and graphical display device 160 is provided, which is connected to the processing device 120. On the display device 160, the reference mark 155 can be shown and preferably also moved. In addition, the device 100 includes a camera 165, which is preferably mounted close to the reference mark 155, and more preferably to the head portion 145.

FIG. 2 shows the device 100 in a further embodiment in dialogue with the user 105. By way of example, two reference marks 155 are provided here, which are preferably controlled in concert so that they are perceived by the user 105 as an eye pair.

The device 100 is configured to support an interaction between the user 105 and the home appliance 110 by supporting a natural language dialog between the user 105 and the device 100 by aligning the optical reference mark 155 with the user 105. The goal is to give the user 105 the impression of attention of the device 100 while the dialogue continues. For this purpose, the reference mark 155 can be controlled in particular so that it mimics the eye of a human interlocutor with whom the user 105 speaks. In a first step, the device 100 may determine, by means of the camera 165, a relative position or direction of the user 105 with respect to the device 100 and align the head portion 145 and / or the reference mark 155 with the user 105. In a refinement, an eye 170 of the user 105 may also be detected within the image of the camera 165, and the reference mark 155 may be specifically aligned with the eye 170. When the user 105 moves in space, the alignment of the reference mark 155 may follow the user 105. If the dialog is ended, the tracking can be aborted or the alignment can be destroyed.

It is further preferred that the device 100 conduct a natural language dialogue with the user 105. An utterance spoken by the user 105 may be scanned by the acoustic input device 125 and analyzed by the processing device 120. Depending on the result, for example, a request to one of the household appliances 1 10 via the interface 135 emitted become. A message of a domestic appliance 110, for example regarding its operating status, may be received via the interface 135 and converted into a natural language utterance by the processing means 120. Then the utterance is preferably delivered to the user via the acoustic output device 130 105 issued.

FIG. 3 shows an exemplary representation of optical reference marks 155 of the device 100 of the two preceding figures. The reference marks 155 can each be reproduced plastically or displayed on the display 160. The reference marks 155 are preferably set up to support the dialogue with the user 105. For this it is helpful if the device 100 can be perceived as a person or as a person-like machine. For example, the illustrated reference marks 155 are shown as a pair of female eyes. A natural language output by means of the acoustic output device 130 preferably uses a likewise female voice.

The reference mark 155 may have further details of a human eye, which details may be used either statically, to enhance the human-like impression, or dynamically, such as to express simple emotions. The illustrated reference mark 155 includes optical replicas of an eyeball 305, an iris 310, a pupil 315, an upper eyelid 320, a lower eyelid 325, upper and lower eyelashes 330, and an eyebrow 335. In other embodiments, some of the mentioned elements or Features be different or missing. It is also possible to stylistically only indicate features 305 to 335.

FIG. 4 shows exemplary positions of a pair of reference marks 155 of the device 100 of FIGS. 1 and 2. In alternative embodiments, only one reference mark 155 may be provided. In the illustrated embodiment, the visual impression of an eye is essentially made by displaying a black circle in a white circle. As a result, different viewing directions can already be simulated if the reference mark 155 is aligned with the user 105. The viewing directions can already express feelings, can be supported by the information transfer between the user 155 and the device 100.

In further embodiments, other elements of the eye or eye area described above, for example, with reference to FIG. 3, may be displayed for expressing a sensation.

In FIG. 4 a, the reference marks 155 are directed directly at the user 105. Viewing directions are indicated in FIGS. 4b and 4c, which approach the user 105 when the latter is laterally downwards (FIG. 4b) or laterally upwards (FIG. 4c) of the device 100 or the display device 160. In Fig. 4d, the dark circles in the upper half of the bright circles are covered and a horizontal line is seen to give the impression of semi-closed eyelids. As a result, the device 100 can convey a feeling of tiredness or lack of concentration. In Figure 4e the dark circles are completely missing, so that the impression of closed eyes can be established. It can thereby be clarified that a dialogue with the device 100 is not currently taking place - the device 100 "sleeps".

FIG. 5 shows a flowchart of an exemplary method 500 for controlling the interaction of the device 100 of FIGS. 1 and 2 with the user 105. The method 500 can be carried out in particular by means of the processing device 120.

In a step 502, it is detected that the user 105 has spoken a keyword. The keyword may include a name of the device 100. An indication of the keyword may be given in textual form on the device 100. In a step 504, it is checked whether the user 105 is already known. If this is the case, then in a step 506, a corresponding internal mark can be set, which is here called user_detectable by way of example. Subsequently, or if the user 105 is not known, it is determined in a step 508 from which direction the acoustic signal was received. If several acoustic input devices 125 are provided, an approximate direction of the user 105 can be determined on the basis of the stereo effect. In a step 510, the head section 145 is preferably moved in this direction in order to change an image section of the camera 165 so that it contains the user 105 as much as possible. In a step 512, it is checked whether a face can be recognized on an image of the camera 165. If this is the case, preferably in a step 514 a relative position of the user 105 is determined on the basis of the camera image. Subsequently, in a step 516, the reference mark 155 is aligned with the user 105 and, if possible, with an eye 170 of the user 105 detected in the image of the camera 165. In a following step 518, the user 105 is tracked by continuously determining his relative position to the device 100 and aligning the reference mark 155 and, if appropriate, the camera 165, in each case to the specific position.

In a step 520, an utterance of the user 105 is detected. If no utterance can be detected, then the method 500 may return to step 512. Otherwise, in step 522, the sound source of the utterance may be determined. In a step 524, the previously determined positions or orientations of the user 105 with respect to the device 100 may be fused together. The orientation of the reference mark 155 and / or the camera 165 - optionally with the assistance of the orientation of the header 145 - may be updated in a step 526. Thereafter, the method 500 may proceed to step 512.

If no face of the user 105 was detected in the camera image in step 512, it may be checked in a step 528 whether the internal mark user_detectable is set. If this is not the case, the method 500 can proceed to step 512. Otherwise, in a step 530, the camera 165 is preferably pivoted in a predetermined window around the specific acoustic position. Subsequently, it is checked in a step 532 whether a face has been recognized and in this case branches to step 514. Otherwise, it is checked in a step 534 whether the predetermined window has already been completely analyzed. If this is not the case, method 500 preferably proceeds to step 530. Otherwise, in step 536, an internal variable called c here may be set to 0. In a subsequent step 538, the user 105 can be asked, for example in the natural language way by means of the output device 130, or in an optical or textual manner by means of the display device 160. whether his face just can not be recognized by the camera 165. The variable c is incremented.

In a step 540, it is checked whether the user 105 has answered yes. In this case, the internal user_detectable flag is set to 0 in step 542, and method 500 proceeds to step 520. If no response of the user 105 can be determined, then a predetermined time may be waited in a step 544 before checking in a step 546 whether the variable c has the value 3. If this is not the case, then in step 538 the question is reissued to the user 105 and the variable c is again incremented. If the value of the variable c is 3 in step 546, the method 500 may end in step 548.

If it is determined in step 540 that the response of the user 105 is no, then in a step 550 the variable c is set to the value 0 and in a step 552 the user 105 is asked to repeat the keyword. The variable c is incremented. In a subsequent step 554, it is checked whether the user 105 the

Keyword has spoken. In this case, the method 500 preferably proceeds to step 508. Otherwise, a predetermined time may be waited in a step 556 before checking in a step 558 whether the variable c has the value 3. If this is not the case, then the method 500 proceeds to the step 552 and again asks the user 105 to speak the keyword. Otherwise, the method 500 may end in step 548.

reference numeral

100 device

105 users

1 10 domestic appliance

1 15 interface

120 processing device

125 acoustic input device

130 acoustic output device

135 interface

140 base section

145 head section

150 drive device

155 reference mark

160 display device

165 camera

170 user's eye

305 eyeball

310 Iris

315 pupil

320 upper eyelid

325 lower eyelid

330 eyelash

335 eyebrow

500 procedures

502 keyword recorded

504 users known?

506 put brand user_detectable

508 Determine acoustic direction of the user

510 Swivel head section, create camera image

512 face recognized? Determine rel. Position of the user from camera image Align reference mark

Track users

Capture user input

Determine sound source expression

Merge positions

Adjust alignment

Brand user_detectable set?

Move the camera in a predetermined window

Face recognized?

Searched the window completely?

c = o

Question to user

Evaluate answer

user_detectable = 0

waiting

c _{= 3?}

The End

c _{= 0}

Ask user to repeat keyword

Keyword spoken?

waiting

c _{= 3?}

Claims

1 . Apparatus (100) for controlling interaction between a user (105) and a home appliance (110), the apparatus (100) comprising:

a base portion (140);

- an optical reference mark (155) movable relative to the base portion (140) and adapted to convey to the user (105) the visual impression of an eye;

a camera (165);

an acoustic input device (125);

an acoustic output device (130);

- An interface (135) for connection to the household appliance (1 10); and

a processing device (120), which is set up to:

to detect the beginning of a natural language dialogue between the user (105) and the device (100);

- take a picture of the user (105) by means of the camera (165);

- align the reference mark (155) with the user (105) during the dialogue; and

- To control the home appliance (1 10) by means of the interface (135) based on the dialogue.

The apparatus (100) of claim 1, wherein the processing means (120) is further adapted to detect an eye (170) of the user (105) based on the image and reference the mark (155) to the eye (170). of the user (105).

The apparatus (100) of claim 1 or 2, further comprising a head portion (145) movable in at least one dimension relative to the base portion (140) and to which the reference mark (155) is attached, the processing means (120). adapted to move the head portion (145) opposite the base portion (140) and the reference mark (155) opposite the head portion (145) to align the reference mark (155) with the user (105). Device (100) according to one of the preceding claims, wherein the reference mark (155) is formed three-dimensionally and a drive device (150) for moving the reference mark (155) is provided.

The apparatus (100) of any of claims 1 to 3, further comprising a two-dimensional graphical display (160), wherein the processing means (120) is adapted to display and align the reference mark (155) on the display (160).

Device (100) according to one of the preceding claims, wherein the acoustic input device (125) comprises a microphone array with the aid of which an acoustic position of the user (105) can be determined.

Apparatus (100) according to the preceding claim, wherein the camera (165) is adapted to be pivoted in a predetermined window around the determined acoustic position of the user (105) to find a face of the user (105).

A method (500) for controlling a device (100) for controlling an interaction between a user (105) and a home appliance (110), the device (100) comprising an optical reference mark (155) adapted to the user (105) to give the visual impression of an eye, the method (500) comprising the steps of:

- detecting (502) the beginning of a natural language dialogue between the user (105) and the device (100);

- scanning (510) an image of the user (105);

- aligning (510) the reference mark (155) with the user (105) during the dialogue; and

- controlling the household appliance (1 10) based on the dialogue.

Method (500) according to one of the preceding claims, wherein the dialogue comprises at least one natural-language utterance of the user (105) and a

natural language utterance of the device (100) and the two utterances be in context with each other.

The method (500) of any one of the preceding claims, wherein detecting the beginning of the dialog comprises detecting (502) a keyword spoken by the user (105).

1 1. The method (500) of any one of the preceding claims, further comprising determining the end of the dialogue and deselecting the alignment of the reference mark (155) with the user (105).

The method (500) according to one of the preceding claims, wherein the dialogue is determined to have ended when no longer than a predetermined duration

natural language utterance was exchanged with the user (105).

13. domestic appliance (1 10) with a device (100) according to one of claims 1 to 7.