DE102016216409A1 - Interactive control device - Google Patents

Abstract

An apparatus for controlling an interaction between a user and a home appliance comprises a base portion; an optical reference mark movable relative to the base portion and arranged to convey the visual impression of an eye to the user; a camera; an acoustic input device; an acoustic output device; an interface for connection to the household appliance; and a processing device. The processing means is adapted to detect the beginning of a natural language dialogue between a user and the device; to take a picture of the user by means of the camera; align the reference mark to the user during the dialog; and control the home appliance via the interface based on the dialogue.

Description

The invention relates to a device for operating a household appliance. In particular, the invention relates to an interactive device adapted to maintain a dialogue with a user.

State of the art

A home appliance is adapted to perform a predetermined task within a household. For example, the domestic appliance may comprise a vacuum cleaner, a stove or an extractor hood. Some home appliances can operate autonomously and rarely require input or output. Other home appliances must be interactively controlled. For example, a vacuum cleaner may be autonomous, while a cooker usually needs to be operated by a user during cooking.

The operation of a domestic appliance can be difficult if, for example, the user has dirty hands, is under time pressure or tries to concatenate several processes involving different household appliances. One option for advanced home appliance control includes voice input. The home appliance may comprise a processing device which is set up to convert a natural language instruction of the user into a control function of the domestic appliance. Similarly, for example, an operating state of the domestic appliance by means of a corresponding processing device as a spoken text, just natural language, are output.

Nevertheless, the interaction between the user and the home appliance is often difficult. For example, it is not always clear when the home appliance is expecting an input, whether it has understood the input, or is performing an activity that corresponds to the activity intended by the user.

US 2010/0023163 A1 Proposes to equip a robot to help a user comply with given eating habits so that eye contact with the user can be simulated.

It is an object of the present invention to provide a technique that allows for improved interaction between a user and a home appliance. The invention solves this problem by means of the subjects of the independent claims. Subclaims give preferred embodiments again.

Disclosure of the invention

A device according to the invention for controlling an interaction between a user and a household appliance comprises a base section; an optical reference mark movable relative to the base portion and arranged to convey the visual impression of an eye to the user; a camera; an acoustic input device; an acoustic output device; an interface for connection to the household appliance and a processing device. The processing means is adapted to detect the beginning of a natural language dialogue between a user and the device; to take a picture of the user by means of the camera; align the reference mark to the user during the dialog; and control the home appliance via the interface based on the dialogue.

The device may allow to visually support and design the dialogue with the user. As a result, the dialogue can be made more intuitive and the teaching of certain dialog patterns by the user can be omitted. By aligning the reference mark with the user, it can automatically give the impression of attention, so that he knows that he is in a dialogue. The user can then improve an instruction that can be processed by the device. Conversely, turning away the reference mark from the user can signal to the user that no dialogue is currently taking place. This allows an immediate feedback to the user that a possible linguistic utterance is not processed by him and is not interpreted as an instruction. The feeling of surveillance therefore can not be adjusted by the user.

Preferably, the processing device is further configured to capture an eye of the user based on the image and to align the reference mark with the user's eye. This can give an enhanced impression of the visual contact between the device and the user. In further embodiments, the dialogue can be additionally solidified by the way the reference mark is displayed. For example, the visual impression of the eye can be changed in such a way that simple feelings such as attention, absence, concentration, etc. are displayed in a supportive manner.

In yet another embodiment, the apparatus includes a head portion that is movable in at least one dimension relative to the base portion and to which the reference mark is attached. It is the Processing means adapted to move the head portion relative to the base portion and the reference mark relative to the head portion to align the reference mark on the user. Optionally, the reference mark is aligned with an eye of the user, as explained above. Through the head section, the impression of an understanding being with the user can increase, so that, for example, reservations of the user with respect to a technical device can not tarnish the dialogue.

In a first variant, the reference mark is formed three-dimensionally and a drive device is provided for moving the reference mark. A realistic representation of an eye can be used so easily to realize the above-described fixation on the user. A graphic display is not required. If, nevertheless, a graphical display device is provided, it can be used to output other information, in particular visual or textual information.

In another variant, the device comprises a two-dimensional graphic display device, and the processing device is configured to display and align the reference mark on the display device. In this variant, increased flexibility in the representation of the reference mark can be achieved. In particular, simple feelings can be simulated better by adjusting the representation of the reference mark accordingly.

The acoustic input device may comprise a microphone array with the aid of which an acoustic position of the user can be determined. Preferably, the camera is then adapted to be pivoted in a predetermined window around the user's particular acoustic position to find a user's face.

An apparatus for controlling an interaction between a user and a home appliance includes an optical reference mark arranged to convey to the user the visual impression of an eye. A method of controlling the device according to the invention comprises steps of detecting the beginning of a natural language dialogue between the user and the device; scanning an image of the user; aligning the reference mark to the user during the dialogue and controlling the home appliance based on the dialogue.

The method according to the invention can be carried out in particular on the device described above. For this purpose, the device may have a processing device which comprises a programmable microcomputer or microcontroller. The method can be realized in particular as a computer program product with program code means. The method can be used to control one or more domestic appliances. It is preferred that the method additionally performs transforming a natural language input of the user into a machine intelligible signal and / or converting a machine intelligible signal to a natural language utterance to the user. Thereby, the method can more effectively control the interaction between the user and a device, while a device function of a domestic appliance is mechanically controlled in a known manner. In particular, the machine controller may include the exchange of status, command or request information.

The dialogue preferably comprises at least one natural language utterance of the user and a natural language utterance of the device, the two utterances being related to each other. For example, the user may issue a request, which, according to the method, returns an acknowledgment of the understood request. In another example, the user may request a device state of the home appliance, and according to the method, the device state is obtained and output in natural language.

The beginning of the dialogue can be detected after a user-spoken keyword has been captured. The keyword may be fixed and may include, for example, a name of a device performing the method. In another embodiment, the keyword may be predetermined by the user.

It is further preferred that the alignment of the reference mark with the user be canceled when the end of the dialogue has been determined. An impression of being observed by the user can thereby be avoided. The improved delimitation of a dialog phase from a dialog-free phase can improve the user's overall interaction with the method or an executing device.

The dialogue is preferably determined to have ended when no natural language utterance has been exchanged with the user for longer than a predetermined period. The utterance can alternatively be made by the method or the executing device or by the user. In an alternative embodiment, the dialogue may also be determined to have ended when it emerges from the dialogue or sequence of natural language utterances.

In addition, the invention comprises a domestic appliance with a device according to the invention.

The invention has been explained with reference to a device, a domestic appliance and a method. Unless otherwise specified, features of one claim category are analogously applicable to the other claim categories.

Brief description of the figures

The invention will now be described in more detail with reference to the attached figures, in which:

1 a schematic representation of a device for controlling an interaction between a user and a home appliance;

2 the device in dialogue with a user;

3 an exemplary representation of an optical reference mark of the device;

4 exemplary positions of a pair of reference marks; and

5 FIG. 5 is a flowchart of a method for controlling the interaction of the device with a user.

Detailed description of an embodiment

1 shows a schematic representation of a device 100 to control an interaction between a user 105 and a household appliance 110 , In the exemplary illustration are two home appliances 110 presented in the form of a stove and an extractor fan. Both have a preferred wireless interface 115 for connection to the device 100 , However, there may be other home appliances 110 with the device 100 be connected, in particular kitchen appliances or cleaning equipment.

The device 100 comprises a processing device 120 that with an acoustic input device 125 , an acoustic output device 130 and a preferably wireless interface 135 for connection to one or more domestic appliances 110 connected is. The device 100 further comprises a base portion 140 which can be used, for example, to the device 100 on a stand or on a wall, a piece of furniture, a household appliance 110 or another object. Elements of the device 100 can within the base section 140 be included. The device preferably comprises 100 in addition a head section 145 placed in at least one dimension opposite the base section 140 is movably mounted. To move the head section 145 opposite the base section 140 may be a drive device 150 be provided by the processing device 120 can be controlled. For example, the head section 145 around a vertical axis through the base section 140 be swiveling. The head section 145 may also be pivotable about a transverse axis, relative to the base portion 140 to pitch.

At the base section 140 , or, if present, at the head section 145 , is a movable optical reference mark 155 that is set up by the user 105 appropriate to be perceived as an eye. The reference mark 155 can be formed plastically and, for example by means of another drive device 150 be mechanically mobile. In the illustrated embodiment, a preferred two-dimensional and graphical display device 160 provided with the processing device 120 connected is. On the display 160 can be the reference mark 155 represented and preferably also moved. In addition, the device includes 100 a camera 165 that prefers close to the reference mark 155 and most preferably at the head portion 145 is appropriate.

2 shows the device 100 in a further embodiment in dialogue with the user 105 , Here are two reference marks by way of example 155 provided that preferably concerted be controlled such that they are by the user 105 be perceived as a pair of eyes.

The device 100 is set up to interact between the user 105 and the household appliance 110 by supporting a natural language dialogue between the user 105 and the device 100 by an alignment of the optical reference mark 155 on the user 105 supported. The goal is to the user 105 the impression of an attention of the device 100 as the dialogue continues. For this, the reference mark 155 in particular be controlled so that it mimics the eye of a human interlocutor with whom the user 105 speaks. In a first step, the device can 100 by means of the camera 165 a relative position or direction of the user 105 with respect to the device 100 determine and the head section 145 and / or the reference mark 155 on the user 105 align. In a refinement can be within the image of the camera 165 also an eye 170 the user 105 be captured and the reference mark 155 May be specific to the eye 170 be aligned. Moves the user 105 in space, so can the orientation of the reference mark 155 the user 105 consequences. If the dialog is ended, the tracking can be aborted or the alignment can be destroyed.

It is further preferred that the device 100 a natural language dialogue with the user 105 performs. One by the user 105 spoken utterance can by means of the acoustic input device 125 sampled and by means of the processing device 120 to be analyzed. Depending on the result, for example, a request for one of the household appliances 110 over the interface 135 to be sent out. A message of a household appliance 110 , for example, regarding its operating state, can via the interface 135 received and by the processing device 120 into a natural language utterance ("text to speech"). Subsequently, the utterance is preferred via the acoustic output device 130 to the user 105 output.

3 shows an exemplary representation of optical reference marks 155 the device 100 the two preceding figures. The reference marks 155 can each be modeled plastically or on the display device 160 being represented. The reference marks 155 are preferably set to dialogue with the user 105 to support. It is helpful if the device 100 can be perceived as a person or as a person-like machine. For example, the illustrated reference marks 155 shown as a female pair of eyes. A natural language output by means of the acoustic output device 130 preferably uses a female voice as well.

The reference mark 155 may have further details of a human eye, where the details may be used either statically, to enhance the human-like impression, or dynamically, such as to express simple emotions. The illustrated reference mark 155 includes optical replicas of an eyeball 305 , an iris 310 , a pupil 315 , an upper eyelid 320 , a lower eyelid 325 , upper and lower eyelashes 330 and an eyebrow 335 , In other embodiments, some of the elements or features may be otherwise designed or lacking. It is also possible characteristics 305 to 335 stylistic only hint.

4 shows exemplary positions of a pair of reference marks 155 the device 100 of the 1 and 2 , In alternative embodiments, only one reference mark can be used 155 be provided. In the illustrated embodiment, the visual impression of an eye is essentially made by displaying a black circle in a white circle. As a result, different viewing directions can already be simulated if the reference mark 155 on the user 105 is aligned. The viewing directions can already express feelings, thereby facilitating the transmission of information between the user 155 and the device 100 can be supported.

In further embodiments, other elements of eyes or an eye area, for example, the above with reference to 3 described to express a feeling.

In 4a are the reference marks 155 directly to the user 105 directed. In the 4b and 4c Gaze directions are hinted at the user 105 run when it is laterally down ( 4b ) or laterally above ( 4c ) of the device 100 or the display device 160 located. In 4d The dark circles in the upper half of the bright circles are covered and a horizontal line can be seen to create the impression of half-closed eyelids. This can be done by the device 100 a feeling of tiredness or lack of concentration can be conveyed. In 4e the dark circles are missing completely, so that the impression of closed eyes can adjust. This can be used to clarify that a dialogue with the device 100 currently not taking place - the device 100 "is sleeping".

5 shows a flowchart of an exemplary method 500 for controlling the interaction of the device 100 of the 1 and 2 with the user 105 , The procedure 500 can in particular by means of the processing device 120 be performed.

In one step 502 is detected that the user 105 has spoken a keyword. The keyword can be a name of the device 100 include. An indication of the keyword may be in textual form on the device 100 be specified. In one step 504 it checks if the user 105 already known. If this is the case, so in one step 506 a corresponding internal mark can be set, which is exemplified here user_detectable. Subsequently or if the user 105 is not known, in one step 508 determines from which direction the acoustic signal was received. Are several acoustic input devices 125 provided, based on the stereo effect, an approximate direction of the user 105 be determined.

In one step 510 is preferably the head portion 145 moved in that direction to one Image section of the camera 165 to change so that he is the user 105 possibly contains. In one step 512 will check if on a picture of the camera 165 a face can be recognized. If this is the case, it is preferred in one step 514 a relative position of the user 105 determined on the basis of the camera image. Subsequently, in one step 516 the reference mark 155 on the user 105 and further, if possible, on a picture in the camera 165 captured eye 170 the user 105 aligned. In a following step 518 becomes the user 105 tracked by its relative position to the device 100 continuously determined and the reference mark 155 and, if necessary, the camera 165 each aligned to the specific position.

In one step 520 becomes an utterance of the user 105 detected. If no statement can be made, the procedure may be 500 to the step 512 to return. Otherwise, in one step 522 the sound source of the utterance can be determined. In one step 524 can the previously determined positions or orientations of the user 105 with respect to the device 100 be merged with each other. The orientation of the reference mark 155 and / or the camera 165 - if necessary, with support for the orientation of the head section 145 - can in one step 526 to be updated. Subsequently, the process can 500 with the step 512 Continue.

Was in the step 512 no face of the user 105 detected in the camera image, so can in one step 528 checking if the internal user_detectable brand is set. If this is not the case, then the procedure can 500 with the step 512 Continue. Otherwise, it is preferred in one step 530 the camera 165 pivoted in a predetermined window to the specific acoustic position. Subsequently, in one step 532 Check if a face has been detected and in this case to step 514 branched. Otherwise, in one step 534 checks if the predetermined window has already been fully analyzed. If this is not the case, the procedure continues 500 preferably with the step 530 continued. Otherwise, in one step 536 an internal variable, which is called c here, will be set to 0. In a subsequent step 538 can the user 105 in the natural language way by means of the output device 130 , or in an optical or textual manner by means of the display device 160 to be asked if his face is straight from the camera 165 can not be detected. The variable c is incremented.

In one step 540 it checks if the user 105 answered yes. In this case, the internal brand user_detectable becomes one step 542 set to 0 and the procedure 500 moves to the step 520 continued. Can not answer the user 105 can be determined, so in one step 544. to be serviced a predetermined time before in one step 546 it is checked whether the variable c has the value 3. If this is not the case, then in step 538 the question to the user 105 reissued and the variable c is incremented again. Is the value of the variable c in step 546 3, so the procedure 500 in one step 548 end up.

Will in step 540 found that the answer of the user 105 No, it will be in one step 550 the variable c is set to the value 0 and in one step 552 the user 105 asked to repeat the keyword. The variable c is incremented. In a subsequent step 554 it checks if the user 105 the keyword has spoken. In this case, the procedure continues 500 preferably with the step 508 continued. Otherwise, in one step 556 to be serviced a predetermined time before in one step 558 it is checked whether the variable c has the value 3. If this is not the case, the procedure continues 500 with the step 552 continue and ask the user 105 again, the keyword to speak. Otherwise, the procedure may 500 in step 548 end up.

LIST OF REFERENCE NUMBERS

100

 contraption

105

 user

110

 household appliance

115

 interface

120

 processing device

125

 acoustic input device

130

 acoustic output device

135

 interface

140

 base section

145

 header

150

 driving means

155

 reference mark

160

 display

165

 camera

170

 Eye of the user

305

 eyeball

310

 iris

315

 pupil

320

 upper eyelid

325

 lower eyelid

330

 eyelash

335

 eyebrow

500

 method

502

 Keyword captured

504

 User known?

506

 put brand user_detectable

508

 Determine acoustic direction of the user

510

 Swivel head section, create camera image

512

 Face recognized?

514

 Determine rel. Position of the user from camera image

516

 Align reference mark

518

 Track users

520

 Capture user input

522

 Determine sound source expression

524

 Merge positions

526

 Adjust alignment

528

 Brand user_detectable set?

530

 Move the camera in a predetermined window

532

 Face recognized?

534

 Searched the window completely?

536

 c = 0

538

 Question to user

540

 Evaluate answer

542

 user_detectable = 0

544

 waiting

546

 c = 3?

548

 The End

550

 c = 0

552

 Ask user to repeat keyword

554

 Keyword spoken?

556

 waiting

558

 c = 3?

QUOTES INCLUDE IN THE DESCRIPTION

This list of the documents listed by the applicant has been generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• US 2010/0023163 A1 [0005]

Claims (13)

Contraption ( 100 ) for controlling an interaction between a user ( 105 ) and a household appliance ( 110 ), the device ( 100 ) comprises: - a base section ( 140 ); One opposite the base section ( 140 ) movable optical reference mark ( 155 ), which is adapted to the user ( 105 ) to convey the visual impression of an eye; - a camera ( 165 ); An acoustic input device ( 125 ); An acoustic output device ( 130 ); An interface ( 135 ) for connection to the domestic appliance ( 110 ); and a processing device ( 120 ), which is set up to: - initiate a natural language dialogue between the user ( 105 ) and the device ( 100 ) capture; - a picture of the user ( 105 ) by means of the camera ( 165 ); - the reference mark ( 155 ) during the dialog to the user ( 105 ) to align; and - the household appliance ( 110 ) by means of the interface ( 135 ) based on the dialogue. Contraption ( 100 ) according to claim 1, wherein the processing device ( 120 ) is further adapted to create an eye based on the image ( 170 ) of the user ( 105 ) and the reference mark ( 155 ) on the eye ( 170 ) of the user ( 105 ). Contraption ( 100 ) according to claim 1 or 2, further comprising a head portion ( 145 ), which in at least one dimension opposite the base portion ( 140 ) is movable and on which the reference mark ( 155 ), the processing device ( 120 ) is adapted to the head section ( 145 ) opposite the base section ( 140 ) and the reference mark ( 155 ) opposite the head section ( 145 ) to move the reference mark ( 155 ) on the user ( 105 ). Contraption ( 100 ) according to one of the preceding claims, wherein the reference mark ( 155 ) is formed three-dimensionally and a drive device ( 150 ) for moving the reference mark ( 155 ) is provided. Contraption ( 100 ) according to one of claims 1 to 3, further comprising a two-dimensional graphic display device ( 160 ), the processing device ( 120 ) is adapted to the reference mark ( 155 ) on the display device ( 160 ) to represent and align. Contraption ( 100 ) according to one of the preceding claims, wherein the acoustic input device ( 125 ) comprises a microphone array, with the aid of which an acoustic position of the user ( 105 ) can be determined. Contraption ( 100 ) according to the preceding claim, wherein the camera ( 165 ) is set up in a predetermined window around the particular acoustic position of the user ( 105 ) to be pivoted to a face of the user ( 105 ) to find. Procedure ( 500 ) for controlling a device ( 100 ) for controlling an interaction between a user ( 105 ) and a household appliance ( 110 ), the device ( 100 ) an optical reference mark ( 155 ), which is adapted to the user ( 105 ) to give the visual impression of an eye, the method ( 500 ) comprises the following steps: - detecting ( 502 ) of the beginning of a natural language dialogue between the user ( 105 ) and the device ( 100 ); - scanning ( 510 ) of an image of the user ( 105 ); - Align ( 510 ) of the reference mark ( 155 ) on the user ( 105 ) during the dialogue; and - controlling the household appliance ( 110 ) based on the dialogue. Procedure ( 500 ) according to any one of the preceding claims, wherein the dialogue comprises at least one natural language utterance of the user ( 105 ) and a natural language utterance of the device ( 100 ) and the two utterances are in context with each other. Procedure ( 500 ) according to any one of the preceding claims, wherein detecting the beginning of the dialogue comprises detecting ( 502 ) one by the user ( 105 ) spoken keyword. Procedure ( 500 ) according to one of the preceding claims, further comprising determining the end of the dialogue and canceling the orientation of the reference mark ( 155 ) on the user ( 105 ). Procedure ( 500 ) according to one of the preceding claims, wherein the dialogue is determined to have ended if no natural-language utterance with the user (more than a predetermined duration) 105 ) was exchanged. Domestic appliance ( 110 ) with a device ( 100 ) according to one of claims 1 to 7.

Images