WO2016202544A1

WO2016202544A1 - Method for operating a household appliance, corresponding household appliance and system comprising a household appliance

Info

Publication number: WO2016202544A1
Application number: PCT/EP2016/061802
Authority: WO
Inventors: Jochen Grimminger; Astrid Klingshirn; Martin Gebler
Original assignee: BSH Hausgeräte GmbH
Priority date: 2015-06-15
Filing date: 2016-05-25
Publication date: 2016-12-22
Also published as: CN107735636A; US20180172343A1; DE102015210899A1

Abstract

The invention relates to a method which comprises the following steps: taking a picture of the interior of a household appliance by means of a camera, identifying at least one object in the picture, and editing said object. The invention also relates to corresponding devices and systems.

Description

Method for operating a household appliance and corresponding household appliance and system with household appliance

The invention relates to a method for operating a household appliance and a corresponding household appliance and a system with a household appliance.

It is known in a household appliance, e.g. a refrigerator, to use a camera which at certain times takes up the inside of the refrigerator or a part of it. The camera may e.g. when opening and / or closing the refrigerator door, make a picture from the inside of the refrigerator, e.g. from the food or other items currently in the fridge. Such a recording can be made accordingly also in the closed state of the refrigerator. For example, it is possible that a sensor is provided for this purpose, which detects an opening and / or closing of the refrigerator door. This can e.g. done by means of a magnetic sensor or other switch. The shot taken by the camera is e.g. stored in a memory via a wireless or wired connection. The memory may e.g. be a memory of the household appliance or a memory of a computer or a computer network. In particular, the recording may be made via a network interface on a central database, e.g. Tethered via the Internet.

If at least one recording of the contents of the household appliance is made with each opening and / or closing, the changes between different recordings which occur during the use of the domestic appliance can lead to the setting of goods in the domestic appliance and to a corresponding removal of the goods be inferred. This information can be further processed for different purposes.

Here it is problematic that there are a variety of reusable containers that appear on the recording and provide no conclusion on their content. This applies, for example, for universally usable resealable containers. The task is to improve an existing household appliance so that an efficient handling of different containers is possible.

This object is achieved according to the features of the independent claims. Further developments of the invention will become apparent from the dependent claims.

To solve the problem, a method is specified with the following steps:

Taking a picture of the interior of a household appliance by means of a camera,

Detecting at least one object in the recording,

Editing the object.

The method may e.g. run completely or partially in the household appliance. In particular, it is possible that the recognition of the object and / or the editing of the object in or on the household appliance are performed. Optionally, these steps may also be performed separately from the domestic appliance, in at least one external component, e.g. connected via a communication interface with the household appliance. For example, the detection of the object can be performed in a central computer, which is connected to the Internet as well as the household appliance. The editing of the object may e.g. be performed on or by means of the central computer. Optionally, editing may also be performed on another device, e.g. a mobile terminal (e.g., smartphone). According to a further option, the recognition can take place at least partially or completely by means of a program running on the terminal, with or without the assistance of the user.

The recording can be made by a camera of the household appliance. The recording can be stored in a memory of the household appliance, in an external memory and / or in a network memory. In principle, it is possible to arbitrarily combine the applications mentioned here by way of example. Thus, the recording could be transmitted via a WLAN interface from the household appliance via the Internet directly to a smartphone, which can be connected to the Internet either via a WLAN interface or via another wireless communication connection (eg 3G or 4G). Detecting the object and editing the object Object could be performed on the mobile terminal according to this example. For this purpose, the mobile terminal may have a special program (a so-called "app"). A further development is that the editing of the object comprises: creating, changing or deleting information associated with the object.

For editing, the home appliance may have a display device (with a screen and an input interface). Also, an external display device which is couplable to the home appliance can be used. Furthermore, it is an option that the external display device can be coupled to a memory on which the recordings are stored.

Another development is that the information comprises at least one of the following:

a content of the object;

a purpose of the object;

an owner of the object;

an instruction concerning the object;

a condition associated with the object;

a default action to take if a condition is met.

The given action can be varied. For example, a communication with other services can be established automatically. In particular, it can be seen that at least one item (eg, milk) is missing in the refrigerator and a reordering of the item can be automatically initiated. Thus, it is an option that the item, if any, occupy a particular space in the refrigerator or are located in a particular area of the refrigerator. This can be automatically detected if the item is missing or was consumed. For example, when removing the item from the refrigerator, a timer (timer) is started. If the item is not returned within a predetermined time (eg 1 hour), it is assumed that the item has been consumed and the reordering can be triggered. Accordingly, there is one Variety of options for detecting items in the refrigerator as well as possible options that may be coupled to detection or non-detection.

In particular, it is possible that at least one condition is linked as information. This condition may be regular, e.g. every day or depending on certain parameters, e.g. a temperature or other objects, are automatically checked. If the condition is met, a predetermined action is initiated. An example is the display of goods whose expiration date has expired and / or is about to expire. Another example is the issue of a warning when the temperature in the home appliance reaches a critical level for this product.

In particular, it is a development that the household appliance is one of the following devices:

a kitchen appliance,

- a cooking appliance,

a refrigerator and / or freezer.

The refrigerator and / or freezer is also referred to here as a refrigerator. It is also a development that the camera is arranged such that when opening and / or closing a door of the household appliance, the recording is created.

To solve the problem, a household appliance is further proposed with a processing unit for carrying out the method described here.

Furthermore, the object is achieved by means of a system comprising a household appliance and at least one further component,

wherein the household appliance has a camera for taking a picture of the interior of a domestic appliance,

- wherein the at least one further component comprises a processing unit for

Recognize at least one object in the recording,

Editing the object. The processing unit mentioned here may in particular be embodied as a processor unit and / or an at least partially hard-wired or logical circuit arrangement, which is set up, for example, in such a way that the method can be carried out as described herein. Said processing unit may be or include any type of processor or computer or computer with correspondingly necessary peripherals (memory, input / output interfaces, input / output devices, etc.).

The above explanations regarding the method apply to the device or the system accordingly.

A development is that the at least one further component comprises a memory which is accessible via a network interface and on which the recording of the camera can be stored. Furthermore, it is a development that the at least one further component has a display device with an input interface and a screen, wherein the object can be edited on the basis of the input interface and wherein the object and information can be displayed on the basis of the screen. As part of an additional development, the display device is a smartphone or a computer.

Embodiments of the invention are illustrated and explained below with reference to the drawing.

It shows:

1 shows by way of example a schematic block diagram comprising a refrigerator with a door, a camera and a processing unit in a symbolic sectional view.

The proposal described here allows a user to recognize what is contained in any container (eg, a universal resealable can, packaging, etc.) that is in a household appliance. For this purpose is the container taken by a camera, subjected to the inclusion of an object detection and recognized the container as an object of the recording. The camera can be arranged, for example, in the household appliance. When opening and / or closing the door of the household appliance, the recording can be created.

The recognized object is edited, i. e.g. with an information assigned or linked. The information may include a marker, a text, a symbol, a color, etc. The information may be associated with a content of the container for the user. Based on the information, it is thus possible for the user to recognize the contents of the container.

The object is edited by e.g. the recognized object is colored and / or filled with text. For example, the recording can be supplemented with the information and the information is stored linked to the recording. If the user looks at the picture, he also sees the information and can immediately recognize the contents of the container. Optionally, it is possible that the user makes the information visible through an action or the information is displayed as a result of the action. Thus, the clarity of the presentation can be improved by not always displaying all the information.

In this case, it is advantageous that the information can be offered to any user who uses the household appliance. For example, the information can be displayed on a display device. The display device may be part of the household appliance or it may be part of a computer, e.g. a smartphone or any (mobile or stationary) computer. The recording with or without information may e.g. be forwarded by the household appliance via an interface and stored in a network. The information may already be provided on the home appliance or it may be supplemented or changed after the recording (with or without information) e.g. has been stored in the household appliance, in a connected storage and / or in the network.

The information may contain the contents of the container, an indication of purpose (eg in the context of a recipe), a possible use, an owner, a condition (eg expiration date) or the like. include. The information can be linked to at least one predetermined action: If the user selects an expiration date as information, a predefined action can be initiated before, with or after reaching this expiration date: For example, a message can be displayed two days before the expiration date (eg the display device) that the goods should be consumed in the container (in this case, for example, the container and the goods and their pending expiration date or the time to expiry are displayed). In this way, a countdown for goods expiring in an upcoming period, eg the next ten days, could be displayed. This can be effectively ensured that the user can consume the goods in good time before reaching the expiration date or that there are no goods in the refrigerator, whose expiry of the expiration date, the user has not noticed.

In another example, the temperature of the refrigerator may be adjusted depending on the information: if the user indicates as information a particular commodity for which a preferred temperature (or temperature range) is stored in a table of the processing unit, the refrigerator may e.g. for an area where the container is located, adjust the temperature accordingly.

One advantage is that also a displacement of the object in the refrigerator can be detected and thus also the information with the object in the refrigerator virtually mitwandert. Thus, on the display device, the contents of the refrigerator, so the detected objects, always up to date presented together with the information. Optionally, the information about the object can be deleted or updated. This allows a varied use of the recognized object as a container for different goods.

The information can be linked in different ways to objects detected in the recording. For example, the information can be displayed transparently, enlarged, reduced, shifted, with arrows or lines pointing to the object, etc. It is also possible for the contents of the refrigerator to be displayed virtually on the display device by virtue of the detected objects being virtually moved on the screen. Accordingly, the information together with the virtual operations are rebuilt and presented. It is also possible that one of the actual position of the object in the refrigerator independent (possibly sorted list) is displayed, which sorts the detected objects with the information, for example, tabular or otherwise. The information may serve as a sorting criterion for such a presentation.

In addition, it should be noted that the information can be edited in different representations on the display device. For example, the information can be supplemented, created or deleted for the first time. The information can also be provided with conditions or previously created conditions can be deleted. Furthermore, the information can be presented to all users or to a selection of users. For example, it is possible to display to certain users the content "biscuits" of a container and to display other information to other users rather than the content, e.g. "Owner: Karl, please do not eat!".

Furthermore, it is possible for the information to include suggestions for storage (location of storage, duration of storage, temperature of storage, etc.) of an item in the household appliance, in particular for changing the storage. The display device may be part of the household appliance. It is also possible that the display device is a conventional computer (e.g., laptop, personal computer, etc.). Furthermore, a smartphone can be used as a display device. For example, an application (also: application program, app) can be installed on the smartphone, so that the smartphone can be used as a display device for the household appliance.

The information can be edited via the display device. Also, the contents of the household appliance (current content according to last taken recording, if necessary, taking into account previous recordings) can be virtually displayed on the display device. Furthermore, evaluations (eg with regard to the expiry date or with regard to possible recipes) can be carried out with the aid of the display device. In this sense, the display device includes, for example, a processor unit that allows editing and / or processing of the information by means of a program and recording. In this case, part of the processing, for example detection of objects in the recordings of the camera, can already be performed by the household appliance or it can be transmitted from the household appliance recordings to an external processing unit, for example in a network. There, processing steps (eg, alone or in addition) can be carried out, for example, to detect objects in the recordings. The results of the processing may be provided to the different display devices, eg via a wireless or a wired interface.

One option is that the home appliance has a connection to the Internet and loads the recordings into a storage connected to the Internet. The display devices can also access the memory via the Internet and thus the recordings stored there, possibly together with the stored information.

The information may be stored centrally, stored together or separately from the recordings, stored in a memory directly connected to or located in the home appliance. Furthermore, the information can be stored in the display device. Distributing the information from one user to at least one user is possible in all the cases mentioned, and optionally can also be provided with restrictions concerning the access (for example in the form of user rights).

1 shows by way of example a schematic block diagram comprising a refrigerator 101 with a door 102 in a symbolic sectional view. In Figure 1, the door 102 is closed. A sensor 106 detects whether the door 102 is open or closed. Upon opening and / or closing the door 102, the sensor 106 transmits a corresponding signal to a processing unit 104. The processing unit 104 then controls a camera 103 so that it creates a receptacle from the interior of the refrigerator 101. By way of example, the processing unit 104 stores the recording in one Memory 107. Memory 107 may be part of or separate from processing unit 104.

A display device 108 may be external to the refrigerator 101. Also, the display device 108 may use the processing unit 104 or it may be part of the display device 108. The display device 108 may e.g. be arranged outside of the door 102 of the refrigerator 101 and so operated by the user with the door closed (or even open) 102. For example, the display device 108 includes an input interface, e.g. a keyboard, a (for example, touch-sensitive) screen, a pointing device, at least one button or other navigation options. Thus, the user can be presented via a screen of the display device 108 recording with or without (editable) information. In particular, the user can edit the information and thus, for example, associate information with a recognized object of the recording. Thus, the recording can be superimposed on the screen of the display device with virtual information, wherein the information can supplement the recognized object by way of example.

For example, the processing unit 104 may be configured to detect at least one object 105 in the shot taken by the camera. The object 105 in the example of Fig. 1 is a reclosable universal opaque can. This can is recognized in the recording and can be edited as an object by the user. Such editing includes, by way of example, assigning information to the object 105. Thus, the user can link the information "biscuits" with the object 105 by means of the display device 108. This information "biscuits" may then be displayed on the screen of the display device 108 along with the shot.

For example, it is also possible for the camera 103 and the processing unit 104 to be implemented as one component. In particular, such a component may be a e.g. have a wireless network connection and store the recordings made directly on a connected via the Internet memory. Such storage is also referred to as "cloud storage."

FIG. 1 further shows, by way of example, an interface between the processing unit 104 and a smartphone 123. It should be noted that the dashed lines 1 symbolically represent a transmission of signals or data. The transmission of data to the smartphone 123 may, for example, take place via a radio interface which at least partially uses the Internet. For this purpose, the processing unit 104 may itself have a wireless or wired connection to the Internet.

The smartphone 123 represents another example of a display device. It includes an input option in the form of a (eg virtual) keyboard 123 and a screen 121. On the screen of the smartphone, the recognized object of the can 105 can be virtualized, for example, visualized as a detached image 122 of the can 105. The user has the opportunity to enter information for this image 122. In the example of FIG. 1, the string "biscuits" is entered as information 124 and linked to the image 122. Thus, each user or an authorized selection of users may be provided with the ability to also experience what is in the object 105 while viewing a shot from the inside of the refrigerator 101. For this purpose, the information "biscuits" is displayed on the display device 108 or 123 together with the recording.

LIST OF REFERENCE NUMBERS

101 refrigerator

102 door

103 camera

104 processing unit

105 object

106 sensor

107 memory

108 display device

121 screen

122 illustration

123 smartphone

124 information

Claims

Method comprising the steps:

- Making a recording from the interior of a household appliance by means of a camera,

Detecting at least one object in the recording,

- Editing the object.

The method of claim 1, wherein editing the object comprises: creating, changing or deleting information associated with the object.

The method of claim 2, wherein the information comprises at least one of the following:

- a content of the object;

- a purpose of the object;

- an owner of the object;

an instruction concerning the object;

a condition associated with the object;

a default action to be taken if a condition is met.

Method according to one of the preceding claims, in which the domestic appliance is one of the following appliances:

- a kitchen appliance,

- a cooking appliance,

- a refrigerator and / or freezer.

Method according to one of the preceding claims, wherein the camera is arranged such that when opening and / or closing a door of the household appliance, the recording is created.

Domestic appliance with a processing unit for carrying out the method according to one of the preceding claims.

7. system comprising a household appliance and at least one further component,

- wherein the household appliance has a camera for taking a picture of the interior of the household appliance,

- wherein the at least one further component comprises a processing unit for

Detecting at least one object in the recording,

- Editing the object.

8. The system of claim 7, wherein the at least one further component comprises a memory which is accessible via a network interface and on which the recording of the camera can be stored.

9. System according to one of claims 7 or 8, wherein the at least one further component comprises a display device with an input interface and a screen, wherein based on the input interface, the object is editable and wherein based on the screen, the object and information can be displayed.

The system of claim 9, wherein the display device is a smartphone or a computer.