US20180172343A1

US20180172343A1 - Method for operating a household appliance, corresponding household appliance and system with a household appliance

Info

Publication number: US20180172343A1
Application number: US15/736,111
Authority: US
Inventors: Jochen Grimminger; Astrid Klingshirn; Martin Gebler
Original assignee: BSH Hausgeraete GmbH
Current assignee: BSH Hausgeraete GmbH
Priority date: 2015-06-15
Filing date: 2016-05-25
Publication date: 2018-06-21
Also published as: WO2016202544A1; CN107735636A; DE102015210899A1

Abstract

A method includes taking a picture of the interior of a household appliance by using a camera, identifying at least one object in the picture, and editing the object. A household appliance having a processing unit for performing the method and a system having a household appliance and at least one further component are also provided.

Description

The invention relates to a method for operating a household appliance, a corresponding household appliance and a system with a household appliance.
It is known that a camera can be used in a household appliance, e.g. a refrigerator, to record the interior of the refrigerator or part thereof at specific times. The camera can take a picture of the interior of the refrigerator, e.g. the food or other items currently located in the refrigerator, e.g. when the refrigerator door is opened or closed. Correspondingly, such a picture can also be taken when the refrigerator is in a closed state.
A sensor can be provided for this purpose, for example, and detect any opening and/or closing of the refrigerator door. This can be achieved e.g. by means of a magnetic sensor or other type of switch. The picture taken by the camera is saved in a memory, e.g. via a wireless or hard-wired connection. The memory can be a memory of the household appliance or a memory of a computer or computer network, for example. In particular, the picture can be stored via a network interface in a central database which is attached via the Internet, for example.
If at least one picture of the content of the household appliance is taken each time it is opened and/or closed, it is then possible on the basis of the changes between different pictures produced during the course of using the household appliance to infer an insertion of goods into the household appliance and a corresponding removal of goods. This information can be further processed for various purposes.
It is problematic in this context that a multiplicity of reusable containers are present, appearing on the picture but giving no indication of their content. This applies to general-purpose resealable containers, for example.
The object of the invention is to improve an existing household appliance such that efficient use is also possible with different containers.
This object is achieved in accordance with the features in the independent claims. Developments of the invention are derived from the dependent claims.
A method comprising the following steps is specified for the purpose of achieving said object:

- taking a picture of the interior of a household appliance by means of a camera,
- identifying at least one object in the picture,
- editing the object.

The method can be executed e.g. entirely or partly in the household appliance. In particular, the identification of the object and/or the editing the object can be performed in or at the household appliance. As further option, these steps can also be performed separately from the household appliance in at least one external component which is connected to the household appliance e.g. via a communication interface. For example, the identification of the object can be performed in a central computer, which like the household appliance is connected to the Internet. The editing of the object can be performed e.g. on or with the aid of the central computer. As a further option, the editing can also be performed on another device, e.g. a mobile terminal (e.g. smartphone). According to a further option, the identification can be effected at least partially or even entirely by means of a program which is executed on the terminal with or without user input.
The picture can be taken by a camera of the household appliance. The picture can be stored in a memory of the household appliance, in an external memory and/or in a network memory. In principle, any of the application cases cited here by way of example can be combined as desired. The picture could therefore be transferred via a WLAN interface from the household appliance via the Internet directly to a smartphone, which can be connected to the Internet either via a WLAN interface or other wireless communication connection (e.g. 3G or 4G). The identification of the object and the editing of the object could be performed on the mobile terminal in this example. The mobile terminal can have a special program for this purpose (a so-called “app”).
According to a development, the editing of the object comprises: creating, modifying or deleting information which is associated with the object.
The household appliance can have a display device (comprising a screen and an input interface) for the purpose of editing. It is also possible to use an external display device which can be linked to the household appliance. As a further option, the external display device can be connected to a memory in which the pictures are stored.
According to a further development, the information comprises at least one of the following:

- a content of the object;
- an intended use of the object;
- an owner of the object;
- an instruction relating to the object;
- a condition associated with the object;
- a predefined action to be executed if a condition is met.

The predefined action can be very diverse. For example, communication can be automatically established with other services. In particular, it is possible to detect that at least one item (e.g. milk) is missing from the refrigerator and reordering of the item can automatically be triggered. Accordingly, the item (where present) can optionally occupy a specific space in the refrigerator or be arranged in a specific region of the refrigerator. It is then possible automatically to determine when the item is missing or has been consumed. For example, a timer is started when the item is removed from the refrigerator. If the item is not replaced within a predefined time (e.g. 1 hour), it is assumed that the item has been consumed and the reordering can be triggered. A multiplicity of options are available for identifying items in the refrigerator accordingly, and possible options which can be associated with the identification or non-identification.
In particular, at least one condition can be associated as information. This condition can be checked automatically on a regular basis, e.g. every day, or as a function of specific parameters, e.g. a temperature, or other objects. If the condition is met, a predefined action is initiated. One example is the display of goods whose use-by date has expired and/or will soon expire. A further example is the output of a warning if the temperature in the household appliance reaches a critical threshold for this product.
According to a development, the household appliance is one of the following appliances in particular:

- a kitchen appliance,
- a cooking appliance,
- a fridge/freezer appliance.

The fridge/freezer appliance is also referred to as a refrigerator here.
According to a further development, the camera is so arranged that the picture is taken when a door of the household appliance is opened and/or closed.
The object is further achieved by a proposed household appliance comprising a processing unit for performing the method described here.
The object is further achieved by a system which comprises a household appliance and at least one further component,

- wherein the household appliance has a camera for taking a picture of the interior of a household appliance,
- wherein the at least one further component has a processing unit for
  - identifying at least one object in the picture,
  - editing the object.

The processing unit cited here can be embodied in particular as a processor unit and/or a circuit arrangement that is at least partly hard-wired or a logical circuit arrangement, which is so configured that the method can be performed as described here, for example. Said processing unit may be or comprise any type of processor or computer or computer with peripherals (memory, input/output interfaces, input/output devices, etc.) that are correspondingly required.
The present explanations relating to the method apply respectively to the device and the system correspondingly.
According to a development, the at least one further component comprises a memory which can be accessed via a network interface and in which the picture can be stored by the camera.
According to a further development, the at least one further component has a display device with an input interface and a screen, wherein the object can be edited using the input interface and wherein the object and information can be represented by means of the screen.
According to a further development, the display device is a smartphone or a computer.

Exemplary embodiments of the invention are illustrated and explained below with reference to the drawing, in which:

FIG. 1 shows an exemplary schematic block diagram comprising a refrigerator with a door, a camera and a processing unit in a symbolic side view.

The proposal described here allows a user to identify what is contained in any container (e.g. a general-purpose resealable box, packaging, etc.) which is situated in a household appliance. To this end, the container is recorded by a camera, the picture undergoes object identification and the container is identified as the object of the picture. The camera can be arranged in the household appliance, for example. The picture can be taken when the door of the household appliance is opened or closed, for example.
The identified object is edited, i.e. annotated or associated with information, for example. The information may comprise a marking, a text, a symbol, a color, etc. The information can be associated with a content of the container for the user. It is therefore possible for the user to identify the content of the container on the basis of the information.
The object is edited, e.g. by coloring the identified object and/or annotating it with a text. For example, the picture can be supplemented by the information and the information stored in association with the picture. When looking at the picture, the user also sees the information and can immediately identify the content of the container. It is optionally possible for the user to make the information visible by means of an action, or for the information to be represented as a result of the action. The clarity of the representation can be improved if not all information is represented at all times.
It is advantageous in this context that the information can be made available to every user who uses the household appliance. For example, the information can be represented on a display device. The display device can be part of the household appliance or part of a computer, e.g. a smartphone or any (mobile or stationary) computer. The picture with or without information can be forwarded e.g. from the household appliance via an interface and stored in a network. The information can be provided directly at the household appliance, or it can be supplemented or modified after the picture (with or without information) has been stored in the household appliance, in an associated memory and/or in the network, for example.
The information can comprise the content of the container, an intended purpose (e.g. in the context of a recipe), a possible use, an owner, a condition (e.g. use-by date), or similar.
The information can be associated with at least one predefined action: if the user selects a use-by date as information, a predefined action may be initiated before, upon or after reaching this use-by date: for example, a message may be displayed (e.g. on the display device) two days before reaching the use-by date to the effect that the goods in the container should be consumed (e.g. the container and the goods can be displayed in this case with the imminent use-by date or the time until expiry). It is also possible in this way to display a countdown for goods due to expire in a forthcoming time period, e.g. the next ten days. It is thereby possible effectively to ensure that the user can consume the goods in a timely manner before reaching the use-by date, and/or that the refrigerator does not contain any goods in respect of which the user is not aware that the use-by date has expired.
In a further example, the temperature of the refrigerator can be adapted as a function of the information: if information specified by the user relates to specific goods for which a preferred temperature (or a temperature range) is stored in a table of the processing unit, the refrigerator can adapt the temperature accordingly for a region in which the container is located, for example.
It is advantageous that displacement of the object in the refrigerator can also be detected and therefore in a virtual sense the information moves with the object in the refrigerator. This means that at all times the most recent content of the refrigerator, i.e. the identified objects, can be represented on the display device with the information. The information about the object can optionally be deleted or updated. This allows the identified object to be used in a wide variety of ways as a container for different goods.
The information can be associated in different ways with the objects identified in the picture. For example, the information can be represented in a manner which is transparent, magnified, reduced, shifted, pointing to the object with arrows or lines, etc. It is also possible in a virtual manner to represent the content of the refrigerator on the display device, by virtually moving the identified objects on the screen. The information can be recreated and represented in conjunction with the virtual operations. It is also possible to represent a (possibly sorted) list which is independent of the actual position of the object in the refrigerator, and shows the identified objects with the information in the form of a table or otherwise sorted, for example. The information can be used as a sorting criterion for such a representation.
It should also be noted that the information can be edited in different representations on the display device. For example, the information can be supplemented, entered for the first time or deleted. The information can also be associated with conditions, or previously entered conditions can be deleted. Furthermore, the information can be represented to all users or a selection of users. For example, it is possible to display the content “Biscuits” of a container to specific users, and instead of the content to display different information to other users, e.g. “Owner: Karl, please do not eat!”.
The information can also comprise suggestions in respect of storage (place of storage, duration of storage, temperature of storage, etc.) of an item in the household appliance, in particular for the purpose of changing the storage.
The display device can be part of the household appliance. The display device can also be a conventional computer (e.g. laptop, personal computer, etc.). Furthermore, a smartphone can be used as a display device. For example, an application (also known as an application program or app) can be installed on the smartphone, so that the smartphone can be used as a display device for the household appliance.
The information can be edited via the display device. The content of the household appliance (current content according to most recent picture, possibly with reference to previous pictures) can also be represented virtually on the display device. Furthermore, it is possible to perform analyses (e.g. in respect of the use-by date or in respect of possible recipes) using the display device.
Accordingly, the display device comprises e.g. a processor unit which allows the information to be edited and/or processed by means of a program and with reference to the picture.
In this context, part of the processing such as e.g. identification of objects in the pictures of the camera can already be performed by the household appliance, or pictures can be transmitted from the household appliance pictures to an external processing unit, e.g. in a network. Processing steps e.g. for identifying objects in the pictures can be performed there, either exclusively or additionally. The results of the processing can be provided to the different display devices, e.g. via a wireless or hard-wired interface.
As an option, the household appliance can have a connection to the Internet and load the pictures into a memory which is connected to the Internet. The display devices can also access the memory via the Internet, and thereby access the pictures stored there, together with the stored information if applicable.
The information can be stored centrally, it can be stored together with or separately from the pictures, it can also be stored in a memory which is directly connected to or arranged within the household appliance. Furthermore, the information can be stored in the display device. Distribution of the information from one user to at least one other user is possible in all of the cited cases, and optionally with restrictions relating to the access (e.g. on the form of user permissions).
By way of example, FIG. 1 shows a schematic block diagram comprising a refrigerator 101 with a door 102 in a symbolic side view. The door 102 is closed in FIG. 1. A sensor 106 detects whether the door 102 is open or closed. When the door 102 is opened or closed, the sensor 106 transmits a corresponding signal to a processing unit 104. The processing unit 104 then activates a camera 103, whereby this takes a picture of the interior of the refrigerator 101. By way of example, the processing unit 104 stores the picture in a memory 107. The memory 107 can be embodied as part of the processing unit 104 or separate from this.
A display device 108 can be embodied externally relative to the refrigerator 101. The display device 108 can also make use of the processing unit 104, or this can be part of the display device 108. The display device 108 can be arranged e.g. externally on the door 102 of the refrigerator 101, and therefore operated by the user when the door 102 is closed (or open). For example, the display device 108 has an input interface, e.g. a keypad, a (e.g. touch-sensitive) screen, a pointer device, at least one control button or other navigation tool. The picture with or without (editable) information can therefore be represented for the user via a screen of the display device 108. In particular, the user can edit the information and, for example, assign information to an identified object in the picture in this way. This means that virtual information can be superimposed on the picture on the screen of the display device, said information supplementing the identified object, for example.
The processing unit 104 can be configured, for example, to identify at least one object 105 in the picture taken by the camera. The object 105 in the example of FIG. 1 is a resealable general-purpose opaque box. This box is identified in the picture and can be edited by the user as an object. Such editing comprises the assignment of information to the object 105, for example. The user can therefore associate the information “Biscuits” with the object 105 by means of the display device 108. This information “Biscuits” can then be displayed with the picture on the screen of the display device 108.
It is also possible, for example, to embody the camera 103 and the processing unit 104 as one component. In particular, such a component may have a wireless network connection, for example, whereby the pictures that have been taken are stored directly in a memory which is attached via the Internet. Such a memory is also referred to as “cloud storage”.
By way of example, FIG. 1 also shows an interface between the processing unit 104 and a smartphone 123. It should be noted that the broken lines in FIG. 1 symbolically represent a transfer of signals or data. The transfer of data to the smartphone 123 can be effected e.g. via a radio interface which at least partially uses the Internet. To this end, the processing unit 104 itself can have a wireless or hard-wired connection to the Internet.
The smartphone 123 represents a further example of a display device. It comprises input means in the form of a (e.g. virtual) keypad 123 and a screen 121. The identified object of the box 105 can be visualized on the screen of the smartphone, e.g. as an optional image 122 of the box 105. The user has the means to enter information for this image 122. In the example of FIG. 1, the character string “Biscuits” is entered as information 124 and associated with the image 122.
This means that each user or an authorized selection of users can be given the option, when looking at a picture of the interior of the refrigerator 101, also to find out what is contained in the object 105. For this purpose, the information “Biscuits” is also shown with the picture on the display device 108 or 123.

LIST OF REFERENCE SIGNS

101 Refrigerator
102 Door
103 Camera
104 Processing unit
105 Object
106 Sensor
107 Memory
108 Display device
121 Screen
122 Image
123 Smartphone
124 Information

Claims

1-10. (canceled)

11. A method, comprising the following steps:

taking a picture of an interior of a household appliance by using a camera;

identifying at least one object in the picture; and

editing the object.

12. The method according to claim 11, which further comprises carrying out the step of editing the object by creating, modifying or deleting information associated with the object.

13. The method according to claim 12, wherein the information includes at least one of the following:

a content of the object;

an intended use of the object;

an owner of the object;

an instruction relating to the object;

a condition associated with the object; or

a predefined action to be executed if a condition is met.

14. The method according to claim 11, wherein the household appliance is one of the following appliances:

a kitchen appliance,

a cooking appliance, or

a refrigerator/freezer appliance.

15. The method according to claim 11, which further comprises carrying out the step of taking the picture by using the camera when a door of the household appliance is open or closed.

16. A household appliance, comprising a processing unit for performing the following steps:

taking a picture of an interior of a household appliance by using a camera;

identifying at least one object in the picture; and

editing the object.

17. A system, comprising:

a household appliance having a camera for taking a picture of an interior of the household appliance; and

at least one further component having a processing unit for:

identifying at least one object in the picture and

editing the object.

18. The system according to claim 17, wherein said at least one further component includes a memory configured to be accessed by a network interface and configured to store the picture by the camera.

19. The system according to claim 17, wherein said at least one further component has a display device with an input interface and a screen, said input interface is configured to edit the object, and said screen is configured to show the object and information.

20. The system according to claim 19, wherein said display device is a smartphone or a computer.