DE102015210900A1 - Method for processing data of a household appliance - Google Patents

Abstract

A method is proposed, comprising the steps of: creating a recording by means of a mobile terminal, detecting a household appliance in the receptacle and replacing the household appliance or a part of the household appliance in the recording by a content image from the inside of the household appliance. Furthermore, a corresponding system comprising the household appliance and the mobile terminal is specified.

Description

The invention relates to a method for processing image data provided by a household appliance on a mobile terminal and to a corresponding system.

It is known in a household appliance, e.g. a refrigerator, to use a camera which at certain times takes up the inside of the refrigerator or a part of it. The camera may e.g. when opening and / or closing the refrigerator door, make a picture from the inside of the refrigerator, e.g. from the food or other items currently in the fridge. Such a recording can be made accordingly also in the closed state of the refrigerator.

For example, it is possible that a sensor is provided for this purpose, which detects an opening and / or closing of the refrigerator door. This can e.g. done by means of a magnetic sensor or other switch. The shot taken by the camera is e.g. stored in a memory via a wireless or wired connection. The memory may e.g. be a memory of the household appliance or a memory of a computer or a computer network. In particular, the recording may be made via a network interface on a central database, e.g. Tethered via the Internet.

If at least one recording of the contents of the household appliance is made with each opening and / or closing, the changes between different recordings that occur in the course of use of the household appliance can lead to a setting of goods in the household appliance and to a corresponding removal of the goods be inferred. This information can be further processed for different purposes.

Here it is problematic that there are a variety of reusable containers that appear on the recording and provide no conclusion on their content. This is for example for universally usable resealable containers.

The task is to improve the handling of a household appliance.

This object is achieved according to the features of the independent claims. Further developments of the invention will become apparent from the dependent claims.

• – Erstellen einer Aufnahme mittels eines mobilen Endgeräts,
• – Erkennen eines Haushaltsgeräts in der Aufnahme,
• – Ersetzen des Haushaltsgeräts oder eines Teil des Haushaltsgeräts in der Aufnahme durch ein Inhaltsbild vom Inneren des Haushaltsgeräts.

To solve the problem, a method is proposed comprising the steps:

• Create a recording by means of a mobile terminal,
• Detecting a household appliance in the receptacle,
• - Replacing the household appliance or a part of the household appliance in the recording by a content image of the interior of the household appliance.

The detection of the household appliance as an object in the recording can be performed by the mobile terminal. Preferably, the mobile terminal and the household appliance are at least temporarily connected to each other via a network or via a communication link, so that the content image of the household appliance to the mobile terminal is transferable.

The content image may include a single image, an image sequence or a moving image sequence (ie a movie). The content image may e.g. be taken by at least one camera, which is located on or in the household appliance. This camera takes pictures of the interior of the household appliance. The content images may e.g. temporarily stored in a memory. As such memory, e.g. be used: a memory in the household appliance, a network storage and / or storage of the mobile terminal.

For example, the content image may be transmitted via a WLAN interface from the home appliance over the Internet to a smartphone that is connectable to the Internet either via a WLAN interface or via another wireless communication link (e.g., 3G or 4G). To recognize the object, the mobile terminal may have a special program (a so-called "app").

The content image or a part of the content image can be used to replace the household appliance in the recording that was or is created by the mobile terminal and thus to present an impression of an augmented reality on the mobile terminal by the user into the interior of the mobile device Household appliance can look.

A development is that the recording includes the household appliance.

Preferably, the household appliance (from the outside) is shown on the recording. This part of the recording can be replaced or overlaid by the content image.

• – Erstellen des Inhaltsbilds vom Innenraum des Haushaltsgeräts.

Another development is that before replacing the household appliance or the part of the household appliance in the recording the step is carried out:

• - Creating the content image of the interior of the household appliance.

For this, e.g. a camera of the household appliance can be used. This camera can provide the content image to a memory via a processing unit. Optionally, the camera may be combined with the processing unit so that the camera with processing unit e.g. send the content image over a network at certain times.

In particular, it is a further development that the content image comprises at least one item of information which can be selected and displayed on the basis of the mobile terminal.

Preferably, the content image may include the information in addition to the image taken by the camera of the household appliance. This information can be displayed as an icon. Optionally, the information may be displayed in more detail upon selection, e.g. opens a window containing explanations. The information is not limited to colors, symbols, text, etc. Also, the information may be associated with a condition, e.g. a selection, so that the information is displayed when the content image or part of the content image is clicked (e.g., by touching with a finger in the case of a touch screen mobile terminal).

• – einem Datum, das angibt, seit wann der Gegenstand in dem Kühlschrank gelagert wird,
• – einem Datum, das zeigt, seit wann der Gegenstand nicht mehr bewegt wurde,
• – eine Dauer bis zu einem Verfallsdatum des Gegenstands,
• – ein Farb-Code für bestimmte Lebensmittel,
• – einer Nährwertinformation,
• – einer zweckgebundenen Information,
• – einem Rezept.

It is also a further development that the information comprises at least one of the following:

• A date indicating since when the item is stored in the refrigerator,
• - a date that shows since when the item was not moved,
• - a period up to an expiry date of the item,
• - a color code for certain foods,
• - nutritional information,
• - a specific information,
• - a recipe.

Furthermore, it is a development that based on the information a predetermined action is executable.

• – Auslösung einer Bestellung;
• – Anzeige, ob ein Rezept oder eine Speise zubereitbar ist oder nicht;
• – Auswahl aus Rezepten, die aufgrund der mindestens einen Information zubereitbar sind;
• – Anweisung zur Platzierung eines Gegenstands in dem Haushaltsgerät.

In the context of an additional development, the predetermined action comprises at least one of the following:

• - triggering an order;
• Indication of whether a recipe or a dish is prepared or not;
• - Selection of recipes that are preparable based on the at least one information;
• - instruction to place an item in the household appliance.

• – Ersetzen des Haushaltsgeräts oder eines Teil des Haushaltsgeräts in der Aufnahme durch das Inhaltsbild, wobei das Inhaltsbild abhängig von einer Navigationsinformation ausgewählt wird.

A next development is that the method further comprises:

• Replacing the home appliance or a part of the household appliance in the recording by the content image, wherein the content image is selected depending on a navigation information.

It should be noted that the selection of the displayed content image also includes a part of the content image. So a navigation inside the household appliance by means of the content image (or multiple content images) is possible.

An embodiment is that the navigation information is provided by the mobile terminal, so that a virtual navigation inside the household appliance is made possible.

• – ein Küchengerät,
• – ein Gargerät,
• – ein Kühl- und/oder Gefriergerät.

An alternative embodiment is that the household appliance is one of the following appliances:

• - a kitchen appliance,
• - a cooking appliance,
• - a refrigerator and / or freezer.

The refrigerator and / or freezer is also referred to here as a refrigerator.

A next embodiment is that the content image is created from the interior of the household appliance by means of a camera arranged in the household appliance, wherein when opening and / or closing a door of the household appliance, the content image is created.

• – wobei das Haushaltsgerät eine Kamera aufweist zum Erstellen eines Inhaltsbilds vom Innenraum des Haushaltsgeräts, wobei das Inhaltsbild dem mobilen Endgerät bereitstellbar ist,
• – wobei das mobile Endgerät – eine Kamera aufweist zum Erstellen einer Aufnahme, – eine Verarbeitungseinheit aufweist zum Erkennen eines Haushaltsgeräts in der Aufnahme und zum Ersetzen des Haushaltsgeräts oder eines Teil des Haushaltsgeräts in der Aufnahme durch das Inhaltsbild vom Inneren des Haushaltsgeräts.

Furthermore, a system is proposed with a household appliance and a mobile terminal,

• - wherein the household appliance has a camera for creating a content image of the interior of the household appliance, wherein the content image is provided to the mobile terminal,
• - wherein the mobile terminal - has a camera for taking a picture, - a processing unit for detecting a household appliance in the receptacle and for replacing the household appliance or a part of the household appliance in the recording by the content image from the inside of the household appliance.

The processing unit mentioned here may in particular be embodied as a processor unit and / or an at least partially hard-wired or logical circuit arrangement, which is set up, for example, in such a way that the method can be carried out as described herein. Said processing unit may include any type of processor or computer or computer necessary peripherals (memory, input / output interfaces, I / O devices, etc.).

The above explanations regarding the method apply to the device or the system accordingly.

Embodiments of the invention are illustrated and explained below with reference to the drawings.

Show it:

1 a schematic arrangement of a kitchen unit with several units, including a refrigerator as an example home appliance and a smartphone that creates a recording of the kitchenette;

2 a screen on the smartphone comprising the kitchen unit with the content picture showing the inside of the refrigerator instead of the refrigerator;

3 by way of example a schematic block diagram comprising a refrigerator with a door, a camera and a processing unit in a symbolic sectional view.

The examples described herein allow a user to display the contents of a home appliance on a terminal when an image taken by the camera of the terminal captures and / or displays the home appliance or a portion of the home appliance.

The terminal is e.g. to a mobile device, a smartphone, a smartwatch, a tablet computer, a portable computer, etc. The terminal has a wireless or a wired connection to a network. The network may e.g. include the Internet or connect to the Internet.

This solution can thus be understood as a form of augmented reality in the field of household appliances. Extended reality is understood according to Wikipedia ( http://de.wikipedia.org/wiki/Erweiterte_Realität ) the computer-aided extension of the perception of reality.

In the present case, a given pattern, e.g. a door of the household appliance, recognized and displayed instead of the door of the household appliance, the contents of the household appliance on the terminal. For this purpose, a recording from the inside of the household appliance can be transmitted to the terminal and displayed on the screen of the terminal e.g. instead of the given pattern. Thus, the pattern, e.g. replaced the door of the household appliance by the inclusion of the contents of the household appliance.

In this case, it is an option that, with a movement of the terminal device, a viewing angle corresponding to the movement on the contents of the household appliance is displayed as a recording. This can e.g. be achieved by different shots from different angles, from different cameras, or by displaying different sections of at least one shot. For example, if the user moves the terminal from top to bottom, the corresponding different areas inside the home appliance may be displayed on the screen. Instead of moving the terminal itself from top to bottom, an input means (e.g., virtual joystick, scrollbar, touch screen, etc.) could also be provided to simulate this movement (or any other movement). It is also possible that e.g. by moving the terminal and / or by the input means a zooming in or out is realized. Basically, a (nearly) free navigation inside the household appliance by means of such input means and / or corresponding movement of the terminal can be realized.

For example, the household appliance may be a refrigerator. The terminal may be, for example, a smartphone. If the user directs the camera of the smartphone to the part of the kitchen in which the refrigerator is located, then the refrigerator is recognized as an object. This item can be replaced by including the contents of the refrigerator (also referred to as the "content image"). The content image has either been previously created or is being created by a camera located in the refrigerator and, e.g. over a network, the smartphone provided. As a result, for example, on the screen of the smartphone, the (or part of) the kitchen is displayed unchanged except for the refrigerator; Instead of the refrigerator, however, the user now looks at the inside of the refrigerator on the basis of the content image and can thus recognize which foods and objects are in the refrigerator. The door of the refrigerator appears by means of this measure of augmented reality as if it were transparent.

Optionally, depending on the size of the detected "fridge" object, the content image can be displayed in different resolutions and / or cut-outs.

• – einem Datum, das angibt, seit wann der Gegenstand in dem Kühlschrank gelagert wird,
• – einem Datum, das zeigt, seit wann der Gegenstand nicht mehr bewegt wurde,
• – eine Dauer bis zu einem Verfallsdatum des Gegenstands,
• – ein Farb-Code für bestimmte Lebensmittel; z.B. können Lebensmittel mit kurzer Haltbarkeit (von wenigen Tagen, z.B. weniger als einer Woche) mit einer roten Farbe oder einer roten Markierung versehen sein,
• – einer Nährwertinformation,
• – einer zweckgebundenen Information, z.B. einer Verwendbarkeit für eine Party,
• – einem Rezept.

The contents image of the interior of the refrigerator may be provided with different additional information, eg

• A date indicating since when the item is stored in the refrigerator,
• - a date that shows since when the item was not moved,
• - a period up to an expiry date of the item,
• - a color code for certain foods; For example, foods with short shelf life (of a few days, eg less than a week) may be provided with a red color or a red marking,
• - nutritional information,
• - purpose-specific information, eg usability for a party,
• - a recipe.

For example, items that are part of a recipe may be marked (or otherwise highlighted) with a color. The user can select a recipe on the smartphone and sees the ingredients in the refrigerator (for example, these may be marked with a green color or a green mark on the screen). The recipes may be automatically offered to the user depending on his preference and / or depending on the time of day. Also, e.g. be noted per recipe, whether the necessary ingredients that would normally be stored in the refrigerator, are also present. In such a case, e.g. the recipes are marked green, for which all the necessary ingredients appear to be present in the refrigerator.

It is also an option to provide an ideal storage location in the refrigerator for an item. For example, it could be so marked on the content image that an apple should be stored in the area intended for fruit.

An advantage of this approach is that the user can efficiently use the contents of the refrigerator in a variety of ways without having to open the refrigerator door each time. As a result, on the one hand saves energy, on the other hand eliminates the annoying and time-consuming search of the refrigerator contents.

1 shows a schematic arrangement of a kitchenette 101 with several units, including a refrigerator 102 as an exemplary household appliance. Furthermore, a smartphone 103 with a display 104 (which is designed as a touch screen, for example) and a camera 105 shown, with the camera 105 on the display 104 opposite side is installed. The user can use the camera 105 the kitchenette 101 take up. The recording can comprise at least one image or a sequence of images, ie a film. In particular, the recording can be created and displayed as a movie in real time.

The smartphone 103 is an example of a mobile device. In the present example, it is connected to a network via a radio interface, for example a mobile radio interface or a WLAN connection. Likewise, the fridge 102 connected to the network. Exemplary infrastructure details are also provided below with regard to 3 explained.

The recording on the smartphone is analyzed and the refrigerator 102 will be in the kitchenette 101 recognized. The detection can be automatic; Also, it is possible for the user to be the one on the smartphone 103 has informed running software, such as the refrigerator 102 looks like (this can be done in the form of a recording of the refrigerator from the front, which is then passed to the software or read from this).

Will the fridge 102 detected, the recognized object "refrigerator" is automatically replaced in the recording at least partially, completely or symbolically by a recording of the contents of the refrigerator. This inclusion of the contents of the refrigerator (hereinafter referred to as "content image") is provided by a camera located in the refrigerator (to distinguish the camera in the refrigerator from the terminal's camera, the former is also referred to as CIF (" camera in fridge ").

So the content image partially overlays the camera's shot 105 , This augmented reality 106 includes the illustration of the kitchenette with a content image 107 , By way of illustration, this augmented reality 106 again in 2 shown enlarged. On the smartphone 103 the user can in the fridge 102 look without having to open the refrigerator door.

3 shows by way of example a schematic block diagram comprising a refrigerator 102 with a door 302 in a symbolic section view. In 3 is the door 302 closed. A sensor 306 Detects if the door 302 open or closed. When opening and / or closing the door 302 the sensor transmits 306 a corresponding signal to a processing unit 304 , The processing unit 304 then controls a camera 303 (CIF), so that these take a picture of the interior of the refrigerator 102 created. By way of example, the processing unit stores 304 the recording in a store 307 , The memory 307 can be part of the processing unit 304 or be carried out separately to this.

The processing unit 304 For example, it can be set up to at least one object 305 in the from the camera 303 detected recording. In 3 are exemplary a variety of objects 305 represented, the items inside the refrigerator, such as goods or food, may be.

For example, it is also possible that the camera 303 and the processing unit 304 as a component. In particular, such a component can have, for example, a wireless network connection and store the recordings made directly on a memory connected via the Internet. Such storage is also referred to as "cloud storage."

In 3 is further exemplified an interface between the processing unit 304 and the smartphone 103 shown. It should be noted that the dashed lines in 3 symbolically represent a transmission of signals or data. The transfer of data to the smartphone 103 can be done for example via a radio interface, which at least partially uses the Internet. For this purpose, the processing unit 304 even have a wireless or wired connection to the Internet.

The smartphone 103 represents another example of a display device. It includes an input option in the form of a (eg virtual) keyboard 321 as well as a screen 104 , keyboard 321 and screen 104 can be combined as a touch screen.

The content image can be viewed at different times from the CIF 303 be included, for example, if the door 302 of the refrigerator 102 closed and / or opened. Also, at predetermined times or on certain occasions (eg as a result of a query) a content image can be created. The content image can be saved in different ways. The store can be in the fridge 102 , in a network or in the mobile terminal 103 be located. Also, the content image can be saved multiple times.

In an exemplary scenario, the refrigerator has 102 via a connection to a network, eg the internet. This connection can be wireless or wired. The smartphone 103 also has a connection to the network (preferably wireless). So can the fridge 102 and the smartphone 103 communicate with each other directly or via the network; For example, the content image can be used by the smartphone 103 to be provided.

If the recording is a real-time recording, the smartphone 103 from the kitchenette 101 created, the refrigerator may be at least partially replaced by the content image or by an object associated with the content image. Depending on an orientation and / or movement of the smartphone 103 a different section of the content image can be displayed or it can be switched between several content images (also extracts of several content images). Thus it can be achieved that the user by the movement of the smartphone 103 the contents of the refrigerator 102 looks from different angles or with different magnifications. Thus, in particular, it can be achieved that the user inside the refrigerator 102 navigated. Optionally, the navigation - in addition to the movement of the smartphone itself - also (possibly alternatively or additionally) by means of an input device (eg a virtual joystick or a touch screen) are performed.

LIST OF REFERENCE NUMBERS

101

 kitchenette

102

 fridge

103

 Smartphone

104

 display

105

 camera

106

 expanded reality

107

 content image

302

 door

303

 Camera (CIF)

304

 processing unit

305

 object

306

 sensor

307

 Storage

321

 keyboard

QUOTES INCLUDE IN THE DESCRIPTION

This list of the documents listed by the applicant has been generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited non-patent literature

• http://en.wikipedia.org/wiki/Advanced_Reality [0038]

Claims (12)

Method comprising the steps: Create a recording by means of a mobile terminal, Detecting a household appliance in the receptacle, - Replacing the household appliance or a part of the household appliance in the recording by a content image of the interior of the household appliance. The method of claim 1, wherein the receptacle comprises the household appliance. Method according to one of the preceding claims, wherein the step is carried out before replacing the household appliance or the part of the household appliance in the receptacle: - Creating the content image of the interior of the household appliance. Method according to one of the preceding claims, wherein the content image comprises at least one information that can be selected and displayed on the basis of the mobile terminal. The method of claim 4, wherein the information comprises at least one of A date indicating since when the item is stored in the refrigerator, - a date that shows since when the item was not moved, - a period up to an expiry date of the item, - a color code for certain foods, - nutritional information, - a specific information, - a recipe. Method according to one of claims 4 or 5, wherein based on the information, a predetermined action is executable. The method of claim 6, wherein the predetermined action comprises at least one of - triggering an order; Indication of whether a recipe or a dish is prepared or not; - Selection of recipes that are preparable based on the at least one information; - instruction to place an item in the household appliance. Method according to one of the preceding claims, comprising the step: Replacing the home appliance or a part of the household appliance in the recording by the content image, wherein the content image is selected depending on a navigation information. The method of claim 8, wherein the navigation information is provided by means of the mobile terminal, so that a virtual navigation inside the household appliance is made possible. Method according to one of the preceding claims, in which the domestic appliance is one of the following appliances: - a kitchen appliance, - a cooking appliance, - a refrigerator and / or freezer. Method according to one of the preceding claims, wherein the content image is created from the interior of the household appliance by means of a camera arranged in the household appliance, wherein when opening and / or closing a door of the household appliance, the content image is created. System with a household appliance and a mobile terminal, - wherein the household appliance has a camera for creating a content image of the interior of the household appliance, wherein the content image is provided to the mobile terminal, - the mobile terminal - has a camera to take a picture, - A processing unit for detecting a household appliance in the recording and replacing the household appliance or a part of the household appliance in the recording by the content image from the inside of the household appliance.

Images