EP3742051B1 - Lighting control of food products - Google Patents

Description

The invention relates to a method for automatic exposure control in a household appliance with an integrated camera, in particular for items to be cooked in cooking chambers of cooking appliances with an integrated camera, and a corresponding cooking appliance.

Camera systems are currently being integrated into networked household appliances in order to capture images, e.g. from the cooking chamber of an oven or the interior of a refrigerator, and to send them to a mobile end device (e.g. a smartphone). The images are usually transmitted from the home appliance to the end device via a local network or via a cloud server owned by the home appliance manufacturer. The images are then displayed, for example, in a special app or, with a web server implemented on the device, using a browser on a mobile device. In order to adapt the recording quality to different objects, the exposure time of the camera must be adjusted according to the object to be imaged. Various strategies are already known from digital photography and are used there. These include center-weighted exposure control, spot metering, etc.

Although these lighting controls, which are known in principle, can be used in the cooking chamber of the oven, they often do not deliver satisfactory results or do not reliably deliver satisfactory results. The reason for this is that a user is free to place the food to be cooked in the cooking chamber and the minimum or maximum size as well as the shape and number of food to be cooked can be very variable. In the worst case, when using conventional exposure methods, the actually interesting food would be overexposed and the uninteresting food carrier, e.g. a baking tray, would be optimally exposed. Poorly exposed images of items to be cooked are disadvantageous for the user during use, since he cannot correctly identify the degree of doneness in the course of a cooking program, for example. He will also expect optimal image quality from a high-quality cooking appliance with an integrated camera system, which, however, cannot be reliably ensured with conventional exposure methods.

State of the art is the documents WO2018033383 and US2011002677 refer to.

The invention therefore addresses the problem of providing an improved exposure method for a cooking appliance with a camera. According to the invention, this problem is solved by a method and a cooking appliance according to the independent claims. Advantageous refinements and developments of the invention result from the dependent claims.

• Aufnehmen eines Startbilds mit einer vorgegebenen Grundbelichtungszeit;
• Segmentieren des Startbilds in Gargut-Flächenanteile und Nicht-Gargut-Flächenanteile;
• Erstellen einer Belichtungs-Maske für die Gargut-Flächenanteile, welche die Nicht-Gargut-Flächenanteile ausblendet;
• Anpassen der Belichtungszeit unter Verwendung der Belichtungs-Maske, so dass eine vorgegebene Zielhelligkeit der Gargut-Flächenanteile erreicht wird; und
• Aufnehmen eines Zielbilds mit der angepassten Belichtungszeit.

According to a first aspect, a method for controlling exposure in a cooking appliance with an integrated camera is provided, comprising:

• taking a start image with a predetermined basic exposure time;
• segmenting the start image into cooked product area portions and non-cooked product area portions;
• Creation of an exposure mask for the cooking product areas, which hides the non-cooking product areas;
• Adjusting the exposure time using the exposure mask so that a predetermined target brightness of the cooking product surface portions is achieved; and
• Capture a target image with the adjusted exposure time.

This ensures that the food to be cooked is exposed correctly, regardless of where the user places the food in the cooking chamber/on the food support and which food support he uses. The use of baking paper, baking foil and the like, which can normally influence the exposure due to their color, can also be compensated with this.

• Randerkennung, um einfach zusammenhängende Gebiete zu erkennen;
• Berücksichtigung von Farbwerten, wobei Gebiete ähnlicher Farbe bevorzugt als zusammenhängende Gebiete eingestuft werden;
• Berücksichtigung einfach zusammenhängender Gebiete, wobei einfach zusammenhängende Gebiete bevorzugt als Gargut-Flächenanteile eingestuft werden und nicht einfach zusammenhängende Gebiete bevorzugt als Nicht-Gargut-Flächenanteile eingestuft werden;
• Berücksichtigung des Abstands von Gebieten von der Mitte und/oder dem Rand des Gargutträgers, wobei Gebiete in der Mitte bevorzugt als Gargut-Flächenanteile eingestuft werden und Flächenanteile am Rand bevorzugt als Nicht-Gargut-Flächenanteile eingestuft werden;
• Berücksichtigen der Form und Farbe bekannter Gargutträger, wobei Gebiete mit dieser Form und Farbe bevorzugt als Nicht-Gargut-Flächenanteile eingestuft werden;
• Erkennung von Objektstrukturen im Bild und dadurch Unterscheidung zu weniger strukturierten Nicht-Gargut-Flächenanteilen (z.B. mathematische Ableitung des Bildes).

There are various options for segmenting into cooked food surface areas and non-cooked food surface areas, which can be used individually or in any combination:

• edge detection to easily detect contiguous areas;
• Consideration of color values, with areas of similar color being preferentially classified as contiguous areas;
• Taking into account simply contiguous areas, where simply contiguous areas are preferably classified as portions of the cooking area and areas that are not simply contiguous are preferably classified as portions of the area that are not to be cooked;
• Taking into account the distance of areas from the center and/or the edge of the food support, areas in the center preferably being classified as portions of the surface of the food to be cooked and portions of the surface at the edge being preferably classified as portions of the surface not being cooked;
• Taking into account the shape and color of known food supports, areas with this shape and color being preferably classified as non-cooking surface portions;
• Recognition of object structures in the image and thus differentiation from less structured non-cooking surface areas (e.g. mathematical derivation of the image).

In an exemplary embodiment, it can Segmenting takes place as follows: A border or edge operator generates a mask, the borders or edges in the image being recognized using methods known per se, for example by jumps in brightness/color changes at the edges of objects. These are then blurred, and the blurred contours are closed into larger figures using a dilation operator.

Many items to be cooked, such as roasts, cakes and the like, simply form contiguous areas. Therefore, simply contiguous areas that are located in the center or at a distance from the edge of the cooking space/food support can be recognized preferably as cooking product surface portions. Conversely, an area that is not simply connected, which extends to the edge of the cooking space/food support, can preferably be recognized as a non-cooking surface portion. Exceptions are items to be cooked that are not simply connected, such as donuts, pretzels and the like. This can be compensated for, for example, by additionally considering the position, i.e. more at the edge or more in the middle. Or the combination of an inner, simply connected region with an outer, not simply connected region, both of which together in turn form a simply connected region, can preferably be recognized as a portion of the surface of the cooking product.

• Ermitteln eines ersten Durchschnittshelligkeitswerts der Bildpunkte der Gargut-Flächenanteile; und
• Ermitteln, abhängig von der Grundbelichtungszeit und dem ersten Durchschnittshelligkeitswert, der Belichtungszeit für die vorgegebene Zielhelligkeit.

Adjusting exposure time includes:

• determining a first average brightness value of the pixels of the cooking product surface portions; and
• Determine the exposure time for the specified target brightness, depending on the basic exposure time and the first average brightness value.

This can be done, for example, using a look-up table (LUT) that links exposure times and target brightness.

In a preferred embodiment, the pixels of the surface portions of the cooking product whose brightness lies outside a predefined brightness range are not taken into account when determining the first average brightness value.

In this way it can be achieved that pixels that are already overexposed or underexposed in the basic exposure of the start image do not falsify further processing. These pixels are therefore not taken into account when determining the first average brightness value.

• Vorgeben der Zielhelligkeit in Abhängigkeit von dem ersten Durchschnittshelligkeitswert.

In a preferred, alternative embodiment, the method further comprises:

• Specification of the target brightness as a function of the first average brightness value.

This can be used to distinguish between light (muffins, cake batter, casserole) and dark food (chocolate cake, roast). Adjusted target brightness levels can be specified depending on whether the food is light or dark. This prevents the food being cooked from being over- or underexposed due to its colour.

• Ermitteln eines zweiten Durchschnittshelligkeitswerts der Bildpunkte der Nicht-Gargut-Flächenanteile; und
• Vergleichen des ersten mit dem zweiten Durchschnittshelligkeitswert und Vorgeben der Zielhelligkeit in Abhängigkeit davon.

In a preferred embodiment, the method further comprises:

• determining a second average brightness value of the pixels of the non-cooking product surface portions; and
• Comparing the first with the second average brightness value and specifying the target brightness depending thereon.

This comparison enables an even more precise differentiation between light and dark items to be cooked. However, this embodiment depends on the type and color of the non-cooking surface portions, so it works best if the color of the non-cooking surface portions, ie the cooking space or the food support, is known. This is easy to ensure in the case of food supports associated with the household appliance, such as trays and grates.

In a preferred embodiment, the steps of adjusting the exposure time and recording a target image are repeated if the actual brightness of the cooking product surface portions of the target image deviates from the target brightness by more than a threshold value.

In this way, the specified target brightness can be maintained as precisely as possible. Alternatively, to increase the speed of the method, an iterative approach to the target brightness can be dispensed with, with deviations from the specified target brightness being accepted.

• einen Garraum;
• eine integrierte Kamera, die dazu eingerichtet ist, Bilder vom Garraum aufzunehmen; und
• eine Steuereinheit, die mit der Kamera verbunden und dazu eingerichtet ist, ein Verfahren wie vorstehend beschrieben auszuführen.

According to a further aspect, a cooking appliance is provided, comprising:

• a cooking chamber;
• an integrated camera configured to capture images of the cooking cavity; and
• a control unit connected to the camera and configured to carry out a method as described above.

In a preferred embodiment, the cooking chamber can be closed with a door and the control unit is also set up to carry out the method in response to the door being closed.

Since placing items to be cooked in such a household appliance is inevitably connected with the closing (and possibly previous opening) of the door, it makes sense to make the method dependent on the closing.

In a preferred embodiment, the control unit is further set up to store the adjusted exposure time in response to the start of a cooking program and to use the stored exposure time for further images during the execution of the cooking program.

Alternatively or additionally, the start of a cooking program can be used as a trigger for the method. A combination is also possible, such as starting the method by recognizing that the door is closing, and storing the exposure mask produced for use during the cooking program. While the cooking program is running, door openings and closings can be used to (re)start the process. Alternatively, the saved exposure mask can be retained throughout the entire cooking program.

Figur 1

ein Flußdiagramm einer Ausführungsform des Verfahrens der Erfindung;

Figur 2

eine schematische Darstellung einer Ausführungsform eines Haushaltsgeräts der Erfindung; und

Figur 3

eine schematische Darstellung von Teilschritten einer Ausführungsform des Verfahrens der Erfindung.

Embodiments of the invention are shown purely schematically in the drawings and described in more detail below. It shows

figure 1

a flow chart of an embodiment of the method of the invention;

figure 2

a schematic representation of an embodiment of a household appliance of the invention; and

figure 3

a schematic representation of partial steps of an embodiment of the method of the invention.

figure 1 Figure 12 shows, in flow chart form, steps of an embodiment of the invention. In step 102, a start image of the item to be cooked in the cooking space or on the item to be cooked (such as a baking tray or grid) is recorded with a basic exposure time.

In step 104, segmentation is performed on the starting image. The aim of the segmentation is to divide the start image into portions of the surface of the food to be cooked and portions of the surface that are not to be cooked. There are various options for segmentation, either individually or in combination can be used. A preferred variant is associated with figure 3 are explained below.

In step 106, based on the result of the segmentation, an exposure mask for the cooking product surface portions is created, which hides the non-cooking product surface portions.

In step 108 a target brightness is specified. This can optionally be made dependent on whether the cooking product surface portions are light or dark cooking products. The distinction between light and dark can be based on an absolute brightness value of the cooking product area portions or on a brightness difference between cooking product surface portions and non-cooking product surface portions. The specification can be made here by selecting from a look-up table (LUT).

In step 110, the exposure time is adjusted using the exposure mask, so that the specified target brightness of the surface portions of the cooking item is achieved. This can be done by means of calculation/extrapolation if the dependence on brightness and exposure time is known, or by means of a look-up table (LUT).

In step 112, a target image is captured with the adjusted exposure time. In a simple variant, the method can be ended here, i.e. the target image is used further, e.g. displayed on the user's mobile device and/or the display of the relevant household appliance.

In an optional variant, a step 114 follows in which the actual brightness of the target image is checked. If the actual brightness matches the target brightness, the target image is used. However, if the actual brightness does not match the target brightness, steps 110 and 112 are repeated, preferably until there is a match between the actual brightness and the target brightness.

In figure 2 a cooking appliance 10, such as an oven, is shown schematically according to one embodiment. The cooking appliance 10 has a cooking chamber 6 which can be closed with a door 14 . A camera 12 is provided to record items to be cooked. The camera 12 does not necessarily have to be arranged in the cooking chamber ceiling as shown here, but can in principle be arranged in any way, so that the food to be cooked can be photographed from all sides.

For the sake of simplicity, it is shown here that an item 4 to be cooked, such as a piece of meat, is lying on the cooking chamber floor. However, the cooking appliance usually has a holder for one Food support such as a baking tray or a grate, so that food to be cooked is then placed on the (not shown here) food support. The cooking appliance 10 has a control unit 8 which is connected to the camera 12 and is set up to carry out an exposure control method as disclosed in this description.

In figure 3 partial steps of an embodiment are shown, relating to the segmentation into cooking product surface portions and non-cooking product surface portions. In the upper part of the figure, a food support 2 with a food 4 to be cooked is shown schematically. An edge operator is applied to the image captured by the camera. Methods known per se can be used as the edge operator, such as the Sobel, Scharr, Laplace, Prewitt, Roberts, Kirsch, Canny, Marr-Hildreht algorithms, contrast enhancers, active contours or the like. In a simple variant, the contrast differences, which are usually greatest at edges, are used.

Using an edge operator, an image can be obtained in the bottom row on the left (shown filled here). The edges are softened in a next step, so that an image like that in the lower row is obtained in the middle. In the final step, a dilation operator is applied to obtain an image as shown in the bottom row on the right. The exposure mask can now be created from this image, with the areas shown here in black corresponding to portions of the surface of the food to be cooked and the areas shown in white to portions of the surface of the food not to be cooked.

Claims (8)

1. Method for controlling exposure in a cooking appliance that has an integrated camera, the method comprising:

   - recording a start image with a specified basic exposure time;

   - segmenting the start image into food surface portions and non-food surface portions;

   - creating an exposure mask for the food surface portions, which exposure mask hides the non-food surface portions;

   - adjusting the exposure time using the exposure mask, so that a specified target brightness of the food surface portions is achieved; and

   - recording a target image with the adjusted exposure time;

   wherein the segmentation of the start image into food surface portions and non-food surface portions involves one or more of the following steps:

   - edge detection, for detecting singly contiguous regions;

   - taking into account colour values, wherein regions of similar colour are preferably classified as contiguous regions;

   - taking into account singly contiguous regions, wherein singly contiguous regions are preferably classified as food surface portions and non-singly contiguous regions are preferably classified as non-food surface portions;

   - taking into account the distance of regions from the centre and/or the edge of the food support, wherein regions in the centre are preferably classified as food surface portions and surface portions at the edge are preferably classified as non-food surface portions;

   - taking into account the shape and colour of known food carriers, wherein regions having this shape and colour are preferably classified as non-food surface portions;

   - detecting the structure of the food and distinguishing from non-food surface content;

   and wherein adjusting the exposure time comprises:

   - determining a first average brightness value of the pixels of the food surface portions; and

   - determining, on the basis of the basic exposure time and the first average brightness value, the exposure time for the specified target brightness.
2. Method according to claim 1, wherein the pixels of the food surface portions of which the brightness is outside a specified brightness range is not taken into account when determining the first average brightness value.
3. Method according to either claim 1 or claim 2, further comprising:

   - determining a second average brightness value of the pixels of the non-food surface portions; and

   - comparing the first with the second average brightness value and specifying the target brightness on the basis thereof.
4. Method according to either claim 1 or claim 2, further comprising:

   - specifying the target brightness on the basis of the first average brightness value.
5. Method according to any of the preceding claims, wherein the steps of adapting the exposure time and recording a target image are repeated if the actual brightness of the food surface portions of the target image deviates by more than one threshold value from the target brightness.
6. Cooking appliance, comprising:

   - a cooking chamber;

   - an integrated camera configured to record images of the cooking chamber; and

   - a control unit, which is connected to the camera and configured to carry out a method according to any of claims 1 to 5.
7. Cooking appliance according to claim 6, wherein the cooking chamber can be closed by a door and the control unit is further configured to carry out the method in response to the door being closed.
8. Cooking appliance according to either claim 6 or claim 7, wherein the control unit is further configured to store the adjusted exposure time in response to the start of a cooking program and to use the stored exposure time for further images when the cooking program is being carried out.

Images