DE102022210321A1 - Identifying food to be cooked in a thermal image - Google Patents

Abstract

The invention relates to a method (S1-S9) for identifying food (G) to be cooked in at least one pixel-based thermal image (W1, W2) recorded from a cooking chamber (2) of a microwave cooking appliance (1), in which microwave energy is radiated into the cooking chamber (2), a thermal image (W2) is recorded, pixels are identified which have temperature values increased by a predetermined temperature level in relation to the set of all pixels, and a filling algorithm is executed which assigns pixels to the food (G) which belong to an image area bordered by at least some of the identified pixels. The invention also relates to a microwave cooking appliance (1) having a cooking chamber (2) to which microwaves can be applied, a thermal imaging camera (11) provided for recording thermal images (W1, W2) from the cooking chamber (2), and a control device (13), wherein the microwave cooking appliance (1) is set up to carry out the method (S1-S9; S1-S10). The invention is particularly advantageously applicable to household microwave cooking appliances.

Description

The invention relates to a method for identifying food to be cooked in at least one pixel-based thermal image recorded from a cooking chamber of a microwave cooking appliance, in which microwave energy is radiated into the cooking chamber, a thermal image is recorded and pixels are identified which have temperature values increased by a predetermined temperature level in relation to the set of all pixels. The invention also relates to a method for operating a microwave cooking appliance, in which a position of the food to be cooked in a thermal image is determined using the method and a cooking process is controlled using an evaluation of the pixels assigned to the food to be cooked. The invention further relates to a microwave cooking appliance, having a cooking chamber to which microwaves can be applied, a thermal imaging camera provided for recording thermal images from the cooking chamber and a control device, wherein the microwave cooking appliance is set up to carry out the method. The invention is particularly advantageously applicable to household microwave cooking appliances.

For advanced control of cooking processes in microwave ovens to improve preparation results based on image-based infrared (IR) or When heat radiation measurements are being carried out, precise determination of the food to be cooked is important. This means in particular that a precise distinction should be made as to whether a pixel in the image of the viewing area of a thermal imaging camera is assigned or belongs to a sub-area of the food to be cooked or - irrelevant for the control of the cooking process - to an environment of the food to be cooked, such as a food support (e.g. plate, grill rack, baking tray...) or a cooking chamber wall.

The identification of an area assigned to the food in an image of the cooking chamber can be done by a user by manually marking it on a touch-sensitive screen ("touchscreen"). However, this is not very user-friendly and is often inaccurate. Complex image analysis methods, particularly based on artificial intelligence ("AI") methods, can also be used to automatically recognize the area of the food in an image. However, this is very computationally and training-intensive.

WO 2020/156928 A1 reveals a household cooking appliance. The household cooking appliance has a cooking space heating device which is set up for localized heating of a cooking space and which can be operated with at least two configurations which generate different energy distributions in the cooking space, a temperature detection device which is set up for contactless detection of a heat distribution in the cooking space, and a data processing device , which is set up to distinguish a non-food area from at least one area of the cooking space occupied by food from the detected heat distribution and a control device which is used to set a current configuration of the cooking space heating device with a view to increasing an energy release into the detected food and for Controlling the cooking space heating device, the temperature detection device and the data processing device is set up, wherein the data processing device is set up to distinguish the non-cooked food area from the cooked food in the detected heat distribution based on a temperature difference between the non-cooked food area and the cooked food.

WO 2020/156929 A1 discloses a microwave oven. The microwave device has a microwave device which is set up to generate microwaves and to introduce the microwaves into a cooking space and which can be operated with at least two configurations which generate different field distributions of the microwaves in the cooking space, a temperature detection device which is used to detect a heat distribution in a contactless manner the cooking chamber is set up, a data processing device that is set up to recognize a non-food area in the cooking chamber from the detected heat distribution and a control device that is set up to set a current configuration of the microwave device and to operate the microwave device, whereby the Control device is set up to select or adjust at least one configuration of the microwave device with a view to reducing the power of the microwaves in the detected non-cooked food area.

EP 3 767 580 A1 discloses a control unit for a household appliance that has at least one reference mark in an interior, the reference mark having reference values for one or more different properties. The control unit is set up to capture image data relating to the interior of the household appliance using a camera of the household appliance and to identify the reference mark in the image data. Furthermore, the control unit is set up to determine actual values for the one or more properties of the reference mark based on the image data. The control unit is further set up to process the image data depending on the actual values and depending on the reference values in order to determine object information relating to an object in the interior of the household appliance and/or to create an object to provide an evaluated image in relation to the interior of the household appliance.

EN 10 2017 101 183 A1 discloses a method for operating a cooking appliance and a cooking appliance in which food is heated in a cooking chamber using a heating device. The food is recorded using a camera device. At least one cooking parameter is determined based on the recording of the food. The heating device comprises a heat source with a plurality of separately controllable heating means. One spatial segment of a plurality of spatial segments in the cooking chamber is specifically heated using at least one heating means. The individual heating means are controlled depending on the cooking parameter.

WO 2016 170 734 A1 discloses a cooking apparatus comprising: a microwave generating unit that generates microwaves; a heating chamber for accommodating an object to be heated; an infrared sensor installed inside the heating chamber; a scanning unit that moves the infrared sensor for scanning; and a control unit that controls the microwave generation unit based on the output of the infrared sensor. The infrared sensor obtains a variety of temperature distributions by obtaining a temperature distribution each time the infrared sensor is scanned over a predetermined distance. The control unit controls the microwave generation unit according to a temperature distribution obtained by summing the plurality of temperature distributions.

US 10 219 330 B2 discloses an electronic oven and an accompanying control system that avoids boiling or splattering in a heating chamber of the oven while heating an object in the chamber. A method that may be performed by the control system includes evaluating sensor data from a visible light sensor and sensor data from an infrared light sensor. The controller is communicatively coupled to the visible light sensor and the infrared light sensor. The method is directed to generating a splatter prediction in response to evaluating the sensor data from the visible light sensor and the sensor data from the infrared light sensor. The method is further directed to reducing a power level of the microwave energy source in response to the splatter prediction. The controller is also communicatively coupled to the microwave energy source.

EP 2 618 634 A1 discloses a microwave heating device and a method for heating a load using microwaves. The microwave heating device comprises a cavity arranged to receive a load, a plurality of supply openings for supplying microwaves from a plurality of microwave generators to the cavity, and a control unit. The control unit is configured to obtain a desired temperature pattern within the cavity based on information about a plurality of regions of the load, determine a heating pattern comprising corresponding zones of different intensity to adjust to the desired temperature pattern, and control at least some of the plurality of microwave generators to provide the heating pattern within the cavity.

WO 2012/109634 A1 discloses an apparatus for processing objects with RF energy. The apparatus may include a display for displaying to a user an image of an object to be processed, the image including at least a first portion and a second portion of the object. The apparatus may also include an input unit and at least one processor configured to: receive information based on an input provided to the input unit; and generate processing information based on the received information for use in processing the object to achieve a first processing result in the first portion of the object and a second processing result in the second portion of the object.

EP 1 997 349 B1 discloses an electromagnetic heating device for heating an irregularly shaped object, comprising: a cavity in which an object is to be placed; at least one feeder that delivers UHF or microwave energy to the cavity; and a controller that controls one or more characteristics of the cavity or the energy to ensure that the UHF or microwave energy is delivered to the object evenly within ±30% over at least 80% of the volume of the object.

WO2020/200913 A1 discloses a household appliance. The household appliance has a treatment room for treating goods, in particular food to be cooked, at least one pattern lamp which is designed to irradiate at least one light pattern into the treatment room, and at least one image sensor directed into the treatment room for recording the at least one light pattern reflected from the treatment room, wherein the pattern lamp can be rotated by means of a motor and the household appliance is designed to determine at least one contour information of the material irradiated by the light pattern from at least two reflected light patterns belonging to different angles of rotation of the at least one pattern lamp. A method is used to determine contour information of goods located in a treatment room of a household appliance. WO2020/200913 A1 is particularly advantageously applicable to determining contour information of food to be cooked in an oven.

EP 3 574 711 A1 discloses methods and systems relating to improved human-machine interfaces for electronic ovens. Various methods of displaying information to the user are disclosed. Various methods for dividing segmentation and identification tasks between a user and a control system are disclosed. In one example, an electronic oven includes a touch display, a heating chamber for heating an object, a light sensor having a field of view of at least a portion of the heating chamber, and a microwave energy source coupled to the heating chamber. The oven also includes a computer-readable medium that stores instructions for displaying the portion of the heating chamber as an image on the touch display using information from the light sensor and processes touch input on the image.

EP 3 767 183 A1 discloses a method for detecting contamination of a household appliance. The method comprises the steps of providing a reference image of a cavity or section of the household appliance, capturing a current image of the same cavity or section, comparing the current image with the reference image and generating a difference image of the same cavity or section and checking whether a pixel and/or a group of neighboring pixels of the difference image exceeds a predefined threshold value.

EP 2 055 146 B1 discloses how by showing how the geometric shape of an object can be determined by measuring with RF energy.

DE 10 2020 215 681 A1 discloses a household microwave appliance that is operated successively under several parameter configurations that treat the food to be cooked locally differently in order to carry out an initial scan by means of a thermal image sensor directed into the cooking space to determine temperature distributions on a surface of the food to be cooked in order to obtain change patterns from differences in different temperature distributions , for which an evaluation value is calculated which, based on a target temperature distribution, which results from a standardized target state and a current temperature distribution, determines the heating pattern which best approximates the current temperature distribution to the target temperature distribution and, as a result, that Microwave power is applied to the food to be cooked with the parameter configuration associated with the heating pattern.

It is the object of the present invention to at least partially overcome the disadvantages of the prior art and in particular to provide an improved possibility of recognizing food to be cooked in a pixel-based image taken from a cooking chamber of a microwave cooking appliance.

This object is achieved according to the features of the independent claims. Preferred embodiments can be found in particular in the dependent claims.

• - Mikrowellenenergie in den Garraum eingestrahlt wird,
• - ein Wärmebild aufgenommen wird,
• - Bildpunkte identifiziert werden, die in Bezug auf die Menge aller Bildpunkte um ein vorgegebenes Temperaturniveau erhöhte Temperaturwerte aufweisen,
• - ein Füllalgorithmus ausgeführt wird, der Bildpunkte dem Gargut zurechnet, die zu einer durch zumindest einige der identifizierten Bildpunkte umrandeten Bildfläche gehören.

The task is solved by a method for identifying food to be cooked in at least one pixel-based thermal image recorded from a treatment room (“cooking room”) of a microwave oven, in which

• - microwave energy is radiated into the cooking space,
• - a thermal image is recorded,
• - pixels are identified that have temperature values that are increased by a predetermined temperature level in relation to the amount of all pixels,
• - A filling algorithm is carried out, which assigns pixels to the food to be cooked that belong to an image area surrounded by at least some of the identified pixels.

This process has the advantage that - for example, in contrast to manual marking on a touchscreen or automated using Kl-image recognition methods - it is very precise, user-friendly, requires little computation and is therefore very fast and also runs automatically.

The method uses the so-called “edge overheating effect”, in which a heating image shows the frequently occurring behavior that the edges of the food are heated up strongly in some places when exposed to microwaves, especially initially, while the inner area of the food surface usually heats up significantly less or hardly at all. The “edge overheating effect” occurs with practically all food containing water. In principle, this effect makes the cooking process more difficult, as there is an increased energy input at the edges, while the center heats up only with difficulty. However, this phenomenon now enables a possibility with little computational effort to record the outer contour of the food and use this to distinguish between pixels belonging to the food and pixels not belonging to the food. This assignment can also be referred to as "area masking".

The identification of food in a thermal image corresponds in particular to the allocation of the image points as to whether a temperature is recorded on a surface of the food or - conversely - a temperature not on a surface of the food but, for example, a temperature on a surface of another object such as a food carrier, a cooking chamber wall, etc. The identification therefore corresponds in particular to identifying the position of the food, not its type.

The cooking chamber is used to accommodate items to be treated, particularly cooked, using microwaves. The cooking chamber can be exposed to microwaves. It often has a loading opening at the front that can be closed off microwave-tight using a cooking chamber door.

The microwaves are generated using at least one microwave generator, for example a magnetron or a semiconductor-based microwave generator. The microwaves can be introduced into the cooking chamber directly from the microwave generator or via a respective microwave guide. It is a further development that microwaves are introduced into the cooking space via a microwave feed point (“microwave port”) or via several microwave ports, if necessary phase-shifted in the case of several microwave ports, in particular with a variably adjustable phase shift. Rotatable rotary antennas and/or stirrers can also be present at the microwave port in order to vary the mode pattern of the microwaves in the cooking chamber. In addition, the fashion pattern in the area of the food to be cooked can be varied by providing a turntable, if available. The frequency of the microwaves can be in the range between 2.4 GHz and 2.5 GHz, in particular around 2.45 GHz, or in the range between 902 MHz and 928 MHz, in particular around 915 MHz. In particular, a semiconductor-based microwave generator makes it possible to vary a frequency of the microwaves in a targeted manner.

The pixel-based thermal image is in particular composed of a matrix-shaped arrangement of pixels, whereby the values associated with the pixels correspond to a temperature in the surface area measured by the respective pixels.

The pixel-based thermal image is typically recorded by a digital thermal imaging camera. In order to capture the food to be cooked over a large area, the thermal imaging camera is advantageously arranged in the area of a ceiling of the wall of the cooking space and is oriented in particular downwards or obliquely downwards. A field of view of the thermal imaging camera advantageously includes at least the bottom of the cooking chamber wall, possibly also parts of the sides of the cooking chamber wall.

The microwave oven can also have several thermal imaging cameras, which advantageously take images of the cooking space from different angles.

It is a further development that the microwave cooking appliance has, in addition to the at least one digital thermal imaging camera, at least one digital "optical" cooking chamber camera that takes images in the optical spectral range. This is advantageous in order to be able to control a cooking process based on an optical property of the surface of the food being cooked, such as a degree of browning, a color change, a change in volume, etc. The images taken by this can also be displayed to a user. The optical cooking chamber camera can be a digital RGB color camera, for example. It is a further development that is advantageous for capturing the food over a large area that the optical cooking chamber camera is arranged in the area of a ceiling of the wall of the cooking chamber and is directed downwards or diagonally downwards in particular. It is a further development that the thermal imaging camera and the optical cooking chamber camera are arranged close to one another, which has the advantage that the thermal images and the optical images show particularly similar areas of the cooking chamber and are particularly easy to compare.

By radiating microwave energy into the cooking chamber, the food inside is heated, whereby the edge of the food is not only heated more than its outer "non-food" surroundings, but due to the "edge overheating effect" it is also usually heated more than the middle of the food. After a certain irradiation period, the thermal image of the cooking chamber is recorded, which typically shows a temperature distribution of the surface of the food.

All pixels of the thermal image have corresponding temperature values, whereby at least some pixels associated with the edge of the food being cooked can be identified by the fact that they have a significantly higher temperature level or show a significantly higher temperature than the other pixels. The pixels with a higher temperature level can be differentiated from other pixels and thus identified by comparing threshold values, for example. These Pixels with a higher temperature level show at least sections of the edge or the outer contour of the food being cooked from the perspective of the thermal imaging camera.

The filling algorithm then fills the area in the image plane that is limited by the pixels assigned to the edge of the food or identifies the pixels that lie in such an area. After the filling algorithm has been carried out, those pixels that lie in the area, including its edge, are assigned to the food. Conversely, those pixels that lie outside the area, including its edge, are not assigned to the food.

It is a further development that the filling algorithm is designed in such a way that pixels previously assigned to the edge of the food are not assigned to the area corresponding to the food after the filling algorithm has been carried out, but to the non-food area, because they did not satisfy certain criteria of the filling algorithm for assignment to the area.

One embodiment involves recording a first thermal image before the microwave energy is radiated into the cooking chamber, recording a second thermal image after the microwave energy is radiated into the cooking chamber, creating a differential thermal image in which a temperature difference between the temperature value recorded with the second thermal image and the temperature value recorded with the first thermal image is calculated for each of the pixels, and identifying the pixels in the differential thermal image. This has the advantage that after the cooking chamber has been loaded with the food, existing temperature differences between the food and the area surrounding the food or even within the food are taken into account and their influences are neutralized.

One embodiment is that the temperature differences are mapped or "normalized" to a value range [0; 1] (or other specified value range). This has the advantage that the method can be more easily adapted to different types of food and/or to different microwave operating parameters during microwave feed.

It is an embodiment that those pixels that reach or exceed a predetermined temperature threshold are identified as assigned to the edge of the food to be cooked. This is advantageously particularly easy to implement.

It is an embodiment that when the temperature differences are within a value range [0; 1], the temperature threshold value is in a range [0.2; 0.5]. This has proven to be a particularly good compromise between precise detection of the edge of the food to be cooked and a sufficiently closed edge. If the value of the temperature threshold value is too low, for example 0.1, the food G to be cooked is usually completely detected, but then large areas of the food support or the cooking chamber wall may also be incorrectly recognized as belonging to the food to be cooked. If the temperature threshold value is chosen too high, e.g. 0.7, only individual, particularly heated edge segments of the food G are detected, which may be too little to reliably define an edge that is filled by the Starfill algorithm. In addition, it is advantageous that by mapping the range of values to a fixed range, the same threshold value can be appropriately applied to many different items to be cooked and/or different microwave operating parameters.

One embodiment uses an algorithm (hereinafter referred to as the "starfill" algorithm without loss of generality) in which a pixel is assigned to the food if a number of directions radiating out from this pixel in a star shape, in which identified pixels are located, reaches or exceeds a minimum number. In particular, in contrast to the so-called "floodfill" algorithm (which is often used in graphics programs to fill in bordered areas with color), this avoids gaps in the identified edge leading to areas outside the edge also being assigned to the food. In addition, the "floodfill" algorithm requires one of the inner points of the bordered area as a starting point, which are, by definition, unknown at the start of the evaluation.

The starfill algorithm checks for an image point under consideration whether image points assigned to the food to be cooked lie in different directions in the image plane of the thermal image, in particular the differential heat image, which emanate from this image point in a star shape. The image point under consideration is then assigned to the food to be cooked if a predetermined minimum number of directions meet this condition. The Starfill algorithm can carry out this check and subsequent assignment for all image points of the thermal image, in particular the differential thermal image, for example row and column wise.

In a (x, y) matrix-shaped thermal image, directions radiating “star-shaped” from a pixel refer to the symmetrical directions along the x-extension and the y-extension as well as, if applicable, the oblique directions lying evenly between these (main) extensions.

• - in x-Richtung,
• - in (-x)-Richtung,
• - in y-Richtung und
• - in (-y)-Richtung.

The smallest number of star-shaped directions for an interior point within the (x, y) matrix is four, namely starting from this image point

• - in X direction,
• - in X direction,
• - in y-direction and
• - in (-y) direction.

The next largest number of star-shaped directions is eight, namely the additional four directions starting from this pixel through the nearest neighboring pixels, which lie obliquely to the x and y extensions. This can basically be extended to 12, 16, 20 etc. directions. The higher the number of directions, the higher the accuracy of the assignment on slanted edges of the edge.

A reduced number of possible directions are available for edge points and corner points, namely two directions for corner points and three directions for edge points, if general - for interior points - four directions are available, three directions for corner points and five directions for edge points, if general - for Interior points - eight directions available, etc.

It is a design that for a total of eight star-shaped directions, at least one of the following boundary conditions of the Starfill algorithm is set: for corner points, a minimum number of two applies; for edge points, a minimum number of three applies; for interior points, a minimum number of six applies. This has proven to be a particularly good compromise between completely filling the volume delimited by the edge and avoiding assigning pixels outside the edge to food. With a total of eight star-shaped directions, a maximum of three directions can emanate from a corner point, a maximum of five directions from an edge point, etc.)

It is a further development that the Starfill algorithm is applied to a specific pixel and is subsequently applied to a next pixel, including the result for all previous pixels. In particular, the starfill algorithm can be carried out row-wise and column-wise.

It is a further development that already existing identified pixels remain identified unchanged, i.e. an assignment to the food once made is not resolved, e.g. even if the specified minimum number of directions is no longer given for this pixel or the conditions of the Starfill algorithm are no longer met.

It is a further development that an identified pixel or a pixel assigned to the food is no longer assigned to the food (i.e. the assignment is canceled) if the specified minimum number of directions is no longer met.

One embodiment is that the filling algorithm is executed multiple times (“recursively”) for all pixels until there is no further change in the identified pixels. The method can therefore assign the pixels iteratively. This is particularly advantageous in order to achieve the most complete assignment of the pixels to the food or not to the food.

Alternatively or additionally, the filling algorithm can be executed multiple times until a maximum number of runs or iterations has been reached.

It is an embodiment that the pixels assigned to the food to be cooked are additionally converted to pixels of an optical digital camera of the microwave oven. This is particularly advantageous in order to improve optical monitoring (in the visible spectral range) of food to be cooked, e.g. in relation to a degree of browning, a change in volume, etc., by comparing it with the present method, as this provides information about the position of the food to be cooked for optical monitoring becomes usable. This takes advantage of the fact that pixels of the thermal imaging camera can be correlated with pixels of an optical digital camera, so that when a pixel of the thermal imaging camera is assigned to the food to be cooked, one or more pixels of the optical digital camera, which show the same area of the food to be cooked, are also assigned to the food to be cooked can be. It is a further development that pixels of the thermal imaging camera that are not assigned to the food to be cooked are converted to pixels of an optical digital camera of the microwave oven.

It is a further development that this can also be done the other way around, i.e. that pixels of an optical digital camera that are attributed to the food to be cooked and/or pixels of an optical digital camera that are not assigned to the food to be cooked are converted to corresponding pixels of the thermal imaging camera. This means that if at least one pixel of the optical digital camera is not assigned to the food to be cooked, a pixel of the optical thermal imaging camera that shows the same area of the food to be cooked is also not assigned to the food to be cooked.

It is an embodiment that identifying the position of the food to be cooked is carried out in an initial phase of a cooking process. This is particularly advantageous because the food is not yet warmed through and the “edge overheating effect” is particularly pronounced. This in turn increases the reliability of the process.

It is an embodiment that while the microwave energy is radiated into the cooking chamber for the purpose of carrying out the method, at least one microwave operating parameter that changes a mode image of the microwaves in the cooking chamber is varied. This advantageously avoids narrowly localized spatial areas with high microwave energy (so-called “hot spots”) or at least varies them to such an extent that heating of the food to be cooked over a larger area is achieved. This in turn is advantageous in order to be able to recognize the increased heating of the edge of the food caused by the “Edge Overheating Effect” particularly well using the thermal imaging camera, especially over the entire edge if possible, or to be able to detect the number and/or length of the food not heating up particularly strongly Keep edge sections small. This configuration can be particularly advantageous in connection with the DE 10 2020 215 681 A1 be carried out, especially within the framework of the DE 10 2020 215 681 A1 mentioned “initial scans”, in which microwaves are fed into the cooking chamber under different parameter configurations, temperature distributions belonging to the parameter configurations are measured on the surface of the food to be cooked using the thermal imaging camera and heating patterns are determined from differences in different temperature distributions. The parameter configurations can then correspond in particular to the above microwave operating parameters.

• - bei Vorhandensein einer Drehantenne: Drehwinkel der Drehantenne, z.B. im Bereich [0°; 180°] oder im Bereich [0°; 360°], bspw. in Schritten von 1°, 5° oder 10°;
• - bei Vorhandensein eines Modenrührers bzw. „Stirrers“: Drehwinkel des Modenrührers, z.B. im Bereich [0°; 180°] oder im Bereich [0°; 360°], bspw. in Schritten von 1°, 5° oder 10°;
• - bei einem Mikrowellengenerator mit variabler Mikrowellenfrequenz, insbesondere halbleiterbasiertem Mikrowellengenerator: Mikrowellenfrequenz, z.B. im Bereich [2,4 GHz; 2,5 GHz] bspw. in Schritten von 10 MHz;
• - bei mehreren Mikrowellenports mit der Möglichkeit einer Variation einer Phasendifferenz zwischen den von diesen Mikrowellenports abgestrahlten Mikrowellen: Phasendifferenz, z.B. im Bereich [0°, 360°]

umfasst. Es ist vorteilhaft, wenn sich der Drehteller (falls vorhanden) mindestens einmal um sich selbst gedreht hat, sich die Drehantenne (falls vorhanden) mindestens einmal um sich selbst gedreht hat, die Mikrowellenfrequenz möglichst einmal vollständig variiert worden ist (falls möglich), die Phasendifferenzen einmal durchlaufen worden sind (falls möglich), usw. Die Aufnahme des zweiten Wärmebildes erfolgt bei Verwendung eines Drehtellers bevorzugt nach einer ganzzahligen Anzahl von vollständigen Umdrehungen, um zu erreichen, dass sich das Gargut bei Aufnahme des ersten und des zweiten Wärmebilds an der identischen Position befindet. Andernfalls müssten die Wärmebilder mittels einer geeigneten Rotationsmatrix zueinander ausgerichtet werden, was grundsätzlich auch möglich ist.It is a further development that a microwave operating parameter contains at least one parameter from the group

• - if a rotating antenna is present: angle of rotation of the rotating antenna, e.g. in the range [0°; 180°] or in the range [0°; 360°], e.g. in steps of 1°, 5° or 10°;
• - if a stirrer is present: angle of rotation of the stirrer, e.g. in the range [0°; 180°] or in the range [0°; 360°], e.g. in steps of 1°, 5° or 10°;
• - for a microwave generator with variable microwave frequency, in particular a semiconductor-based microwave generator: microwave frequency, e.g. in the range [2.4 GHz; 2.5 GHz], e.g. in steps of 10 MHz;
• - in the case of several microwave ports with the possibility of a variation of a phase difference between the microwaves emitted by these microwave ports: phase difference, e.g. in the range [0°, 360°]

It is advantageous if the turntable (if present) has rotated at least once around itself, the rotating antenna (if present) has rotated at least once around itself, the microwave frequency has been varied completely once (if possible), the phase differences have been run through once (if possible), etc. When using a turntable, the second thermal image is preferably recorded after an integer number of complete revolutions in order to ensure that the food being cooked is in the same position when the first and second thermal images are recorded. Otherwise, the thermal images would have to be aligned with one another using a suitable rotation matrix, which is also possible in principle.

The object is also achieved by a method for operating a microwave cooking device, in which the position of the food to be cooked is carried out using the method as described above and a cooking process is controlled based on an evaluation of at least the image points assigned to the food to be cooked, possibly also based on a combination of an evaluation of the pixels assigned to the food to be cooked and an evaluation of the pixels that are not assigned to the food to be cooked. The method for operating the microwave cooking device can be designed analogously to the method for identifying food to be cooked described above, and vice versa, and has the same advantages. For example, the determination of the position of the food to be cooked can be used to identify a target cooking state more precisely and thus increase the reliability of success and improve the cooking result. When the target cooking state is reached, at least one action can be triggered, e.g. feeding microwaves can be stopped and/or a message can be issued to a user, etc.

The object is also achieved by a microwave cooking device, having a cooking chamber that can be exposed to microwaves, a thermal imaging camera provided for recording thermal images from the cooking chamber and a control device, the microwave cooking device, in particular its control device, being set up to carry out the method or methods as described above . The microwave cooking device can be designed analogously to the methods and vice versa, and has the same advantages.

The microwave oven is in particular a household appliance. The microwave oven can be a standalone microwave oven or a com Combination device, for example a microwave device with additional heat radiator, for example an electrical resistance heating element, especially in the form of a table device, or an oven with microwave functionality.

• - eine mikrowellendichte Garraumtür zum mikrowellendichten Verschließen einer frontseitigen Beschickungsöffnung des Garraums;
• - mindestens einen Mikrowellengenerator;
• - mindestens einen Mikrowelleneinspeisungspunkt;
• - eine Drehantenne;
• - einen Modenrührer;
• - einen Drehteller.

In particular, the microwave cooking device can have one or more of the following devices:

• - a microwave-tight cooking chamber door for microwave-tight closing of a front loading opening of the cooking chamber;
• - at least one microwave generator;
• - at least one microwave feed point;
• - a rotary antenna;
• - a fashion stirrer;
• - a turntable.

• 1 zeigt als Schnittdarstellung in Seitenansicht eine Skizze eines Mikrowellengargeräts, das zum Ablauf eines Verfahrens zum Identifizieren von Gargut in einem aus seinem Garraum aufgenommenen bildpunktbasierten Wärmebild eingerichtet ist;
• 2 zeigt einen möglichen Ablauf des Verfahrens;
• 3 zeigt ein bildpunktbasiertes Wärmebild eines auf einem Teller abgelegten Garguts vor Einspeisung von Mikrowellen;
• 4 zeigt ein bildpunktbasiertes Wärmebild des auf dem Teller abgelegten Garguts aus 2 nach Einspeisung von Mikrowellen;
• 5 zeigt ein aus dem Wärmebildern nach 3 und 4 erzeugtes Temperaturdifferenzbild;
• 6 zeigt eine aus dem Temperaturdifferenzbild abgeleitete Zuordnung von Bildpunkten zu dem Gargut vor Anwendung des Starfill-Algorithmus; und
• 7A zeigt eine Anwendung des Starfill-Algorithmus für einen Innenpunkt des Temperaturdifferenzbilds;
• 7B zeigt eine Anwendung des Starfill-Algorithmus für einen Randpunkt des Temperaturdifferenzbilds;
• 7C zeigt eine Anwendung des Starfill-Algorithmus für einen Eckpunkt des Temperaturdifferenzbilds;
• 8 zeigt eine aus dem Temperaturdifferenzbild abgeleitete Zuordnung von Bildpunkten zu dem Gargut nach erster Anwendung des Starfill-Algorithmus; und
• 9 zeigt eine aus dem Temperaturdifferenzbild abgeleitete Zuordnung von Bildpunkten zu dem Gargut nach weiterer Anwendung des Starfill-Algorithmus.

The characteristics, features and advantages of this invention described above, as well as the manner in which they are achieved, will be more clearly and clearly understood in connection with the following schematic description of an exemplary embodiment, which will be explained in more detail in connection with the drawings.

• 1 shows, as a sectional view in side view, a sketch of a microwave oven which is set up to carry out a method for identifying food to be cooked in a pixel-based thermal image recorded from its cooking space;
• 2 shows a possible course of the procedure;
• 3 shows a pixel-based thermal image of a food placed on a plate before microwaves are fed in;
• 4 shows a pixel-based thermal image of the food placed on the plate 2 after feeding microwaves;
• 5 shows one from the thermal images 3 and 4 generated temperature difference image;
• 6 shows an assignment of image points to the food to be cooked, derived from the temperature difference image, before application of the Starfill algorithm; and
• 7A shows an application of the Starfill algorithm for an interior point of the temperature difference image;
• 7B shows an application of the Starfill algorithm for an edge point of the temperature difference image;
• 7C shows an application of the Starfill algorithm for a corner point of the temperature difference image;
• 8th shows an assignment of image points to the food to be cooked, derived from the temperature difference image, after the first application of the Starfill algorithm; and
• 9 shows an assignment of image points to the food derived from the temperature difference image after further application of the Starfill algorithm.

1 shows a sectional side view of a sketch of a microwave cooking appliance 1 with a cooking chamber 2, which is used to carry out a method for identifying food G in a pixel-based thermal image W1, W2 taken from a cooking chamber 2 (see 3 and 4 ). The cooking chamber 2 can be closed at its front loading opening by a cooking chamber door 3. In the present case, the microwave cooking appliance 1 is designed purely as an example as an oven with microwave functionality, in which the food G to be cooked can typically be placed on a food carrier 4 located on a certain shelf level, such as a baking tray or wire rack, etc., e.g. lying on a cooking dish such as a plate 5.

The microwave cooking device 1 has a, in particular semiconductor-based, microwave generator 6 for generating microwaves, in particular in a range [2.4 GHz; 2.5 GHz]. The microwaves generated are guided via a microwave guide 7, in particular in the form of a waveguide, to a microwave feed point 8 and fed there into the cooking chamber 2. At the microwave feed point 8 there is a rotatable rotary antenna 9 for the purpose of mode variation.

Furthermore, a pixel-based thermal imaging camera 11 for taking thermal images W1, W2 is located on a ceiling 10 of the cooking chamber 2, which is directed into the cooking chamber 2 from above and has the cooking chamber 2 with the food G to be cooked therein in its field of view. Optionally, the microwave cooking appliance 1 can also have an optical digital camera 12, in particular also arranged on the ceiling 10 of the cooking chamber 2, specifically close to the thermal imaging camera 11.

The microwave generator 6, a motor of the rotating antenna 9, as well as the thermal imaging camera 11 and, if applicable, the optical digital camera 12 can be controlled by means of a control device 13. The control device 13 can also be set up to carry out the method described below.

2 shows a possible sequence of the method for identifying the position of the food G in a thermal image W1, W2.

In a step S1, the food G to be cooked is placed on the food support 4, possibly lying on or in a dish such as a plate 5, a bowl, etc. A microwave power is also set.

In a step S2, a first thermal image W1 of the not yet heated food G is taken from the cooking chamber 2, which 3 with a temperature scale on the right-hand edge. For example, the thermal image W1 shows a plate with food G at approximately room temperature (here: ravioli with sauce in a round soup plate). The thermal image records the IR pixels arranged in the form of an (x, y) matrix, the values of which correspond to the local surface temperatures of the objects in the field of view of the thermal imaging camera 11 or at least correlate with the surface temperatures. Consequently, a temperature distribution in the cooking chamber 2 is mapped onto a matrix of (x . y) IR pixels, whereby the appropriately usable thermal imaging cameras 11 can have several hundred to several thousand individual pixels. The resolution of the thermal image W1 shown is (24 . 32) = 768 pixels. The method can also be used for spatially distributed and separated food, e.g. individual potatoes with a piece of meat next to them.

In a step S3, microwaves are radiated into the cooking chamber 2 with the set microwave power, with at least one microwave operating parameter that changes a mode image of the microwaves in the cooking chamber 2 being varied during the irradiation, e.g. a rotation angle of the rotating antenna 9 and/or a microwave frequency. For example, the rotating antenna 9 can be rotated continuously during this initial phase or this “initial scan”. It is advantageous here that as many different field distributions or mode images as possible are run through in order to cause a temperature increase in as many areas of the food G as possible.

Step S3 can be carried out for a predetermined duration, for example between 10 s and 30 s, but this (for example analogous to that in DE 10 2020 215 681 A1 described initial scan) can be dependent, for example, on the set power, a thermal mass of the food G and the absorption capacity of the food G. It is advantageous if in step S3 the rotary antenna 9 has rotated around itself at least once, the microwave frequency has been varied completely if possible once (if possible), the phase differences have been passed through once (if possible), etc.

After step S3, a second thermal image W2 is recorded in step S4, which is in 4 is shown. It turns out that the food G to be cooked heats up significantly more than the walls of the cooking space or visible parts of the plate on which the food G to be cooked is positioned. In particular, the “edge overheating effect” can also be seen, in which an edge of the food G to be cooked is heated particularly strongly, while the inner area of the surface of the food G to be cooked usually heats up significantly less or hardly noticeably.

In step S5, differential heat image DW is created (see 5 ), in which a temperature difference (“temperature swing”) between the temperature value recorded with the second thermal image W2 and the temperature value recorded with the first thermal image W1 is calculated for each of the image points.

In step S6, pixels are identified in the differential thermal image DW that have temperature values that are increased by a predetermined temperature level in relation to the set of all pixels in the differential thermal image DW. These pixels form a corresponding image pattern BM0 in the (x, y) matrix M, as shown in 6 shown.

Step S6 can be implemented, for example, in such a way that in a first sub-step S6A the temperature differences are mapped to a value range [0; 1]. The lowest value of the temperature differences of all 768 pixels is set to the value zero, and the highest value of the temperature differences of all pixels is set to the value one. If, for example, the lowest value were 3 °C and the highest value were 13 °C, the set of temperature differences {3; 5; 10; 13} would be mapped to the set {0; 0.2; 0.7; 1}.

In a subsequent sub-step S6B, those pixels which reach or exceed a predetermined temperature threshold value, here for example 0.27, are identified as belonging to the edge of the food G to be cooked. These pixels form the in 6 shown initial image pattern BM0 before execution of the filling algorithm.

The pixels identified as belonging to the food G in 6 already show very clearly the - here almost circular - edge or the outer contour of the food G. The inner area of the food G is not filled and there are still openings in the edge in places that have remained relatively cold. This cannot be solved by a universally valid temperature threshold.

In step S7, a filling algorithm in the form of a starfill algorithm is therefore applied to the initial image pattern BM0, in which a pixel is assigned to the food item G if a number of directions radiating from this pixel in a star shape, in which identified pixels are located, reaches or exceeds a minimum number. In the following, a starfill algorithm with a total of eight star-shaped directions is considered as an example.

This is in 7A for an image point located as a corner point in the matrix M and marked by a filled circle: the corner point has three possible directions extending from it in a star shape, namely the x-direction, the y-direction and an (x, y) oblique direction. Here, for example, image points identified as belonging to the food to be cooked are (filled) in the x-direction and in the y-direction, as indicated by the solid arrows. A distance between the identified pixels and the corner point is irrelevant in the present exemplary embodiment, but can generally be taken into account. However, there is no identified image point in the (x, y) oblique direction, as indicated by the dotted arrow. If it applies to the corner point that it is assigned to the food G to be cooked if there is at least one identified image point in at least two of the directions, this would be the case for the corner point shown.

In 7B The starfill algorithm is indicated for an image point located as an edge point in the matrix M and marked by a filled circle: the edge point has five possible directions extending from it in a star shape, namely the x-direction, the (-x) direction, the y direction, the (x, y) oblique direction and the (-x, y) oblique direction. Here, for example, image points identified as belonging to the food to be cooked lie in the (-x) direction, in the y-direction and in the (x, y) oblique direction, as indicated by the solid arrows. However, there are no identified image points in the x direction and in the (-x, y) oblique direction, as indicated by the dotted arrow. If the edge point is assigned to the food G to be cooked if there is at least one identified image point in at least three of the directions, this would be the case for the edge point shown.

In 7C the starfill algorithm is indicated for an image point lying as an interior point in the matrix M and marked by a filled circle: the interior point has all eight possible directions emanating from it in a star shape, namely the x-direction, the (-x)-direction, the y-direction, the (-y)-direction, the (x, y)-oblique direction, the (-x, y)-oblique direction, the (x, -y)-oblique direction and the (-x, -y)-oblique direction. Here, for example, image points identified as belonging to the food to be cooked lie in the x-direction, in the (x)-direction, in the y-direction, in the (-y)-direction, in the (-x, y)-oblique direction and in the (x, -y)-oblique direction, as indicated by the solid arrows. However, there are no identified pixels in the (x, y) oblique direction and in the (-x, -y) oblique direction, as indicated by the dotted arrow. If the interior point is assigned to the food G if there is at least one identified pixel in at least six of the directions, this would be the case for the interior point shown.

The Starfill algorithm can be applied to all pixels of the matrix M in one pass. In particular, the Starfill algorithm can be applied to a specific pixel (e.g. first to the pixel x = 0, y = 0) and then applied to the next pixel, including the result for all previous pixels. Existing identified pixels remain assigned unchanged, even if they subsequently meet the conditions of the Starfill algorithm. In particular, the Starfill algorithm can be carried out row by row and column by column.

The Starfill algorithm is particularly suitable for filling edge contours with openings. However, simple filling algorithms, such as “Floodfill”, in which areas of contiguous pixels of one color are captured, are unsuitable for contours that are not closed. The Starfill algorithm is particularly suitable for images or pixel matrices with relatively low resolution.

8th shows the (x, y) matrix M with the image pattern BM1 after a single application of the Starfill algorithm to the image pattern BMO from 6 , where starting from the image point x = 0, y = 0 the Starfill algorithm was first applied column by column in the x-direction and then row by row in the y-direction.

In a step S8, it is checked whether (a) a maximum number (“recursion number”) of applications or runs of the Starfill algorithm has been reached or whether (b) there are no more changes in successive runs of the Starfill algorithm, if necessary .depending on which condition occurs first.

If this is not the case (“N”), the process branches back to step S7, otherwise (“Y”) the process continues to step S9. 9 shows the (x, y) matrix M with an image pattern BMn, where the Starfill algorithm has been applied n times with n > 1.

In step S9, the method is terminated, whereby the pixels identified as belonging to the food to be cooked are then subjected to the following method steps S10 can be used as a basis for at least one method for operating the microwave cooking appliance 1, e.g. a method for detecting a target cooking state of the food G. For this purpose, the identified pixels can serve as a "mask", for example, which are used further in the following method(s), while all pixels that do not belong to the mask are disregarded or "masked out".

The method described enables extremely precise masking of the food G to be cooked. The method avoids in particular surfaces that only heat up slightly, such as the wide rim of the soup plate in the embodiment or other tableware items.

Of course, the present invention is not limited to the exemplary embodiment shown.

In general, “a”, “an”, etc. can be understood to mean a singular or a plural, in particular in the sense of “at least one” or “one or more”, etc., unless this is explicitly excluded, e.g. by the expression “exactly one”, etc.

A numerical value may also include the exact number stated as well as a usual tolerance range, as long as this is not explicitly excluded.

Reference symbol list

1

Microwave oven

2

cooking space

3

Cooking chamber door

4

Food carrier

5

Plate

6

Microwave generator

7

Microwave guidance

8th

Microwave feed point

9

Rotating antenna

10

Ceiling of the cooking space

11

Thermal camera

12

Optical digital camera

13

Control device

BM0

Initial image pattern

BM1

Image pattern

BMn

Image pattern

DW

Differential thermal image

G

Food to be cooked

M

matrix

S1-S10

Procedural steps

T

temperature

W1

First thermal image

W2

Second thermal image

x

x-direction

y

y direction

QUOTES INCLUDED IN THE DESCRIPTION

This list of documents listed by the applicant was generated automatically and is included solely to provide the reader with better information. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions.

Cited patent literature

• WO 2020156928 A1 [0004]
• WO 2020156929 A1 [0005]
• EP 3767580 A1 [0006]
• DE 102017101183 A1 [0007]
• WO 2016170734 A1 [0008]
• US 10219330 B2 [0009]
• EP 2618634 A1 [0010]
• WO 2012109634 A1 [0011]
• EP 1997349 B1 [0012]
• WO 2020200913 A1 [0013]
• EP 3574711 A1 [0014]
• EP 3767183 A1 [0015]
• EP 2055146 B1 [0016]
• DE 102020215681 A1 [0017, 0053, 0068]

Claims (13)

Method (S1-S9) for identifying food to be cooked (G) in at least one pixel-based thermal image (W1, W2) recorded from a cooking chamber (2) of a microwave oven (1), in which - microwave energy is radiated into the cooking chamber (2) (S3), - at least one thermal image (W1, W2) is recorded (S2, S4), - Pixels are identified (S6) which have temperature values that are increased by a predetermined temperature level in relation to the amount of all pixels, and - A filling algorithm is carried out, which assigns pixels to the food to be cooked (G), which belong to an image area surrounded by at least some of the identified pixels. Procedure (S1-S9). Claim 1 , in which - a first thermal image (W1) is recorded before the microwave energy is radiated into the cooking space (2), - a second thermal image (W2) is recorded after the microwave energy is radiated into the cooking space (2), - a differential heat image (DW ) is created, in which a temperature difference between the temperature value recorded with the second thermal image (W2) and the temperature value recorded with the first thermal image (W1) is calculated for each of the image points and - the image points are identified in the differential thermal image (DW). Procedure (S1-S9) according to Claim 2 , in which the temperature differences are mapped to a value range [0; 1]. Method (S1-S9) according to one of the preceding claims, in which those pixels which reach or exceed a predetermined temperature threshold value are identified as belonging to the edge of the food (G). Procedures (S1-S9) according to Claims 3 and 4 , where the temperature threshold is in a range [0.2; 0.5]. Method (S1-S9) according to one of the preceding claims, in which a starfill algorithm is used as the filling algorithm, in which a pixel is assigned to the food to be cooked (G) if a number of directions departing from this pixel in a star shape in which identified pixels a minimum number is reached or exceeded. Procedure (S1-S9) according to Claim 5 , in which at least one of the following boundary conditions of the Starfill algorithm is set: - a minimum number of two applies for corner points; - a minimum number of three applies for edge points; - a minimum number of six applies for interior points. Method (S1-S9) according to one of the preceding claims, in which the filling algorithm is executed several times until there is no further change in the identified pixels. Method (S1-S9) according to one of the preceding claims, in which the pixels assigned to the food (G) are additionally converted to pixels of an optical digital camera (12) of the microwave cooking appliance (1). Method (S1-S9) according to one of the preceding claims, in which the identification of the position of the food to be cooked (G) is carried out in an initial phase of a cooking process. Procedure (S1-S9) according to Claim 10 in which, during the irradiation of the microwave energy into the cooking chamber (2) for the purpose of carrying out the method, at least one microwave operating parameter which changes a mode image of the microwaves in the cooking chamber (2) is varied. Method (S1-S10) for operating a microwave cooking device (1) according to one of the preceding claims, in which a position of the food to be cooked (G) in a thermal image (W1, W2) is carried out using the method (S1-S9) according to one of the preceding claims and a cooking process is controlled based on an evaluation of the image points assigned to the food (G). Microwave cooking appliance (1), comprising a cooking space (2) which can be subjected to microwaves, a thermal imaging camera (11) provided for taking thermal images (W1, W2) from the cooking space (2) and a control device (13), the microwave cooking appliance (1) also is set up to carry out the method according to one of the preceding claims (S1-S9; S1-S10).

Images