EP3152498B1 - Gargerät mit lichtmusterprojektor und kamera - Google Patents

Gargerät mit lichtmusterprojektor und kamera Download PDF

Info

Publication number
EP3152498B1
EP3152498B1 EP15727630.4A EP15727630A EP3152498B1 EP 3152498 B1 EP3152498 B1 EP 3152498B1 EP 15727630 A EP15727630 A EP 15727630A EP 3152498 B1 EP3152498 B1 EP 3152498B1
Authority
EP
European Patent Office
Prior art keywords
light pattern
cooking
cooking appliance
camera
food
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
EP15727630.4A
Other languages
German (de)
English (en)
French (fr)
Other versions
EP3152498A1 (de
Inventor
Sebastian Erbe
Robert KÜHN
Dan Neumayer
Daniel Vollmar
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
BSH Hausgeraete GmbH
Original Assignee
BSH Hausgeraete GmbH
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by BSH Hausgeraete GmbH filed Critical BSH Hausgeraete GmbH
Priority to PL15727630T priority Critical patent/PL3152498T3/pl
Publication of EP3152498A1 publication Critical patent/EP3152498A1/de
Application granted granted Critical
Publication of EP3152498B1 publication Critical patent/EP3152498B1/de
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • F—MECHANICAL ENGINEERING; LIGHTING; HEATING; WEAPONS; BLASTING
    • F24—HEATING; RANGES; VENTILATING
    • F24C—DOMESTIC STOVES OR RANGES ; DETAILS OF DOMESTIC STOVES OR RANGES, OF GENERAL APPLICATION
    • F24C7/00—Stoves or ranges heated by electric energy
    • F24C7/08—Arrangement or mounting of control or safety devices
    • F24C7/081—Arrangement or mounting of control or safety devices on stoves
    • F—MECHANICAL ENGINEERING; LIGHTING; HEATING; WEAPONS; BLASTING
    • F24—HEATING; RANGES; VENTILATING
    • F24C—DOMESTIC STOVES OR RANGES ; DETAILS OF DOMESTIC STOVES OR RANGES, OF GENERAL APPLICATION
    • F24C7/00—Stoves or ranges heated by electric energy
    • F24C7/08—Arrangement or mounting of control or safety devices
    • F24C7/082—Arrangement or mounting of control or safety devices on ranges, e.g. control panels, illumination
    • F24C7/085—Arrangement or mounting of control or safety devices on ranges, e.g. control panels, illumination on baking ovens

Definitions

  • the invention relates to a cooking appliance, comprising a cooking chamber with a loading opening that can be closed by means of a door, a light pattern projector fixed in relation to the cooking chamber for generating a light pattern, a camera for taking pictures from an area that can be irradiated by the light pattern and one coupled to the camera Evaluation device for calculating a three-dimensional shape of at least one object which is located in the area that can be irradiated by the light pattern, by means of a light pattern evaluation.
  • the invention is particularly advantageously applicable to ovens.
  • the invention is particularly advantageously applicable to household appliances.
  • EP 2 530 387 A1 discloses an oven with a device for detecting a three-dimensional shape of food on a baking tray of the oven.
  • the device contains at least one laser which is arranged or can be arranged over a cooking space of the oven. A laser beam from the laser is directed downwards.
  • the device further comprises at least one camera which is arranged or can be arranged above a baking sheet of the oven. The camera is arranged or can be arranged in a front section of the furnace.
  • the baking tray and the camera are mechanically coupled so that the camera and the baking tray can be moved synchronously.
  • An upper side of the baking sheet is in a field of view of the camera.
  • An angle between a central axis of a field of view of the camera and the laser beam is predetermined.
  • the device for detecting the three-dimensional shape of the food on the baking sheet is also disclosed.
  • EP 2 149 755 A1 discloses an oven for heating food products comprising a cooking chamber to receive the product via a loading port and a product feature corn extraction system configured to extract at least one product feature representative of a configuration of the product, the system comprising : at least one camera, which is designed and arranged to record top views of the product, and at least one contour plane unit that is used to extract or highlight contour planes at least a portion of the product and, as the case may be, an object that to introduce together is provided with the product in the cooking space, and a product feature extraction unit for extracting the at least one product feature on the basis of the top view of the product and contour planes of the product.
  • a method for operating an oven for heating a food product comprising the steps: a) extracting a product feature of a product that is to be heated in a chamber of the oven by recording at least one top view of a product using at least one camera, extracting and / or Highlighting of contour planes of at least one section of the product and, as may be the case, an object that is intended to be introduced together with the product into the cooking chamber, using at least one contour plane unit, and b) extracting the at least a product feature based on the plan views and contour planes, based on at least one product feature, and optionally secondary data representing a physical configuration of the product, preferably at least one of the product temperature, a product weight and a product density, for automatic control or for Heating the Product.
  • EP 1 921 384 A1 discloses a device for determining the temperature inside a product to be cooked.
  • the device has at least one temperature sensor for detecting at least one surface temperature of the item to be cooked and / or an ambient temperature of the item to be cooked, in particular at a measuring location within a cooking space surrounding the item to be cooked, preferably with an ambient temperature sensor arranged at the measuring location.
  • the device comprises at least one distance sensor for detecting one or a plurality of distances between the distance sensor on the one hand and one or a plurality of distance measuring points on the surface of the food to be cooked on the other hand.
  • the device comprises at least one time measuring device for recording the time during preparation of the food and at least one calculation device for calculating the temperature inside the food from the surface temperature of the food and / or ambient temperature, the interval or the plurality of intervals, the time and one Starting temperature of the food.
  • a method for determining the temperature inside a product to be cooked is also disclosed.
  • DE 197 48 062 A1 discloses a method and a device for three-dimensional, optical measurement of objects. Afterwards, the measuring system must be calibrated in the case of optical, areally working, three-dimensional measuring methods, since the geometric parameters of the system must be known in order to carry out the triangulation calculation. After calibration, the lenses must no longer be adjusted, as this changes the imaging errors of the optics in an uncontrollable manner. The method enables the measuring system to be set to a different measuring field size even after calibration.
  • the measuring system By determining the inner bundle of rays from the projector and camera by means of a device which simultaneously serves to focus on a wide variety of measuring distances, the measuring system is adapted to different measuring field sizes in such a way that the geometric changes made to the system can be precisely determined and those that are decisive for the triangulation Parameters can be calculated without recalibration.
  • the calibration now takes place with a measurement field size which is selected solely from the point of view of the ease of manufacture of the calibration device and easily manageable dimensions. Once the system has been calibrated, it can then be set to the most varied, in particular very large, measurement distances and measurement volumes. An application to household appliances or cooking appliances is not disclosed.
  • WO 00/70303 discloses a method and an apparatus for the imaging of three-dimensional objects, comprising a structural light source which projects a focused image onto an object by passing light either continuously or in a stroboscopic manner through an optical grating and a downstream projection lens. An application to household appliances or cooking appliances is not disclosed.
  • DE 10 2006 005 874 A1 discloses a device and a method for contactless measurement of, in particular, cylindrical objects on surfaces. To this end, it is proposed to use a laser to generate a line on the surface, the reflection of which is measured by a camera. After the line has been recorded, it is shifted parallel to itself several times and the recording is repeated. In this way, a shadow image of the object arranged on the surface is generated by successively displacing the line. It is also possible to separate the multiline triangulation and the shadow formation from one another.
  • a stationary laser or another radiation source can be used for multiline triangulation.
  • the Shadow formation can be carried out by two also stationary radiation sources, for example a row of LEDs, simultaneously or one after the other. The use of a stationary structure of radiation sources and camera simplifies and makes the mechanical structure cheaper. An application to household appliances or cooking appliances is not disclosed.
  • WO 2010/102261 A1 discloses a food treatment device in which food located in the device is treated by radiation.
  • the food treatment device has a camera for image recognition.
  • a type of food eg pizza
  • the image recognition can identify its location, occupancy, etc., for example.
  • a light pattern can also be used to identify the height or other expansion of the pizza.
  • a cooking device having a cooking chamber with a loading opening that can be closed by means of a door, (at least) one projector (referred to below as a "light pattern projector" without restricting generality) for generating a light pattern, at least) one camera for recording images from an area that can be irradiated by the light pattern and an evaluation device coupled to the camera for calculating a three-dimensional shape of at least one object that is located in the area that can be irradiated by the light pattern by means of a light pattern evaluation, the The light pattern projector is arranged to radiate a light pattern into the cooking chamber, the camera is arranged in particular fixedly with respect to the cooking chamber, the camera is arranged for taking pictures from an area of the cooking chamber that can be irradiated by the light pattern, even when the cooking chamber is closed and the Evaluation device is set up for repeated calculation of the three-dimensional shape of the at least one object, which is located in the region of the cooking chamber that can be irradiated by the light pattern, during
  • the cooking appliance has the advantage that the depth information can be used as a parameter for automatic programs.
  • a possible change in volume of the item to be cooked during the cooking process when the cooking appliance is in operation can thus be recorded and a type of item to be cooked can be determined.
  • the control of the cooking parameters can be influenced, for example the cooking space temperature. So For example, a rising behavior of a bread and a shrinking behavior of a piece of meat may be recorded and possibly used to control the cooking appliance.
  • the basically known method of patterned or structured light (“structure light”) is used in particular.
  • a defined light pattern is projected onto the object to be recorded or measured by means of the light pattern projector and recorded by the camera.
  • a three-dimensional model of this object can be calculated by means of the evaluation device.
  • a depth resolution of a specific image point depends on the angle between a light beam for generating this image point and a normal vector to a plane or to an optical axis of the camera. A theoretical optimum resolution would be at the largest possible angle.
  • the cooking appliance may be or have an oven, in particular an oven.
  • the cooking space may then also be referred to as the oven space.
  • the oven may be a stand-alone oven or it may be part of an oven / hob combination or range.
  • the oven may have a microwave and / or steam treatment functionality.
  • the cooking appliance is a household appliance, in particular in the sense of “white goods”.
  • the light pattern projector may also be referred to as a so-called “3D scanner”.
  • the light pattern projector emits at least one light pattern, for example a stripe and / or point pattern, but is not limited thereto. In this way, any other light pattern can be generated, for example ring-shaped Pattern, wave pattern, etc.
  • a pattern is selected in particular in such a way that it matches the desired resolution of the three-dimensional image.
  • the camera may be a digital camera. It may take individual pictures and / or picture sequences, especially videos.
  • the evaluation device may be an independent device of the cooking appliance, e.g. in the form of electronics, in particular on a separate circuit board. Alternatively, it may be integrated into a further device of the cooking appliance, e.g. into a central control device. This further device may then also be able to carry out the evaluation in particular.
  • an optical axis of the light pattern projector and an optical axis of the camera are at an angle between 20 ° and 30 ° to one another. This achieves good visibility of the projected light pattern with good depth resolution and consequently a particularly reliable determination of the three-dimensional shape of the at least one object.
  • the light pattern projector and the camera are arranged behind a wall or muffle of the cooking chamber, in particular at a predefined distance. In this way, these two components can be sufficiently thermally insulated from the cooking space.
  • the cooking chamber wall may have a respective window for the light pattern projector and for the camera.
  • the windows may be covered with transparent glass.
  • the light pattern projector and the camera are arranged behind a ceiling of the cooking space.
  • a food support eg a baking sheet or a wire rack
  • This in turn enables particularly precise images and measurements to be generated.
  • This position has the further advantage that cooling air conducted over the ceiling (eg for cooling electronics arranged above the cooking chamber) can also be used to cool the light pattern projector and the camera.
  • the distance between the light pattern projector and the camera behind the cooking chamber wall or muffle can be muffle-specific.
  • different light patterns can be radiated into the cooking space by means of the light pattern projector.
  • the three-dimensional shape of the at least one object can be determined with a particularly small error.
  • alternating point-like and strip-like light patterns can be irradiated and evaluated.
  • Different dot patterns and / or different stripe patterns can also be radiated into the cooking space. This can take place in a predetermined sequence or if a measured depth resolution does not produce sufficient results.
  • the light pattern projector has at least one pixel-like screen or screen for shaping the light pattern.
  • the pixel-like screen may e.g. be a liquid crystal screen or an LCD screen.
  • the pixel-like screen may itself generate light as a structural unit in order to adequately illuminate the cooking space with the light pattern.
  • the pixel-like screen may e.g. can also be backlit by at least one separate light source so that it can be used as a 'variable diaphragm'. The latter case enables particularly high luminous fluxes.
  • the light emitted by the light pattern projector and received by the camera may be visible light and / or infrared light.
  • the advantage of infrared light is that an observer looking into the cooking space cannot see the light pattern.
  • the 3D scanner can be calibrated.
  • At least one calibration marking is located on a food support, for example on a baking sheet or a grate etc.
  • it may be located on a surface for use or storage of the food support for food.
  • This calibration mark especially likes have a known size, so that a distance to the camera can be determined by means of the size recorded by the camera.
  • a calibration marking may be, for example, a colored marking and / or a predetermined shaped marking.
  • the calibration marks can also be defined geometrical features, e.g. Functional areas of the cooking chamber wall or muffle such as insertion protrusions.
  • the calibration mark (s) can also serve to determine the insertion level on which the object to be measured is located.
  • the calibration preferably takes place in the closed muffle or with the cooking space closed, in particular at the start of a cooking process. This minimizes any influence on the measurement from the environment.
  • 3D scan preferred insertion levels in the cooking space for the 3D measurement (“3D scan”). They are preferably located in a lower third of the muffle. This has the advantage that large-area and / or large-volume objects can also be reliably identified and measured.
  • the 3D measurement of the object advantageously takes place after the calibration. In principle, however, calibration can also be dispensed with.
  • the cooking appliance is equipped with an insertion identifier.
  • the cooking appliance may then, when it detects that a food carrier is located on an insertion level that is unfavorable for a 3D measurement, to output a warning signal and / or a display to a user.
  • the cooking appliance may then also prevent a 3D scan.
  • the evaluation device is set up to recognize a type of item to be cooked on the basis of its shape and shape change calculated by the light pattern evaluation.
  • This enables, among other things, an automatic adaptation of cooking parameters to the food (e.g. within the framework of a cooking program) and / or an adaptation of user guidance to the food (e.g. by displaying cooking parameters and / or cooking programs suitable for the recognized food).
  • the cooking appliance is set up to recognize the food being cooked on the basis of an image evaluation of images recorded by the camera (without 3D measurement, also referred to as “image recognition” below) and the 3D scan. Recognition of the food to be cooked on the basis of a combined image recognition and a 3D measurement enables a higher recognition probability through the additional height and depth information. This may e.g. enter as information in an image recognition algorithm.
  • the evaluation device is set up to recognize a type of food carrier, e.g. whether the food is on a wire rack or on a baking sheet.
  • This information can be used by the cooking appliance, for example, to set or adapt cooking parameters, for example a heating output of upper and / or lower heating or an activation and / or setting of a heating output of a circulating air heater.
  • the evaluation device is set up to recognize a type of accessory, in particular cooking utensils, for example a roasting pan or the like placed on a food support, in which the food is located. So it may be measurable whether the food to be cooked is in an open roaster or the roaster is closed. This information may also be used by the cooking appliance to set or adapt cooking parameters, possibly including the option of selecting the cooking method used. If the roaster is closed, the cooking device may be dependent on input from the user about the type of content.
  • a type of accessory in particular cooking utensils, for example a roasting pan or the like placed on a food support, in which the food is located. So it may be measurable whether the food to be cooked is in an open roaster or the roaster is closed. This information may also be used by the cooking appliance to set or adapt cooking parameters, possibly including the option of selecting the cooking method used. If the roaster is closed, the cooking device may be dependent on input from the user about the type of content.
  • the evaluation device is set up to detect a core temperature of an object.
  • the core temperature can be calculated using a correlation with a change in volume determined by the 3D scan during a cooking process with knowledge of the type of food being cooked. There is no need for a separate core temperature probe or roast skewer. For a particularly high level of reliability in determining the core temperature, it is a preferred development that the food to be cooked has an almost homogeneous structure.
  • the core temperature is determined by means of a 3D scan, not as in EP 1 921 384 A1 described by means of one or more distance sensors.
  • the evaluation device is coupled to a control device of the cooking device and the control device is set up to adapt an operation of the cooking device on the basis of at least one object parameter determined by the evaluation device.
  • the associated object can be food, an accessory and / or a food carrier.
  • Object parameters can e.g. be a position, shape, volume or type, etc., of the object.
  • the 3D information determined by one or more 3D scans can be used in particular for the automation of cooking, cooking and baking processes. As already mentioned, it is possible to carry out the 3D scan during a cooking process. The 3D information or 3D data that is then determined is not only used to identify the food being cooked, but also to adapt the cooking parameters if necessary.
  • a cooking process or cooking process can thus be individually tailored to the food being cooked. If an exact detection of the food to be cooked is not possible, inputs from a user can in particular be taken into account. For this purpose, the cooking appliance may request the user to enter further information about the food to be cooked into the cooking appliance.
  • the cooking appliance has a screen on which at least one three-dimensional image of at least one object recorded by the camera can be displayed.
  • a three-dimensional representation of the contents of the cooking space is offered on a screen. This enables a particularly informative presentation of information for a user.
  • the screen may e.g. be present on a front side or a top side of the cooking appliance.
  • the screen may be a touch-sensitive sensor screen or touch screen.
  • One way of operating the cooking device is to carry out a calibration before starting a food treatment (e.g. a cooking process). Furthermore, shortly before the start of a food treatment, a 3D measurement or a 3D scan of the food to be cooked may be carried out in order to record its initial geometry. An object detection with regard to a type of item to be cooked, a type of an accessory and / or a type of a food carrier may also be carried out before the start of a food treatment.
  • At least one 3D scan may also take place during the food treatment, in particular several 3D scans at, for example, periodic intervals.
  • a product to be cooked is identified and, for example, the end of treatment and / or a core temperature can be identified.
  • the light pattern projector is also provided for illuminating the cooking space. For example, it may illuminate the cooking space for viewing by a user and only shine the light pattern into the cooking space for comparatively short periods in between. This means that there is no need for a separate light source for lighting the cooking space.
  • Fig.1 shows an arrangement (“3D scanner”) for determining a three-dimensional shape of at least one object O (“3D scanner”), having a light pattern projector 1 directed at the object, a camera 2 directed at the object O, and a control device C for operation the light pattern projector 1 and for calculating a three-dimensional shape of the object O on the basis of at least one image received by the camera 2 by means of a light pattern evaluation.
  • a screen 3 for viewing the object O ′ calculated by the control device C is optionally available.
  • the light pattern projector 1 generates a predetermined light pattern L, e.g. a stripe or dot pattern.
  • the light pattern projector 1 emits its light in a light beam with a first optical axis A1.
  • the camera 2 typically a digital camera, has a field of view F with a second optical axis A2, which is oriented at an angle to the first optical axis A1 of the light pattern projector 1. In other words, the camera 2 is oriented at an angle to the light pattern projector 1. It observes a region of the object O that is or can be irradiated by the light pattern L.
  • Fig. 2 shows a sketch of a reconstruction of a shape of the object O measured by means of the 3D scanner 1, 2, C.
  • the light pattern projector 1 here has a light source Q, e.g. an array of light-emitting diodes, which is followed by a pattern-generating element in the form of a freely programmable LCD surface D that can be radiated through. Depending on the pattern M generated on the LCD surface D, a corresponding, in particular complementary, light pattern L is emitted from the LCD surface D.
  • a light source Q e.g. an array of light-emitting diodes
  • a pattern-generating element in the form of a freely programmable LCD surface D that can be radiated through.
  • a corresponding, in particular complementary, light pattern L is emitted from the LCD surface D.
  • an LED screen may already serve as the light source (not shown), in which case a separate light source can then be dispensed with because of the background lighting (“backlighting”) integrated therein.
  • Fig. 2 is shown by way of example how light in the form of a vertical column or line G is radiated from the light pattern projector 1 onto the object O.
  • a projection P (G) of this line G which is distorted by the surface contour of the object O, appears on the object O.
  • the camera 2 takes an image of this projection P (G), which reproduces the distortion, due to its inclination with respect to the light pattern projector 1.
  • the camera 2 stores the projection P (G) as appropriately positioned image points B or "pixels" of a matrix that results from a matrix-like structure of individual sensors of a sensor array S of the camera, for example a CCD sensor array.
  • the height or depth information is given by the deviation of the image points B from a vertical line.
  • the depth resolution depends on an angle W between the light beam r leading to the image point B and the direction of the column or line G.
  • a theoretical optimum in terms of resolution would be at the largest possible angle W.
  • the visibility of the projection P (G) on the surface of the object O and thus its detectability in the camera image deteriorate with increasing approach to this optimum. Since a reconstruction is only possible for those points on the surface of the object O which on the one hand are visible from the camera 2 and on the other hand can be illuminated by the light pattern projector 1, a compromise is made here.
  • Such a 3D measurement or 3D scan is known in principle and is therefore not discussed further below.
  • Fig. 3 shows a sectional view in side view of a cooking device equipped with a 3D scanner in the form of an oven 4.
  • the oven 4 has a cooking space 6 delimited by an oven muffle 5.
  • the oven muffle 5 has at the front a loading opening 8 which can be closed by means of an oven door 7 and through which objects, in particular in the form of food O1, can be brought into the cooking space 6.
  • a cooking space temperature T can be set by means of one or more, in particular electrically operated, heating elements (not shown).
  • a ceiling 9 of the furnace muffle 5 there are two viewing windows 10 and 11, which can be covered, for example, with transparent glass panes.
  • a light pattern projector 1 e.g. with an LCD display for pattern generation
  • a camera 2 behind the viewing window 11 their distance from the furnace muffle 5 is thermally protected.
  • a flow of cooling air may sweep over the ceiling 9, for example for cooling components arranged there, such as a control device.
  • the light pattern projector 1 and the camera 2 can also be further cooled by this cooling air flow.
  • the light pattern projector 1 and the camera 2 are arranged laterally offset from one another.
  • their optical axes A1 and A2 assume an angle ⁇ between 20 ° and 30 °, which enables high depth resolution with good visibility.
  • the light pattern projector 1 and the camera 2 are fixedly arranged in relation to the cooking space 6 and therefore do not move with it, for example when the oven door 7 is actuated.
  • the light pattern projector 1 emits a light pattern L through the viewing window 10 into the cooking space 6, in such a way that, from a predetermined distance from the ceiling 9, practically the entire horizontal surface of the cooking space 6 can be illuminated with the light pattern L. This may e.g. be the case in a lower half or in a lower third of the cooking space 6.
  • the camera 2 records images from an area of the cooking chamber 6 that can at least partially be irradiated by the light pattern
  • the oven 4 also has an evaluation device 12 coupled to the camera 2 for calculating a three-dimensional shape, for example of the food O1 and a food carrier O2, which are located in the area that can be irradiated by the light pattern L, by means of a light pattern evaluation. This is based on a 3D measurement based on at least one image recorded by the camera 2.
  • the light pattern projector 1, the camera 2 and the evaluation device 12 together form the 3D scanner.
  • the evaluation device 12 may be functionally integrated into a central control device of the baking oven 4 or it may be coupled to a control device as an independent unit.
  • a 3D scan comprising e.g. a recording of an image of the projection P (G) of the light pattern L by means of the camera 2 and a calculation of the three-dimensional shape of the food O1 and the food support O2 can be carried out with the oven door 7 or loading opening 8 closed.
  • the food carrier O2 has one or more, for example colored, calibration markings K on its upper side, which have a known size and can be easily identified.
  • the size of the calibration mark (s) K recorded by the camera 2 allows a distance from the ceiling 9 and thus, for example, the insertion level used identify.
  • the oven 4 may output a message to a user, e.g. a display on a front screen 3 of a control panel 13. At least one calibration mark may also be located on the oven muffle 4.
  • an initial 3D measurement of the item to be cooked O1 may be carried out by means of the 3D scanner 1, 2, 12 in order to calculate its original shape.
  • the calculated shape may be displayed on the screen 3.
  • the calculated shape is used to determine the item to be cooked O1 by the cooking appliance 4, specifically for image recognition of the item to be cooked O1, which can also be carried out by the camera 2.
  • a type of food carrier O2 may also be recognized by means of the 3D scanner 1, 2, 12.
  • One or more cooking parameters of the cooking process may be adapted on the basis of a recognition of the food O1 and possibly the carrier O2.
  • a cooking time and / or the cooking space temperature T may thus be adapted on the basis of the recognized type and / or the recognized volume of the item to be cooked and / or the recognized item carrier O2.
  • 3D measurements can be carried out repeatedly in order to determine a change in shape and / or volume of the item to be cooked O1.
  • the oven 4 may adapt the cooking sequence, e.g. change the cooking time and / or the cooking space temperature, including switching off the heaters.
  • the light pattern projector 1 may project different light patterns L into the cooking space 6.
  • Fig. 4 shows a diagram of a profile of a temperature T and a volume V of the item to be cooked O1 with a determination of the item to be cooked by means of the cooking appliance 4, for example.
  • the food O1 to be cooked is purely by way of example a dough product, for example a round tarte flambée or a biscuit in a springform pan.
  • a distinction between these two types of food O1 cannot be distinguished only by means of image recognition by the camera 2 because in a two-dimensional view from above (top view) both the tarte flambée and the sponge cake look circular. In addition, both are similar in color. However, both types of food require a different specific baking environment. If the tarte flambée is treated like the sponge cake, the result is unsatisfactory, which also applies to the reverse case.
  • the cooking appliance 4 Through the 3D measurement by means of the 3D scanner 1, 2, 12, the cooking appliance 4 also receives spatial information about the food to be cooked O1.
  • This spatial information of the initial state of the item to be cooked O1 before the cooking process may already be sufficient to distinguish the flat tarte flambée from the higher biscuit.
  • the type of item to be cooked O1 is determined by means of the 3D scanner 1, 2 12 from the change in its shape, in particular a change ⁇ V in its volume V.
  • the cooking space temperature T has an initial value Ts at an initial point in time ts of the cooking process, e.g. Room temperature.
  • the cooking space temperature T increases due to at least one activated heater, specifically for tarte flambée and sponge cake, as shown by the curve T1 + T2. If the cooking space temperature T reaches a target temperature Td1 for biscuits at a point in time td, which is below a target temperature Td2 for tarte flambée, a further 3D measurement is carried out in order to determine the type of item O1 to be cooked.
  • the cooking appliance 4 can then, for example, increase its cooking space temperature T to the associated setpoint Td2, as indicated by the temperature curve T2.
  • the cooking process ends at an associated end time te2.
  • the cooking appliance 4 may subsequently keep its cooking space temperature T at the associated setpoint Td1, as indicated by the temperature curve T1. The cooking process ends at an associated end time te1.
  • a clear distinguishing feature for identifying the food to be cooked can therefore be provided.

Landscapes

  • Engineering & Computer Science (AREA)
  • Chemical & Material Sciences (AREA)
  • Combustion & Propulsion (AREA)
  • Mechanical Engineering (AREA)
  • General Engineering & Computer Science (AREA)
  • Length Measuring Devices By Optical Means (AREA)
EP15727630.4A 2014-06-05 2015-06-03 Gargerät mit lichtmusterprojektor und kamera Active EP3152498B1 (de)

Priority Applications (1)

Application Number Priority Date Filing Date Title
PL15727630T PL3152498T3 (pl) 2014-06-05 2015-06-03 Urządzenie do obróbki termicznej z projektorem wzorca świetlnego i kamerą

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
DE102014210672.9A DE102014210672A1 (de) 2014-06-05 2014-06-05 Gargerät mit Lichtmusterprojektor und Kamera
PCT/EP2015/062349 WO2015185608A1 (de) 2014-06-05 2015-06-03 Gargerät mit lichtmusterprojektor und kamera

Publications (2)

Publication Number Publication Date
EP3152498A1 EP3152498A1 (de) 2017-04-12
EP3152498B1 true EP3152498B1 (de) 2020-11-11

Family

ID=53366015

Family Applications (1)

Application Number Title Priority Date Filing Date
EP15727630.4A Active EP3152498B1 (de) 2014-06-05 2015-06-03 Gargerät mit lichtmusterprojektor und kamera

Country Status (7)

Country Link
US (1) US10228145B2 (pl)
EP (1) EP3152498B1 (pl)
CN (1) CN106461230B (pl)
DE (1) DE102014210672A1 (pl)
ES (1) ES2835724T3 (pl)
PL (1) PL3152498T3 (pl)
WO (1) WO2015185608A1 (pl)

Families Citing this family (65)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
DE102014113389A1 (de) * 2014-09-17 2016-03-17 Pilz Gmbh & Co. Kg Verfahren und Vorrichtung zum Identifizieren von Strukturelementen eines projizierten Strukturmusters in Kamerabildern
USD787041S1 (en) 2015-09-17 2017-05-16 Whirlpool Corporation Gas burner
US10837651B2 (en) 2015-09-24 2020-11-17 Whirlpool Corporation Oven cavity connector for operating power accessory trays for cooking appliance
US11777190B2 (en) 2015-12-29 2023-10-03 Whirlpool Corporation Appliance including an antenna using a portion of appliance as a ground plane
WO2017170519A1 (ja) * 2016-03-29 2017-10-05 パナソニックＩｐマネジメント株式会社 加熱調理器
JPWO2017170319A1 (ja) * 2016-03-29 2019-02-14 パナソニックＩｐマネジメント株式会社 加熱調理器
WO2017170318A1 (ja) * 2016-03-29 2017-10-05 パナソニックＩｐマネジメント株式会社 加熱調理器
CN107296507A (zh) * 2016-04-15 2017-10-27 松下知识产权经营株式会社 烹调辅助方法以及烹调辅助系统
DE102016206483A1 (de) * 2016-04-18 2017-10-19 Convotherm Elektrogeräte GmbH Verfahren zum Feststellen einer Reinigungsnotwendigkeit und Qualitätsmanagement-Überwachungssystem eines gewerblichen Gargeräts, und gewerbliches Gargerät
AU2016403900B2 (en) * 2016-04-20 2022-09-29 Vorwerk & Co. Interholding Gmbh System for the preparation of at least one food product and method for operating the relevant system
US10145568B2 (en) 2016-06-27 2018-12-04 Whirlpool Corporation High efficiency high power inner flame burner
DE102016012036A1 (de) 2016-10-11 2018-04-12 Diehl Ako Stiftung & Co. Kg Gargutträger zum Aufnehmen von Gargut in einem Gargerät und Gargerät mit einem solchen Gargutträger
DE102016221446A1 (de) * 2016-11-02 2018-05-03 BSH Hausgeräte GmbH Kalibrieren eines Sauerstoffsensors eines Haushaltsgeräts
EP3346190B1 (en) 2017-01-10 2020-05-06 Electrolux Appliances Aktiebolag Food preparation entity
US10551056B2 (en) 2017-02-23 2020-02-04 Whirlpool Corporation Burner base
US10451290B2 (en) 2017-03-07 2019-10-22 Whirlpool Corporation Forced convection steam assembly
US10660162B2 (en) 2017-03-16 2020-05-19 Whirlpool Corporation Power delivery system for an induction cooktop with multi-output inverters
DE102017206056A1 (de) 2017-04-10 2018-10-11 BSH Hausgeräte GmbH Betreiben eines Gargeräts
KR102366006B1 (ko) * 2017-06-20 2022-02-23 삼성전자주식회사 오븐
WO2019033089A1 (en) * 2017-08-11 2019-02-14 Brava Home, Inc. CONFIGURABLE COOKING SYSTEMS AND METHODS
CN109166054A (zh) * 2017-08-16 2019-01-08 嘉易烤株式会社 烹饪鱼肉类时碳化时点管理服务提供系统
CN107692840A (zh) * 2017-09-06 2018-02-16 珠海格力电器股份有限公司 电器具的显示方法及装置，电器具
DE102017121084A1 (de) * 2017-09-12 2019-03-14 Rational Aktiengesellschaft Gargerät sowie Verfahren zur Erkennung der Belegung eines Einschubes in einem Garraum
KR102400018B1 (ko) * 2017-09-29 2022-05-19 삼성전자주식회사 자동 조리 장치 및 방법
US20190110638A1 (en) * 2017-10-16 2019-04-18 Midea Group Co., Ltd Machine learning control of cooking appliances
US10591218B2 (en) 2017-10-27 2020-03-17 Whirlpool Corporation Oven having an imaging device
US10605463B2 (en) 2017-10-27 2020-03-31 Whirlpool Corporation Cooking appliance with a user interface
US10523851B2 (en) 2018-02-19 2019-12-31 Haier Us Appliance Solutions, Inc. Camera assembly for an oven appliance
JPWO2019198621A1 (ja) * 2018-04-09 2021-04-15 パナソニックＩｐマネジメント株式会社 加熱調理器
CN108742170B (zh) * 2018-05-08 2020-08-18 华南理工大学 一种烤箱的智能识别烹饪系统
WO2019246490A1 (en) 2018-06-21 2019-12-26 Prince Castle LLC Infrared toaster
US10627116B2 (en) 2018-06-26 2020-04-21 Whirlpool Corporation Ventilation system for cooking appliance
US10619862B2 (en) 2018-06-28 2020-04-14 Whirlpool Corporation Frontal cooling towers for a ventilation system of a cooking appliance
CN113170545B (zh) * 2018-07-09 2023-12-29 博莱沃家居公司 烤箱内摄像头以及计算机视觉系统和方法
US10837652B2 (en) 2018-07-18 2020-11-17 Whirlpool Corporation Appliance secondary door
DE102018124378B4 (de) * 2018-10-02 2023-03-23 BIBA - Bremer Institut für Produktion und Logistik GmbH Vorrichtung und Verfahren zur Prozessüberwachung mehrerer Teiglinge in einer Prozesskammer sowie eine Prozesskammer mit einer derartigen Vorrichtung
CN109330356A (zh) * 2018-11-09 2019-02-15 珠海格力电器股份有限公司 烹饪器具
WO2020116814A1 (en) * 2018-12-06 2020-06-11 Samsung Electronics Co., Ltd. Heating cooker including three dimensional measuring device
JP7290415B2 (ja) * 2018-12-06 2023-06-13 三星電子株式会社 三次元計測装置および加熱調理器
DE102018221749A1 (de) * 2018-12-14 2020-06-18 BSH Hausgeräte GmbH Backofen und Steuerverfahren
US11287140B2 (en) 2019-01-04 2022-03-29 Whirlpool Corporation Cooking appliance with an imaging device
DE102019201332A1 (de) * 2019-02-01 2020-08-06 BSH Hausgeräte GmbH Haushalts-Gargerät und Verfahren zum Betreiben eines Haushalts-Gargeräts
DE102019203259A1 (de) 2019-03-11 2020-09-17 BSH Hausgeräte GmbH Optisches Erkennen eines Garguts
US11677901B2 (en) * 2019-03-12 2023-06-13 Marmon Foodservice Technologies, Inc. Infrared toaster
DE102019107846A1 (de) * 2019-03-27 2020-07-16 Miele & Cie. Kg Verfahren zum Betreiben eines Gargeräts und Gargerät
DE102019107812A1 (de) * 2019-03-27 2020-10-01 Miele & Cie. Kg Verfahren zum Betreiben eines Gargeräts und Gargerät
DE102019107859A1 (de) * 2019-03-27 2020-07-09 Miele & Cie. Kg Verfahren zum Betreiben eines Gargeräts und Gargerät
DE102019107834A1 (de) * 2019-03-27 2020-07-16 Miele & Cie. Kg Verfahren zum Betreiben eines Gargeräts und Gargerät
DE102019204533A1 (de) * 2019-04-01 2020-10-01 BSH Hausgeräte GmbH Verfahren zum Zubereiten eines Garguts mit optisch angezeigten Gargutzonen des Garguts, Gargerät und Computerprogrammprodukt
DE102019204531A1 (de) 2019-04-01 2020-10-01 BSH Hausgeräte GmbH Haushaltsgerät und Verfahren zum Bestimmen von Konturinformation von Gut
EP3758439B1 (en) * 2019-06-25 2022-10-26 Electrolux Appliances Aktiebolag Method and system for controlling an oven, and oven for heating food items
DE102019209198A1 (de) * 2019-06-26 2020-12-31 Robert Bosch Gmbh Haushaltsgerät
DE102019210426B3 (de) * 2019-07-15 2020-12-10 BSH Hausgeräte GmbH Steuereinheit und Verfahren zur Auswertung von Bilddaten in einem Hausgerät
US10819905B1 (en) * 2019-09-13 2020-10-27 Guangdong Media Kitchen Appliance Manufacturing Co., Ltd. System and method for temperature sensing in cooking appliance with data fusion
JP7236644B2 (ja) * 2019-10-01 2023-03-10 パナソニックＩｐマネジメント株式会社 加熱調理器
DE102019216682A1 (de) * 2019-10-29 2021-04-29 BSH Hausgeräte GmbH Bestimmen eines Ziel-Verarbeitungszustands eines zu behandelnden Garguts
US20210186032A1 (en) * 2019-12-19 2021-06-24 Whirlpool Corporation Monitoring system
DE102020107568B4 (de) * 2020-03-19 2022-02-03 Miele & Cie. Kg Verfahren zur Ansteuerung eines Gargeräts und Gargerät
CN113491454A (zh) 2020-04-06 2021-10-12 沙克忍者运营有限责任公司 能定位在支撑表面上的烹饪系统
KR102341238B1 (ko) * 2020-08-21 2021-12-20 주식회사 비욘드허니컴 사람간 협업을 위한 자동화 조리기기 시스템
DE102020126249A1 (de) * 2020-10-07 2022-04-07 Welbilt Deutschland GmbH Gargerät, insbesondere gewerbliches Gargerät
US11940153B2 (en) 2020-12-01 2024-03-26 GMG Products, LLC Fuel conditioner for grill
US11747087B2 (en) * 2021-01-06 2023-09-05 Bsh Home Appliances Corporation Household appliance including reflective door
CN113208449B (zh) * 2021-05-31 2022-10-11 广东美的厨房电器制造有限公司 用于烹饪设备的控制方法、控制装置以及烹饪设备
USD1005769S1 (en) 2021-09-08 2023-11-28 Newage Products Inc. Oven

Family Cites Families (20)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4224743A (en) * 1978-06-19 1980-09-30 Alternative Pioneering Systems, Inc. Food dehydrating machine
US4588297A (en) 1982-06-14 1986-05-13 Nippon Steel Corporation Optical profile measuring method
DE19748062C2 (de) 1997-10-31 2000-10-05 Bernward Maehner Verfahren und Einrichtung zur dreidimensionalen optischen Vermessung von Objekten
JP2002544510A (ja) 1999-05-14 2002-12-24 ３ディーメトリックス，インコーポレイテッド カラー構造化された光３ｄ撮像システム
JP2001099615A (ja) * 1999-09-30 2001-04-13 Nippon Crucible Co Ltd 対象物距離測定装置および対象物３次元形状測定装置
DE50113144D1 (de) 2000-06-28 2007-11-29 Bosch Gmbh Robert Vorrichtung zum bildlichen Erfassen von Stückgütern
JP3826111B2 (ja) * 2003-06-06 2006-09-27 株式会社東芝 加熱調理器
US7131529B2 (en) * 2003-07-01 2006-11-07 Casa Herrera, Inc. Oven conveyor alignment system apparatus and method
US10687391B2 (en) * 2004-12-03 2020-06-16 Pressco Ip Llc Method and system for digital narrowband, wavelength specific cooking, curing, food preparation, and processing
DE102006005874C5 (de) 2005-05-11 2017-05-18 Carl Zeiss Automated Inspection GmbH Verfahren zum berührungsfreien Vermessen
JP2007192518A (ja) * 2006-01-23 2007-08-02 Matsushita Electric Ind Co Ltd 高周波加熱装置
ATE432446T1 (de) 2006-11-02 2009-06-15 Electrolux Home Prod Corp Vorrichtung und verfahren zum bestimmen der temperatur im inneren eines garguts
DE102008024731B4 (de) 2008-05-19 2020-08-20 BAM Bundesanstalt für Materialforschung und -prüfung Verfahren und Vorrichtung zur Sinterung eines Objektes unter Bestimmung des geometrischen Oberflächenprofils des Objekts
EP2149755B1 (en) 2008-07-30 2012-12-05 Electrolux Home Products Corporation N.V. Oven and method of operating the same
UA106979C2 (uk) * 2009-03-05 2014-11-10 Пресско Текнолоджі, Інк. Система обробки для нагріву їжі, спосіб приготування, висушування або консервації харчового об'єкта та спосіб обробки харчового об'єкта
DE102010003115A1 (de) * 2010-03-22 2011-09-22 BSH Bosch und Siemens Hausgeräte GmbH Gargerät
GB2486165A (en) * 2010-11-30 2012-06-13 St Microelectronics Res & Dev Oven using a Single Photon Avalanche Diode (SPAD) array
EP3220058A1 (en) 2011-06-03 2017-09-20 Electrolux Home Products Corporation N.V. A cooking oven including an apparatus for detecting the three-dimensional shape of food stuff on a food stuff carrier
EP2798273A1 (en) * 2011-12-26 2014-11-05 Arçelik Anonim Sirketi Oven with optical detection means
EP2844187B1 (en) * 2012-05-03 2016-10-12 3Shape A/S Automated production of dental restoration

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
None *

Also Published As

Publication number Publication date
EP3152498A1 (de) 2017-04-12
US10228145B2 (en) 2019-03-12
US20170115008A1 (en) 2017-04-27
ES2835724T3 (es) 2021-06-23
CN106461230A (zh) 2017-02-22
WO2015185608A1 (de) 2015-12-10
DE102014210672A1 (de) 2015-12-17
PL3152498T3 (pl) 2021-05-31
CN106461230B (zh) 2019-04-12

Similar Documents

Publication Publication Date Title
EP3152498B1 (de) Gargerät mit lichtmusterprojektor und kamera
DE102016215550A1 (de) Feststellen eines Bräunungsgrads von Gargut
WO2018188913A1 (de) Betreiben eines gargeräts
DE102019120008B4 (de) Verfahren zum Betreiben eines Gargeräts und Gargerät
DE102016107617A1 (de) Verfahren zum Betreiben eines Gargeräts und Gargerät
DE102015211954B4 (de) Verfahren und Vorrichtung zum berührungslosen Vermessen einer Objektoberfläche
DE102013110644B3 (de) Verfahren zum Garen von Lebensmitteln in einem Gargerät
DE102014109432B4 (de) Laserscanner und Verfahren
DE102014114901A1 (de) Gargerät sowie Verfahren zur Erfassung eines Prozessparameters eines Garprozesses
EP3918880A1 (de) Haushalts-gargerät und verfahren zum betreiben eines haushalts-gargeräts
EP4229336A1 (de) Verfahren zur bestimmung des zeitpunktes für die reinigung eines garraums eines gargeräts
WO2010130567A2 (de) Kochfeld und verfahren zum erhitzen von auf dem kochfeld aufgestellten kochgefässen
DE102013114227A1 (de) Gargerät mit Beschickungserkennung
DE102017220884A1 (de) Gargerät mit Bildaufnahmeeinrichtung
DE102014210673A1 (de) Bestimmen einer Einschubebene eines Gargutträgers
DE102011075187B3 (de) Kochfeld
DE102019201330A1 (de) Mikrowellengerät und Verfahren zum Betreiben eines Mikrowellengeräts
DE10125247C1 (de) Haushaltsgerät mit einem Garraum
EP3904769A1 (de) Gargerät mit elektronischer bedienfehlererkennung
DE102015101174A1 (de) Kochfeld und Verfahren zum Projizieren eines Bildes
DE102019107828B4 (de) Verfahren zum Betreiben eines Gargeräts und Gargerät
DE102015105128A1 (de) Verfahren und Vorrichtung zur Messung des Glanzgrads und/oder der Mattheit von Gegenständen
WO2019170447A1 (de) Interaktionsmodul
US20220287508A1 (en) Vision system for a toaster
DE102023123163A1 (de) Beleuchtungsvorrichtung und Beleuchtungsverfahren mit Messung und Anzeige des Abstands

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20170105

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

DAV Request for validation of the european patent (deleted)
DAX Request for extension of the european patent (deleted)
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

17Q First examination report despatched

Effective date: 20190506

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

INTG Intention to grant announced

Effective date: 20200619

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE PATENT HAS BEEN GRANTED

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

Free format text: NOT ENGLISH

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: AT

Ref legal event code: REF

Ref document number: 1333844

Country of ref document: AT

Kind code of ref document: T

Effective date: 20201115

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 502015013793

Country of ref document: DE

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

Free format text: LANGUAGE OF EP DOCUMENT: GERMAN

REG Reference to a national code

Ref country code: NL

Ref legal event code: MP

Effective date: 20201111

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210311

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210211

Ref country code: RS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20201111

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20201111

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210212

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210311

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20201111

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20201111

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210211

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG9D

REG Reference to a national code

Ref country code: ES

Ref legal event code: FG2A

Ref document number: 2835724

Country of ref document: ES

Kind code of ref document: T3

Effective date: 20210623

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20201111

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20201111

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20201111

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20201111

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20201111

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20201111

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20201111

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 502015013793

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20201111

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

26N No opposition filed

Effective date: 20210812

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: NL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20201111

Ref country code: AL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20201111

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20201111

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20201111

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

REG Reference to a national code

Ref country code: BE

Ref legal event code: MM

Effective date: 20210630

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20210603

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LI

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20210630

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20210603

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20210630

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210311

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20210630

REG Reference to a national code

Ref country code: AT

Ref legal event code: MM01

Ref document number: 1333844

Country of ref document: AT

Kind code of ref document: T

Effective date: 20210603

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: AT

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20210603

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: HU

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO

Effective date: 20150603

P01 Opt-out of the competence of the unified patent court (upc) registered

Effective date: 20230504

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20201111

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: FR

Payment date: 20230622

Year of fee payment: 9

Ref country code: DE

Payment date: 20230630

Year of fee payment: 9

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: TR

Payment date: 20230530

Year of fee payment: 9

Ref country code: PL

Payment date: 20230518

Year of fee payment: 9

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: IT

Payment date: 20230630

Year of fee payment: 9

Ref country code: GB

Payment date: 20230622

Year of fee payment: 9

Ref country code: ES

Payment date: 20230719

Year of fee payment: 9

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20201111