BE1030689B1 - Method for controlling a flushing device, method for training a model, device and flushing device - Google Patents

Abstract

The invention relates to a method for controlling a washing device (100), wherein the washing device (100) comprises at least one camera unit (140) for recording an interior (107) of the washing device (100). The method comprises a step of reading in a camera signal (145) from the camera unit (140), wherein the camera signal (145) represents an image of the interior (107) captured by the camera unit (140), and a step of evaluating the image using the camera signal (145) and using a model through which at least two different loading states of the interior (107) can be distinguished. The method further comprises a step of outputting a control signal (150) for controlling a washing program for washing objects (120) in the interior (107) if the image represents a loading state of the interior (107) which corresponds to at least a starting load quantity.

Description

Description

Method for controlling a flushing device, method for training a model,

Device and flushing device

The invention relates to a method for controlling a flushing device, a method for training a model, a device and a flushing device.

In common household washing appliances, such as dishwashers

Interior cameras can be integrated to record a load status. In addition, you can

There will be options for automatically starting a washing program, in which a specific load level is assigned a respective fill level number, or

Threshold values can be set in the form of numbers.

The invention aims to provide an improved method for controlling a

Flushing device, an improved method for training a model, an improved

To create a device and an improved dishwashing device. According to the invention this is

Task solved by a method for controlling a flushing device, a method for training a model, a device and a flushing device with the features of the main claims. Advantageous refinements and further developments of the invention result from the following subclaims.

The advantages that can be achieved with the invention include, in addition to an optimized starting process of a washing cycle in a washing machine, the simple application and customizability of the method.

A method for controlling a washing device is presented, wherein the washing device comprises at least one camera unit for detecting an interior of the washing device.

The method includes a step of reading in a camera signal from the

Camera unit, wherein the camera signal is an image of the image captured by the camera unit

Interior represents, and a step of evaluating the image using the

Camera signal and using a model through which at least two different loading states of the interior from the image of the

Interior can be distinguished. The method further includes a step of:

Outputting a control signal for controlling a washing program for washing 30 objects in the interior if the image represents a loading state of the interior that corresponds to at least a starting load quantity of items to be cleaned arranged in the interior.

The dishwashing device can be, for example, a dishwasher, which can be designed to accommodate dishes, cutlery and other household utensils in the interior, for example in appropriate baskets and additionally or alternatively

Cutlery drawer. The dishwashing device can be electronically networked with other devices and additionally or alternatively with a server outside the household in order to receive, output and automate various data in the form of signals

Processes, such as selecting a suitable washing program, can be carried out largely without manual intervention by users. The dishwashing device can, for example, be integrated or able to be integrated into the network of a so-called smart kitchen or a smart home. The washing device can be designed to carry out an automatic washing start for cleaning the objects arranged in the interior, for example dishes and cooking utensils such as pots and pans. The

The start of such a rinsing process can depend on the load status of the interior or the baskets arranged therein. Such a loading condition can be detected, for example, by one or more cameras arranged in or on the interior. For example, the camera unit can be attached to a

Be arranged inside a door closing the interior and be designed to take a photo of the interior each time the door is closed.

Using the camera signal, such an image can be sent to, for example

Device for evaluating image data are provided, which can be arranged, for example, as an electronic device in the dishwashing device or which can be accessed, for example, via a cloud. The image provided by the camera unit can now be evaluated using a model to determine the current

Check loading status. The model can be, for example, one

Algorithm or a neural network act, which can, for example, have been trained to use the camera signal to compare a quantity of recorded items to be washed in the interior with similar, stored image data and to decide whether the recorded quantity requires a flushing start or not. For example, such a neural network can have been trained with training images from experimental households

Operation of the flushing device can be compared with currently captured images. It can accordingly be designed to make a binary decision as to whether a

Camera image taken when the door was closed shows a flushing interior that is sufficiently filled for an autonomous flushing start. If a loading condition is reached that corresponds to a learned starting loading quantity, then a control signal is output to determine this

to control the start of a rinse cycle. Such a signal can, for example, be output to a device of the washing device, which can be designed to automatically initiate a washing cycle. This advantageously allows an optimal time to be determined at which the dishwasher is sufficiently full to start it automatically,

so that users always have enough clean dishes available without unnecessary use of resources. In addition, the simple

The distinction between “full” and “not full” simplifies the application of the method, since there is no need to manually assign numbers to the camera images, for example, as accurately as possible, which could represent the degree of loading. All that remains is to check whether the captured image shows a sufficiently filled sink interior. Just with this information ("Starting makes sense" or "Starting is not worth it yet") and the associated training images, an algorithm can be enabled to optimally control a flushing process.

According to one embodiment, in the step of evaluating, the image of a first

Image category or a second image category can be assigned. The first one can

Image category corresponds to a not full loading state of the interior and the second image category can correspond to the full loading state. For example, images captured by the camera unit, for example enriched with machine data, can be automatically assigned the class labels “start” or “no start”.

Images that correspond to a starting load quantity trained into the model, for example when all baskets of the dishwasher are at least eighty percent full, receive the label “Start” and the control signal can be output accordingly. Other images that can arise when the dishwasher door is closed, for example with a time interval (for example a predetermined or predeterminable time interval

Minimum distance, approximately 30 minutes, before a start) or after

Clearing out can also be taken into account and automatically assigned the class label “none.”

Start". This offers the advantage that a simple, binary decision can be made without additional intermediate values.

According to a further embodiment, in the reading step a

Image information signal can be read in, wherein the image information signal can represent at least one additional piece of information about the image. In the step of

Evaluating the image can be evaluated using the image information signal.

For example, in the reading step, in addition to the individual images of the 3o sink interior, metadata can also be read in, such as time stamps or the

Information as to whether the image was created after the door was closed manually or by a flush start or end. Advantageously, using the

Image information signal a current state of the dishwashing device can be optimally estimated.

According to a further embodiment, the image can be discarded in the evaluation step if a predetermined image parameter is not met. For example, images can be discarded that have a technical defect and cannot be meaningfully evaluated, for example if they were taken over- or underexposed. This has the advantage that only image data that can be used for the process can be saved, which saves energy and storage space. Optionally, methods of so-called undersampling or oversampling can also be used.

According to a further embodiment, in the reading step, a user signal can be read in, which represents an input from a user on a user interface. In the output step, the control signal can be generated using the

User signal is output. For example, dates such as days of the week or

Times for a desired rinse cycle can be set by users in order to advantageously individualize the method to different locations with different users. This can give different households the opportunity to

Preferred times for the automatic start, or for the desired one

Define program end time (desired weekday and time). Optionally can also

Data on usage behavior (loading, unloading or starting the machine), etc. are included in order to be able to advantageously calculate the probability of a sensible start based on these general conditions.

According to a further embodiment, in the reading step, a sensor signal can be read from a sensor interface to a sensor, wherein the sensor signal

At least one environmental parameter of an environment detected by the sensor

Washing device can represent. In the output step, the control signal can be included

Using the sensor signal. For example, the flushing device can be combined with an existing photovoltaic system, wherein the sensor can be designed to detect a current flow produced by the system. In such a case, using the sensor signal, an automatic start of a washing cycle can be coupled to high power production in order to advantageously make optimal use of the available power. In addition, an option is also conceivable in which the presence of at least one household member can be a prerequisite for a flush to start

Do not let the dishwasher start automatically when no one is at home. Both for the

For the option of sunshine-related electricity production as well as for the presence condition, additional sources of information can optionally be used, such as

Connection to a PV system, local weather data, data from a SmartHome for the

Presence detection using motion detectors or the location of mobile devices, for example

Example using so-called geo-fencings.

According to a further embodiment, in the step of outputting, the control signal can be output to a user interface to a user. This can be the case

Rinsing program can be controlled in response to a response signal, whereby the

Response signal can represent an input from the user on the user interface.

For example, the user interface can be designed as a display on the dishwashing device or as a corresponding app for a mobile device, for example a smartphone. 5 Using the control signal, a user can, for example, be given a push button

Notification is sent to the mobile device and the options “Start” and “Move” are offered together with the current camera image. If a household member chooses the

If you use the “Postpone” option, the start of the flushing can be postponed for a configurable time interval and requested again later via a push notification.

Advantageously, a flushing start can thereby be optimized for a user's current situation.

In addition, a method for training a model to control a flushing device is presented. The method includes a step of reading in a first image data set, which represents a plurality of first images of an interior of a washing device, which are assigned to a first image category, and reading in a second image data set, which represents a plurality of second images of an interior of a washing device, which are assigned to a second Image category are assigned. The method further includes a step of:

Adjusting the model so that the first images entered into the model are the first

Image category are assigned and the second images entered into the model are assigned to the second image category in order to train the model.

The model to be trained, which can also be referred to as a basic model, can be an algorithm or a neural network. After

The network can advantageously be pre-trained in a variant of the one described above

Method for controlling a dishwashing device can be used. The model can be used for this

Example can be trained with training images from test households. For this purpose, the necessary training data can first be collected. These can, for example,

Collected in the laboratory or in experimental households or, if necessary, in the field.

The time course of the images and, for example, additional ones can be changed

Metadata represent a cyclical profile that can be roughly divided into three repeating phases: The first phase, the loading phase, can ideally be done with one

Start with an image that shows an empty sink interior, and then images that show an increasingly full sink interior as time progresses. On the

The loading phase can be followed by a rinsing phase, which can be triggered by the dishwasher starting the rinsing process. The first image of this phase can therefore ideally show a sink interior fully filled with soiled loads. Images optionally collected during rinsing are no longer relevant for this application. This

Phase can conclude with an image of a wash interior fully filled with clean loads, automatically triggered by the end of the wash cycle. Optionally, there is now an unloading phase that can contain images that can be triggered by opening and closing the dishwasher door without the dishwasher having been emptied.

Ideally, this phase does not exist and a new loading phase follows, which ideally begins with a picture of the completely empty sink interior. The

Images from the loading and unloading phase can, for example, be assigned to the first image data set and thus to the first image category (“no start”) and the images from the rinsing phase can be assigned to the second image data set and thus to the second image category (“start”). 11 Since these phases are due to user behavior that is not always predictable or due to possible technical problems (e.g. offline times due to internet failure).

To meet ideal conditions, manual control of the images and data by a human may be required. However, as soon as a first functioning model/network exists, it can be used for pre-checking in order to carry out the manual

Reduce verification effort.

According to one embodiment, the method may include a step of retraining the

Model after the fitting step. In the step of

After training, at least one further first image data set must be read in, which is one

A plurality of further first images of the interior can represent those of the first

Image category can be assigned. In addition, at least another second one can

Image data set is read in, which contains a plurality of further second images of the

Can represent interior space, which can be assigned to the second image category.

The model can be adjusted in such a way that the other first images correspond to the first

Image category are assigned and the other second images are assigned to the second image category in order to retrain the basic model.

The basic model can advantageously be specific through the step of retraining

The needs of a user or household can be adapted. For example, according to the same principle as when training the basic model by the dishwasher in the user's household, start and non-start images can be collected and automatically checked by existing neural networks with the, for example local, basic model for the correct assignment of the class labels. For example, slight inaccuracies may be acceptable in order to approximate individual usage behavior. Once enough images over a certain period of time for both

Class labels are present, this can be done (if necessary after approval by the user).

Basic model can be retrained with the collected data using transfer learning.

As long as the computing and storage capacities of a dishwasher are not sufficient for training neural networks, this step can be carried out on the provider's servers (in the cloud), for example. This would have the additional advantage that in

Data collected from target households could further refine and improve the basic model. The easiest and most obvious way is to retrain the basic model with at least one data set enriched with further training examples from customer households. Alternatively, several retrained models (possibly together with the basic model) can be combined to form so-called ensembles, the overall result of which can correspond to the mean of the individual results. According to the

The “knowledge of the crowd” principle can in many cases classify ensembles better than individual neural networks. Overall, the step of retraining offers the advantage that an individually retrained network adapts to the respective ones in a much more targeted manner

can be adapted to the circumstances and needs of individual households. In addition, a neural network that is individually retrained for each user can

Optimally reflect the circumstances of a household, as there are differences from household to household in terms of loading methods (for example, large items are put away rather roughly at the bottom) and types of items to be washed (sometimes more porcelain, sometimes more plastic, sometimes more pots, etc.).

pans). In addition, specialized ones can advantageously be developed over time

Develop basic models for different countries and food cultures.

The approach presented here also creates a device that is designed to

To carry out, control or implement steps of a variant of a method presented here in appropriate facilities. Also through this one

Embodiment variant of the invention in the form of a device, the object on which the invention is based can be solved quickly and efficiently.

The device can be designed to read in input signals and

Using the input signals to determine and provide output signals. A

Input signal can, for example, represent a sensor signal that can be read in via an input interface of the device. An output signal can be a control signal or a

Represent data signal that can be provided at an output interface of the device. The device can be designed to receive the output signals

To determine the use of a processing rule implemented in hardware or software. For example, the device can have a logic circuit, an integrated one

Include a circuit or a software module and, for example, as a discrete one

Component can be realized or encompassed by a discrete component.

In addition, a dishwashing device with a variant of the previously presented device and with

Presented at least one camera unit for recording an interior of the dishwashing device.

For example, the dishwashing device can be a dishwasher for cleaning

Act dishes that can be designed to network with other household appliances within a home network. In other words, the dishwashing device can be a dishwasher with a camera and image-dependent, configurable autostart. Through this combination of a dishwashing device with a camera unit and a device, the advantages mentioned above can be optimally implemented.

A computer program product or computer program is also advantageous

Program code stored on a machine-readable carrier or storage medium such as a

Semiconductor memory, a hard drive memory or an optical memory can be stored. Is the program product or program on a computer or a

Device executed, the program product or program for implementation,

Implementation and/or control of the steps of the method according to one of the embodiments described above can be used.

Even if the described approach is described using a household appliance, the device/method described here can accordingly be used in connection with a commercial or professional device, for example a medical device, such as a cleaning or disinfection device, a small sterilizer, a

Large-scale disinfector or a container washing system can be used.

An exemplary embodiment of the invention is shown purely schematically in the drawings and is described in more detail below. It shows

Figure 1 shows a schematic representation of an exemplary embodiment of a dishwashing device with a device for controlling a method for controlling the

dishwashing device;

Figure 2 shows a flowchart of a method for controlling a flushing device according to an exemplary embodiment;

Figure 3 is a block diagram of a device according to an exemplary embodiment;

Figure 4 is a flowchart of a method for training a model

Controlling a flushing device according to an exemplary embodiment;

Figure 5 shows a flowchart of another method for training a model for controlling a flushing device according to an exemplary embodiment; and

Figure 6 is a flowchart of a method for retraining a model

Controlling a dishwashing device according to an exemplary embodiment

Figure 1 shows a schematic representation of an exemplary embodiment of a washing device 100 with a device 105 for controlling a method for controlling the washing device 100. The washing device 100 shown here is an example

Dishwasher. The dishwashing device 100 has an interior 107 in this

Embodiment can be closed with the door 110. Within the interior 107 there are three baskets 115, 116, 117 for holding various objects as items 120a, 120b to be washed, which are, for example, dishes and cooking utensils. In order to clean these objects, the washing device includes 100 in this

Embodiment of a spray device with, for example, three rotatable spray arms 125, 126, 127. By means of spray jets emerging from the spray arms 125, 126, 127, the

Objects 120 can be cleaned during a washing program. The pressure for the spray jets emerging from the spray arms 125, 126, 127 can be generated, for example, by a circulation pump 130 11. A water switch 131 is also an example between the individual ones

Spray arms 125, 126, 127 switchable. Such and other actions of the flushing device 100 can be controlled in one exemplary embodiment by means of a device control device 135.

In order to automatically control a cleaning program optimized for a current load status of the washing device 100, the washing device also has the device 105 and a camera unit 140 for recording the interior 107. In one exemplary embodiment, the

Camera unit 140 arranged on the door 110 and designed to be responsive to a

Opening or closing the door 110 takes an image of the interior 107 and thereby also of objects 120 arranged therein. The image captured by the camera unit 140

Interior 107 can be provided to device 105 using a camera signal 145.

The device 105 is designed to read the camera signal 145 and that from the

Camera unit 140 captured image using the camera signal and under

Evaluate the use of a model. In one embodiment, the model is the

Device 105 for this purpose is designed as an example of a pre-trained neural network in order to distinguish between two different loading states of the interior 107. In one exemplary embodiment, a not fully loaded interior 107, in which, for example, only a few or no objects 120 are arranged, can be distinguished from a fully loaded interior 107, which, for example, has at least eighty

Percent filled with objects to be washed 120.

If the image captured by the camera unit 140 and read in using the camera signal 145 represents a loading state of the interior that corresponds to at least a starting loading quantity, then a control signal 150 can be output by the device 105 in order to control a washing program for washing the objects 120. In one

In the exemplary embodiment, the control signal 150 can be provided to the device control device 135 in order to control a corresponding actuator system of the washing device 100.

Figure 2 shows a flowchart of a method 200 for controlling a flushing device according to an exemplary embodiment. The method 200 shown here can be used to control a flushing device, as described in the previous figure.

The method 200 includes a step 205 of reading a camera signal from the

Camera unit, wherein the camera signal is an image of the image captured by the camera unit

The interior of the dishwasher is represented.

This is followed by a step 210 of evaluating the image using the

Camera signal and using a model through which at least two different loading states of the interior can be distinguished.

Furthermore, the method includes a step 215 of outputting a control signal

Controlling a washing program to wash objects in the interior when the image has a

Represents the loading state of the interior, which corresponds to at least a starting load quantity. ı5 In one exemplary embodiment, in step 205 of reading in, a

Image information signal is read in, which represents at least one additional piece of information about the image, for example a recording time of the captured image. Only as an example in step 210 of evaluating the image using the

Image information signal evaluated.

Additionally or alternatively, in one exemplary embodiment, in the reading step

User signal read in, which represents an input from a user on a user interface. An example is the user interface, which is a display on the

Washing device or, for example, an app on a mobile device can act, information regarding the desired washing program is entered, for example a desired one

Time and/or a day of the week. In the output step 215, the control signal is then output using the user signal, merely as an example.

Optionally, in step 205 of reading, a sensor signal from a

Sensor interface read to a sensor, wherein the sensor signal represents at least one environmental parameter of an environment of the dishwashing device detected by the sensor. By way of example, the sensor is a device for detecting a person. In step 215 of outputting, the control signal is only used as an example

Using the sensor signal output, for example under the premise that a

The flushing program is only started when a person has been detected. In another

In the exemplary embodiment, the sensor can also detect other environmental parameters,

for example, an available power flow or a mobile device located within a predetermined area.

In one exemplary embodiment, the image is also assigned to a first image category or a second image category in step 210 of evaluation, the first image category corresponding to an incomplete loading state of the interior and the second

Image category corresponds to the full loading status.

The image is simply discarded as an example if a predetermined image parameter is not met, for example if the image is overexposed or underexposed.

In one embodiment, the control signal is also output to a user interface to a user in step 215 of output. The user interface is, for example, an app on the user's mobile device, whereby using the control signal, for example, a push message is generated which informs the user about a possible flushing start. The flushing program is then controlled in response to a response signal merely by way of example, the response signal representing an input from the user at the user interface. In one embodiment, under

Using the response signal, a desired setting of the user is implemented, for example a confirmation of the start of the washing program or a rejection of the same.

In other words, the process presented here can be described as follows:

In contrast to other methods for controlling a flushing device, in the method 200 shown here, the degree of loading is not determined as a number and the

Comparison with a threshold value (which can be customized by the user) is required to determine the optimal time for the dishwasher to start rinsing autonomously. The

Method 200 does not require a loading level in the form of a number, enables the

Users can still adapt the process to their personal needs

Usage preferences.

This is realized with an algorithm (neural network), for example with

training images from test households. Enriched by

Machine data can contain images where the dishwasher is actually from

When the experimental household was started, the class label “Start” must have been automatically assigned to it. Remaining images, which were created when the dishwasher door was closed, with a time interval (for example a predetermined or predefinable time interval

Minimum distance, approximately 30 minutes, before a start) or after

Cleanouts were also taken into account and automatically assigned the class label “None.”

Start". A basic model trained with such examples can be used with sufficient

Data availability and sufficient tests are already included in the delivery condition of the dishwashers

Camera in the device or via the cloud can be made equally available to all users for use. It is able to make the binary decision as to whether a camera image that was created when the door was closed shows a flushing interior that is sufficiently filled for an autonomous flushing start, without having to determine a load level.

In order to make the operation of multiple networks (in an ensemble) more efficient, this can be done

Overall behavior of an ensemble can optionally be reduced to a single one using “Knowledge Distillation”.

network. Knowledge Distillation (KD) was originally developed to...

To transfer the behavior of large, resource-hungry networks to smaller, more efficient networks without significantly affecting classification accuracy. The behavior of ensembles can also be transferred to individual networks using KD.

Users can therefore program their dishwasher using the dishwashing scheduler - but with one or more preferred times. As soon as the upper basic model or an individualized model classifies a camera image with "Start", within the next

Preferential time the dishwasher starts automatically according to the parameters set in the flush scheduler.

Figure 3 shows a block diagram of a device 105 according to an exemplary embodiment.

The device 105 shown here corresponds to or is similar to that in the previous one

Figure 1 described device, with the difference that the device 105 in this

Embodiment based on one described in the previous Figure 1

Flushing device 100 can be operated on an external server or in a so-called cloud.

The device 105 includes a model 300 designed, for example, as a neural network and is designed to control a method as described in the previous Figure 2.

For this purpose, the device 105 is designed to read in a camera signal 145 from the camera unit 140, the camera signal 145 representing an image of an interior of the dishwasher 100 captured by the camera unit 140. Using the camera signal 145 and the model 300, the image provided by the camera unit 140 is through the

Device 105 can be evaluated. The 300 model makes it possible to distinguish between at least two different load states in the interior. In one embodiment, the image is of a first image category or a second

Image category can be assigned, whereby the first image category corresponds to a not fully loaded state of the interior and the second image category corresponds to a fully loaded state.

The device 105 is further designed to output a control signal 150 in order to

To activate the washing program for washing objects in the interior when the picture shows one

Represents the loading state of the interior, which corresponds to at least a starting load quantity. In one exemplary embodiment, the control signal 150 can be output to a device control device 135 of the washing device 100, whereby the

Device control device 135 is designed, for example, to control an actuator of the washing device 100 to carry out a washing program.

In one embodiment, the camera unit 140 is designed to additionally

Camera signal 145 to provide an image information signal 305, the 101 image information signal 305 exemplifying additional information to that provided

Image represents, for example, a timestamp. The device 105 is designed to do this

Read in image information signal 305 and evaluate the image information, for example, using model 300.

In addition, the device 105 is designed in one exemplary embodiment to read in a user signal 310, which represents an input from a user at a user interface 315a. In one embodiment, the user interface 315 is exemplary as

Touch display is formed on the dishwashing device 100. Various user-specific inputs can only be entered into the user interface 315 by way of example, for example desired times or days of the week for carrying out a washing program

Or the preference for a certain program, for example a water-saving option. Using the user signal 310 are such and similar

Inputs applicable when controlling the flushing device 100 by the device 105.

In one embodiment, the device 105 is further designed to be a

Sensor signal 320 is read in from a sensor interface 324 to a sensor 328, wherein the sensor signal 320 detects at least one of the sensor 328

Environmental parameters of an environment of the flushing device 100 are represented. The sensor 328 is designed in one embodiment to detect current production of an existing one

To record photovoltaic systems and provide relevant information using the

Sensor signal 320 to be provided to the device 105. An automatic start of one

The washing program can therefore be linked to high electricity production and that

Control signal 150 can be output in this case using sensor signal 320.

Additionally or alternatively, the control signal 150 can also be output to a user interface 315b. By way of example only, a message can be generated on the user interface in response to the control signal 150, informing the user about a possible

Rinse start informed. For example, the user has the choice of agreeing to the proposed flush start or rejecting or postponing it. The selected response is in one embodiment using a response signal 330 from the

Device 105 can be read.

In the exemplary embodiment shown here, the user interface 315b, on which a flushing start can be selected or rejected, for example, is a different user interface 315b than the previously described user interface 315a for outputting the user signal 310. The former is only an example App on a mobile device, while the latter is integrated into the dishwashing device 100. The representation chosen in this figure is chosen for the sake of clarity. In other

In exemplary embodiments, only one of the two user interfaces 315a, 315b can be available or, alternatively, both user interfaces 315a, 315b can be designed to read in both the control signal and the user signal 310 as well

Output response signal 330.

Figure 4 shows a flowchart of a method 400 for training a model

Controlling a dishwashing device according to an exemplary embodiment. The method 400 can be used, for example, for training a model, as described in the previous Figure 3, in order to control a flushing device, as described in the previous Figure 1.

The method 400 includes a step 405 of reading in a first image data set, which represents a plurality of first images of an interior of a washing device, which are assigned to a first image category, and reading in a second image data set, which represents a plurality of second images of an interior of a washing device, which assigned to a second image category.

Furthermore, the method 400 includes a step 410 of adapting the model such that the first images are assigned to the first image category and the second images are assigned to the second image category in order to train the model.

In one embodiment, the method 400 also includes a step 415 of

Retraining the model after step 410 of fitting. In the exemplary step 415 of retraining, at least one further first image data set is read in, which represents a plurality of further first images of the interior, those of the first

Image category are assigned. In addition, at least one further second image data set is read in, which represents a plurality of further second images of the interior that are assigned to the second image category. The model is adapted in such a way that the further first images are assigned to the first image category and the further second images are assigned to the second image category in order to retrain the basic model.

In other words, in the method 400 presented here for training a neural network, images in which the dishwasher was actually started by the test household, enriched by machine data, are automatically given the class label “Start”. Other images, which are created when the dishwasher door is closed, are included s a time interval (for example a predetermined or predeterminable time interval

Minimum distance, approximately 30 minutes, before a start) or after

Clearouts are also taken into account and automatically given the class label “None.”

Start". Before training the basic model (neural network), these images should first be taken from people, and later - at least partially - from already existing neural ones

networks are checked. In the latter case, human testing should only be carried out in borderline cases where an already trained network fails.

In order to adapt this basic model to usage-specific needs, the

Step 415 of retraining can be carried out. This uses the same principle as when training the basic model with the user's dishwasher

Non-start images are collected and automatically checked by existing neural networks with the (local) basic model for the correct assignment of the class labels. Slight inaccuracies are acceptable in order to be able to approximate individual usage behavior. As soon as there are enough images for both class labels over a certain period of time, the basic model can be retrained with the collected data using transfer learning (if necessary after approval by the user). As long as the calculation and

If the memory capacity of a dishwasher is not sufficient for training neural networks, this step should be carried out on the manufacturer's servers (in the cloud), for example. Purely theoretically, there is nothing stopping you from re-training locally in the dishwasher. However, there is a legitimate interest in post-training for development purposes

models, as well as the training data used for this. This makes it possible

Further refine and improve the basic model. The easiest and most obvious way is to retrain the basic model with a data set enriched with additional training examples from different households. Alternatively, several retrained models (possibly together with the basic model) can be created into so-called

Combine ensembles whose overall result corresponds to the average of the individual results. In accordance with the “knowledge of the crowd” principle, ensembles classify better in many cases than individual neural networks.

The described method 400 reduces the amount required for implementation

Energy expenditure by eliminating the need to create images that are as accurate as possible by hand

Assign numbers that represent the degree of loading. All that remains is to check whether the collected start images actually show a sufficiently filled dishwasher interior or whether the dishwasher was started prematurely without being sufficiently filled. With just this information ("Starting makes sense" or "Starting is not worth it yet") and the associated training images, it is already possible for an algorithm to determine an optimal point in time at which the dishwasher is sufficiently full to start it automatically so that users always have enough clean dishes available without unnecessary use of resources. Also the

The algorithm can be customized and adapted to specific preferences, even without the user having to specify an abstract number

Loading level would have to be set - the algorithms learn automatically from the user's own

Examples, including automatic control of the individualized algorithms. Such an individually retrainable network can be tailored specifically to the respective circumstances

Adapt to the needs of individual households.

Figure 5 shows a flowchart of another method 500 for training a

Model for controlling a dishwashing device according to an exemplary embodiment. The other method 500 shown here is similar to the method for training a model described in the previous FIG. 4, with the difference that the method 500 shown here includes additional and alternative steps.

The method 500 includes a step 505 of recording. In this step, an image is taken as an example when the door is closed or when the dishwashing device starts a (manual) program and, for example, into a cloud

Transmitted.

In the following step 510 of sorting, a plurality of images are according to

Machine data sorted by the manufacturer within the cloud as “Start” and “No Start” images.

The method 500 further includes a step 515 of checking, in which the in the

Cloud sorted images can be checked. This step is optionally carried out by humans and/or an existing algorithm.

This is followed in step 520 by training the model, which can also be referred to as the basic model, with the collected and processed data. This step is also only carried out in the cloud as an example.

In the following step 525 of providing, the checked basic model finally becomes available

Use via the cloud or delivered on the device.

In one exemplary embodiment, after step 525 of providing, the previous steps 505, 510, 515, 520 are carried out again, so that method 500 is used as a closed loop for continuous improvement of the basic model.

Figure 6 shows a flowchart of a method 500 for retraining a model for controlling a flushing device according to an exemplary embodiment. The other method 600 shown here can be used as a retraining step, as in the previous one

Figure 4 was described can be applied.

The method 600 includes a step 605 of recording. In this step, for example, an image is recorded when the door is closed or when the dishwasher program starts (manually). This and others recorded

For example, images are both stored in the device and transferred to a cloud in a step 607 for quality assurance and development purposes.

In the following sorting step 610, the images are sorted as “Start” and “No Start” images according to machine data.

This is followed by a step 615 of checking, in which the images and their sorting are checked by an existing basic model, with slight inaccuracies being tolerated. For example, if a start time is too far ahead of the one specified by the

Basic model has been determined, the start image is not taken into account.

The method 600 further includes a step 820 of retraining, in which the

The basic model is retrained with the collected and processed data using transfer learning.

In the subsequent step 625 of the use, the retrained model is made available for use via the cloud or directly in the device. Additionally, in a transfer step 630, the model is transferred to the cloud for quality assurance and development purposes.

In one embodiment, after step 625 of providing, the previous steps 605, 610, 615, 620 are carried out again, so that method 600 is used as a closed loop for continuous improvement of the model. All steps 605, 610, 615, 620, 625 can optionally be carried out in the device or in the cloud.

Overall, the procedures described above can be described as follows:

First, training data is collected to train a basic model (neural network). These are first collected in the laboratory or in experimental households or, if necessary, in the field. This doesn't just include the individual images

The interior of the sink, but also metadata, such as timestamps and whether the image is after

caused by the user closing the door or by starting/ending a flush. The temporal progression of the images and data thus forms a cyclical profile that can be roughly divided into three repeating phases:

The first phase, the loading phase, ideally begins with an empty image

Sink interior shows and includes images that show an increasingly full s sink interior as time progresses. The images of the loading phase are exclusively provided by

User loading interactions generated. The loading phase is followed by the rinsing phase, which is triggered by the user starting the dishwasher. The first picture of this phase therefore ideally shows a fully filled container with a dirty load

Sink interior. Images collected during rinsing are optional for this one

Use case no longer relevant. This phase ends with an image of a wash interior fully filled with clean loads, automatically triggered by the end of the

rinse cycle. Optionally, there is now a discharging phase that contains images that were triggered by the user opening and closing the dishwasher door without the dishwasher being emptied. Ideally, this phase does not exist and a new loading phase follows, which ideally begins with a picture of the completely empty sink interior.

Since these phases are caused by user behavior that is not always predictable or by possible technical problems (e.g. offline times due to internet failure).

If the conditions correspond to ideal conditions, manual control of the images and data by a human is always required. However, as soon as a first functioning model/net exists, it can be used for pre-checking in order to carry out the manual

Reduce verification effort.

Ideally, a start image of a fully loaded wash interior and several images from the loading phase are generated per wash cycle. The start image receives (after checking, for

example, whether contrary to expectations the flushing was not carried out with a full load) the class label “Start” and the images of the loading phase the class label “No Start”.

A data set of images is gradually created that is divided into two classes. Since significantly more images are generated during the loading phase, this class is, as expected, overrepresented. In order to correct this imbalance,

Methods of so-called under- or oversampling are used so that the neural network can learn a balanced distribution during training. Optionally, a rejection class can be added for images in which an overexposed or underexposed image was taken that cannot be meaningfully evaluated (this can be done later in the

During operation, the dishwasher may attempt to take another image in order to capture an evaluable image).

This data set can now be used to train a neural network. If newer and/or better training examples are available, this process can be repeated iteratively to further improve and, if necessary, change the neural network

To adapt and expand conditions (for example as an adaptation to international markets with different loading parts). Ultimately, this network is operated either in the cloud or on the device in the respective household.

Aside from the fact that such a basic model can be available for each target market, it can be individualized for each user. There are several different approaches for this.

The first option uses a small amount of sample data from each

household has its own network retrained using transfer learning. In principle, this can be done both in the cloud and on the user's device (assuming appropriate computing power "training on-line"). This retrained network can then either be used in the

Cloud or in the device of the respective household. In order not to have a separate network in the cloud for every household, which is only requested occasionally, this is the case

If possible, operation in the device is probably preferable.

To improve and ensure the robustness of the retrained individual

Network, the already described method of the ensemble (both the general

Basic model evaluates each new image as well as the individual network). As long as both networks are similar within a certain framework (in terms of time and classification results).

deliver results, robust operation can be assumed. Does the individual network diverge from the

Basic model, so re-training is advisable. Several diverge at the same time

If individual networks of several (many) households differ from the respective basic model, an update of the basic model is advisable. The specialist literature speaks of “model drift” in such cases.

Another possibility for individualization would be not to individualize the basic model directly, but rather to create a significantly smaller network, which can easily be trained with fewer computing resources in the device in the respective household. As

The input data here is not directly the captured images, but rather the

Evaluation results of the basic model. Additionally, data such as day of the week,

Time, usage behavior (loading, unloading or starting the machine), etc. are taken into account in order to be able to calculate the probability of a sensible start based on these general conditions. This network can be based on a history of the described

Data is continuously re-trained or re-trained in order to keep the current data up to date

To be able to model and imitate usage behavior.

However, both options require the device to be networked in order to do this

To be able to monitor the behavior of the neural networks and assess their quality. A possible malfunction of the networks (e.g. model drift, see 0.) should be identified at an early stage in order to take appropriate countermeasures (e.g. a reset of an individualized system).

network to the basic model or update of the basic model).

As a third option, every household can be given the opportunity to have a certain amount of their own

To be able to carry out individualization - not least in order to avoid the need to take countermeasures in the event of faulty individual networks. Manual individualization by the respective household is therefore also possible with the implementation options described above.

Households could be given the opportunity to set preferred times for the automatic start or for the desired program end time (desired

day and time of the week). Also would be available in combination with one

It is conceivable for a photovoltaic system to link an automatic start to high electricity production. Since users may have concerns about having the dishwasher start automatically when no one is at home, an option is also conceivable that requires the presence of at least one household member. There are further sources of information (connection to PV system, local weather data, data from a...

SmartHomes for presence detection using motion detectors or the location of

Mobile devices, for example using “geo-fencing”).

The last, but probably easiest option would be to give the household a push

Send notification to mobile devices along with the current

Camera image to offer the options “Start” and “Move”. Choose one

If a household member selects the “Postpone” option, the flush start will be set by a configurable time

Time interval postponed and asked again later via push notification.

Claims (13)

Patent claims 1. Method (200) for controlling a washing device (100), wherein the washing device (100) comprises at least one camera unit (140) for detecting an interior (107) of the washing device (100), the method (200) comprising the following steps (205 , 210, 215): reading in (205) a camera signal (145) from the camera unit (140), the camera signal (145) representing an image of the interior (107) captured by the camera unit (140); Evaluating (210) the image using the camera signal (145) and using a model (300) through which at least two different loading states of the interior (107) can be distinguished from the image of the interior (107); and outputting (215) a control signal (150) for controlling a washing program for washing objects (120) in the interior (107) if the image represents a loading state of the interior (107) of at least one starting load quantity arranged in the interior (107). of items to be washed (120). 2. Method (200) according to claim 1, wherein in the step (210) of evaluating the image is assigned to a first image category or a second image category, the first image category corresponding to a not full loading state of the interior (107) and the second image category to the full one corresponds to the loading condition. 3. Method (200) according to one of the preceding claims, wherein in step (205) of reading an image information signal (305) is read in, the image information signal (305) representing at least one additional piece of information about the image, wherein in step (210). Evaluating the image using the image information signal (305). 4. Method (200) according to one of the preceding claims, wherein in step (210) of evaluating the image is discarded if a predetermined image parameter is not met. 5. Method (200) according to one of the preceding claims, wherein in the step (205) of reading in, a user signal (310) is read in, which represents an input from a user at a user interface (315), wherein in the step (215) of outputting Control signal (150) is output using the user signal (310). 6. Method (200) according to one of the preceding claims, wherein in the step (205) of reading, a sensor signal (320) is read from a sensor interface (324) to a sensor (328), the sensor signal (320) being at least one of the Sensor (328) represents environmental parameters of an environment of the washing device (100), wherein in step (215) of output, the control signal (150) is output using the sensor signal (320). 7. Method (200) according to one of the preceding claims, wherein in step (215) of outputting the control signal (150) is output to a user interface (315) to a user, the flushing program being controlled in response to a response signal (330), wherein the response signal (330) represents an input from the user on the user interface (315). 8. Method (400) for training a model (300) for controlling a flushing device (100), the method (400) having the following steps (405, 410): reading in (405) of a first image data set, which contains a plurality of first images an interior (107) of a washing device (100) which is assigned to a first image category and reading in a second image data set which represents a plurality of second images of an interior (107) of a washing device (100) which are assigned to a second image category; and adjusting (410) the model (300) such that the first images input to the model (300) are assigned to the first image category and the second images input to the model (300) are assigned to the second image category to create the model (300). train. 9. Method (400) according to claim 8, with a step (415) of retraining the model (300) after the step (410) of adapting, wherein in the step (415) of retraining at least one further first image data set is read in, which is a represents a plurality of further first images of the interior (107) which are assigned to the first image category and at least one further second image data set is read in, which represents a plurality of further second images of the interior (107) which are assigned to the second image category, wherein the model (300) is adapted in such a way that the further first images are assigned to the first image category and the further second images are assigned to the second image category in order to retrain the basic model. 10. Device (105) which is designed to carry out the steps (205, 210, 215; 405, 410) of one of the methods (200; 400) according to one of the preceding claims in corresponding units. 11. Washing device (100) with a device (105) according to claim 10 and with at least one camera unit (140) for detecting an interior (107) of the washing device (100). 12. Computer program product with program code for carrying out one of the methods (200; 400) according to one of the preceding claims 1 to 9, if the computer program product is executed on a device (105) according to claim 10. 13. Machine-readable storage medium on which the computer program according to claim 12 is stored.